[Feature][connector-v2] Add Sensorsdata Connector Support #9323 #9432

zhangqs0205 · 2025-06-12T08:01:12Z

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config

zhilinli123 · 2025-06-12T10:56:09Z

docs/en/connector-v2/sink/SensorsData.md

+| name                      | type    | required | default value |
+|---------------------------|---------|----------|---------------|
+| server_url                | string  | yes      | -             |
+| bulk_size                 | int     | no       | 50            |
+| max_cache_row_size        | int     | no       | 0             |
+| consumer                  | string  | no       | -             |
+| entity_name               | string  | yes      | users         |
+| record_type               | string  | yes      | users         |
+| schema                    | string  | yes      | users         |
+| distinct_id_column        | string  | yes      | -             |
+| identity_fields           | array   | yes      | -             |
+| property_fields           | array   | yes      | -             |


Reference connector v2 doc format
https://seatunnel.apache.org/docs/2.3.11/connector-v2/source/Mysql

The content in the "Description" column is relatively complex, it is explained separately below the table. The format of Jdbc.md was referenced.

@zhilinli123 Please review

.../apache/seatunnel/connectors/seatunnel/sensorsdata/sdk/config/SensorsDataSDKSinkOptions.java

...nnector-v2-e2e/connector-sensorsdata-e2e/src/test/resources/fake_to_sensorsdata_details.conf

Copilot

Pull Request Overview

This PR introduces initial support for the SensorsData connector (v2), including module setup, SDK-based writer implementation, distribution registration, end-to-end test scaffold, and documentation.

Added a new connector-sensorsdata module with config, sink, writer, and error handling classes
Registered the connector in seatunnel-dist, plugin mappings, and labeler config
Provided E2E test scaffold and comprehensive user documentation

Reviewed Changes

Copilot reviewed 53 out of 53 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-sensorsdata-e2e/README.md	Added E2E test stub README
seatunnel-dist/pom.xml	Added `connector-sensorsdata` as provided dependency
seatunnel-connectors-v2/pom.xml	Included `connector-sensorsdata` module
seatunnel-connectors-v2/connector-sensorsdata/src/test/.../SensorsDataSDKFactoryTest.java	Added basic factory optionRule unit test
seatunnel-connectors-v2/connector-sensorsdata/src/main/java/.../state/*.java	Added state and commit-info classes
seatunnel-connectors-v2/connector-sensorsdata/src/main/java/.../sdk/sink/SensorsDataSDKWriter.java	Implemented SDK-based SinkWriter
seatunnel-connectors-v2/connector-sensorsdata/src/main/java/.../sdk/sink/SensorsDataSDKSinkFactory.java	Hooked into SeaTunnel TableSinkFactory
seatunnel-connectors-v2/connector-sensorsdata/src/main/java/.../sdk/sink/SensorsDataSDKSink.java	Implemented `SeaTunnelSink` wrapper
seatunnel-connectors-v2/connector-sensorsdata/src/main/java/.../sdk/exception/*.java	Defined connector-specific exceptions and error codes
seatunnel-connectors-v2/connector-sensorsdata/src/main/java/.../sdk/config/*.java	Added sink options and config classes
seatunnel-connectors-v2/connector-sensorsdata/pom.xml	New Maven POM for connector-sensorsdata module
plugin-mapping.properties	Registered `seatunnel.sink.SensorsData` mapping
config/plugin_config	Listed `connector-sensorsdata`
.github/workflows/labeler/label-scope-conf.yml	Added labeler rules for connector-sensorsdata
docs/en/connector-v2/sink/SensorsData.md	Written connector user guide
docs/en/connector-v2/changelog/connector-sensorsdata.md	Stubbed changelog for connector-sensorsdata

Comments suppressed due to low confidence (4)

seatunnel-connectors-v2/connector-sensorsdata/src/test/java/org/apache/seatunnel/connectors/seatunnel/sensorsdata/sdk/SensorsDataSDKFactoryTest.java:29

[nitpick] This test only verifies optionRule is non-null; consider adding tests for write logic, record-type dispatching, and error handling to ensure full coverage.

Assertions.assertNotNull((new SensorsDataSDKSinkFactory()).optionRule());

docs/en/connector-v2/sink/SensorsData.md:28

The default for 'consumer' in code is 'batch'; update the default value in the docs table to 'batch' for consistency.

| consumer                  | string  | no       | -             |

seatunnel-e2e/seatunnel-connector-v2-e2e/connector-sensorsdata-e2e/README.md:1

[nitpick] Rename the header to '# SensorsData connector E2E test' to match the module name and avoid confusion.

# SDH connector E2E test

docs/en/connector-v2/sink/SensorsData.md:127

The type for 'skip_error_record' should be [boolean], not [string], to align with the code.

### skip_error_record[string]

...ava/org/apache/seatunnel/connectors/seatunnel/sensorsdata/sdk/sink/SensorsDataSDKWriter.java

docs/en/connector-v2/sink/SensorsData.md

zhangqs0205 · 2025-06-13T07:14:11Z

I have a question and would appreciate your advice:
As seen in this PR, two modules are submitted:

seatunnel-connector-v2/connector-sensorsdata
seatunnel-format/seatunnel-format-sensorsdata

Here, connector-sensorsdata depends on seatunnel-format-sensorsdata.

Our scenario is:

On our private branch, besides connector-sensorsdata, we have several transform modules. We extracted common dependencies for these modules into seatunnel-format-sensorsdata.

For the code submitted to the community, since seatunnel-format-sensorsdata is only used by connector-sensorsdata, it seems more reasonable to consolidate them into a single module. However, if we do this, each transform module in our private branch would need to duplicate these base classes, which isn’t practical.

My questions are:

Should we submit seatunnel-format-sensorsdata as an independent module to the community?
If it shouldn’t be a separate module, do you have any suggestions?
One idea I have:
Release seatunnel-format-sensorsdata as a separate Maven Central artifact.
But this artifact would be specifically for Seatunnel since it depends on seatunnel-api.

@zhilinli123

Copilot

Pull Request Overview

This PR introduces the new SensorsData connector (v2) for SeaTunnel, implementing end-to-end support in code, configuration, documentation, and CI.

Adds connector-sensorsdata as a Maven module and registers it in distribution and plugin mappings
Implements core connector classes: config, factory, sink, writer, state, and exceptions
Provides unit test skeleton, detailed documentation, changelog, and updates CI workflows and plugin configs

Reviewed Changes

Copilot reviewed 54 out of 54 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
seatunnel-dist/pom.xml	Added `connector-sensorsdata` dependency
seatunnel-connectors-v2/pom.xml	Registered `connector-sensorsdata` module
connector-sensorsdata/src/main/java/...	Implemented config, factory, sink, writer, state, exception classes
connector-sensorsdata/src/test/java/...	Added basic `SensorsDataSDKFactoryTest`
plugin-mapping.properties	Mapped `SensorsData` to `connector-sensorsdata`
docs/en/connector-v2/sink/SensorsData.md	Created connector documentation and examples
docs/en/connector-v2/changelog/connector-sensorsdata.md	Added changelog scaffold
config/plugin_config	Added `connector-sensorsdata` entry
.github/workflows/labeler/label-scope-conf.yml	Added labeling rules for `connector-sensorsdata`
.github/workflows/backend.yml	Introduced CI job for sensorsdata integration tests

Comments suppressed due to low confidence (4)

docs/en/connector-v2/sink/SensorsData.md:28

The default value for consumer in code is "batch", but the docs show -. Please update the default in the table to batch to match the implementation.

| consumer                  | string  | no       | -             |

seatunnel-connectors-v2/connector-sensorsdata/src/test/java/org/apache/seatunnel/connectors/seatunnel/sensorsdata/sdk/SensorsDataSDKFactoryTest.java:29

[nitpick] The test only asserts that optionRule() is non-null. To improve coverage, add assertions that required options (e.g., SERVER_URL) are present in the returned rule.

        Assertions.assertNotNull((new SensorsDataSDKSinkFactory()).optionRule());

.github/workflows/backend.yml:1460

The CI step refers to :connector-sensorsdata-e2e, but no such module was added. Either add the corresponding e2e test module or remove/update this CI job reference.

          ./mvnw -B -T 1 verify -DskipUT=true -DskipIT=false -D"license.skipAddThirdParty"=true -D"skip.ui"=true --no-snapshot-updates -pl :connector-sensorsdata-e2e -am -Pci

seatunnel-connectors-v2/connector-sensorsdata/src/main/java/org/apache/seatunnel/connectors/seatunnel/sensorsdata/sdk/exception/SensorsDataConnectorErrorCode.java:24

This enum constant UNKNOWN_RECORD_TYPE is never referenced in the code; consider removing it or aligning with used error codes to avoid dead entries.

    UNKNOWN_RECORD_TYPE("SENSORS_DATA-02", "Unknown record type"),

zhilinli123 · 2025-06-19T02:53:24Z

Author

I have a question and would appreciate your advice: As seen in this PR, two modules are submitted:
seatunnel-connector-v2/connector-sensorsdata
seatunnel-format/seatunnel-format-sensorsdata
Here, connector-sensorsdata depends on seatunnel-format-sensorsdata.

Our scenario is:

On our private branch, besides connector-sensorsdata, we have several transform modules. We extracted common dependencies for these modules into seatunnel-format-sensorsdata.

For the code submitted to the community, since seatunnel-format-sensorsdata is only used by connector-sensorsdata, it seems more reasonable to consolidate them into a single module. However, if we do this, each transform module in our private branch would need to duplicate these base classes, which isn’t practical.

My questions are:

Should we submit seatunnel-format-sensorsdata as an independent module to the community?

If it shouldn’t be a separate module, do you have any suggestions?
One idea I have:
Release seatunnel-format-sensorsdata as a separate Maven Central artifact.
But this artifact would be specifically for Seatunnel since it depends on seatunnel-api.

@zhilinli123

PTAL : @hailin0 @Hisoka-X

Hisoka-X · 2025-06-19T03:21:56Z

Please move the code of seatunnel-format-sensorsdata to connector-sensorsdata

zhangqs0205 · 2025-06-24T02:18:07Z

Please move the code of seatunnel-format-sensorsdata to connector-sensorsdata

Indeed, based on the currently submitted code, moving seatunnel-format-sensorsdata into connector-sensorsdata appears to be correct.

However, in our internal branch, several other transforms also depend on seatunnel-format-sensorsdata. As a common dependency, keeping it as a separate module would be easier to maintain.

If we move it, merging new community code into our internal branch in the future might become more difficult. Do you have any good suggestions?

@Hisoka-X

Hisoka-X · 2025-06-24T03:26:43Z

Please move the code of seatunnel-format-sensorsdata to connector-sensorsdata

Indeed, based on the currently submitted code, moving seatunnel-format-sensorsdata into connector-sensorsdata appears to be correct.

However, in our internal branch, several other transforms also depend on seatunnel-format-sensorsdata. As a common dependency, keeping it as a separate module would be easier to maintain.

If we move it, merging new community code into our internal branch in the future might become more difficult. Do you have any good suggestions?

@Hisoka-X

You can add the dependency connector-sensorsdata to your own transform module pom.

zhangqs0205 · 2025-07-02T08:15:33Z

Please move the code of seatunnel-format-sensorsdata to connector-sensorsdata

Indeed, based on the currently submitted code, moving seatunnel-format-sensorsdata into connector-sensorsdata appears to be correct.
However, in our internal branch, several other transforms also depend on seatunnel-format-sensorsdata. As a common dependency, keeping it as a separate module would be easier to maintain.
If we move it, merging new community code into our internal branch in the future might become more difficult. Do you have any good suggestions?
@Hisoka-X

You can add the dependency connector-sensorsdata to your own transform module pom.

Moved

Hisoka-X · 2025-07-02T13:05:26Z

seatunnel-e2e/seatunnel-connector-v2-e2e/connector-sensorsdata-e2e/README.md

@@ -0,0 +1,2 @@
+# SDH connector E2E test


Please remove this.

Hisoka-X · 2025-07-02T13:06:22Z

.github/workflows/backend.yml

@@ -1435,3 +1435,28 @@ jobs:
          ./mvnw -B -T 1 verify -DskipUT=true -DskipIT=false -D"license.skipAddThirdParty"=true -D"skip.ui"=true --no-snapshot-updates -pl :connector-redis-e2e -am -Pci
        env:
          MAVEN_OPTS: -Xmx4096m
+


Please update

seatunnel/tools/update_modules_check/update_modules_check.py

Line 146 in 86c5ae4

modules_arr.remove("connector-jdbc-e2e")

Hisoka-X · 2025-07-03T11:59:17Z

wating test case passed.

zhangqs0205 · 2025-08-13T08:50:49Z

wating test case passed.

passed

Hisoka-X

Thanks @zhangqs0205

1 1 1 1 1 update doc [Improve][Csv] support configurable CSV delimiter in file connector (apache#9660) [Fix][Doc] Fix rest api finished-jobs doc miss `SAVEPOINT_DONE` (apache#9676) [Fix] [connector-jdbc] prevent precision loss in Float to BigDecimal conversion (apache#9670) [Feature][Connector-File-Hadoop]Support multi table sink feature for HdfsFile (apache#9651) [HotFix][CI] The Some CI subtasks are not executed (apache#9684) [Docs][Connector-Iceberg] Update Iceberg doc with support S3 Tables rest catalogs (apache#9686) [Docs] update jdbc related document (apache#9679) [Fix][Connector-V2] Fix misleading parameter name in DorisStreamLoad (apache#9685) [Feature] [connector-file] Add configurable sheet_max_rows support for Excel sink connector (apache#9668) [fix][connectors-v2] repeated commit cause task exceptions (apache#9665) [Feature][connector-v2] Add Sensorsdata Connector Support apache#9323 (apache#9432) [Improve][API] Optimize the enumerator API semantics and reduce lock calls at the connector level (apache#9671) [Hotfix][Zeta] Fix custom https port configuration not working (apache#9705) [Improve][Doc] Add SeaTunnel tools into Readme (apache#9707) [Improve][Core] Unify the aws-sdk-v2 version to 2.31.30 (apache#9698) [Feature][Transform-V2] Add `TRIM_SCALE` function for sql transform (apache#9700) [Improve][Core] Update apache common to apache common lang3 (apache#9694) Co-authored-by: litiliu <[email protected]>

…apache#9432)

王琛 and others added 5 commits June 11, 2025 14:37

Add SensorsData Connector

06d27de

SensorsData Sink 添加参数 instant_events，用于支持上报即时事件

97ecfd5

Add SensorsData Connector

4a5e37c

[SDH-18165] 兼容 distinctId 字段没有数据的场景, 使用 identites 数据补全

1bd4bd2

[SDH-18196] sink 用户支持 profile_unset

6c2bf02

github-actions bot added document CI&CD core SeaTunnel core module connectors-v2 Transform-v2 e2e format labels Jun 12, 2025

zhilinli123 reviewed Jun 12, 2025

View reviewed changes

.../apache/seatunnel/connectors/seatunnel/sensorsdata/sdk/config/SensorsDataSDKSinkOptions.java Outdated Show resolved Hide resolved

zhilinli123 reviewed Jun 12, 2025

View reviewed changes

...nnector-v2-e2e/connector-sensorsdata-e2e/src/test/resources/fake_to_sensorsdata_details.conf Outdated Show resolved Hide resolved

hailin0 requested a review from Copilot June 13, 2025 01:44

Copilot AI reviewed Jun 13, 2025

View reviewed changes

nielifeng requested a review from Copilot June 16, 2025 01:38

This comment was marked as outdated.

Sign in to view

zhangqs0205 force-pushed the add_connector branch from 385af94 to 4ffbdc5 Compare June 16, 2025 10:47

nielifeng requested a review from Copilot June 17, 2025 02:02

Copilot AI reviewed Jun 17, 2025

View reviewed changes

zhangqs0205 marked this pull request as ready for review June 19, 2025 01:45

github-actions bot removed the format label Jun 30, 2025

Add SensorsData Connector

f8a5205

zhangqs0205 force-pushed the add_connector branch from 1eb5095 to f8a5205 Compare July 1, 2025 14:36

Hisoka-X reviewed Jul 2, 2025

View reviewed changes

Add SensorsData Connector

f532785

zhangqs0205 requested a review from Hisoka-X July 3, 2025 02:50

Add SensorsData Connector

439b396

zhangqs0205 requested a review from zhilinli123 August 13, 2025 08:49

Hisoka-X approved these changes Aug 14, 2025

View reviewed changes

github-actions bot added approved reviewed labels Aug 14, 2025

corgy-w approved these changes Aug 14, 2025

View reviewed changes

corgy-w merged commit bb53f77 into apache:dev Aug 14, 2025
6 checks passed

HighandLight pushed a commit to HighandLight/seatunnel that referenced this pull request Aug 23, 2025

[Feature][connector-v2] Add Sensorsdata Connector Support apache#9323 (…

d47f244

…apache#9432)

[Feature][connector-v2] Add Sensorsdata Connector Support #9323 #9432

[Feature][connector-v2] Add Sensorsdata Connector Support #9323 #9432

Uh oh!

Conversation

zhangqs0205 commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

zhilinli123 Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

zhangqs0205 Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

zhangqs0205 Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhangqs0205 commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Our scenario is:

My questions are:

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

zhilinli123 commented Jun 19, 2025

Our scenario is:

My questions are:

Uh oh!

Hisoka-X commented Jun 19, 2025

Uh oh!

zhangqs0205 commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hisoka-X commented Jun 24, 2025

Uh oh!

zhangqs0205 commented Jul 2, 2025

Uh oh!

Hisoka-X Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Hisoka-X Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Hisoka-X commented Jul 3, 2025

Uh oh!

zhangqs0205 commented Aug 13, 2025

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zhangqs0205 commented Jun 12, 2025 •

edited

Loading

zhangqs0205 commented Jun 13, 2025 •

edited

Loading

zhangqs0205 commented Jun 24, 2025 •

edited

Loading