Skip to content

Commit 728a16f

Browse files
xiaochen-zhoulitiliu
xiaochen-zhou
and
litiliu
committed
update code
1 1 1 1 1 update doc [Improve][Csv] support configurable CSV delimiter in file connector (apache#9660) [Fix][Doc] Fix rest api finished-jobs doc miss `SAVEPOINT_DONE` (apache#9676) [Fix] [connector-jdbc] prevent precision loss in Float to BigDecimal conversion (apache#9670) [Feature][Connector-File-Hadoop]Support multi table sink feature for HdfsFile (apache#9651) [HotFix][CI] The Some CI subtasks are not executed (apache#9684) [Docs][Connector-Iceberg] Update Iceberg doc with support S3 Tables rest catalogs (apache#9686) [Docs] update jdbc related document (apache#9679) [Fix][Connector-V2] Fix misleading parameter name in DorisStreamLoad (apache#9685) [Feature] [connector-file] Add configurable sheet_max_rows support for Excel sink connector (apache#9668) [fix][connectors-v2] repeated commit cause task exceptions (apache#9665) [Feature][connector-v2] Add Sensorsdata Connector Support apache#9323 (apache#9432) [Improve][API] Optimize the enumerator API semantics and reduce lock calls at the connector level (apache#9671) [Hotfix][Zeta] Fix custom https port configuration not working (apache#9705) [Improve][Doc] Add SeaTunnel tools into Readme (apache#9707) [Improve][Core] Unify the aws-sdk-v2 version to 2.31.30 (apache#9698) [Feature][Transform-V2] Add `TRIM_SCALE` function for sql transform (apache#9700) [Improve][Core] Update apache common to apache common lang3 (apache#9694) Co-authored-by: litiliu <[email protected]>
1 parent ad697a8 commit 728a16f

File tree

226 files changed

+8176
-1174
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

226 files changed

+8176
-1174
lines changed

‎.github/workflows/backend.yml

Lines changed: 36 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -480,7 +480,7 @@ jobs:
480480
matrix:
481481
java: [ '8', '11' ]
482482
os: [ 'ubuntu-latest' ]
483-
timeout-minutes: 150
483+
timeout-minutes: 200
484484
steps:
485485
- uses: actions/checkout@v2
486486
- name: Set up JDK ${{ matrix.java }}
@@ -1182,7 +1182,7 @@ jobs:
11821182

11831183
kudu-connector-it:
11841184
needs: [ changes, sanity-check ]
1185-
if: needs.changes.outputs.api == 'true' || contains(needs.changes.outputs.it-modules, 'connector-kudu-e2e')
1185+
if: needs.changes.outputs.api == 'true' || needs.changes.outputs.engine == 'true' || contains(needs.changes.outputs.it-modules, 'connector-kudu-e2e')
11861186
runs-on: ${{ matrix.os }}
11871187
strategy:
11881188
matrix:
@@ -1207,7 +1207,7 @@ jobs:
12071207

12081208
amazonSqs-connector-it:
12091209
needs: [ changes, sanity-check ]
1210-
if: needs.changes.outputs.api == 'true' || contains(needs.changes.outputs.it-modules, 'connector-amazonsqs-e2e')
1210+
if: needs.changes.outputs.api == 'true' || needs.changes.outputs.engine == 'true' || contains(needs.changes.outputs.it-modules, 'connector-amazonsqs-e2e')
12111211
runs-on: ${{ matrix.os }}
12121212
strategy:
12131213
matrix:
@@ -1232,7 +1232,7 @@ jobs:
12321232

12331233
kafka-connector-it:
12341234
needs: [ changes, sanity-check ]
1235-
if: needs.changes.outputs.api == 'true' || contains(needs.changes.outputs.it-modules, 'connector-kafka-e2e')
1235+
if: needs.changes.outputs.api == 'true' || needs.changes.outputs.engine == 'true' || contains(needs.changes.outputs.it-modules, 'connector-kafka-e2e')
12361236
runs-on: ${{ matrix.os }}
12371237
strategy:
12381238
matrix:
@@ -1257,7 +1257,7 @@ jobs:
12571257

12581258
rocketmq-connector-it:
12591259
needs: [ changes, sanity-check ]
1260-
if: needs.changes.outputs.api == 'true' || contains(needs.changes.outputs.it-modules, 'connector-rocketmq-e2e')
1260+
if: needs.changes.outputs.api == 'true' || needs.changes.outputs.engine == 'true' || contains(needs.changes.outputs.it-modules, 'connector-rocketmq-e2e')
12611261
runs-on: ${{ matrix.os }}
12621262
strategy:
12631263
matrix:
@@ -1283,7 +1283,7 @@ jobs:
12831283

12841284
doris-connector-it:
12851285
needs: [ changes, sanity-check ]
1286-
if: needs.changes.outputs.api == 'true' || contains(needs.changes.outputs.it-modules, 'connector-doris-e2e')
1286+
if: needs.changes.outputs.api == 'true' || needs.changes.outputs.engine == 'true' || contains(needs.changes.outputs.it-modules, 'connector-doris-e2e')
12871287
runs-on: ${{ matrix.os }}
12881288
strategy:
12891289
matrix:
@@ -1308,7 +1308,7 @@ jobs:
13081308

13091309
paimon-connector-it:
13101310
needs: [ changes, sanity-check ]
1311-
if: needs.changes.outputs.api == 'true' || contains(needs.changes.outputs.it-modules, 'connector-paimon-e2e')
1311+
if: needs.changes.outputs.api == 'true' || needs.changes.outputs.engine == 'true' || contains(needs.changes.outputs.it-modules, 'connector-paimon-e2e')
13121312
runs-on: ${{ matrix.os }}
13131313
strategy:
13141314
matrix:
@@ -1333,7 +1333,7 @@ jobs:
13331333

13341334
oracle-cdc-connector-it:
13351335
needs: [ changes, sanity-check ]
1336-
if: needs.changes.outputs.api == 'true' || contains(needs.changes.outputs.it-modules, 'connector-cdc-oracle-e2e')
1336+
if: needs.changes.outputs.api == 'true' || needs.changes.outputs.engine == 'true' || contains(needs.changes.outputs.it-modules, 'connector-cdc-oracle-e2e')
13371337
runs-on: ${{ matrix.os }}
13381338
strategy:
13391339
matrix:
@@ -1363,7 +1363,7 @@ jobs:
13631363
13641364
connector-file-local-it:
13651365
needs: [ changes, sanity-check ]
1366-
if: needs.changes.outputs.api == 'true' || contains(needs.changes.outputs.it-modules, 'connector-file-local-e2e')
1366+
if: needs.changes.outputs.api == 'true' || needs.changes.outputs.engine == 'true' || contains(needs.changes.outputs.it-modules, 'connector-file-local-e2e')
13671367
runs-on: ${{ matrix.os }}
13681368
strategy:
13691369
matrix:
@@ -1388,7 +1388,7 @@ jobs:
13881388

13891389
connector-file-sftp-it:
13901390
needs: [ changes, sanity-check ]
1391-
if: needs.changes.outputs.api == 'true' || contains(needs.changes.outputs.it-modules, 'connector-file-sftp-e2e')
1391+
if: needs.changes.outputs.api == 'true' || needs.changes.outputs.engine == 'true' || contains(needs.changes.outputs.it-modules, 'connector-file-sftp-e2e')
13921392
runs-on: ${{ matrix.os }}
13931393
strategy:
13941394
matrix:
@@ -1413,7 +1413,7 @@ jobs:
14131413

14141414
connector-redis-it:
14151415
needs: [ changes, sanity-check ]
1416-
if: needs.changes.outputs.api == 'true' || contains(needs.changes.outputs.it-modules, 'connector-redis-e2e')
1416+
if: needs.changes.outputs.api == 'true' || needs.changes.outputs.engine == 'true' || contains(needs.changes.outputs.it-modules, 'connector-redis-e2e')
14171417
runs-on: ${{ matrix.os }}
14181418
strategy:
14191419
matrix:
@@ -1435,3 +1435,28 @@ jobs:
14351435
./mvnw -B -T 1 verify -DskipUT=true -DskipIT=false -D"license.skipAddThirdParty"=true -D"skip.ui"=true --no-snapshot-updates -pl :connector-redis-e2e -am -Pci
14361436
env:
14371437
MAVEN_OPTS: -Xmx4096m
1438+
1439+
connector-sensorsdata-it:
1440+
needs: [ changes, sanity-check ]
1441+
if: needs.changes.outputs.api == 'true' || contains(needs.changes.outputs.it-modules, 'connector-sensorsdata-e2e')
1442+
runs-on: ${{ matrix.os }}
1443+
strategy:
1444+
matrix:
1445+
java: [ '8', '11' ]
1446+
os: [ 'ubuntu-latest' ]
1447+
timeout-minutes: 120
1448+
steps:
1449+
- uses: actions/checkout@v2
1450+
- name: Set up JDK ${{ matrix.java }}
1451+
uses: actions/setup-java@v3
1452+
with:
1453+
java-version: ${{ matrix.java }}
1454+
distribution: 'temurin'
1455+
cache: 'maven'
1456+
- name: free disk space
1457+
run: tools/github/free_disk_space.sh
1458+
- name: run sensorsdata connector integration test
1459+
run: |
1460+
./mvnw -B -T 1 verify -DskipUT=true -DskipIT=false -D"license.skipAddThirdParty"=true -D"skip.ui"=true --no-snapshot-updates -pl :connector-sensorsdata-e2e -am -Pci
1461+
env:
1462+
MAVEN_OPTS: -Xmx4096m

‎.github/workflows/labeler/label-scope-conf.yml

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -320,4 +320,11 @@ aerospike:
320320
- all:
321321
- changed-files:
322322
- any-glob-to-any-file: seatunnel-connectors-v2/connector-aerospike/**
323-
- all-globs-to-all-files: '!seatunnel-connectors-v2/connector-!(aerospike)/**'
323+
- all-globs-to-all-files: '!seatunnel-connectors-v2/connector-!(aerospike)/**'
324+
325+
sensorsdata:
326+
- all:
327+
- changed-files:
328+
- any-glob-to-any-file: seatunnel-connectors-v2/connector-sensorsdata/**
329+
- all-globs-to-all-files: '!seatunnel-connectors-v2/connector-!(sensorsdata)/**'
330+

‎README.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
- [Supported Connectors](#supported-connectors)
1616
- [Getting Started](#getting-started)
1717
- [Multimodal Data Integration](#multimodal-data-integration)
18+
- [Apache SeaTunnel Tools](#apache-seatunnel-tools)
1819
- [Use Cases](#use-cases)
1920
- [Code of Conduct](#code-of-conduct)
2021
- [Contributors](#contributors)
@@ -90,6 +91,10 @@ Choose your runtime execution engine:
9091

9192
- For integrating video, images, and binary files with SeaTunnel, please refer to the documentation for detailed instructions.
9293

94+
## Apache SeaTunnel Tools
95+
96+
SeaTunnel Tools provides a range of peripheral tools, including Apache SeaTunnel Mcp Server, etc,please refer to [SeaTunnel Tools](https://github.com/apache/seatunnel-tools).
97+
9398
## Use Cases
9499

95100
Explore real-world use cases of SeaTunnel, such as Weibo, Tencent Cloud, Sina, Sogou, and Yonghui Superstores. More use cases can be found on the [SeaTunnel Blog](https://seatunnel.apache.org/blog).

‎config/plugin_config

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,3 +95,4 @@ connector-sls
9595
connector-qdrant
9696
connector-typesense
9797
connector-cdc-opengauss
98+
connector-sensorsdata

‎docs/en/connector-v2/changelog/connector-sensorsdata.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
<details><summary> Change Log </summary>
2+
3+
| Change | Commit | Version |
4+
| --- | --- | --- |
5+
6+
7+
</details>

‎docs/en/connector-v2/sink/CosFile.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -53,8 +53,8 @@ To use this connector you need put hadoop-cos-{hadoop.version}-{version}.jar and
5353
| filename_time_format | string | no | "yyyy.MM.dd" | Only used when custom_filename is true |
5454
| file_format_type | string | no | "csv" | |
5555
| filename_extension | string | no | - | Override the default file name extensions with custom file name extensions. E.g. `.xml`, `.json`, `dat`, `.customtype` |
56-
| field_delimiter | string | no | '\001' | Only used when file_format is text |
57-
| row_delimiter | string | no | "\n" | Only used when file_format is `text`, `csv` and `json` |
56+
| field_delimiter | string | no | '\001' for text and ',' for csv | Only used when file_format is text and csv |
57+
| row_delimiter | string | no | "\n" | Only used when file_format is `text`, `csv` and `json` |
5858
| have_partition | boolean | no | false | Whether you need processing partitions. |
5959
| partition_by | array | no | - | Only used then have_partition is true |
6060
| partition_dir_expression | string | no | "${k0}=${v0}/${k1}=${v1}/.../${kn}=${vn}/" | Only used then have_partition is true |
@@ -65,6 +65,7 @@ To use this connector you need put hadoop-cos-{hadoop.version}-{version}.jar and
6565
| compress_codec | string | no | none | |
6666
| common-options | object | no | - | |
6767
| max_rows_in_memory | int | no | - | Only used when file_format is excel. |
68+
| sheet_max_rows | int | no | 1048576 | Only used when file_format is excel. |
6869
| sheet_name | string | no | Sheet${Random number} | Only used when file_format is excel. |
6970
| csv_string_quote_mode | enum | no | MINIMAL | Only used when file_format is csv. |
7071
| xml_root_tag | string | no | RECORDS | Only used when file_format is xml. |
@@ -134,7 +135,7 @@ Please note that, The final file name will end with the file_format's suffix, th
134135

135136
### field_delimiter [string]
136137

137-
The separator between columns in a row of data. Only needed by `text` file format.
138+
The separator between columns in a row of data. Only needed by `text` and `csv` file format.
138139

139140
### row_delimiter [string]
140141

@@ -203,6 +204,10 @@ Sink plugin common parameters, please refer to [Sink Common Options](../sink-com
203204

204205
When File Format is Excel,The maximum number of data items that can be cached in the memory.
205206

207+
### sheet_max_rows [int]
208+
209+
When file format is Excel, the maximum number of rows per sheet.
210+
206211
### sheet_name [string]
207212

208213
Writer the sheet of the workbook

‎docs/en/connector-v2/sink/DB2.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ semantics (using XA transaction guarantee).
6363
|-------------------------------------------|---------|----------|---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
6464
| url | String | Yes | - | The URL of the JDBC connection. Refer to a case: jdbc:db2://127.0.0.1:50000/dbname |
6565
| driver | String | Yes | - | The jdbc class name used to connect to the remote data source,<br/> if you use DB2 the value is `com.ibm.db2.jdbc.app.DB2Driver`. |
66-
| user | String | No | - | Connection instance user name |
66+
| username | String | No | - | Connection instance user name |
6767
| password | String | No | - | Connection instance password |
6868
| query | String | No | - | Use this sql write upstream input datas to database. e.g `INSERT ...`,`query` have the higher priority |
6969
| database | String | No | - | Use this `database` and `table-name` auto-generate sql and receive upstream input datas write to database.<br/>This option is mutually exclusive with `query` and has a higher priority. |
@@ -124,7 +124,7 @@ sink {
124124
jdbc {
125125
url = "jdbc:db2://127.0.0.1:50000/dbname"
126126
driver = "com.ibm.db2.jdbc.app.DB2Driver"
127-
user = "root"
127+
username = "root"
128128
password = "123456"
129129
query = "insert into test_table(name,age) values(?,?)"
130130
}
@@ -142,7 +142,7 @@ sink {
142142
jdbc {
143143
url = "jdbc:db2://127.0.0.1:50000/dbname"
144144
driver = "com.ibm.db2.jdbc.app.DB2Driver"
145-
user = "root"
145+
username = "root"
146146
password = "123456"
147147
# Automatically generate sql statements based on database table names
148148
generate_sink_sql = true
@@ -163,7 +163,7 @@ sink {
163163
driver = "com.ibm.db2.jdbc.app.DB2Driver"
164164
165165
max_retries = 0
166-
user = "root"
166+
username = "root"
167167
password = "123456"
168168
query = "insert into test_table(name,age) values(?,?)"
169169

0 commit comments

Comments
 (0)