Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Connector-V2] Support multi-table sink feature for paimon #5652 #6449

Merged
merged 8 commits into from
Mar 19, 2024
Merged
82 changes: 59 additions & 23 deletions docs/en/connector-v2/sink/Paimon.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,48 +4,84 @@

## Description

Write data to Apache Paimon.
Sink connector for Apache Paimon. It can support cdc mode 、auto create table.

## Key features

- [x] [exactly-once](../../concept/connector-v2-features.md)

## Options

| name | type | required | default value |
|----------------|--------|----------|---------------|
| warehouse | String | Yes | - |
| database | String | Yes | - |
| table | String | Yes | - |
| hdfs_site_path | String | No | - |
| name | type | required | default value | Description |
|------------------|--------|----------|------------------------------|---------------------------------|
| warehouse | String | Yes | - | Paimon warehouse path |
| database | String | Yes | - | The database you want to access |
| table | String | Yes | - | The table you want to access |
| hdfs_site_path | String | No | - | |
| schema_save_mode | Enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | The schema save mode |
| data_save_mode | Enum | no | APPEND_DATA | The data save mode |

### warehouse [string]

Paimon warehouse path

### database [string]
## Examples

The database you want to access
### Single table

### table [String]
```hocon
env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 5000
}

The table you want to access
source {
Mysql-CDC {
base-url = "jdbc:mysql://127.0.0.1:3306/seatunnel"
username = "root"
password = "******"
table-names = ["seatunnel.role"]
}
}

## Examples
transform {
}

```hocon
sink {
Paimon {
warehouse = "/tmp/paimon"
database = "default"
table = "st_test"
catalog_name="seatunnel_test"
warehouse="file:///tmp/seatunnel/paimon/hadoop-sink/"
database="seatunnel"
table="role"
}
}
```

## Changelog
### Multiple table

```hocon
env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 5000
}

source {
Mysql-CDC {
base-url = "jdbc:mysql://127.0.0.1:3306/seatunnel"
username = "root"
password = "******"
table-names = ["seatunnel.role","seatunnel.user","galileo.Bucket"]
}
}

### next version
transform {
}

- Add Paimon Sink Connector
sink {
Paimon {
catalog_name="seatunnel_test"
warehouse="file:///tmp/seatunnel/paimon/hadoop-sink/"
database="${database_name}"
table="${table_name}"
}
}
```

87 changes: 87 additions & 0 deletions docs/zh/connector-v2/sink/Paimon.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# Paimon

> Paimon 数据连接器

## 描述

Apache Paimon数据连接器。支持cdc写以及自动建表。

## 主要特性

- [x] [exactly-once](../../concept/connector-v2-features.md)

## 连接器选项

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There seem to be some problems with the format

Copy link
Contributor Author

@dailai dailai Mar 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There seem to be some problems with the format

I've already previewed it and there don't seem to be any formatting issues here.

截屏2024-03-12 13 37 46

| 名称 | 类型 | 是否必须 | 默认值 | 描述 |
|------------------|--------|------|------------------------------|--------------------|
| warehouse | String | Yes | - | Paimon warehouse路径 |
| database | String | Yes | - | 数据库名称 |
| table | String | Yes | - | 表名 |
| hdfs_site_path | String | No | - | |
| schema_save_mode | Enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | schema保存模式 |
| data_save_mode | Enum | no | APPEND_DATA | 数据保存模式 |

## 示例

### 单表

```hocon
env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 5000
}

source {
Mysql-CDC {
base-url = "jdbc:mysql://127.0.0.1:3306/seatunnel"
username = "root"
password = "******"
table-names = ["seatunnel.role"]
}
}

transform {
}

sink {
Paimon {
catalog_name="seatunnel_test"
warehouse="file:///tmp/seatunnel/paimon/hadoop-sink/"
database="seatunnel"
table="role"
}
}
```

### 多表

```hocon
env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 5000
}

source {
Mysql-CDC {
base-url = "jdbc:mysql://127.0.0.1:3306/seatunnel"
username = "root"
password = "******"
table-names = ["seatunnel.role","seatunnel.user","galileo.Bucket"]
}
}

transform {
}

sink {
Paimon {
catalog_name="seatunnel_test"
warehouse="file:///tmp/seatunnel/paimon/hadoop-sink/"
database="${database_name}"
table="${table_name}"
}
}
```

5 changes: 5 additions & 0 deletions seatunnel-connectors-v2/connector-paimon/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,11 @@
</properties>

<dependencies>
<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>connector-common</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>org.apache.paimon</groupId>
Expand Down
Loading
Loading