Skip to content

Commit

Permalink
[create-pull-request] automated change (#660)
Browse files Browse the repository at this point in the history
Co-authored-by: ozgrakkurt <ozgrakkurt@users.noreply.github.com>
  • Loading branch information
github-actions[bot] and ozgrakkurt authored Aug 4, 2023
1 parent 9b9cfbb commit b3e4bee
Showing 1 changed file with 51 additions and 1 deletion.
52 changes: 51 additions & 1 deletion embeds/connectors/outbound/sql.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ meta:
sql:
url: ${{ secrets.DATABASE_URL }}
```
## Usage Example
## Insert Usage Example
Let's look at the example of the connector with one transformation named [infinyon/json-sql](https://github.com/infinyon/fluvio-connectors/blob/main/smartmodules/json-sql/README.md). The transformation takes
records in JSON format and creates SQL insert operation to `topic_message` table. The value from `device.device_id`
JSON field will be put to `device_id` column and the entire json body to `record` column.
Expand Down Expand Up @@ -139,3 +139,53 @@ cdk deploy shutdown --config connector-config.yaml
After you run the connector you will see records in your database table.

See more in our [Build MQTT to SQL Pipeline](https://www.fluvio.io/docs/tutorials/mqtt-to-sql/) and [Build HTTP to SQL Pipeline](https://www.fluvio.io/docs/tutorials/data-pipeline/) tutorials.

## Upsert Usage Example

Every step would be same except the connector config and the behavior of the connector after deployment.

We have a `operation` parameter which defaults to `insert` but we can pass `upsert` here to specify we want to do an upsert operation.

Upsert additionaly takes an `unique-columns` argument. `unique-columns` specifies the list indices or column names to check for uniqueness of a record.
If a record with same value in `unique-columns` exists in the database, it will be updated. If no record exists with same value, the given record will
be inserted.

Connector configuration file for upsert (assuming `device_id` is a unique column or an index in the database):

```yaml
# connector-config.yaml
apiVersion: 0.1.0
meta:
version: 0.3.1
name: json-sql-connector
type: sql-sink
topic: sql-topic
create-topic: true
secrets:
- name: DATABASE_URL
sql:
url: ${{ secrets.DATABASE_URL }}
transforms:
- uses: infinyon/json-sql
with:
mapping:
operation: "upsert"
unique-columns:
- "device_id"
table: "topic_message"
map-columns:
"device_id":
json-key: "device.device_id"
value:
type: "int"
default: "0"
required: true
"record":
json-key: "$"
value:
type: "jsonb"
required: true
```

See more about upsert in [our blog](https://infinyon.com/blog/2023/07/sql-upsert/).
Note: the blog doesn't use `json-sql` smartmodule and has hardcoded records for demonstration. `sql-connector` is intended to be used with `json-sql`.

0 comments on commit b3e4bee

Please sign in to comment.