Skip to content

Commit

Permalink
update connector file to match the the project Readme.
Browse files Browse the repository at this point in the history
  • Loading branch information
ajhunyady committed Dec 27, 2023
1 parent fc56358 commit 8993ca9
Showing 1 changed file with 33 additions and 7 deletions.
40 changes: 33 additions & 7 deletions embeds/connectors/inbound/http.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
# Fluvio HTTP Inbound Connector

Read HTTP Responses given input HTTP request configuration options and `interval` x and produces them to Fluvio topics.
Read HTTP Responses given input HTTP request configuration options and produce them
to Fluvio topics.

Supports both polling of atomic endpoints and streaming of HTTP responses when given the `stream` configuration option.
When polling, an `interval` configuration option is accepted. When streaming a `delimiter` configuration option, which
defaults to `'\n'`, is accepted.

Supports HTTP/1.0, HTTP/1.1, HTTP/2.0 protocols.

Expand All @@ -18,6 +23,8 @@ Tutorial for [HTTP to SQL Pipeline](https://www.fluvio.io/docs/tutorials/data-pi
| user-agent | "fluvio/http-source 0.1.0" | String | Request user-agent |
| output_type | text | String | `text` = UTF-8 String Output, `json` = UTF-8 JSON Serialized String |
| output_parts | body | String | `body` = body only, `full` = all status, header and body parts |
| stream | false | bool | Flag to indicate HTTP streaming mode |
| delimiter | '\n' | String | Delimiter to separate records when producing from an HTTP streaming endpoint |

#### Record Type Output
| Matrix | Output |
Expand All @@ -33,10 +40,10 @@ Tutorial for [HTTP to SQL Pipeline](https://www.fluvio.io/docs/tutorials/data-pi
This is an example of simple connector config file:

```yaml
# sample-config.yaml
# config-example.yaml
apiVersion: 0.1.0
meta:
version: 0.2.5
version: 0.3.0
name: cat-facts
type: http-source
topic: cat-facts
Expand All @@ -63,10 +70,10 @@ The produced record in Fluvio topic will be:
Fluvio HTTP Source Connector supports Secrets in the `endpoint` and in the `headers` parameters:

```yaml
# sample-config.yaml
# config-example.yaml
apiVersion: 0.1.0
meta:
version: 0.2.5
version: 0.3.0
name: cat-facts
type: http-source
topic: cat-facts
Expand All @@ -89,10 +96,10 @@ Fluvio HTTP Source Connector supports [Transformations](https://www.fluvio.io/do
The previous example can be extended to add extra transformations to outgoing records:
```yaml
# sample-config.yaml
# config-example.yaml
apiVersion: 0.1.0
meta:
version: 0.2.5
version: 0.3.0
name: cat-facts
type: http-source
topic: cat-facts
Expand Down Expand Up @@ -123,3 +130,22 @@ Now produced records will have a different shape, for example:
```

Read more about [JSON to JSON transformations](https://www.fluvio.io/smartmodules/certified/jolt/).

### Streaming Mode

Provide the `stream` configuration option to enable streaming mode with `delimiter` to determine how the incoming records are separated.

```yaml
# config-example.yaml
apiVersion: 0.1.0
meta:
version: 0.3.0
name: wiki-updates
type: http-source
topic: wiki-updates
http:
endpoint: "https://stream.wikimedia.org/v2/stream/recentchange"
method: GET
stream: true
delimiter: "\n\n"
```

0 comments on commit 8993ca9

Please sign in to comment.