diff --git a/embeds/connectors/inbound/http.md b/embeds/connectors/inbound/http.md index d06ddad5..9053d1fb 100644 --- a/embeds/connectors/inbound/http.md +++ b/embeds/connectors/inbound/http.md @@ -1,6 +1,11 @@ # Fluvio HTTP Inbound Connector -Read HTTP Responses given input HTTP request configuration options and `interval` x and produces them to Fluvio topics. +Read HTTP Responses given input HTTP request configuration options and produce them +to Fluvio topics. + +Supports both polling of atomic endpoints and streaming of HTTP responses when given the `stream` configuration option. +When polling, an `interval` configuration option is accepted. When streaming a `delimiter` configuration option, which +defaults to `'\n'`, is accepted. Supports HTTP/1.0, HTTP/1.1, HTTP/2.0 protocols. @@ -18,6 +23,8 @@ Tutorial for [HTTP to SQL Pipeline](https://www.fluvio.io/docs/tutorials/data-pi | user-agent | "fluvio/http-source 0.1.0" | String | Request user-agent | | output_type | text | String | `text` = UTF-8 String Output, `json` = UTF-8 JSON Serialized String | | output_parts | body | String | `body` = body only, `full` = all status, header and body parts | +| stream | false | bool | Flag to indicate HTTP streaming mode | +| delimiter | '\n' | String | Delimiter to separate records when producing from an HTTP streaming endpoint | #### Record Type Output | Matrix | Output | @@ -33,10 +40,10 @@ Tutorial for [HTTP to SQL Pipeline](https://www.fluvio.io/docs/tutorials/data-pi This is an example of simple connector config file: ```yaml -# sample-config.yaml +# config-example.yaml apiVersion: 0.1.0 meta: - version: 0.2.5 + version: 0.3.0 name: cat-facts type: http-source topic: cat-facts @@ -63,10 +70,10 @@ The produced record in Fluvio topic will be: Fluvio HTTP Source Connector supports Secrets in the `endpoint` and in the `headers` parameters: ```yaml -# sample-config.yaml +# config-example.yaml apiVersion: 0.1.0 meta: - version: 0.2.5 + version: 0.3.0 name: cat-facts type: http-source topic: cat-facts @@ -89,10 +96,10 @@ Fluvio HTTP Source Connector supports [Transformations](https://www.fluvio.io/do The previous example can be extended to add extra transformations to outgoing records: ```yaml -# sample-config.yaml +# config-example.yaml apiVersion: 0.1.0 meta: - version: 0.2.5 + version: 0.3.0 name: cat-facts type: http-source topic: cat-facts @@ -123,3 +130,22 @@ Now produced records will have a different shape, for example: ``` Read more about [JSON to JSON transformations](https://www.fluvio.io/smartmodules/certified/jolt/). + +### Streaming Mode + +Provide the `stream` configuration option to enable streaming mode with `delimiter` to determine how the incoming records are separated. + +```yaml +# config-example.yaml +apiVersion: 0.1.0 +meta: + version: 0.3.0 + name: wiki-updates + type: http-source + topic: wiki-updates +http: + endpoint: "https://stream.wikimedia.org/v2/stream/recentchange" + method: GET + stream: true + delimiter: "\n\n" +``` \ No newline at end of file