Skip to content

Commit

Permalink
Cleanup sm (#764)
Browse files Browse the repository at this point in the history
* Update Fluvio live site with master updates (#733)

* docs: Cli update (#721)

* Update CLI docs

* Rearrange nav a bit
* Removing the infinyon banner too

* Update command

* Update labels

* First pass updates to docs page (#726)

* docs cleanup (#732)

* Modify  `smdk test` examples and documentation to use `--stdin` (#666)

* Remove deprecated script from readme

* Modify `smdk test` examples and doc to use `--stdin`

---------

Co-authored-by: T.J. Telan <t.telan@gmail.com>

---------

Co-authored-by: T.J. Telan <t.telan@gmail.com>
Co-authored-by: Consoli <matheus.consoli7@gmail.com>

* Merge from master (#747)

* docs: Cli update (#721)

* Update CLI docs

* Rearrange nav a bit
* Removing the infinyon banner too

* Update command

* Update labels

* First pass updates to docs page (#726)

* docs cleanup (#732)

* Modify  `smdk test` examples and documentation to use `--stdin` (#666)

* Remove deprecated script from readme

* Modify `smdk test` examples and doc to use `--stdin`

---------

Co-authored-by: T.J. Telan <t.telan@gmail.com>

* docs: fix link for go package (#734)

Prefers the doc page hosted by fluvio since none is directly offered by package

* rename config-example.yml to config-sample.yml, (#744)

that's the name of the file generated by `cdk generate`

* Clarified the term SPU and linked to appropriate page (#746)

* chore: fix lint ci

---------

Co-authored-by: T.J. Telan <t.telan@gmail.com>
Co-authored-by: Consoli <matheus.consoli7@gmail.com>
Co-authored-by: Tom Carrio <tom@carrio.dev>
Co-authored-by: mrasputin <150076078+mrasputin@users.noreply.github.com>
Co-authored-by: John Landahl <john@landahl.org>

* fix: list-log samples (#749) (#750)

closes #745

* fix: log-and-list navigation (#751) (#752)

* fix: log-and-list navigation

fix log nav consistency

* embeds/client-examples, use python 3.11

* update connector file to match the the project Readme.

* cleanup connectors

* renamed connector hello to github-examples-

* change use hello to use github examples

* remove steps from examples

* add community section

* clean-up hub, and make minor edits to github section

* fix indentation

* removed smdk install

---------

Co-authored-by: Deb <122817612+drc-infinyon@users.noreply.github.com>
Co-authored-by: T.J. Telan <t.telan@gmail.com>
Co-authored-by: Consoli <matheus.consoli7@gmail.com>
Co-authored-by: Alan Chen <argot@digikata.com>
Co-authored-by: Tom Carrio <tom@carrio.dev>
Co-authored-by: mrasputin <150076078+mrasputin@users.noreply.github.com>
Co-authored-by: John Landahl <john@landahl.org>
  • Loading branch information
8 people authored Jan 9, 2024
1 parent bf94856 commit a172aee
Show file tree
Hide file tree
Showing 15 changed files with 31 additions and 137 deletions.
3 changes: 2 additions & 1 deletion content/connectors/cdk/publish.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,8 @@ Use connector `.ipkg` package file and run it with the `--ipkg` option:
$ cdk deploy start --ipkg acme/my-connector0.1.0.ipkg --config ../sample-config.yaml
```

This command assumes that the sample config is the parent directory.
This command assumes that the sample config is in the parent directory.



#### Run an Official Connector
Expand Down
11 changes: 1 addition & 10 deletions content/docs/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,18 +153,9 @@ This will display data sent to the topic.

SmartModules are user-defined functions that process data streams.

### Install SMDK

The SmartModules Development Kit (SMDK) is required for building SmartModules:

%copy first-line%
```bash
$ fluvio install smdk
```

### Build a Filter SmartModule

Create and navigate to your SmartModule project:
Use the SmartModules Development Kit (smdk) binary to create your SmartModule project:

%copy%
```bash
Expand Down
4 changes: 0 additions & 4 deletions content/smartmodules/analytics/aggregate.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,6 @@ each new value into the accumulator.
Let's set up a new SmartModule project so that we can look at some code while
introducing aggregators.

##### Prerequisites

This section assumes that SMDK is [installed].

## Generic Example: Aggregate Numbers

Expand Down Expand Up @@ -244,7 +241,6 @@ Congratulations! :tada: Your SmartModule is now available for download in the Sm
- [Writing a map to transform records]({{< ref "/smartmodules/transform/map" >}})


[installed]: {{< ref "smartmodules/smdk/install" >}}
[publish]: {{< ref "smartmodules/smdk/publish" >}}
[InfinyOn Cloud]: https://infinyon.cloud
[`current profile`]: {{< ref "cli/client/profile" >}}
Expand Down
11 changes: 3 additions & 8 deletions content/smartmodules/lookback.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,6 @@ If Fluvio topic is empty, `look_back` is never called.

-> Lookback is only supported for SmartModules that are run on SPU. This implies that Source Connectors don't support it.

## Examples

This section assumes that SMDK is [installed].


### Monotonically increasing values

This is an example of SmartModule that leverages Lookback functionality. It reads the last record from a topic and
Expand All @@ -62,7 +57,9 @@ Using hub https://hub-dev.infinyon.cloud
[2/7] Done: Cargo.toml
[3/7] Done: README.md
[4/7] Done: SmartModule.toml
[5/7] Done: rust-toolchain.toml [6/7] Done: src/lib.rs [7/7] Done: src
[5/7] Done: rust-toolchain.toml
[6/7] Done: src/lib.rs
[7/7] Done: src
```
Then, put the code snippet from above into `src/lib.rs` file.

Expand Down Expand Up @@ -118,5 +115,3 @@ $ smdk test --text 1 --lookback-last 1 --record 2 --record 3
```
the record will be filtered out as expected.

[installed]: {{< ref "smartmodules/smdk/install" >}}

8 changes: 1 addition & 7 deletions content/smartmodules/smdk/build-test.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,8 @@ weight: 50
toc: false
---

SmartModules are binaries developed for real-time data stream processing, requiring in-line and live testing facilities. This section covers in-line testing, and the [next section] describes how to load SmartModules on the Cluster for live traffic testing.
This section covers building and testing SmartModules locally, and the [next section] describes how to load them on the Cluster for live traffic testing.

##### Prerequisites

This section assumes that SMDK is [installed] and `my-filter` project has been [generated].

### Build - Operation

Expand Down Expand Up @@ -55,15 +52,12 @@ Checkout the [next section] on how upload the SmartModule to your local cluster

### Steps

1. [Install SMDK]({{< ref "install" >}})
2. [Generate a SmartModule]({{< ref "generate" >}})
3. **[Build and Test]({{< ref "build-test" >}})**
4. [Load to your Cluster]({{< ref "load" >}})
5. [Publish to SmartModule Hub]({{< ref "publish" >}})

[next section]: {{< ref "load" >}}
[installed]: {{< ref "install" >}}
[generated]: {{< ref "generate" >}}
[Load to your Cluster]: {{< ref "load" >}}
[Publish to SmartModule Hub]: {{< ref "publish" >}}
[InfinyOn Cloud]: https://infinyon.cloud
15 changes: 5 additions & 10 deletions content/smartmodules/smdk/generate.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,6 @@ toc: false

SMDK generate helps developers build a sample SmartModule project by answering a few simple questions

##### Prerequisites

This section assumes that SMDK is [installed]

### Generate - Operation

SMDK generate commands runs a wizard and builds a sample project in a subdirectory
Expand Down Expand Up @@ -89,13 +85,12 @@ The project is ready to build and test. Checkout the next section for instructio

### Steps

1. [Install SMDK]({{< ref "install" >}})
2. **[Generate a SmartModule]({{< ref "generate" >}})**
3. [Build and Test]({{< ref "build-test" >}})
4. [Load to your Cluster]({{< ref "load" >}})
5. [Publish to SmartModule Hub]({{< ref "publish" >}})
1. **[Generate a SmartModule]({{< ref "generate" >}})**
2. [Build and Test]({{< ref "build-test" >}})
3. [Load to your Cluster]({{< ref "load" >}})
4. [Publish to SmartModule Hub]({{< ref "publish" >}})


[installed]: {{< ref "install" >}}
[Build and Test]: {{< ref "build-test" >}}
[Load to your Cluster]: {{< ref "load" >}}
[Publish to SmartModule Hub]: {{< ref "publish" >}}
41 changes: 0 additions & 41 deletions content/smartmodules/smdk/install.md

This file was deleted.

16 changes: 6 additions & 10 deletions content/smartmodules/smdk/load.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,11 @@ toc: false

SMDK load give developers the ability to upload their own SmartModules to a local or [InfinyOn Cloud] clusters.

##### Prerequisites

This section assumes that SMDK is [installed] and `my-filter` project has been [built].
This section assumes that `my-filter` project has been [built].

### Load - Operation

Navigate to your `my-filter` directory and use the `load` command:
From your `my-filter` directory and use the `load` command:

%copy first-line%
```bash
Expand Down Expand Up @@ -115,13 +113,11 @@ The conector creates a topic called `cats` and filters out all records without l

### Steps

1. [Install SMDK]({{< ref "install" >}})
2. [Generate a SmartModule]({{< ref "generate" >}})
3. [Build and Test]({{< ref "build-test" >}})
4. **[Load to your Cluster]({{< ref "load" >}})**
5. [Publish to SmartModule Hub]({{< ref "publish" >}})
1. [Generate a SmartModule]({{< ref "generate" >}})
2. [Build and Test]({{< ref "build-test" >}})
3. **[Load to your Cluster]({{< ref "load" >}})**
4. [Publish to SmartModule Hub]({{< ref "publish" >}})

[InfinyOn Cloud]: https://infinyon.cloud
[installed]: {{< ref "install" >}}
[built]: {{< ref "build-test/#build---operation" >}}
[`current profile`]: {{< ref "cli/client/profile" >}}
9 changes: 4 additions & 5 deletions content/smartmodules/smdk/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,10 @@ SMDK helps developers build, test in-line, load to local or cloud Cluster, then

### Steps

1. **[Install SMDK]({{< ref "install" >}})**
2. [Generate a SmartModule]({{< ref "generate" >}})
3. [Build and Test]({{< ref "build-test" >}})
4. [Load to your Cluster]({{< ref "load" >}})
5. [Publish to SmartModule Hub]({{< ref "publish" >}})
1. [Generate a SmartModule]({{< ref "generate" >}})
2. [Build and Test]({{< ref "build-test" >}})
3. [Load to your Cluster]({{< ref "load" >}})
4. [Publish to SmartModule Hub]({{< ref "publish" >}})

[`Fluvio CLI`]: {{< ref "/cli/smartmodules/smdk" >}}
[`Install Rust`]: https://www.rust-lang.org/tools/install
Expand Down
18 changes: 7 additions & 11 deletions content/smartmodules/smdk/publish.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,13 @@ weight: 70
toc: false
---

SMDK load give developers the ability to publish their own SmartModules to the [Hub].
SMDK publish give developers the ability to publish their own SmartModules to the [Hub].

##### Prerequisites

This section assumes that SMDK is [installed] and `my-filter` project has been [built].
This section assumes that `my-filter` project has been [built].

### Publish - Operation

Navigate to your `my-filter` directory and use the `publish` command:
From your `my-filter` directory and use the `publish` command:

%copy first-line%
```bash
Expand Down Expand Up @@ -44,14 +42,12 @@ Congratulations :tada:! You are now ready to build, test, and publish your own

### Steps

1. [Install SMDK]({{< ref "install" >}})
2. [Generate a SmartModule]({{< ref "generate" >}})
3. [Build and Test]({{< ref "build-test" >}})
4. [Load to your Cluster]({{< ref "load" >}})
5. **[Publish to SmartModule Hub]({{< ref "publish" >}})**
1. [Generate a SmartModule]({{< ref "generate" >}})
2. [Build and Test]({{< ref "build-test" >}})
3. [Load to your Cluster]({{< ref "load" >}})
4. **[Publish to SmartModule Hub]({{< ref "publish" >}})**

[Hub]: {{< ref "/smartmodules/hub/overview" >}}
[SmartModule types]: {{< ref "/smartmodules/transform/overview" >}}
[installed]: {{< ref "install" >}}
[built]: {{< ref "build-test/#build---operation" >}}
[download]: {{< ref "../hub/download" >}}
11 changes: 2 additions & 9 deletions content/smartmodules/smdk/smartmodulerecord.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,9 @@ weight: 40
toc: false
---

Inside the SmartModule Context, Fluvio `Record` becomes `SmartModuleRecord`.
`SmartModuleRecord` is a wrapper on the standard `Record`, which provides extra details on the `Record` such as the moment when the `Record` was produced.

`SmartModuleRecord` is a wrapper on the standard `Record`, which provides extra
details on the `Record` such as the moment when the `Record` was produced.

-> `SmartModuleRecord` was introduced on [Fluvio v0.10.14][1], we recommend upgrading to `SmartModuleRecord` as future versions will completely deprecate `Record` in the SmartModule Context.

## Usage
### Usage

When building any type of SmartModules you will have access to each record, in
the SmartModule function these records are represented with the
Expand Down Expand Up @@ -54,5 +49,3 @@ pub fn map(record: &SmartModuleRecord) -> Result<(Option<RecordData>, RecordData
With this small change we are injecting insightful data to our record. This is
a simple use case but we can have more based on the implementation and business
requirements.

[1]: https://github.com/infinyon/fluvio/blob/master/CHANGELOG.md#platform-version-01014---2023-07-28
5 changes: 0 additions & 5 deletions content/smartmodules/transform/array-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,6 @@ SmartModule ArrayMaps are used to break apart Records into smaller pieces. This

Let's take a look at an example ArrayMap and walk through how it works and what some sample input and output data might look like.

##### Prerequisites

This section assumes that SMDK is [installed].

## Generic Example: Transform JSON arrays to records

A common use case is to transform JSON arrays and produce a stream of the records of those arrays. For example, suppose that each element we receive in Fluvio is a JSON array, though we want to interact with the _elements_ of these arrays rather than the arrays themselves. Then, using an ArrayMap, we can transform composite arrays to records that look like this (where this line is a single record):
Expand Down Expand Up @@ -234,7 +230,6 @@ Congratulations! :tada: Your SmartModule is now available for download in the Sm
- [Writing a JSON filter]({{< ref "/smartmodules/transform/filter" >}})
- [Writing an aggregate to sum numbers]({{< ref "/smartmodules/analytics/aggregate" >}})

[installed]: {{< ref "smartmodules/smdk/install" >}}
[publish]: {{< ref "smartmodules/smdk/publish" >}}
[InfinyOn Cloud]: https://infinyon.cloud
[`current profile`]: {{< ref "cli/client/profile" >}}
Expand Down
5 changes: 0 additions & 5 deletions content/smartmodules/transform/filter-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,6 @@ the record according to your needs and return it as `Some(record)`.

Let's dive in and see how to use this in practice. You can find the full code for this doc's example [in the fluvio-smartmodule-examples repository][1].

##### Prerequisites

This section assumes that SMDK is [installed].


## Getting Practical: Transform `Order Ready` events in a mobile shopping App

Expand Down Expand Up @@ -278,7 +274,6 @@ Congratulations! :tada: Your SmartModule is now available for download in the Sm
- [Writing an aggregate to sum numbers]({{< ref "aggregate" >}})


[installed]: {{< ref "smartmodules/smdk/install" >}}
[publish]: {{< ref "smartmodules/smdk/publish" >}}
[InfinyOn Cloud]: https://infinyon.cloud
[`current profile`]: {{< ref "cli/client/profile" >}}
Expand Down
5 changes: 0 additions & 5 deletions content/smartmodules/transform/filter.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,6 @@ The simplest type of SmartModule is a filter, which can examine each record in a

<img src="/smartmodules/images/smartmodule-filter.svg" alt="SmartModule Filter" justify="center" height="190">

##### Prerequisites

This section assumes that SMDK is [installed].


## Getting Practical: Filter Records by JSON fields

Expand Down Expand Up @@ -336,7 +332,6 @@ Congratulations! :tada: Your SmartModule is now available for download in the Sm
- [Writing an aggregate to sum numbers]({{< ref "/smartmodules/analytics/aggregate" >}})


[installed]: {{< ref "smartmodules/smdk/install" >}}
[publish]: {{< ref "smartmodules/smdk/publish" >}}
[InfinyOn Cloud]: https://infinyon.cloud
[`current profile`]: {{< ref "cli/client/profile" >}}
Expand Down
6 changes: 0 additions & 6 deletions content/smartmodules/transform/map.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,6 @@ example looks like.
<img src="/smartmodules/images/smartmodule-map.svg" alt="SmartModule Map" justify="center" height="190">


##### Prerequisites

This section assumes that SMDK is [installed].


### Create a SmartModule Project

Run `smdk generate` with the name of the map and choose the "map" options:
Expand Down Expand Up @@ -227,7 +222,6 @@ Congratulations! :tada: Your SmartModule is now available for download in the Sm
- [Writing an aggregate to sum numbers]({{< ref "/smartmodules/analytics/aggregate" >}})


[installed]: {{< ref "smartmodules/smdk/install" >}}
[publish]: {{< ref "smartmodules/smdk/publish" >}}
[InfinyOn Cloud]: https://infinyon.cloud
[`current profile`]: {{< ref "cli/client/profile" >}}
Expand Down

0 comments on commit a172aee

Please sign in to comment.