Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add initial event feed support #290

Merged
merged 18 commits into from
Oct 28, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 40 additions & 38 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,11 @@ See the [Fauna Documentation](https://docs.fauna.com/fauna/current/) for additio
- [Iterate on a stream](#iterate-on-a-stream)
- [Close a stream](#close-a-stream)
- [Stream options](#stream-options)
- [Change Feeds](#change-feeds-beta)
- [Request a Change Feed](#request-a-change-feed)
- [Iterate on a Change Feed](#iterate-on-a-change-feed)
- [Event Feeds](#event-feeds)
- [Request an Event Feed](#request-a-event-feed)
- [Iterate on an Event Feed](#iterate-on-a-event-feed)
- [Error handling](#error-handling)
- [Change Feed options](#change-feed-options)
- [Event Feed options](#event-feed-options)
- [Contributing](#contributing)
- [Set up the repo](#set-up-the-repo)
- [Run tests](#run-tests)
Expand Down Expand Up @@ -476,11 +476,11 @@ The driver supports [Event Streaming](https://docs.fauna.com/fauna/current/learn
### Start a stream

To get a stream token, append
ecooper marked this conversation as resolved.
Show resolved Hide resolved
[`toStream()`](https://docs.fauna.com/fauna/current/reference/reference/schema_entities/set/tostream)
[`eventSource()`](https://docs.fauna.com/fauna/current/reference/reference/schema_entities/set/eventsource)
or
[`changesOn()`](https://docs.fauna.com/fauna/current/reference/reference/schema_entities/set/changeson)
[`eventsOn()`](https://docs.fauna.com/fauna/current/reference/reference/schema_entities/set/eventson)
to a set from a [supported
source](https://docs.fauna.com/fauna/current/reference/streaming_reference/#supported-sources).
source](https://docs.fauna.com/fauna/current/reference/streaming_reference/#sets).
ecooper marked this conversation as resolved.
Show resolved Hide resolved

To start and subscribe to the stream, pass the stream token to `stream()`:
ecooper marked this conversation as resolved.
Show resolved Hide resolved

Expand All @@ -490,7 +490,7 @@ const response = await client.query(fql`

{
initialPage: set.pageSize(10),
streamToken: set.toStream()
streamToken: set.eventSource()
ecooper marked this conversation as resolved.
Show resolved Hide resolved
}
`);
const { initialPage, streamToken } = response.data;
Expand All @@ -501,7 +501,7 @@ client.stream(streamToken);
You can also pass a query that produces a stream token directly to `stream()`:
ecooper marked this conversation as resolved.
Show resolved Hide resolved

```javascript
const query = fql`Product.all().changesOn(.price, .stock)`;
const query = fql`Product.all().eventsOn(.price, .stock)`;

client.stream(query);
```
Expand Down Expand Up @@ -558,7 +558,7 @@ stream.start(
Use `close()` to close a stream:

```javascript
const stream = await client.stream(fql`Product.all().toStream()`);
const stream = await client.stream(fql`Product.all().eventSource()`);

let count = 0;
for await (const event of stream) {
Expand Down Expand Up @@ -591,59 +591,59 @@ const options: StreamClientConfiguration = {
cursor: null,
};

client.stream(fql`Product.all().toStream()`, options);
client.stream(fql`Product.all().eventSource()`, options);
```

For supported properties, see
[StreamClientConfiguration](https://fauna.github.io/fauna-js/latest/types/StreamClientConfiguration.html)
in the API reference.

## Change Feeds (beta)
## Event Feeds

The driver supports [Change Feeds](https://docs.fauna.com/fauna/current/learn/track-changes/streaming/#change-feeds).
The driver supports [Event Feeds](https://docs.fauna.com/fauna/current/learn/cdc/#event-feeds).

### Request a Change Feed
### Request a Event Feed
ecooper marked this conversation as resolved.
Show resolved Hide resolved

A Change Feed asynchronously polls an [event stream](https://docs.fauna.com/fauna/current/learn/streaming),
An Event Feed asynchronously polls an [event stream](https://docs.fauna.com/fauna/current/learn/cdc/#event-feeds),
represented by a stream token, for events.

To get a stream token, append `toStream()` or `changesOn()` to a set from a
To get a stream token, append `eventSource()` or `eventsOn()` to a set from a
[supported source](https://docs.fauna.com/fauna/current/reference/streaming_reference/#supported-sources).

To get paginated events for the stream, pass the stream token to
`changeFeed()`:
`feed()`:

```javascript
const response = await client.query(fql`
let set = Product.all()

{
initialPage: set.pageSize(10),
streamToken: set.toStream()
streamToken: set.eventSource()
}
`);
const { initialPage, streamToken } = response.data;

const changeFeed = client.changeFeed(streamToken);
const feed = client.feed(streamToken);
```

You can also pass a query that produces a stream token directly to `changeFeed()`:
You can also pass a query that produces a stream token directly to `feed()`:
ecooper marked this conversation as resolved.
Show resolved Hide resolved

```javascript
const query = fql`Product.all().changesOn(.price, .stock)`;
const query = fql`Product.all().eventsOn(.price, .stock)`;

const changeFeed = client.changeFeed(query);
const feed = client.feed(query);
```

### Iterate on a Change Feed
### Iterate on a Event Feed
ecooper marked this conversation as resolved.
Show resolved Hide resolved

`changeFeed()` returns a `ChangeFeedClient` instance that can act as an `AsyncIterator`. You can use `for await...of` to iterate through all the pages:
`feed()` returns a `FeedClient` instance that can act as an `AsyncIterator`. You can use `for await...of` to iterate through all the pages:

```ts
const query = fql`Product.all().changesOn(.price, .stock)`;
const changeFeed = client.changeFeed(query);
const query = fql`Product.all().eventsOn(.price, .stock)`;
const feed = client.feed(query);

for await (const page of changeFeed) {
for await (const page of feed) {
console.log("Page stats", page.stats);

for (event in page.events) {
Expand All @@ -662,10 +662,10 @@ for await (const page of changeFeed) {
Alternatively, use `flatten()` to get paginated results as a single, flat array:

```ts
const query = fql`Product.all().changesOn(.price, .stock)`;
const changeFeed = client.changeFeed(query);
const query = fql`Product.all().eventsOn(.price, .stock)`;
const feed = client.feed(query);

for await (const event of changeFeed.flatten()) {
for await (const event of feed.flatten()) {
console.log("Stream event:", event);
ecooper marked this conversation as resolved.
Show resolved Hide resolved
}
```
Expand All @@ -681,12 +681,12 @@ This distinction allows for you to ignore errors originating from event processi
For example:

```ts
const changeFeed = client.changeFeed(fql`
Product.all().map(.details.toUpperCase()).toStream()
const feed = client.feed(fql`
Product.all().map(.details.toUpperCase()).eventSource()
`);

try {
for await (const page of changeFeed) {
for await (const page of feed) {
// Pages will stop at the first error encountered.
// Therefore, its safe to handle an event failures
// and then pull more pages.
Expand All @@ -703,26 +703,28 @@ try {
}
```

### Change Feed options
### Event Feed options

The client configuration sets the default options for `changeFeed()`. You can pass a `ChangeFeedClientConfiguration` object to override these defaults:
The client configuration sets the default options for `feed()`. You can pass a `FeedClientConfiguration` object to override these defaults:

```ts
const options: ChangeFeedClientConfiguration = {
const options: FeedClientConfiguration = {
long_type: "number",
max_attempts: 5,
max_backoff: 1000,
query_timeout_ms: 5000,
client_timeout_buffer_ms: 5000,
secret: "FAUNA_SECRET",
cursor: undefined,
start_ts: undefined,
};

client.changeFeed(fql`Product.all().toStream()`, options);
client.feed(fql`Product.all().eventSource()`, options);
```

## Contributing

Any contributions are from the community are greatly appreciated!
Any contributions from the community are greatly appreciated!

If you have a suggestion that would make this better, please fork the repo and create a pull request. You may also simply open an issue. We provide templates, so please complete those to the best of your ability.

Expand Down
18 changes: 9 additions & 9 deletions __tests__/functional/client-configuration.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ describe("ClientConfiguration", () => {
process.env["FAUNA_SECRET"] = "foo";
const client = new Client();
expect(Buffer.from(JSON.stringify(client)).toString()).not.toContain(
"secret"
"secret",
);
});

Expand All @@ -33,11 +33,11 @@ describe("ClientConfiguration", () => {

const client = new Client({ secret: "secret" });
expect(client.clientConfiguration.endpoint?.toString()).toEqual(
"https://localhost:9999/"
"https://localhost:9999/",
);
});

it("Client respectes passed in client configuration over defaults", () => {
it("Client respects passed in client configuration over defaults", () => {
// TODO: when the Client accepts an http client add a mock that validates
// the configuration changes were applied.
});
Expand All @@ -51,7 +51,7 @@ describe("ClientConfiguration", () => {
if ("message" in e) {
expect(e.message).toEqual(
"You must provide a secret to the driver. Set it in \
an environmental variable named FAUNA_SECRET or pass it to the Client constructor."
an environmental variable named FAUNA_SECRET or pass it to the Client constructor.",
);
}
}
Expand Down Expand Up @@ -117,10 +117,10 @@ an environmental variable named FAUNA_SECRET or pass it to the Client constructo
expect(req.headers["x-query-timeout-ms"]).toEqual("5000");
const _expectedHeader = expectedHeader;
expect(req.headers[_expectedHeader.key]).toEqual(
_expectedHeader.value
_expectedHeader.value,
);
return getDefaultHTTPClient(getDefaultHTTPClientOptions()).request(
req
req,
);
},

Expand All @@ -132,11 +132,11 @@ an environmental variable named FAUNA_SECRET or pass it to the Client constructo
query_timeout_ms: 5000,
[fieldName]: fieldValue,
},
httpClient
httpClient,
);
await client.query<number>(fql`"taco".length`);
client.close();
}
},
);

it("can accept endpoints with or without a trailing slash.", async () => {
Expand Down Expand Up @@ -173,7 +173,7 @@ an environmental variable named FAUNA_SECRET or pass it to the Client constructo
} finally {
client?.close();
}
}
},
);

it("throws a RangeError if 'client_timeout_buffer_ms' is less than or equal to zero", async () => {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,40 +1,38 @@
import {
StreamToken,
getDefaultHTTPClient,
ChangeFeedClientConfiguration,
ChangeFeedClient,
FeedClientConfiguration,
FeedClient,
} from "../../src";
import { getDefaultHTTPClientOptions } from "../client";

const defaultHttpClient = getDefaultHTTPClient(getDefaultHTTPClientOptions());
const defaultConfig: ChangeFeedClientConfiguration = {
const defaultConfig: FeedClientConfiguration = {
secret: "secret",
long_type: "number",
max_attempts: 3,
max_backoff: 20,
query_timeout_ms: 5000,
client_timeout_buffer_ms: 5000,
httpClient: defaultHttpClient,
};
const dummyStreamToken = new StreamToken("dummy");

describe("ChangeFeedClientConfiguration", () => {
describe("FeedClientConfiguration", () => {
it("can be instantiated directly with a token", () => {
new ChangeFeedClient(dummyStreamToken, defaultConfig);
new FeedClient(dummyStreamToken, defaultConfig);
});

it("can be instantiated directly with a lambda", async () => {
new ChangeFeedClient(
() => Promise.resolve(dummyStreamToken),
defaultConfig,
);
new FeedClient(() => Promise.resolve(dummyStreamToken), defaultConfig);
});

it("throws a RangeError if 'max_backoff' is less than or equal to zero", async () => {
expect.assertions(1);

const config = { ...defaultConfig, max_backoff: 0 };
try {
new ChangeFeedClient(dummyStreamToken, config);
new FeedClient(dummyStreamToken, config);
} catch (e: any) {
expect(e).toBeInstanceOf(RangeError);
}
Expand All @@ -46,21 +44,18 @@ describe("ChangeFeedClientConfiguration", () => {
${"httpClient"}
${"max_backoff"}
${"max_attempts"}
${"client_timeout_buffer_ms"}
${"query_timeout_ms"}
${"secret"}
`(
"throws a TypeError if $fieldName provided is undefined",
async ({
fieldName,
}: {
fieldName: keyof ChangeFeedClientConfiguration;
}) => {
async ({ fieldName }: { fieldName: keyof FeedClientConfiguration }) => {
expect.assertions(1);

const config = { ...defaultConfig };
delete config[fieldName];
try {
new ChangeFeedClient(dummyStreamToken, config);
new FeedClient(dummyStreamToken, config);
} catch (e: any) {
expect(e).toBeInstanceOf(TypeError);
}
Expand All @@ -72,7 +67,7 @@ describe("ChangeFeedClientConfiguration", () => {

const config = { ...defaultConfig, max_attempts: 0 };
try {
new ChangeFeedClient(dummyStreamToken, config);
new FeedClient(dummyStreamToken, config);
} catch (e: any) {
expect(e).toBeInstanceOf(RangeError);
}
Expand All @@ -81,7 +76,25 @@ describe("ChangeFeedClientConfiguration", () => {
it("throws a TypeError is start_ts and cursor are both provided", async () => {
const config = { ...defaultConfig, start_ts: 1, cursor: "cursor" };
expect(() => {
new ChangeFeedClient(dummyStreamToken, config);
new FeedClient(dummyStreamToken, config);
}).toThrow(TypeError);
});

it("throws a RangeError if 'query_timeout_ms' is less than or equal to zero", async () => {
const config = { ...defaultConfig, query_timeout_ms: 0 };
try {
new FeedClient(dummyStreamToken, config);
} catch (e: any) {
expect(e).toBeInstanceOf(RangeError);
}
});

it("throws a RangeError if 'client_timeout_buffer_ms' is less than or equal to zero", async () => {
const config = { ...defaultConfig, client_timeout_buffer_ms: 0 };
try {
new FeedClient(dummyStreamToken, config);
} catch (e: any) {
expect(e).toBeInstanceOf(RangeError);
}
});
});
Loading
Loading