Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨ Source-Stripe: Add incremental stream support to accounts stream #46864

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
+34 −9
Open

✨ Source-Stripe: Add incremental stream support to accounts stream #46864

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
44cecf8
Accounts incremental
williamkaper Oct 12, 2024
40e12ab
chore: format code
williamkaper Oct 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions airbyte-integrations/connectors/source-stripe/erd/discovered_catalog.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3289,6 +3289,10 @@
"description": "The timestamp when the account was created.",
"type": ["null", "integer"]
},
"updated": {
"description": "The timestamp when the account was updated.",
"type": ["null", "integer"]
},
"default_currency": {
"description": "The default currency used for transactions.",
"type": ["null", "string"]
Expand Down Expand Up @@ -3839,9 +3843,10 @@
}
}
},
"supported_sync_modes": ["full_refresh"],
"source_defined_primary_key": [["id"]],
"is_resumable": true
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"default_cursor_field": ["updated"],
"source_defined_primary_key": [["id"]]
},
{
"name": "shipping_rates",
Expand Down
3 changes: 2 additions & 1 deletion airbyte-integrations/connectors/source-stripe/erd/source.dbml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,7 @@ Table "accounts" {
"company" object
"country" string
"created" integer
"updated" integer
"default_currency" string
"details_submitted" boolean
"email" string
Expand Down Expand Up @@ -1634,4 +1635,4 @@ Ref {

Ref {
"usage_records"."subscription_item" <> "subscription_items"."id"
}
}
4 changes: 3 additions & 1 deletion airbyte-integrations/connectors/source-stripe/integration_tests/configured_catalog.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,12 @@
"stream": {
"name": "accounts",
"json_schema": {},
"supported_sync_modes": ["full_refresh"],
"supported_sync_modes": ["full_refresh", "incremental"],
"source_defined_cursor": true,
"source_defined_primary_key": [["id"]]
},
"primary_key": [["id"]],
"cursor_field": ["updated"],
"sync_mode": "full_refresh",
"destination_sync_mode": "overwrite"
},
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-stripe/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: e094cb9a-26de-4645-8761-65c0c425d1de
dockerImageTag: 5.6.2
dockerImageTag: 5.6.3
dockerRepository: airbyte/source-stripe
documentationUrl: https://docs.airbyte.com/integrations/sources/stripe
erdUrl: https://dbdocs.io/airbyteio/source-stripe?view=relationships
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-stripe/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ requires = [ "poetry-core>=1.0.0",]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
version = "5.6.2"
version = "5.6.3"
name = "source-stripe"
description = "Source implementation for Stripe."
authors = [ "Airbyte <contact@airbyte.io>",]
Expand Down
4 changes: 4 additions & 0 deletions airbyte-integrations/connectors/source-stripe/source_stripe/schemas/accounts.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,10 @@
"description": "The timestamp when the account was created.",
"type": ["null", "integer"]
},
"updated": {
"description": "The timestamp when the account was updated.",
"type": ["null", "integer"]
},
"default_currency": {
"description": "The default currency used for transactions.",
"type": ["null", "string"]
Expand Down
14 changes: 13 additions & 1 deletion airbyte-integrations/connectors/source-stripe/source_stripe/source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,19 @@ def streams(self, config: MutableMapping[str, Any]) -> List[Stream]:
**args,
),
SetupAttempts(**incremental_args),
StripeStream(name="accounts", path="accounts", use_cache=USE_CACHE, **args),
UpdatedCursorIncrementalStripeStream(
name="accounts",
path="accounts",
legacy_cursor_field="created",
event_types=[
"account.updated",
"account.external_account.created",
"account.external_account.updated",
"account.external_account.deleted",
],
use_cache=USE_CACHE,
**args,
),
CreatedCursorIncrementalStripeStream(name="shipping_rates", path="shipping_rates", **incremental_args),
CreatedCursorIncrementalStripeStream(name="balance_transactions", path="balance_transactions", **incremental_args),
CreatedCursorIncrementalStripeStream(name="files", path="files", **incremental_args),
Expand Down
3 changes: 2 additions & 1 deletion docs/integrations/sources/stripe.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ The Stripe source connector supports the following streams:
- [Transactions](https://stripe.com/docs/api/transfers/list) \(Incremental\)
- [Transfers](https://stripe.com/docs/api/transfers/list) \(Incremental\)
- [Transfer Reversals](https://stripe.com/docs/api/transfer_reversals/list)
- [Usage Records](https://stripe.com/docs/api/usage_records/subscription_item_summary_list)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was causing the tests to fail - not sure how this got here...

- [Usage Records](https://stripe.com/docs/api/usage_records/)

### Entity-Relationship Diagram (ERD)
<EntityRelationshipDiagram></EntityRelationshipDiagram>
Expand Down Expand Up @@ -239,6 +239,7 @@ Each record is marked with `is_deleted` flag when the appropriate event happens

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:----------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 5.6.3 | 2024-10-12 | [46864](https://github.com/airbytehq/airbyte/pull/46864) | Add incremental stream support to `accounts` stream |
| 5.6.2 | 2024-10-05 | [43881](https://github.com/airbytehq/airbyte/pull/43881) | Update dependencies |
| 5.6.1 | 2024-10-03 | [46327](https://github.com/airbytehq/airbyte/pull/46327) | Bump the cdk to 5.10.2 to stop using PrintBuffer optimization due to record count mismatches |
| 5.6.0 | 2024-09-10 | [44891](https://github.com/airbytehq/airbyte/pull/44891) | Update `Payment Methods` stream |
Expand Down
Loading