0.15.0 (2020-04-09)

FastSync

• To Snowflake: Support for IAM roles, AWS Session Tokens and to pass credentials as environment variables

Tap Kafka

• Bump pipelinewise-tap-kafka to 3.0.0
  • Add local storage of consumed messages and instant commit kafka offsets
  • Add more configurable options: consumer_timeout_ms, session_timeout_ms, heartbeat_interval_ms, max_poll_interval_ms
  • Add two new fixed output columns: MESSAGE_PARTITION and MESSAGE_OFFSET

Tap Snowflake

• Bump pipelinewise-tap-snowflake to 2.0.0
  • Discover only the required tables to avoid issues when too many tables in the database causing SHOW COLUMNS column to return more than the maximum 10000 rows

Target Snowflake

• Bump pipelinewise-target-snowflake to 1.6.3
  • Generate compressed CSV files by default. Optionally can be disabled by the no_compression config option

0.14.3 (2020-03-25)

• Support tap/target config files with .yaml extension when importing config
• Fixed dependency conflict in install script
• Fixed an issue when add_metadata_columns was not defined in inheritable_config.json

FastSync

• From MySQL: Increased default batch size to 50.000 rows when fastsync exporting data from MySQL tables
• To Snowflake: Log inserts, updates and csv file sizes in the same format to target-snowflake connector

Tap Kafka

• Bump pipelinewise-tap-kafka to 2.1.1
  • Commit offset from the state file and not from the consumed messages

Tap Snowflake

• Bump pipelinewise-tap-snowflake to 1.1.2
  • Fixed some dependency conflicts

Target Snowflake

• Bump pipelinewise-target-snowflake to 1.6.2
  • Log inserts, updates and csv file sizes in a more consumable format

0.14.2 (2020-03-19)

Singer transformation

Make tranformation consistent between FastSync and Singer by updating transform-field to transform without trimming.

tap-snowflake

Remove PIPELINEWISE.COLUMNS cache table.

0.14.1 (2020-03-13)

FastSync S3-csv to Snowflake

• Fix bug when date_overrides is present.

FastSync and singer target-snowflake

• Remove PIPELINEWISE.COLUMNS cache table.

0.14.0 (2020-03-10)

FastSync from Postgres

• Support reserved words as table names.

Tap Postgres

• Bump pipelinewise-tap-postgres to 1.5.1
  • Support per session wal_sender_timeout

CLI

• Fixed an issue when install script didn't show installed components correctly

0.13.3 (2020-03-09)

FastSync from Postgres & Mysql

• Fixed 'NoneType' object has no attribute 'upper' that happens when table has no PK.

FastSync to Snowflake

• Fixed an issue when parallel running fastsync(s) processes sometimes caused Information schema query returned too much data error in Snowflake

0.13.2 (2020-03-05)

FastSync Postgres

• Handle reserved words in column names in FastSync from PostgreSQL

0.13.1 (2020-03-02)

Bump ansible to 2.7.16

FastSync MySQL

• Handle reserved words in column names in FastSync from MySQL
• Fixed issue when parallelism and parallelism_max parameters were not used in tap YAML files

Tap Postgres

• Bump tap-postgres to 1.4.1
  • Remove unused timestamps in log

0.13.0 (2020-02-26)

• Structured logs in Pipelinewise, FastSync and majority of plugins.
• Include a logging config file in Pipelinewise repository and package.
• Ability to provide a custom logging config by setting the env variable LOGGING_CONF_FILE to be the path to the .conf file.

0.12.4 (2020-02-19)

Merge pull request #325 from transferwise/release-0.12.4

[RELEASE] Bump to 0.12.4 and add changelog