Releases: transferwise/pipelinewise
Releases · transferwise/pipelinewise
0.15.0 (2020-04-09)
FastSync
- To Snowflake: Support for IAM roles, AWS Session Tokens and to pass credentials as environment variables
Tap Kafka
- Bump
pipelinewise-tap-kafka
to 3.0.0- Add local storage of consumed messages and instant commit kafka offsets
- Add more configurable options:
consumer_timeout_ms
,session_timeout_ms
,heartbeat_interval_ms
,max_poll_interval_ms
- Add two new fixed output columns:
MESSAGE_PARTITION
andMESSAGE_OFFSET
Tap Snowflake
- Bump
pipelinewise-tap-snowflake
to 2.0.0- Discover only the required tables to avoid issues when too many tables in the database causing
SHOW COLUMNS
column to return more than the maximum 10000 rows
- Discover only the required tables to avoid issues when too many tables in the database causing
Target Snowflake
- Bump
pipelinewise-target-snowflake
to 1.6.3- Generate compressed CSV files by default. Optionally can be disabled by the
no_compression
config option
- Generate compressed CSV files by default. Optionally can be disabled by the
0.14.3 (2020-03-25)
- Support tap/target config files with
.yaml
extension when importing config - Fixed dependency conflict in install script
- Fixed an issue when
add_metadata_columns
was not defined ininheritable_config.json
FastSync
- From MySQL: Increased default batch size to 50.000 rows when fastsync exporting data from MySQL tables
- To Snowflake: Log inserts, updates and csv file sizes in the same format to target-snowflake connector
Tap Kafka
- Bump
pipelinewise-tap-kafka
to 2.1.1- Commit offset from the state file and not from the consumed messages
Tap Snowflake
- Bump
pipelinewise-tap-snowflake
to 1.1.2- Fixed some dependency conflicts
Target Snowflake
- Bump
pipelinewise-target-snowflake
to 1.6.2- Log inserts, updates and csv file sizes in a more consumable format
0.14.2 (2020-03-19)
Singer transformation
Make tranformation consistent between FastSync and Singer by updating transform-field to transform without trimming.
tap-snowflake
Remove PIPELINEWISE.COLUMNS
cache table.
0.14.1 (2020-03-13)
FastSync S3-csv to Snowflake
- Fix bug when
date_overrides
is present.
FastSync and singer target-snowflake
- Remove PIPELINEWISE.COLUMNS cache table.
0.14.0 (2020-03-10)
FastSync from Postgres
- Support reserved words as table names.
Tap Postgres
- Bump
pipelinewise-tap-postgres
to 1.5.1- Support per session
wal_sender_timeout
- Support per session
CLI
- Fixed an issue when install script didn't show installed components correctly
0.13.3 (2020-03-09)
FastSync from Postgres & Mysql
- Fixed
'NoneType' object has no attribute 'upper'
that happens when table has no PK.
FastSync to Snowflake
- Fixed an issue when parallel running fastsync(s) processes sometimes caused
Information schema query returned too much data
error in Snowflake
0.13.2 (2020-03-05)
FastSync Postgres
- Handle reserved words in column names in FastSync from PostgreSQL
0.13.1 (2020-03-02)
Bump ansible to 2.7.16
FastSync MySQL
- Handle reserved words in column names in FastSync from MySQL
- Fixed issue when
parallelism
andparallelism_max
parameters were not used in tap YAML files
Tap Postgres
- Bump tap-postgres to 1.4.1
- Remove unused timestamps in log
0.13.0 (2020-02-26)
- Structured logs in Pipelinewise, FastSync and majority of plugins.
- Include a logging config file in Pipelinewise repository and package.
- Ability to provide a custom logging config by setting the env variable
LOGGING_CONF_FILE
to be the path to the.conf
file.
0.12.4 (2020-02-19)
Merge pull request #325 from transferwise/release-0.12.4 [RELEASE] Bump to 0.12.4 and add changelog