-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Enhance bulk insert speed using copy command #370
Open
kinghuang
wants to merge
23
commits into
MeltanoLabs:main
Choose a base branch
from
kinghuang:bulk-insert-copy
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Commits on Jun 28, 2024
-
Create a variation of generate_insert_statement that returns a PostgreSQL copy statement, suitable for bulk loading of data formatted as csv.
Configuration menu - View commit details
-
Copy full SHA for a91c94b - Browse repository at this point
Copy the full SHA a91c94bView commit details -
Use copy instead of insert to bulk insert records. In PostgreSQL, copy is the fastest way to insert bulk data.
Configuration menu - View commit details
-
Copy full SHA for c11a929 - Browse repository at this point
Copy the full SHA c11a929View commit details -
Remove generate_insert_statement
The override of generate_insert_statement is no longer used.
Configuration menu - View commit details
-
Copy full SHA for 4af0c97 - Browse repository at this point
Copy the full SHA 4af0c97View commit details -
Directly handle csv generation
Use the type bind processors to generate values to ensure that values are represented correctly to PostgreSQL, especially for things like ARRAY and JSONB. Also, handle null values.
Configuration menu - View commit details
-
Copy full SHA for 943419b - Browse repository at this point
Copy the full SHA 943419bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 734c9b5 - Browse repository at this point
Copy the full SHA 734c9b5View commit details -
Configuration menu - View commit details
-
Copy full SHA for a97ab70 - Browse repository at this point
Copy the full SHA a97ab70View commit details -
Always quote strings and handle array values.
Configuration menu - View commit details
-
Copy full SHA for 72f2b37 - Browse repository at this point
Copy the full SHA 72f2b37View commit details -
Configuration menu - View commit details
-
Copy full SHA for e8b438c - Browse repository at this point
Copy the full SHA e8b438cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 20cdb55 - Browse repository at this point
Copy the full SHA 20cdb55View commit details -
Configuration menu - View commit details
-
Copy full SHA for 534a38a - Browse repository at this point
Copy the full SHA 534a38aView commit details -
Configuration menu - View commit details
-
Copy full SHA for ed4838f - Browse repository at this point
Copy the full SHA ed4838fView commit details
Commits on Jul 3, 2024
-
Configuration menu - View commit details
-
Copy full SHA for c9f4e3f - Browse repository at this point
Copy the full SHA c9f4e3fView commit details
Commits on Jul 17, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 373f2a3 - Browse repository at this point
Copy the full SHA 373f2a3View commit details -
Configuration menu - View commit details
-
Copy full SHA for b09b4c4 - Browse repository at this point
Copy the full SHA b09b4c4View commit details
Commits on Jul 18, 2024
-
Configuration menu - View commit details
-
Copy full SHA for dd1622a - Browse repository at this point
Copy the full SHA dd1622aView commit details
Commits on Jul 30, 2024
-
Configuration menu - View commit details
-
Copy full SHA for cc1c6bc - Browse repository at this point
Copy the full SHA cc1c6bcView commit details
Commits on Aug 14, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 4ca89fa - Browse repository at this point
Copy the full SHA 4ca89faView commit details -
Configuration menu - View commit details
-
Copy full SHA for 12a85b2 - Browse repository at this point
Copy the full SHA 12a85b2View commit details
Commits on Aug 29, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 519d15b - Browse repository at this point
Copy the full SHA 519d15bView commit details
Commits on Sep 3, 2024
-
Configuration menu - View commit details
-
Copy full SHA for f325009 - Browse repository at this point
Copy the full SHA f325009View commit details
Commits on Sep 19, 2024
-
Configuration menu - View commit details
-
Copy full SHA for a7d4263 - Browse repository at this point
Copy the full SHA a7d4263View commit details
Commits on Sep 27, 2024
-
Configuration menu - View commit details
-
Copy full SHA for a99021e - Browse repository at this point
Copy the full SHA a99021eView commit details -
Configuration menu - View commit details
-
Copy full SHA for a11c76b - Browse repository at this point
Copy the full SHA a11c76bView commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.