Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Enhance bulk insert speed using copy command #370

Open
wants to merge 23 commits into
base: main
Choose a base branch
from

Commits on Jun 28, 2024

  1. Generate a copy statement

    Create a variation of generate_insert_statement that returns a PostgreSQL copy statement, suitable for bulk loading of data formatted as csv.
    kinghuang committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    a91c94b View commit details
    Browse the repository at this point in the history
  2. Bulk insert data using copy

    Use copy instead of insert to bulk insert records. In PostgreSQL, copy is the fastest way to insert bulk data.
    kinghuang committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    c11a929 View commit details
    Browse the repository at this point in the history
  3. Remove generate_insert_statement

    The override of generate_insert_statement is no longer used.
    kinghuang committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    4af0c97 View commit details
    Browse the repository at this point in the history
  4. Directly handle csv generation

    Use the type bind processors to generate values to ensure that values are represented correctly to PostgreSQL, especially for things like ARRAY and JSONB. Also, handle null values.
    kinghuang committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    943419b View commit details
    Browse the repository at this point in the history
  5. Quote table name

    kinghuang committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    734c9b5 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    a97ab70 View commit details
    Browse the repository at this point in the history
  7. Improve csv generation

    Always quote strings and handle array values.
    kinghuang committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    72f2b37 View commit details
    Browse the repository at this point in the history
  8. Remove unused imports

    kinghuang committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    e8b438c View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    20cdb55 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    534a38a View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    ed4838f View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. Configuration menu
    Copy the full SHA
    c9f4e3f View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. Configuration menu
    Copy the full SHA
    373f2a3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b09b4c4 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. Configuration menu
    Copy the full SHA
    dd1622a View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. Configuration menu
    Copy the full SHA
    cc1c6bc View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. Configuration menu
    Copy the full SHA
    4ca89fa View commit details
    Browse the repository at this point in the history
  2. Update imports

    edgarrmondragon committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    12a85b2 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. Configuration menu
    Copy the full SHA
    519d15b View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. Configuration menu
    Copy the full SHA
    f325009 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. Configuration menu
    Copy the full SHA
    a7d4263 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. Configuration menu
    Copy the full SHA
    a99021e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a11c76b View commit details
    Browse the repository at this point in the history