Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use PENDING mode in BigQuery writes #20852

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Use PENDING mode in BigQuery writes #20852

wants to merge 1 commit into from
+47 −31

Conversation

ebyhr
Copy link
Member

@ebyhr ebyhr commented Feb 28, 2024 •
edited by hashhar
Loading

Description

Use PENDING mode in BigQuery writes to improve the write performance.
The elapsed time became 2/3 when testing with tpch.tiny.orders table.

Fixes #15158

Release notes

(x) Release notes are required, with the following suggested text:

# BigQuery
* Improve performance when writing data. ({issue}`issuenumber`)

@cla-bot cla-bot bot added the cla-signed label Feb 28, 2024
@github-actions github-actions bot added the bigquery BigQuery connector label Feb 28, 2024
@ebyhr ebyhr self-assigned this Feb 28, 2024
hashhar
hashhar reviewed Feb 28, 2024
View reviewed changes
Copy link
Member

@hashhar hashhar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why stream is still managed in the page-sink?

I had thought it would make more sense to move write-stream management to coordinator in beginInsert and finalize the streams in finishInsert?

@hashhar hashhar mentioned this pull request Feb 28, 2024
Open
@github-actions GitHub Actions

This comment was marked as resolved.

Sign in to view
@github-actions github-actions bot added the stale label Mar 20, 2024
@wendigo
Copy link
Contributor

wendigo commented Mar 20, 2024

@ebyhr are you still working on that?

@ebyhr

This comment was marked as outdated.

Sign in to view
@ebyhr ebyhr closed this Mar 21, 2024
@ebyhr ebyhr deleted the ebi/bigquery-write-pending branch March 21, 2024 01:37
@ebyhr ebyhr restored the ebi/bigquery-write-pending branch November 6, 2024 10:18
@ebyhr ebyhr reopened this Nov 6, 2024
@ebyhr ebyhr force-pushed the ebi/bigquery-write-pending branch 2 times, most recently from 67bf3c4 to d19d6cf Compare November 6, 2024 10:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hashhar hashhar hashhar left review comments

@findepi findepi findepi approved these changes

Assignees

@ebyhr ebyhr

Labels
bigquery BigQuery connector cla-signed stale
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Switch to Storage write API in BigQuery
4 participants
@ebyhr @wendigo @findepi @hashhar