Skip to content

feat(source/bigquery): attach SQLCommenter attributes as BigQuery job labels - #3843

Open
spark2ignite wants to merge 3 commits into
googleapis:mainfrom
spark2ignite:feat/bigquery-sqlcommenter-job-labels
Open

feat(source/bigquery): attach SQLCommenter attributes as BigQuery job labels#3843
spark2ignite wants to merge 3 commits into
googleapis:mainfrom
spark2ignite:feat/bigquery-sqlcommenter-job-labels

Conversation

@spark2ignite

Copy link
Copy Markdown

Description

Enabling --sql-commenter currently has no effect on the BigQuery source:
the commenter is only wired into postgres/mysql-family sources, and BigQuery
would not retain SQL-text comments in job metadata anyway. As noted on the
issue, today exactly one attribute reaches the job — the automatic
mcp-toolbox-tool label added in v1.5.0 (#1975), which carries the tool
type (e.g. bigquery-execute-sql). The seven remaining SQLCommenter
attributes — per-user, per-agent, per-model attribution and the trace
linkage — never reach the job. This PR adds them. The SQLCommenter
tool.name attribute (the MCP tool name, e.g. execute_sql) is distinct
from the existing type label; both are kept, and explicit tool-supplied
labels always win on key collisions.

Following the direction discussed in the linked issue, this PR attaches the
SQLCommenter attributes as native job labels instead of prepending a
comment, so they surface first-class in INFORMATION_SCHEMA.JOBS, audit
logs, and billing exports — no query text parsing needed to recover them.

  • New sqlcommenter.Labels(...) returns the exact attribute set
    PrependComment uses (tool.name, client, client.model,
    client.user.id, client.agent.id, traceparent, server,
    db.system.name), sanitized to BigQuery label constraints: lowercase
    [a-z0-9_-], max 63 chars, keys start with a letter, dots mapped to
    underscores (tool.nametool_name).
  • Labels are merged in Source.RunSQL, so both bigquery-execute-sql and
    bigquery-sql are covered with no per-tool changes; ML/AI tools that don't
    flow through RunSQL are unaffected, matching the scope note on the issue.
    Explicit tool-supplied labels (e.g. mcp-toolbox-tool) win on collisions.
  • Gated identically to other sources: global --sql-commenter flag plus an
    optional per-source sqlCommenter override in tools.yaml, default off.
  • Docs: new "BigQuery: Job Labels" section in the SQL Commenter monitoring
    page (with an example INFORMATION_SCHEMA.JOBS attribution query) and a
    sqlCommenter row in the BigQuery source reference.

Tests: table-driven unit tests for key/value sanitization (uppercase,
illegal characters, truncation, leading non-letter keys), enable/override
precedence mirroring the existing PrependComment cases, label merge
precedence, and YAML config parsing. go test -race ./cmd/... ./internal/...
and golangci-lint run pass.

Also verified end-to-end against a live BigQuery project: with
sqlCommenter: true, a query issued through the MCP protocol (JSON-RPC
tools/call with _meta["dev.mcp-toolbox/telemetry"] set) produced a job
carrying all eight attributes as labels alongside the existing
mcp-toolbox-tool:

"labels": {
  "client": "select-e2e-client_0_0_1",
  "client_agent_id": "agent-e2e-42",
  "client_model": "test-model-1",
  "client_user_id": "vadim-e2e",
  "db_system_name": "bigquery",
  "mcp-toolbox-tool": "bigquery-execute-sql",
  "server": "genai-toolbox_1_9_0_dev_darwin_arm64",
  "tool_name": "execute_sql",
  "traceparent": "00-f58710a2aea7bf757cf9292b026a2bce-de6a2f6214d3f085-01"
}

With the flag off, the job carries only mcp-toolbox-tool — byte-identical
to current behavior. The labels are queryable in
INFORMATION_SCHEMA.JOBS_BY_PROJECT (grouping bytes billed by tool_name
and client works as expected).

PR Checklist

  • Make sure you reviewed
    CONTRIBUTING.md
  • Make sure to open an issue as a
    bug/issue
    before writing your code! That way we can discuss the change, evaluate
    designs, and agree on the general idea
  • Ensure you have manually reviewed the entire diff before requesting a
    review
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)
  • Make sure to add ! if this involve a breaking change

🛠️ Fixes #3286

BigQuery does not retain SQL-text comments in job metadata, so instead of
prepending a comment the SQLCommenter attributes (tool.name, client,
client.model, client.user.id, client.agent.id, traceparent, server,
db.system.name) are attached as native job labels on queries executed
through RunSQL. Labels surface in INFORMATION_SCHEMA.JOBS, audit logs, and
billing exports without query text parsing.

Keys and values are sanitized to BigQuery label constraints (lowercase
[a-z0-9_-], max 63 chars, keys start with a letter); explicit tool-supplied
labels win on collisions. Gated exactly like other sources: the global
--sql-commenter flag plus an optional per-source sqlCommenter override.
@spark2ignite
spark2ignite requested review from a team as code owners August 18, 2026 20:57

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces SQL Commenter support for the BigQuery source. Because BigQuery does not retain SQL-text comments in its logs, SQL Commenter attributes are attached as native job labels instead. The implementation includes sanitization logic to satisfy BigQuery's label constraints, a helper to merge explicit and commenter-derived labels, configuration updates, and comprehensive unit tests and documentation. No review comments were provided, so there is no additional feedback.

@spark2ignite

Copy link
Copy Markdown
Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for SQL Commenter in BigQuery by attaching telemetry attributes as native job labels instead of SQL-text comments, updating the configuration, documentation, and adding sanitization logic for BigQuery label constraints. The review feedback highlights a potential issue where sanitized label values starting with an underscore or dash would be rejected by BigQuery, and suggests prefixing them with 'x' along with adding corresponding test cases.

Comment thread internal/sources/sqlcommenter/sqlcommenter.go
Comment thread internal/sources/sqlcommenter/sqlcommenter_test.go
Label keys must begin with a lowercase letter, but values carry no
leading-character requirement — BigQuery accepts values starting with an
underscore, dash, or digit. Assert that the sanitizer preserves such
values rather than prefixing them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(tools/bigquery): Wire SQLCommenter for BigQuery source (mirror postgres/mysql pattern)

2 participants