Skip to content

fix(knowledge): harden ingestion pipeline - #7077

Merged
waleedlatif1 merged 5 commits into
stagingfrom
codex/knowledge-ingestion-hardening
Aug 26, 2026
Merged

fix(knowledge): harden ingestion pipeline#7077
waleedlatif1 merged 5 commits into
stagingfrom
codex/knowledge-ingestion-hardening

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • Harden connector pagination, hydration, reconciliation, and queue ownership boundaries
  • Bound parser, archive, chunking, embedding, and upstream error handling
  • Add durable quota continuation and explicit skipped and failed processing outcomes
  • Add backward-compatible queue-generation, deferred-processing, and sync-metric schema

Type of Change

  • Bug fix

Testing

  • bun run lint
  • bun run check:audits
  • bun run check:migrations origin/staging
  • bunx turbo run type-check
  • bun run test

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@greptile-apps

greptile-apps Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Too many files changed for review (128 files, 100 file limit).

Bypass the limit by tagging @greptile-apps to review.

@vercel

vercel Bot commented Aug 25, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview Aug 26, 2026 12:49am

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 128 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread apps/sim/lib/core/security/redaction.ts Outdated
Comment thread apps/sim/connectors/google-drive/google-drive-errors.ts
Comment thread apps/sim/lib/chunkers/json-yaml-chunker.ts Outdated
Comment thread apps/sim/lib/chunkers/structured-data-chunker.ts
Comment thread apps/sim/lib/file-parsers/yaml-parser.ts
Comment thread apps/sim/connectors/google-docs/google-docs.ts
Comment thread apps/sim/lib/file-parsers/opendocument-parser.ts
Comment thread apps/sim/lib/knowledge/documents/utils.ts Outdated
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 25, 2026

Copy link
Copy Markdown

@cubic

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 128 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread apps/sim/lib/core/security/redaction.ts Outdated
Comment thread apps/sim/lib/core/security/redaction.ts Outdated
Comment thread apps/sim/lib/core/security/redaction.ts Outdated
Comment thread apps/sim/lib/knowledge/documents/utils.ts
Comment thread apps/sim/lib/chunkers/structured-data-chunker.ts Outdated
Comment thread apps/sim/lib/knowledge/documents/utils.test.ts Outdated
Comment thread apps/sim/lib/chunkers/json-yaml-chunker.ts Outdated
@waleedlatif1
waleedlatif1 force-pushed the codex/knowledge-ingestion-hardening branch from 94fbb94 to 9c2d9d3 Compare August 25, 2026 22:00
@gitguardian

gitguardian Bot commented Aug 25, 2026

Copy link
Copy Markdown

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 25, 2026

Copy link
Copy Markdown

@cubic

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 25, 2026

Copy link
Copy Markdown

@cubic

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 129 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread apps/sim/lib/core/security/redaction.ts Outdated
Comment thread apps/sim/lib/chunkers/json-yaml-chunker.ts Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 129 files

Requires human review: Auto-approval blocked by 2 unresolved issues from previous reviews.

Re-trigger cubic

Comment thread apps/sim/lib/core/security/redaction.ts Outdated
Comment thread apps/sim/lib/chunkers/structured-data-chunker.ts Outdated
Comment thread apps/sim/lib/chunkers/json-yaml-chunker.ts Outdated
Comment thread apps/sim/lib/knowledge/documents/utils.test.ts
@waleedlatif1
waleedlatif1 force-pushed the codex/knowledge-ingestion-hardening branch from 28171cb to 1320c71 Compare August 25, 2026 22:40
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 25, 2026

Copy link
Copy Markdown

@cubic

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 129 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread apps/sim/lib/chunkers/json-yaml-chunker.ts Outdated
@waleedlatif1
waleedlatif1 force-pushed the codex/knowledge-ingestion-hardening branch 2 times, most recently from e0a511f to 8e034ed Compare August 25, 2026 23:17
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 25, 2026

Copy link
Copy Markdown

@cubic

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 129 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Re-trigger cubic

@waleedlatif1
waleedlatif1 force-pushed the codex/knowledge-ingestion-hardening branch from 8e034ed to f83396c Compare August 26, 2026 00:26
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 26, 2026

Copy link
Copy Markdown

@cubic

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 131 files

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Re-trigger cubic

Comment thread apps/sim/lib/file-parsers/json-parser.ts
@waleedlatif1
waleedlatif1 force-pushed the codex/knowledge-ingestion-hardening branch from f83396c to f13a7ef Compare August 26, 2026 00:44
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 26, 2026

Copy link
Copy Markdown

@cubic

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 133 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Re-trigger cubic

@waleedlatif1
waleedlatif1 merged commit 0c38037 into staging Aug 26, 2026
29 checks passed
@waleedlatif1
waleedlatif1 deleted the codex/knowledge-ingestion-hardening branch August 26, 2026 00:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant