Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingest pipeline breaks order after last file #63

Open
juagargi opened this issue Apr 17, 2024 · 0 comments
Open

Ingest pipeline breaks order after last file #63

juagargi opened this issue Apr 17, 2024 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@juagargi
Copy link
Member

juagargi commented Apr 17, 2024

The ingest command ingests files into the DB.

The processing pipeline is:

  1. Open file in Processor
  2. Parse file, send each certificate to CertificateProcessor
  3. In multiple routines, create batches of certificates, in CertificateProcessor
  4. For each batch, in an independent routine, update the DB.
  5. The CertificateProcessor waits until all its batches are updated.
  6. The Processor waits until the CertificateProcessor is done
  7. The Processor calls OnBundleFinished if the # of certs is a bundle, or at end

The bug appears:

  • When there is no bundle size defined: the call to OnBundleFinished is done before the CertProcessor is done.
  • When there is a defined bundle size: the CertProcessor is still running.

This bug makes the last batches not to be ready for the next steps, done on OnBundleFinished,
thus creating an incomplete view of the certificate landscape represented by the files.

We need to:

  • Just move the code to solve the first bug.
  • On each bundle:
    • Close the parsedCertCh and wait for CertProcessor to finish.
    • Create a new parsedCertCh and a new CertProcessor
@juagargi juagargi added the bug Something isn't working label Apr 17, 2024
@juagargi juagargi self-assigned this Apr 17, 2024
juagargi added a commit that referenced this issue Apr 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant