Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

driver_matrix: Multi-stage pipeline tracking #289

Open
4 of 5 tasks
k0machi opened this issue Oct 24, 2023 · 6 comments · Fixed by #387
Open
4 of 5 tasks

driver_matrix: Multi-stage pipeline tracking #289

k0machi opened this issue Oct 24, 2023 · 6 comments · Fixed by #387
Assignees
@k0machi
Labels
Argus enhancement New feature or request refactor

Comments

@k0machi
Copy link
Collaborator

k0machi commented Oct 24, 2023 •
edited
Loading

Currently, driver matrix tests rely on filename parsing to determine which test they are running. In addition to that, the submission happens at the end of pipeline, losing some of the potentially useful information - like start time, overall pipeline health is guessed instead.

The replacement mechanism should function akin to SCT and provide following features:

  • Maintain a unique job id per matrix run
  • Create an empty entity when a run is started
  • Submit each test result separately as they complete
  • Use pipeline health indicator to provide general status - Failed, Passed, etc
  • Pass the test information (kind, driver) to the API.
@k0machi k0machi mentioned this issue Oct 24, 2023
Merged
@k0machi k0machi mentioned this issue Dec 20, 2023
Closed
@fruch fruch added enhancement New feature or request Argus labels Dec 20, 2023
@k0machi k0machi mentioned this issue Dec 21, 2023
Open
@fruch
Copy link
Contributor

fruch commented Jan 18, 2024

@k0machi

can you expend a bit, this sound like a change more then Argus itself.
can you estimate how long it would take ?

@k0machi k0machi changed the title driver_matrix: Replace filename parsing with job API similar to SCT driver_matrix: Adding Job API similar to SCT Jan 23, 2024
@k0machi
Copy link
Collaborator Author

k0machi commented Jan 23, 2024

Better title and time estimation added.

@k0machi k0machi changed the title driver_matrix: Adding Job API similar to SCT driver_matrix: Multi-stage pipeline tracking Jun 5, 2024
@k0machi
Copy link
Collaborator Author

k0machi commented Jun 25, 2024 •
edited
Loading

Current plans:

  • Switching from docker container to a standalone client - assuming the ec2-asg-x86 node pool we have has python 3.10
  • Making a cli interface for the driver results - skipping need for a custom python script that has to be maintained separately
  • Keeping legacy endpoint by checking schema version on the request
  • The new API will contain this flow:
    • Create run - Requires UUID (pipeline manager) and build environment information (separate parameters, undecided which ones are needed). Run will be created with status "CREATED"
    • Update run with build information if available
    • Submit result - Requires raw XML encoded as base64 inside the request json, as well as job id. First submission will set test status to "RUNNING" and apply heartbeat.
    • Fail result - Same as submit, probably under same command except different type, needs failure reason, run id. Won't change the status yet.
    • Finalize - Indicate that we have finished the pipeline. Backend will determine the results from that. Will need just the id.

@k0machi
Copy link
Collaborator Author

k0machi commented Jun 27, 2024 •
edited
Loading

As a note, we will need to update all driver matrix repositories to generate some kind of metadata file, as the current implementation runs every version on the same process and only outputs results at the end, so we will need some way to collect which file belongs to which version.

Additionally, adding a way to store logs from driver matrix results would be very useful, akin to how sct does it. Saving scylla nodes logs might prove tricky as that is more on the driver's test framework side than ours.

k0machi added a commit to k0machi/argus that referenced this issue Jul 15, 2024
@k0machi
refactor(plugins/driver_matrix_tests): Multi-stage pipeline for Drivers
bd0cdd2 
This commit reworks the way Driver Matrix Runs are submitted to Argus,
replacing previously one-shot architecture with a multi-stage pipeline
akin to SCT, where driver matrix run first submits itself, and then
submits results sequentially (either a pass or failure), finalizing with
a call that makes backend determine status based on resulting payload.
Patch failures are now supported. Entire logic was moved from the client
to the backend, removing the need to update the client in case of large
changes to the API / submitted data.

Fixes scylladb#289
@k0machi k0machi mentioned this issue Jul 15, 2024
Merged
k0machi added a commit to k0machi/argus that referenced this issue Jul 15, 2024
@k0machi
refactor(plugins/driver_matrix_tests): Multi-stage pipeline for Drivers
bd19c0d 
This commit reworks the way Driver Matrix Runs are submitted to Argus,
replacing previously one-shot architecture with a multi-stage pipeline
akin to SCT, where driver matrix run first submits itself, and then
submits results sequentially (either a pass or failure), finalizing with
a call that makes backend determine status based on resulting payload.
Patch failures are now supported. Entire logic was moved from the client
to the backend, removing the need to update the client in case of large
changes to the API / submitted data.

Fixes scylladb#289
k0machi added a commit to k0machi/argus that referenced this issue Jul 15, 2024
@k0machi
refactor(plugins/driver_matrix_tests): Multi-stage pipeline for Drivers
0883557 
This commit reworks the way Driver Matrix Runs are submitted to Argus,
replacing previously one-shot architecture with a multi-stage pipeline
akin to SCT, where driver matrix run first submits itself, and then
submits results sequentially (either a pass or failure), finalizing with
a call that makes backend determine status based on resulting payload.
Patch failures are now supported. Entire logic was moved from the client
to the backend, removing the need to update the client in case of large
changes to the API / submitted data.

Fixes scylladb#289
k0machi added a commit to k0machi/argus that referenced this issue Jul 15, 2024
@k0machi
refactor(plugins/driver_matrix_tests): Multi-stage pipeline for Drivers
96c74d7 
This commit reworks the way Driver Matrix Runs are submitted to Argus,
replacing previously one-shot architecture with a multi-stage pipeline
akin to SCT, where driver matrix run first submits itself, and then
submits results sequentially (either a pass or failure), finalizing with
a call that makes backend determine status based on resulting payload.
Patch failures are now supported. Entire logic was moved from the client
to the backend, removing the need to update the client in case of large
changes to the API / submitted data.

Fixes scylladb#289
k0machi added a commit to k0machi/argus that referenced this issue Jul 15, 2024
@k0machi
refactor(plugins/driver_matrix_tests): Multi-stage pipeline for Drivers
46df51e 
This commit reworks the way Driver Matrix Runs are submitted to Argus,
replacing previously one-shot architecture with a multi-stage pipeline
akin to SCT, where driver matrix run first submits itself, and then
submits results sequentially (either a pass or failure), finalizing with
a call that makes backend determine status based on resulting payload.
Patch failures are now supported. Entire logic was moved from the client
to the backend, removing the need to update the client in case of large
changes to the API / submitted data.

Fixes scylladb#289
k0machi added a commit to k0machi/argus that referenced this issue Jul 22, 2024
@k0machi
refactor(plugins/driver_matrix_tests): Multi-stage pipeline for Drivers
01016c9 
This commit reworks the way Driver Matrix Runs are submitted to Argus,
replacing previously one-shot architecture with a multi-stage pipeline
akin to SCT, where driver matrix run first submits itself, and then
submits results sequentially (either a pass or failure), finalizing with
a call that makes backend determine status based on resulting payload.
Patch failures are now supported. Entire logic was moved from the client
to the backend, removing the need to update the client in case of large
changes to the API / submitted data.

Fixes scylladb#289
k0machi added a commit to k0machi/python-driver-matrix that referenced this issue Jul 22, 2024
@k0machi
run.py: Introduce metadata files for runs
11c2a20 
This commit adds a new file emitted by the matrix runs, intended to
provide metadata about the run for reporting tools that are executed
later, such as Argus. Currently, the metadata file always contains the
driver name (derived from result xml name), type (in this case it's
always python), and either failure_reason key (which contains exception
stack trace that happened during .run() or path to the resulting junit
xml file relative to the metadata file.

This commit is part of ongoing task to improve matrix pipeline reporting
to Argus.

Task: scylladb/argus#289
@k0machi k0machi mentioned this issue Jul 22, 2024
Merged
k0machi added a commit to k0machi/argus that referenced this issue Jul 22, 2024
@k0machi
refactor(plugins/driver_matrix_tests): Multi-stage pipeline for Drivers
0227a70 
This commit reworks the way Driver Matrix Runs are submitted to Argus,
replacing previously one-shot architecture with a multi-stage pipeline
akin to SCT, where driver matrix run first submits itself, and then
submits results sequentially (either a pass or failure), finalizing with
a call that makes backend determine status based on resulting payload.
Patch failures are now supported. Entire logic was moved from the client
to the backend, removing the need to update the client in case of large
changes to the API / submitted data.

Fixes scylladb#289
k0machi added a commit to k0machi/python-driver-matrix that referenced this issue Jul 22, 2024
@k0machi
run.py: Introduce metadata files for runs
ac9dddb 
This commit adds a new file emitted by the matrix runs, intended to
provide metadata about the run for reporting tools that are executed
later, such as Argus. Currently, the metadata file always contains the
driver name (derived from result xml name), type (in this case it's
always python), and either failure_reason key (which contains exception
stack trace that happened during .run() or path to the resulting junit
xml file relative to the metadata file.

This commit is part of ongoing task to improve matrix pipeline reporting
to Argus.

Task: scylladb/argus#289
k0machi added a commit to k0machi/argus that referenced this issue Jul 25, 2024
@k0machi
refactor(plugins/driver_matrix_tests): Multi-stage pipeline for Drivers
32cd16d 
This commit reworks the way Driver Matrix Runs are submitted to Argus,
replacing previously one-shot architecture with a multi-stage pipeline
akin to SCT, where driver matrix run first submits itself, and then
submits results sequentially (either a pass or failure), finalizing with
a call that makes backend determine status based on resulting payload.
Patch failures are now supported. Entire logic was moved from the client
to the backend, removing the need to update the client in case of large
changes to the API / submitted data.

Fixes scylladb#289
fruch pushed a commit to scylladb/python-driver-matrix that referenced this issue Aug 26, 2024
@k0machi @fruch
run.py: Introduce metadata files for runs
dcbe423 
This commit adds a new file emitted by the matrix runs, intended to
provide metadata about the run for reporting tools that are executed
later, such as Argus. Currently, the metadata file always contains the
driver name (derived from result xml name), type (in this case it's
always python), and either failure_reason key (which contains exception
stack trace that happened during .run() or path to the resulting junit
xml file relative to the metadata file.

This commit is part of ongoing task to improve matrix pipeline reporting
to Argus.

Task: scylladb/argus#289
k0machi added a commit that referenced this issue Aug 27, 2024
@k0machi
refactor(plugins/driver_matrix_tests): Multi-stage pipeline for Drivers
32c2080 
This commit reworks the way Driver Matrix Runs are submitted to Argus,
replacing previously one-shot architecture with a multi-stage pipeline
akin to SCT, where driver matrix run first submits itself, and then
submits results sequentially (either a pass or failure), finalizing with
a call that makes backend determine status based on resulting payload.
Patch failures are now supported. Entire logic was moved from the client
to the backend, removing the need to update the client in case of large
changes to the API / submitted data.

Fixes #289
@fruch fruch reopened this Sep 12, 2024
@fruch
Copy link
Contributor

fruch commented Sep 12, 2024

@k0machi

reopen this one, since we didn't finish the work for all of the matrixes,
please point from here to the PRs or other related work.

k0machi added a commit to k0machi/scylla-java-driver-matrix that referenced this issue Sep 13, 2024
@k0machi
run.py: Introduce metadata files for results
d47b95e 
This commit adds metadata files to java matrix reports, allowing argus
to correctly collect each version separately.

Task: scylladb/argus#289
@k0machi k0machi mentioned this issue Sep 13, 2024
Merged
k0machi added a commit to k0machi/scylla-java-driver-matrix that referenced this issue Sep 13, 2024
@k0machi
run.py: Introduce metadata files for results
c8c79f7 
This commit adds metadata files to java matrix reports, allowing argus
to correctly collect each version separately.

Task: scylladb/argus#289
k0machi added a commit to k0machi/scylla-cpp-driver-matrix that referenced this issue Sep 13, 2024
@k0machi
run.py: Introduce metadata files for results
695aef3 
This commit introduces results metadata files for collecting by argus
scripts within scylla-pkg.

Task: scylladb/argus#289
@k0machi k0machi mentioned this issue Sep 13, 2024
Merged
k0machi added a commit to k0machi/scylla-java-driver-matrix that referenced this issue Sep 13, 2024
@k0machi
run.py: Introduce metadata files for results
9146549 
This commit adds metadata files to java matrix reports, allowing argus
to correctly collect each version separately.

Task: scylladb/argus#289
k0machi added a commit to k0machi/gocql-driver-matrix that referenced this issue Sep 17, 2024
@k0machi
run.py: Add metadata file support
d054cb3 
This commit adds metadata files support to make it possible for Argus to
upload individual runs to itself using the client application.

Task: scylladb/argus#289
@k0machi k0machi mentioned this issue Sep 17, 2024
Merged
fruch pushed a commit to scylladb/scylla-java-driver-matrix that referenced this issue Sep 17, 2024
@k0machi @fruch
run.py: Introduce metadata files for results
6d35c54 
This commit adds metadata files to java matrix reports, allowing argus
to correctly collect each version separately.

Task: scylladb/argus#289
fruch pushed a commit to scylladb/gocql-driver-matrix that referenced this issue Sep 17, 2024
@k0machi @fruch
run.py: Add metadata file support
12f0b52 
This commit adds metadata files support to make it possible for Argus to
upload individual runs to itself using the client application.

Task: scylladb/argus#289
fruch pushed a commit to scylladb/scylla-cpp-driver-matrix that referenced this issue Sep 19, 2024
@k0machi @fruch
run.py: Introduce metadata files for results
488f3df 
This commit introduces results metadata files for collecting by argus
scripts within scylla-pkg.

Task: scylladb/argus#289
@fruch
Copy link
Contributor

fruch commented Sep 23, 2024

  • Waiting on for rust
  • and then update all of the pipelines

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@k0machi k0machi

Labels
Argus enhancement New feature or request refactor
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

refactor(plugins/driver_matrix_tests): Multi-stage pipeline for Drivers k0machi/argus
2 participants
@fruch @k0machi