[SDK] test: Add e2e test for tune function. #2399

Electronic-Waste · 2024-07-30T08:40:28Z

What this PR does / why we need it:

Adds an e2e test in the run_e2e_experiment.py for the tune function.
Change the error field name from metrics_collector to metrics_collector_spec

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2383

Checklist:

Docs included if any changes are user facing

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste · 2024-07-30T08:41:33Z

PTAL👀 @andreyvelich @johnugeorge @tenzen-y @tariq-hasan

Signed-off-by: Electronic-Waste <2690692950@qq.com>

andreyvelich

Thanks a lot for doing this @Electronic-Waste!
This should help us to make this API more stable.

test/e2e/v1beta1/scripts/gh-actions/run-e2e-experiment.sh

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste · 2024-08-01T09:23:18Z

Now run-e2e-tune-api.py has been added, and the error in CI/CD test is addressed.

Also, I found function verify_experiment_results is used in both the tune API test and the original test. So I move this function to verify.py and import it separately in run-e2e-tune-api.py and run-e2e-experiment.py.

Could you take a look?👀 @andreyvelich

test/e2e/v1beta1/scripts/gh-actions/run-e2e-experiment.sh

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste · 2024-08-01T14:56:39Z

@andreyvelich I've added a new workflow file e2e-test-tune-api.yaml to .github/workflows, and also, e2e-test-tune-api.sh to test/e2e/v1beta1/scripts/gh-actions. Now, the test for tune API is separate from other tests.

andreyvelich

Overall looks good, thank you @tenzen-y, just a few comments.
/assign @johnugeorge @tenzen-y

test/e2e/v1beta1/scripts/gh-actions/build-load.sh

test/e2e/v1beta1/scripts/gh-actions/run-e2e-tune-api.sh

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste · 2024-08-01T17:31:23Z

This PR is also related to my GSoC project.

ref issue: #2340

.github/workflows/e2e-test-tune-api.yaml

test/e2e/v1beta1/scripts/gh-actions/verify.py

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste · 2024-08-02T08:43:40Z

I've removed the time.sleep and refactored the workflow file with template.

PTAL👀 @tenzen-y if you are available.

.github/workflows/e2e-test-tune-api.yaml

andreyvelich · 2024-08-05T18:04:28Z

.github/workflows/e2e-test-tune-api.yaml

+          experiments: ""
+          # Comma Delimited
+          trial-images: ""
+          tune-api: true


@tenzen-y Does it look to you on how we separate the template-e2e-test ?

test/e2e/v1beta1/scripts/gh-actions/build-load.sh

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste · 2024-08-06T17:15:41Z

@andreyvelich I've modified this PR as you suggested in the last few comments. Could you plz take a look when you are available? Thanks so much!

andreyvelich

Thanks a lot for doing this @Electronic-Waste!
It should improve tune API stability.
/lgtm
/approve

google-oss-prow · 2024-08-06T17:28:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Electronic-Waste added 2 commits July 30, 2024 06:55

fix(sdk): fix error field metrics_collector in tune function.

e195c74

Signed-off-by: Electronic-Waste <2690692950@qq.com>

test(sdk): Add e2e tests for tune function.

87a3a65

Signed-off-by: Electronic-Waste <2690692950@qq.com>

google-oss-prow bot requested review from anencore94, gaocegege and johnugeorge July 30, 2024 08:40

google-oss-prow bot added the size/M label Jul 30, 2024

test(sdk): add missing field parameters.

153cdef

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste mentioned this pull request Jul 30, 2024

[GSoC] Project6: Push-based Metrics Collection for Katib #2340

Open

6 tasks

andreyvelich reviewed Jul 30, 2024

View reviewed changes

test/e2e/v1beta1/scripts/gh-actions/run-e2e-experiment.sh Outdated Show resolved Hide resolved

refactor(test/sdk): add run-e2e-tune-api.py.

b43f603

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste force-pushed the fix/tune branch from ec4f166 to b43f603 Compare July 30, 2024 18:04

google-oss-prow bot added size/L and removed size/M labels Jul 30, 2024

Electronic-Waste force-pushed the fix/tune branch 3 times, most recently from 6235a39 to b371915 Compare July 30, 2024 18:10

test(sdk): delete tune testing code in run-e2e-experiment.

6925087

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste force-pushed the fix/tune branch from b371915 to 6925087 Compare July 30, 2024 18:10

Electronic-Waste added 4 commits July 30, 2024 18:11

test(sdk): add blank lines.

c69cda6

Signed-off-by: Electronic-Waste <2690692950@qq.com>

test(sdk): add verbose and temporarily delete e2e-experiment test.

4edc40c

Signed-off-by: Electronic-Waste <2690692950@qq.com>

test(sdk): add namespace_labels.

92a3bea

Signed-off-by: Electronic-Waste <2690692950@qq.com>

test(sdk): add time.sleep(5).

e61bc6d

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste force-pushed the fix/tune branch from 9afd5c0 to e61bc6d Compare August 1, 2024 07:15

Electronic-Waste requested a review from andreyvelich August 1, 2024 07:58

Electronic-Waste added 3 commits August 1, 2024 08:27

test(sdk): add error output.

71d3aac

Signed-off-by: Electronic-Waste <2690692950@qq.com>

test(sdk): build random image for tune.

6ae4616

Signed-off-by: Electronic-Waste <2690692950@qq.com>

test(sdk): delete extra debug log.

72b3b48

Signed-off-by: Electronic-Waste <2690692950@qq.com>

andreyvelich reviewed Aug 1, 2024

View reviewed changes

test/e2e/v1beta1/scripts/gh-actions/run-e2e-experiment.sh Outdated Show resolved Hide resolved

Electronic-Waste added 3 commits August 1, 2024 14:25

refactor(test/sdk): create separate workflow for tune.

a86a0ae

Signed-off-by: Electronic-Waste <2690692950@qq.com>

test(sdk): change api to API.

02b90b2

Signed-off-by: Electronic-Waste <2690692950@qq.com>

test(sdk): change the permission of scripts.

aec2649

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste requested a review from andreyvelich August 1, 2024 14:49

andreyvelich reviewed Aug 1, 2024

View reviewed changes

google-oss-prow bot assigned johnugeorge and tenzen-y Aug 1, 2024

Electronic-Waste added 2 commits August 1, 2024 16:03

test(sdk): delete exit code & comment image pulling.

b7140d2

Signed-off-by: Electronic-Waste <2690692950@qq.com>

test(sdk): delete image pulling phase.

108ec4a

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste requested a review from andreyvelich August 1, 2024 16:24

tenzen-y reviewed Aug 1, 2024

View reviewed changes

.github/workflows/e2e-test-tune-api.yaml Outdated Show resolved Hide resolved

test/e2e/v1beta1/scripts/gh-actions/verify.py Outdated Show resolved Hide resolved

test(sdk): refactor workflow file to use template.

ef7a05c

Signed-off-by: Electronic-Waste <2690692950@qq.com>

Electronic-Waste requested a review from tenzen-y August 2, 2024 06:48

andreyvelich reviewed Aug 5, 2024

View reviewed changes

Electronic-Waste requested a review from andreyvelich August 6, 2024 14:01

Electronic-Waste added 5 commits August 6, 2024 14:09

test(sdk): mark experiments and trial-images as not required.

33fec7b

Signed-off-by: Electronic-Waste <2690692950@qq.com>

test(sdk): pass tune-api param to setup-minikube.sh.

42a5d3e

Signed-off-by: Electronic-Waste <2690692950@qq.com>

test(sdk): fix err in template-e2e-test.

6beeeaa

Signed-off-by: Electronic-Waste <2690692950@qq.com>

test(sdk): add debug logs.

dde4402

Signed-off-by: Electronic-Waste <2690692950@qq.com>

test(sdk): reorder params and delete logs.

c962aa5

Signed-off-by: Electronic-Waste <2690692950@qq.com>

andreyvelich approved these changes Aug 6, 2024

View reviewed changes

google-oss-prow bot assigned andreyvelich Aug 6, 2024

google-oss-prow bot added the lgtm label Aug 6, 2024

google-oss-prow bot added the approved label Aug 6, 2024

google-oss-prow bot merged commit b6f7cfd into kubeflow:master Aug 6, 2024
63 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SDK] test: Add e2e test for tune function. #2399

[SDK] test: Add e2e test for tune function. #2399

Electronic-Waste commented Jul 30, 2024

Electronic-Waste commented Jul 30, 2024

andreyvelich left a comment

Electronic-Waste commented Aug 1, 2024

Electronic-Waste commented Aug 1, 2024

andreyvelich left a comment

Electronic-Waste commented Aug 1, 2024

Electronic-Waste commented Aug 2, 2024

andreyvelich Aug 5, 2024

Electronic-Waste commented Aug 6, 2024

andreyvelich left a comment

google-oss-prow bot commented Aug 6, 2024

[SDK] test: Add e2e test for tune function. #2399

[SDK] test: Add e2e test for tune function. #2399

Conversation

Electronic-Waste commented Jul 30, 2024

Electronic-Waste commented Jul 30, 2024

andreyvelich left a comment

Choose a reason for hiding this comment

Electronic-Waste commented Aug 1, 2024

Electronic-Waste commented Aug 1, 2024

andreyvelich left a comment

Choose a reason for hiding this comment

Electronic-Waste commented Aug 1, 2024

Electronic-Waste commented Aug 2, 2024

andreyvelich Aug 5, 2024

Choose a reason for hiding this comment

Electronic-Waste commented Aug 6, 2024

andreyvelich left a comment

Choose a reason for hiding this comment

google-oss-prow bot commented Aug 6, 2024