-
Notifications
You must be signed in to change notification settings - Fork 441
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SDK] test: Add e2e test for tune function. #2399
Merged
Merged
Changes from all commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
e195c74
fix(sdk): fix error field metrics_collector in tune function.
Electronic-Waste 87a3a65
test(sdk): Add e2e tests for tune function.
Electronic-Waste 153cdef
test(sdk): add missing field parameters.
Electronic-Waste b43f603
refactor(test/sdk): add run-e2e-tune-api.py.
Electronic-Waste 6925087
test(sdk): delete tune testing code in run-e2e-experiment.
Electronic-Waste c69cda6
test(sdk): add blank lines.
Electronic-Waste 4edc40c
test(sdk): add verbose and temporarily delete e2e-experiment test.
Electronic-Waste 92a3bea
test(sdk): add namespace_labels.
Electronic-Waste e61bc6d
test(sdk): add time.sleep(5).
Electronic-Waste 71d3aac
test(sdk): add error output.
Electronic-Waste 6ae4616
test(sdk): build random image for tune.
Electronic-Waste 72b3b48
test(sdk): delete extra debug log.
Electronic-Waste a86a0ae
refactor(test/sdk): create separate workflow for tune.
Electronic-Waste 02b90b2
test(sdk): change api to API.
Electronic-Waste aec2649
test(sdk): change the permission of scripts.
Electronic-Waste b7140d2
test(sdk): delete exit code & comment image pulling.
Electronic-Waste 108ec4a
test(sdk): delete image pulling phase.
Electronic-Waste ef7a05c
test(sdk): refactor workflow file to use template.
Electronic-Waste 33fec7b
test(sdk): mark experiments and trial-images as not required.
Electronic-Waste 42a5d3e
test(sdk): pass tune-api param to setup-minikube.sh.
Electronic-Waste 6beeeaa
test(sdk): fix err in template-e2e-test.
Electronic-Waste dde4402
test(sdk): add debug logs.
Electronic-Waste c962aa5
test(sdk): reorder params and delete logs.
Electronic-Waste File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
name: E2E Test with tune API | ||
|
||
on: | ||
pull_request: | ||
paths-ignore: | ||
- "pkg/ui/v1beta1/frontend/**" | ||
|
||
concurrency: | ||
group: ${{ github.workflow }}-${{ github.ref }} | ||
cancel-in-progress: true | ||
|
||
jobs: | ||
e2e: | ||
runs-on: ubuntu-22.04 | ||
timeout-minutes: 120 | ||
steps: | ||
- name: Checkout | ||
uses: actions/checkout@v4 | ||
|
||
- name: Setup Test Env | ||
uses: ./.github/workflows/template-setup-e2e-test | ||
with: | ||
kubernetes-version: ${{ matrix.kubernetes-version }} | ||
|
||
- name: Run e2e test with tune API | ||
uses: ./.github/workflows/template-e2e-test | ||
with: | ||
tune-api: true | ||
|
||
strategy: | ||
fail-fast: false | ||
matrix: | ||
# Detail: https://hub.docker.com/r/kindest/node | ||
kubernetes-version: ["v1.27.11", "v1.28.7", "v1.29.2"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,97 @@ | ||
import argparse | ||
import logging | ||
|
||
from kubeflow.katib import KatibClient | ||
from kubeflow.katib import search | ||
from kubernetes import client | ||
from verify import verify_experiment_results | ||
|
||
# Experiment timeout is 40 min. | ||
EXPERIMENT_TIMEOUT = 60 * 40 | ||
|
||
# The default logging config. | ||
logging.basicConfig(level=logging.INFO) | ||
|
||
|
||
def run_e2e_experiment_create_by_tune( | ||
katib_client: KatibClient, | ||
exp_name: str, | ||
exp_namespace: str, | ||
): | ||
# Create Katib Experiment and wait until it is finished. | ||
logging.debug("Creating Experiment: {}/{}".format(exp_namespace, exp_name)) | ||
|
||
# Use the test case from get-started tutorial. | ||
# https://www.kubeflow.org/docs/components/katib/getting-started/#getting-started-with-katib-python-sdk | ||
# [1] Create an objective function. | ||
def objective(parameters): | ||
import time | ||
time.sleep(5) | ||
result = 4 * int(parameters["a"]) - float(parameters["b"]) ** 2 | ||
print(f"result={result}") | ||
|
||
# [2] Create hyperparameter search space. | ||
parameters = { | ||
"a": search.int(min=10, max=20), | ||
"b": search.double(min=0.1, max=0.2) | ||
} | ||
|
||
# [3] Create Katib Experiment with 4 Trials and 2 CPUs per Trial. | ||
# And Wait until Experiment reaches Succeeded condition. | ||
katib_client.tune( | ||
name=exp_name, | ||
namespace=exp_namespace, | ||
objective=objective, | ||
parameters=parameters, | ||
objective_metric_name="result", | ||
max_trial_count=4, | ||
resources_per_trial={"cpu": "2"}, | ||
) | ||
experiment = katib_client.wait_for_experiment_condition( | ||
exp_name, exp_namespace, timeout=EXPERIMENT_TIMEOUT | ||
) | ||
|
||
# Verify the Experiment results. | ||
verify_experiment_results(katib_client, experiment, exp_name, exp_namespace) | ||
|
||
# Print the Experiment and Suggestion. | ||
logging.debug(katib_client.get_experiment(exp_name, exp_namespace)) | ||
logging.debug(katib_client.get_suggestion(exp_name, exp_namespace)) | ||
|
||
|
||
if __name__ == "__main__": | ||
parser = argparse.ArgumentParser() | ||
parser.add_argument( | ||
"--namespace", type=str, required=True, help="Namespace for the Katib E2E test", | ||
) | ||
parser.add_argument( | ||
"--verbose", action="store_true", help="Verbose output for the Katib E2E test", | ||
) | ||
args = parser.parse_args() | ||
|
||
if args.verbose: | ||
logging.getLogger().setLevel(logging.DEBUG) | ||
|
||
katib_client = KatibClient() | ||
|
||
namespace_labels = client.CoreV1Api().read_namespace(args.namespace).metadata.labels | ||
if 'katib.kubeflow.org/metrics-collector-injection' not in namespace_labels: | ||
namespace_labels['katib.kubeflow.org/metrics-collector-injection'] = 'enabled' | ||
client.CoreV1Api().patch_namespace(args.namespace, {'metadata': {'labels': namespace_labels}}) | ||
|
||
# Test with run_e2e_experiment_create_by_tune | ||
exp_name = "tune-example" | ||
exp_namespace = args.namespace | ||
try: | ||
run_e2e_experiment_create_by_tune(katib_client, exp_name, exp_namespace) | ||
logging.info("---------------------------------------------------------------") | ||
logging.info(f"E2E is succeeded for Experiment created by tune: {exp_namespace}/{exp_name}") | ||
except Exception as e: | ||
logging.info("---------------------------------------------------------------") | ||
logging.info(f"E2E is failed for Experiment created by tune: {exp_namespace}/{exp_name}") | ||
raise e | ||
finally: | ||
# Delete the Experiment. | ||
logging.info("---------------------------------------------------------------") | ||
logging.info("---------------------------------------------------------------") | ||
katib_client.delete_experiment(exp_name, exp_namespace) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tenzen-y Does it look to you on how we separate the
template-e2e-test
?