Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Start triggering testing at finer granularity in the repo #595

Merged
merged 65 commits into from
Sep 19, 2024
Merged

Start triggering testing at finer granularity in the repo #595

merged 65 commits into from
Sep 19, 2024

Conversation

daw3rd
Copy link
Member

@daw3rd daw3rd commented Sep 17, 2024 •
edited
Loading

This PR splits out the tests from test.yml into separate tests for each transform and kfp and the core libraries.
Each workflow is now targeted to a subdirectory of the repository so fewer full builds should happen.
The workflows directory contains a Makefile and test-transforms.template file that is used to generate
the test-*-*.yml workflows for each of the transforms in the repository.

When adding a new transform, the Makefile will need to be edited to include the new transform and make will need to be run to create the transform's workflow.

Also, this should help the space problems as each workflow only handles 1 transform.

Also :), this should make the build (of transform changes) faster since only the changed transform is processed.

Why are these changes needed?

To speed up the ci/cd builds.

Related issue number (if any).

#515

@daw3rd
disable test workflow when none code files change
3fe8d2c 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
one more path-ignore in test.yml
86951f8 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
one more fix for path-ignore in test.yml
d368dc7 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
test universal transform separately
6d7c186 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
rename test universal workflow
35f9f31 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
add comments to noop src to trigger new universal test workflow
37cd7ef 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
fix paths in test universal workflow
a7fa50b 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
addj back ignore paths in test universal workflow
300d06a 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
another noop comment
0154179 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
move ignored paths to paths in univesal test workflow
5dd78e6 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
test-universal workflow name changes
d91da0d 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop comments
1be3292 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop readme change'
54d48b6 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
change test universal not paths
288ffdf 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
disable all but new noop and doc_id test workflows
668b70d 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
code change in noop
aa24ee7 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
remake test transforms
1d72cf5 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
Merge branch 'dev' into cicd-opt
142da80
@daw3rd
add individual test transform workflows
6bee51b 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop README change
db18c74 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
better ignore of .md on test transform workflows
42e18fc 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop readme change
ca09d75 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop test transform worklow 1 ignore
f16780b 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop readme
189bdd7 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
split out the tests into test-kfp/lib/misc and remove test.yml, add r…
54b3c69 
…eadme

Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
test-kfp only on kfp/**
d530ac9 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop code change to trigger build
781113f 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
comments in workflows
f09fbee 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
fix typo in test template on check_images
684ae7a 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop src change
f419059 
Signed-off-by: David Wood <dawood@us.ibm.com>
roytman
roytman reviewed Sep 18, 2024
View reviewed changes
transforms/universal/filter/python/src/filter_local.py Outdated Show resolved Hide resolved
transforms/universal/noop/python/src/noop_transform.py Outdated Show resolved Hide resolved
.github/workflows/test.yml Outdated Show resolved Hide resolved
.github/workflows/Makefile Show resolved Hide resolved
.github/workflows/README.md Outdated Show resolved Hide resolved
.github/workflows/README.md Outdated Show resolved Hide resolved
.github/workflows/build-library.yml Show resolved Hide resolved
.github/workflows/build-library.yml Show resolved Hide resolved
.github/workflows/test-code-code2parquet.yml Show resolved Hide resolved
@daw3rd
check for makefile in test transform workflow
a887bf9 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
automatically determine transforms in transforms directory for which …
f9399c1 
…to generate test workflows

Signed-off-by: David Wood <dawood@us.ibm.com>
roytman
roytman approved these changes Sep 18, 2024
View reviewed changes
Copy link
Member

@roytman roytman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but leave it to @revit13 to check her comments

@daw3rd
worklow readme, transform existence verification, disable build-libra…
e064e00 
…ry, and tools tests

Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
workflow readme details on kfp and misc tests
3a81d21 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
backing out change to dpk lib code
5d957ed 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
restore filter code
b02b52c 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
restore noop code
7cb4390 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
workflow readme
b780cf5 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
really restore noop code
9a5ccdc 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
check for makefile in transform test-src testing
93c56c2 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
don't include lib test dependencies in transform test workflows
8f8592d 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop code change
40ce888 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
disable noop, don't include lib test-data in transform dependencies
cb0d1f9 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
use job.id.if on Makefile to enable transform test job
d6c5784 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
use job.id.if on Makefile to enable transform test job
4f6b2f2 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
restore noop Makefile
c0f4935 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
exclude kfp_ray from transfor test workflow and change noop code
0a09daa 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
remove if: from test workflows
1bfb33d 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
backout noop code change
a80c0fd 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
backout noop code change
ee08033 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd daw3rd merged commit e4dac0b into dev Sep 19, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@roytman roytman roytman approved these changes

@revit13 revit13 revit13 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@daw3rd @roytman @revit13