Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Various fixes to workflows, especially kfp #664

Merged
merged 87 commits into from
Oct 3, 2024
Merged

Various fixes to workflows, especially kfp #664

merged 87 commits into from
Oct 3, 2024

Conversation

daw3rd
Copy link
Member

@daw3rd daw3rd commented Oct 3, 2024

  1. Change kfp workflow to only run workflow-build on target transform
  2. Change transform Makefiles to check for kfp_ray dir before trying to recurse
  3. Add .make.* to most workflow dependencies
  4. Switch to ubuntu-latest (probably not needed, but was trying to workaround an action queuing problem).
  5. Move kfp blacklist definition to check-workflows.sh script.

Why are these changes needed?

To improve/fix ci/cd bugs

Related issue number (if any).

@daw3rd
disable test workflow when none code files change
3fe8d2c 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
one more path-ignore in test.yml
86951f8 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
one more fix for path-ignore in test.yml
d368dc7 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
test universal transform separately
6d7c186 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
rename test universal workflow
35f9f31 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
add comments to noop src to trigger new universal test workflow
37cd7ef 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
fix paths in test universal workflow
a7fa50b 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
addj back ignore paths in test universal workflow
300d06a 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
another noop comment
0154179 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
move ignored paths to paths in univesal test workflow
5dd78e6 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
test-universal workflow name changes
d91da0d 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop comments
1be3292 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop readme change'
54d48b6 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
change test universal not paths
288ffdf 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
disable all but new noop and doc_id test workflows
668b70d 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
code change in noop
aa24ee7 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
remake test transforms
1d72cf5 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
Merge branch 'dev' into cicd-opt
142da80
@daw3rd
add individual test transform workflows
6bee51b 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop README change
db18c74 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
better ignore of .md on test transform workflows
42e18fc 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop readme change
ca09d75 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop test transform worklow 1 ignore
f16780b 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop readme
189bdd7 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
split out the tests into test-kfp/lib/misc and remove test.yml, add r…
54b3c69 
…eadme

Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
test-kfp only on kfp/**
d530ac9 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
noop code change to trigger build
781113f 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
comments in workflows
f09fbee 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
updated workflow readme
193f439 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
only run build-library workflow on data-processing-lib changes
3ed0fbd 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
use job.id.if on Makefile to enable transform test job
4f6b2f2 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
restore noop Makefile
c0f4935 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
exclude kfp_ray from transfor test workflow and change noop code
0a09daa 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
remove if: from test workflows
1bfb33d 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
backout noop code change
a80c0fd 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
backout noop code change
ee08033 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
only build spark image for transform image tests that need it
30a1524 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
header_cleanser code change
6d6a233 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
Merge branch 'dev' into cicd-opt
b062ae0
@daw3rd
only build spark image for transform image tests that need it
f495f82 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
Merge branch 'dev' into cicd-opt
eac1c9c
@daw3rd
update workflows to trigger on .make.* changes
85f9bed 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
blank line added to .make.versions to test new workflows
19d3edf 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
remove change to .make.versions
1086fc8 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
remove test-universal-html2parquet.yml since it is now moved to language
5a44432 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
mv kfp blacklist definition to check-workflows.sh script and have it …
ad3347c 
…check for kfp workflows

Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
license_select_wf.py comment to trigger kfp tests here
e8611eb 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
have transform top level makefile check for kfp_ray directory before …
e369340 
…recursing into

Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
fix doc_quality Makefile kfp rules
602719a 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
fix tabbing in recent Makefile updates
5dcc092 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
fix kfp workflows to only build the target transform
7947986 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
switch workflows from ubuntu-22.04 to ubuntu-latest to try and avoid …
0fbb40e 
…excessive job queuing

Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
remove non-change from license_select_wf.py to trigger new ci/cd
a81b65a 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
Merge branch 'dev' into cicd-opt
a47656c
@daw3rd
one more fix to kfp transform workflows to not workflow-build in all …
6fd07b3 
…transforms

Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
fix repo_level_ordering/ray/Makefile to support kind/kfp testing targets
148b762 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
change kfp test workflows to not test if transform's Makefile or kfp_…
ed0556d 
…ray dir is not present

Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd
set cancel-in-progress=true in workflows
eadce89 
Signed-off-by: David Wood <dawood@us.ibm.com>
@daw3rd daw3rd merged commit a5f86da into dev Oct 3, 2024
108 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@revit13 revit13 revit13 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@daw3rd @revit13