Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

General setup overhaul #45

Merged
merged 33 commits into from
Jul 13, 2024
Merged

General setup overhaul #45

merged 33 commits into from
Jul 13, 2024

Conversation

tvoigtlaender
Copy link
Contributor

@tvoigtlaender tvoigtlaender commented Jun 18, 2024 •
edited
Loading

Overhaul of the KingMaker setup and environments

  • Update environments
  • Rework docker image workflow
  • remove clutter from setup.sh
  • Drop support for Centos8 and Ubuntu2004
  • Clean up unused files
  • Bugfixes

TODO:

  • Change back default tag of used docker images (used for testing)
  • Switch to main branch in submodules after they are merged
  • Rename envs to more meaningful names
  • Set up new /cvmfs envs for local use
  • Figure out bug with AlmaLinux9 setup
  • Update documentation

tvoigtlaender and others added 26 commits June 12, 2024 13:38
@tvoigtlaender
Remove unused files.
a12b536
@tvoigtlaender
update law
9285574
@tvoigtlaender
Update envs and configs
6865dfc
@tvoigtlaender
Update and clean up framework.py
ab9ee81
@tvoigtlaender
Clean up env setup of setup.sh\n Only set up starting env through cvm…
9c7f9ea 
…fs or miniforge. Give option to provide alternative source script in remote job. Remote tasks are assumed to have built in envs or use already available software like cvmfs LCG stacks.
@tvoigtlaender
Minor changes
ffcbc07
@tvoigtlaender
Clean up setup.sh further
761036f
@tvoigtlaender
Remove OS check during setup and move software for sample_manager int…
a3597b9 
…o KingMaker env. Update env names.
@tvoigtlaender
Change docker image tag to dev branch
54d1151
@tvoigtlaender
Update envs yml files
35ea7d2
@tvoigtlaender
update re-update KingMaker env based on original env. Fix issue in ta…
4529815 
…rball command.
Add rucio py-module to KingMaker env for sample_manager
e6f94a4
Seperate ML_train env into ROOT env and ML env due to size issues of …
6578578 
…single image
@tvoigtlaender
Return to previous naming of envs
6d25b13
@tvoigtlaender
Update ML_LAW env. Fix names
63df83a
@tvoigtlaender
Move env files to kingmaker-images. Add warning to sample manager. Ad…
523c56d 
…d kingmaker-images as submodule. Drop support for Centos8 and Ubuntu2004.
@tvoigtlaender
Enforce accepted OS
5676dc7
@tvoigtlaender
Syntax error
24cb5d2
@tvoigtlaender
Remove repeated print
1596efc
@tvoigtlaender
Disable usage of cvmfs for local envs until new envs are rolled out.
05df15b
@tvoigtlaender
Fix submodule
cd52fa3
@tvoigtlaender
Fix OS query for Alma9
145a241
@tvoigtlaender
Make OS parsing in framework.py more similar to parsing in os-version.sh
7237053
@tvoigtlaender
use same name for AlmaLinux in both OS checks (Alma)
21b503c
@tvoigtlaender
Update env submodule. Generalize compile script path fixing. Formating
d05ec84
@tvoigtlaender
Update xrootd to 5.7
4875acb
harrypuuter
harrypuuter approved these changes Jul 11, 2024
View reviewed changes
Copy link
Contributor

@harrypuuter harrypuuter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me!

processor/framework.py Outdated Show resolved Hide resolved
scripts/os-version.sh Outdated Show resolved Hide resolved
@tvoigtlaender
shortened target path for local cvmfs envs. Note: does not exist as o…
68be54a 
…f commit. Proper implementation of versioning remains to be discussed.
@tvoigtlaender
Copy link
Contributor Author

Local environments under /cvmfs/etp.kit.edu are not yet updated, and the alternative setup will be applied until it is.
This does not influence the code of this PR, as the env will be placed at the predefined path in the setup.
Proper setup for versioning of cvmfs setup remains to be discussed.

@tvoigtlaender
Copy link
Contributor Author

Waiting for KIT-CMS/kingmaker-images#8 to be merged to fix issue of cached docker images.

@tvoigtlaender tvoigtlaender merged commit 8766bee into main Jul 13, 2024
1 of 2 checks passed
@tvoigtlaender tvoigtlaender mentioned this pull request Jul 13, 2024
Closed
@tvoigtlaender tvoigtlaender deleted the update_setup_PR branch July 18, 2024 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@harrypuuter harrypuuter harrypuuter approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tvoigtlaender @harrypuuter