-
Notifications
You must be signed in to change notification settings - Fork 697
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Rel-Eng]: Adds script to automatically generate GHCup metadata for Cabal releases #9411
[Rel-Eng]: Adds script to automatically generate GHCup metadata for Cabal releases #9411
Conversation
The usage and current output result after executing the script can be seen as below:- usage
or, without the quotes
both should work. The outputThe script being used as mentioned above generates the metadata for the release in the following format, which can be used by GHCup drop-in. The output of the script for release version "3.10.2.0" can be seen below.
Notes on output :-The output for release "3.10.2.0" from running this script is almost identical when compared to what is currently deployed at GHCup for this release . The SHA256 hashes match and the urls are identical (except for the ~cabal part). |
|
||
RELEASE=$1 | ||
## FixMe:// What dir to use here? | ||
## cd "gh-release-artifacts/Cabal-${RELEASE}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@hasufell what would the dir here be?
It just occurred to me that they (@Kleidukos , @hasufell ) might have notifications off for draft PRs. |
@arjunkathuria indeed, drafts are not meant to be reviewed. :) |
@arjunkathuria can you run |
@Kleidukos just did, came out clean. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only one change and it should be good 👍
This script only works well in conjunction with a script that downloads artifacts into a specific dir. |
Yup, that's what i was wondering, if we'd need to explicitly download them or run this script on a dir that already contains them somehow. I'll need to add that one too right ? |
The other option is to run it in CI, but that's not worth it imo. One has to download the artifacts either way to upload them to downloads.haskell.org. I don't think haskell infra allows automatic uploads through machine keys (I hope they don't), so it's a manual step. |
I see. |
Added the script to download the binary release files on a release number basis and verify the downloaded checksums on them. To be used in conjunction with the GHCup metadata generating script. |
|
||
## Download release files | ||
echo "Downloading form: \"https://downloads.haskell.org/~cabal/cabal-install-${RELEASE}/\"" | ||
wget --no-parent -r --reject "index.html*" --no-directories "https://downloads.haskell.org/~cabal/cabal-install-${RELEASE}/" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the wrong way around and would mean we download twice: once manually and once after we uploaded (why?).
The artifacts are built on gitlab CI. The release manager will have to download them to their local machine in order to upload them to downloads.haskell.org. And they want to perform the yaml step after they downloaded from gitlab, not after they uploaded to the download server.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. Since we don't seem to do github binary releases for cabal-install, i suppose i'd have to download them from upstream links like https://downloads.haskell.org/~cabal/cabal-install-3.10.1.0/ ?
could've mentioned here when i asked to confirm.
What's the correct url here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chreekat might know
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🕯️ * summons @chreekat * 🕯️
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👻
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you think that this is better suited as its own issue/PR pair?
The scope of this particular PR was to make a script that generates metadata for GHCup for a cabal release.
I think we can iterate on the how-to and where-to download parts that make releases easier as its own rel-eng issue as it would evolve into its own discussion/interface requirements.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're asking for the URL to download artifacts from gitlab.
That's why I suggest talking to a past/current Cabal release manager. I imagine they already have a workflow for this. While I can point to release pipelines and explain in general how to find the artifacts, it would be better to know what the actual current process is. Maybe @Kleidukos can help.
Do you think that this is better suited as its own issue/PR pair?
The scope of this particular PR was to make a script that generates metadata for GHCup for a cabal release.
Yeah, I totally agree with that.
To actually automate it, it would be a lot easier to run the script in the GitLab pipeline that generates the bindists in the first place. That's what I can help with. https://github.com/haskell/cabal/blob/master/.gitlab-ci.yml is the entrypoint.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the best course of action is to purge gitlab CI.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You may want to re-purpose the tools of the GHC release engineering team:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, they do it in a couple of steps.
- get the job ids from a pipeline id.
- try to get artifacts from these job ids and error if anyone of those failed.
- If successful, generate metadata from these artifacts.
step 3. is mostly done here.
i suppose i can try repurposing the logic in steps 1. and 2. from their python scripts in the shell script here.
If that's not required or if that needs to be a separate issue/PR pair or a broader discussion, please do let me know.
09ca001
to
677d7e8
Compare
I migrated release CI to github and also added the yaml snippet and download scripts. That makes this PR obsolete. |
You mean, "obsolete in the future"? That's what all PRs are. :) I'm guessing (@Kleidukos, correct me if I'm wrong), in the best case, we'd migrate to exclusively github for release after a couple of release using both gitlab and github and fine-tuning the new (github) method. For these couple of releases, this PR is still likely to be a great help (and possibly for the fine-tuning). Still, it's fair heads-up to @arjunkathuria, in case this PR needs a lot of work still and, presumably, it's life-span would be shorter than initially expected. So, @arjunkathuria, it's your call and, in any case, the work you spent on that is very much appreciated. |
Thanks @Mikolaj Ah well. I'd be alright to close this when #9437 gets merged, as it probably supersedes this. I'd like to thank everyone involved @Mikolaj @hasufell @Kleidukos @chreekat @ulysses4ever and @Bodigrim for their time and effort put into providing great feedback and reviewing this. |
just looked at #9437 to find that @hasufell closed it? unsure where this leaves this PR, any inputs @Mikolaj @Kleidukos @hasufell ? i'd be pretty alright to close this if the script isn't required/useful anymore or if the underlying assumptions and scope changed. |
I won't be using the bindists from cabal CI for ghcup, except for the |
@arjunkathuria: It seems we are going to have many releases soon, so your script is likely to be very useful, but the current release manager (@Kleidukos) would know best. |
Bump. cc: @Kleidukos |
Ok let's bring this in. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will, obviously, be refined on actual usage, but that's a solid start at least. Thank you!
[BTW, I haven't informed the git police yet, but the convention is that commit messages are sentences in imperative present tense without the ending dot ("Tweak wibbles in fobuscator").] |
CI seems stuck. A rebase would probably help. |
677d7e8
to
6f8c0b9
Compare
updated the commit messages as suggested, thanks!
Done, rebased to latest master
That's quite encouraging. Thanks again, for all your time and feedback on this @Kleidukos @Mikolaj ! |
@arjunkathuria: could you mark this PR as "Maintainers are allowed to edit this pull request. " so that mergify can rebase and merge it? Let me know if that's problematic. |
6f8c0b9
to
c5be3f5
Compare
Hi @Mikolaj, hope you've been well.
Hardly, Done! |
Adds the initial version of the release script that generates the metadata for a given release automatically in the correct format which can be used by GHCup as-is which makes it way easier to add that release to be distributed via GHCup. This would evolve based on the feedback from the maintainers but the basic core functionality still works. To Resolve: haskell#9298
Modifies the top shebang from being #!/bin/bash to #!/usr/bin/env bash as requested in the PR review
The script to generate GHCup metadata needs the artifacts already downloaded. This script was made to be used in conjunction to the ghcup metadata generating script. This script:- * Makes the directories that ./create-release-metadata-for-ghcup.sh CDs into * Downloads the binary release files given a particular release number into correct directories. * Verifies that the downloaded checksums match the expected values upstream.
… metadata generating script Checks if binary-release folders for the release passed exist. "create-release-metadata-for-ghcup.sh" needs the release files pre-downloaded. Echos user to run the download script first if correct binaries folder not found.
We do not want to commit the downloaded binary files for a cabal release. This prevents us from doing that, even on accident.
…load script Modifies the top shebang from being #!/bin/bash to #!/usr/bin/env bash in the 'download-cabal-install-release-binaries.sh' The later seems more portable as learned from the PR review comments.
c5be3f5
to
d5c49d4
Compare
Great. That was enough for mergify. Thank you again for your contribution! |
…abal releases (#9411) * [Rel-Eng]: Add initial version of the release metadata script Adds the initial version of the release script that generates the metadata for a given release automatically in the correct format which can be used by GHCup as-is which makes it way easier to add that release to be distributed via GHCup. This would evolve based on the feedback from the maintainers but the basic core functionality still works. To Resolve: #9298 * [Rel-Eng]: Change top shebang from '/bin/bash' to '/usr/bin/env bash' Modifies the top shebang from being #!/bin/bash to #!/usr/bin/env bash as requested in the PR review * [Rel-Eng]: Add script to download cabal-install binaries for a release The script to generate GHCup metadata needs the artifacts already downloaded. This script was made to be used in conjunction to the ghcup metadata generating script. This script:- * Makes the directories that ./create-release-metadata-for-ghcup.sh CDs into * Downloads the binary release files given a particular release number into correct directories. * Verifies that the downloaded checksums match the expected values upstream. * [Rel-Eng]: Integrate the release binary downloading script into GHCup metadata generating script Checks if binary-release folders for the release passed exist. "create-release-metadata-for-ghcup.sh" needs the release files pre-downloaded. Echos user to run the download script first if correct binaries folder not found. * [Rel-Eng]: Add release-scripts binary-downloads to gitignore We do not want to commit the downloaded binary files for a cabal release. This prevents us from doing that, even on accident. * [Rel-Eng]: Change from '/bin/bash' to '/usr/bin/env bash' in the download script Modifies the top shebang from being #!/bin/bash to #!/usr/bin/env bash in the 'download-cabal-install-release-binaries.sh' The later seems more portable as learned from the PR review comments.
…abal releases (haskell#9411) * [Rel-Eng]: Add initial version of the release metadata script Adds the initial version of the release script that generates the metadata for a given release automatically in the correct format which can be used by GHCup as-is which makes it way easier to add that release to be distributed via GHCup. This would evolve based on the feedback from the maintainers but the basic core functionality still works. To Resolve: haskell#9298 * [Rel-Eng]: Change top shebang from '/bin/bash' to '/usr/bin/env bash' Modifies the top shebang from being #!/bin/bash to #!/usr/bin/env bash as requested in the PR review * [Rel-Eng]: Add script to download cabal-install binaries for a release The script to generate GHCup metadata needs the artifacts already downloaded. This script was made to be used in conjunction to the ghcup metadata generating script. This script:- * Makes the directories that ./create-release-metadata-for-ghcup.sh CDs into * Downloads the binary release files given a particular release number into correct directories. * Verifies that the downloaded checksums match the expected values upstream. * [Rel-Eng]: Integrate the release binary downloading script into GHCup metadata generating script Checks if binary-release folders for the release passed exist. "create-release-metadata-for-ghcup.sh" needs the release files pre-downloaded. Echos user to run the download script first if correct binaries folder not found. * [Rel-Eng]: Add release-scripts binary-downloads to gitignore We do not want to commit the downloaded binary files for a cabal release. This prevents us from doing that, even on accident. * [Rel-Eng]: Change from '/bin/bash' to '/usr/bin/env bash' in the download script Modifies the top shebang from being #!/bin/bash to #!/usr/bin/env bash in the 'download-cabal-install-release-binaries.sh' The later seems more portable as learned from the PR review comments.
Draft PR for feedback and suggestion for issue #9298
The problem
Currently, on a cabal release, for it to be released by GHCup, the metadata of the release required by GHCup is manually generated. The process seems quite arduous and is an overhead for a release manager.
The solution
This PR intends to introduce a script that would help make cabal releases easier and more streamlined by automatically generating the required cabal release metadata in the correct format that is usable by GHCup as-is. It then essentially becomes a simpler affair for the release manager to introduce the new release in GHCup.
The link in the issue 9298 thread was used as a reference for this script.
Current Progress
The basic core functionality of the script works, as in, if you give it a release number as it's first and only argument, it generates the metadata for that release which can be used drop-in by GHCup. An example of the output to follow in the first comment of the PR.
More sections and info TBD
Todos
scripts/release
directory.