GH-40924: [MATLAB][Packaging] Add script for uploading Release Candidate (RC) MLTBX packages for the MATLAB bindings to the Apache Arrow GitHub Releases area #40956

sgilmore10 · 2024-04-02T18:53:39Z

Rationale for this change

This is a follow up to #38660 and this mailing list discussion.

We would like to upload the MLTBX packages for the MATLAB bindings to the GitHub Releases area for the Apache Arrow GitHub project.

Using GitHub Releases will enable tight integration with the MathWorks File Exchange, which should make it easy for users to discover and install the MATLAB interface using MATLAB.

To make this possible, we need to update the release scripts under /dev/release to upload the MLTBX packages built by crossbow to the GitHub Releases area.

What changes are included in this PR?

Modified dev/release/05-binary-upload.sh to sign and upload release candidate MLTBX files to apache/arrow's GitHub Releases Area using the gh release create command.
Marked the created GitHub Release as a prerelease to differentiate between release candidates and official releases.
Compared the format of the created GitHub Releases to the format of apache/arrow-flight-sql-postgresql's GitHub Releases to validate our changes made sense.

Are these changes tested?

I was not able to run dev/release/05-binary-upload.sh directly, but I did test the gh create release, gpg, and shasum commands worked as expected by creating a "dummy" release for this repository: https://github.com/sgilmore10/TestRepo/releases/tag/1.0.0rc1

Are there any user-facing changes?

No.

Future Directions

Modify dev/release/post-02-upload.sh to create a non-prerelease GitHub Release for the MATLAB bindings.
Create an Apache Arrow entry on the MathWorks File Exchange to facilitate a one-click install workflow for the MATLAB bindings.
Refactor the changes made to 05-binary-upload.sh into a separate function that can be reused by post-02-upload.sh to reduce code duplication.

Open Questions

Should we include the list of issues resolved in the release notes for prereleases?
@kou, @assignUser, or @raulcd, do you have suggestions on how we can validate the changes to dev/release/05-binary-upload.sh?

GitHub Issue: [MATLAB][Packaging] Add script for uploading Release Candidate (RC) MLTBX packages for the MATLAB bindings to the Apache Arrow GitHub Releases area #40924

… Releases Area

github-actions · 2024-04-02T18:54:07Z

⚠️ GitHub issue #40924 has been automatically assigned in GitHub to PR creator.

2. Update git tag 3. Fix target branch name

kou · 2024-04-03T21:34:25Z

Ah, sorry for not sharing how to implement this in https://lists.apache.org/thread/hm7cbrkggomhx7nxdzr5xkrmkrztmmwo ...

How about using git tag -a apache-arrow-X.Y.Z-rcN, git push apache apache-arrow-X.Y.Z-rcN and GitHub Actions for the RC tag?

We can detect whether this push is for tag or commit by if: github.ref_type == 'tag'.
See also: https://docs.github.com/en/actions/learn-github-actions/contexts#github-context

If this push is for tag, we can download the built .mltbx file and upload it to GitHub Releases by GitHub Actions. We can't sign by GitHub Actions. So we need to sign on local later. We can do it by a release script.

This is a workflow what arrow-adbc already uses.
See also:

git tag: https://github.com/apache/arrow-adbc/blob/a35fae0f11c29b6207c1361986d7900441562d64/dev/release/01-prepare.sh#L76
Upload artifacts to GitHub Releases by GitHub Actions: https://github.com/apache/arrow-adbc/blob/a35fae0f11c29b6207c1361986d7900441562d64/.github/workflows/packaging.yml#L945-L999
Download artifacts from GitHub Releases, sign them and upload to GitHub Releases: https://github.com/apache/arrow-adbc/blob/main/dev/release/02-sign.sh
Change the associated tag of GitHub Releases from apache-arrow-adbc-X.Y.Z-rcN to apache-arrow-adbc-X.Y.Z after a vote is passed: https://github.com/apache/arrow-adbc/blob/main/dev/release/post-02-binary.sh
- I want to use different approach in apache/arrow for this. I don't want to change existing GitHub Releases for this. I want to create a new GitHub Releases for apache-arrow-X.Y.Z instead of reusing a GitHub Releases for apache-arrow-X.Y.Z-rcN.
- arrow-flight-sql-postgresql uses the different approach: https://github.com/apache/arrow-flight-sql-postgresql/blob/main/.github/workflows/release.yaml
  - git tag -a apache-arrow-X.Y.Z and git push apache apache-arrow-X.Y.Z trigger this GitHub Actions workflow.

assignUser · 2024-04-03T22:20:42Z

We can't sign by GitHub Actions

Not yet but the option exists! We can sign via gha workflow (in apache/arrow), INFRA has to review and approve the workflow doing the signing and will add the key to be used as a repo secret. See https://www.apache.org/legal/release-policy.html#release-signing

This change was added sometime last year but I never had the time to follow up but with the current discussions about making releases easier and this new addition it might be the right time to tackle it! (for both the monorepo and subprojects)

kou · 2024-04-04T02:03:33Z

#37350 is a related issue for automated release signing.

sgilmore10 · 2024-04-04T17:49:11Z

Hi @kou and @assignUser,

Thanks for sending us a lot of helpful resources on how to best cut a GitHub Release. @kevingurney and I will review them and re-work our implementation to use GitHub Actions.

To start, we may use the manual signing approach @kou laid out just to get something working, but we would like to transition to an automated signing approach as @assignUser suggested. We'd be happy to investigate how we can get approval from the ASF to automate signing our release artifacts.

Thanks again!
Sarah

assignUser · 2024-04-04T17:55:24Z

I just wanted to say I really appreciate your work on this and your style of communication!

Just to clarify the comment about the automated release signing was more a tangent and we would likely implement it for all things in the monorepo at once. So you can continue with the manual way for now with no problems!

sgilmore10 · 2024-04-04T18:09:02Z

Thanks for clarifying @assignUser! We're excited to be working with you guys on this!

sgilmore10 · 2024-04-04T18:46:08Z

As we were looking into this, @kevingurney and I realized we need to be able to connect the crossbow job ID that created the MLTBX file with the GitHub Actions workflow. We aren't aware of any existing mechanism that can provide inputs to a GitHub Actions Workflow that is trigged automatically. As an aside, it is possible to supply inputs to a GitHub Actions Workflow that is triggered manually (see this article).

One workaround to enable the automatic approach would be to embed the crossbow job ID in the git tag's message and then extract the ID in the workflow. Using this ID and the archery crossbow download-artifacts command, we could download the appropriate MLTBX file.

We recognize that embedding the ID in the git tag message is not ideal, so please let us know if you have any other suggestions.

2. Fix syntax errors

kou · 2024-04-05T02:42:42Z

We can use dev/release/04-binary-download.sh ${MAJOR}.${MINOR}.${PATCH} ${RC} --task-filter matlab to download the .mltbx file. :-)

raulcd · 2024-04-05T10:36:36Z

dev/release/05-binary-upload.sh

+  release_notes="Release Candidate: ${version} RC${rc}"
+  title="Apache Arrow ${version} RC${rc}"
+  repository="https://github.com/apache/arrow"
+  gh release create \


this means we requires the gh CLI installed locally, right?
Can we add a note on the requirements for the release tasks that we require gh CLI installed?
https://github.com/apache/arrow/blob/main/docs/source/developers/release.rst

Hi @raulcd,

This change is out of date - sorry about that. Based on @kou's feedback, we going to use a GitHub Actions Workflow - triggered by pushing the tag apache-arrow-X.Y.Z-rc{N} - to create release instead of using gh. I just haven't updated 05-binary-upload.sh yet, but will do so soon!

Actually, I just realized you are right. We will need to add a note about requiring the GItHub CLI to be installed locally. While we are no longer using gh to create the release in 04-binary-upload.sh, we will need to use gh once the GitHub release is created to upload the signed MLTBX files.

I'll make sure to add a note about this requirement to the release documentation.

Sorry about the confusion!

…idate Number as Environment Variables

2. Fix archery install path

2. ./dev/release/04-binary-download.sh -> ./arrow/release/04-binary-download.sh 3. Fix directory path provided to find

2. Set environment variable GH_TOKEN in Create GitHub Release for Release Candidate step

kou · 2024-04-05T23:29:47Z

dev/release/05-binary-upload.sh

+  release_tag="apache-arrow-${version_with_rc}"
+  tag_message="Release candidate: ${version_with_rc}"
+  git tag -a ${release_tag} -m ${tag_message}
+  git push apache ${release_tag}


Could you do this in dev/release/01-prepare.sh something like the following?

diff --git a/dev/release/01-prepare.sh b/dev/release/01-prepare.sh index 01fa2f3d80..f5ec0a8e26 100755 --- a/dev/release/01-prepare.sh +++ b/dev/release/01-prepare.sh @@ -33,7 +33,7 @@ next_version=$2 next_version_snapshot="${next_version}-SNAPSHOT" rc_number=$3 -release_tag="apache-arrow-${version}" +rc_tag="apache-arrow-${version}-rc${rc_number}" release_branch="release-${version}" release_candidate_branch="release-${version}-rc${rc_number}" @@ -45,9 +45,9 @@ release_candidate_branch="release-${version}-rc${rc_number}" : ${PREPARE_TAG:=${PREPARE_DEFAULT}} if [ ${PREPARE_TAG} -gt 0 ]; then - if [ $(git tag -l "${release_tag}") ]; then - echo "Delete existing git tag $release_tag" - git tag -d "${release_tag}" + if [ $(git tag -l "${rc_tag}") ]; then + echo "Delete existing git tag $rc_tag" + git tag -d "${rc_tag}" fi fi @@ -91,7 +91,7 @@ if [ ${PREPARE_LINUX_PACKAGES} -gt 0 ]; then fi if [ ${PREPARE_VERSION_PRE_TAG} -gt 0 ]; then - echo "Prepare release ${version} on tag ${release_tag} then reset to version ${next_version_snapshot}" + echo "Prepare release ${version} on tag ${rc_tag} then reset to version ${next_version_snapshot}" update_versions "${version}" "${next_version}" "release" git commit -m "MINOR: [Release] Update versions for ${version}" @@ -100,5 +100,5 @@ fi ############################## Tag the Release ############################## if [ ${PREPARE_TAG} -gt 0 ]; then - git tag -a "${release_tag}" -m "[Release] Apache Arrow Release ${version}" + git tag -a "${rc_tag}" -m "[Release] Apache Arrow Release ${version}" fi

Hi @kou,

Thanks for sharing this diff! We just have one clarification question:

Based on the diff you shared, it looks like you want us to replace the release_tag with the rc_tag. Did you mean for us to actually add the rc_tag in addition to the release_tag? We asking because we assume we still want to create a release_tag.

Did you mean for us to actually add the rc_tag in addition to the release_tag?

No. I mean for us to replace the release_tag with rc_tag.

We asking because we assume we still want to create a release_tag.

I want to change the current workflow:

Create a apache-arrow-X.Y.Z tag for X.Y.Z RC0

(Found a problem for X.Y.Z RC0)

Remove the apache-arrow-X.Y.Z tag for X.Y.Z RC0

Create a apache-arrow-X.Y.Z tag for X.Y.Z RC1

...

to the following:

Create a apache-arrow-X.Y.Z-rc0 tag for X.Y.Z RC0

(Found a problem for X.Y.Z RC0)

Create a apache-arrow-X.Y.Z-rc1 tag for X.Y.Z RC1

Vote

Passed

Create a apache-arrow-X.Y.Z tag from apache-arrow-X.Y.Z-rc1 like apache/arrow-adbc and apache/arrow-flight-sql-postgresql do

Hmm. It's better that we handle the workflow change related task in a separated issue...

Thanks for the clarification! I've created a issue #41102 to track this work. Once this PR is merged, I'll take ownership of that issue and make the required changes.

Hi @kou,

On second thought, we think it actually makes more sense to tackle #41102 first since the new GitHub Actions Workflow (package.yml) requires the release candidate tag (apache-arrow.X.Y.Z-rcN) to be pushed. We'll put this PR on pause for a bit until that #41102 is closed.

Best,
Sarah

It makes sense!

assignUser · 2024-04-25T16:43:10Z

Superseeded by #41131

…didates (e.g. apache-arrow-{MAJOR}.{MINOR}.{PATCH}-rc{RC_NUM}) (#41131) ### Rationale for this change As per @ kou's [suggestion](#40956 (comment)) in #40956, we should create unique git tags (e.g. `apache-arrow-{MAJOR}.{MINOR}.{VERSION}-rc{RC_NUM}`) instead re-using the same git tag (`apache-arrow-{MAJOR}.{MINOR}.{VERSION}`) for each release candidate. The official release candidate tag (`apache-arrow-{MAJOR}.{MINOR}.{VERSION}`) should be created **only** after a release candidate is voted on and accepted. This "official" release tag should point to the same object in the git database as the accepted release candidate tag. The new release workflow could look like the following: > 1. Create a apache-arrow-X.Y.Z-rc0 tag for X.Y.Z RC0 > 2. (Found a problem for X.Y.Z RC0) > 3. Create a apache-arrow-X.Y.Z-rc1 tag for X.Y.Z RC1 > 4. Vote > 5. Passed > 6. Create a apache-arrow-X.Y.Z tag from apache-arrow-X.Y.Z-rc1 ike apache/arrow-adbc and apache/arrow-flight-sql-postgresql do See @ kou's [comment](#40956 (comment)) for more details. ### What changes are included in this PR? 1. Updated `dev/release/01-prepare.sh` to create release-candidate-specific git tags (e.g. `apache-arrow-{MAJOR}.{MINOR}.{PATCH}-rc{RC_NUM}`). 2. Updated scripts in `dev/release` to use the new git tag name. 3. Added GitHub Workflow file `publish_release_candidate.yml`. This workflow is triggered when a release candidate git tag is pushed and creates a Prerelease GitHub Release. 4. Added logic to `dev/release/02-post-binary.sh` to create and push the release git tag (i.e. `apache-arrow-{MAJOR}.{MINOR}.{PATCH}`). 5. Added GitHub Workflow `publish_release.yml`. This workflow is triggered when the release tag is pushed and creates a GitHub Release for the approved release (i.e. the voted upon release). 6. Added `dev/release/post-16-delete-release-candidates.sh` to delete the release candidate git tags and their associated GitHub Releases. 7. Updated `docs/developers/release.rst` with the new steps. ### Are these changes tested? 1. We were not able to verify the changes made to the scripts in `dev/release`. Any suggestions on how we can verify these scripts would be much appreciated :) 2. We did test the new GitHub Workflows (`publish_release_candidate.yml` and `publish_release.yml`) work as intended by pushing git tags to [`mathworks/arrow`](https://github.com/mathworks/arrow). ### Are there any user-facing changes? No. ### Open Questions 1. We noticed that [apache/arrow-flight-sql-postgresql](https://github.com/apache/arrow-flight-sql-postgresql/releases) does **not** delete the release candidate Prereleases from their GitHub Releases area. Should we be doing the same? Or would it be preferable to just delete the the release candidates **without** deleting the release candidate tags. 2. We're not that familiar with ruby, so we're not sure if the changes we made to `dev/release/02-source-test.rb` make sense. ### Future Directions 1. Continue working on #40956 2. Add logic to auto-sign release artifacts in GitHub Actions Workflows. * GitHub Issue: #41102 Lead-authored-by: Sarah Gilmore <sgilmore@mathworks.com> Co-authored-by: Sarah Gilmore <74676073+sgilmore10@users.noreply.github.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Sarah Gilmore <sgilmore@mathworks.com>

sgilmore10 and others added 6 commits April 2, 2024 13:54

Add UPLOAD_MATLAB env variable to 05-binary-upload.sh

d5adac5

Add logic to upload MLTBX release candidates to apache/arrow's GitHub…

386c50b

… Releases Area

Fix indentation

378af3a

Split gh command into multiple lines

ee9c171

Add quotes around release notes argument.

6f6d67d

Add gpg signing and SHA512 checksum calculation for MLTBX file.

db4ed2a

github-actions bot added the awaiting review Awaiting review label Apr 2, 2024

1. Update Release Title

9dc7015

2. Update git tag 3. Fix target branch name

sgilmore10 marked this pull request as ready for review April 2, 2024 19:03

sgilmore10 requested review from assignUser, kou and raulcd as code owners April 2, 2024 19:03

kevingurney added Component: MATLAB Component: Packaging labels Apr 2, 2024

Add initial .github/workflows/package.yml workflow

f9d9f65

github-actions bot removed the Component: MATLAB label Apr 4, 2024

Add Cut Prerelease step to package.yml workflow

e108422

sgilmore10 marked this pull request as draft April 4, 2024 20:27

sgilmore10 added 2 commits April 4, 2024 16:37

Fix syntax error when appending env var to GITHUB_ENV

6c6bb92

1. Comment package.yml steps

2fb2d55

2. Fix syntax errors

raulcd reviewed Apr 5, 2024

View reviewed changes

Remove step extracting crossbow id from git tag message

d489d60

kevingurney added the Component: MATLAB label Apr 5, 2024

sgilmore10 added 9 commits April 5, 2024 16:12

Change job name from release to publish-release-candidate

8fd9b4c

add comment explaining when the workflow is triggered

37171a5

Add step to checkout crossbow

1623ede

Remove step updating git tags

c0a9d8e

Rename env var RELEASE_TAG_NAME to ARROW_RELEASE_TAG_NAME

2ea8b11

Rename Extract version strings step to Store Version and Release Cand…

bc0b411

…idate Number as Environment Variables

1. Rename Setup Archery step to Install Archery

61f62ec

2. Fix archery install path

1. Rename Download artifacts step to Download MLTBX Release Candidate

47e213a

2. ./dev/release/04-binary-download.sh -> ./arrow/release/04-binary-download.sh 3. Fix directory path provided to find

1. Rename Cut Pre-Release to Create GitHub Release for Release Candidate

e619b8b

2. Set environment variable GH_TOKEN in Create GitHub Release for Release Candidate step

github-actions bot removed the Component: MATLAB label Apr 5, 2024

sgilmore10 added 5 commits April 5, 2024 17:01

Fix indentation in package.yml

592e168

Fix call to 04-binary-download.sh

bdf2a77

Update old references to RELEASE_TAG_NAME to ARROW_RELEASE_TAG_NAME

153ec84

Update 05-binary-upload.sh to trigger the Package workflow

243ea58

Add stencil utils-sign-artifact.sh file

af1910a

kou reviewed Apr 5, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Apr 5, 2024

Remove utils-sign-artifact.sh

a6e6624

Update release documentaion

b1e8e15

github-actions bot added Component: Documentation awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Apr 16, 2024

assignUser closed this Apr 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-40924: [MATLAB][Packaging] Add script for uploading Release Candidate (RC) MLTBX packages for the MATLAB bindings to the Apache Arrow GitHub Releases area #40956

GH-40924: [MATLAB][Packaging] Add script for uploading Release Candidate (RC) MLTBX packages for the MATLAB bindings to the Apache Arrow GitHub Releases area #40956

sgilmore10 commented Apr 2, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Apr 2, 2024

kou commented Apr 3, 2024

assignUser commented Apr 3, 2024

kou commented Apr 4, 2024

sgilmore10 commented Apr 4, 2024

assignUser commented Apr 4, 2024

sgilmore10 commented Apr 4, 2024 •

edited

Loading

sgilmore10 commented Apr 4, 2024 •

edited

Loading

kou commented Apr 5, 2024

raulcd Apr 5, 2024

sgilmore10 Apr 5, 2024

sgilmore10 Apr 5, 2024 •

edited

Loading

kou Apr 5, 2024 •

edited

Loading

sgilmore10 Apr 8, 2024

kou Apr 8, 2024

sgilmore10 Apr 9, 2024

sgilmore10 Apr 9, 2024

kou Apr 9, 2024

assignUser commented Apr 25, 2024

GH-40924: [MATLAB][Packaging] Add script for uploading Release Candidate (RC) MLTBX packages for the MATLAB bindings to the Apache Arrow GitHub Releases area #40956

GH-40924: [MATLAB][Packaging] Add script for uploading Release Candidate (RC) MLTBX packages for the MATLAB bindings to the Apache Arrow GitHub Releases area #40956

Conversation

sgilmore10 commented Apr 2, 2024 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Future Directions

Open Questions

github-actions bot commented Apr 2, 2024

kou commented Apr 3, 2024

assignUser commented Apr 3, 2024

kou commented Apr 4, 2024

sgilmore10 commented Apr 4, 2024

assignUser commented Apr 4, 2024

sgilmore10 commented Apr 4, 2024 • edited Loading

sgilmore10 commented Apr 4, 2024 • edited Loading

kou commented Apr 5, 2024

raulcd Apr 5, 2024

Choose a reason for hiding this comment

sgilmore10 Apr 5, 2024

Choose a reason for hiding this comment

sgilmore10 Apr 5, 2024 • edited Loading

Choose a reason for hiding this comment

kou Apr 5, 2024 • edited Loading

Choose a reason for hiding this comment

sgilmore10 Apr 8, 2024

Choose a reason for hiding this comment

kou Apr 8, 2024

Choose a reason for hiding this comment

sgilmore10 Apr 9, 2024

Choose a reason for hiding this comment

sgilmore10 Apr 9, 2024

Choose a reason for hiding this comment

kou Apr 9, 2024

Choose a reason for hiding this comment

assignUser commented Apr 25, 2024

sgilmore10 commented Apr 2, 2024 •

edited by github-actions bot

Loading

sgilmore10 commented Apr 4, 2024 •

edited

Loading

sgilmore10 commented Apr 4, 2024 •

edited

Loading

sgilmore10 Apr 5, 2024 •

edited

Loading

kou Apr 5, 2024 •

edited

Loading