Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-40924: [MATLAB][Packaging] Add script for uploading Release Candidate (RC) MLTBX packages for the MATLAB bindings to the Apache Arrow GitHub Releases area #40956

Closed
wants to merge 29 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
d5adac5
Add UPLOAD_MATLAB env variable to 05-binary-upload.sh
sgilmore10 Apr 1, 2024
386c50b
Add logic to upload MLTBX release candidates to apache/arrow's GitHub…
sgilmore10 Apr 1, 2024
378af3a
Fix indentation
sgilmore10 Apr 1, 2024
ee9c171
Split gh command into multiple lines
sgilmore10 Apr 1, 2024
6f6d67d
Add quotes around release notes argument.
kevingurney Apr 1, 2024
db4ed2a
Add gpg signing and SHA512 checksum calculation for MLTBX file.
kevingurney Apr 1, 2024
9dc7015
1. Update Release Title
sgilmore10 Apr 2, 2024
f9d9f65
Add initial .github/workflows/package.yml workflow
sgilmore10 Apr 4, 2024
e108422
Add Cut Prerelease step to package.yml workflow
sgilmore10 Apr 4, 2024
6c6bb92
Fix syntax error when appending env var to GITHUB_ENV
sgilmore10 Apr 4, 2024
2fb2d55
1. Comment package.yml steps
sgilmore10 Apr 4, 2024
ac77f5b
Use dev/release/04-binary-download.sh to download the release candida…
sgilmore10 Apr 5, 2024
d489d60
Remove step extracting crossbow id from git tag message
sgilmore10 Apr 5, 2024
8fd9b4c
Change job name from release to publish-release-candidate
sgilmore10 Apr 5, 2024
37171a5
add comment explaining when the workflow is triggered
sgilmore10 Apr 5, 2024
1623ede
Add step to checkout crossbow
sgilmore10 Apr 5, 2024
c0a9d8e
Remove step updating git tags
sgilmore10 Apr 5, 2024
2ea8b11
Rename env var RELEASE_TAG_NAME to ARROW_RELEASE_TAG_NAME
sgilmore10 Apr 5, 2024
bc0b411
Rename Extract version strings step to Store Version and Release Cand…
sgilmore10 Apr 5, 2024
61f62ec
1. Rename Setup Archery step to Install Archery
sgilmore10 Apr 5, 2024
47e213a
1. Rename Download artifacts step to Download MLTBX Release Candidate
sgilmore10 Apr 5, 2024
e619b8b
1. Rename Cut Pre-Release to Create GitHub Release for Release Candidate
sgilmore10 Apr 5, 2024
592e168
Fix indentation in package.yml
sgilmore10 Apr 5, 2024
bdf2a77
Fix call to 04-binary-download.sh
sgilmore10 Apr 5, 2024
153ec84
Update old references to RELEASE_TAG_NAME to ARROW_RELEASE_TAG_NAME
sgilmore10 Apr 5, 2024
243ea58
Update 05-binary-upload.sh to trigger the Package workflow
sgilmore10 Apr 5, 2024
af1910a
Add stencil utils-sign-artifact.sh file
sgilmore10 Apr 5, 2024
a6e6624
Remove utils-sign-artifact.sh
sgilmore10 Apr 6, 2024
b1e8e15
Update release documentaion
sgilmore10 Apr 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions .github/workflows/package.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

name: Package

on:
push:
tags:
# A release candidate should be published whenever a tag with
# the name below is pushed to the apache/arrow repository.
- "apache-arrow-*-rc*"

jobs:
publish-release-candidate:
name: Publish Release Candidate to GitHub Releases
runs-on: ubuntu-latest
if: github.ref_type == 'tag'
steps:
- name: Checkout apache/arrow
uses: actions/checkout@v4
with:
path: arrow
fetch-depth: 0
- name: Checkout ursacomputing/crossbow
uses: actions/checkout@v4
with:
repository: ursacomputing/crossbow
path: crossbow
fetch-depth: 0
- name: Store Release Tag Name as Environment Variable
# For convenience, store the release tag name as an environment
# variable for use in subsequent steps.
run: |
echo "ARROW_RELEASE_TAG_NAME=${{github.ref_name}}" >> "$GITHUB_ENV"
- name: Store Version and Release Candidate Number as Environment Variables
# From environment variable ARROW_RELEASE_TAG_NAME, extract the version and
# release candidate number. Store these values as environment variables.
run: |
version_with_rc=${ARROW_RELEASE_TAG_NAME#apache-arrow-}
version=${version_with_rc%-rc*}
rc_num=${version_with_rc#${version}-rc}
echo "ARROW_VERSION_WITH_RC=${version_with_rc}" >> "${GITHUB_ENV}"
echo "ARROW_VERSION=${version}" >> ${GITHUB_ENV}
echo "ARROW_RC_NUM=${rc_num}" >> ${GITHUB_ENV}
- name: Install Archery
run: |
python3 -m pip install -e arrow/dev/archery[crossbow]
- name: Download Release Candidate MLTBX
# Download the MLTBX file from the crossbow GitHub Release's area.
run: |
./arrow/dev/release/04-binary-download.sh ${ARROW_VERSION} ${ARROW_RC_NUM} --task-filter matlab
mltbx_file=$(find arrow/packages -name '*.mltbx' -type f)
echo "ARROW_MLTBX_FILE=${mltbx_file}" >> $GITHUB_ENV
- name: Create GitHub Release for Release Candidate
# Create a pre-release in apache/arrow's GitHub Releases area.
# Attach the release candidate MLTBX file as an asset.
run: |
target_branch=release-${ARROW_VERSION_WITH_RC}
title="Apache Arrow ${ARROW_VERSION} RC${ARROW_RC_NUM}"
repository=${{github.repository}}
release_notes="Release Candidate: ${ARROW_VERSION} RC${ARROW_RC_NUM}"
gh release create \
${ARROW_RELEASE_TAG_NAME} \
${ARROW_MLTBX_FILE} \
--prerelease \
--target ${target_branch} \
--notes "${release_notes}" \
--title "${title}" \
--repo ${repository}
env:
GH_TOKEN: ${{ github.token }}
16 changes: 16 additions & 0 deletions dev/release/05-binary-upload.sh
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ fi
: ${UPLOAD_CENTOS:=${UPLOAD_DEFAULT}}
: ${UPLOAD_DEBIAN:=${UPLOAD_DEFAULT}}
: ${UPLOAD_DOCS:=${UPLOAD_DEFAULT}}
: ${UPLOAD_MATLAB:=${UPLOAD_DEFAULT}}
: ${UPLOAD_NUGET:=${UPLOAD_DEFAULT}}
: ${UPLOAD_PYTHON:=${UPLOAD_DEFAULT}}
: ${UPLOAD_R:=${UPLOAD_DEFAULT}}
Expand Down Expand Up @@ -135,3 +136,18 @@ docker_run \
VERBOSE=${VERBOSE:-no} \
VERSION=${version} \
YUM_TARGETS=$(IFS=,; echo "${yum_targets[*]}")

# Create a git tag to associate with a GitHub Release that will
# have the release candidate MLTBX file as an asset.
#
# Push this tag (e.g. apache-arrow-15.0.2-rc0) to the remote
# apache/arrow repository to trigger the GitHub Actions Workflow
# that creates the GitHub Release and uploads the MLTBX file.
#
# See .github/workflows/package.yml for details.
if [ ${UPLOAD_MATLAB} -gt 0 ]; then
release_tag="apache-arrow-${version_with_rc}"
tag_message="Release candidate: ${version_with_rc}"
git tag -a ${release_tag} -m ${tag_message}
git push apache ${release_tag}
Copy link
Member

@kou kou Apr 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you do this in dev/release/01-prepare.sh something like the following?

diff --git a/dev/release/01-prepare.sh b/dev/release/01-prepare.sh
index 01fa2f3d80..f5ec0a8e26 100755
--- a/dev/release/01-prepare.sh
+++ b/dev/release/01-prepare.sh
@@ -33,7 +33,7 @@ next_version=$2
 next_version_snapshot="${next_version}-SNAPSHOT"
 rc_number=$3
 
-release_tag="apache-arrow-${version}"
+rc_tag="apache-arrow-${version}-rc${rc_number}"
 release_branch="release-${version}"
 release_candidate_branch="release-${version}-rc${rc_number}"
 
@@ -45,9 +45,9 @@ release_candidate_branch="release-${version}-rc${rc_number}"
 : ${PREPARE_TAG:=${PREPARE_DEFAULT}}
 
 if [ ${PREPARE_TAG} -gt 0 ]; then
-  if [ $(git tag -l "${release_tag}") ]; then
-    echo "Delete existing git tag $release_tag"
-    git tag -d "${release_tag}"
+  if [ $(git tag -l "${rc_tag}") ]; then
+    echo "Delete existing git tag $rc_tag"
+    git tag -d "${rc_tag}"
   fi
 fi
 
@@ -91,7 +91,7 @@ if [ ${PREPARE_LINUX_PACKAGES} -gt 0 ]; then
 fi
 
 if [ ${PREPARE_VERSION_PRE_TAG} -gt 0 ]; then
-  echo "Prepare release ${version} on tag ${release_tag} then reset to version ${next_version_snapshot}"
+  echo "Prepare release ${version} on tag ${rc_tag} then reset to version ${next_version_snapshot}"
 
   update_versions "${version}" "${next_version}" "release"
   git commit -m "MINOR: [Release] Update versions for ${version}"
@@ -100,5 +100,5 @@ fi
 ############################## Tag the Release ##############################
 
 if [ ${PREPARE_TAG} -gt 0 ]; then
-  git tag -a "${release_tag}" -m "[Release] Apache Arrow Release ${version}"
+  git tag -a "${rc_tag}" -m "[Release] Apache Arrow Release ${version}"
 fi

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @kou,

Thanks for sharing this diff! We just have one clarification question:

Based on the diff you shared, it looks like you want us to replace the release_tag with the rc_tag. Did you mean for us to actually add the rc_tag in addition to the release_tag? We asking because we assume we still want to create a release_tag.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you mean for us to actually add the rc_tag in addition to the release_tag?

No. I mean for us to replace the release_tag with rc_tag.

We asking because we assume we still want to create a release_tag.

I want to change the current workflow:

  1. Create a apache-arrow-X.Y.Z tag for X.Y.Z RC0
  2. (Found a problem for X.Y.Z RC0)
  3. Remove the apache-arrow-X.Y.Z tag for X.Y.Z RC0
  4. Create a apache-arrow-X.Y.Z tag for X.Y.Z RC1
  5. ...

to the following:

  1. Create a apache-arrow-X.Y.Z-rc0 tag for X.Y.Z RC0
  2. (Found a problem for X.Y.Z RC0)
  3. Create a apache-arrow-X.Y.Z-rc1 tag for X.Y.Z RC1
  4. Vote
  5. Passed
  6. Create a apache-arrow-X.Y.Z tag from apache-arrow-X.Y.Z-rc1 like apache/arrow-adbc and apache/arrow-flight-sql-postgresql do

Hmm. It's better that we handle the workflow change related task in a separated issue...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification! I've created a issue #41102 to track this work. Once this PR is merged, I'll take ownership of that issue and make the required changes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @kou,

On second thought, we think it actually makes more sense to tackle #41102 first since the new GitHub Actions Workflow (package.yml) requires the release candidate tag (apache-arrow.X.Y.Z-rcN) to be pushed. We'll put this PR on pause for a bit until that #41102 is closed.

Best,
Sarah

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes sense!

fi
5 changes: 5 additions & 0 deletions docs/source/developers/release.rst
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,11 @@ Build source and binaries and submit them
# Start verifications for binaries and wheels
dev/release/07-binary-verify.sh <version> <rc-number>

# Sign and upload MATLAB artifacts to the GitHub Releases area.
#
# NOTE: You must have GitHub CLI installed to run this script.
dev/release/08-matlab-upload.sh <version> <rc-number>

Verify the Release
------------------

Expand Down
Loading