unified release process

[pngwn]

closes #4851.

This PR overhauls our versioning and release process, and unifies it across the JS and Python files/libraries.

---

Before I explain this PR I just want to point out that change will increase the frequency with which you have to pull changes from a PR branch because the bot will be pushing to the PR. There should never be conflicts because it will only create changeset files (so .changeset/something.md) but you may get errors if you try to push and you have pulled when the bot has added a commit.

In order to work around this, I recommend setting up some global git config to make pulling easier:

git config --global push.autosetupremote true # not required but is nice
git config --global push.default upstream
git config --global branch.autosetupmerge simple # i think this is the default
git config --global pull.rebase true # rebase pull is cleaner than merge

These settings will automatically setup up the tracking branch for you (as long as you are working off this repo). If you need to pull you'll just be able to git pull without tracking remote explicitly and you'll also be able to just git push instead of git push origin whatever. You don't have to do this but it'll make life way easier. Being able to just push / pull is super nice.

---

There are quite a few moving parts in this PR (honestly it was a much more enormous amount of work than i expected), so I'll will explain what happens and then i will explain how.

What does it do

This PR introduces two distinct things: automatic versioning/ changelog generation/ release and automatic changeset generation. This will make sense in a sec.

Whenever we make a PR that requires a release (so any code change) instead of updating the changelog manually, we just add a changeset file that describe the change. The only information needed is:

• the package(s) that have changed
• the type of change (patch, minor or major)
• a description of the change (for the changelog, so friendly descriptions)
• the 'type' of change (fix, feat, highlight). This is to allow us to format our changelogs nicely for each releases. We can add more types if we thing they are necessary.

A changeset file looks like this:

---
<package-name>: <patch|minor|major>
<package-name>: <patch|minor|major>
...
---
<fix|feat>: description

A changeset with a highlight type ha slightly different requirements:

---
<package-name>: <patch|minor|major>
<package-name>: <patch|minor|major>
...
---
highlight: 

#### a title

some content

highlight types must have a level 4 markdown heading on a separate line from the highlight: text. This looks nicer and is validated in CI.

When this file is added to a PR and that PR is merged, a robot will create a new 'release' PR this PR takes care of figuring our the correct version bump for each package based on the combined changesets file, it also generates a changelog from those changesets. It link the PR, the merge commit and thanks the user (with a link) in the change logs. Normal entries look like this:

- [#1234](https//link.to.pr) [`abcd123`](https://link.to.ref) - The description of the change. Thanks [@user](https://link.to.user)!

Rendered:

• #1234 abcd123 - The description of the change. Thanks @user!

highlight entries look like this:

#### The description of the change / title ([#1234](https//link.to.pr) [`abcd123`](https://link.to.ref))

any content

Thanks [@user](https://link.to.user)!

Rendered:

The description of the change / title (#1234 abcd123)

any content

Thanks @user!

This generated PR also bumps all of the version files (package.json + version.txt) and bumps the version of any local dependencies (so when gradio_client gets bumped, the requirements.txt of gradio gets bumped to match this version).

When we merged this PR, the second part of teh same CI workflow will see that the changesets have been removed + the version updated. It will then check that version does not exist, run any build scripts and release the package. Python packages are expected to have a build_pypi.sh script in their directory (we can move this to scripts/ if needed).

changesets will also generate tags + github releases but I need to double check that is enabled.

The second part os the automatic changeset generation. Generating the changesets can be a bit tedious so we have a bot that does it for you. The bot does this:

• By default, it will check the changed packages, check any linked issues (closes The ability to not have all those extra image editing tools, only cropping #123) and generate the correct changeset based on that. If there are no release it will set the bump as minor and the type as feat.
• We can change the version and type with labels. v: (patch|major|minor) for the bump type and t: (fix|feat|highlight) for the 'type'.
• We can select the packages that should be affected if the bot is wrong. There is a PR comment that we can interact with to enter "manual package selection mode". Then we will get a checklist of all packages that we can choose from
• We can enter full manual mode byt editing the generated changeset file. The bot will stop doing stuff (other than updating the comment) if the changeset file has been edited manually.
• We can delete the changeset file and remove all labels to start again.

How does it work

Changeset generation

• When a PR is opened, reopened, synchronised, edited or labelled, or the issue generated by the job is edited a CI job runs. This job does nothing and only exists to trigger the action that does the work (pull_request event or issues_edited is the event used here).
• Another action (with the actual logic in it) will run when the above action has finished (workflow_run event). This is for security reasons. We need access to secrets to do anything but we don't want PRs from forks to have access to those secrets. pull_request evens do not expose secrets. This 'other' action runs in the context of main, so a PR would need to be merged any code within it could have any impact.
• This action figures out the PR via a reusbale action which can be found here. This action uses the source repo + branch name and cross-references that with any open PRs to find the PR number.
• When it has the PR number, this is passed into the changeset generation / update action which is also reusable and can be found here. I can go through how this works if needed but there is a bit of complexity there.
• One difference between vanilla changesets and our version is how we handle dependencies. Pretty much any change we make should end up in the gradio changelog in order to be useful to users. This isn't typically how it works in JS. The default behaviour is for individual packages to contain their changelog (which is good) but for packages that depend on those packages to simple contain an 'dependencies updated' changelog entry (which is not very useful). So I have introduced a main_changeset field to the package.json of any package whose changelog should be added to the 'main' package, in this case gradio. What this mean is that if you change the Plot component's JS/Svelte code with a title of "ensure the plot uses the correct styling in dark mode" that description will be added to the @gradio/plot package (which is nice for us to keep track) but it will also be added to the gradio package (which is nice for our users).

Versioning

• We build on top of changesets to do this, this is why i have added package.json files to the gradio libs, to kind of treat them as 'packages' (changesets only really works with npm packages).
• We follow the normal changesets flow with a few customisation. The changeset files we generate match what changesets expects, so we don't need to do anything clever there. All we really need to do is add a little metadata to introduce the concept of type (fix, feat, highlight). So when making PRs and merging them into main the flow is pretty consistent with the changesets documentation.
• After we have merged into main we do a bunch of more custom stuff. There are two scripts that handle this. ./changeset/changeset.cjs and ./changeset/fix_changelogs.cjs.
• changesets doesn't have an official API to create a custom changelog format, so we need to hack around it a bit.
• ./changeset/changeset.cjs has a function or two that run for every 'release line', i.e. every changelog entry for each package. We do some github api lookups to get some details about a PR, parse the changeset file to see if it is a fix, feat, or highlight and then write those details to a JSON file. These function also return a string that becomes the new changelog for each package but we don't care about that because we overwrite them in the next step.
• The above command runs as part of the changeset version command, which also bumps the JS packages. As we are storing the version for python packages in package.json files they get bumped correctly, however they are not in the correct place (version.txt), so we need to copy those new version into a few places to play nice with the python conventions. This is also when see if any python packages depend on another python package locally. In this case gradio depends on gradio_client. If gradio_client has been bumped, then we will update the requirements.txt of gradio to match this new version. This is because that is how we tests in CI, latest with latest, so that is what we pin to.
• This is also where we reformat the changelogs to respect our custom format (which is with Highlights, "Features", and "Fixes" sections). We use the JSON file we saved earlier to know what has changed for each package.
• The changesets action creates a descriptive PR message from the changelogs themselves, so this pretty much works without any intervention.

Publishing

This is also part of the 'changesets' CI job.

• The JS stuff 'just works' by default because that is what changesets does.
• The python stuff does not. We have some custom stuff to make that go.
• The logic lives in this reusable action. This is basically the same as the pypi action we used to use except faster (and not quite as feature rich).
• It checks that the version have not already been published. If they have it bails.
• It runs the build_pypi.sh script that ever package is expected to have
• It uploads the new version with twine
• (I can go through this action in more detail if needed as well)

---

checklist for me as i go through.

• change pngwn/gradio reference to ``gradio-app/gradio`
• check all python modules have a build script in the correct place
• remove the current changesets ci thing
• reformat existing changelogs (i'll explain this)
• check all secrets (i think i named them the same but need to double check)
• check all package names + package.json are correct
• explain wtf this does and where everything is in this PR body

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
gradio	✅ Ready (Inspect)	Visit Preview	Jul 21, 2023 6:08pm

[gradio-pr-bot]

All the demos for this PR have been deployed at https://huggingface.co/spaces/gradio-pr-deploys/pr-4986-all-demos

---

You can install the changes in this PR by running:

pip install https://gradio-builds.s3.amazonaws.com/d51f61692b0bdb001bf902bbc02a24ed6fc1109d/gradio-3.38.0-py3-none-any.whl

[abidlabs]

Very cool @pngwn! Thanks for putting this together. I looked through the changes, I couldn't follow everything, but changes look good at a high level. Down to test it out with the next release

[freddyaboulton]

I just checked and @gradio-pr-bot is in the org so I think we can use secrets.COMMENT_TOKEN here.

[freddyaboulton]

These tokens look good to me.

[freddyaboulton]

@pngwn the CI master !!! Excited to test this out!!

[hannahblair]

Thanks so much for taking the time to explain the changes here so elaborately! It'd be great to keep this documentation somewhere as an easily accessible reference for contributors instead of digging up this PR.

[hannahblair]

i think we need to remove my boop here 😁

[pngwn]

ah, yes.

[hannahblair]

is this doing the same as the 8BitJonny/gh-get-current-pr@2.2.0 action in deploy-chromatic.yml?

[pngwn]

It's similar but not quite the same. I'm not 100% certain if that action would work for us as ours is tailored for workflow runs (and doesn't require any input), we use the branch name + repo owner to check the PR rather than a commit sha which is what i think that action is doing. It could work tho, not certain.

[gradio-pr-bot]

🎉 Chromatic build completed!

There are 0 visual changes to review.
There are 0 failed tests to fix.

• Storybook Preview
• Build Review