[Upgrades] Document learnings from the last upgrade

[okdas]

Summary

Performed the first upgrade on the Alpha TestNet. Add some documentation changes to prevent some issues in the future.

Issue

N/A

Type of change

Select one or more from the following:

• New feature, functionality or library
• Consensus breaking; add the consensus-breaking label if so. See [Infra] Automatically add the consensus-breaking label #791 for details
• Bug fix
• Code health or cleanup
• Documentation
• Other (specify)

Testing

• Documentation: make docusaurus_start; only needed if you make doc changes
• Unit Tests: make go_develop_and_test
• LocalNet E2E Tests: make test_e2e
• DevNet E2E Tests: Add the devnet-test-e2e label to the PR.

Sanity Checklist

• I have tested my changes using the available tooling
• I have commented my code
• I have performed a self-review of my own code; both comments & source code
• I create and reference any new tickets, if applicable
• I have left TODOs throughout the codebase, if applicable

[okdas]

Items to add:

• Configure cosmovisor to create backups before upgrade
• Have more bump validators to 5.
• Instructions on how to test the upgrade locally. Thinking a script w/o LocalNet.

[Olshansk]

@okdas Let's add the details here as well: https://x.com/olshansky/status/1846211059989778741