Skip to content

Commit

Permalink
Revert "Tutorial experiment update (#569)" (#570)
Browse files Browse the repository at this point in the history 
This reverts commit 87b9710.
  • Loading branch information
@ngrayluna
ngrayluna authored Aug 22, 2024
1 parent fbda426 commit fe6ab55
Showing 1 changed file with 69 additions and 75 deletions.
144 changes: 69 additions & 75 deletions colabs/intro/Intro_to_Weights_&_Biases.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,18 +20,44 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Use [W&B](https://wandb.ai/site?utm_source=intro_colab&utm_medium=code&utm_campaign=intro) for machine learning experiment tracking, model checkpointing, collaboration with your team and more. See the full W&B Documentation [here](https://docs.wandb.ai/).\n",
"# 🏃‍♀️ Quickstart\n",
"Use **[Weights & Biases](https://wandb.ai/site?utm_source=intro_colab&utm_medium=code&utm_campaign=intro)** for machine learning experiment tracking, model checkpointing, and collaboration with your team. See the full Weights & Biases Documentation **[here](https://docs.wandb.ai/quickstart)**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🤩 A shared dashboard for your experiments\n",
"\n",
"In this notebook, you will create and track a machine learning experiment using a simple PyTorch model. By the end of the notebook, you will have an interactive project dashboard that you can share and customize with other members of your team. [View an example dashboard here](https://wandb.ai/wandb/wandb_example)."
"With just a few lines of code,\n",
"you'll get rich, interactive, shareable dashboards [which you can see yourself here](https://wandb.ai/wandb/wandb_example).\n",
"![](https://i.imgur.com/Pell4Oo.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"Install the W&B Python SDK and log in:"
"## 🔒 Data & Privacy\n",
"\n",
"We take security very seriously, and our cloud-hosted dashboard uses industry standard best practices for encryption. If you're working with models and datasets that cannot leave your enterprise cluster, we have [on-prem](https://docs.wandb.com/self-hosted) installations available.\n",
"\n",
"It's also easy to download all your data and export it to other tools — like custom analysis in a Jupyter notebook. Here's [more on our API](https://docs.wandb.com/library/api).\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🪄 Install `wandb` library and login\n",
"\n",
"\n",
"Start by installing the library and logging in to your free account.\n",
"\n"
]
},
{
Expand All @@ -51,8 +77,6 @@
"source": [
"# Log in to your W&B account\n",
"import wandb\n",
"import random\n",
"import math\n",
"\n",
"# Use wandb-core, temporary for wandb's new backend \n",
"wandb.require(\"core\")"
Expand All @@ -71,12 +95,12 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Simulate and track a machine learning experiment with W&B\n",
"## 👟 Run an experiment\n",
"1️⃣. **Start a new run** and pass in hyperparameters to track\n",
"\n",
"Create, track, and visualize a machine learning experiment. To do this:\n",
"2️⃣. **Log metrics** from training or evaluation\n",
"\n",
"1. Initialize a [W&B run](https://docs.wandb.ai/guides/runs) and pass in the hyperparameters you want to track.\n",
"2. Within your training loop, log metrics such as the accuruacy and loss."
"3️⃣. **Visualize results** in the dashboard"
]
},
{
Expand All @@ -91,7 +115,7 @@
"# Launch 5 simulated experiments\n",
"total_runs = 5\n",
"for run in range(total_runs):\n",
" # 1️. Start a new run to track this script\n",
" # 🐝 1️⃣ Start a new run to track this script\n",
" wandb.init(\n",
" # Set the project where this run will be logged\n",
" project=\"basic-intro\",\n",
Expand All @@ -112,7 +136,7 @@
" acc = 1 - 2 ** -epoch - random.random() / epoch - offset\n",
" loss = 2 ** -epoch + random.random() / epoch + offset\n",
"\n",
" # 2️. Log metrics from your script to W&B\n",
" # 🐝 2️⃣ Log metrics from your script to W&B\n",
" wandb.log({\"acc\": acc, \"loss\": loss})\n",
"\n",
" # Mark the run as finished\n",
Expand All @@ -123,37 +147,33 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"View how your machine learning peformed in your W&B project. Copy and paste the URL link that is printed from the previous cell. The URL will redirect you to a W&B project that contains a dashboard showing graphs the show how \n",
"\n",
"The following image shows what a dashboard can look like:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](https://i.imgur.com/Pell4Oo.png)"
"3️⃣ You can find your interactive dashboard by clicking any of the 👆 wandb links above."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now that we know how to integrate W&B into a psuedo machine learning training loop, let's track a machine learning experiment using a basic PyTorch neural network. The following code will also upload model checkpoints to W&B that you can then share with other teams in in your organization.\n",
"# 🔥 Simple Pytorch Neural Network\n",
"\n",
"## Track a machine learning experiment using Pytorch\n",
"💪 Run this model to train a simple MNIST classifier, and click on the project page link to see your results stream in live to a W&B project.\n",
"\n",
"The following code cell defines and trains a simple MNIST classifier. During training, you will see W&B prints out URLs. Click on the project page link to see your results stream in live to a W&B project.\n",
"\n",
"W&B runs automatically log [metrics](https://docs.wandb.ai/ref/app/pages/run-page#charts-tab),\n",
"Any run in `wandb` automatically logs [metrics](https://docs.wandb.ai/ref/app/pages/run-page#charts-tab),\n",
"[system information](https://docs.wandb.ai/ref/app/pages/run-page#system-tab),\n",
"[hyperparameters](https://docs.wandb.ai/ref/app/pages/run-page#overview-tab),\n",
"[terminal output](https://docs.wandb.ai/ref/app/pages/run-page#logs-tab) and\n",
"you'll see an [interactive table](https://docs.wandb.ai/guides/data-vis)\n",
"with model inputs and outputs. \n",
"\n",
"### Set up PyTorch Dataloader\n",
"The following cell defines some useful functions that we will need to train our machine learning model. The functions themselves are not unique to W&B so we'll not cover them in detail here. See the PyTorch documentation for more information on how to define [forward and backward training loop](https://pytorch.org/tutorials/beginner/nn_tutorial.html), how to use [PyTorch DataLoaders](https://pytorch.org/tutorials/beginner/basics/data_tutorial.html) to load data in for training, and how define PyTorch models using the [`torch.nn.Sequential` Class](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html). "
"This code will also save and upload Model checkpoints to W&B where you can then use them in the [W&B Registry](https://docs.wandb.ai/guides/registry)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set up Dataloader"
]
},
{
Expand Down Expand Up @@ -212,31 +232,11 @@
" # Log one batch of images to the dashboard, always same batch_idx.\n",
" if i==batch_idx and log_images:\n",
" log_image_table(images, predicted, labels, outputs.softmax(dim=1))\n",
" return val_loss / len(valid_dl.dataset), correct / len(valid_dl.dataset)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create a teble to compare the predicted values versus the true value\n",
"\n",
"The following cell is unique to W&B, so let's go over it.\n",
" return val_loss / len(valid_dl.dataset), correct / len(valid_dl.dataset)\n",
"\n",
"In the cell we define a function called `log_image_table`. Though technically, optional, this function creates a W&B Table object. We will use the table object to create a table that shows what the model predicted for each image. \n",
"\n",
"More specifically, each row will conists of the image fed to the model, along with predicted value and the actual value (label). "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def log_image_table(images, predicted, labels, probs):\n",
" \"Log a wandb.Table with (img, pred, target, scores)\"\n",
" # Create a wandb Table to log images, labels and predictions to\n",
" # 🐝 Create a wandb Table to log images, labels and predictions to\n",
" table = wandb.Table(columns=[\"image\", \"pred\", \"target\"]+[f\"score_{i}\" for i in range(10)])\n",
" for img, pred, targ, prob in zip(images.to(\"cpu\"), predicted.to(\"cpu\"), labels.to(\"cpu\"), probs.to(\"cpu\")):\n",
" table.add_data(wandb.Image(img[0].numpy()*255), pred, targ, *prob.numpy())\n",
Expand All @@ -247,11 +247,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Train your model and upload checkpoints\n",
"\n",
"The following code trains and saves model checkpoints to your project. Use model checkpoints like you normally would to assess how the model performed during training. \n",
"\n",
"W&B also makes it easy to share your saved models and model checkpoints with other members of your team or organization. To learn how to share your model and model checkpoints with members outside of your team, see [W&B Registry](https://docs.wandb.ai/guides/registry)."
"## Train Your Model & Upload Checkpoints"
]
},
{
Expand All @@ -262,7 +258,7 @@
"source": [
"# Launch 3 experiments, trying different dropout rates\n",
"for _ in range(3):\n",
" # initialise a wandb run\n",
" # 🐝 initialise a wandb run\n",
" wandb.init(\n",
" project=\"pytorch-intro\",\n",
" config={\n",
Expand Down Expand Up @@ -307,50 +303,49 @@
" \"train/example_ct\": example_ct}\n",
"\n",
" if step + 1 < n_steps_per_epoch:\n",
" # Log train metrics to wandb\n",
" # 🐝 Log train metrics to wandb\n",
" wandb.log(metrics)\n",
"\n",
" step_ct += 1\n",
"\n",
" val_loss, accuracy = validate_model(model, valid_dl, loss_func, log_images=(epoch==(config.epochs-1)))\n",
"\n",
" # Log train and validation metrics to wandb\n",
" # 🐝 Log train and validation metrics to wandb\n",
" val_metrics = {\"val/val_loss\": val_loss,\n",
" \"val/val_accuracy\": accuracy}\n",
" wandb.log({**metrics, **val_metrics})\n",
"\n",
" # Save the model checkpoint to wandb\n",
" # 🐝 Save the model checkpoint to wandb\n",
" torch.save(model, \"my_model.pt\")\n",
" wandb.log_model(\"./my_model.pt\", \"my_mnist_model\", aliases=[f\"epoch-{epoch+1}_dropout-{round(wandb.config.dropout, 4)}\"])\n",
"\n",
" print(f\"Epoch: {epoch+1}, Train Loss: {train_loss:.3f}, Valid Loss: {val_loss:3f}, Accuracy: {accuracy:.2f}\")\n",
"\n",
" # If you had a test set, this is how you could log it as a Summary metric\n",
" # 🐝 If you had a test set, this is how you could log it as a Summary metric\n",
" wandb.summary['test_accuracy'] = 0.8\n",
"\n",
" # Close your wandb run\n",
" # 🐝 Close your wandb run\n",
" wandb.finish()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You have now trained your first model using W&B. Click on one of the links above to see your metrics and see your saved model checkpoints in the Artifacts tab in the W&B App UI"
"You have now trained your first model using wandb! 👆 Click on the wandb link above to see your metrics and see your saved Model checkpoints in the Artifacts tab in the UI"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## (Optional) Set up a W&B Alert\n",
"\n",
"Create a [W&B Alerts](https://docs.wandb.ai/guides/track/alert) to send alerts to your Slack or email from your Python code. \n",
"# 🔔 Try W&B Alerts\n",
"\n",
"There are 2 steps to follow the first time you'd like to send a Slack or email alert, triggered from your code:\n",
"**[W&B Alerts](https://docs.wandb.ai/guides/track/alert)** allows you to send alerts, triggered from your Python code, to your Slack or email. There are 2 steps to follow the first time you'd like to send a Slack or email alert, triggered from your code:\n",
"\n",
"1) Turn on Alerts in your W&B [User Settings](https://wandb.ai/settings)\n",
"2) Add `wandb.alert()` to your code. For example:\n",
"\n",
"2) Add `wandb.alert()` to your code:\n",
"\n",
"```python\n",
"wandb.alert(\n",
Expand All @@ -359,7 +354,7 @@
")\n",
"```\n",
"\n",
"The following cell shows a minimal example below to see how to use `wandb.alert`"
"See the minimal example below to see how to use `wandb.alert`. You can find the full docs for **[W&B Alerts here](https://docs.wandb.ai/guides/track/alert)**"
]
},
{
Expand All @@ -379,12 +374,12 @@
" accuracy = round(random.random() + random.random(), 3)\n",
" print(f'Accuracy is: {accuracy}, {acc_threshold}')\n",
"\n",
" # Log accuracy to wandb\n",
" # 🐝 Log accuracy to wandb\n",
" wandb.log({\"Accuracy\": accuracy})\n",
"\n",
" # If the accuracy is below the threshold, fire a W&B Alert and stop the run\n",
" # 🔔 If the accuracy is below the threshold, fire a W&B Alert and stop the run\n",
" if accuracy <= acc_threshold:\n",
" # Send the wandb Alert\n",
" # 🐝 Send the wandb Alert\n",
" wandb.alert(\n",
" title='Low Accuracy',\n",
" text=f'Accuracy {accuracy} at step {training_step} is below the acceptable theshold, {acc_threshold}',\n",
Expand All @@ -400,11 +395,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"You can find the full docs for [W&B Alerts here](https://docs.wandb.ai/guides/track/alert).\n",
"\n",
"## Next steps\n",
"# What's next 🚀 ?\n",
"The next tutorial you will learn how to do hyperparameter optimization using W&B Sweeps:\n",
"[Hyperparameters sweeps using PyTorch](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/pytorch/Organizing_Hyperparameter_Sweeps_in_PyTorch_with_W%26B.ipynb)"
"## 👉 [Hyperparameters sweeps using PyTorch](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/pytorch/Organizing_Hyperparameter_Sweeps_in_PyTorch_with_W%26B.ipynb)"
]
}
],
Expand All @@ -421,5 +415,5 @@
}
},
"nbformat": 4,
"nbformat_minor": 4
"nbformat_minor": 0
}

0 comments on commit fe6ab55

Please sign in to comment.