This repo contains notes and guidance on setting up Azure for use in a big data class. These notes are primarily to help me remember the process and are not necessarily correct. I was going through the process as a university entity, which does make some elements different.
You can choose to buy Azure time directly or go through a reseller. Our university has a relationship with SHI, and I elected to use SHI to purchase time on Azure. When you go through SHI, it looks like you have to figure out some settings on enterprise agreement portal before your portal.azure.com will be connected. After 15 minutes with two Microsoft employees, we got that working. I wish I remembered the details.
There were options to pre-purchase time with a one-year or three-year expiration. The time chunks were sold in ~3k units.
Once in portal.azure.com, you will need to complete the following steps.
- Connect your subscription
- Create a resource group using that subscription (What is a resource group?)
- Within that resource group, add a resource (e.g., Databricks, PostgreSQL, storage) or functionality from the Azure Marketplace by clicking the
+ add
link.
- For our big data class, we use
Azure Databricks
,Azure Database for PostgreSQL
, and the MicrosoftStorage account
resources. You will need to go to each and thenCreate
the resource. When you create each resource, you will need to pick a region. The region names change a little at almost every drop-down, so be careful.Azure Databricks
: I accepted the defaults. You will build your spark clusters in Databricks.Azure Database for PostgreSQL
: This element has a revolving cost that can be relatively high. The minimum price for one CPU with 100 GB of storage is ~$40 a month (see steps below for this resource).Storage account
:
- Click the
Create
button.
- You will then want to select
single server
to get the lowest monthly cost.
- Now, you will be able to set up the server. Make sure you pick the same Location as your other resources. You will want to click on
configure server
- You will then see the
General Purpose
tab. Click on theBasic
tab.
- On the
Basic
tab, you can adjust to one core and reduce the store to the size you require. My database is about 30 gigs. Then clickok
.
- Now, you can go through the additional steps and then click
Create
at the bottom.