Merge branch 'main' into article-conda-project-space

uabrc · Oct 9, 2024 · 50e5535 · 50e5535
2 parents f13430a + b40d538
commit 50e5535
Show file tree

Hide file tree

Showing 11 changed files with 137 additions and 74 deletions.
diff --git a/docs/account_management/cheaha_account.md b/docs/account_management/cheaha_account.md
@@ -1,3 +1,7 @@
+---
+toc_depth: 3
+---
+
 # Cheaha Account Management
 
 These instructions are intended to guide researchers on creating new accounts and managing existing accounts.
@@ -78,11 +82,38 @@ Periodically, we review all researcher accounts to ensure they are authorized to
 
 If you believe this to be in error, please [Contact Support](../help/support.md).
 
+## What can I do with my account?
+
+Research Computing offers services addressing a wide range of needs for researchers at UAB, including students, staff, and faculty, as well as for both Labs and research cores.
+
+We're always happy to provide support for your Research Computing needs, you need only [Contact Support](../index.md#how-to-contact-us).
+
+### For Students, Staff, and Faculty
+
+- [Get Started with Open OnDemand](../cheaha/open_ondemand/index.md)
+- [Additional Learning Resources](../education/training_resources.md)
+- [Data Science Journal Club Course](../education/courses.md#data-science-journal-club-course)
+
+### For Lab PIs and Core Directors
+
+- [No-cost storage offerings](../data_management/storage.md#what-type-of-storage-do-i-need)
+    - [GPFS](../data_management/storage.md#what-shared-storage-solutions-are-available): Hot storage, compute adjacent, directly accessible from Cheaha
+    - [LTS](../data_management/lts/index.md): Cool storage, large capacity
+    - [Transfer data with Globus](../data_management/transfer/globus.md)
+- [Batch computing](../cheaha/slurm/introduction.md)
+    - [Desktop](../cheaha/open_ondemand/hpc_desktop.md), [Jupyter](../cheaha/open_ondemand/ood_jupyter.md), [RStudio](../cheaha/open_ondemand/ood_rstudio.md), [Matlab](../cheaha/open_ondemand/ood_matlab.md), and more
+    - [GPUs](../cheaha/slurm/gpu.md)
+- [On-prem cloud computing](../uab_cloud/index.md)
+    - [Tutorial](../uab_cloud/tutorial/index.md)
+    - [Web servers](../uab_cloud/remote_access.md#make-instances-publically-accessible-from-the-internet)
+
+  If you are unable to find what you need, please feel free to [Contact Support](../index.md#how-to-contact-us).
+
 ## Cheaha Account and Group Membership FAQ
 
 Our Cheaha system is robust, but errors may occur due to general platform connectivity issues or missing components. Below are FAQs for self-service Cheaha account creation and a troubleshooting guide for common issues:
 
-- **Which credentials should I use?** Please visit [how do I login to research computing services](../account_management/index.md#how-do-i-login-to-research-computing-services).
+- **Which credentials should I use?** Please visit [How Do I Login to Research Computing Services](../account_management/index.md#how-do-i-login-to-research-computing-services).
 - **What do I do if I'm waiting for it to finish for longer than a couple of minutes?**
 
     - Try closing and restarting your browser, then trying again.
@@ -99,7 +130,7 @@ Our Cheaha system is robust, but errors may occur due to general platform connec
 
     - **Do you have an active Open OnDemand session?**
 
-        - In Open OnDemand ([https://rc.uab.edu](https://rc.uab.edu)),  navigate to the green navigation bar in the top right corner. Look for the `Help` or `Developer` dropdown menu and click on it. Then, click `Restart Web Server`. Once the restart is complete, please try again.
+        - In Open OnDemand ([https://rc.uab.edu](https://rc.uab.edu)), navigate to the green navigation bar in the top right corner. Look for the `Help` or `Developer` dropdown menu and click on it. Then, click `Restart Web Server`. Once the restart is complete, please try again.
 
     - **Do you have one or more OOD HPC Desktops running?**
 

diff --git a/docs/data_management/lts/policies.md b/docs/data_management/lts/policies.md
@@ -2,7 +2,7 @@
 
 A major use for LTS is storage of data that should be accessible to multiple users from a lab or research group. By default, buckets are only visible and accessible to the owner of the bucket, and no mechanism exists to search for buckets other users have created.
 
-Instead, sharing buckets must be done through the command line using [bucket policies](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-policies.html). A bucket policy is a JSON formatted file that assigns user read and write permissions to the bucket and to objects within the bucket. If you have not worked with JSON files before, a brief explantion can be found [here](https://docs.fileformat.com/web/json/). It's important to note that the bucket owner will always retain the ability to perform all actions on a bucket and its contents and so do not need to be explicitly granted permissions.
+Instead, sharing buckets must be done through the command line using [bucket policies](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-policies.html). A bucket policy is a JSON formatted file that assigns user read and write permissions to the bucket and to objects within the bucket. If you have not worked with JSON files before, a brief explanation can be found [here](https://docs.fileformat.com/web/json/). It's important to note that the bucket owner will always retain the ability to perform all actions on a bucket and its contents and so do not need to be explicitly granted permissions.
 
 <!-- markdownlint-disable MD046 -->
 !!! important
@@ -248,6 +248,74 @@ In some instances, the bucket owner (i.e. ideally the PI for the lab if this is
 }
 ```
 
+## Comments in S3 IAM Policies
+
+IAM policies for `S3`, used by LTS for object-level access control within buckets, are written in `JSON` format. Since `JSON` does not support comments natively, `AWS` does not provide a dedicated comment field in their IAM policy schema.
+
+The optional `SID` field in IAM policies, though intended for uniquely identifying statements, can also be used as an ad-hoc comment. In the example below, the `SID` field provides a description of the statement, serving as a comment.
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "This statement grants read access to all objects in bucket1",
+      "Effect": "Allow",
+      "Principal": {
+        "AWS":[
+            "arn:aws:iam:::user/bob",
+            "arn:aws:iam:::user/jane@uab.edu"
+        ]
+      },
+      "Action": [
+        "s3:GetObject"
+      ],
+      "Resource": [
+        "arn:aws:s3:::bucket1",
+        "arn:aws:s3:::bucket1/*"
+      ]
+    }
+  ]
+}
+```
+
+## Specifying "all actions" in IAM Policies
+
+To allow or deny all actions on a specific resource, such as an `S3` bucket or object, use the following `Action` block as part of a `Statement` object to specify that all actions are affected by the statement:
+
+```Json
+    "Action": [
+    "s3:*"
+    ],
+```
+
+Here is an example IAM policy that grants all `S3` actions on bucket1 and all its objects:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "This statement grants access to all S3 actions in bucket1",
+      "Effect": "Allow",
+      "Principal": {
+        "AWS":[
+             "arn:aws:iam:::user/bob",
+             "arn:aws:iam:::user/jane@uab.edu"
+        ]
+      },
+      "Action": [
+        "s3:*"
+      ],
+      "Resource": [
+        "arn:aws:s3:::bucket1",
+        "arn:aws:s3:::bucket1/*"
+      ]
+    }
+  ]
+}
+```
+
 ## Applying a Policy
 
 Policies can be applied to a bucket either by the owner or by a user who has been given the `s3:PutBucketPolicy` permission. Use s3cmd to apply policies.

diff --git a/docs/data_management/res/storage_overview.csv b/docs/data_management/res/storage_overview.csv
@@ -1,10 +1,10 @@
-Platform                      ,[Long-term Storage](lts/index.md)                                                         ,[User Data and Home Directories](storage.md#user-data-and-home-directories) ,[Project Directories](storage.md#project-directory)                                                                                                                                                     ,[User Scratch](./storage.md#user-scratch)                               ,[Local Scratch](storage.md#local-scratch)
-Data Use-Case                 ,Data that rarely or never changes. Static data hosting. Acquired data pick-up repository. ,General-purpose data storage.                                               ,Data to be shared among collaborator group and ready for analysis.                                                                                                                                      ,Temporary files created during analysis. Rolling file deletion over time. ,Small files created during jobs. Must be deleted at the end of a job.
-Default Quota                 ,Individual: 5 TB<br>Group: 75 TB                                                          ,5 TB                                                                        ,25 TB                                                                                                                                                                                                   ,100 TB                                                                    ,Small
-Owner                         ,Individual: per researcher<br>Group: Lab PI or Core Director                              ,Individual Researchers                                                      ,Lab PI or Core Director                                                                                                                                                                                 ,Individual Researchers                                                    ,Individual Researchers
-Accessible From               ,- Cheaha<br>- Globus                                                                      ,- Cheaha<br>- Globus                                                        ,- Cheaha<br>- Globus                                                                                                                                                                                    ,Cheaha                                                                    ,Cheaha
-Cheaha Path                   ,No path; see [LTS Interfaces](lts/interfaces.md).                                         ,`/home/<BlazerID>` (`$HOME`) and `/data/user/<BlazerID>` (`$USER_DATA`)     ,`/data/project/<name>`                                                                                                                                                                                  ,`/scratch/<BlazerID>` (`$USER_SCRATCH`)                                   ,`/local/$SLURM_JOB_ID`
-Read/Write (IO) Speed         ,Slower                                                                                    ,Fast                                                                        ,Fast                                                                                                                                                                                                    ,Fast                                                                      ,Fastest
-Responsibilities & Procedures ,PI/Director responsible for data and access control.                                      ,                                                                            ,PI/Director responsible for data and access control.                                                                                                                                                    ,Data deleted after 30 days.                                               ,Data deleted as needed.
-Access Control                ,[Bucket policies](./lts/policies.md)                                                      ,Self only                                                                   ,[`chmod`](../workflow_solutions/shell.md#manage-permissions-of-files-and-directores-chmod) and [ACLs](../workflow_solutions/shell.md#manage-researcher-access-to-files-and-directories-getfacl-setfacl) ,Self only                                                                 ,Self only
-How to Request                ,Upon request.                                                                             ,Comes with Cheaha account.                                                  ,Upon request.                                                                                                                                                                                           ,Comes with Cheaha account.                                                ,Comes with Cheaha account.
+Platform,[Long-term Storage](lts/index.md),[User Data and Home Directories](storage.md#user-data-and-home-directories),[Project Directories](storage.md#project-directory),[User Scratch](./storage.md#user-scratch),[Local Scratch](storage.md#local-scratch)
+Data Use-Case,Data that rarely or never changes. Static data hosting. Acquired data pick-up repository.,General-purpose data storage.,Data to be shared among collaborator group and ready for analysis.,Temporary files created during analysis. Rolling file deletion over time.,Small files created during jobs. Must be deleted at the end of a job.
+Default Quota,Individual: 5 TB<br>Group: 75 TB,5 TB,25 TB,100 TB,Small
+Owner,Individual: per researcher<br>Group: Lab PI or Core Director,Individual Researchers,Lab PI or Core Director,Individual Researchers,Individual Researchers
+Accessible From,- Cheaha<br>- Globus,- Cheaha<br>- Globus,- Cheaha<br>- Globus,Cheaha,Cheaha
+Cheaha Path,No path; see [LTS Interfaces](lts/interfaces.md).,`/home/<BlazerID>` (`$HOME`) and `/data/user/<BlazerID>` (`$USER_DATA`),`/data/project/<name>`,`/scratch/<BlazerID>` (`$USER_SCRATCH`),`/local/$SLURM_JOB_ID`
+Read/Write (IO) Speed,Slower,Fast,Fast,Fast,Fastest
+Responsibilities & Procedures,PI/Director responsible for data and access control.,,PI/Director responsible for data and access control.,Data deleted after 30 days.,Data deleted as needed.
+Access Control,[Bucket policies](./lts/policies.md),Self only,[`chmod`](../workflow_solutions/shell.md#manage-permissions-of-files-and-directores-chmod) and [ACLs](../workflow_solutions/shell.md#manage-researcher-access-to-files-and-directories-getfacl-setfacl),Self only,Self only
+How to Request,Upon request.,Comes with Cheaha account.,Upon request.,Comes with Cheaha account.,Comes with Cheaha account.
diff --git a/docs/data_management/storage.md b/docs/data_management/storage.md
@@ -225,15 +225,23 @@ rm -rf "$TMPDIR"
     Using `/local/$SLURM_JOB_ID` with MPI jobs takes additional consideration. If you do not need MPI, please use the `#SBATCH --nodes=1` slurm directive to specify that all requested cores are on the same node. If you need the performance of `/local/$SLURM_JOB_ID` in an MPI job, please contact [Support](../help/support.md) and read about the Slurm commands `sbcast` and `sgather`.
 <!-- markdownlint-enable MD046 -->
 
-## Temporary Files (`tmp`)
+<!-- markdownlint-disable MD046 -->
+!!! note
+
+    By default the variable `$TMPDIR` points to `/scratch/local/` which in turn is a symbolic link to `/local/`. The variable `$LOCAL_SCRATCH` is identical to `$TMPDIR`.
+
+    We recommend overwriting `$TMPDIR`, as above, because it will ensure that jobs always write to a safe location on the same node the work is being done.
+<!-- markdownlint-enable MD046 -->
+
+## Temporary Files (`/tmp/` directory)
 
-Please do not use the directory `tmp` as storage for temporary files. The `tmp` directory is local to each node, and a full `tmp` directory harms compute performance on that node for all users. Instead, please use [`/local/$SLURM_JOB_ID`](#local-scratch) for fast access and [`$USER_SCRATCH`](#user-scratch) for larger space.
+Please do not use the directory `/tmp/` as storage for temporary files. The `/tmp/` directory is local to each node, and a full `/tmp/` directory harms compute performance on that node for all users. Instead, please use [`/local/$SLURM_JOB_ID`](#local-scratch) for fast access and [`$USER_SCRATCH`](#user-scratch) for larger space.
 
-Some software defaults to using `tmp` without any warning or documentation, especially software designed for personal computers. We may reach out to inform you if your software fills `tmp`, as it can harm performance on that compute node. If that happens we will work with you to identify ways of redirecting temporary storage to one of the scratch spaces.
+Some software defaults to using `/tmp/` without any warning or documentation, especially software designed for personal computers. We may reach out to inform you if your software fills `/tmp/`, as it can harm performance on that compute node. If that happens we will work with you to identify ways of redirecting temporary storage to one of the scratch spaces.
 
-### Software Known to Use `tmp`
+### Software Known to Use `/tmp/`
 
-The following software are known to use `tmp` by default, and can be worked around by using the listed flags. See [Local Scratch](#local-scratch) for more information about creating a local temporary directory.
+The following software are known to use `/tmp/` by default, and can be worked around by using the listed flags. See [Local Scratch](#local-scratch) for more information about creating a local temporary directory.
 
 - [Java](https://docs.oracle.com/cd/E63231_01/doc/BIAIN/GUID-94C6B992-1488-4FC7-85EC-91E410D6E7D1.htm#BIAIN-GUID-94C6B992-1488-4FC7-85EC-91E410D6E7D1): `java * -Djava.io.tmpdir=/local/$SLURM_JOB_ID`
 - [UMI Tools](https://umi-tools.readthedocs.io/en/latest/common_options.html): `umi_tools * --temp-dir=/local/$SLURM_JOB_ID`
@@ -243,7 +251,7 @@ The following software are known to use `tmp` by default, and can be worked arou
 - [FastQC](https://home.cc.umanitoba.ca/~psgendb/doc/fastqc.help): `fastqc * -d /local/$SLURM_JOB_ID`
 - [MACS2](https://manpages.org/macs2_callpeak): `macs2 callpeak * --tempdir /local/$SLURM_JOB_ID`
 
-Software known to use `tmp` by default with no know workaround.
+Software known to use `/tmp/` by default with no know workaround.
 
 - [Keras](https://github.com/tensorflow/tensorflow/blob/5bb81b7b0dd140a4304b92530614502c0c61a150/tensorflow/python/keras/utils/data_utils.py#L205) has `/tmp/.keras` hardcoded as a fallback cache directory if `~/.keras` is inaccessible. See [here](https://github.com/tensorflow/tensorflow/issues/38831) for a discussion of the issue.
 

diff --git a/docs/grants/budgets.md b/docs/grants/budgets.md
diff --git a/docs/grants/facilities.md b/docs/grants/facilities.md
@@ -5,3 +5,7 @@ The link below contains a plain-text facilities document designed to support res
 [Click Here To Download The Research Computing Facilities Grant Document.](res/uab-rc-facilities.txt)
 
 If you believe that information within is inaccurate or there are missing details, please feel free to contact [Support](../help/support.md).
+
+## Detailed Hardware Information
+
+For more detailed information on compute hardware please see: [Detailed Hardware Information](../cheaha/hardware.md#details)
diff --git a/docs/grants/opportunities.md b/docs/grants/opportunities.md
@@ -0,0 +1,3 @@
+# Funding Opportunities
+
+- Research Software Funding Opportunities: <https://www.researchsoft.org/funding-opportunities/>
diff --git a/docs/grants/overview.md b/docs/grants/overview.md