Merge pull request #142 from /issues/36

Issues/36: Subsetting
podaac · Aug 22, 2023 · 2e4565b · 2e4565b
2 parents aebd818 + 12ca7b4
commit 2e4565b
Show file tree

Hide file tree

Showing 14 changed files with 954 additions and 331 deletions.
diff --git a/.github/workflows/python-app.yml b/.github/workflows/python-app.yml
@@ -16,7 +16,7 @@ jobs:
       max-parallel: 2
       matrix:
         python-version: [ "3.7", "3.8", "3.9", "3.10" ]
-        poetry-version: [ "1.1.14" ]
+        poetry-version: [ "1.5.1" ]
         os: [ ubuntu-22.04, macos-latest, windows-latest ]
     runs-on: ${{ matrix.os }}
     steps:
@@ -63,7 +63,7 @@ jobs:
       max-parallel: 1
       matrix:
         python-version: [ "3.7", "3.8", "3.9", "3.10" ]
-        poetry-version: [ "1.1.14" ]
+        poetry-version: [ "1.5.1" ]
         os: [ ubuntu-22.04, macos-latest, windows-latest ]
     runs-on: ${{ matrix.os }}
     steps:

diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -20,7 +20,7 @@ jobs:
     - name: Install Poetry
       uses: abatilo/actions-poetry@v2.0.0
       with:
-        poetry-version: 1.1.14
+        poetry-version: '1.5.1'
     - name: Install dependencies
       run: |
         poetry install

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,6 +3,10 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
+## [Unreleased]
+### Added
+- Added support for Harmony subsetting
+
 ## [1.14.0]
 ### Added
 - Added support for wildcard search patterns in podaac-data-downloader when executed with the -gr option (i.e. search/download by CMR Granule Ur/Id). Also, added usage details to Downloader.md to describe this new feature [138](https://github.com/podaac/data-subscriber/pull/138).

diff --git a/Downloader.md b/Downloader.md
@@ -41,6 +41,7 @@ optional arguments:
                         Specify a provider for collection search. Default is POCLOUD.
   --limit LIMIT         Integer limit for number of granules to download. Useful in testing. Defaults to no limit.
   --dry-run             Search and identify files to download, but do not actually download them
+  --subset              Flag to enable subsetting on the specified collection 
 
 ```
 
@@ -236,7 +237,9 @@ and
 
 Using the `--process` option, you can run a simple command agaisnt the "just" downloaded file. This will take the format of "<command> <path/to/file>". This means you can run a command like `--process gzip` to gzip all downloaded files. We do not support more advanced processes at this time (piping, running a process on a directory, etc).
 
+### Granule subsetting
 
+To enable granule subsetting, include the `--subset` flag in your request. This will invoke the NASA Harmony API to subset the granules in the specified collection. The collection must have subsetting enabled for this feature to function. If it does not, the data will be downloaded normally. 
 
 ### In need of Help?
 The PO.DAAC User Services Office is the primary point of contact for answering your questions concerning data and information held by the PO.DAAC. User Services staff members are knowledgeable about both the data ordering system and the data products themselves. We answer questions about data, route requests to other DAACs, and direct questions we cannot answer to the appropriate information source.

diff --git a/Subscriber.md b/Subscriber.md
@@ -35,6 +35,7 @@ $> podaac-data-subscriber -h
 -p PROVIDER, --provider PROVIDER
                       Specify a provider for collection search. Default is POCLOUD.
 --dry-run             Search and identify files to download, but do not actually download them
+--subset              Flag to enable subsetting on the specified collection
 ```
 
 ## Run the Script
@@ -212,6 +213,9 @@ and
 
 Using the `--process` option, you can run a simple command agaisnt the "just" downloaded file. This will take the format of "<command> <path/to/file>". This means you can run a command like `--process gzip` to gzip all downloaded files. We do not support more advanced processes at this time (piping, running a process on a directory, etc).
 
+### Granule subsetting
+
+To enable granule subsetting, include the `--subset` flag in your request. This will invoke the NASA Harmony API to subset the granules in the specified collection. The collection must have subsetting enabled for this feature to function. If it does not, the data will be downloaded normally.
 
 ### In need of Help?
 The PO.DAAC User Services Office is the primary point of contact for answering your questions concerning data and information held by the PO.DAAC. User Services staff members are knowledgeable about both the data ordering system and the data products themselves. We answer questions about data, route requests to other DAACs, and direct questions we cannot answer to the appropriate information source.

diff --git a/poetry.lock b/poetry.lock
diff --git a/pyproject.toml b/pyproject.toml
@@ -16,6 +16,7 @@ python = "^3.7"
 requests = "^2.27.1"
 tenacity = "^8.0.1"
 packaging = "^23.0"
+harmony-py = "^0.4.9"
 
 [tool.poetry.dev-dependencies]
 pytest = "^7.1.2"