Skip to content

Commit

Permalink
Merge branch 'dev' into nf-core-template-merge-2.14.1
Browse files Browse the repository at this point in the history
  • Loading branch information
Felix-Kummer committed May 24, 2024
2 parents d6e1398 + 6fb5757 commit edf0b33
Show file tree
Hide file tree
Showing 40 changed files with 1,787 additions and 600 deletions.
1 change: 0 additions & 1 deletion .github/workflows/awsfulltest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ jobs:
steps:
- name: Launch workflow via Seqera Platform
uses: seqeralabs/action-tower-launch@v2
# TODO nf-core: You can customise AWS full pipeline tests as required
# Add full size test data (but still relatively small datasets for few samples)
# on the `test_full.config` test runs with only one set of parameters
with:
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ jobs:
uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1

- name: Run pipeline with test data
# TODO nf-core: You can customise CI pipeline run tests as required
# For example: adding multiple test runs with different parameters
# Remember that you can parallelise this by using strategy.matrix
run: |
Expand Down
3 changes: 3 additions & 0 deletions .nf-core.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,6 @@ template:
prefix: nf-core
skip:
- igenomes
lint:
files_exist:
- conf/igenomes.config
10 changes: 8 additions & 2 deletions CITATIONS.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# nf-core/rangeland: Citations

## [FORCE on Nextflow](https://www.informatik.hu-berlin.de/de/forschung/gebiete/wbi/research/publications/2021/force_nextflow.pdf/@@download/file/force_nextflow.pdf)

This paper describes technical details of deploying the FORCE tool in a Nextflow workflow. The workflow developed in this paper is the foundation for this nf-core pipeline.

> Lehmann, F., Frantz, D., Becker, S., Leser, U., Hostert, P. (2021). FORCE on Nextflow: Scalable Analysis of Earth Observation Data on Commodity Clusters. In CIKM Workshops.
## [nf-core](https://pubmed.ncbi.nlm.nih.gov/32055031/)

> Ewels PA, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol. 2020 Mar;38(3):276-278. doi: 10.1038/s41587-020-0439-x. PubMed PMID: 32055031.
Expand All @@ -10,9 +16,9 @@
## Pipeline tools

- [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/)
- [FORCE](https://www.mdpi.com/2072-4292/11/9/1124)

> Andrews, S. (2010). FastQC: A Quality Control Tool for High Throughput Sequence Data [Online].
> Frantz, D. (2019). FORCE—Landsat + Sentinel-2 Analysis Ready Data and Beyond. Remote Sensing, 11, 1124
- [MultiQC](https://pubmed.ncbi.nlm.nih.gov/27312411/)

Expand Down
82 changes: 47 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,50 +19,36 @@

## Introduction

**nf-core/rangeland** is a bioinformatics pipeline that ...
**nf-core/rangeland** is a geographical best-practice analysis pipeline for remotely sensed imagery. The pipeline processes satellite imagery alongside auxiliary data in multiple steps to arrive at a set of trend files related to land-cover changes. The main pipeline steps are:

<!-- TODO nf-core:
Complete this sentence with a 2-3 sentence summary of what types of data the pipeline ingests, a brief overview of the
major pipeline sections and the types of output it produces. You're giving an overview to someone new
to nf-core here, in 15-20 seconds. For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#introduction
-->
1. Read satellite imagery, digital elevation model, endmember definition, water vapor database and area of interest definition
2. Generate allow list and analysis mask to determine which pixels from the satellite data can be used
3. Preprocess data to obtain atmospherically corrected images alongside quality assurance information
4. Classify pixels by applying linear spectral unmixing
5. Time series analyses to obtain trends in vegetation dynamics
6. Create mosaic and pyramid visualizations of the results

<!-- TODO nf-core: Include a figure that guides the user through the major workflow steps. Many nf-core
workflows use the "tube map" design for that. See https://nf-co.re/docs/contributing/design_guidelines#examples for examples. -->
<!-- TODO nf-core: Fill in short bullet-pointed list of the default steps in the pipeline -->

1. Read QC ([`FastQC`](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/))
2. Present QC for raw reads ([`MultiQC`](http://multiqc.info/))
7. Read QC ([`FastQC`](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/))
8. Present QC for raw reads ([`MultiQC`](http://multiqc.info/))

## Usage

> [!NOTE]
> If you are new to Nextflow and nf-core, please refer to [this page](https://nf-co.re/docs/usage/installation) on how to set-up Nextflow. Make sure to [test your setup](https://nf-co.re/docs/usage/introduction#how-to-run-a-pipeline) with `-profile test` before running the workflow on actual data.
<!-- TODO nf-core: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
Explain what rows and columns represent. For instance (please edit as appropriate):
First, prepare a samplesheet with your input data that looks as follows:
`samplesheet.csv`:
```csv
sample,fastq_1,fastq_2
CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz
```
Each row represents a fastq file (single-end) or a pair of fastq files (paired end).
-->
To run the pipeline on real data, input data needs to be acquired. Concretely, satellite imagery, water vapor data, a digital elevation model, endmember definitions, a datacube specification, and a area-of-interest specification are required. Please refer to the [usage documentation](https://nf-co.re/rangeland/usage) for details on the input structure.

Now, you can run the pipeline using:

<!-- TODO nf-core: update the following command to include all required parameters for a minimal example -->

```bash
nextflow run nf-core/rangeland \
nextflow run nf-core/rangeland/main.nf \
-profile <docker/singularity/.../institute> \
--input samplesheet.csv \
--input <SATELLITE IMAGES> \
--dem <DIGITAL ELEVATION MODEL> \
--wvdb <WATER VAPOR DATA> \
--data_cube <DATA CUBE> \
--aoi <AREA OF INTEREST> \
--endmember <ENDMEMBER SPECIFICATION> \
--outdir <OUTDIR>
```

Expand All @@ -80,11 +66,33 @@ For more details about the output files and reports, please refer to the

## Credits

nf-core/rangeland was originally written by Fabian Lehmann, David Frantz, Felix Kummer.
The rangeland workflow was originally written by:

- [Fabian Lehmann](https://github.com/Lehmann-Fabian)
- [David Frantz](https://github.com/davidfrantz)

The original workflow can be found on [github](https://github.com/CRC-FONDA/FORCE2NXF-Rangeland).

Transformation to nf-core/rangeland was conducted by [Felix Kummer](https://github.com/Felix-Kummer). nf-core alignment started on the [nf-core branch of the original repository](https://github.com/CRC-FONDA/FORCE2NXF-Rangeland/tree/nf-core).

We thank the following people for their extensive assistance in the development of this pipeline:

<!-- TODO nf-core: If applicable, make list of people who have also contributed -->
- [Fabian Lehmann](https://github.com/Lehmann-Fabian)
- [Katarzyna Ewa Lewinska](https://github.com/kelewinska).

## Acknowledgements

This pipeline was developed and aligned with nf-core as part of the [Foundations of Workflows for Large-Scale Scientific Data Analysis (FONDA)](https://fonda.hu-berlin.de/) initiative.

[![FONDA](docs/images/fonda_logo2_cropped.png)](https://fonda.hu-berlin.de/)

FONDA can be cited as follows:

> **The Collaborative Research Center FONDA.**
>
> Ulf Leser, Marcus Hilbrich, Claudia Draxl, Peter Eisert, Lars Grunske, Patrick Hostert, Dagmar Kainmüller, Odej Kao, Birte Kehr, Timo Kehrer, Christoph Koch, Volker Markl, Henning Meyerhenke, Tilmann Rabl, Alexander Reinefeld, Knut Reinert, Kerstin Ritter, Björn Scheuermann, Florian Schintke, Nicole Schweikardt, Matthias Weidlich.
>
> _Datenbank Spektrum_ 2021 doi: [10.1007/s13222-021-00397-5](https://doi.org/10.1007/s13222-021-00397-5)
## Contributions and Support

Expand All @@ -97,8 +105,6 @@ For further information or help, don't hesitate to get in touch on the [Slack `#
<!-- TODO nf-core: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file. -->
<!-- If you use nf-core/rangeland for your analysis, please cite it using the following doi: [10.5281/zenodo.XXXXXX](https://doi.org/10.5281/zenodo.XXXXXX) -->

<!-- TODO nf-core: Add bibliography of tools and data used in your pipeline -->

An extensive list of references for the tools used by the pipeline can be found in the [`CITATIONS.md`](CITATIONS.md) file.

You can cite the `nf-core` publication as follows:
Expand All @@ -108,3 +114,9 @@ You can cite the `nf-core` publication as follows:
> Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.
>
> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x).
This pipeline is based one the publication listed below. The publication can be cited as follows:

> **FORCE on Nextflow: Scalable Analysis of Earth Observation Data on Commodity Clusters**
>
> [Lehmann, F., Frantz, D., Becker, S., Leser, U., Hostert, P. (2021). FORCE on Nextflow: Scalable Analysis of Earth Observation Data on Commodity Clusters. In CIKM Workshops.](https://www.informatik.hu-berlin.de/de/forschung/gebiete/wbi/research/publications/2021/force_nextflow.pdf/@@download/file/force_nextflow.pdf)
44 changes: 44 additions & 0 deletions bin/merge_boa.r
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
#!/usr/bin/env Rscript

args = commandArgs(trailingOnly=TRUE)


if (length(args) < 3) {
stop("\nthis program needs at least 3 inputs\n1: output filename\n2-*: input files", call.=FALSE)
}

fout <- args[1]
finp <- args[2:length(args)]
nf <- length(finp)

require(raster)


img <- brick(finp[1])
nc <- ncell(img)
nb <- nbands(img)


sum <- matrix(0, nc, nb)
num <- matrix(0, nc, nb)

for (i in 1:nf){

data <- brick(finp[i])[]

num <- num + !is.na(data)

data[is.na(data)] <- 0
sum <- sum + data

}

mean <- sum/num
img[] <- mean


writeRaster(img, filename = fout, format = "GTiff", datatype = "INT2S",
options = c("INTERLEAVE=BAND", "COMPRESS=LZW", "PREDICTOR=2",
"NUM_THREADS=ALL_CPUS", "BIGTIFF=YES",
sprintf("BLOCKXSIZE=%s", img@file@blockcols[1]),
sprintf("BLOCKYSIZE=%s", img@file@blockrows[1])))
38 changes: 38 additions & 0 deletions bin/merge_qai.r
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#!/usr/bin/env Rscript

args = commandArgs(trailingOnly=TRUE)


if (length(args) < 3) {
stop("\nthis program needs at least 3 inputs\n1: output filename\n2-*: input files", call.=FALSE)
}

fout <- args[1]
finp <- args[2:length(args)]
nf <- length(finp)

require(raster)


img <- raster(finp[1])
nc <- ncell(img)


last <- rep(1, nc)

for (i in 1:nf){

data <- raster(finp[i])[]

last[!is.na(data)] <- data[!is.na(data)]

}

img[] <- last


writeRaster(img, filename = fout, format = "GTiff", datatype = "INT2S",
options = c("INTERLEAVE=BAND", "COMPRESS=LZW", "PREDICTOR=2",
"NUM_THREADS=ALL_CPUS", "BIGTIFF=YES",
sprintf("BLOCKXSIZE=%s", img@file@blockcols[1]),
sprintf("BLOCKYSIZE=%s", img@file@blockrows[1])))
143 changes: 143 additions & 0 deletions bin/test.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
#!/usr/bin/env Rscript

args = commandArgs(trailingOnly=TRUE)


if (length(args) != 7) {
stop("\ngive input directory (mosaic) as 1st arg\ngive reference rasters (*.tif) as 2nd-7th args in order:
woody cover change, woody cover year of change,
herbaceous cover change, herbaceous cover year of change,
peak change, peak year of change, ", call.=FALSE)
}

dinp <- args[1]

# load package
require(terra)



# LOAD REFERENCE
#######################################################################
woody_cover_changes_ref <- rast(args[2])
woody_cover_year_of_change_ref <- rast(args[3])

herbaceous_cover_changes_ref <- rast(args[4])
herbaceous_cover_year_of_change_ref <- rast(args[5])

peak_changes_ref <- rast(args[6])
peak_year_of_change_ref <- rast(args[7])


# WOODY COVER CHANGE (VALUE OF BASE LEVEL)
#######################################################################

fname <- dir(dinp, ".*HL_TSA_LNDLG_SMA_VBL-CAO.vrt$", full.names=TRUE)

woody_cover_rast <- rast(fname)

woody_cover_changes <- woody_cover_rast$CHANGE
woody_cover_year_of_change <- woody_cover_rast["YEAR-OF-CHANGE"]



# HERBACEOUS COVER CHANGE (VALUE OF SEASONAL APLITUDE)
#######################################################################


fname <- dir(dinp, ".*HL_TSA_LNDLG_SMA_VSA-CAO.vrt$", full.names=TRUE)

herbaceous_cover_rast <- rast(fname)

herbaceous_cover_changes <- herbaceous_cover_rast$CHANGE
herbaceous_cover_year_of_change <- herbaceous_cover_rast["YEAR-OF-CHANGE"]



# VALUE OF PEAK SEASON
#######################################################################

fname <- dir(dinp, ".*HL_TSA_LNDLG_SMA_VPS-CAO.vrt$", full.names=TRUE)

peak_rast <- rast(fname)

peak_changes <- peak_rast$CHANGE
peak_year_of_change <- peak_rast["YEAR-OF-CHANGE"]



# FOR REFERENCE: SAVE RASTERS
#######################################################################

#writeRaster(woody_cover_changes, "woody_cover_chg_ref.tif")
#writeRaster(woody_cover_year_of_change, "woody_cover_yoc_ref.tif")

#writeRaster(herbaceous_cover_changes, "herbaceous_cover_chg_ref.tif")
#writeRaster(herbaceous_cover_year_of_change, "herbaceous_cover_yoc_ref.tif")

#writeRaster(peak_changes, "peak_chg_ref.tif")
#writeRaster(peak_year_of_change, "peak_yoc_ref.tif")




# COMPARE TESTRUN WITH REFERENCE EXECUTION
#######################################################################
failure <- FALSE

woody_cover_changes_result <- all.equal(woody_cover_changes, woody_cover_changes_ref)
if (is.character(woody_cover_changes_result)) {
print(paste0("Error: ", woody_cover_changes_result, " for woody cover changes."))
failure <- TRUE
} else {
print("Woody cover change check passed.")
}

woody_cover_year_of_change_result <- all.equal(woody_cover_year_of_change, woody_cover_year_of_change_ref)
if (is.character(woody_cover_year_of_change_result)) {
print(paste0("Error: ", woody_cover_year_of_change_result, " for woody cover year of change."))
failure <- TRUE
} else {
print("Woody cover year of change check passed.")
}


herbaceous_cover_changes_result <- all.equal(herbaceous_cover_changes, herbaceous_cover_changes_ref)
if (is.character(herbaceous_cover_changes_result)) {
print(paste0("Error: ",herbaceous_cover_changes_result, " for herbaceous cover changes."))
failure <- TRUE
} else {
print("Herbaceous cover change check passed.")
}

herbaceous_cover_year_of_change_result <- all.equal(herbaceous_cover_year_of_change, herbaceous_cover_year_of_change_ref)
if (is.character(herbaceous_cover_year_of_change_result)) {
print(paste0("Error: ", herbaceous_cover_year_of_change_result, " for herbaceous cover year of change."))
failure <- TRUE
} else {
print("Herbaceous cover year of change check passed.")
}


peak_changes_result <- all.equal(peak_changes, peak_changes_ref)
if (is.character(peak_changes_result)) {
print(paste0("Error: ", peak_changes_result, " for peak changes."))
failure <- TRUE
} else {
print("Peak change check passed.")
}


peak_year_of_change_result <- all.equal(peak_year_of_change, peak_year_of_change_ref)
if (is.character(peak_year_of_change_result)) {
print(paste0("Error: ", peak_year_of_change_result, " for peak year of change."))
failure <- TRUE
} else {
print("Peak year of change check passed.")
}

if (failure) {
stop("Some test failed.")
} else {
print("All checks passed.")
}
Loading

0 comments on commit edf0b33

Please sign in to comment.