Plugin update: apply-flatfield

transforms/images/apply-flatfield-plugin/.bumpversion.cfg

@@ -0,0 +1,29 @@
+[bumpversion]
+current_version = 2.0.0-dev9
+commit = False
+tag = False
+parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\-(?P<release>[a-z]+)(?P<dev>\d+))?
+serialize = 
+	{major}.{minor}.{patch}-{release}{dev}
+	{major}.{minor}.{patch}
+
+[bumpversion:part:release]
+optional_value = _
+first_value = dev
+values = 
+	dev
+	_
+
+[bumpversion:part:dev]
+
+[bumpversion:file:pyproject.toml]
+search = version = "{current_version}"
+replace = version = "{new_version}"
+
+[bumpversion:file:plugin.json]
+
+[bumpversion:file:VERSION]
+
+[bumpversion:file:README.md]
+
+[bumpversion:file:src/polus/plugins/transforms/images/apply_flatfield/__init__.py]

transforms/images/apply-flatfield-plugin/Dockerfile

@@ -0,0 +1,20 @@
+FROM polusai/bfio:2.3.3
+
+# environment variables defined in polusai/bfio
+ENV EXEC_DIR="/opt/executables"
+ENV POLUS_IMG_EXT=".ome.tif"
+ENV POLUS_TAB_EXT=".csv"
+ENV POLUS_LOG="INFO"
+
+# Work directory defined in the base container
+WORKDIR ${EXEC_DIR}
+
+COPY pyproject.toml ${EXEC_DIR}
+COPY VERSION ${EXEC_DIR}
+COPY README.md ${EXEC_DIR}
+COPY src ${EXEC_DIR}/src
+
+RUN pip3 install ${EXEC_DIR} --no-cache-dir
+
+ENTRYPOINT ["python3", "-m", "polus.plugins.transforms.images.apply_flatfield"]
+CMD ["--help"]

transforms/images/apply-flatfield-plugin/README.md

@@ -0,0 +1,57 @@
+# Apply Flatfield Plugin (v2.0.0-dev9)
+
+This WIPP plugin applies a flatfield operation on every image in a collection.
+The algorithm used to apply the flatfield is as follows:
+
+![Corrected = \frac{Original - Darkfield}{Brightfield} - Photobleach + Offset](https://render.githubusercontent.com/render/math?math=Corrected%20%3D%20%5Cfrac%7BOriginal%20-%20Darkfield%7D%7BBrightfield%7D%20-%20Photobleach%20%2B%20Offset)
+
+A brief description of the variables:
+1. ![Corrected](https://render.githubusercontent.com/render/math?math=Corrected) is the flatfield corrected image.
+2. ![Darkfield](https://render.githubusercontent.com/render/math?math=Darkfield) is the darkfield image (sometimes referred to as offset, dark current, or dark  noise). This is an image collected by the camera when the shutter is closed.
+3. ![Brightfield](https://render.githubusercontent.com/render/math?math=Brightfield) is the normalized brightfield image. This is an image collected when the shutter  is open and illumination is on. The image should contain single precision  floating point values, where ![mean(Brightfield)=1](https://render.githubusercontent.com/render/math?math=mean(Brightfield)%3D1).
+4. ![Photobleach](https://render.githubusercontent.com/render/math?math=Photobleach) is a scalar indicating how much the image has been photobleached. This is a per-image scalar offset.
+5. ![Offset](https://render.githubusercontent.com/render/math?math=Offset) is a  scalar applied to all images in the collection. If ![Photobleach](https://render.githubusercontent.com/render/math?math=Photobleach) is specified, then this plugin uses ![Offset=mean(Photobleach)](https://render.githubusercontent.com/render/math?math=Offset%3Dmean(Photobleach)).
+
+For more information on flatfielding, see the paper by [Young](https://currentprotocols.onlinelibrary.wiley.com/doi/full/10.1002/0471142956.cy0211s14).
+This plugin specifically uses the formulation from [Peng et al](https://www.nature.com/articles/ncomms14836).
+
+For more information on WIPP, visit the
+[official WIPP page](https://isg.nist.gov/deepzoomweb/software/wipp).
+
+## TODO
+
+### Additional Flatfield Formulations
+
+Implement additional formulations of flatfield correction. Specifically, the formula specified by Young:
+
+![Corrected = \frac{Original - Darkfield}{Brightfield - Darkfield} ](https://render.githubusercontent.com/render/math?math=Corrected%20%3D%20%5Cfrac%7BOriginal%20-%20Darkfield%7D%7BBrightfield%20-%20Darkfield%7D%20)
+
+Additional formulations may also include reference image free algorithms for flatfield correction, such as the [rolling ball algorithm](https://www.computer.org/csdl/magazine/co/1983/01/01654163/13rRUwwJWBB).
+
+### Photobleach Correction
+
+Since the `basicpy` package and the `basic-flatfield-estimation` tool do not yet support photobleach estimation, this plugin does not yet support photobleach correction.
+Once they add support for photobleach estimation, this plugin should be updated to support it.
+
+## Building
+
+To build the Docker image for the conversion plugin, run `./build-docker.sh`.
+
+## Install WIPP Plugin
+
+If WIPP is running, navigate to the plugins page and add a new plugin.
+Paste the contents of `plugin.json` into the pop-up window and submit.
+
+## Options
+
+Command line options:
+
+| Name             | Description                                                           | I/O    | Type       |
+|------------------|-----------------------------------------------------------------------|--------|------------|
+| `--imgDir`       | Input image collection to be processed by this plugin                 | Input  | collection |
+| `--imgPattern`   | Filename pattern used to separate data and match with flatfield files | Input  | string     |
+| `--ffDir`        | Image collection containing flatfield and/or darkfield images         | Input  | collection |
+| `--ffPattern`    | Filename pattern used to match flatfield files to image files         | Input  | string     |
+| `--dfPattern`    | Filename pattern used to match darkfield files to image files         | Input  | string     |
+| `--outDir`       | Output collection                                                     | Output | collection |
+| `--preview`      | Preview the output images' names without actually running computation | Input  | boolean    |

transforms/images/apply-flatfield-plugin/VERSION

@@ -0,0 +1 @@
+2.0.0-dev9

transforms/images/apply-flatfield-plugin/build-docker.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+
+version=$(<VERSION)
+docker build . -t polusai/apply-flatfield-plugin:${version}

transforms/images/apply-flatfield-plugin/plugin.json

@@ -0,0 +1,94 @@
+{
+  "name": "Apply Flatfield",
+  "version": "2.0.0-dev9",
+  "title": "Apply Flatfield",
+  "description": "Apply a flatfield algorithm to a collection of images.",
+  "author": "Nick Schaub (Nick.Schaub@nih.gov), Najib Ishaq (najib.ishaq@nih.gov)",
+  "institution": "National Center for Advancing Translational Sciences, National Institutes of Health",
+  "repository": "https://github.com/labshare/polus-plugins",
+  "website": "https://ncats.nih.gov/preclinical/core/informatics",
+  "citation": "",
+  "containerId": "polusai/apply-flatfield-plugin:2.0.0-dev9",
+  "baseCommand": [
+    "python3",
+    "-m",
+    "polus.plugins.transforms.images.apply_flatfield"
+  ],
+  "inputs": [
+    {
+      "name": "imgDir",
+      "type": "collection",
+      "description": "Input image collection to be processed by this plugin",
+      "required": true
+    },
+    {
+      "name": "imgPattern",
+      "type": "string",
+      "description": "Filename pattern used to separate data and match with flatfied files",
+      "required": true
+    },
+    {
+      "name": "ffDir",
+      "type": "collection",
+      "description": "Image collection containing flatfield and/or darkfield images",
+      "required": true
+    },
+    {
+      "name": "ffPattern",
+      "type": "string",
+      "description": "Filename pattern used to match flatfield files to image files",
+      "required": true
+    },
+    {
+      "name": "dfPattern",
+      "type": "string",
+      "description": "Filename pattern used to match darkfield files to image files",
+      "required": false
+    },
+    {
+      "name": "preview",
+      "type": "boolean",
+      "description": "Preview the output images' names without actually running computation",
+      "required": false
+    }
+  ],
+  "outputs": [
+    {
+      "name": "outDir",
+      "type": "collection",
+      "description": "Output collection"
+    }
+  ],
+  "ui": [
+    {
+      "key": "inputs.imgDir",
+      "title": "Images to correct",
+      "description": "Input image collection to be processed by this plugin"
+    },
+    {
+      "key": "inputs.imgPattern",
+      "title": "Image pattern",
+      "description": "Filename pattern used to separate data and match with flatfield files"
+    },
+    {
+      "key": "inputs.ffDir",
+      "title": "Background images (flatfield/darkfield)",
+      "description": "Image collection containing flatfield and/or darkfield images"
+    },
+    {
+      "key": "inputs.ffPattern",
+      "title": "Flatfield file pattern",
+      "description": "Filename pattern used to match flatfield files to image files"
+    },
+    {
+      "key": "inputs.dfPattern",
+      "title": "Darkfield file pattern",
+      "description": "Filename pattern used to match darkfield files to image files"
+    },
+    {
+      "key": "inputs.preview",
+      "title": "Preview Output",
+      "description": "Preview the output images' names without actually running computation"
+    }
+  ]
+}

transforms/images/apply-flatfield-plugin/pyproject.toml

@@ -0,0 +1,34 @@
+[tool.poetry]
+name = "polus-plugins-transforms-images-apply-flatfield"
+version = "2.0.0-dev9"
+description = ""
+authors = [
+    "Nick Schaub <nicholas.schaub@nih.gov>",
+    "Najib Ishaq <najib.ishaq@nih.gov>"
+]
+readme = "README.md"
+packages = [{include = "polus", from = "src"}]
+
+[tool.poetry.dependencies]
+python = ">=3.9,<3.12"
+bfio = { version = "^2.3.3", extras = ["all"] }
+filepattern = "^2.0.4"
+typer = { version = "^0.7.0", extras = ["all"] }
+numpy = "^1.24.3"
+tqdm = "^4.65.0"
+preadator = "0.4.0-dev2"
+
+[tool.poetry.group.dev.dependencies]
+bump2version = "^1.0.1"
+pre-commit = "^3.0.4"
+pytest = "^7.2.1"
+pytest-sugar = "^0.9.6"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.pytest.ini_options]
+pythonpath = [
+  "."
+]

transforms/images/apply-flatfield-plugin/run-plugin.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+version=$(<VERSION)
+datapath=$(readlink --canonicalize ./data)
+echo ${datapath}
+
+# Inputs
+imgDir="/data/images"
+imgPattern="p{p:d+}_x{x:d+}_y{y:d+}_wx{r:d+}_wy{z:d+}_c{c:d+}.ome.tif"
+ffDir="/data/estimation"
+ffPattern="p{p:d+}_x\\(01-24\\)_y\\(01-16\\)_wx\\(1-3\\)_wy\\(1-3\\)_c{c:d+}_flatfield.ome.tif"
+dfPattern="p{p:d+}_x\\(01-24\\)_y\(01-16\\)_wx\\(1-3\\)_wy\\(1-3\\)_c{c:d+}_darkfield.ome.tif"
+# photoPattern=""
+
+# Output paths
+outDir=/data/outputs
+
+POLUS_IMG_EXT=".ome.zarr"
+
+docker run --mount type=bind,source=${datapath},target=/data/ \
+            -e POLUS_IMG_EXT=${POLUS_IMG_EXT} \
+            --user $(id -u):$(id -g) \
+            polusai/apply-flatfield-plugin:${version} \
+            --imgDir ${imgDir} \
+            --imgPattern ${imgPattern} \
+            --ffDir ${ffDir} \
+            --ffPattern ${ffPattern} \
+            --ffPattern ${dfPattern} \
+            --outDir ${outDir}

transforms/images/apply-flatfield-plugin/src/polus/plugins/transforms/images/apply_flatfield/__init__.py

@@ -0,0 +1,6 @@
+"""Provides the apply_flatfield module."""
+
+from . import utils
+from .apply_flatfield import apply
+
+__version__ = "2.0.0-dev9"

transforms/images/apply-flatfield-plugin/src/polus/plugins/transforms/images/apply_flatfield/__main__.py

@@ -0,0 +1,108 @@
+"""Provides the CLI for the Apply Flatfield plugin."""
+
+import json
+import logging
+import pathlib
+import typing
+
+import typer
+from polus.plugins.transforms.images.apply_flatfield import apply
+from polus.plugins.transforms.images.apply_flatfield import utils
+
+# Initialize the logger
+logging.basicConfig(
+    format="%(asctime)s - %(name)-8s - %(levelname)-8s - %(message)s",
+    datefmt="%d-%b-%y %H:%M:%S",
+)
+logger = logging.getLogger("polus.plugins.transforms.images.apply_flatfield")
+logger.setLevel(utils.POLUS_LOG)
+
+app = typer.Typer()
+
+
+@app.command()
+def main(  # noqa: PLR0913
+    img_dir: pathlib.Path = typer.Option(
+        ...,
+        "--imgDir",
+        help="Path to input images.",
+        exists=True,
+        readable=True,
+        resolve_path=True,
+        file_okay=False,
+    ),
+    img_pattern: str = typer.Option(
+        ...,
+        "--imgPattern",
+        help="Filename pattern used to select images from imgDir.",
+    ),
+    ff_dir: pathlib.Path = typer.Option(
+        ...,
+        "--ffDir",
+        help="Path to flatfield (and optionally darkfield) images.",
+        exists=True,
+        readable=True,
+        resolve_path=True,
+        file_okay=False,
+    ),
+    ff_pattern: str = typer.Option(
+        ...,
+        "--ffPattern",
+        help="Filename pattern used to select flatfield components from ffDir.",
+    ),
+    df_pattern: typing.Optional[str] = typer.Option(
+        None,
+        "--dfPattern",
+        help="Filename pattern used to select darkfield components from ffDir.",
+    ),
+    out_dir: pathlib.Path = typer.Option(
+        ...,
+        "--outDir",
+        help="Path to output directory.",
+        exists=True,
+        writable=True,
+        resolve_path=True,
+        file_okay=False,
+    ),
+    preview: bool = typer.Option(
+        False,
+        "--preview",
+        help="Preview the output without saving.",
+    ),
+) -> None:
+    """CLI for the Apply Flatfield plugin.
+
+    The variables used in ffPattern and dfPattern must be a subset of those used
+    in imgPattern.
+
+    If dfPattern is not specified, then darkfield correction will not be
+    applied.
+    """
+    logger.info("Starting Apply Flatfield plugin ...")
+
+    logger.info(f"imgDir = {img_dir}")
+    logger.info(f"imgPattern = {img_pattern}")
+    logger.info(f"ffDir = {ff_dir}")
+    logger.info(f"ffPattern = {ff_pattern}")
+    logger.info(f"dfPattern = {df_pattern}")
+    logger.info(f"outDir = {out_dir}")
+    logger.info(f"preview = {preview}")
+
+    out_files = apply(
+        img_dir=img_dir,
+        img_pattern=img_pattern,
+        ff_dir=ff_dir,
+        ff_pattern=ff_pattern,
+        df_pattern=df_pattern,
+        out_dir=out_dir,
+        preview=preview,
+    )
+
+    if preview:
+        with out_dir.joinpath("preview.json").open("w") as writer:
+            out_dict = {"files": [p.name for p in out_files]}
+            json.dump(out_dict, writer, indent=2)
+
+
+if __name__ == "__main__":
+    app()