Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add sentinel 2 script #335

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions sentinel2-processing/nz-150k-tile-index.cpg
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
UTF-8
Binary file added sentinel2-processing/nz-150k-tile-index.dbf
Binary file not shown.
1 change: 1 addition & 0 deletions sentinel2-processing/nz-150k-tile-index.prj
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
PROJCS["NZGD_2000_New_Zealand_Transverse_Mercator",GEOGCS["GCS_NZGD_2000",DATUM["D_NZGD_2000",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",1600000.0],PARAMETER["False_Northing",10000000.0],PARAMETER["Central_Meridian",173.0],PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_Of_Origin",0.0],UNIT["Meter",1.0]]
Binary file added sentinel2-processing/nz-150k-tile-index.shp
Binary file not shown.
Binary file added sentinel2-processing/nz-150k-tile-index.shx
Binary file not shown.
167 changes: 167 additions & 0 deletions sentinel2-processing/prepare_10m_sentinel_imagery.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
#!/usr/bin/env python3

"""
This is a Python version of a bash script written by Rebecca Clarke https://gist.github.com/rebclarke/b9e55a3558be3ec76248c78379b67366
"""

import os
import glob
import shutil
from osgeo import gdal
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't look like you're using this import anywhere. Did you mean to change the subprocess uses below to use this library? That would be preferable.

import subprocess


# user definated variables
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These aren't really user defined anymore if they're in the script. If you want to pass them as arguments, I'd recommend looking into argparse (example).

TILE_INDEX = "nz-150k-tile-index.shp"
MIN = "800"
MAX = "3000"
Comment on lines +16 to +17
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are passed to gdal_translate -scale; based on its documentation, would it be useful to name them something like MIN_SCALE and MAX_SCALE?

CWD = "C:/Temp/image-processing/test"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"CWD" ("current working directory") has a specific meaning in operating systems, being the directory from which a program was executed (not the directory where the script lives). You can get this directory using os.getcwd().

SENTINEL_IMAGE_DIR = "R10m"
GDAL_DIR = "C:/Program Files/QGIS 3.34.4/apps/Python39/Scripts"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very brittle, because the directory path will be different for anyone using a different OS or version of QGIS. This can be made more flexible in a couple of ways:

  • Pass the directory as an option, like --gdal-dir="C:/Program Files/QGIS 3.34.4/apps/Python39/Scripts".
  • If files from that directory are discoverable by the surrounding shell, that is, if you can run for example gdal_merge.py without specifying its directory, then you can use shutil.which("gdal_merge.py") to get its absolute path.



def make_directories(dir_name):
"""
Create directors for the various processed images to be saved into.
"""
path = os.path.join(CWD, dir_name)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you from os.path import join you can shorten these expressions. But this is a stylistic choice, so I'll leave it to you whether you want to do that.

if os.path.exists(path):
shutil.rmtree(path)
Comment on lines +28 to +29
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very risky! It would be pretty easy for a programming mistake to delete a bunch of files which shouldn't be deleted. I'd recommend either:

  • raising an exception telling the user what to do, or
  • using tempfile.mkdtemp to create a new random directory every run and tell the user which directory to look for their data in.

os.mkdir(path)


# make output directories
make_directories("output")
make_directories("output/merged")
make_directories("output/rescaled")
make_directories("output/nearblack")
make_directories("output/reprojected")
make_directories("output/tiled")


path = os.path.join(CWD, SENTINEL_IMAGE_DIR)
rgb_files = glob.glob("R10m/*B0[2-4]_10m.jp2")

blue = rgb_files[0]
green = rgb_files[1]
red = rgb_files[2]
Comment on lines +43 to +47
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If rgb_files is supposed to have exactly three files you can unpack all of them in one go:

Suggested change
rgb_files = glob.glob("R10m/*B0[2-4]_10m.jp2")
blue = rgb_files[0]
green = rgb_files[1]
red = rgb_files[2]
blue, green, red = glob.glob("R10m/*B0[2-4]_10m.jp2")


gdal_merge_path = os.path.join(GDAL_DIR, "gdal_merge.py")

merged_output = os.path.join(CWD, "output/merged", "merged.tif")
translate_output = os.path.join(CWD, "output/rescaled", "rescaled.tif")
nearblack_output = os.path.join(CWD, "output/nearblack", "nearblack.tif")
reprojected_output = os.path.join(CWD, "output/reprojected", "reprojected.tif")
tiled_output_dir = os.path.join(CWD, "output/tiled")
vrt_output = os.path.join(CWD, "output", "overall.vrt")

# merge the three R,G,B images together to create a single RGB image
print("Merging image tiles into one RGB image")
subprocess.run(
[
"python.exe",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you run just "python" instead, it should work on other platforms without changes. But this line shouldn't even be necessary, since gdal_merge.py should be executable.

gdal_merge_path,
"-separate",
"-co",
"PHOTOMETRIC=RGB",
"-o",
merged_output,
red,
green,
blue,
]
)

# rescale the imagery to 8bit unsigned
print("Rescaling to 8bit unsigned")
subprocess.run(
[
"gdal_translate",
"-ot",
"Byte",
"-of",
"GTiff",
"-scale",
MIN,
MAX,
"1",
"255",
Comment on lines +87 to +88
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the documentation, "If omitted the output range is 0 to 255". Do we actually want to use the output range 1 to 255 instead?

"-b",
"1",
"-b",
"2",
"-b",
"3",
"-co",
"compress=lzw",
merged_output,
translate_output,
]
)

# run nearblack
print("Running nearblack and setting alpha channel")
subprocess.run(
[
"nearblack",
"-of",
"GTiff",
"-setalpha",
"-o",
nearblack_output,
translate_output,
]
)

# reproject to NZTM 2000
print("Reprojecting to NZTM 2000")
subprocess.run(
[
"gdalwarp",
"-t_srs",
"EPSG:2193",
"-r",
"bilinear",
"-tr",
"10",
"-10",
"-co",
"compress=lzw",
nearblack_output,
reprojected_output,
]
)

# mosaic into a single image
print("Building VRT")
subprocess.run(["gdalbuildvrt", vrt_output, reprojected_output])

print("Tiling to Topo 50")
ogr_info = subprocess.Popen(
["ogrinfo", "-ro", "-al", TILE_INDEX], stdout=subprocess.PIPE
)

# tile into Topo50 sheet extents
ogr_output = ogr_info.communicate()[0].decode("utf-8")
Comment on lines +140 to +145
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like you run this and then just wait for it to finish immediately. In that case I'd recommend ogr_output = subprocess.run([…], capture_output=True, check=True, encoding="utf-8").output.

for line in ogr_output.splitlines():
if "sheet_code" in line:
sheet_code = line[len(line) - 4 : len(line) - 0]
subprocess.run(
[
"gdalwarp",
vrt_output,
f"{tiled_output_dir}/{sheet_code}.tif",
"-cutline",
TILE_INDEX,
"-cwhere",
f"sheet_code = '{sheet_code}'",
"-tr",
"10",
"-10",
"-co",
"compress=lzw",
"-t_srs",
"EPSG:2193",
"-crop_to_cutline",
]
)