Skip to content

Commit

Permalink
Include empty resources in summary stats (#61)
Browse files Browse the repository at this point in the history
* template update

* include empty resource classes in statistics

* remove auto readme because of custom content

* version bump

* adapt template changes

* more template-related updates

* fix

* roll requirements back to main

* fix typo

* adapt Dockerfile
  • Loading branch information
dontseyit authored Jan 23, 2024
1 parent d7ad4b1 commit 4295d0e
Show file tree
Hide file tree
Showing 104 changed files with 783 additions and 364 deletions.
5 changes: 0 additions & 5 deletions .coveragerc

This file was deleted.

2 changes: 0 additions & 2 deletions .deprecated_files_ignore

This file was deleted.

2 changes: 1 addition & 1 deletion .devcontainer/dev_install
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ cd /workspace
python -m pip install --upgrade pip

# install or upgrade dependencies for development and testing
pip install --no-deps -r requirements-dev.txt
pip install --no-deps -r ./lock/requirements-dev.txt

# install the package itself in edit mode:
pip install --no-deps -e .
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/check_config_docs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
- uses: actions/checkout@v3

- id: common
uses: ghga-de/gh-action-common@v3
uses: ghga-de/gh-action-common@v4

- name: Check config docs
run: |
Expand Down
16 changes: 16 additions & 0 deletions .github/workflows/check_pyproject.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: Check if the config schema and the example are up to date.

on: push

jobs:
static-code-analysis:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3

- id: common
uses: ghga-de/gh-action-common@v4

- name: Check pyproject.toml
run: |
./scripts/update_pyproject.py --check
2 changes: 1 addition & 1 deletion .github/workflows/check_readme.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
- uses: actions/checkout@v3

- id: common
uses: ghga-de/gh-action-common@v3
uses: ghga-de/gh-action-common@v4

- name: Check readme
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/static_code_analysis.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
- uses: actions/checkout@v3

- id: common
uses: ghga-de/gh-action-common@v3
uses: ghga-de/gh-action-common@v4

- uses: pre-commit/action@v3.0.0
env:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
- uses: actions/checkout@v3

- id: common
uses: ghga-de/gh-action-common@v3
uses: ghga-de/gh-action-common@v4

- id: pytest
run: |
Expand Down
12 changes: 0 additions & 12 deletions .mypy.ini

This file was deleted.

8 changes: 1 addition & 7 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,18 +47,12 @@ repos:
args: [--fix=lf]
- id: no-commit-to-branch
args: [--branch, dev, --branch, int, --branch, main]
- id: debug-statements
- id: debug-statements
- id: debug-statements
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.1.7
hooks:
- id: ruff
args: [--fix, --exit-non-zero-on-fix]
- repo: https://github.com/psf/black
rev: 23.9.1
hooks:
- id: black
- id: ruff-format
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.7.1
hooks:
Expand Down
29 changes: 29 additions & 0 deletions .pyproject_generation/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
<!--
Copyright 2021 - 2023 Universität Tübingen, DKFZ, EMBL, and Universität zu Köln
for the German Human Genome-Phenome Archive (GHGA)
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

# Generating the pyproject.toml

The pyproject.toml of the service is generated by combining static configuration
captured in [`./pyproject_template.toml`](./pyproject_template.toml) and custom
package metadata specified in [`./pyproject_custom.toml`](./pyproject_custom.toml).

The `./pyproject_template.toml` is managed by the template, please do not edit manually.

You may specify properties in the `./pyproject_custom.toml` which are already specified
in the `./pyproject_template.toml`. In that case, the `./pyproject_custom.toml` takes
priority.
19 changes: 19 additions & 0 deletions .pyproject_generation/pyproject_custom.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
[project]
name = "metldata"
version = "1.2.0"
description = "metldata - A framework for handling metadata based on ETL, CQRS, and event sourcing."
dependencies = [
"hexkit[akafka,mongodb]~=1.0.0",
"ghga-service-commons[api,auth]~=1.1.0",
"ghga-event-schemas~=1.0.0",
"typer~=0.9.0",
"linkml==1.6.1",
"linkml-runtime==1.6.0",
"linkml-validator==0.4.5",
]

[project.urls]
Repository = "https://github.com/ghga-de/metldata"

[project.scripts]
metldata = "metldata.__main__:run"
107 changes: 107 additions & 0 deletions .pyproject_generation/pyproject_template.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
[build-system]
requires = ["setuptools>=67.7.2"]
build-backend = "setuptools.build_meta"

[project]
readme = "README.md"
authors = [
{ name = "German Human Genome Phenome Archive (GHGA)", email = "contact@ghga.de" },
]
requires-python = ">=3.9"
license = { text = "Apache 2.0" }
classifiers = [
"Development Status :: 1 - Planning",
"Operating System :: POSIX :: Linux",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"License :: OSI Approved :: Apache Software License",
"Topic :: Internet :: WWW/HTTP :: HTTP Servers",
"Topic :: Software Development :: Libraries",
"Intended Audience :: Developers",
]

[tool.setuptools.packages.find]
where = ["src"]

[tool.ruff]
exclude = [
".git",
".devcontainer",
"__pycache__",
"build",
"dist",
]
ignore = [
"E", # pycodestyle errors
"W", # pycodestyle warnings - pycodestyle covered by black
"PLW", # pylint warnings
"RUF001", # ambiguous unicode character strings
"RUF010", # explicit conversion to string or repr: !s or !r
"RUF012", # mutable class variables need typing.ClassVar annotation
"N818", # Errors need to have Error suffix
"B008", # function call in arg defaults,
"PLR2004", # magic numbers should be constants
"D205", # blank-line-after-summary
"D400", # first doc line ends in period
"D401", # non-imperative-mood
"D107", # missing docstring in __init__
"D206", # indent-with-spaces (ignored for formatter)
"D300", # triple-single-quotes (ignored for formatter)
]
line-length = 88
select = [
"C90", # McCabe Complexity
"F", # pyflakes codes
"I", # isort
"S", # flake8-bandit
"B", # flake8-bugbear
"N", # pep8-naming
"UP", # pyupgrade
"PL", # pylint
"RUF", # ruff
"SIM", # flake8-simplify
"D", # pydocstyle
]
fixable = [
"UP", # e.g. List -> list
"I", # sort imports
"D", # pydocstyle
]
src = ["src", "tests", "examples", "scripts"]
target-version = "py39"

[tool.ruff.mccabe]
max-complexity = 10

[tool.ruff.per-file-ignores]
"scripts/*" = ["PL", "S", "SIM", "D"]
"tests/*" = ["S", "SIM", "PLR", "B011"]
".devcontainer/*" = ["S", "SIM", "D"]
"examples/*" = ["S", "D"]
"__init__.py" = ["D"]

[tool.ruff.pydocstyle]
convention = "pep257"

[tool.mypy]
disable_error_code = "import"
show_error_codes = true
exclude = [
'build/lib/',
]
warn_redundant_casts = true
warn_unused_ignores = true
check_untyped_defs = true
no_site_packages = false

[tool.pytest.ini_options]
minversion = "7.1"
asyncio_mode = "strict"

[tool.coverage.paths]
source = [
"src",
"/workspace/src",
"**/lib/python*/site-packages",
]
22 changes: 11 additions & 11 deletions readme_generation.md → .readme_generation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,30 +18,30 @@

# Readme Generation

The README file is generated by collecting information from different sources as
The Repository README is generated by collecting information from different sources as
outlined in the following.

- name: The full name of the package is derived from the remote origin Git repository.
- title: A title case representation of the name.
- shortname: An abbreviation of the full name. This is derived from the name mentioned
in the [`./pyproject.toml`](./pyproject.toml).
in the [`../pyproject.toml`](../pyproject.toml).
- summary: A short 1-2 sentence summary derived from the description in the
[`./pyproject.toml`](./pyproject.toml).
[`../pyproject.toml`](../pyproject.toml).
- version: The package version derived from the version specified in the
[`./pyproject.toml`](./pyproject.toml).
[`../pyproject.toml`](../pyproject.toml).
- description: A markdown-formatted description of the features and use cases of this
service or package. Obtained from the [`./.description.md`](./.description.md).
service or package. Obtained from the [`./description.md`](./description.md).
- design_description: A markdown-formatted description of the overall architecture and
design of the package. Obtained from the [`./.design.md`](./.design.md).
design of the package. Obtained from the [`./design.md`](./design.md).
- config_description: A markdown-formatted description of all config parameters.
This is autogenerated from the [`./config_schema.json`](./config_schema.json).
This is autogenerated from the [`../config_schema.json`](../config_schema.json).
- openapi_doc: A markdown-formatted description of the HTTP API. This is autogenerated
and links to the [`./openapi.yaml`](./openapi.yaml). If the openapi.yaml is not
and links to the [`../openapi.yaml`](../openapi.yaml). If the openapi.yaml is not
this documentation is empty.

The [`./.readme_template.md`](./.readme_template.md) serves as a template where the
The [`./readme_template.md`](./readme_template.md) serves as a template where the
above variable can be filled in using Pythons `string.Template` utility from the
standard library.

The [`./scripts/update_readme.py`] script can be used to collect all information and
fill it into the template to generate the README file.
The [`../scripts/update_readme.py`](../scripts/update_readme.py) script can be used to
collect all information and fill it into the template to generate the README file.
16 changes: 6 additions & 10 deletions .description.md → .readme_generation/description.md
Original file line number Diff line number Diff line change
@@ -1,44 +1,40 @@
<!-- Please provide a short overview of the features of this service.-->

### Scope and Features:

Metldata is a framework for handling the entire lifetime of metadata by addressing the
a complex combination of challenges that makes it suitable especially for public
archives for sensitive data:

![Schematic reprensentation of challenges.](./images/challenges.png)
**Figure 1| Overview of the combination of challenges during metadata handling.**

#### Immutability
### Immutability
It is guaranteed that data entries do not change over time making reproducibility
possible without having to rely on local snapshots.

#### Accessibility
### Accessibility
A stable accession is assigned to each resource. Together with the immutability
property, this guarantees that you will always get the same data when querying with the
same accession.

#### Corrections, Improvements, Extensions
### Corrections, Improvements, Extensions

Even though data is stored in an immutable way, the metldata still allows for
corrections, improvements, and extensions of submitted data. This is achieved my not
just storing the current state of a submission but by persisting a version history.
Thereby, modifications are realized by issuing a new version of the submission without
affecting the content of existing versions.

#### Transparency
### Transparency

The version history not only resolved the conflict between immutability and the need to
evolve and adapt data, it also make the changes transparent to user relying on the data.

#### Multiple Representations
### Multiple Representations

Often, the requirements regarding the structure and content of data differs depending
the use case and the audience. Metldata accounts for this by proving a configurable
workflow engine for transforming submitted metadata into multiple representation of that
data.

#### GDPR Compliance
### GDPR Compliance

The GDPR gives data subjects the right to issue a request to delete data. Metldata
complies with this demand. Thereby, only entire versions of a submission can be deleted.
Expand Down
4 changes: 0 additions & 4 deletions .design.md → .readme_generation/design.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,3 @@
<!-- Please provide an overview of the architecture and design of the code base.
Mention anything that deviates from the standard triple hexagonal architecture and
the corresponding structure. -->

The framework uses a combination of ETL, CQRS, and event sourcing. Currently it is
designed to mostly run as a CLI application for managing metadata on the local file
system. However, later, it will be translated into a microservice based-architecture.
Expand Down
5 changes: 2 additions & 3 deletions .readme_template.md → .readme_generation/readme_template.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@

[![tests](https://github.com/ghga-de/$name/actions/workflows/tests.yaml/badge.svg)](https://github.com/ghga-de/$name/actions/workflows/tests.yaml)
[![Coverage Status](https://coveralls.io/repos/github/ghga-de/$name/badge.svg?branch=main)](https://coveralls.io/github/ghga-de/$name?branch=main)
[![tests](https://github.com/ghga-de/$repo_name/actions/workflows/tests.yaml/badge.svg)](https://github.com/ghga-de/$repo_name/actions/workflows/tests.yaml)
[![Coverage Status](https://coveralls.io/repos/github/ghga-de/$repo_name/badge.svg?branch=main)](https://coveralls.io/github/ghga-de/$repo_name?branch=main)

# $title

Expand Down
Loading

0 comments on commit 4295d0e

Please sign in to comment.