Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate dcs #8

Closed
wants to merge 76 commits into from
Closed

Integrate dcs #8

wants to merge 76 commits into from

Conversation

TheByronHimes
Copy link
Member

No functional changes made to DCS tests, just reorganization of fixtures into a conftest.py file.
The license and license header files were out of date, so I copied them over from the auth-service and made the updates to all the files. (a script to update license headers might be a nice-to-have, by the way).
There was another bug in the update_readme_services script that was unable to handle pyproject description fields with more than the initial hyphen.
There was also a typo in the EKSS pyproject where 'service' was spelled 'sevice'

KerstenBreuer and others added 30 commits November 4, 2021 10:20
…V-315)

Initialized the microservice by modifying the template
Added localstack and psql containers for development
Added usable code from sandbox-storage
Added a database DAO interface for handling DRS object
entries in the database:
  - get DRS objects by id
  - register new DRS objects
  - unregister existing DRS objects by id
 A Prostgres-specific implementation of this DAO is provided.
The S3-specific object storage was imported from the
ghga_service_chassis_lib.
Moreover, request/reply models were refactored.
Checks outbox if file exists after download request

Checks the outbocks after an API download request, if a file exists
Returns a presigned url if it does exist
Returns a 202 Statement and publishes a non_staged_file_request topic if not
Adds Unit tests for pubsub and core functionality
Subscribes to file_staged_for_download, updates database entry, if ncessary

Subscribes to file_staged_for_download
Checks, if the file really exists in outbox
Updates the file related information in the database, if necessary
…istered

Subscribes to file_internally_registered
Adds the file to the database
Publishes to drs_object_registered
- used messages from package instead of keeping local copies
- adapted database to keep additional file attributes
- updated functions and tests to new messages
- updated licence headers
- incorporated changes from teplate repo
- in test states, the messages are now create form file_info instead of from scratch
Refactor to use message schemas from ghga-messages package (GDEV-436)
- dev_cd action now runs from main
- switch from codecov to coveralls
- do not pin ports in devcontainer
prepare to deprecate dev branch (GDEV-792) (#9)
Fixed CI/CD tags
So that checks pass again.

Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de>
Renamed to dcs

Changed the name of the service & the package
Updated Libraries
Bumped Version to 0.2.0
to prepare for refactoring, the following pieces were removed/cleaned up:
- integration tests (except for core targeting ones)
- pub/sub logic based on RabbitMQ
- REST API based on pyramid
- database migration
- not needed software deps
- not needed config params

The outbound infrastructure dependencies (S3 and postgres) remain
unchanged since the current core implementation depends on them.

Co-authored-by: Moritz Hahn <75744178+MoritzHahn1337@users.noreply.github.com>
Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de>
- restructure core into hexagonal architecture
- replaced PostgreSQL-based DAO by MongoDB-based
   protocol/provider from hexkit
- migrated object storage abstraction from the chassis lib
   to hexkit
- switched to async python
- included api level conformance tests


Co-authored-by: Moritz Hahn <75744178+MoritzHahn1337@users.noreply.github.com>
Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de>
Added FastAPI Adapter (GDEV-1317)
Used httpyexcept exception handling (GDEV-1318)
Custom Title in Swagger UI (GDEV-1444)

Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de>
Co-authored-by: KerstenBreuer <kersten-breuer@outlook.com>
Added more OpenAPI customization
and update to current event schemas.

Bumps version to 0.4.0.

Co-authored-by: Moritz Hahn <75744178+MoritzHahn1337@users.noreply.github.com>
Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de>
Added container scanning and signing
TheByronHimes and others added 25 commits June 13, 2023 17:15
Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
* Delete test container module

* Add endpoints handler

* Remove MockAPI from JointFixture

Change EKSS base url

* Convert mock app to use EndpointHandler

* Update test_typical_journey to use httpx_mock

* Update template files

---------

Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
* Update dependencies

* Replace EndpointsHandler with MockRouter

* Use exception base from httpyexpect in commons

* Remove commented-out httpx_mock.add_response()

* Fix imports for hexkit fixtures

* Use httpy_exception_handler

Remove vestigial HttpException class

* Bump version from 0.6.4 -> 0.6.5

* Update README and openapi spec

* Remove outdated httpyexpect reference

* Import fixture from mock_router

assert_all_responses_were_requested fixture is available from there now

---------

Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
Bumps version to 0.6.6.
Allow testcontainers to be run on the docker host deamon.

This is just a prototype and should be implemented in the template repository.
A first draft of how DI could work without a framework,

In brief:
 - one async context manager constructs the core and resolves all its dependencies
 - dedicated async context managers are provided to construct inbound adapters using the core dependencies

Also removes the requirement to override dependencies in the initialized container.

Some utils collected in `dcs.utils` might be placed in the ghga-service-commons as they are not specific to the service.

Upgrades to pydantic v2.

This PR had to upgrade to the latest ghga-event-schemas which also introduce the s3_endpoint_alias fields.
Since this service is not yet ready to handle them, I just hard-coded the field where required.
Has to be cleaned up in a dedicated PR.

Bumps version to 1.0.0.
* Update template files

* Fix description.md

* Add new line

---------

Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
* Use configured logging and update docs

* Bump version from 1.1.0 -> 1.2.0

* Remove unused utils file

* Fix some typos

* Update openapi spec

---------

Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
Added file_id db check to registration
Added logging to data_repository implementation
Co-authored-by: Byron Himes <byro93@live.com>
TheByronHimes added a commit that referenced this pull request Jun 6, 2024
* Update template files

Incorporate changes suggested by ruff/pre-commit

* Update README and openapi doc

* Silence mypy errors

Fix some pyupgrade stuff

Add a docstring to joint.py

---------

Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
@TheByronHimes TheByronHimes deleted the integrate_dcs branch June 6, 2024 07:34
TheByronHimes added a commit that referenced this pull request Jun 10, 2024
* Initial commit

* Feature/init repo gdev 1274 (#2)

* Initial config changes, replacing repo name, removing unneeded content


Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de>

* Feature/upload functionality gdev 1275 (#3)

* Implemented FastAPI Endpoint for extracttion of file secrets and content offsets
* Currently store file secrets internally in MongoDB
* Inject GHGA keypair from environment

Co-authored-by: KerstenBreuer <kersten-breuer@outlook.com>
Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de>

* SecretStr adjustments (#5)

Config now uses appropriate fields (like SecretStr).
Simplified test code

Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de>

* Feature/eks download functionality gdev 1276 (#6)

Added Download functionality (GDEV-1276)
Wrote Integration tests (GDEV-1279)
Bumped chassis lib, hexkit version and updated to new template version

Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de>
Co-authored-by: Kersten Breuer <kersten-breuer@outlook.com>

* Feature/simplify encoding gdev 1384 (#7)

Changed encoding of base64 bytes objects to use utf-8 directly instead of hex representation for conversion to string

Co-authored-by: KerstenBreuer <kersten-breuer@outlook.com

* Feature/vault testcontainer gdev 1439 (#8)

Added testcontainer setup for HashiCorp vault
Added vault client wrapper

Co-authored-by: KerstenBreuer <kersten-breuer@outlook.com>
Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de>

* Feature/replace mongodb gdev 1416 (#9)

Replaced MongoDB with HashiCorp Vault using hvac

Co-authored-by: KerstenBreuer <kersten-breuer@outlook.com>
Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de>

* Prettier swagger gdev 1445 (#11)

Added Customization for OpenAPI/Swagger

* Fixed oversights, updated dependencies (#12)

Co-authored-by: KerstenBreuer <kersten-breuer@outlook.com>

* Updated chassis-lib and openapi.yaml (#13)

* Fixed missing async context for run_server (#14)

* Update Dockerfile

* Added container scanning and signing (#15)

* Update dev_cd.yaml (#16)

Co-authored-by: Thomas J. Zajac <thomas-jakob.zajac@uni-tuebingen.de>

* Added new secret (#17)

Added creation of a new secret in preparation of re-encryption

* Fix/current dependencies (#18)

* Updates from template
* Updated license headers
* Updated hexkit and chassis-lib
* httpx for fastpi issues

* Fix/enhanced error handling gdev 1590 (#19)

Added errors for vault-specific issues in upload path with proposed error codes

Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de>

* Update Dockerfile (#20)

* Fix/change credential type gdev 1613 (#21)

Changed vault authentication engine to approle

Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de>

* Added global config for vault over http (#22)

* Updated mypy to 1.0.0 (#23)

Updated mypy to 1.0.0

* Update Template (#24)

Updated Template Files

* Added manual run of dev_cd (#25)

* Added manual run of dev_cd

Co-authored-by: Moritz Hahn <moritz.hahn@uni-tuebingen.de>

* Added Handover Documents (#26)

Switched to new Readme - Generation
Wrote Service description

Co-authored-by: Kersten Breuer <kersten-breuer@outlook.com>
Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de>

* Add file secret deletion handling gsi 132 (#27)

Added deletion Endpoint
Removed High-level business logic from fastapi adapter

* Add deletion description (#28)

Document the previous PR

* Fixed vault.delete_secret (#29)

Made v2 explicit instead of setting it in constructor
Replaced v1 delete method with v2 one nuking all key info for path
Added test for correct deletion

* update dependencies (GSI-181) (#30)

update hexkit

* Apply template changes (GSI 397) (#31)

* Update template files

* Deal with mypy

* Update openapi docs/readme

* Resolve deprecation warning for hashicorp

---------

Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>

* add health endpoint to service api (#32)

* add health endpoint

* version bump

* update openapi yaml

* Added ca_bundle_location config, simplified vault URL (#33)

Co-authored-by: Christoph Zwerschke <cito@online.de>

* Fix dependencies (#34)

* Expose vault path as option (#35)

* Upgrade pydantic and devcontainer (GSI-438)

Upgrades to pydantic v2.

Switches to docker-outside-of-docker for the devcontainer environment.

Bumps version to 1.0.0.

* Validate setting for vault CA bundle (GSI-462) (#37)

* Add Kubernetes authentication (#38)

* Remove black and update ruff

* Support Kubernetes auth

* Template updates

* Ignore type error

* Move `if not all` to `else` block

* Fix mypy ignores default values

* Use configured logging (GSI-567) (#39)

* Update template files

* Use configured logging

* Update config, readme, openapi spec

---------

Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>

* Add 'lock' to reqs file path (#40)

Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>

* Add configurable mount point for secret engines (#41)

* Add configurable mount point for secret engines

* Consolidate service files into subdir for ekss

* Integrate the EKSS

* Update the util script for openapi in ekss

* Fix some readme and docs issues

* Prevent double dashes in README

---------

Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de>
Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de>
Co-authored-by: KerstenBreuer <kersten-breuer@outlook.com>
Co-authored-by: Moritz Hahn <75744178+MoritzHahn1337@users.noreply.github.com>
Co-authored-by: Alexander Kyumurkov <akyumurkov@tutanota.com>
Co-authored-by: Alexander Kyumurkov <kyumurkov@uni-koeln.de>
Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
Co-authored-by: Seyit Zor <seyitzor@gmail.com>
Co-authored-by: Christoph Zwerschke <cito@online.de>
Co-authored-by: Christoph Zwerschke <c.zwerschke@dkfz-heidelberg.de>
Co-authored-by: Camill Kaipf <87362681+ckaipf@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants