Integrate dcs #8
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…V-315) Initialized the microservice by modifying the template Added localstack and psql containers for development Added usable code from sandbox-storage
Added a database DAO interface for handling DRS object entries in the database: - get DRS objects by id - register new DRS objects - unregister existing DRS objects by id A Prostgres-specific implementation of this DAO is provided. The S3-specific object storage was imported from the ghga_service_chassis_lib. Moreover, request/reply models were refactored.
Checks outbox if file exists after download request Checks the outbocks after an API download request, if a file exists Returns a presigned url if it does exist Returns a 202 Statement and publishes a non_staged_file_request topic if not Adds Unit tests for pubsub and core functionality
Subscribes to file_staged_for_download, updates database entry, if ncessary Subscribes to file_staged_for_download Checks, if the file really exists in outbox Updates the file related information in the database, if necessary
…istered Subscribes to file_internally_registered Adds the file to the database Publishes to drs_object_registered
- used messages from package instead of keeping local copies - adapted database to keep additional file attributes - updated functions and tests to new messages - updated licence headers - incorporated changes from teplate repo - in test states, the messages are now create form file_info instead of from scratch
Refactor to use message schemas from ghga-messages package (GDEV-436)
- dev_cd action now runs from main - switch from codecov to coveralls - do not pin ports in devcontainer
prepare to deprecate dev branch (GDEV-792) (#9)
Fixed CI/CD tags
So that checks pass again. Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de>
Renamed to dcs Changed the name of the service & the package Updated Libraries Bumped Version to 0.2.0
to prepare for refactoring, the following pieces were removed/cleaned up: - integration tests (except for core targeting ones) - pub/sub logic based on RabbitMQ - REST API based on pyramid - database migration - not needed software deps - not needed config params The outbound infrastructure dependencies (S3 and postgres) remain unchanged since the current core implementation depends on them. Co-authored-by: Moritz Hahn <75744178+MoritzHahn1337@users.noreply.github.com> Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de>
- restructure core into hexagonal architecture - replaced PostgreSQL-based DAO by MongoDB-based protocol/provider from hexkit - migrated object storage abstraction from the chassis lib to hexkit - switched to async python - included api level conformance tests Co-authored-by: Moritz Hahn <75744178+MoritzHahn1337@users.noreply.github.com> Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de>
Added FastAPI Adapter (GDEV-1317) Used httpyexcept exception handling (GDEV-1318) Custom Title in Swagger UI (GDEV-1444) Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de> Co-authored-by: KerstenBreuer <kersten-breuer@outlook.com>
Added more OpenAPI customization
and update to current event schemas. Bumps version to 0.4.0. Co-authored-by: Moritz Hahn <75744178+MoritzHahn1337@users.noreply.github.com> Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de>
Added container scanning and signing
Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
* Delete test container module * Add endpoints handler * Remove MockAPI from JointFixture Change EKSS base url * Convert mock app to use EndpointHandler * Update test_typical_journey to use httpx_mock * Update template files --------- Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
* Update dependencies * Replace EndpointsHandler with MockRouter * Use exception base from httpyexpect in commons * Remove commented-out httpx_mock.add_response() * Fix imports for hexkit fixtures * Use httpy_exception_handler Remove vestigial HttpException class * Bump version from 0.6.4 -> 0.6.5 * Update README and openapi spec * Remove outdated httpyexpect reference * Import fixture from mock_router assert_all_responses_were_requested fixture is available from there now --------- Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
Bumps version to 0.6.6.
Allow testcontainers to be run on the docker host deamon. This is just a prototype and should be implemented in the template repository.
A first draft of how DI could work without a framework, In brief: - one async context manager constructs the core and resolves all its dependencies - dedicated async context managers are provided to construct inbound adapters using the core dependencies Also removes the requirement to override dependencies in the initialized container. Some utils collected in `dcs.utils` might be placed in the ghga-service-commons as they are not specific to the service. Upgrades to pydantic v2. This PR had to upgrade to the latest ghga-event-schemas which also introduce the s3_endpoint_alias fields. Since this service is not yet ready to handle them, I just hard-coded the field where required. Has to be cleaned up in a dedicated PR. Bumps version to 1.0.0.
* Update template files * Fix description.md * Add new line --------- Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
* Use configured logging and update docs * Bump version from 1.1.0 -> 1.2.0 * Remove unused utils file * Fix some typos * Update openapi spec --------- Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
Added file_id db check to registration Added logging to data_repository implementation
Co-authored-by: Byron Himes <byro93@live.com>
TheByronHimes
added a commit
that referenced
this pull request
Jun 6, 2024
* Update template files Incorporate changes suggested by ruff/pre-commit * Update README and openapi doc * Silence mypy errors Fix some pyupgrade stuff Add a docstring to joint.py --------- Co-authored-by: TheByronHimes <TheByronHimes@gmail.com>
TheByronHimes
added a commit
that referenced
this pull request
Jun 10, 2024
* Initial commit * Feature/init repo gdev 1274 (#2) * Initial config changes, replacing repo name, removing unneeded content Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de> * Feature/upload functionality gdev 1275 (#3) * Implemented FastAPI Endpoint for extracttion of file secrets and content offsets * Currently store file secrets internally in MongoDB * Inject GHGA keypair from environment Co-authored-by: KerstenBreuer <kersten-breuer@outlook.com> Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de> * SecretStr adjustments (#5) Config now uses appropriate fields (like SecretStr). Simplified test code Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de> * Feature/eks download functionality gdev 1276 (#6) Added Download functionality (GDEV-1276) Wrote Integration tests (GDEV-1279) Bumped chassis lib, hexkit version and updated to new template version Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de> Co-authored-by: Kersten Breuer <kersten-breuer@outlook.com> * Feature/simplify encoding gdev 1384 (#7) Changed encoding of base64 bytes objects to use utf-8 directly instead of hex representation for conversion to string Co-authored-by: KerstenBreuer <kersten-breuer@outlook.com * Feature/vault testcontainer gdev 1439 (#8) Added testcontainer setup for HashiCorp vault Added vault client wrapper Co-authored-by: KerstenBreuer <kersten-breuer@outlook.com> Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de> * Feature/replace mongodb gdev 1416 (#9) Replaced MongoDB with HashiCorp Vault using hvac Co-authored-by: KerstenBreuer <kersten-breuer@outlook.com> Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de> * Prettier swagger gdev 1445 (#11) Added Customization for OpenAPI/Swagger * Fixed oversights, updated dependencies (#12) Co-authored-by: KerstenBreuer <kersten-breuer@outlook.com> * Updated chassis-lib and openapi.yaml (#13) * Fixed missing async context for run_server (#14) * Update Dockerfile * Added container scanning and signing (#15) * Update dev_cd.yaml (#16) Co-authored-by: Thomas J. Zajac <thomas-jakob.zajac@uni-tuebingen.de> * Added new secret (#17) Added creation of a new secret in preparation of re-encryption * Fix/current dependencies (#18) * Updates from template * Updated license headers * Updated hexkit and chassis-lib * httpx for fastpi issues * Fix/enhanced error handling gdev 1590 (#19) Added errors for vault-specific issues in upload path with proposed error codes Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de> * Update Dockerfile (#20) * Fix/change credential type gdev 1613 (#21) Changed vault authentication engine to approle Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de> * Added global config for vault over http (#22) * Updated mypy to 1.0.0 (#23) Updated mypy to 1.0.0 * Update Template (#24) Updated Template Files * Added manual run of dev_cd (#25) * Added manual run of dev_cd Co-authored-by: Moritz Hahn <moritz.hahn@uni-tuebingen.de> * Added Handover Documents (#26) Switched to new Readme - Generation Wrote Service description Co-authored-by: Kersten Breuer <kersten-breuer@outlook.com> Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de> * Add file secret deletion handling gsi 132 (#27) Added deletion Endpoint Removed High-level business logic from fastapi adapter * Add deletion description (#28) Document the previous PR * Fixed vault.delete_secret (#29) Made v2 explicit instead of setting it in constructor Replaced v1 delete method with v2 one nuking all key info for path Added test for correct deletion * update dependencies (GSI-181) (#30) update hexkit * Apply template changes (GSI 397) (#31) * Update template files * Deal with mypy * Update openapi docs/readme * Resolve deprecation warning for hashicorp --------- Co-authored-by: TheByronHimes <TheByronHimes@gmail.com> * add health endpoint to service api (#32) * add health endpoint * version bump * update openapi yaml * Added ca_bundle_location config, simplified vault URL (#33) Co-authored-by: Christoph Zwerschke <cito@online.de> * Fix dependencies (#34) * Expose vault path as option (#35) * Upgrade pydantic and devcontainer (GSI-438) Upgrades to pydantic v2. Switches to docker-outside-of-docker for the devcontainer environment. Bumps version to 1.0.0. * Validate setting for vault CA bundle (GSI-462) (#37) * Add Kubernetes authentication (#38) * Remove black and update ruff * Support Kubernetes auth * Template updates * Ignore type error * Move `if not all` to `else` block * Fix mypy ignores default values * Use configured logging (GSI-567) (#39) * Update template files * Use configured logging * Update config, readme, openapi spec --------- Co-authored-by: TheByronHimes <TheByronHimes@gmail.com> * Add 'lock' to reqs file path (#40) Co-authored-by: TheByronHimes <TheByronHimes@gmail.com> * Add configurable mount point for secret engines (#41) * Add configurable mount point for secret engines * Consolidate service files into subdir for ekss * Integrate the EKSS * Update the util script for openapi in ekss * Fix some readme and docs issues * Prevent double dashes in README --------- Co-authored-by: Thomas Zajac <thomas-jakob.zajac@uni-tuebingen.de> Co-authored-by: Moritz Hahn <Moritz.Hahn@uni-tuebingen.de> Co-authored-by: KerstenBreuer <kersten-breuer@outlook.com> Co-authored-by: Moritz Hahn <75744178+MoritzHahn1337@users.noreply.github.com> Co-authored-by: Alexander Kyumurkov <akyumurkov@tutanota.com> Co-authored-by: Alexander Kyumurkov <kyumurkov@uni-koeln.de> Co-authored-by: TheByronHimes <TheByronHimes@gmail.com> Co-authored-by: Seyit Zor <seyitzor@gmail.com> Co-authored-by: Christoph Zwerschke <cito@online.de> Co-authored-by: Christoph Zwerschke <c.zwerschke@dkfz-heidelberg.de> Co-authored-by: Camill Kaipf <87362681+ckaipf@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No functional changes made to DCS tests, just reorganization of fixtures into a
conftest.pyfile.The license and license header files were out of date, so I copied them over from the auth-service and made the updates to all the files. (a script to update license headers might be a nice-to-have, by the way).
There was another bug in the update_readme_services script that was unable to handle pyproject description fields with more than the initial hyphen.
There was also a typo in the EKSS pyproject where 'service' was spelled 'sevice'