From f19eaa1d1948e407f9b82fe4dbbc66537e491f4b Mon Sep 17 00:00:00 2001 From: Bogdan Marc Date: Fri, 6 Sep 2024 12:56:48 +0100 Subject: [PATCH 1/7] Moved all README information into the wiki --- README.md | 250 +----------------------------------------------------- 1 file changed, 1 insertion(+), 249 deletions(-) diff --git a/README.md b/README.md index c057e42..08d086e 100644 --- a/README.md +++ b/README.md @@ -14,252 +14,4 @@ Please see the other repositories in the [HM Land Registry Open Data](https://github.com/epimorphics/hmlr-linked-data/) project for more details. -## Running the Standard Reports Manager locally - -The application connects to the Standard Reports Manager service to submit -requests. - -The easiest way to do this is as a local docker container. The image can be -built from [standard reports -manager](https://github.com/epimorphics/standard-reports-manager/) repository. -or pulled from the Amazon Elastic Container Registry -[ECR](https://eu-west-1.console.aws.amazon.com/ecr/repositories/private/018852084843/epimorphics/standard-reports-manager/dev?region=eu-west-1). - -### Building and running from [standard reports manager](https://github.com/epimorphics/standard-reports-manager/) repository - -To build and a run a new docker image check out the [standard reports -manager](https://github.com/epimorphics/standard-reports-manager/) repository -and run - -```sh -make image run -``` - -### Running an existing [ECR](https://eu-west-1.console.aws.amazon.com/ecr/repositories/private/018852084843/epimorphics/standard-reports-manager/dev?region=eu-west-1) image - -Obtaining an ECR image requires: - -- AWS IAM credentials to connect to the HMLR AWS account -- the ECR credentials helper installed locally (see - [here](https://github.com/awslabs/amazon-ecr-credential-helper)) -- Set the contents of your `~/.docker/config.json` file to be: - -```sh -{ - "credsStore": "ecr-login" -} -``` - -This configures the Docker daemon to use the credential helper for all Amazon -ECR registries. - -To use a credential helper for a specific ECR registry[^1], create a -`credHelpers` section with the URI of your ECR registry: - -```sh -{ - [...] - "credHelpers": { - "public.ecr.aws": "ecr-login", - "018852084843.dkr.ecr.eu-west-1.amazonaws.com": "ecr-login" - } -} -``` - -Once you have a local copy of you required image, it is advisable to run a local -docker bridge network to mirror production and development environments. - -Running a client application as a docker image from their respective `Makefile`s -will set this up automatically, but to confirm run - -```sh -docker network inspect dnet -``` - -To create the docker network run - -```sh -docker network create dnet -``` - -### Running as a docker container - -Take a copy of the latest Manager development configuation file - -```sh -wget -O test/fixtures/conf/app.conf https://raw.githubusercontent.com/epimorphics/standard-reports-manager/master/dev/app.conf -``` - -```sh -docker run --network dnet -p 8081:8080 --rm --name standard-reports-manager \ - -v $(pwd)/test/fixtures/conf/app.conf:/etc/standard-reports/app.conf \ - 018852084843.dkr.ecr.eu-west-1.amazonaws.com/epimorphics/standard-reports-manager/dev:0.1.1_5ebbea4_00000030 -``` - -the latest image can be found here -[dev](https://github.com/epimorphics/hmlr-ansible-deployment/blob/master/ansible/group_vars/dev/tags.yml) -and -[production](https://github.com/epimorphics/hmlr-ansible-deployment/blob/master/ansible/group_vars/prod/tags.yml). - -The full list of versions can be found at [AWS -ECR](https://eu-west-1.console.aws.amazon.com/ecr/repositories/private/018852084843/epimorphics/standard-reports-manager/dev?region=eu-west-1) - -Note: port 8080 should be avoided to allow for a reverse proxy to run on this -port. - -With this set up, the api service is available on `http://localhost:8081` from -the host or `http://standard-reports-manager:8080` from inside other docker containers. - -## Running the app - -This application can be run stand-alone as a rails server in `development` mode. -However, when deployed, applications will run behind a reverse proxy. - -This enables request to be routed to the appropriate application base on the -request path. In order to simplifiy the proxy configuration we retain the -original path where possible. - -For information on how to running a proxy to mimic production and run multple -services together read through the information in our -[simple-web-proxy](https://github.com/epimorphics/simple-web-proxy/edit/main/README.md) -repository. - -If running more than one application locally ensure that each is listerning on a -separate port and separate path. In the case of running local docker images, the -required configuration is captured in the `Makefile`. - -### Locally - -For developing rails applications you can start the server locally using the -following command: - -```sh -rails server -``` - -and visit in your browser. - -To change to using `production` mode use the `-e` option; or to change to a -different port use the `-p` option. - -Note: In `production` mode, `SECRET_KEY_BASE` is also required. It is -insufficient to just set this as the value must be exported. e.g. - -```sh -export SECRET_KEY_BASE=$(./bin/rails secret) -``` - -#### Running Rails as a server with a sub-directory via Makefile - -```sh -API_SERVICE_URL= RAILS_ENV= RAILS_RELATIVE_URL_ROOT=/ make server -``` - -The default for `RAILS_ENV` here is `development`. - -### As a docker container - -It can be useful to run the compiled Docker image, that will mirror the -production installation, locally yourself. Assuming you have the [Standard -Reports Manager running](#running-the-standard-reports-manager-locally), then -you can run the Docker image for the app itself as follows: - -```sh -make image run -``` - -or, if the image is already built, simply - -```sh -make run -``` - -Docker images run in `production` mode. - -To test the running application visit `localhost:/`. - -### Development and Production mode - -Applications running in `development` mode default to *not* use a sub-directory -to aid stand-alone development. - -Applications running in `production` mode *do* use a sub-directory i.e. -`/app/standard-reports`. - -In each case the is achieved by setting `config.relative_url_root` property to -this sub-directory within the file -`config/environments/(development|production).rb`. - -If need be, `config.relative_url_root` may by overridden by means of the -`RAILS_RELATIVE_URL_ROOT` environment variable, althought this could also -require rebuilding the assets or docker image. - -## Additional Information - -### Coding standards - -Rubocop should complete with no warnings. - -### Tests - -Simply: - -```sh -API_SERVICE_URL=http://localhost:8081 bundle exec rails test -``` - -Passing in the `API_SERVICE_URL` is required to ensure the tests run against the -`standard-reports-manager` service running locally. - -### Issues - -Please add issues to the [shared issues -list](https://github.com/epimorphics/hmlr-linked-data/issues) - -### Runtime Configuration environment variables - -We use a number of environment variables to determine the runtime behaviour of -the application: - -| name | description | default value | -| -------------------------- | ----------------------------------------------------------------------- | -------------------------- | -| `API_SERVICE_URL` | The base URL from which data is accessed, including the HTTP scheme eg. | None | -| | if running a `standard-reports-manager service` locally | | -| | if running a `standard-reports-manager docker` image locally | | -| `SECRET_KEY_BASE` | See [description](https://api.rubyonrails.org/classes/Rails/Application.html#method-i-secret_key_base). | | -| | For `development` mode a acceptable value is already configured, in production mode this should be set to the output of `rails secret`. | | -| | This is handled automatically when starting a docker container, or the `server` `make` target | | -| `SENTRY_API_KEY` | The DSN for sending reports to the PPD Sentry account | None | - -### Deployment - -The detailed deployment mapping is described in `deployment.yml`. At the time of -writing, using the new infrastructure, the deployment process is as follows: - -- commits to the `dev-infrastructure` branch will deploy the dev server -- commits to the `preprod` branch will deploy the pre-production server -- any commit on the `prod` branch will deploy the production server as a new - release - -If the commit is a "new" release, the deployment should be tagged with the same -semantic version number matching the `BREAKING.FEATURE.PATCH` format, e.g. -`v1.2.3`, the same as should be set in the `/app/lib/version.rb`; also, a short -annotation summarising the updates should be included in the tag as well. - -Once the production deployment has been completed and verified, please create a -release on the repository using the same semantic version number. Utilise the -`Generate release notes from commit log` option to create specific notes on the -contained changes as well as the ability to diff agains the previous version. - -#### `entrypoint.sh` features - -- Workaround to removing the PID lock of the Rails process in the event of the - application crashing and not releasing the process. -- Guards to ensure the required environment variables are set accordingly and - trigger the build to fail noisily and log to the system. -- Rails secret creation for `SECRET_KEY_BASE` assignment; see [Runtime - Configuration environment - variables](#runtime-configuration-environment-variables). - -[^1]: With Docker 1.13.0 or greater, you can configure Docker to use different -credential helpers for different registries. +For more information about this project visit [the wiki](https://github.com/epimorphics/standard-reports-ui/wiki). From 87581951b841d70573043fc9097239b134b91f98 Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Tue, 17 Sep 2024 16:48:42 +0100 Subject: [PATCH 2/7] refactor: Updates `config/puma.rb` to include metrics plugin and port info Implements port number as environment variable, with default, to enable running multiple sibling HMLR apps locally if needed --- config/puma.rb | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/config/puma.rb b/config/puma.rb index 7bdabbf..378775d 100644 --- a/config/puma.rb +++ b/config/puma.rb @@ -10,9 +10,13 @@ min_threads_count = ENV.fetch('RAILS_MIN_THREADS', max_threads_count) threads min_threads_count, max_threads_count -# Specifies the `port` that Puma will listen on to receive requests; default is 3000. -# -port ENV.fetch('PORT', 3000) +# Specifies the `port` that Puma will listen on to receive requests; +# default is 3000. +port ENV.fetch('PORT', 3000) + +# Specifies the `metrics_port` that Puma will listen on to export metrics; +# default is 9393. +metrics_port = ENV.fetch('METRICS_PORT', 9393) # Specifies the `environment` that Puma will run in. # @@ -36,9 +40,13 @@ # # preload_app! +# Enable the metrics plugin to export Puma's internal statistics as Prometheus metrics +plugin :metrics +# Bind the metric server to "url". "tcp://" is the only accepted protocol. +metrics_url "tcp://0.0.0.0:#{metrics_port}" if Rails.env.development? + # Allow puma to be restarted by `rails restart` command. plugin :tmp_restart -plugin :metrics # Use a custom log formatter to emit Puma log messages in a JSON format log_formatter do |str| From a4dcf8915b8de4c8228fec66206e005690cd091a Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Tue, 17 Sep 2024 16:49:23 +0100 Subject: [PATCH 3/7] refactor: Updates `config/initializers/prometheus.rb` Adds `Middleware instrumentation` fix for 0 memory bug by notifying Action Dispatch subscribers on Prometheus initialise --- config/initializers/prometheus.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/config/initializers/prometheus.rb b/config/initializers/prometheus.rb index 5691075..648ed3b 100644 --- a/config/initializers/prometheus.rb +++ b/config/initializers/prometheus.rb @@ -48,3 +48,7 @@ docstring: 'Histogram of response times for API requests', buckets: Prometheus::Client::Histogram.exponential_buckets(start: 0.005, count: 16) ) + +# Middleware instrumentation + # This fixes the 0 memory bug by notifying Action Dispatch subscribers on Prometheus initialise + ActiveSupport::Notifications.instrument('process_middleware.action_dispatch') From 84e1c44615362b652227c84208ed10fe7e8f8e52 Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Tue, 17 Sep 2024 16:49:52 +0100 Subject: [PATCH 4/7] refactor: Updates the application exceptions controller Instruments `ActiveSupport::Notifications` for internal errors --- app/controllers/application_controller.rb | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/app/controllers/application_controller.rb b/app/controllers/application_controller.rb index 36f271f..31e9aca 100644 --- a/app/controllers/application_controller.rb +++ b/app/controllers/application_controller.rb @@ -27,6 +27,9 @@ def log_request_result # or attempt to render a generic error page if no specific error page exists unless Rails.application.config.consider_all_requests_local rescue_from StandardError do |e| + # Instrument ActiveSupport::Notifications for internal errors: + ActiveSupport::Notifications.instrument('internal_error.application', exception: e) + # Trigger the appropriate error handling method based on the exception case e.class when ActionController::RoutingError, ActionView::MissingTemplate :render404 @@ -41,10 +44,7 @@ def log_request_result end def handle_internal_error(exception) - # Notify subscribers of the internal error event and render the appropriate error page - # or attempt to render a generic error page if no specific error page exists - # unless the exception is a 404, in which case do nothing - instrument_internal_error(exception) unless exception.status == 404 + # Render the appropriate error page based on the exception if exception.instance_of? ArgumentError render_error(400) else @@ -118,13 +118,4 @@ def detailed_request_log(duration) end end # rubocop:enable Metrics/AbcSize, Metrics/MethodLength - - # Notify subscriber(s) of an internal error event with the payload of the - # exception once done - # @param [Exception] exp the exception that caused the error - # @return [ActiveSupport::Notifications::Event] provides an object-oriented - # interface to the event - def instrument_internal_error(exception) - ActiveSupport::Notifications.instrument('internal_error.application', exception: exception) - end end From a1ef4d32d14607595ea9fa8e912d267a3063cb1f Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Tue, 17 Sep 2024 16:59:36 +0100 Subject: [PATCH 5/7] docs: Updated CHANGELOG --- CHANGELOG.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 112763f..d27778a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,19 @@ # Standard Reports UI: change log -## 1.5.4 - 2024-09 - +## Unreleased + +- (Jon) Updated the application exceptions controller to instrument the + `ActiveSupport::Notifications` for internal errors + [GH-139](https://github.com/epimorphics/standard-reports-ui/issues/139) +- (Jon) Updated `config/initializers/prometheus.rb` to include the `Middleware + instrumentation` fix for the 0 memory bug by notifying Action Dispatch + subscribers on Prometheus initialise + [GH-139](https://github.com/epimorphics/standard-reports-ui/issues/139) +- (Jon) Updated `config/puma.rb` to include metrics plugin and port information + for the metrics endpoint as environment variable, with default, to enable + running multiple sibling HMLR apps locally if needed without port conflicts + [GH-139](https://github.com/epimorphics/standard-reports-ui/issues/139) +- (Jon) Updated the `lr_common_styles` gem to the latest 1.9.9 patch release. - (Jon) Moved all mirrored configuration settings from individual environments into the application configuration to reduce the need to manage multiple sources of truth From 4e415b88196dc511894aaad59b0f152385dc7703 Mon Sep 17 00:00:00 2001 From: "Jon R. Humphrey" Date: Tue, 17 Sep 2024 17:05:59 +0100 Subject: [PATCH 6/7] build: Updated to LR_Common_Styles gem v1.9.9 --- Gemfile.lock | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Gemfile.lock b/Gemfile.lock index 70dde98..f412b4d 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -367,7 +367,7 @@ GEM json lograge railties - lr_common_styles (1.9.8) + lr_common_styles (1.9.9) bootstrap-sass (~> 3.4.0) font-awesome-rails (~> 4.7.0.1) govuk_elements_rails (~> 2.0.0) From 9867fcc68754a7ea97332f168a5745c9c79e8431 Mon Sep 17 00:00:00 2001 From: Bogdan Marc Date: Tue, 24 Sep 2024 14:51:27 +0100 Subject: [PATCH 7/7] Added version to unreleased changes in changelog --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d27778a..1f65c9e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # Standard Reports UI: change log -## Unreleased +## 1.5.4 - 2024-09 - (Jon) Updated the application exceptions controller to instrument the `ActiveSupport::Notifications` for internal errors