* Remove UUID from attrs sent_to metrics. (#21630)

* Remove UUID from attrs sent_to metrics.

* pushing with some linting issues for visibility.

* Change config implementation.

* Remove UUID from spec

* remove uuid from update_appellant job

* diable rubocop for open_telemetry init

* OTEL fixes

* rescue, errors

---------

Co-authored-by: mikefinneran <110622959+mikefinneran@users.noreply.github.com>

* add otel reqs

* Remove throw

* comment out otel file loop

* try vendor suggested config

* update otel config to match vendor

* update pg gem

* otel include dt_host file

* otel require bundler and rubygems

* otel subset of instrumentation

* Revert "update pg gem"

This reverts commit fc1a45d.

* update redis and minimize otel instrumentation to just rails, rack, and activerecord

* otel add use_all except pg and redis

* otel require instruments

* otel fix typo

* otel fix typo

* otel comment out net_http

* otel silence aws sdk internals

* otel silence aws sdk internals

* otel add net http

* Individually use OTEL instruments (#22082)

* Individually use OTEL instruments
with options.

* disable AwsSdk only. Checking Rack options.

* re-require awssdk even while disabled.

* disable awssdk

* disable datadog for testing

* change sequence factory to properly seed

* updated opentelemetry and datadog configs

* rack context getter initalizer

* use one at a time

* add curly braces

* Revert change

* revert change

* Revert change

* Revert change

* Remove gemfile grouping

* Remove datadog.

* ActionPack and Actionview changes

* APPEALS-44287: Excluding disposition held and select that appeal for distribution  (#22277)

* first run at an SQL query removing duplicate appeals from distribution

* code refactor and excluding disposition held and select that appeal for distribution

* automated test for the duplicate hearing bug

* fix rubocop offense SpaceInsideBlockBraces

---------

Co-authored-by: Sean Parker <Sean.Parker3@va.gov>
Co-authored-by: samasudhirreddy <sudhirreddy.sama@va.gov>

---------

Co-authored-by: Noelle Adkin <98478937+NoelleAd@users.noreply.github.com>
Co-authored-by: Raymond Hughes <131811099+raymond-hughes@users.noreply.github.com>
Co-authored-by: Dani <danielle.coleman1@va.gov>
Co-authored-by: raymond-hughes <raymond.hughes2@va.gov>
Co-authored-by: Craig Reese <109101548+craigrva@users.noreply.github.com>
Co-authored-by: Sean Parker <Sean.Parker3@va.gov>
Co-authored-by: samasudhirreddy <sudhirreddy.sama@va.gov>

… errors

…PEALS-50882

Alex/APPEALS-52242

…w/APPEALS-49677

APPEALS-49677: Implement dynatrace custom metric dimension bug fix in Caseflow

…PEALS-55812

APPEALS-55812 Remove New Relic gem in Caseflow

… assigned_to_location CSS IDs

…restricted status state

…staging to RSpec file

  Add search query service for the api response for the `/search` page

…lease) (#22776)

* 🔧 Assume defaults for `config.action_dispatch.use_cookies_with_metadata` and `config.action_mailer.delivery_job`

The following config settings are not backwards compatible:

- config.action_dispatch.use_cookies_with_metadata
- config.action_mailer.delivery_job

Now that Rails 6.0 is stable on production, we can assume
their default values going forward.

* ✅ Fix flakey spec

* 🔧 Assume default for `config.action_dispatch.use_authenticated_cookie_encryption`

Since we are making other cookie configuration changes in this PR for Rails 6.0,
this is an opportune time to migrate this Rails 5.2 cookie setting to its
default value as well.

* ⏪️ Restore overrides for `config.action_dispatch.use_authenticated_cookie_encryption` and `config.action_dispatch.use_cookies_with_metadata`

While testing in PreProd, we discovered that, without these cookie config overrides,
re-authentication was broken -- after logging out, a user could not log back in.

Since the default settings are still optional going forward, we can restore these
overrides and devise a solution to migrate cookies later.

For more details, see Jira story APPEALS-54897:

https://jira.devops.va.gov/browse/APPEALS-54897

* ✨ Add new utility module for adding DB indexes concurrently

Introduces `Caseflow::Migrations::AddIndexConcurrently` as a replacement
for `Caseflow::Migration` for migrations on ActiveRecord 6.0 and beyond,
since `Caseflow::Migration` is forever coupled to ActiveRecord 5.1 due
to its extensive use on legacy migrations and should be deprecated moving
forward.

* 🗑️ Deprecate `Caseflow::Migration`

* 🔧 Add instructive error message for non-concurrent `add_index` migrations

* 🚨 Address linter / codeclimate complaints

* ✨ Introduce `SslRedirectExclusionPolicy`

To be used in the environment configuration settings for excluding exempt
request paths from SSL redirects when `config. force_ssl = true`

* ♻️ Replace deprecated controller-level `force_ssl`

Replace deprecated controller-level `force_ssl` with
equivalent configuration settings in preparation for
the Rails 6.1 upgrade.

* 🔥 Remove deprecated config setting `config.active_record.sqlite3.represent_boolean_as_integer`

This will have no implications for Caseflow, since we are only using the `sqlite3`
adapter nominally for the `demo_vacols` database, which is not actually being used
in our demo environments (demo environments are deployed as `development` envs).

* ⬆️ Update `caseflow-commons` to resolve sub-dependency conflicts

Removes unneeded gems `bourbon` and `neat`, which had a sub-dependency conflict on `thor`.

* ⬆️ Update rails and other gems as necessary

* 🐛 Fix 'uninitialized constant' error when loading app

* ⬆️ bin/rails app:update - Apply relevant changes

* 🔧 Override default for `config.active_record.has_many_inversing`

* 🔧 Assume default for `config.active_storage.track_variants`

We're not currently using ActiveStorage in Caseflow, so it
is safe to just assume the default here.

* 🔧 Override default for `config.active_job.retry_jitter`

The default jitter is probably safe, however, I'm not 100% sure that we
don't have any jobs that need to be requeued with exact wait times. So
we let's override this for now to stay on the safe side.

* 🔧 Assume default for `config.active_job.skip_after_callbacks_if_terminated`

We're not currently using `throw :abort` within any `before_enqueue`/`before_perform` 
callbacks on existing Caseflow jobs, so the default should be fine here.

For more background, see
https://lilyreile.medium.com/rails-6-1-new-framework-defaults-what-they-do-and-how-to-safely-uncomment-them-c546b70f0c5e#4c60

* 🔧 Assume default for `config.action_dispatch.cookies_same_site_protection`

This setting controls the `SameSite` optional attribute for the
`Set-Cookie` header.

`SameSite=Lax` means that the cookie is not sent on cross-site requests,
such as on requests to load images or frames, but is sent when a user is
navigating to the origin site from an external site (for example, when
following a link). This is the default behavior if the SameSite attribute
is not specified.

`Lax` is currently the default assumed by both Chrome and Edge browsers
when this attribute is left unspecified, so assuming this value
should be sensible. It allows us to have our cake (blocking CSRF attacks)
and eat it too (providing a logged-in experience when users navigate to
Caseflow  across origins).

For more background, see
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie#samesitesamesite-value
- https://lilyreile.medium.com/rails-6-1-new-framework-defaults-what-they-do-and-how-to-safely-uncomment-them-c546b70f0c5e#1f15

* 🔧 Assume default for `config.action_controller.urlsafe_csrf_tokens`

* 🔧 Assume default for `ActiveSupport.utc_to_local_returns_utc_offset_times`

We're not using `ActiveSupport::TimeZone.utc_to_local` anywhere,
so the default is safe to assume here.

* 🔧 Assume default for `config.action_dispatch.ssl_default_redirect_status`

The default is safe to assume.

For more background, see
https://lilyreile.medium.com/rails-6-1-new-framework-defaults-what-they-do-and-how-to-safely-uncomment-them-c546b70f0c5e#4c3e

* 🔧 Assume default for `config.active_record.legacy_connection_handling`

The default should be safe to assume here, as we do not do any
role or shard switching on database connections.

For more background, see
https://lilyreile.medium.com/rails-6-1-new-framework-defaults-what-they-do-and-how-to-safely-uncomment-them-c546b70f0c5e#8007

* 🔧 Assume default for `config.action_view.form_with_generates_remote_forms`

We don't use the `form_with` helper anywhere, so this behavior change is
inconsequential for us, and we can safely assume the new default.

* 🔧 Assume default for `config.active_storage.queues.analysis`

We do not use ActiveStorage, so the default is safe to assume here.

* 🔧 Assume default for `config.active_storage.queues.purge`

We do not use ActiveStorage, so the default is safe to assume here.

* 🔧 Assume default for `config.action_mailbox.queues.incineration`

We don't use ActionMailbox, so the new default is safe to assume here.

* 🔧 Assume default for `config.action_mailbox.queues.routing`

We do not use ActionMailbox, so the default is safe to assume here.

* 🔧 Assume default for `config.action_mailer.deliver_later_queue_name`

We're not using `ActionMailer::MessageDelivery #deliver_later` anywhere,
so the default is safe to assume.

* 🔧 Assume default for `config.action_view.preload_links_header`

This flag can be safely uncommented. Browsers that support Link
headers will get a performance boost. Browsers that don’t will
ignore them.

We override in `development` environments to avoid an edge case
leading to an HTTP response header overflow.

For more background, see
https://lilyreile.medium.com/rails-6-1-new-framework-defaults-what-they-do-and-how-to-safely-uncomment-them-c546b70f0c5e#3679

* 🔥 Remove 'new_framework_defaults_6_1.rb'

* 🔧 Load defaults for Rails 6.1

* ♻️ Extract constant

* ♻️ Migrate to new Rails deprecation config where applicable

* ♻️ Push members down now that there is only one subclass

* 🩹 Add forgotten disallowed deprecation warning

This deprecation warning was addressed by the following
PR, but we forgot to add it to the list of disallowed
deprecation warnings:

#21614

* 💡 Update comment

Task `rake routes` has been replaced with `rails routes`

* ✅ Update test to account for change to `ActionDispatch::Response#content_type` 

`ActionDispatch::Response#content_type` now returns the full Content-Type header

* 🚨 Exclude 'config.ru' from Rubocop cops

* 🚚 Move 'db/etl/migrate' to 'db/etl_migrate'

* 🚚 Move 'db/etl/schema.rb' to 'db/etl_schema.rb'

* ♻️ Arrange 'database.yml' configs by environment

Group DB configs by environment in anticipation of
reformatting for Rails 6+ multi-DB configuration.

* 🔧 Reformat 'database.yml' to Rails 6+ multi-DB conventions

* 🔧 Add etl migration paths to DB config

* 🔧 Update DB connection names in 'database_cleaner' config

* ♻️ Use new database-specific rake tasks

After migrating to the Rails 6+ native multi-database configuration,
the behavior of some DB management tasks, such as `rake db:migrate`
changed such that they now act on ALL databases and not just the
primary database. So we must replace the invocations of these
tasks with their new, database-specific counterparts.

* ➖ Remove 'multiverse' gem

Now that we have fiully transitioned to Rails-native
multi-database support, we are no longer reliant on
the 'multiverse' gem and can remove it.

* 🗃️ Prohibit execution of vacols DB and non-DB-specific rake tasks

After transitioning to Rails-native multi-DB support,
the behavior of some DB tasks changed such that they will now
act on ALL databases and not just the primary database
(ex. `rake db:migrate` will now migrate ALL databases).

To avoid accidents, we re-define these tasks here to no-op
and output a helpful message to redirect developers toward
using their new database-specific counterparts instead.

* ♻️ Create new environment for GH workflow 'Make-docs-to-webpage'

Instead of performing a bunch of hard-to-maintain `sed` gymnastics
to modify the existing 'test' environment, let's create a new
'make_docs' environment (based off of 'test') and configure it
appropriately for use by the 'Make-docs-to-webpage' GH workflow.

* 💚 Remove redundant DB migrations from CI workflow

Task `db:schema:load` already loads the checked in schema,
so there should be no need to run `db:migrate` afterwards.

* 🐛 Fix `spec/mailers/hearing_mailer_spec.rb` - `NoMethodError`

Addresses the following error:

  NoMethodError: undefined method `build_lookup_context' for ActionView::Base:Class

* 🐛 Fix `spec/workflows/post_decision_motion_updater_spec.rb` - `FrozenError`

Addresses the following error:

  FrozenError: can't modify frozen Hash: {}

* ✅ Add test for `RoSchedulePeriod`

* 🐛 Fix `spec/models/schedule_period_spec.rb` - `ActiveRecord::RecordInvalid`

Apparently, there were some changes to the inner workings of
`ActiveModel::Errors` in Rails 6.1, causing a model to be
considered invalid in the case that `errors[:base] == [[]]`.
This makes sense, as `[[]]` is not considered "empty".

Unfortunately, this was causing `RoSchedulePeriod #validate_spreadsheet`
to inadvertently mark the model as invalid upon creation.
`HearingSchedule::ValidateRoSpreadsheet #validate` returns an
empty array (`[]`) when valid, which gets pushes onto the
`RoSchedulePeriod` `errors[:base]` array, resulting in a
non-empty array (`[[]]`) and an erroneously invalid disposition.

Furthermore, calling `<<` to an `ActiveModel::Errors` message array
in order to add an error is a deprecated, so we can take this
opportunity to use the new `#add` API to hit two birds with one stone.

The change implemented here is not a pure refactoring, however the
end-user experience is unchanged in terms of how errors are presented
when attempting to upload a spreadsheet with multiple non-conformities.

Down the road, we may want to consider moving
`HearingSchedule::ValidateRoSpreadsheet` toward using
`ActiveModel::Validations` in order to leverage the
full `ActiveModel::Errors` API and construct the
errors object in the prescribed manner.

For more details see
- https://api.rubyonrails.org/v6.1.7.7/classes/ActiveModel/Validations.html
- https://api.rubyonrails.org/v6.1.7.7/classes/ActiveModel/Errors.html

* 🐛 Fix `spec/mailers/hearing_mailer_spec.rb` - `ActionView::Template::Error`

* ✅ Fix `spec/models/veteran_spec.rb`

* ✅ Fix `spec/sql/ama_cases_sql_spec.rb`

Addresses failures such as the below:

  0) AMA Cases Tableau data source expected report calculates age and AOD based on person.dob
     Failure/Error: expect(aod_case["aod_veteran.age"]).to eq("76")

       expected: "76"
            got: 0.76e2

* ✅ Fix multiple specs - `Minitest::UnexpectedError`

Test helper method `#perform_enqueued_jobs` now wraps exceptions
in an `Minitest::UnexpectedError`:

https://github.com/rails/rails/blob/914caca2d31bd753f47f9168f2a375921d9e91cc/activejob/lib/active_job/test_helper.rb#L591

So, to assert that a specific
exception is raised during execution of the `#perform_enqueued_jobs`
block, we must rescue the `Minitest::UnexpectedError` and make the
assertion on its error message instead.

* ✅ Fix `spec/lib/helpers/association_wrapper_spec.rb`

* ✅ Fix `spec/controllers/api/v1/jobs_controller_spec.rb`

In Rails 6.1, `ActiveJob #perform_now` was changed to
behave as it did once before (at the behest of GitHub),
returning the value fo the job instead of true/false.

See related GH issue: rails/rails#38040

* 🐛 Fix `spec/controllers/appeals_controller_spec.rb` - `NoMethodError`

Addresses error:

  NoMethodError: undefined method `workflow' for #<CaseSearchResultsForVeteranFileNumber:0x00007f9a030966c8>

  0) AppealsController GET appeals when current user is a System Admin when request header does not contain Veteran ID responds with an error
     Failure/Error: errors: errors.messages[:workflow],

     NoMethodError:
       undefined method `workflow' for #<CaseSearchResultsForVeteranFileNumber:0x00007f9a030966c8>
     # ./app/workflows/case_search_results_base.rb:31:in `search_call'

* 🐛 Fix `CaseSearchResultsBase` validations

Addresses test failures in `spec/controllers/appeals_controller_spec.rb`
similar to the below:

  AppealsController GET appeals when current user is a System Admin when request header does not contain Veteran ID responds with an error
     Failure/Error: expect(response_body["errors"][0]["title"]).to eq "Veteran file number missing"

       expected: "Veteran file number missing"
            got: nil

Using `ActiveModel::Errors` to store error data in an arbitrary format may have been
somewhat permissible in the past, but it is an abuse of the object's intended use and is
also proving incompatible with the more formalized `ActiveModels::Errors` API in Rails 6.1.

In order to preserve the existing response shape of the affected JSON endpoints,
we need to move away from the `ActiveModel::Validations` implementation on `CaseSearchResultsBase`
(and its descendent classes) to a more bespoke method of performing validations and aggregating
errors, since Rails 6.1 `ActiveModel::Errors` is no longer appropriate for  our needs here.

* ✅ Fix `spec/controllers/application_controller_spec.rb` -- Cache-Control error

Addresses the test failure below:

    ApplicationController no cache headers when toggle set sets Cache-Control etc
     Failure/Error: expect(response.headers["Cache-Control"]).to eq "no-cache, no-store"

       expected: "no-cache, no-store"
            got: "no-store"

       (compared using ==)
     # ./spec/controllers/application_controller_spec.rb:59:in `block (4 levels) in <top (required)>'

In Rails 6.1, the `no-store` directive is exclusive of any others
that are set on the `Cache-Control` header, which makes sense given
the specification https://datatracker.ietf.org/doc/html/rfc7234#section-3

This change was implemented in PR rails/rails#39461

Since it no longer makese sense to set both `no-store` and `no-cache` directives,
we will only set `no-store` here, as that is the stronger of the two.

* 🐛 Fix multiple specs - `ActiveRecord::EagerLoadPolymorphicError`

Addresses multiple test failures caused by the error below:

  QueueConfig.to_hash title when assigned to an org is formatted as expected
     Failure/Error:
       tasks.with_assignees.group("assignees.display_name").count(:all).each_pair.map do |option, count|
         label = self.class.format_option_label(option, count)
         self.class.filter_option_hash(option, label)
       end

     ActiveRecord::EagerLoadPolymorphicError:
       Cannot eagerly load the polymorphic association :appeal
     # ./app/models/queue_column.rb:110:in `assignee_options'

* 🐛 Fix `spec/models/task_spec.rb` - `update_all` clears query cache

In Rails 6.1.7.7, the method `ActiveRecord::Relation #update_all`
will now clear any records cached by the calling relation.

This was altering the behavior of `Task #cancel_task_and_child_subtasks`
and causing the following test failure:

  Task#cancel_task_and_child_subtasks cancels all tasks and child subtasks
     Failure/Error: expect(second_level_tasks[0].versions.count).to eq(initial_versions + 2)

       expected: 3
            got: 2

       (compared using ==)
     # ./spec/models/task_spec.rb:368:in `block (3 levels) in <top (required)>'

To remedy, we will now cache the necessary Task records in an Array, which can
be used for generating PaperTrail versions both before and after the `update_all`.

* 🐛 Fix `spec/services/hearings/calendar_service_spec.rb` - template rendering error

Addresses the following test failure:

  Hearings::CalendarService.confirmation_calendar_invite returns appropriate iCalendar event
     Failure/Error: expect(ical_event.description).to eq(expected_description)

       expected: "You're scheduled for a virtual hearing with a Veterans Law Judge of the Board of Veterans' Appeals.\...to reschedule or cancel your virtual hearing, contact us by email at bvahearingteamhotline@va.gov\n"
            got: #<Icalendar::Values::Text("You're scheduled for a virtual hearing with a Veterans Law Judge of the Bo... reschedule or cancel your virtual hearing, contact us by email at bvahearingteamhotline@va.gov\n")>

* 🐛 Fix YAML syntax error caused by whitespace in ENV var

Address the following error, found during demo deployment:

  rake aborted!
  Cannot load database configuration:
  YAML syntax error occurred while parsing /caseflow/config/database.yml. Please note that YAML must be consistently indented using spaces. Tabs are not allowed. Error: (<unknown>): could not find expected ':' while scanning a simple key at line 49 column 5

* ⬆️ Update `caseflow-commons` dependency to latest ref

Removes `bourbon` and `neat` dependencies.