…thod for others to leverage.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Most of this is just playing, but making the StreamManager implement AutoCloseable gives a place to end spans to show how long a serializer/connection factory was relevant for.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…to the collector to prometheus, zipkin, etc

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…e config hierarchy.

This was broken from the merge https://github.com/opensearch-project/opensearch-migrations/pull/376/files#diff-430f89dc33402ecf692b9a8372f66e585bb2f9215596433216580efc2a56795c.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…lotted within the same graph in prometheus.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…ics into an optional.

Dropping the optionals makes the code simpler and if we don't want to do logging, we can just not fill in the configuration for the SDK.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…troduce some more typesafe wrappers for contexts.

Lots more to come.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…explicitly passing strongly typed context objects.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Make sure that the context is using the right requestKey, which also will have the appropriate indices as per the test context.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…ayerRequestContext to the replayer

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Don't bother showing the Kakfa offloader just buffering (was called recordStream).  Now the offloader span is a child span of the connection span from the handler, so we can see the handler gathering the request/response (or waiting for the response).

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

That makes it a separate state for the logging handler superclass.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…rocessor.

Prometheus metrics already have an export_name that is unique, the processors weren't doing anything useful, & the namespace was appending EVERYTHING from one of the two services.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…d (less so for now) metrics can be exported across more of the lifetime of a request/connection.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

… a test bug.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

# Conflicts:
#	TrafficCapture/nettyWireLogging/src/main/java/org/opensearch/migrations/trafficcapture/netty/ConditionallyReliableLoggingHttpRequestHandler.java
#	TrafficCapture/nettyWireLogging/src/main/java/org/opensearch/migrations/trafficcapture/netty/LoggingHttpRequestHandler.java
#	TrafficCapture/nettyWireLogging/src/test/java/org/opensearch/migrations/trafficcapture/netty/ConditionallyReliableLoggingHttpRequestHandlerTest.java
#	TrafficCapture/trafficCaptureProxyServer/src/main/java/org/opensearch/migrations/trafficcapture/proxyserver/netty/ProxyChannelInitializer.java
#	TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/Accumulation.java
#	TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/CapturedTrafficToHttpTransactionAccumulator.java
#	TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/RequestResponsePacketPair.java
#	TrafficCapture/trafficReplayer/src/main/java/org/opensearch/migrations/replay/RequestSenderOrchestrator.java
#	TrafficCapture/trafficReplayer/src/test/java/org/opensearch/migrations/replay/SimpleCapturedTrafficToHttpTransactionAccumulatorTest.java

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…when the request was ignored and remove the now-unused file for responses.

I'd like to revisit this eventually to make sure that it's as efficient as possible and to organize it better.  However, it does get the job done for now and tests were updated to confirm.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…uests and responses.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…ifferent levels of them.

This has the minimal amount of work to get those relationships to simply compile.  Nearly every unit test fails and the code is more clunky than it needs to be, but getting to this point was alone a major lift.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…in, but the replayer isn't crashing.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…scovered by trace inspection.

1) close was called AFTER the RRPair was rotated, so there were no traffic streams being committed, resulting in a perpetual hole in the commit log.
2) close was being scheduled immediately, before all requests (in most cases) because the channelInteractionNumber was being miscalculated.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…using the same ConnectionReplaySession to be returned for every connection, which resulted in serious corruption of ordering and tests never completing.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…rafficTest runs (-1, false) is still missing a message.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…n earlier edit around flushing the streams held list on close.

That should only happen when the close was on an accumulated pair that had a broken request that wasn't rotated.  Otherwise, the traffic streams would be committed twice.
I also tweaked makeTrafficStreams to keep stable ids on the individual streams to make debugging runs a LOT simpler.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

It could be called from one thread while the nextCommit maps were being modified by other (Kafka Consumer) threads.  Now there's a lock and a quick copy to protect against that race.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…/kafka interactions + preempt blocking in the BlockingTrafficSource when there's keys (offsets) to commit.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…tationAttributes.

This allows for passing root contexts around that have attributes but don't have an associated span.  This helps make the code have less assumptions about how it is situated.
The change also opens the door to removing all of the static factories for spans and metrics.  Those factories can be chained from a top-level context that is passed throughout the callstack, rooted from these new IInstrumentationAttributes classes.  That change isn't here, but will probably be completed in the near future & this makes it easier.
I'm also in the process of adding contexts and spans to more places (like the high-level traffic source work), which is what precipitated the greater change.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…e target transaction.

Unfortunately, the TrafficReplayer fails to load because of a race condition around statically initializing the Otel SDK.
However, all unit tests are working, so this is a checkpoint release before I remove the reliance on static otel initialization and move toward it being done via contexts.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

The static initializer race conditions have been resolved (no more static otel for tracing and metering) and the E2E solution is functional again.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…rouping more contexts together in files.

1) RootOtelContext creates all SimpleMeteringClosures and IInstrumentationAttributes has default methods to wire together all metric emissions (making it a lot more convenient).
2) All IInstrumentationAttributes have a scopeName now so that it's consistent and can be used to make metrics and spans.
3) All replayer contexts were split into interfaces and implementations (bridge or PIMPL pattern).  I've moved them around so that they now all reside in the tracing directory in the same type of structure.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…s to bring it up with or w/out InMemory exporters.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…e a context into it so that span verifications can happen within unit tests.

The FullTrafficReplayerTest now has a test that is verifying the number of spans that were reported.  That was usefuly to work through to fix the bugs in double counting some spans.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

I'm not adding one BETWEEN writes yet because that code is too complicated and should be simplified.  I'm also not sure that that isn't going to flood the traces for limited value.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

… be used for some metrics too.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…mit more.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

… a metric name had changed and meter delta events as upDownCounters.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…ntrol over which attributes are included within metrics and spans.

I've also added "activeConnection" for capture connections.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…rity to 2.11.1.0

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

I've personally been using jaeger more and zipkin has been crashing with an OOM on startup (probably due to my docker setup) and I suspect that it has been causing greater problems with the otel collector.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…so that the connectionId is now emitted from the activeConnections metric.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…nd adding some TODOs for future research.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

I still need to continue to knock the dimensionality of the data (unique attributes) down considerably, but this at least mitigates the grpc overflow errors that I was seeing.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…c can be emitted with the span data as its exemplar.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…o preserve new work first.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…ic instruments for each of the contexts' needs.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

… metric instruments into the context classes.

I'm not happy with the scope type propagation through so much of the interface hierarchy, so I'm going to revisit that next.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…k) are problematic.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…f the instrumentation interfaces.

That was only needed for one call, which is easily inlined into the implementation classes by putting createChildContext() calls into each parent.  That allows the implementations to go back to their own root scopes and do whatever is necessary.  Lots of code is a lot easier to read and maintain now.
There are still other changes that I'm in the process of making, including supporting linked spans and squashing compilation errors (still).

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…-level Capture proxy is remaining).

I need to stitch together a couple root context classes to provide disparate top-level contexts.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…nection id naming bug that was causing FullReplayerTests to fail.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…trics flag to the prometheus exporter for the otel collector to support exemplars.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…to its own span. I've also updated linked spans to allow for more than one (will test shortly)

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

… class so that I can initialize and teardown the context (and do some sanity checks in the process).

I've already found and fixed a couple issues with scoped contexts being doubly closed, which has resulted in UpDownCounters being corrupted.
The final check on the TestContext to make sure that all scopes are closed isn't active yet since the results are "862 tests completed, 808 failed".

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

I had been pulling activity names from the enclosing class, but in cases of inheritance, that was unmaintainable.  That had caused a number of automatically-generated metrics to be missing because subclasses would use the superclass ACTIVITY_NAME definition.  Now, all the MetricInstruments constructors are private and the objects are created via factory functions that are alongside the inner class definitions.  That moves the naming responsibilities to the actual part of the code that cares and makes the system more foolproof.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…t setup method was corrupted.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…tials.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…toring to rely upon values within contexts rather than passing redundant copies for things like request/channel keys.

Logging comparison metrics as logs is being phased out, being replaced by using metric and span attributes to directly pick up values (and letting those downstream dashboards to add additional logic to determine if values were matching/not, though statusMatch does still exist as a metric - only because its quick and easy on both ends).
All tests pass, but I haven't done any testing with the dockerSolution.
I need to re-enable the kafka container test because I need to be validating constantly that that test is working, but as of now, it throws an OOM error, which needs to be investigated.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…nstruments class when it should not have been

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

… as an api dependency for coreUtilities, since that's a tight coupling.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…sues.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…rfile was hardcoded for x86_64.

Install awscli via pip now, which should work on both environments.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Support 3 different otel-collector container configurations for docker-compose.
* Prometheus + Jaeger + OpenSearch (for metrics, traces, and 'analytics/logs') - all of which are local
* AWS CloudWatch + AWS X-Ray (using a mounted credentials file if present, or reliant how well the base image can resolve credentials) + OpenSearch (local)
* All of the above

`./gradlew :dockerSolution:composeUp` will launch the first configuration by default.  Passing the flag `-Potel-collector=otel-aws.yml` will use the AWS configuration and `-Potel-collector=otel-everything.yml` will use the 'everything' configuration.

The otel-collector container itself still uses the custom-built collector from mikaylathompson as the base image.  However, the extended image normalizes the user and entrypoint so that further extensions and applications of the container can behave as if the base image was the AWS Distro for OpenTelemetry collector base image (amazon/aws-otel-collector, see https://aws-otel.github.io/docs/setup/docker-images).  That also includes being able to convert the credentials file into environment variables since the otel-collector was otherwise struggling to use the credentials file directly.

The otel-collector configuration is contained within one yaml file and the code within otelConfigs now allows one to fully materialize an otel-collector config file.  The otel-config*.yaml files within the dockerSolution have been created via the makeConfigFiles.sh script.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Use System.nanotime() instead of Instant.now() for duration calculations.
Call the fillAttributes() method from the super (class or interface) before doing any more puts.
Correct the order that attributes are pulled from the scope hierarchy (and add a test)
Use the ActivityNames pattern for IWireCaptureContexts
StreamLifecycleManager no longer implements Autoclosable.  I didn't need it to close down contexts and no classes had implemented a non-empty version except to log that it was called.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…d OS analytics engine by default.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

One for attributes that should be present in all sub-spans and another where the attributes should only be present within the current one (and not children)

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…eivers' for metrics, traces and logs templates.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…aws config file

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

… xray

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…instead of https. I've also removed the 'no-ssl' option and deduce it from the protocol

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

# Conflicts:
#	test/awsE2ESolutionSetup.sh

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

… That's been pushed down to its subclass IScopedInstrumentationAttributes.

Attributes for spans are now filled when the span is closed rather than when it is created.  This gives less leeway to being able to override the value w/out changing the value within the context class, but the context class should be the ground truth, record of value, so this seems like it's the right behavior anyway.
With those changes, I did some cleanup on the attribute values that were being tracked for tuple comparison.  Now status codes are tracked as metric AND span attributes.  That should make it MUCH easier to search metrics for specific patterns that popped up in metrics.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

# Conflicts:
#	TrafficCapture/captureOffloader/src/test/java/org/opensearch/migrations/trafficcapture/StreamChannelConnectionCaptureSerializerTest.java
#	TrafficCapture/dockerSolution/src/main/docker/migrationConsole/runTestBenchmarks.sh
#	deployment/cdk/opensearch-service-migration/lib/service-stacks/migration-analytics-stack.ts

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…t constructor override, throw an IllegalArgumentException.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>

…losed.

Signed-off-by: Greg Schohn <greg.schohn@gmail.com>