3 changes/fixes around how tuples are output by the replayer. #513
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Copy the bytes into the tuple output rather than try to use the array (which could be very big and full of padded zeros). Don't use a json layout for log4j output - that cases the tuples to be escaped. Instead, just output the json-coded contents + a newline. Fix the humanReadables script to work with the above change. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
gregschohn
requested review from
AndreKurait,
chelma,
kartg,
lewijacn,
mikaylathompson,
okhasawn and
sumobrian
as code owners
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #513 +/- ##
============================================
+ Coverage 76.71% 76.93% +0.21%
- Complexity 1385 1388 +3
============================================
Files 158 158
Lines 6086 6105 +19
Branches 532 532
============================================
+ Hits 4669 4697 +28
Misses 1059 1059
+ Partials 358 349 -9
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
lewijacn
approved these changes
Feb 20, 2024
sumobrian
reviewed
Feb 20, 2024
| @@ -156,7 +156,7 @@ public void testOutputterForGet() throws IOException { | |||
| " \"Status-Code\": 200,\n" + | |||
| " \"Reason-Phrase\": \"OK\",\n" + | |||
| " \"response_time_ms\": 0,\n" + | |||
| " \"body\": \"SFRUUC8xLjEgMjAwIE9LDQpDb250ZW50LXRyYW5zZmVyLWVuY29kaW5nOiBjaHVua2VkDQpEYXRlOiBUaHUsIDA4IEp1biAyMDIzIDIzOjA2OjIzIEdNVA0KVHJhbnNmZXItZW5jb2Rpbmc6IGNodW5rZWQNCkNvbnRlbnQtdHlwZTogdGV4dC9wbGFpbg0KRnVudGltZTogY2hlY2tJdCENCg0KMWUNCkkgc2hvdWxkIGJlIGRlY3J5cHRlZCB0ZXN0ZXIhCg0KMA0KDQo=\",\n" + | |||
| " \"body\": \"SSBzaG91bGQgYmUgZGVjcnlwdGVkIHRlc3RlciEK\",\n" + | |||
There was a problem hiding this comment.
I was hoping there was funny message encoded ◡̈
sumobrian
reviewed
Feb 20, 2024
| @@ -156,7 +156,7 @@ public void testOutputterForGet() throws IOException { | |||
| " \"Status-Code\": 200,\n" + | |||
| " \"Reason-Phrase\": \"OK\",\n" + | |||
| " \"response_time_ms\": 0,\n" + | |||
| " \"body\": \"SFRUUC8xLjEgMjAwIE9LDQpDb250ZW50LXRyYW5zZmVyLWVuY29kaW5nOiBjaHVua2VkDQpEYXRlOiBUaHUsIDA4IEp1biAyMDIzIDIzOjA2OjIzIEdNVA0KVHJhbnNmZXItZW5jb2Rpbmc6IGNodW5rZWQNCkNvbnRlbnQtdHlwZTogdGV4dC9wbGFpbg0KRnVudGltZTogY2hlY2tJdCENCg0KMWUNCkkgc2hvdWxkIGJlIGRlY3J5cHRlZCB0ZXN0ZXIhCg0KMA0KDQo=\",\n" + | |||
| " \"body\": \"SSBzaG91bGQgYmUgZGVjcnlwdGVkIHRlc3RlciEK\",\n" + | |||
There was a problem hiding this comment.
I was hoping there was funny message encoded ◡̈
sumobrian
approved these changes
Feb 20, 2024
|
Was human readable script tested? |
|
Yes, the script was tested against a simple run. There still aren't unit tests, so I'm not sure exactly what the behavior should be. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Simplify and fix the json output of the 'tuples' that the replayer emits. No longer wrap the tuples within a log4j json record, just shoot it straight out, followed by a newline. Furthermore, the bodies often had many null values within them. That's been fixed by copying just the readable bytes rather than taking the whole byte array.
Description
Issues Resolved
#511
https://opensearch.atlassian.net/browse/MIGRATIONS-1510
Testing
Updated unit tests. Ran the docker solution and a standalone replayer with an input stream.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.