Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Centralized all dependencies in TrafficCapture #605

Merged
merged 26 commits into from
May 10, 2024
Merged

Centralized all dependencies in TrafficCapture #605

merged 26 commits into from
May 10, 2024

Conversation

peternied
Copy link
Member

@peternied peternied commented Apr 23, 2024 •
edited
Loading

Description

Create versions property for all shared dependencies. This should make version bumps easier across all the sub projects along.

Testing

./gradlew assemble

Issues

Check List

  • New functionality includes testing
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@peternied
Create versioons property for all shared dependencies
919b83c 
Signed-off-by: Peter Nied <petern@amazon.com>
peternied
peternied commented Apr 23, 2024
View reviewed changes
TrafficCapture/build.gradle Outdated Show resolved Hide resolved
TrafficCapture/nettyWireLogging/build.gradle Outdated Show resolved Hide resolved
@peternied
Add centralized traffic capture dependencies project
acd42f8 
Signed-off-by: Peter Nied <petern@amazon.com>
@peternied
Remove all specific version references
303782d 
Signed-off-by: Peter Nied <petern@amazon.com>
@peternied
Merge from main
f3d5672 
Signed-off-by: Peter Nied <petern@amazon.com>
@peternied
Fix missing missing log4j bom reference
6e05cd1 
Signed-off-by: Peter Nied <petern@amazon.com>
@peternied
Resolve testCompile related failures
7876810 
Signed-off-by: Peter Nied <petern@amazon.com>
@peternied peternied changed the title Create versions property for all shared dependencies Centralized all dependencies in TrafficCapture Apr 30, 2024
@codecov Codecov
Copy link

codecov bot commented Apr 30, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.60%. Comparing base (2ae2959) to head (38cfe19).
Report is 14 commits behind head on main.

Additional details and impacted files 
@@             Coverage Diff              @@
##               main     #605      +/-   ##
============================================
- Coverage     75.84%   75.60%   -0.25%     
+ Complexity     1545     1541       -4     
============================================
  Files           168      168              
  Lines          6446     6455       +9     
  Branches        574      575       +1     
============================================
- Hits           4889     4880       -9     
- Misses         1177     1192      +15     
- Partials        380      383       +3
Flag Coverage Δ
unittests 75.60% <ø> (-0.25%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@peternied
Fix CVE-2024-30171 by updating bouncycastle from 1.74 to 1.78
bd7f76a 
Signed-off-by: Peter Nied <petern@amazon.com>
@peternied
Merge remote-tracking branch 'origin/main' into centralized-dependencies
1c3be04 
Signed-off-by: Peter Nied <petern@amazon.com>
@peternied
Remove BOM references
73da977 
Signed-off-by: Peter Nied <petern@amazon.com>
@peternied
Uniform reference type on dependenices
af4d7e6 
Signed-off-by: Peter Nied <petern@amazon.com>
@peternied
Clean up the dependencies
ea2ac5e 
Signed-off-by: Peter Nied <petern@amazon.com>
@peternied peternied marked this pull request as ready for review May 1, 2024 19:59
peternied and others added 6 commits May 1, 2024 21:05
@peternied
Move opentelemetry-sdk-testing into the same group with the other OTL…
8c040bb 
… dependencies

Signed-off-by: Peter Nied <petern@amazon.com>
@peternied
Merge branch 'main' into centralized-dependencies
3394ebd
@peternied
Allow unsafe access for netty on jdk11 platform
8c2bdae 
Signed-off-by: Peter Nied <peternied@hotmail.com>
@peternied
Revert "Allow unsafe access for netty on jdk11 platform"
a2df582 
This reverts commit 8c2bdae.

Signed-off-by: Peter Nied <peternied@hotmail.com>
@peternied
Upload traffic capture test reports
ead6d8e 
Signed-off-by: Peter Nied <peternied@hotmail.com>
@peternied
Add back info message to Run Tests with Coverage since its stalling
7249bb3 
Signed-off-by: Peter Nied <peternied@hotmail.com>
gregschohn
gregschohn reviewed May 2, 2024
View reviewed changes
Copy link
Collaborator

@gregschohn gregschohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really cool. Thanks for putting this together.
I have a couple questions so that I can better understand how it works.
Thanks!

TrafficCapture/build.gradle Show resolved Hide resolved
TrafficCapture/coreUtilities/build.gradle Show resolved Hide resolved
TrafficCapture/testUtilities/build.gradle Show resolved Hide resolved
TrafficCapture/trafficCaptureDependencies/build.gradle Outdated Show resolved Hide resolved
@peternied
Remove constraints from proxy server build
1f5891b 
Signed-off-by: Peter Nied <petern@amazon.com>
@peternied
Upgrade log4j from 2.20.0 to 2.23.1
55181e9 
Signed-off-by: Peter Nied <petern@amazon.com>
@peternied
Switch to implementation so transitive dependencies aren't included
c8f80ff 
Signed-off-by: Peter Nied <petern@amazon.com>
@AndreKurait
Copy link
Member

I'd be interested in a gradle task to be able to make a diff between changes (at least manual for now).

I was playing around with the following but not sure if it's correct.

allprojects {
    task printResolvedDependencies {
        doLast {
            configurations.findAll { it.canBeResolved && !it.resolvedConfiguration.hasError() }.each { configuration ->
                def resolvedDeps = configuration.resolvedConfiguration.firstLevelModuleDependencies
                if (!resolvedDeps.isEmpty()) {
                    resolvedDeps.each { dependency ->
                        println "Project: ${project.name} Configuration: ${configuration.name} - ${dependency.moduleName}:${dependency.moduleGroup}:${dependency.moduleVersion}"
                    }
                }
            }
        }
    }
}

@peternied
Merge remote-tracking branch 'origin/main' into centralized-dependencies
f051735 
Signed-off-by: Peter Nied <peternied@hotmail.com>
@peternied
Fix usage of protobuf
d9ba33e 
Signed-off-by: Peter Nied <peternied@hotmail.com>
@peternied
Small fixes
f74e186 
Signed-off-by: Peter Nied <peternied@hotmail.com>
@peternied
Switch depedencies to API type so they are applied in test cases as well
6692086 
Signed-off-by: Peter Nied <peternied@hotmail.com>
@peternied
Fix replaying failing to start up
162bea9 
Signed-off-by: Peter Nied <peternied@hotmail.com>
@peternied
exclude netty from security plugin
38cfe19 
Signed-off-by: Peter Nied <peternied@hotmail.com>
@peternied
Copy link
Member Author

@AndreKurait I'm not going to invest in doing much more than :dependencies seems to do the right thing. I did some quick analysis on the classpath generated by the traffic replayer and the e2e test caught when mistakes were made

Comparison on TrafficReplayer classpath

Differences are largely snapping to new versions, the removed fastdoubleparser was dropped by jackson-databine, and netty added a new architecture for riscv64.

  5 /jars/arns-2.23.3.jar:                                                                     <>   5 /jars/arns-2.25.19.jar:
  6 /jars/auth-2.25.19.jar:                                                                    =    6 /jars/auth-2.25.19.jar:
...
 27 /jars/failureaccess-1.0.1.jar:                                                                 27 /jars/failureaccess-1.0.1.jar:
 28 /jars/fastdoubleparser-0.8.0.jar:                                                          +-
 29 /jars/guava-32.0.1-jre.jar:                                                                =   28 /jars/guava-32.0.1-jre.jar:
...
 37 /jars/j2objc-annotations-2.8.jar:                                                              36 /jars/j2objc-annotations-2.8.jar:
 38 /jars/jackson-annotations-2.15.0.jar:                                                      <>  37 /jars/jackson-annotations-2.16.2.jar:
 39 /jars/jackson-core-2.15.0.jar:                                                                 38 /jars/jackson-core-2.16.2.jar:
 40 /jars/jackson-databind-2.15.0.jar:                                                             39 /jars/jackson-databind-2.16.2.jar:
 41 /jars/jackson-dataformat-cbor-2.15.0.jar:                                                      40 /jars/jackson-dataformat-cbor-2.16.2.jar:
 42 /jars/javax.inject-1.jar:                                                                  =   41 /jars/javax.inject-1.jar:
...
 65 /jars/metrics-spi-2.25.19.jar:                                                                 64 /jars/metrics-spi-2.25.19.jar:
 66 /jars/netty-all-4.1.100.Final.jar:                                                         <>  65 /jars/netty-all-4.1.108.Final.jar:
 67 /jars/netty-buffer-4.1.100.Final.jar:                                                          66 /jars/netty-buffer-4.1.108.Final.jar:
 68 /jars/netty-codec-4.1.100.Final.jar:                                                           67 /jars/netty-codec-4.1.108.Final.jar:
 69 /jars/netty-codec-dns-4.1.100.Final.jar:                                                       68 /jars/netty-codec-dns-4.1.108.Final.jar:
 70 /jars/netty-codec-haproxy-4.1.100.Final.jar:                                                   69 /jars/netty-codec-haproxy-4.1.108.Final.jar:
 71 /jars/netty-codec-http-4.1.108.Final.jar:                                                  =   70 /jars/netty-codec-http-4.1.108.Final.jar:
 72 /jars/netty-codec-http2-4.1.100.Final.jar:                                                 <>  71 /jars/netty-codec-http2-4.1.108.Final.jar:
 73 /jars/netty-codec-memcache-4.1.100.Final.jar:                                                  72 /jars/netty-codec-memcache-4.1.108.Final.jar:
 74 /jars/netty-codec-mqtt-4.1.100.Final.jar:                                                      73 /jars/netty-codec-mqtt-4.1.108.Final.jar:
 75 /jars/netty-codec-redis-4.1.100.Final.jar:                                                     74 /jars/netty-codec-redis-4.1.108.Final.jar:
 76 /jars/netty-codec-smtp-4.1.100.Final.jar:                                                      75 /jars/netty-codec-smtp-4.1.108.Final.jar:
 77 /jars/netty-codec-socks-4.1.100.Final.jar:                                                     76 /jars/netty-codec-socks-4.1.108.Final.jar:
 78 /jars/netty-codec-stomp-4.1.100.Final.jar:                                                     77 /jars/netty-codec-stomp-4.1.108.Final.jar:
 79 /jars/netty-codec-xml-4.1.100.Final.jar:                                                       78 /jars/netty-codec-xml-4.1.108.Final.jar:
 80 /jars/netty-common-4.1.100.Final.jar:                                                          79 /jars/netty-common-4.1.108.Final.jar:
 81 /jars/netty-handler-4.1.100.Final.jar:                                                         80 /jars/netty-handler-4.1.108.Final.jar:
 82 /jars/netty-handler-proxy-4.1.100.Final.jar:                                                   81 /jars/netty-handler-proxy-4.1.108.Final.jar:
 83 /jars/netty-handler-ssl-ocsp-4.1.100.Final.jar:                                                82 /jars/netty-handler-ssl-ocsp-4.1.108.Final.jar:
 84 /jars/netty-nio-client-2.25.19.jar:                                                        =   83 /jars/netty-nio-client-2.25.19.jar:
 85 /jars/netty-resolver-4.1.100.Final.jar:                                                    <>  84 /jars/netty-resolver-4.1.108.Final.jar:
 86 /jars/netty-resolver-dns-4.1.100.Final.jar:                                                    85 /jars/netty-resolver-dns-4.1.108.Final.jar:
 87 /jars/netty-resolver-dns-classes-macos-4.1.100.Final.jar:                                      86 /jars/netty-resolver-dns-classes-macos-4.1.108.Final.jar:
 88 /jars/netty-resolver-dns-native-macos-4.1.100.Final-osx-aarch_64.jar:                          87 /jars/netty-resolver-dns-native-macos-4.1.108.Final-osx-aarch_64.jar:
 89 /jars/netty-resolver-dns-native-macos-4.1.100.Final-osx-x86_64.jar:                            88 /jars/netty-resolver-dns-native-macos-4.1.108.Final-osx-x86_64.jar:
 90 /jars/netty-transport-4.1.100.Final.jar:                                                       89 /jars/netty-transport-4.1.108.Final.jar:
 91 /jars/netty-transport-classes-epoll-4.1.100.Final.jar:                                         90 /jars/netty-transport-classes-epoll-4.1.108.Final.jar:
 92 /jars/netty-transport-classes-kqueue-4.1.100.Final.jar:                                        91 /jars/netty-transport-classes-kqueue-4.1.108.Final.jar:
 93 /jars/netty-transport-native-epoll-4.1.100.Final-linux-aarch_64.jar:                           92 /jars/netty-transport-native-epoll-4.1.108.Final-linux-aarch_64.jar:
                                                                                                   93 /jars/netty-transport-native-epoll-4.1.108.Final-linux-riscv64.jar:
 94 /jars/netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:                             94 /jars/netty-transport-native-epoll-4.1.108.Final-linux-x86_64.jar:
 95 /jars/netty-transport-native-kqueue-4.1.100.Final-osx-aarch_64.jar:                            95 /jars/netty-transport-native-kqueue-4.1.108.Final-osx-aarch_64.jar:
 96 /jars/netty-transport-native-kqueue-4.1.100.Final-osx-x86_64.jar:                              96 /jars/netty-transport-native-kqueue-4.1.108.Final-osx-x86_64.jar:
 97 /jars/netty-transport-native-unix-common-4.1.100.Final.jar:                                    97 /jars/netty-transport-native-unix-common-4.1.108.Final.jar:
 98 /jars/netty-transport-rxtx-4.1.100.Final.jar:                                                  98 /jars/netty-transport-rxtx-4.1.108.Final.jar:
 99 /jars/netty-transport-sctp-4.1.100.Final.jar:                                                  99 /jars/netty-transport-sctp-4.1.108.Final.jar:
100 /jars/netty-transport-udt-4.1.100.Final.jar:                                                  100 /jars/netty-transport-udt-4.1.108.Final.jar:
101 /jars/okhttp-4.12.0.jar:                                                                   =  101 /jars/okhttp-4.12.0.jar:
...
134 /jars/third-party-jackson-core-2.25.19.jar:                                                   134 /jars/third-party-jackson-core-2.25.19.jar:
                                                                                               -+ 135 /jars/trafficCaptureDependencies.jar:
135 /jars/trafficReplayer.jar:                                                                 =  136 /jars/trafficReplayer.jar:
...
139 /jars/zstd-jni-1.5.5-1.jar                                                                    140 /jars/zstd-jni-1.5.5-1.jar

@AndreKurait
Copy link
Member

@AndreKurait I'm not going to invest in doing much more than :dependencies seems to do the right thing. I did some quick analysis on the classpath generated by the traffic replayer and the e2e test caught when mistakes were made

Comparison on TrafficReplayer classpath

Differences are largely snapping to new versions, the removed fastdoubleparser was dropped by jackson-databine, and netty added a new architecture for riscv64.

  5 /jars/arns-2.23.3.jar:                                                                     <>   5 /jars/arns-2.25.19.jar:
  6 /jars/auth-2.25.19.jar:                                                                    =    6 /jars/auth-2.25.19.jar:
...
 27 /jars/failureaccess-1.0.1.jar:                                                                 27 /jars/failureaccess-1.0.1.jar:
 28 /jars/fastdoubleparser-0.8.0.jar:                                                          +-
 29 /jars/guava-32.0.1-jre.jar:                                                                =   28 /jars/guava-32.0.1-jre.jar:
...
 37 /jars/j2objc-annotations-2.8.jar:                                                              36 /jars/j2objc-annotations-2.8.jar:
 38 /jars/jackson-annotations-2.15.0.jar:                                                      <>  37 /jars/jackson-annotations-2.16.2.jar:
 39 /jars/jackson-core-2.15.0.jar:                                                                 38 /jars/jackson-core-2.16.2.jar:
 40 /jars/jackson-databind-2.15.0.jar:                                                             39 /jars/jackson-databind-2.16.2.jar:
 41 /jars/jackson-dataformat-cbor-2.15.0.jar:                                                      40 /jars/jackson-dataformat-cbor-2.16.2.jar:
 42 /jars/javax.inject-1.jar:                                                                  =   41 /jars/javax.inject-1.jar:
...
 65 /jars/metrics-spi-2.25.19.jar:                                                                 64 /jars/metrics-spi-2.25.19.jar:
 66 /jars/netty-all-4.1.100.Final.jar:                                                         <>  65 /jars/netty-all-4.1.108.Final.jar:
 67 /jars/netty-buffer-4.1.100.Final.jar:                                                          66 /jars/netty-buffer-4.1.108.Final.jar:
 68 /jars/netty-codec-4.1.100.Final.jar:                                                           67 /jars/netty-codec-4.1.108.Final.jar:
 69 /jars/netty-codec-dns-4.1.100.Final.jar:                                                       68 /jars/netty-codec-dns-4.1.108.Final.jar:
 70 /jars/netty-codec-haproxy-4.1.100.Final.jar:                                                   69 /jars/netty-codec-haproxy-4.1.108.Final.jar:
 71 /jars/netty-codec-http-4.1.108.Final.jar:                                                  =   70 /jars/netty-codec-http-4.1.108.Final.jar:
 72 /jars/netty-codec-http2-4.1.100.Final.jar:                                                 <>  71 /jars/netty-codec-http2-4.1.108.Final.jar:
 73 /jars/netty-codec-memcache-4.1.100.Final.jar:                                                  72 /jars/netty-codec-memcache-4.1.108.Final.jar:
 74 /jars/netty-codec-mqtt-4.1.100.Final.jar:                                                      73 /jars/netty-codec-mqtt-4.1.108.Final.jar:
 75 /jars/netty-codec-redis-4.1.100.Final.jar:                                                     74 /jars/netty-codec-redis-4.1.108.Final.jar:
 76 /jars/netty-codec-smtp-4.1.100.Final.jar:                                                      75 /jars/netty-codec-smtp-4.1.108.Final.jar:
 77 /jars/netty-codec-socks-4.1.100.Final.jar:                                                     76 /jars/netty-codec-socks-4.1.108.Final.jar:
 78 /jars/netty-codec-stomp-4.1.100.Final.jar:                                                     77 /jars/netty-codec-stomp-4.1.108.Final.jar:
 79 /jars/netty-codec-xml-4.1.100.Final.jar:                                                       78 /jars/netty-codec-xml-4.1.108.Final.jar:
 80 /jars/netty-common-4.1.100.Final.jar:                                                          79 /jars/netty-common-4.1.108.Final.jar:
 81 /jars/netty-handler-4.1.100.Final.jar:                                                         80 /jars/netty-handler-4.1.108.Final.jar:
 82 /jars/netty-handler-proxy-4.1.100.Final.jar:                                                   81 /jars/netty-handler-proxy-4.1.108.Final.jar:
 83 /jars/netty-handler-ssl-ocsp-4.1.100.Final.jar:                                                82 /jars/netty-handler-ssl-ocsp-4.1.108.Final.jar:
 84 /jars/netty-nio-client-2.25.19.jar:                                                        =   83 /jars/netty-nio-client-2.25.19.jar:
 85 /jars/netty-resolver-4.1.100.Final.jar:                                                    <>  84 /jars/netty-resolver-4.1.108.Final.jar:
 86 /jars/netty-resolver-dns-4.1.100.Final.jar:                                                    85 /jars/netty-resolver-dns-4.1.108.Final.jar:
 87 /jars/netty-resolver-dns-classes-macos-4.1.100.Final.jar:                                      86 /jars/netty-resolver-dns-classes-macos-4.1.108.Final.jar:
 88 /jars/netty-resolver-dns-native-macos-4.1.100.Final-osx-aarch_64.jar:                          87 /jars/netty-resolver-dns-native-macos-4.1.108.Final-osx-aarch_64.jar:
 89 /jars/netty-resolver-dns-native-macos-4.1.100.Final-osx-x86_64.jar:                            88 /jars/netty-resolver-dns-native-macos-4.1.108.Final-osx-x86_64.jar:
 90 /jars/netty-transport-4.1.100.Final.jar:                                                       89 /jars/netty-transport-4.1.108.Final.jar:
 91 /jars/netty-transport-classes-epoll-4.1.100.Final.jar:                                         90 /jars/netty-transport-classes-epoll-4.1.108.Final.jar:
 92 /jars/netty-transport-classes-kqueue-4.1.100.Final.jar:                                        91 /jars/netty-transport-classes-kqueue-4.1.108.Final.jar:
 93 /jars/netty-transport-native-epoll-4.1.100.Final-linux-aarch_64.jar:                           92 /jars/netty-transport-native-epoll-4.1.108.Final-linux-aarch_64.jar:
                                                                                                   93 /jars/netty-transport-native-epoll-4.1.108.Final-linux-riscv64.jar:
 94 /jars/netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:                             94 /jars/netty-transport-native-epoll-4.1.108.Final-linux-x86_64.jar:
 95 /jars/netty-transport-native-kqueue-4.1.100.Final-osx-aarch_64.jar:                            95 /jars/netty-transport-native-kqueue-4.1.108.Final-osx-aarch_64.jar:
 96 /jars/netty-transport-native-kqueue-4.1.100.Final-osx-x86_64.jar:                              96 /jars/netty-transport-native-kqueue-4.1.108.Final-osx-x86_64.jar:
 97 /jars/netty-transport-native-unix-common-4.1.100.Final.jar:                                    97 /jars/netty-transport-native-unix-common-4.1.108.Final.jar:
 98 /jars/netty-transport-rxtx-4.1.100.Final.jar:                                                  98 /jars/netty-transport-rxtx-4.1.108.Final.jar:
 99 /jars/netty-transport-sctp-4.1.100.Final.jar:                                                  99 /jars/netty-transport-sctp-4.1.108.Final.jar:
100 /jars/netty-transport-udt-4.1.100.Final.jar:                                                  100 /jars/netty-transport-udt-4.1.108.Final.jar:
101 /jars/okhttp-4.12.0.jar:                                                                   =  101 /jars/okhttp-4.12.0.jar:
...
134 /jars/third-party-jackson-core-2.25.19.jar:                                                   134 /jars/third-party-jackson-core-2.25.19.jar:
                                                                                               -+ 135 /jars/trafficCaptureDependencies.jar:
135 /jars/trafficReplayer.jar:                                                                 =  136 /jars/trafficReplayer.jar:
...
139 /jars/zstd-jni-1.5.5-1.jar                                                                    140 /jars/zstd-jni-1.5.5-1.jar

I’m good with that approach. Thanks!

@peternied peternied merged commit cb5e73d into opensearch-project:main May 10, 2024
6 of 7 checks passed
@peternied peternied deleted the centralized-dependencies branch May 17, 2024 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gregschohn gregschohn gregschohn left review comments

@AndreKurait AndreKurait AndreKurait approved these changes

@chelma chelma Awaiting requested review from chelma chelma is a code owner

@kartg kartg Awaiting requested review from kartg

@lewijacn lewijacn Awaiting requested review from lewijacn lewijacn is a code owner

@mikaylathompson mikaylathompson Awaiting requested review from mikaylathompson mikaylathompson is a code owner

@okhasawn okhasawn Awaiting requested review from okhasawn

@sumobrian sumobrian Awaiting requested review from sumobrian sumobrian is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@peternied @AndreKurait @gregschohn