Support for FIPS compliance mode #14912

beanuwave · 2024-07-23T15:34:16Z

Description

FIPS gradle build script is removed.
All BC dependencies are replaces by BCFIPS.
Password matcher inside Identity-Shiro that replies on BC to check if hashed passwords matches with OpenBSDBCrypt, is replaced by password4j implementation.
Adds support for BCFKS format (*.bks) for Key & Truststores.
Refactor parsing private keys with formats EC, PKCS8, PKCS1, DSA, w/wo encryption, w/wo parameters.
FIPS approved-only mode can be configured over opensearch.yml file.
java security file is added to the build.
java policy file is altered to grant neccessary security permissions.

This PR provides FIPS 140-2 support by replacing all BC dependencies with BCFIPS dependencies and making FIPS approved-only mode configurable at launch. Running application in approved-only mode restricts BCFIPS provoder to rely solely on FIPS certified cyphers. Due to replacement of BC libraries, BCrypt password matching and private-key loading from file were replaced by alternative implementations.

Reasons for refactoring PemUtils.java that is used by Reindex API, in case of migrating data from a remote cluster that is TLS protected:

PKCS#8 implementation was not supported by BCFIPS library.
java type security.
Password Based Key Derivation Functions such as PKCS#12 and OpenSSL are not supported in BCFIPS approved-only mode, because only PBKDF2 standard is approved for use in FIPS.
generally good idea to let ASN1 annotation parsing be done by external security libraries.

Related Issues

opensearch-project/security#3420

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-07-23T15:41:17Z

❌ Gradle check result for 6016d5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-07-24T08:51:58Z

❌ Gradle check result for 8e8ed47: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-07-24T08:54:08Z

❌ Gradle check result for 6016d5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

.../identity-shiro/src/main/java/org/opensearch/identity/shiro/realm/BCryptPasswordMatcher.java

dblock · 2024-07-24T15:11:32Z

Could use some help maybe from @cwperks or @peternied reviewing this, please.

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchNode.java

github-actions · 2024-10-24T11:38:53Z

❌ Gradle check result for 90632d0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-24T12:05:38Z

❌ Gradle check result for d5b496e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-24T12:44:45Z

❌ Gradle check result for 52b5345: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-10-24T14:43:39Z

❌ Gradle check result for 89e4b57: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Iwan Igonin <iigonin@sternad.de> # Conflicts: # server/build.gradle

Signed-off-by: Iwan Igonin <iigonin@sternad.de> # Conflicts: # client/rest/build.gradle # distribution/tools/plugin-cli/build.gradle # server/src/main/resources/org/opensearch/bootstrap/test-framework.policy

Signed-off-by: Iwan Igonin <iigonin@sternad.de>

…ional tests. Signed-off-by: Iwan Igonin <iigonin@sternad.de>

Signed-off-by: Iwan Igonin <iigonin@sternad.de>

github-actions · 2024-10-25T12:33:47Z

✅ Gradle check result for 2aa0d1d: SUCCESS

terryquigleysas · 2024-10-25T15:30:05Z

build.gradle

+  // test with FIPS-140-2 enabled
+  plugins.withType(JavaPlugin).configureEach {
+    tasks.withType(Test).configureEach { testTask ->
+      if (System.getenv('OPENSEARCH_CRYPTO_STANDARD') == 'FIPS-140-2') {


With the recent Bouncy Castle 2.x libraries used here it will also be possible to support FIPS-140-3 on Java 21

https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/4743

terryquigleysas · 2024-10-25T15:38:08Z

buildSrc/src/main/java/org/opensearch/gradle/info/GlobalBuildInfoPlugin.java

@@ -112,6 +113,8 @@ public void apply(Project project) {
        BuildParams.init(params -> {
            // Initialize global build parameters
            boolean isInternal = GlobalBuildInfoPlugin.class.getResource("/buildSrc.marker") != null;
+            var cryptoStandard = System.getenv(OPENSEARCH_CRYPTO_STANDARD);
+            var inFipsJvm = cryptoStandard != null && cryptoStandard.equals("FIPS-140-2");


See also https://github.com/opensearch-project/OpenSearch/pull/14912/files#r1816924147

terryquigleysas · 2024-10-25T15:38:20Z

distribution/tools/launchers/src/main/java/org/opensearch/tools/launchers/SystemJvmOptions.java

 import java.util.Arrays;
 import java.util.Collections;
 import java.util.List;
 import java.util.stream.Collectors;

 final class SystemJvmOptions {

-    static List<String> systemJvmOptions() {
+    protected static final String OPENSEARCH_CRYPTO_STANDARD = "OPENSEARCH_CRYPTO_STANDARD";
+    protected static final String FIPS_140_2 = "FIPS-140-2";


See also https://github.com/opensearch-project/OpenSearch/pull/14912/files#r1816924147

terryquigleysas · 2024-10-25T15:38:29Z

plugins/repository-s3/build.gradle

@@ -143,6 +144,13 @@ def fixtureAddress = { fixture, name, port ->
  'http://127.0.0.1:' + ephemeralPort
 }

+def applyFipsConfig(OpenSearchCluster cluster) {
+  if (System.getenv('OPENSEARCH_CRYPTO_STANDARD') == 'FIPS-140-2') {


See also https://github.com/opensearch-project/OpenSearch/pull/14912/files#r1816924147

terryquigleysas · 2024-10-25T15:38:39Z

server/src/main/java/org/opensearch/bootstrap/Bootstrap.java

@@ -195,6 +195,11 @@ private void setup(boolean addShutdownHook, Environment environment) throws Boot
            BootstrapSettings.CTRLHANDLER_SETTING.get(settings)
        );

+        var cryptoStandard = System.getenv("OPENSEARCH_CRYPTO_STANDARD");
+        if (cryptoStandard != null && cryptoStandard.equals("FIPS-140-2")) {


See also https://github.com/opensearch-project/OpenSearch/pull/14912/files#r1816924147

terryquigleysas · 2024-10-25T15:42:16Z

buildSrc/version.properties

@@ -55,7 +55,11 @@ reactivestreams   = 1.0.4
 # when updating this version, you need to ensure compatibility with:
 #  - plugins/ingest-attachment (transitive dependency, check the upstream POM)
 #  - distribution/tools/plugin-cli
-bouncycastle=1.78
+bouncycastle_jce=2.0.0


Is the plan here that there will be two OpenSearch distributions, one to support the current Bouncy Castle libraries, and one containing these FIPS jars?

github-actions · 2024-10-25T16:33:44Z

❌ Gradle check result for 246aeed: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Iwan Igonin <iigonin@sternad.de>

github-actions · 2024-10-25T18:24:45Z

✅ Gradle check result for 7e16ec6: SUCCESS

beanuwave changed the title ~~Draft to allow run in FIPS compliace mode~~ Draft to allow run in FIPS compliance mode Jul 24, 2024

beanuwave force-pushed the fips_compliance2 branch from 8e8ed47 to 6016d5d Compare July 24, 2024 08:46

beanuwave marked this pull request as ready for review July 24, 2024 10:02

beanuwave requested review from peternied, anasalkouz, andrross, ashking94, Bukhtawar, CEHENKLE, dblock, dbwiddis, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami and VachaShah as code owners July 24, 2024 10:02

dancristiancecoi reviewed Jul 24, 2024

View reviewed changes

.../identity-shiro/src/main/java/org/opensearch/identity/shiro/realm/BCryptPasswordMatcher.java Outdated Show resolved Hide resolved

terryquigleysas reviewed Jul 24, 2024

View reviewed changes

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchNode.java Outdated Show resolved Hide resolved

beanuwave force-pushed the fips_compliance2 branch from 3637ccf to 90632d0 Compare October 24, 2024 11:32

beanuwave force-pushed the fips_compliance2 branch from 90632d0 to d5b496e Compare October 24, 2024 11:49

beanuwave force-pushed the fips_compliance2 branch from d5b496e to 52b5345 Compare October 24, 2024 12:31

beanuwave force-pushed the fips_compliance2 branch from 52b5345 to 89e4b57 Compare October 24, 2024 14:18

iigonin added 8 commits October 25, 2024 13:17

Draft to allow run in FIPS compliace mode

a47f4e6

Signed-off-by: Iwan Igonin <iigonin@sternad.de> # Conflicts: # server/build.gradle

make tests run without BC (not BCFIPS) libraries.

0bee0a8

Signed-off-by: Iwan Igonin <iigonin@sternad.de> # Conflicts: # client/rest/build.gradle # distribution/tools/plugin-cli/build.gradle # server/src/main/resources/org/opensearch/bootstrap/test-framework.policy

disable approved-only mode for launch configuration of testcluster

4fc6201

Signed-off-by: Iwan Igonin <iigonin@sternad.de>

update all BC libraries to support JAVA 21

321929f

Signed-off-by: Iwan Igonin <iigonin@sternad.de>

try to pass bwc staged build by adding expected version

7dea462

Signed-off-by: Iwan Igonin <iigonin@sternad.de>

ensure FIPS-mode can be enabled by fixing existent and creating addit…

74011f2

…ional tests. Signed-off-by: Iwan Igonin <iigonin@sternad.de>

fixing additional tests.

58f0400

Signed-off-by: Iwan Igonin <iigonin@sternad.de>

Configure FIPS mode through global ENV VAR instead of thread-based.

b30ebc0

Signed-off-by: Iwan Igonin <iigonin@sternad.de>

beanuwave force-pushed the fips_compliance2 branch from 89e4b57 to 2aa0d1d Compare October 25, 2024 11:18

terryquigleysas reviewed Oct 25, 2024

View reviewed changes

beanuwave force-pushed the fips_compliance2 branch from 2aa0d1d to 246aeed Compare October 25, 2024 15:33

terryquigleysas reviewed Oct 25, 2024

View reviewed changes

write some additional unit tests.

7e16ec6

Signed-off-by: Iwan Igonin <iigonin@sternad.de>

beanuwave force-pushed the fips_compliance2 branch from 246aeed to 7e16ec6 Compare October 25, 2024 17:19

opensearch-ci-bot mentioned this pull request Oct 25, 2024

[AUTOCUT] Gradle Check Flaky Test Report for SpecificClusterManagerNodesIT #15944

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for FIPS compliance mode #14912

Support for FIPS compliance mode #14912

beanuwave commented Jul 23, 2024

github-actions bot commented Jul 23, 2024

github-actions bot commented Jul 24, 2024

github-actions bot commented Jul 24, 2024

dblock commented Jul 24, 2024

github-actions bot commented Oct 24, 2024

github-actions bot commented Oct 24, 2024

github-actions bot commented Oct 24, 2024

github-actions bot commented Oct 24, 2024

github-actions bot commented Oct 25, 2024

terryquigleysas Oct 25, 2024

terryquigleysas Oct 25, 2024

terryquigleysas Oct 25, 2024

terryquigleysas Oct 25, 2024

terryquigleysas Oct 25, 2024

terryquigleysas Oct 25, 2024

github-actions bot commented Oct 25, 2024

github-actions bot commented Oct 25, 2024

Support for FIPS compliance mode #14912

Are you sure you want to change the base?

Support for FIPS compliance mode #14912

Conversation

beanuwave commented Jul 23, 2024

Description

Related Issues

Check List

github-actions bot commented Jul 23, 2024

github-actions bot commented Jul 24, 2024

github-actions bot commented Jul 24, 2024

dblock commented Jul 24, 2024

github-actions bot commented Oct 24, 2024

github-actions bot commented Oct 24, 2024

github-actions bot commented Oct 24, 2024

github-actions bot commented Oct 24, 2024

github-actions bot commented Oct 25, 2024

terryquigleysas Oct 25, 2024

Choose a reason for hiding this comment

terryquigleysas Oct 25, 2024

Choose a reason for hiding this comment

terryquigleysas Oct 25, 2024

Choose a reason for hiding this comment

terryquigleysas Oct 25, 2024

Choose a reason for hiding this comment

terryquigleysas Oct 25, 2024

Choose a reason for hiding this comment

terryquigleysas Oct 25, 2024

Choose a reason for hiding this comment

github-actions bot commented Oct 25, 2024

github-actions bot commented Oct 25, 2024