Skip to content

Commit

Permalink
Merge branch 'main' into listShardsEmptyIndices
Browse files Browse the repository at this point in the history
Signed-off-by: Harsh Garg <gkharsh@amazon.com>
  • Loading branch information
Harsh Garg committed Oct 23, 2024
2 parents b328dbc + 9489a21 commit 016fcaf
Show file tree
Hide file tree
Showing 95 changed files with 5,303 additions and 389 deletions.
12 changes: 9 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Add _list/shards API as paginated alternate to _cat/shards ([#14641](https://github.com/opensearch-project/OpenSearch/pull/14641))
- Latency and Memory allocation improvements to Multi Term Aggregation queries ([#14993](https://github.com/opensearch-project/OpenSearch/pull/14993))
- Flat object field use IndexOrDocValuesQuery to optimize query ([#14383](https://github.com/opensearch-project/OpenSearch/issues/14383))
- Add method to return dynamic SecureTransportParameters from SecureTransportSettingsProvider interface ([#16387](https://github.com/opensearch-project/OpenSearch/pull/16387)
- Add support for renaming aliases during snapshot restore ([#16292](https://github.com/opensearch-project/OpenSearch/pull/16292))
- Add method to return dynamic SecureTransportParameters from SecureTransportSettingsProvider interface ([#16387](https://github.com/opensearch-project/OpenSearch/pull/16387))
- URI path filtering support in cluster stats API ([#15938](https://github.com/opensearch-project/OpenSearch/pull/15938))

### Dependencies
- Bump `com.azure:azure-identity` from 1.13.0 to 1.13.2 ([#15578](https://github.com/opensearch-project/OpenSearch/pull/15578))
- Bump `protobuf` from 3.22.3 to 3.25.4 ([#15684](https://github.com/opensearch-project/OpenSearch/pull/15684))
- Bump `org.apache.logging.log4j:log4j-core` from 2.23.1 to 2.24.1 ([#15858](https://github.com/opensearch-project/OpenSearch/pull/15858), [#16134](https://github.com/opensearch-project/OpenSearch/pull/16134))
- Bump `peter-evans/create-pull-request` from 6 to 7 ([#15863](https://github.com/opensearch-project/OpenSearch/pull/15863))
- Bump `com.nimbusds:oauth2-oidc-sdk` from 11.9.1 to 11.19.1 ([#15862](https://github.com/opensearch-project/OpenSearch/pull/15862))
- Bump `com.microsoft.azure:msal4j` from 1.17.0 to 1.17.1 ([#15945](https://github.com/opensearch-project/OpenSearch/pull/15945))
- Bump `com.microsoft.azure:msal4j` from 1.17.0 to 1.17.2 ([#15945](https://github.com/opensearch-project/OpenSearch/pull/15945), [#16406](https://github.com/opensearch-project/OpenSearch/pull/16406))
- Bump `ch.qos.logback:logback-core` from 1.5.6 to 1.5.10 ([#15946](https://github.com/opensearch-project/OpenSearch/pull/15946), [#16307](https://github.com/opensearch-project/OpenSearch/pull/16307))
- Update protobuf from 3.25.4 to 3.25.5 ([#16011](https://github.com/opensearch-project/OpenSearch/pull/16011))
- Bump `org.roaringbitmap:RoaringBitmap` from 1.2.1 to 1.3.0 ([#16040](https://github.com/opensearch-project/OpenSearch/pull/16040))
Expand Down Expand Up @@ -65,6 +67,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Code cleanup: Remove ApproximateIndexOrDocValuesQuery ([#16273](https://github.com/opensearch-project/OpenSearch/pull/16273))
- Optimise clone operation for incremental full cluster snapshots ([#16296](https://github.com/opensearch-project/OpenSearch/pull/16296))
- Update last seen cluster state in the commit phase ([#16215](https://github.com/opensearch-project/OpenSearch/pull/16215))
- Make multiple settings dynamic for tuning on larger clusters([#16347](https://github.com/opensearch-project/OpenSearch/pull/16347))

### Deprecated

Expand All @@ -85,10 +88,13 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Fix multi-search with template doesn't return status code ([#16265](https://github.com/opensearch-project/OpenSearch/pull/16265))
- [Streaming Indexing] Fix intermittent 'The bulk request must be terminated by a newline [\n]' failures [#16337](https://github.com/opensearch-project/OpenSearch/pull/16337))
- Fix wrong default value when setting `index.number_of_routing_shards` to null on index creation ([#16331](https://github.com/opensearch-project/OpenSearch/pull/16331))
- [Workload Management] Make query groups persistent across process restarts [#16370](https://github.com/opensearch-project/OpenSearch/pull/16370)
- [Workload Management] Make query groups persistent across process restarts ([#16370](https://github.com/opensearch-project/OpenSearch/pull/16370))
- [Workload Management] Enhance rejection mechanism in workload management ([#16417](https://github.com/opensearch-project/OpenSearch/pull/16417))
- Fix inefficient Stream API call chains ending with count() ([#15386](https://github.com/opensearch-project/OpenSearch/pull/15386))
- Fix array hashCode calculation in ResyncReplicationRequest ([#16378](https://github.com/opensearch-project/OpenSearch/pull/16378))
- Fix missing fields in task index mapping to ensure proper task result storage ([#16201](https://github.com/opensearch-project/OpenSearch/pull/16201))
- Fix typo super->sb in method toString() of RemoteStoreNodeAttribute ([#15362](https://github.com/opensearch-project/OpenSearch/pull/15362))
- [Workload Management] Fixing Create/Update QueryGroup TransportActions to execute from non-cluster manager nodes ([16422](https://github.com/opensearch-project/OpenSearch/pull/16422))

### Security

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,9 +77,9 @@
import java.util.stream.Stream;

public class DistroTestPlugin implements Plugin<Project> {
private static final String SYSTEM_JDK_VERSION = "23+37";
private static final String SYSTEM_JDK_VERSION = "23.0.1+11";
private static final String SYSTEM_JDK_VENDOR = "adoptium";
private static final String GRADLE_JDK_VERSION = "23+37";
private static final String GRADLE_JDK_VERSION = "23.0.1+11";
private static final String GRADLE_JDK_VENDOR = "adoptium";

// all distributions used by distro tests. this is temporary until tests are per distribution
Expand Down
2 changes: 1 addition & 1 deletion buildSrc/version.properties
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ opensearch = 3.0.0
lucene = 9.12.0

bundled_jdk_vendor = adoptium
bundled_jdk = 23+37
bundled_jdk = 23.0.1+11

# optional dependencies
spatial4j = 0.7
Expand Down
2 changes: 1 addition & 1 deletion plugins/repository-azure/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ dependencies {
// Start of transitive dependencies for azure-identity
api 'com.microsoft.azure:msal4j-persistence-extension:1.3.0'
api "net.java.dev.jna:jna-platform:${versions.jna}"
api 'com.microsoft.azure:msal4j:1.17.1'
api 'com.microsoft.azure:msal4j:1.17.2'
api 'com.nimbusds:oauth2-oidc-sdk:11.19.1'
api 'com.nimbusds:nimbus-jose-jwt:9.41.1'
api 'com.nimbusds:content-type:2.3'
Expand Down
1 change: 0 additions & 1 deletion plugins/repository-azure/licenses/msal4j-1.17.1.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions plugins/repository-azure/licenses/msal4j-1.17.2.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
a6211e3d71d0388929babaa0ff0951b30d001852
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@

package org.opensearch.plugin.wlm.action;

import org.opensearch.action.ActionRequest;
import org.opensearch.action.ActionRequestValidationException;
import org.opensearch.action.support.clustermanager.ClusterManagerNodeRequest;
import org.opensearch.cluster.metadata.QueryGroup;
import org.opensearch.common.UUIDs;
import org.opensearch.core.common.io.stream.StreamInput;
Expand All @@ -33,7 +33,7 @@
*
* @opensearch.experimental
*/
public class CreateQueryGroupRequest extends ActionRequest {
public class CreateQueryGroupRequest extends ClusterManagerNodeRequest<CreateQueryGroupRequest> {
private final QueryGroup queryGroup;

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,43 +9,82 @@
package org.opensearch.plugin.wlm.action;

import org.opensearch.action.support.ActionFilters;
import org.opensearch.action.support.HandledTransportAction;
import org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction;
import org.opensearch.cluster.ClusterState;
import org.opensearch.cluster.block.ClusterBlockException;
import org.opensearch.cluster.block.ClusterBlockLevel;
import org.opensearch.cluster.metadata.IndexNameExpressionResolver;
import org.opensearch.common.inject.Inject;
import org.opensearch.core.action.ActionListener;
import org.opensearch.core.common.io.stream.StreamInput;
import org.opensearch.plugin.wlm.service.QueryGroupPersistenceService;
import org.opensearch.tasks.Task;
import org.opensearch.threadpool.ThreadPool;
import org.opensearch.transport.TransportService;

import java.io.IOException;

import static org.opensearch.threadpool.ThreadPool.Names.SAME;

/**
* Transport action to create QueryGroup
*
* @opensearch.experimental
*/
public class TransportCreateQueryGroupAction extends HandledTransportAction<CreateQueryGroupRequest, CreateQueryGroupResponse> {
public class TransportCreateQueryGroupAction extends TransportClusterManagerNodeAction<CreateQueryGroupRequest, CreateQueryGroupResponse> {

private final QueryGroupPersistenceService queryGroupPersistenceService;

/**
* Constructor for TransportCreateQueryGroupAction
*
* @param actionName - action name
* @param threadPool - {@link ThreadPool} object
* @param transportService - a {@link TransportService} object
* @param actionFilters - a {@link ActionFilters} object
* @param indexNameExpressionResolver - {@link IndexNameExpressionResolver} object
* @param queryGroupPersistenceService - a {@link QueryGroupPersistenceService} object
*/
@Inject
public TransportCreateQueryGroupAction(
String actionName,
ThreadPool threadPool,
TransportService transportService,
ActionFilters actionFilters,
IndexNameExpressionResolver indexNameExpressionResolver,
QueryGroupPersistenceService queryGroupPersistenceService
) {
super(CreateQueryGroupAction.NAME, transportService, actionFilters, CreateQueryGroupRequest::new);
super(
CreateQueryGroupAction.NAME,
transportService,
queryGroupPersistenceService.getClusterService(),
threadPool,
actionFilters,
CreateQueryGroupRequest::new,
indexNameExpressionResolver
);
this.queryGroupPersistenceService = queryGroupPersistenceService;
}

@Override
protected void doExecute(Task task, CreateQueryGroupRequest request, ActionListener<CreateQueryGroupResponse> listener) {
protected void clusterManagerOperation(
CreateQueryGroupRequest request,
ClusterState clusterState,
ActionListener<CreateQueryGroupResponse> listener
) {
queryGroupPersistenceService.persistInClusterStateMetadata(request.getQueryGroup(), listener);
}

@Override
protected String executor() {
return SAME;
}

@Override
protected CreateQueryGroupResponse read(StreamInput in) throws IOException {
return new CreateQueryGroupResponse(in);
}

@Override
protected ClusterBlockException checkBlock(CreateQueryGroupRequest request, ClusterState state) {
return state.blocks().globalBlockedException(ClusterBlockLevel.METADATA_WRITE);
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -9,43 +9,81 @@
package org.opensearch.plugin.wlm.action;

import org.opensearch.action.support.ActionFilters;
import org.opensearch.action.support.HandledTransportAction;
import org.opensearch.action.support.clustermanager.TransportClusterManagerNodeAction;
import org.opensearch.cluster.ClusterState;
import org.opensearch.cluster.block.ClusterBlockException;
import org.opensearch.cluster.block.ClusterBlockLevel;
import org.opensearch.cluster.metadata.IndexNameExpressionResolver;
import org.opensearch.common.inject.Inject;
import org.opensearch.core.action.ActionListener;
import org.opensearch.core.common.io.stream.StreamInput;
import org.opensearch.plugin.wlm.service.QueryGroupPersistenceService;
import org.opensearch.tasks.Task;
import org.opensearch.threadpool.ThreadPool;
import org.opensearch.transport.TransportService;

import java.io.IOException;

import static org.opensearch.threadpool.ThreadPool.Names.SAME;

/**
* Transport action to update QueryGroup
*
* @opensearch.experimental
*/
public class TransportUpdateQueryGroupAction extends HandledTransportAction<UpdateQueryGroupRequest, UpdateQueryGroupResponse> {
public class TransportUpdateQueryGroupAction extends TransportClusterManagerNodeAction<UpdateQueryGroupRequest, UpdateQueryGroupResponse> {

private final QueryGroupPersistenceService queryGroupPersistenceService;

/**
* Constructor for TransportUpdateQueryGroupAction
*
* @param actionName - action name
* @param threadPool - {@link ThreadPool} object
* @param transportService - a {@link TransportService} object
* @param actionFilters - a {@link ActionFilters} object
* @param indexNameExpressionResolver - {@link IndexNameExpressionResolver} object
* @param queryGroupPersistenceService - a {@link QueryGroupPersistenceService} object
*/
@Inject
public TransportUpdateQueryGroupAction(
String actionName,
ThreadPool threadPool,
TransportService transportService,
ActionFilters actionFilters,
IndexNameExpressionResolver indexNameExpressionResolver,
QueryGroupPersistenceService queryGroupPersistenceService
) {
super(UpdateQueryGroupAction.NAME, transportService, actionFilters, UpdateQueryGroupRequest::new);
super(
UpdateQueryGroupAction.NAME,
transportService,
queryGroupPersistenceService.getClusterService(),
threadPool,
actionFilters,
UpdateQueryGroupRequest::new,
indexNameExpressionResolver
);
this.queryGroupPersistenceService = queryGroupPersistenceService;
}

@Override
protected void doExecute(Task task, UpdateQueryGroupRequest request, ActionListener<UpdateQueryGroupResponse> listener) {
protected void clusterManagerOperation(
UpdateQueryGroupRequest request,
ClusterState clusterState,
ActionListener<UpdateQueryGroupResponse> listener
) {
queryGroupPersistenceService.updateInClusterStateMetadata(request, listener);
}

@Override
protected String executor() {
return SAME;
}

@Override
protected UpdateQueryGroupResponse read(StreamInput in) throws IOException {
return new UpdateQueryGroupResponse(in);
}

@Override
protected ClusterBlockException checkBlock(UpdateQueryGroupRequest request, ClusterState state) {
return state.blocks().globalBlockedException(ClusterBlockLevel.METADATA_WRITE);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@

package org.opensearch.plugin.wlm.action;

import org.opensearch.action.ActionRequest;
import org.opensearch.action.ActionRequestValidationException;
import org.opensearch.action.support.clustermanager.ClusterManagerNodeRequest;
import org.opensearch.cluster.metadata.QueryGroup;
import org.opensearch.core.common.io.stream.StreamInput;
import org.opensearch.core.common.io.stream.StreamOutput;
Expand All @@ -23,7 +23,7 @@
*
* @opensearch.experimental
*/
public class UpdateQueryGroupRequest extends ActionRequest {
public class UpdateQueryGroupRequest extends ClusterManagerNodeRequest<UpdateQueryGroupRequest> {
private final String name;
private final MutableQueryGroupFragment mutableQueryGroupFragment;

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.upgrades;

import org.opensearch.Version;
import org.opensearch.client.Request;
import org.opensearch.client.Response;

import java.io.IOException;
import java.util.Collections;
import java.util.List;
import java.util.Map;

public class ClusterStatsIT extends AbstractRollingTestCase {

private final List<String> nodeStatsMetrics = List.of("os", "process", "jvm", "fs", "plugins", "ingest", "network_types", "discovery_types", "packaging_types");

private final List<String> indicesStatsMetrics = List.of("shards", "docs", "store", "fielddata", "query_cache", "completion", "segments", "analysis", "mappings");

public void testClusterStats() throws IOException {
Response response = client().performRequest(new Request("GET", "/_cluster/stats"));
validateClusterStatsWithFilterResponse(response, nodeStatsMetrics, indicesStatsMetrics);
if (AbstractRollingTestCase.UPGRADE_FROM_VERSION.onOrAfter(Version.V_3_0_0) || (
CLUSTER_TYPE == ClusterType.UPGRADED && Version.CURRENT.onOrAfter(Version.V_3_0_0))) {
response = client().performRequest(new Request("GET", "/_cluster/stats/os/nodes/_all"));
validateClusterStatsWithFilterResponse(response, List.of("os"), Collections.emptyList());
response = client().performRequest(new Request("GET", "/_cluster/stats/indices/mappings/nodes/_all"));
validateClusterStatsWithFilterResponse(response, Collections.emptyList(), List.of("mappings"));
response = client().performRequest(new Request("GET", "/_cluster/stats/os,indices/mappings/nodes/_all"));
validateClusterStatsWithFilterResponse(response, List.of("os"), List.of("mappings"));
}
}

private void validateClusterStatsWithFilterResponse(Response response, List<String> requestedNodesStatsMetrics, List<String> requestedIndicesStatsMetrics) throws IOException {
assertEquals(200, response.getStatusLine().getStatusCode());
Map<String, Object> entity = entityAsMap(response);
if (requestedNodesStatsMetrics != null && !requestedNodesStatsMetrics.isEmpty()) {
assertTrue(entity.containsKey("nodes"));
Map<?, ?> nodesStats = (Map<?, ?>) entity.get("nodes");
for (String metric : nodeStatsMetrics) {
if (requestedNodesStatsMetrics.contains(metric)) {
assertTrue(nodesStats.containsKey(metric));
} else {
assertFalse(nodesStats.containsKey(metric));
}
}
}

if (requestedIndicesStatsMetrics != null && !requestedIndicesStatsMetrics.isEmpty()) {
assertTrue(entity.containsKey("indices"));
Map<?, ?> indicesStats = (Map<?, ?>) entity.get("indices");
for (String metric : indicesStatsMetrics) {
if (requestedIndicesStatsMetrics.contains(metric)) {
assertTrue(indicesStats.containsKey(metric));
} else {
assertFalse(indicesStats.containsKey(metric));
}
}
}
}
}
Loading

0 comments on commit 016fcaf

Please sign in to comment.