From 96dc030c24d44e880f2a1a94cca7929a18562879 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS Date: Thu, 14 Sep 2023 22:42:23 -0500 Subject: [PATCH 01/11] Add segment replication + remote store GA information Signed-off-by: Naarcha-AWS --- .../remote-store/index.md | 2 -- .../remote-segment-backpressure.md | 3 --- .../remote-store/remote-store-stats-api.md | 3 --- .../segment-replication/index.md | 24 ++++++++++++------- 4 files changed, 16 insertions(+), 16 deletions(-) diff --git a/_tuning-your-cluster/availability-and-recovery/remote-store/index.md b/_tuning-your-cluster/availability-and-recovery/remote-store/index.md index aa5eb1f14b..d6fde998af 100644 --- a/_tuning-your-cluster/availability-and-recovery/remote-store/index.md +++ b/_tuning-your-cluster/availability-and-recovery/remote-store/index.md @@ -11,8 +11,6 @@ redirect_from: # Remote-backed storage -This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/1968). -{: .warning} Remote-backed storage offers OpenSearch users a new way to protect against data loss by automatically creating backups of all index transactions and sending them to remote storage. In order to expose this feature, segment replication must also be enabled. See [Segment replication]({{site.url}}{{site.baseurl}}/opensearch/segment-replication/) for additional information. diff --git a/_tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure.md b/_tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure.md index 76844c21e1..18f8444a74 100644 --- a/_tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure.md +++ b/_tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure.md @@ -8,9 +8,6 @@ grand_parent: Availability and recovery # Remote segment backpressure -This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/1968). -{: .warning} - Remote segment backpressure is a shard-level rejection mechanism that dynamically rejects indexing requests when the remote segment store falls behind the local committed segments on the primary shard. With remote segment backpressure, you can prevent the lag between the remote store and the local primary store. The lag can be caused by slow or failed remote store interaction, remote store throttling, long garbage collection pauses, or high CPU utilization. ## Thresholds diff --git a/_tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api.md b/_tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api.md index b5b8765738..ffd9702ce7 100644 --- a/_tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api.md +++ b/_tuning-your-cluster/availability-and-recovery/remote-store/remote-store-stats-api.md @@ -10,9 +10,6 @@ grand_parent: Availability and recovery Introduced 2.8 {: .label .label-purple } -This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/1968). -{: .warning} - Use the Remote Store Stats API to monitor shard-level remote store performance. ## Path and HTTP methods diff --git a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md index 463190a29a..b300407e63 100644 --- a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md +++ b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md @@ -28,6 +28,13 @@ Segment replication can be applied in a variety of scenarios, including: - When experiencing very high loads, you want to add new nodes but don't want to index all data immediately. - OpenSearch cluster deployments with low replica counts, such as those used for log analytics. +## Remote store + +As of OpenSearch 2.10, segment replication now supports two methods for storing replica shards: + +- Using a **Remote Store**, a remote storage solution where you can store shards, your primary replica uploads segments to the remote store. Then, you can down the replica shards from that store. For more information about using a remote store, see [Remote backend storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#segment-replication-and-remote-backed-storage) +- Without a remote store, where replica shards will directly sync segments from the primary shard using node-to-node communication. + ## Segment replication configuration Setting the default replication type for a cluster affects all newly created indexes. You can, however, specify a different replication type when creating an index. Index-level settings override cluster-level settings. @@ -48,7 +55,9 @@ PUT /my-index1 ``` {% include copy-curl.html %} -In segment replication, the primary shard is usually generating more network traffic than the replicas because it copies segment files to the replicas. Thus, it's beneficial to distribute primary shards equally between the nodes. To ensure balanced primary shard distribution, set the dynamic `cluster.routing.allocation.balance.prefer_primary` setting to `true`. For more information, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/). +If you're using a remote store, add the `remote_store` property to the index request body. For more information, see [Create an index]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#create-an-index). + +When using segment replication without a remote store, the primary shard is usually generating more network traffic than the replicas because it copies segment files to the replicas. Thus, it's beneficial to distribute primary shards equally between the nodes. To ensure balanced primary shard distribution, set the dynamic `cluster.routing.allocation.balance.prefer_primary` setting to `true`. For more information, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/). For the best performance, it is recommended that you enable the following settings: @@ -75,8 +84,7 @@ cluster.indices.replication.strategy: 'SEGMENT' ``` {% include copy.html %} -This cluster-level setting cannot be enabled through the [REST API]({{site.url}}{{site.baseurl}}/api-reference/index/). This setting is not applied to system indexes and hidden indexes. By default, all system and hidden indexes in OpenSearch use document replication, even if this setting is enabled. -{: .note} + ### Creating an index with document replication @@ -101,11 +109,11 @@ When using segment replication, consider the following: 1. Enabling segment replication for an existing index requires [reindexing](https://github.com/opensearch-project/OpenSearch/issues/3685). 1. [Cross-cluster replication](https://github.com/opensearch-project/OpenSearch/issues/4090) does not currently use segment replication to copy between clusters. 1. Segment replication is not compatible with [document-level monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/api/#document-level-monitors), which are used with the [Alerting]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/) and [Security Analytics]({{site.url}}{{site.baseurl}}/security-analytics/index/) plugins. The plugins also use the latest available data on replica shards when using the `immediate` refresh policy, and segment replication can delay the policy's availability, resulting in stale replica shards. -1. Segment replication leads to increased network congestion on primary shards. See [Issue - Optimize network bandwidth on primary shards](https://github.com/opensearch-project/OpenSearch/issues/4245). -1. Integration with remote-backed storage as the source of replication is [currently not supported](https://github.com/opensearch-project/OpenSearch/issues/4448). -1. Read-after-write guarantees: Segment replication does not currently support setting the refresh policy to `wait_for`. If you set the `refresh` query parameter to `wait_for` and then ingest documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. We are exploring other mechanisms for providing read-after-write guarantees. For more information, see the corresponding [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/6046). -1. System indexes will continue to use document replication internally until read-after-write guarantees are available. In this case, document replication does not hinder the overall performance because there are few system indexes. - +1. Segment replication leads to increased network congestion on primary shards when not using a remote store. With a remote store, the primary shard can upload segments to the remote store, then you can down replicas from the same store. +1. Read-after-write (RAW) guarantees: As of OpenSearch 2.10, RAW guarantees are supported when using [`get`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/get-documents/) or [`mget`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/multi-get/) API operations, or through searching with `_primary`. +1. As of OpenSearch 2.10, system indexes are now supported inside segment replication. +1. Get, MultiGet, TermVector, and MultiTermVector requests serve strong reads by routing requests to the primary shards. In a read heavy cluster, we recommend setting the `realtime` parameter in these requests to `false`, especially with listed request types. + ## Benchmarks During initial benchmarks, segment replication users reported 40% higher throughput than when using document replication with the same cluster setup. From 779369c767874ad3fa56808b4d9eaf467ebf7c20 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS Date: Thu, 14 Sep 2023 22:46:42 -0500 Subject: [PATCH 02/11] Add introduction labels Signed-off-by: Naarcha-AWS --- .../availability-and-recovery/remote-store/index.md | 3 +++ .../remote-store/remote-segment-backpressure.md | 3 +++ 2 files changed, 6 insertions(+) diff --git a/_tuning-your-cluster/availability-and-recovery/remote-store/index.md b/_tuning-your-cluster/availability-and-recovery/remote-store/index.md index d6fde998af..6066dec238 100644 --- a/_tuning-your-cluster/availability-and-recovery/remote-store/index.md +++ b/_tuning-your-cluster/availability-and-recovery/remote-store/index.md @@ -11,6 +11,9 @@ redirect_from: # Remote-backed storage +Introduced 2.10 +{: .label .label-purple } + Remote-backed storage offers OpenSearch users a new way to protect against data loss by automatically creating backups of all index transactions and sending them to remote storage. In order to expose this feature, segment replication must also be enabled. See [Segment replication]({{site.url}}{{site.baseurl}}/opensearch/segment-replication/) for additional information. diff --git a/_tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure.md b/_tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure.md index 18f8444a74..410545c2d2 100644 --- a/_tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure.md +++ b/_tuning-your-cluster/availability-and-recovery/remote-store/remote-segment-backpressure.md @@ -8,6 +8,9 @@ grand_parent: Availability and recovery # Remote segment backpressure +Introduced 2.10 +{: .label .label-purple } + Remote segment backpressure is a shard-level rejection mechanism that dynamically rejects indexing requests when the remote segment store falls behind the local committed segments on the primary shard. With remote segment backpressure, you can prevent the lag between the remote store and the local primary store. The lag can be caused by slow or failed remote store interaction, remote store throttling, long garbage collection pauses, or high CPU utilization. ## Thresholds From 591ba8fd572e3f9251880f2d2a7d219fc0978a92 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Fri, 15 Sep 2023 12:27:14 -0500 Subject: [PATCH 03/11] Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../availability-and-recovery/segment-replication/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md index b300407e63..7b2b15b8f3 100644 --- a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md +++ b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md @@ -30,9 +30,9 @@ Segment replication can be applied in a variety of scenarios, including: ## Remote store -As of OpenSearch 2.10, segment replication now supports two methods for storing replica shards: +As of OpenSearch 2.10, you can use two methods for segment replication: -- Using a **Remote Store**, a remote storage solution where you can store shards, your primary replica uploads segments to the remote store. Then, you can down the replica shards from that store. For more information about using a remote store, see [Remote backend storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#segment-replication-and-remote-backed-storage) +- Using a **Remote Store**, a remote storage solution where you can store shards, your primary replica uploads segments to the remote store. Then you can download the replica shards from that store. For more information about using a remote store, see [Remote backend storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#segment-replication-and-remote-backed-storage) - Without a remote store, where replica shards will directly sync segments from the primary shard using node-to-node communication. ## Segment replication configuration @@ -109,7 +109,7 @@ When using segment replication, consider the following: 1. Enabling segment replication for an existing index requires [reindexing](https://github.com/opensearch-project/OpenSearch/issues/3685). 1. [Cross-cluster replication](https://github.com/opensearch-project/OpenSearch/issues/4090) does not currently use segment replication to copy between clusters. 1. Segment replication is not compatible with [document-level monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/api/#document-level-monitors), which are used with the [Alerting]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/) and [Security Analytics]({{site.url}}{{site.baseurl}}/security-analytics/index/) plugins. The plugins also use the latest available data on replica shards when using the `immediate` refresh policy, and segment replication can delay the policy's availability, resulting in stale replica shards. -1. Segment replication leads to increased network congestion on primary shards when not using a remote store. With a remote store, the primary shard can upload segments to the remote store, then you can down replicas from the same store. +1. Segment replication leads to increased network congestion on primary shards using node-to-node replication. With a remote store, the primary shard can upload segments to the remote store, then you can download replicas from the same store. 1. Read-after-write (RAW) guarantees: As of OpenSearch 2.10, RAW guarantees are supported when using [`get`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/get-documents/) or [`mget`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/multi-get/) API operations, or through searching with `_primary`. 1. As of OpenSearch 2.10, system indexes are now supported inside segment replication. 1. Get, MultiGet, TermVector, and MultiTermVector requests serve strong reads by routing requests to the primary shards. In a read heavy cluster, we recommend setting the `realtime` parameter in these requests to `false`, especially with listed request types. From b34a484332d8599189dfe9781662df55a1b49c4c Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Fri, 15 Sep 2023 12:27:48 -0500 Subject: [PATCH 04/11] Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../availability-and-recovery/segment-replication/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md index 7b2b15b8f3..d4ca0e63e5 100644 --- a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md +++ b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md @@ -57,7 +57,7 @@ PUT /my-index1 If you're using a remote store, add the `remote_store` property to the index request body. For more information, see [Create an index]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#create-an-index). -When using segment replication without a remote store, the primary shard is usually generating more network traffic than the replicas because it copies segment files to the replicas. Thus, it's beneficial to distribute primary shards equally between the nodes. To ensure balanced primary shard distribution, set the dynamic `cluster.routing.allocation.balance.prefer_primary` setting to `true`. For more information, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/). +When using node-to-node replication, the primary shard is usually generating more network traffic than the replicas because it copies segment files to the replicas. Thus, it's beneficial to distribute primary shards equally between the nodes. To ensure balanced primary shard distribution, set the dynamic `cluster.routing.allocation.balance.prefer_primary` setting to `true`. For more information, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/). For the best performance, it is recommended that you enable the following settings: From 9b668fb933f457af1f4c1c11ef256609a718209e Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Fri, 15 Sep 2023 12:38:39 -0500 Subject: [PATCH 05/11] Update _tuning-your-cluster/availability-and-recovery/segment-replication/index.md Co-authored-by: Suraj Singh Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../availability-and-recovery/segment-replication/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md index d4ca0e63e5..4fb08e0dcd 100644 --- a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md +++ b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md @@ -110,7 +110,7 @@ When using segment replication, consider the following: 1. [Cross-cluster replication](https://github.com/opensearch-project/OpenSearch/issues/4090) does not currently use segment replication to copy between clusters. 1. Segment replication is not compatible with [document-level monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/api/#document-level-monitors), which are used with the [Alerting]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/) and [Security Analytics]({{site.url}}{{site.baseurl}}/security-analytics/index/) plugins. The plugins also use the latest available data on replica shards when using the `immediate` refresh policy, and segment replication can delay the policy's availability, resulting in stale replica shards. 1. Segment replication leads to increased network congestion on primary shards using node-to-node replication. With a remote store, the primary shard can upload segments to the remote store, then you can download replicas from the same store. -1. Read-after-write (RAW) guarantees: As of OpenSearch 2.10, RAW guarantees are supported when using [`get`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/get-documents/) or [`mget`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/multi-get/) API operations, or through searching with `_primary`. +Read-after-write guarantees: Segment replication does not currently support setting the refresh policy to `wait_for`. If you set the `refresh` query parameter to `wait_for` and then ingest documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. If realtime reads are needed, please consider using [`get`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/get-documents/) or [`mget`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/multi-get/) API operations, or through searching with `_primary` preference (documentation link?) 1. As of OpenSearch 2.10, system indexes are now supported inside segment replication. 1. Get, MultiGet, TermVector, and MultiTermVector requests serve strong reads by routing requests to the primary shards. In a read heavy cluster, we recommend setting the `realtime` parameter in these requests to `false`, especially with listed request types. From c5291f7b3c93d8709563f82032f0daadcf6dca17 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Mon, 18 Sep 2023 12:16:30 -0500 Subject: [PATCH 06/11] Apply suggestions from code review Co-authored-by: Bukhtawar Khan Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../segment-replication/index.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md index 4fb08e0dcd..1267f466d7 100644 --- a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md +++ b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md @@ -32,8 +32,8 @@ Segment replication can be applied in a variety of scenarios, including: As of OpenSearch 2.10, you can use two methods for segment replication: -- Using a **Remote Store**, a remote storage solution where you can store shards, your primary replica uploads segments to the remote store. Then you can download the replica shards from that store. For more information about using a remote store, see [Remote backend storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#segment-replication-and-remote-backed-storage) -- Without a remote store, where replica shards will directly sync segments from the primary shard using node-to-node communication. +- Using with a **Remote Store**, a persistent storage solution where you can store data more durably, the primary shard mirrors segments to the remote store and the replica shard hydrates the copy from the same store. For more information about using a remote store, see [Remote backend storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#segment-replication-and-remote-backed-storage) +- Without a remote store, where replica shards will instead sync segments from the primary shard using node-to-node communication. ## Segment replication configuration @@ -57,7 +57,7 @@ PUT /my-index1 If you're using a remote store, add the `remote_store` property to the index request body. For more information, see [Create an index]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#create-an-index). -When using node-to-node replication, the primary shard is usually generating more network traffic than the replicas because it copies segment files to the replicas. Thus, it's beneficial to distribute primary shards equally between the nodes. To ensure balanced primary shard distribution, set the dynamic `cluster.routing.allocation.balance.prefer_primary` setting to `true`. For more information, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/). +When using node-to-node replication, the primary shard consumes higher network bandwidth because it pushes segment files to all the replica shards. Thus, it's beneficial to distribute primary shards equally between the nodes. To ensure balanced primary shard distribution, set the dynamic `cluster.routing.allocation.balance.prefer_primary` setting to `true`. For more information, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/). For the best performance, it is recommended that you enable the following settings: @@ -110,9 +110,9 @@ When using segment replication, consider the following: 1. [Cross-cluster replication](https://github.com/opensearch-project/OpenSearch/issues/4090) does not currently use segment replication to copy between clusters. 1. Segment replication is not compatible with [document-level monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/api/#document-level-monitors), which are used with the [Alerting]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/) and [Security Analytics]({{site.url}}{{site.baseurl}}/security-analytics/index/) plugins. The plugins also use the latest available data on replica shards when using the `immediate` refresh policy, and segment replication can delay the policy's availability, resulting in stale replica shards. 1. Segment replication leads to increased network congestion on primary shards using node-to-node replication. With a remote store, the primary shard can upload segments to the remote store, then you can download replicas from the same store. -Read-after-write guarantees: Segment replication does not currently support setting the refresh policy to `wait_for`. If you set the `refresh` query parameter to `wait_for` and then ingest documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. If realtime reads are needed, please consider using [`get`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/get-documents/) or [`mget`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/multi-get/) API operations, or through searching with `_primary` preference (documentation link?) +Read-after-write guarantees: Segment replication does not currently support setting the refresh policy to `wait_for`. If you set the `refresh` query parameter to `wait_for` and then ingest documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. If realtime reads are needed, consider using [`get`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/get-documents/) or [`mget`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/multi-get/) API operations. 1. As of OpenSearch 2.10, system indexes are now supported inside segment replication. -1. Get, MultiGet, TermVector, and MultiTermVector requests serve strong reads by routing requests to the primary shards. In a read heavy cluster, we recommend setting the `realtime` parameter in these requests to `false`, especially with listed request types. +1. Get, MultiGet, TermVector, and MultiTermVector requests serve strong reads by routing requests to the primary shards. This can hurt performance since more requests are handled by the primary shards versus distributing requests across primary and replicate shards. To help with performance in read heavy clusters, we recommend setting the `realtime` parameter in these requests to `false`, especially with listed request types. For more information, see [Issue #8700](https://github.com/opensearch-project/OpenSearch/issues/8700). ## Benchmarks From 92133af7e1e053e8888e00ea67c0f4eb8a6857bd Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Mon, 18 Sep 2023 12:28:37 -0500 Subject: [PATCH 07/11] Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../availability-and-recovery/segment-replication/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md index 1267f466d7..ac689006a6 100644 --- a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md +++ b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md @@ -32,7 +32,7 @@ Segment replication can be applied in a variety of scenarios, including: As of OpenSearch 2.10, you can use two methods for segment replication: -- Using with a **Remote Store**, a persistent storage solution where you can store data more durably, the primary shard mirrors segments to the remote store and the replica shard hydrates the copy from the same store. For more information about using a remote store, see [Remote backend storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#segment-replication-and-remote-backed-storage) +- Using with a **Remote Store**, a persistent storage solution where you can store data more durably, the primary shard mirrors segments to the remote store and the replica shard hydrates the copy from the same store. For more information about using a remote store, see [Remote backend storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#segment-replication-and-remote-backed-storage). - Without a remote store, where replica shards will instead sync segments from the primary shard using node-to-node communication. ## Segment replication configuration @@ -112,7 +112,7 @@ When using segment replication, consider the following: 1. Segment replication leads to increased network congestion on primary shards using node-to-node replication. With a remote store, the primary shard can upload segments to the remote store, then you can download replicas from the same store. Read-after-write guarantees: Segment replication does not currently support setting the refresh policy to `wait_for`. If you set the `refresh` query parameter to `wait_for` and then ingest documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. If realtime reads are needed, consider using [`get`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/get-documents/) or [`mget`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/multi-get/) API operations. 1. As of OpenSearch 2.10, system indexes are now supported inside segment replication. -1. Get, MultiGet, TermVector, and MultiTermVector requests serve strong reads by routing requests to the primary shards. This can hurt performance since more requests are handled by the primary shards versus distributing requests across primary and replicate shards. To help with performance in read heavy clusters, we recommend setting the `realtime` parameter in these requests to `false`, especially with listed request types. For more information, see [Issue #8700](https://github.com/opensearch-project/OpenSearch/issues/8700). +1. Get, MultiGet, TermVector, and MultiTermVector requests serve strong reads by routing requests to the primary shards. This can hurt performance since more requests are handled by the primary shards compared to distributing requests across primary and replicate shards. To help with performance in read heavy clusters, we recommend setting the `realtime` parameter in these requests to `false`, especially with listed request types. For more information, see [Issue #8700](https://github.com/opensearch-project/OpenSearch/issues/8700). ## Benchmarks From a2e332310a40d3205a9da3cd5188902af7f52050 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Mon, 18 Sep 2023 14:38:14 -0500 Subject: [PATCH 08/11] Update index.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../availability-and-recovery/segment-replication/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md index ac689006a6..e429d583bc 100644 --- a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md +++ b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md @@ -109,7 +109,7 @@ When using segment replication, consider the following: 1. Enabling segment replication for an existing index requires [reindexing](https://github.com/opensearch-project/OpenSearch/issues/3685). 1. [Cross-cluster replication](https://github.com/opensearch-project/OpenSearch/issues/4090) does not currently use segment replication to copy between clusters. 1. Segment replication is not compatible with [document-level monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/api/#document-level-monitors), which are used with the [Alerting]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/) and [Security Analytics]({{site.url}}{{site.baseurl}}/security-analytics/index/) plugins. The plugins also use the latest available data on replica shards when using the `immediate` refresh policy, and segment replication can delay the policy's availability, resulting in stale replica shards. -1. Segment replication leads to increased network congestion on primary shards using node-to-node replication. With a remote store, the primary shard can upload segments to the remote store, then you can download replicas from the same store. +1. Segment replication leads to increased network congestion on primary shards using node-to-node replication, because replica shards will fetch updates from the primary shard. With a remote store, the primary shard can upload segments to the remote store, download replicas from the same store, and fetch updates from the remote store. This helps offload responsbilities from the primary shard to the remote store. Read-after-write guarantees: Segment replication does not currently support setting the refresh policy to `wait_for`. If you set the `refresh` query parameter to `wait_for` and then ingest documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. If realtime reads are needed, consider using [`get`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/get-documents/) or [`mget`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/multi-get/) API operations. 1. As of OpenSearch 2.10, system indexes are now supported inside segment replication. 1. Get, MultiGet, TermVector, and MultiTermVector requests serve strong reads by routing requests to the primary shards. This can hurt performance since more requests are handled by the primary shards compared to distributing requests across primary and replicate shards. To help with performance in read heavy clusters, we recommend setting the `realtime` parameter in these requests to `false`, especially with listed request types. For more information, see [Issue #8700](https://github.com/opensearch-project/OpenSearch/issues/8700). From 0fc62f0c9a796c865281f39ebc01d0450a06384d Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 19 Sep 2023 10:24:25 -0500 Subject: [PATCH 09/11] Apply suggestions from code review Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../segment-replication/index.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md index e429d583bc..fd692f891a 100644 --- a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md +++ b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md @@ -32,8 +32,8 @@ Segment replication can be applied in a variety of scenarios, including: As of OpenSearch 2.10, you can use two methods for segment replication: -- Using with a **Remote Store**, a persistent storage solution where you can store data more durably, the primary shard mirrors segments to the remote store and the replica shard hydrates the copy from the same store. For more information about using a remote store, see [Remote backend storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#segment-replication-and-remote-backed-storage). -- Without a remote store, where replica shards will instead sync segments from the primary shard using node-to-node communication. +- With **remote-backed storage**, a persistent storage solution: The primary shard sends segment files to the remote store and the replica shards source the copy from the same store. For more information about using a remote store, see [Remote-backed storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#segment-replication-and-remote-backed-storage). +- Without remote-backed storage: The primary shard sends segment files directly to the replica shards using node-to-node communication. ## Segment replication configuration @@ -109,10 +109,10 @@ When using segment replication, consider the following: 1. Enabling segment replication for an existing index requires [reindexing](https://github.com/opensearch-project/OpenSearch/issues/3685). 1. [Cross-cluster replication](https://github.com/opensearch-project/OpenSearch/issues/4090) does not currently use segment replication to copy between clusters. 1. Segment replication is not compatible with [document-level monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/api/#document-level-monitors), which are used with the [Alerting]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/) and [Security Analytics]({{site.url}}{{site.baseurl}}/security-analytics/index/) plugins. The plugins also use the latest available data on replica shards when using the `immediate` refresh policy, and segment replication can delay the policy's availability, resulting in stale replica shards. -1. Segment replication leads to increased network congestion on primary shards using node-to-node replication, because replica shards will fetch updates from the primary shard. With a remote store, the primary shard can upload segments to the remote store, download replicas from the same store, and fetch updates from the remote store. This helps offload responsbilities from the primary shard to the remote store. +1. Segment replication leads to increased network congestion on primary shards using node-to-node replication, because replica shards fetch updates from the primary shard. With a remote store, the primary shard can upload segments to the remote store and the replicas can fetch updates from the remote store. This helps offload responsibilities from the primary shard to the remote store. Read-after-write guarantees: Segment replication does not currently support setting the refresh policy to `wait_for`. If you set the `refresh` query parameter to `wait_for` and then ingest documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. If realtime reads are needed, consider using [`get`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/get-documents/) or [`mget`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/multi-get/) API operations. -1. As of OpenSearch 2.10, system indexes are now supported inside segment replication. -1. Get, MultiGet, TermVector, and MultiTermVector requests serve strong reads by routing requests to the primary shards. This can hurt performance since more requests are handled by the primary shards compared to distributing requests across primary and replicate shards. To help with performance in read heavy clusters, we recommend setting the `realtime` parameter in these requests to `false`, especially with listed request types. For more information, see [Issue #8700](https://github.com/opensearch-project/OpenSearch/issues/8700). +1. As of OpenSearch 2.10, system indexes support segment replication. +1. Get, MultiGet, TermVector, and MultiTermVector requests serve strong reads by routing requests to the primary shards. Routing more requests to the primary shards may degrade performance compared to distributing requests across primary and replica shards. To improve performance in read-heavy clusters, we recommend setting the `realtime` parameter in these requests to `false`. For more information, see [Issue #8700](https://github.com/opensearch-project/OpenSearch/issues/8700). ## Benchmarks From f4c8df82aa40de8c3d06d5f8ace936ece7d896b0 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 19 Sep 2023 11:29:02 -0500 Subject: [PATCH 10/11] Replace Remote Store with Remote Backed Storage Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../segment-replication/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md index fd692f891a..1eaed8e818 100644 --- a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md +++ b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md @@ -28,11 +28,11 @@ Segment replication can be applied in a variety of scenarios, including: - When experiencing very high loads, you want to add new nodes but don't want to index all data immediately. - OpenSearch cluster deployments with low replica counts, such as those used for log analytics. -## Remote store +## Remote-backed storage As of OpenSearch 2.10, you can use two methods for segment replication: -- With **remote-backed storage**, a persistent storage solution: The primary shard sends segment files to the remote store and the replica shards source the copy from the same store. For more information about using a remote store, see [Remote-backed storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#segment-replication-and-remote-backed-storage). +- With **remote-backed storage**, a persistent storage solution: The primary shard sends segment files to the remote-backed storage and the replica shards source the copy from the same store. For more information about using a remote-backed storage, see [Remote-backed storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#segment-replication-and-remote-backed-storage). - Without remote-backed storage: The primary shard sends segment files directly to the replica shards using node-to-node communication. ## Segment replication configuration @@ -55,7 +55,7 @@ PUT /my-index1 ``` {% include copy-curl.html %} -If you're using a remote store, add the `remote_store` property to the index request body. For more information, see [Create an index]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#create-an-index). +If you're using a remote-backed storage, add the `remote_store` property to the index request body. For more information, see [Create an index]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#create-an-index). When using node-to-node replication, the primary shard consumes higher network bandwidth because it pushes segment files to all the replica shards. Thus, it's beneficial to distribute primary shards equally between the nodes. To ensure balanced primary shard distribution, set the dynamic `cluster.routing.allocation.balance.prefer_primary` setting to `true`. For more information, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/). @@ -109,7 +109,7 @@ When using segment replication, consider the following: 1. Enabling segment replication for an existing index requires [reindexing](https://github.com/opensearch-project/OpenSearch/issues/3685). 1. [Cross-cluster replication](https://github.com/opensearch-project/OpenSearch/issues/4090) does not currently use segment replication to copy between clusters. 1. Segment replication is not compatible with [document-level monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/api/#document-level-monitors), which are used with the [Alerting]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/) and [Security Analytics]({{site.url}}{{site.baseurl}}/security-analytics/index/) plugins. The plugins also use the latest available data on replica shards when using the `immediate` refresh policy, and segment replication can delay the policy's availability, resulting in stale replica shards. -1. Segment replication leads to increased network congestion on primary shards using node-to-node replication, because replica shards fetch updates from the primary shard. With a remote store, the primary shard can upload segments to the remote store and the replicas can fetch updates from the remote store. This helps offload responsibilities from the primary shard to the remote store. +1. Segment replication leads to increased network congestion on primary shards using node-to-node replication, because replica shards fetch updates from the primary shard. With a remote-backed storage, the primary shard can upload segments to the remote-backed storage and the replicas can fetch updates from the remote-backed storage. This helps offload responsibilities from the primary shard to the remote-backed storage. Read-after-write guarantees: Segment replication does not currently support setting the refresh policy to `wait_for`. If you set the `refresh` query parameter to `wait_for` and then ingest documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. If realtime reads are needed, consider using [`get`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/get-documents/) or [`mget`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/multi-get/) API operations. 1. As of OpenSearch 2.10, system indexes support segment replication. 1. Get, MultiGet, TermVector, and MultiTermVector requests serve strong reads by routing requests to the primary shards. Routing more requests to the primary shards may degrade performance compared to distributing requests across primary and replica shards. To improve performance in read-heavy clusters, we recommend setting the `realtime` parameter in these requests to `false`. For more information, see [Issue #8700](https://github.com/opensearch-project/OpenSearch/issues/8700). From 1b215d26c46b04355578b32fe5ffe68f6733a143 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 19 Sep 2023 16:50:21 -0500 Subject: [PATCH 11/11] Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../segment-replication/index.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md index 1eaed8e818..1927ffac3c 100644 --- a/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md +++ b/_tuning-your-cluster/availability-and-recovery/segment-replication/index.md @@ -32,8 +32,8 @@ Segment replication can be applied in a variety of scenarios, including: As of OpenSearch 2.10, you can use two methods for segment replication: -- With **remote-backed storage**, a persistent storage solution: The primary shard sends segment files to the remote-backed storage and the replica shards source the copy from the same store. For more information about using a remote-backed storage, see [Remote-backed storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#segment-replication-and-remote-backed-storage). -- Without remote-backed storage: The primary shard sends segment files directly to the replica shards using node-to-node communication. +- **Remote-backed storage**, a persistent storage solution: The primary shard sends segment files to the remote-backed storage, and the replica shards source the copy from the same store. For more information about using remote-backed storage, see [Remote-backed storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#segment-replication-and-remote-backed-storage). +- Node-to-node communication: The primary shard sends segment files directly to the replica shards using node-to-node communication. ## Segment replication configuration @@ -55,9 +55,9 @@ PUT /my-index1 ``` {% include copy-curl.html %} -If you're using a remote-backed storage, add the `remote_store` property to the index request body. For more information, see [Create an index]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#create-an-index). +If you're using remote-backed storage, add the `remote_store` property to the index request body. For more information, see [Create an index]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#create-an-index). -When using node-to-node replication, the primary shard consumes higher network bandwidth because it pushes segment files to all the replica shards. Thus, it's beneficial to distribute primary shards equally between the nodes. To ensure balanced primary shard distribution, set the dynamic `cluster.routing.allocation.balance.prefer_primary` setting to `true`. For more information, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/). +When using node-to-node replication, the primary shard consumes more network bandwidth because it pushes segment files to all the replica shards. Thus, it's beneficial to distribute primary shards equally between the nodes. To ensure balanced primary shard distribution, set the dynamic `cluster.routing.allocation.balance.prefer_primary` setting to `true`. For more information, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/). For the best performance, it is recommended that you enable the following settings: @@ -109,10 +109,10 @@ When using segment replication, consider the following: 1. Enabling segment replication for an existing index requires [reindexing](https://github.com/opensearch-project/OpenSearch/issues/3685). 1. [Cross-cluster replication](https://github.com/opensearch-project/OpenSearch/issues/4090) does not currently use segment replication to copy between clusters. 1. Segment replication is not compatible with [document-level monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/api/#document-level-monitors), which are used with the [Alerting]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/) and [Security Analytics]({{site.url}}{{site.baseurl}}/security-analytics/index/) plugins. The plugins also use the latest available data on replica shards when using the `immediate` refresh policy, and segment replication can delay the policy's availability, resulting in stale replica shards. -1. Segment replication leads to increased network congestion on primary shards using node-to-node replication, because replica shards fetch updates from the primary shard. With a remote-backed storage, the primary shard can upload segments to the remote-backed storage and the replicas can fetch updates from the remote-backed storage. This helps offload responsibilities from the primary shard to the remote-backed storage. -Read-after-write guarantees: Segment replication does not currently support setting the refresh policy to `wait_for`. If you set the `refresh` query parameter to `wait_for` and then ingest documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. If realtime reads are needed, consider using [`get`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/get-documents/) or [`mget`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/multi-get/) API operations. +1. Segment replication leads to increased network congestion on primary shards using node-to-node replication because replica shards fetch updates from the primary shard. With remote-backed storage, the primary shard can upload segments to, and the replicas can fetch updates from, the remote-backed storage. This helps offload responsibilities from the primary shard to the remote-backed storage. +Read-after-write guarantees: Segment replication does not currently support setting the refresh policy to `wait_for`. If you set the `refresh` query parameter to `wait_for` and then ingest documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. If real-time reads are needed, consider using the [`get`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/get-documents/) or [`mget`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/multi-get/) API operations. 1. As of OpenSearch 2.10, system indexes support segment replication. -1. Get, MultiGet, TermVector, and MultiTermVector requests serve strong reads by routing requests to the primary shards. Routing more requests to the primary shards may degrade performance compared to distributing requests across primary and replica shards. To improve performance in read-heavy clusters, we recommend setting the `realtime` parameter in these requests to `false`. For more information, see [Issue #8700](https://github.com/opensearch-project/OpenSearch/issues/8700). +1. Get, MultiGet, TermVector, and MultiTermVector requests serve strong reads by routing requests to the primary shards. Routing more requests to the primary shards may degrade performance as compared to distributing requests across primary and replica shards. To improve performance in read-heavy clusters, we recommend setting the `realtime` parameter in these requests to `false`. For more information, see [Issue #8700](https://github.com/opensearch-project/OpenSearch/issues/8700). ## Benchmarks