Top n queries historical queries from local index and time range #84

LilyCaroline17 · 2024-08-30T01:53:18Z

Description

When the local index exporter is enabled, historical top N queries data is stored there. Previously, obtaining these historical queries required manually searching through each index. To simplify this process for users, we've introduced two parameters: 'from' and 'to', to the top n queries request. This allows users to define the time range for retrieving historical data. Please ensure that from and to are in ISO format.

For example, you could make this following call to retrieve the top N queries from August 25, 2024, at 15:00 UTC to August 30, 2024, at 17:00 UTC.
curl -X GET "http://localhost:9200/_insights/top_queries?from=2024-08-25T15:00:00.000Z&to=2024-08-30T17:00:00.000Z"

Issues Resolved

Closes #12

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Emily Guo <emilyguo@amazon.com> Handle indexing errors and accurate search parsing Signed-off-by: Emily Guo <emilyguo@amazon.com> Correctly filter in window queries when from and to exist Signed-off-by: Emily Guo <emilyguo@amazon.com> Remove search requests from top n queries Signed-off-by: Emily Guo <emilyguo@amazon.com> Removed comments Signed-off-by: Emily Guo <emilyguo@amazon.com> Comments for new functions Signed-off-by: Emily Guo <emilyguo@amazon.com> Comments for new functions Signed-off-by: Emily Guo <emilyguo@amazon.com> Updated comments Signed-off-by: Emily Guo <emilyguo@amazon.com>

Signed-off-by: Emily Guo <emilyguo@amazon.com>

Signed-off-by: Emily Guo <35637792+LilyCaroline17@users.noreply.github.com>

deshsidd · 2024-08-30T19:03:14Z

Looks like one of the build is failing please take a look

Signed-off-by: Emily Guo <35637792+LilyCaroline17@users.noreply.github.com>

ansjcy · 2024-09-05T00:04:52Z

src/main/java/org/opensearch/plugin/insights/core/reader/LocalIndexReader.java

+            end = DateTime.now(DateTimeZone.UTC);
+        }
+        DateTime curr = start;
+        while (curr.compareTo(end.plusDays(1).withTimeAtStartOfDay()) < 0) {


Note: We are making assumptions on the index pattern that it will rollover every day. We should add this assumption with a check in the configuration API as well. And we should not allow user to change this rollover period and only allow them to change the prefix pattern. In that case

MatchQueryBuilder excludeQuery = QueryBuilders.matchQuery("indices", "top_queries*");

would become something like

MatchQueryBuilder excludeQuery = QueryBuilders.matchQuery("indices", TOP_N_QUERIES_INDEX_PATTERN + "*");

src/main/java/org/opensearch/plugin/insights/core/service/QueryInsightsService.java

src/main/java/org/opensearch/plugin/insights/core/service/TopQueriesService.java

ansjcy · 2024-09-05T00:16:50Z

src/main/java/org/opensearch/plugin/insights/core/service/TopQueriesService.java

+            return Arrays.stream(indices).noneMatch(index -> {
+                if (index instanceof String) {
+                    String indexString = (String) index;
+                    return indexString.contains("top_queries");


Also we are basically saying ignoring any requests made on the top_queries index. it might be worth discussing it further whether we want to do this.

src/main/java/org/opensearch/plugin/insights/core/service/TopQueriesService.java

...va/org/opensearch/plugin/insights/rules/transport/top_queries/TransportTopQueriesAction.java

…ilters to historical, and rename functions Signed-off-by: Emily Guo <LilyCaroline1717@gmail.com>

Signed-off-by: Emily Guo <35637792+LilyCaroline17@users.noreply.github.com>

Signed-off-by: Emily Guo <LilyCaroline1717@gmail.com>

ansjcy · 2024-09-05T02:24:48Z

Thanks for making the changes!
Overall it looks good to me and I'm merging it now, but we still need to follow up one several things before 2.18 release:

Discuss further on whether we should ignore any requests made on the top_queries index.
Have more strict restriction on index pattern the user can use.

* Added support for historical top n queries Signed-off-by: Emily Guo <emilyguo@amazon.com> Handle indexing errors and accurate search parsing Signed-off-by: Emily Guo <emilyguo@amazon.com> Correctly filter in window queries when from and to exist Signed-off-by: Emily Guo <emilyguo@amazon.com> Remove search requests from top n queries Signed-off-by: Emily Guo <emilyguo@amazon.com> Removed comments Signed-off-by: Emily Guo <emilyguo@amazon.com> Comments for new functions Signed-off-by: Emily Guo <emilyguo@amazon.com> Comments for new functions Signed-off-by: Emily Guo <emilyguo@amazon.com> Updated comments Signed-off-by: Emily Guo <emilyguo@amazon.com> * Added unit tests Signed-off-by: Emily Guo <emilyguo@amazon.com> * Update LocalIndexExporter.java Signed-off-by: Emily Guo <35637792+LilyCaroline17@users.noreply.github.com> * Update LocalIndexReaderTests.java Signed-off-by: Emily Guo <35637792+LilyCaroline17@users.noreply.github.com> * Update QueryInsightsReaderFactoryTests.java Signed-off-by: Emily Guo <35637792+LilyCaroline17@users.noreply.github.com> * Address comments, change getTimeRange into getFrom and getTo, apply filters to historical, and rename functions Signed-off-by: Emily Guo <LilyCaroline1717@gmail.com> * Fix test cases Signed-off-by: Emily Guo <LilyCaroline1717@gmail.com> --------- Signed-off-by: Emily Guo <emilyguo@amazon.com> Signed-off-by: Emily Guo <35637792+LilyCaroline17@users.noreply.github.com> Signed-off-by: Emily Guo <LilyCaroline1717@gmail.com> (cherry picked from commit 4d896cc) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

#105) * Added support for historical top n queries Handle indexing errors and accurate search parsing Correctly filter in window queries when from and to exist Remove search requests from top n queries Removed comments Comments for new functions Comments for new functions Updated comments * Added unit tests * Update LocalIndexExporter.java * Update LocalIndexReaderTests.java * Update QueryInsightsReaderFactoryTests.java * Address comments, change getTimeRange into getFrom and getTo, apply filters to historical, and rename functions * Fix test cases --------- (cherry picked from commit 4d896cc) Signed-off-by: Emily Guo <emilyguo@amazon.com> Signed-off-by: Emily Guo <35637792+LilyCaroline17@users.noreply.github.com> Signed-off-by: Emily Guo <LilyCaroline1717@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

LilyCaroline17 and others added 4 commits August 22, 2024 16:50

Merge branch 'opensearch-project:main' into main

ab44811

Added unit tests

bb3738f

Signed-off-by: Emily Guo <emilyguo@amazon.com>

Merge branch 'opensearch-project:main' into main

8c2e929

LilyCaroline17 requested review from ansjcy, jainankitk, deshsidd and dzane17 as code owners August 30, 2024 01:53

This was referenced Aug 30, 2024

Added from and to parameters to top N queries to specify time range LilyCaroline17/documentation-website#1

Closed

Added from and to parameters to top N queries to specify time range opensearch-project/documentation-website#8132

Open

LilyCaroline17 added 2 commits August 30, 2024 09:41

Update LocalIndexExporter.java

1ba0c80

Signed-off-by: Emily Guo <35637792+LilyCaroline17@users.noreply.github.com>

Update LocalIndexReaderTests.java

d562b8f

Signed-off-by: Emily Guo <35637792+LilyCaroline17@users.noreply.github.com>

Update QueryInsightsReaderFactoryTests.java

9e8f874

Signed-off-by: Emily Guo <35637792+LilyCaroline17@users.noreply.github.com>