[DOC] Spark Support - Dashboard Sources, Materialized Views, and Covering Indexes

[brijos]

What do you want to do?

• [] Request a change to existing documentation
• Add new documentation
• Report a technical problem with the documentation
• Other

Tell us about your request. Provide a summary of the request and all versions that are affected.
In this release, we are building on the opensearch-project/sql#1875. Users in Dashboard will be able to add a new Spark connection to OpenSearch, setup skipping, materialized views, and covering indexes, as well as search external sources from Observability Logs using PPL. Users in Discover who select a Spark data source will be moved to Observability Logs where they will be able to query using PPL.

• Admins will be able to create a new Spark connection and limit access to the connection using OpenSearch FGAC
• Admins can accelerate queries using skipping, materialized views, and covering indexes, limit access to who can manage indexes using OpenSearch FGAC, and set index trim settings to for performance and privacy purposes
• OpenSearch Dashboards users who use discover will be switched to Observability Logs when they select an external source via Spark and will use OpenSearch PPL as a query language (SQL will be supported using APIs)
• Users querying using Observability Logs will be able to direct query external sources

What other resources are available? Provide links to related issues, POCs, steps for testing, etc.
opensearch-project/sql#2027

[vagimeli]

@brijos Where do you think documentation belongs on the doc site? Observability? API Reference? Other?

[vagimeli]

@brijos @shanilpa @derek-ho @hdhalter Before I can get started with writing documentation, I need clarity on the following:

1. Is this documentation issue for the Dashboards feature (that is, frontend)?

2. Is there backend documentation? If so, please provide the link, as I haven't found any on the doc site. If no backend documentation exists, a technical lead will need to put up a draft PR.

3. Is there a Dashboards demo? If not, I'll schedule a demo meeting ASAP.

4. Is there a link to the Figma file for the UX mockups?

This feature appears to be very involved, so having a plan for content development is necessary so that we can meet the release deadline. Thanks for your attention.