Flink Workspace/Statements as a Resource

[MSeal]

Goals

1. Enable visibility into Flink workspace/statements as a resource subpanel option
2. Transfer SQL statements to and from Workspaces in CCloud
3. Submit a SQL statement query for execution (locally or against a cloud compute pool)
4. View temporary table results as you would a topic contents
5. Smart auto-completion of SQL values (stretch)

Non-Goals

1. Visibility into all related Flink resources or API contents (can be a separate discussion)
2. Operational management of Flink abstractions (e.g. not managing compute pools or regional assets)
3. Eliminating Confluent Cloud Workspaces / SQL operations (those has a clear home, value proposition, and roadmap as is)

Background

The ability to execute Flink requests alongside Kafka programs and project templates within a workspace would help expand the streaming abstraction support of the extension substantially. Developing queries is possible in the CCloud environment, but connecting those SQL statements to git projects or as a part of a larger streaming program is more challenging today purely from CCloud interfaces. This would also be a strong supplement for supporting further Flink integrations with abstractions like Functions and Jobs.

Proposal

• Add Flink resource APIs to the sidecar
• Add flink specific query subpanel to the extension UI
• Add SQL file extension awareness for association to elements in panel (perhaps needing a unique suffix extension name?)
• Add UI button + API call for query job submission for files

Initial list of unresolved aspects of the feature

• TBD managing forever queries -- do they need to be always persisted to a Workspace... some visibility and monitoring concerns
• TBD on how to best associate a file to a SQL resource
• TBD UX for local vs remote representations
• TBD on handling out of sync states (remote query changed independent of local representation in file)

The Confluent team is exploring feasibility and prioritization possibilities here, without any strong commitments at the time of discussion creation.