You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Enable visibility into the relationship between topics involved in a workspace's Kafka Stream application
Non-Goals
Manipulating existing project code around KStream relationships
Deviating from KStream Open Source models or requirements (maybe a KIP or smaller change would be needed to better enable better associations down the road)
Remote KStream application tracking (maybe a follow-up or outcome of how to overcome identification, but currently this is easier to solve for local systems first?)
Background
Kafka Streams is a nearly completely client side abstraction which manages topics to a higher abstraction level. Understanding how these topics are being managed for an application under development would be a great tool assist for building these programs. Currently visibility in all contexts about KStreams applications is limited or at the discretion of the program developer.
Proposal
This has more open questions than clear routes to success atm. Since the Kafka Stream applications don't export their model or scope of impact to logs or remote stores it's challenging to associate topics to a given stream. One could implement a naming convention or possibly sniff topic message headers to pull out an attribute? Or maybe we would need to have the application log it's associations explicitly for the extension to parse. Extracting this information from the source code itself would not be easily feasible in all situations, leading to error prone reporting. Ideas and discussion welcome on best potential approaches here to take, though we can lean on the locality of the KStream program and the extension being on the same machine here for solutioning.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Goals
Non-Goals
Background
Kafka Streams is a nearly completely client side abstraction which manages topics to a higher abstraction level. Understanding how these topics are being managed for an application under development would be a great tool assist for building these programs. Currently visibility in all contexts about KStreams applications is limited or at the discretion of the program developer.
Proposal
This has more open questions than clear routes to success atm. Since the Kafka Stream applications don't export their model or scope of impact to logs or remote stores it's challenging to associate topics to a given stream. One could implement a naming convention or possibly sniff topic message headers to pull out an attribute? Or maybe we would need to have the application log it's associations explicitly for the extension to parse. Extracting this information from the source code itself would not be easily feasible in all situations, leading to error prone reporting. Ideas and discussion welcome on best potential approaches here to take, though we can lean on the locality of the KStream program and the extension being on the same machine here for solutioning.
Beta Was this translation helpful? Give feedback.
All reactions