Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

grpc: factor out schema related functionality from bigquery to common #338

Open
wants to merge 17 commits into
base: main
Choose a base branch
from

Conversation

alltilla
Copy link
Member

This will be useful for other drivers that want to support setting the message schema from config.

Depends on #334

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
Out intention is to batch messages that correspond
to the same label set. If between two messages the
label set changes, but they arrive to the same
worker, we should flush the previous batch.

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
Will be useful for templated headers as well.

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
This will be needed for the dynamic headers.

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
It must be enabled in the derived class, and
prepare_context_dynamic() must be used if so.

This is not part of this commit, enabling it will be
done in subsequent commits.

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
It will be needed there as well.

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
We are planning to introduce an opt-in grpc_dest_schema_option
nonterminal.

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
Copy link
Contributor

This Pull Request introduces config grammar changes

axoflow/71be47e0d85d46e497584cf39d346164a7e0c89b -> alltilla/grpc-schema-common

--- a/destination
+++ b/destination

 axosyslog-otlp(
+    frac-digits(<nonnegative-integer>)
     headers(
-        <string> => <string>
+        <string> => <template-content>
+        <string> => <template-reference>
     )
+    local-time-zone(<string>)
+    on-error(<string>)
+    send-time-zone(<string>)
+    template-escape(<yesno>)
+    time-zone(<string>)
+    ts-format(<string>)
 )

 bigquery(
     headers(
-        <string> => <string>
+        <string> => <template-content>
+        <string> => <template-reference>
     )
 )

 loki(
     headers(
-        <string> => <string>
+        <string> => <template-content>
+        <string> => <template-reference>
     )
 )

 opentelemetry(
+    frac-digits(<nonnegative-integer>)
     headers(
-        <string> => <string>
+        <string> => <template-content>
+        <string> => <template-reference>
     )
+    local-time-zone(<string>)
+    on-error(<string>)
+    send-time-zone(<string>)
+    template-escape(<yesno>)
+    time-zone(<string>)
+    ts-format(<string>)
 )

 syslog-ng-otlp(
+    frac-digits(<nonnegative-integer>)
     headers(
-        <string> => <string>
+        <string> => <template-content>
+        <string> => <template-reference>
     )
+    local-time-zone(<string>)
+    on-error(<string>)
+    send-time-zone(<string>)
+    template-escape(<yesno>)
+    time-zone(<string>)
+    ts-format(<string>)
 )

In order to opt-in for schema support, a derived
class must:

 * include grpc_dest_schema_option in the grammar
 * implement the string -> descriptor type mapping
 * store and init a Schema instance in the driver
 * implement the get_schema() method to return
   the stored Schema instance

The derived class can interface with the Schema
by these functions:
  * init()
  * empty()
  * format()
  * get_schema_descriptor()

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant