Skip to content

Home

Jump to bottom
Neil Avery edited this page Oct 3, 2023 · 8 revisions

SpecMesh is an opinionated modelling layer over Apache Kafka resources that combines GitOps, AsyncAPI (modelling), a parser, testing, and provisioning tools as well as chargeback support. By utilizing this methodology and toolset, it enables organizations to adopt Kafka at scale while incorporating simplification guardrails to prevent many typical mistakes. Resource provisioning is concealed beneath the AsyncAPI specification, providing a simplified view that allows both technical and non-technical users to design and build their Kafka applications as data products.

Guides:

In 60 seconds

Here is a simple SpecMesh API spec. When the provision command is executed, a specified set of topics will be provisioned for each app.id + channel - i.e. acme.lifestyle.onboarding._public.user_signed_up. In this case, the domain owner (app-id) is the only principle with ACL permissions to write to it, however, all principles can consume from the _public topic. ACLs are implied through the _private, _protected, _public convention. The structural requirement enforces clear topic and acl ownership (including schemas), but also means the ecosytem resources can not be mined to extract storage and consumption metrics for billing purposes.

asyncapi: '2.5.0'
id: 'urn:acme.lifestyle.onboarding'
info:
  title: ACME Lifestyle Onboarding
  version: '1.0.0'
  description: |
    The ACME lifestyle onboarding app that allows stuff - see this url for more detail.. etc

channels:
  _public.user_signed_up:
    bindings:
      kafka:
        partitions: 3
        replicas: 1
        configs:
          cleanup.policy: delete
          retention.ms: 999000

    publish:
      message:
        payload:
          $ref: "/schema/simple.schema_demo._public.user_signed_up.avsc"

  _private.user_checkout:
    publish:
      message:
        payload:
          $ref: "/schema/simple.schema_demo._public.user_checkout.yml"


  _protected.purchased:
    publish:
      summary: Humans purchasing food - note - restricting access to other domain principles
      tags: [
        name: "grant-access:some.other.domain.root"
      ]
      message:
        name: Food Item
        tags: [
          name: "human",
          name: "purchase"
        ]

Apache Kafka Best Practices enforced by SpecMesh

SpecMesh enables the following best practices

  1. Topic structures that reflect ownership using Aggregate/AppId concepts
  2. Clearly named and configured topic properties (partitioning, replication retention etc)
  3. GitOps as the source of truth for configuration and change management
  4. Simplified, self-governance modelling by prefixing topic _private, _protected,_public concepts. i.e. acme:lifestyle.onboarding._public.user_signed_up
  5. Systematic & consistent provisioning of structured topics and their associated Kafka resources
  6. Industry standard based provisioning process
  7. Combining disparate resources into a simplified, yet logical model (topics, schemas, ACLs)
  8. Prevention of incorrectly named topics or other resources
  9. Bundling together sets of topics into a single, publishable API that models well known concepts
  10. Support for metadata modelling and attachment (AsyncAPI tags)
  11. Chargeback capability that breaks down consumption and production metrics on a per app/aggregate-id basis
  12. Export AsyncAPI specs of apps using the app/aggregate-id (currently in draft form)

Adoption

SpecMesh is designed for users (or applications) that are eager to build new Apache Kafka apps using this approach (resource structuring/modelling). Existing apps that do not conform to the required structure captured in the AsyncAPI spec cannot be adapted at this time; however, we have plans to support such apps in the future.

Compared to?

  • Ansible & Terraform are DevOps tools tools - they maintain their own state of provisioned resources against which their execution will check as the scripts are run. It is generally advisable to use Ansible and Terraform for Server-like infrastructure (i.e. Brokers, Load balancers, Firewalls) as opposed to App-like resources (data infra - such as topics where GitOps is more suitable). Confluent support both Ansible and Terraform.
  • JulieOps is a Kafka GitOps tool with similarities to SpecMesh; both utilize GitOps to provision and maintain Kafka resources (Topics etc). However, their underlying philosophies differ. SpecMesh serves as a data modeling abstraction layer based upon the AsyncAPI, while JulieOps does not hide or model the underlying resources. Instead of being conflated, they are independently configured, including ACLs, topics, and schemas. JulieOps also extends into server-side infrastructure provisioning, such as connectors and ksqlDB.

GitOps versus DevOps and SpecMesh

This section is included to encourage both developers and DevOps teams to consider their differences. Technically, there isn't much distinction between them besides the state changelog mechanism: Self-managed versus Git-centric sources of truth. However, in practice, we often see DevOps teams managing server infrastructure, such as spinning up clusters, server processes, and databases. This long-lived infrastructure underpins the application runtime. Conversely, developers typically follow the GitOps process as part of standard development procedures, including pull requests, code analysis, and peer reviews.

In this context, SpecMesh is developer-centric and rooted in GitOps, empowering developers to self-govern. Developers can create _private, _protected, and _public topics, modify and evolve schemas (event models), and write their streaming apps accordingly.

Clone this wiki locally