Skip to content

KafkaFlow is a .NET framework to create Kafka based applications, simple to use and extend.

License

Notifications You must be signed in to change notification settings

andrefilipegsilva/kafka-flow

 
 

Repository files navigation

KafkaFlow

Build Master Codacy Badge Slack

KafkaFlow is a .NET framework to create Kafka based applications, simple to use and extend.

KafkaFlow uses Confluent Kafka Client.

Features

  • Multi-threaded consumer with message order guarantee
  • Middlewares support for producing and consuming messages
  • Support topics with different message types
  • Consumers with many topics
  • Serializer middleware with ApacheAvro, ProtoBuf and Json algorithms
  • Schema Registry support
  • Compression using native Confluent Kafka client compression or compressor middlewares
  • Graceful shutdown (wait to finish processing to shutdown)
  • Store offset when processing ends, avoiding message loss
  • Supports .NET Core and .NET Framework
  • Can be used with any dependency injection framework (see here)
  • Fluent configuration
  • Admin Web API that allows pause, resume and restart consumers, change workers count and rewind offsets, all at runtime
  • Dashboard UI that allows to visualize relevant informations about all consumers and manage them

Installation

Check the setup page

Usage

Build for .NET Core 2.1 and later using Hosted Service

public static void Main(string[] args)
{
    Host
        .CreateDefaultBuilder(args)
        .ConfigureServices((hostContext, services) =>
        {
            services.AddKafkaFlowHostedService(kafka => kafka
                .UseConsoleLog()
                .AddCluster(cluster => cluster
                    .WithBrokers(new[] { "localhost:9092" })
                    .AddConsumer(consumer => consumer
                        .Topic("sample-topic")
                        .WithGroupId("sample-group")
                        .WithBufferSize(100)
                        .WithWorkersCount(10)
                        .AddMiddlewares(middlewares => middlewares
                            .AddSerializer<NewtonsoftJsonMessageSerializer>()
                            .AddTypedHandlers(handlers => handlers
                                .AddHandler<SampleMessageHandler>())
                        )
                    )
                    .AddProducer("producer-name", producer => producer
                        .DefaultTopic("sample-topic")
                        .AddMiddlewares(middlewares => middlewares
                            .AddSerializer<NewtonsoftJsonMessageSerializer>()
                        )
                    )
                )
            );
        })
        .Build()
        .Run();
}

See samples for more details

Documentation

Wiki Page

Contributing

Read the Contributing guidelines

Authors

We are available through Issues, Discussions or Slack

License

MIT

About

KafkaFlow is a .NET framework to create Kafka based applications, simple to use and extend.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • C# 85.5%
  • TypeScript 11.8%
  • HTML 2.3%
  • Other 0.4%