arkflow-rs/arkflow

What is it?

Key features

• 01Stream pipelines defined in YAML with input, processors, output and error output
• 02SQL, Python UDF, VRL and Protobuf processing steps
• 03Connectors for Kafka, MQTT, NATS, Redis, SQL databases and many more
• 04Embedding, LLM and vector search processors for AI workloads
• 05At-least-once delivery by default via per-stream WAL

Use cases

• Filtering and transforming Kafka topics in real time with SQL
• Running embedding or LLM enrichment over a message stream before writing to a vector database
• Ingesting IoT sensor data from MQTT and writing it to InfluxDB or SQL
• Syncing database change events from Debezium-formatted Kafka topics

Quick start

1. 1

   Clone the repository

   git clone https://github.com/arkflow-rs/arkflow.git
2. 2

   Enter the project directory

   cd arkflow
3. 3

   Build the project in release mode

   cargo build --release
4. 4

   Run ArkFlow with the config file

   ./target/release/arkflow --config config.yaml

Things to know

• !Requires a Rust toolchain to build from source; no package-manager install is documented in the README.
• !The Quick Start config must be created by hand as config.yaml before running.

In collections

Topics

FAQ

What is arkflow?

Rust stream processing engine that runs SQL, Python UDFs and ML/LLM steps over real-time data from Kafka, MQTT, NATS and other sources. ArkFlow is a stream processing engine written in Rust on the Tokio async runtime. Streams are defined in YAML, with an input, a processing pipeline, an output and an optional error output for failed messages. Inputs include Kafka, MQTT, NATS, Pulsar, Redis, WebSocket, HTTP, files and SQL databases, and outputs include Kafka, MongoDB, InfluxDB, SQL databases and vector stores such as Qdrant, Milvus and pgvector.

Is arkflow free to use? What license does it have?

arkflow is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.

Is arkflow still maintained?

Actively maintained. The last commit to arkflow was on Oct 22, 2026.

What can I use arkflow for?

Filtering and transforming Kafka topics in real time with SQL Running embedding or LLM enrichment over a message stream before writing to a vector database Ingesting IoT sensor data from MQTT and writing it to InfluxDB or SQL Syncing database change events from Debezium-formatted Kafka topics

How do I install arkflow?

Clone the repository: git clone https://github.com/arkflow-rs/arkflow.git Enter the project directory: cd arkflow Build the project in release mode: cargo build --release Run ArkFlow with the config file: ./target/release/arkflow --config config.yaml

Where can I get arkflow?

The source code and installation instructions are on GitHub: https://github.com/arkflow-rs/arkflow