pathwaycom/pathway

What is it?

Key features

• 01Unified engine for batch and streaming data with the same Python code.
• 02Connectors for Kafka, PostgreSQL, Google Drive, SharePoint and more than 300 sources via Airbyte.
• 03Stateful transformations such as joins, windowing and sorting, plus support for arbitrary Python functions.
• 04Persistence to save computation state so pipelines can restart after updates or crashes.
• 05Rust engine supporting multithreading, multiprocessing and distributed computation.
• 06Built-in LLM tooling with an in-memory real-time vector index and LangChain and LlamaIndex integrations.

Use cases

• Real-time ETL that moves and transforms data from Kafka or databases.
• Event-driven pipelines with alerting, such as log monitoring.
• Live RAG applications that keep a vector index updated as documents change.
• Switching an existing batch data job to streaming without rewriting it.

Quick start

1. 1

   Install Pathway from PyPI with pip

   pip install -U pathway
2. 2

   Save the example code as main.py and run it with the Pathway runner

   pathway spawn python main.py

Things to know

• !Available only on macOS and Linux; other systems need a virtual machine.
• !Requires Python 3.10 or above.
• !The license is listed as NOASSERTION in the metadata and the README badge points to BSL, so check the license terms before commercial use.
• !Exactly-once consistency is only in the enterprise version; the free version provides at-least-once.

Topics

FAQ

What is pathway?

Python ETL framework for stream processing, real-time analytics, LLM pipelines and RAG, powered by a Rust engine. Pathway is a Python ETL framework for building pipelines that process both batch and streaming data. The same code can run for local development, CI tests, batch jobs, stream replays and live processing, so developers do not need to maintain separate batch and streaming versions of a pipeline.

Is pathway free to use? What license does it have?

pathway is open source under the Other license. Check the license terms before using it in commercial products.

Is pathway still maintained?

Actively maintained. The last commit to pathway was on Oct 20, 2026.

What can I use pathway for?

Real-time ETL that moves and transforms data from Kafka or databases. Event-driven pipelines with alerting, such as log monitoring. Live RAG applications that keep a vector index updated as documents change. Switching an existing batch data job to streaming without rewriting it.

How do I install pathway?

Install Pathway from PyPI with pip: pip install -U pathway Save the example code as main.py and run it with the Pathway runner: pathway spawn python main.py

Where can I get pathway?

The source code and installation instructions are on GitHub: https://github.com/pathwaycom/pathway