Repolode
Submit
vaexio

vaexio/vaex

Python DataFrame library for lazily exploring and visualizing billion-row tabular datasets without loading them fully into memory.

Open-source alternative to:Tableau

View on GitHubWebsite
Stars
8,516
Forks
599
Open issues
556
Last commit
yesterday

What is it?

Vaex is a Python library for working with large tabular datasets that do not fit in memory. It offers a Pandas-like DataFrame API, but keeps data on disk through memory mapping and evaluates expressions lazily, so filtering and transformations do not create copies of the data.

The library computes statistics such as mean, sum, count and standard deviation over N-dimensional grids, and supports visualization through histograms, density plots and 3D volume rendering. HDF5 and Apache Arrow files are opened via memory mapping, and lazy streaming from S3 is supported in combination with it. Joins avoid materializing the right-hand table.

Vaex integrates with Jupyter and Voila for notebooks and dashboards, and includes machine learning utilities that work without explicit pipelines. It is installable with pip or conda.

Key features

  • 01Memory-mapped opening of HDF5 and Apache Arrow files
  • 02Lazy expression evaluation that avoids copying data
  • 03Parallelized groupby and aggregations, including on categories
  • 04Joins that do not materialize the right-hand table
  • 05Histogram, density plot and 3D volume visualization

Use cases

  • Exploring multi-gigabyte or billion-row tables on a single laptop
  • Creating interactive density plots and histograms of large datasets
  • Training scikit-learn models on very large tabular data
  • Building interactive dashboards in Jupyter or Voila

Quick start

  1. 1

    Install Vaex with pip

    pip install vaex
  2. 2

    Or install Vaex with conda from conda-forge

    conda install -c conda-forge vaex

In collections

Diagrams, charts & animated visualizationsData science courses & tools

Topics

#dataframe#big-data#data-analysis#out-of-core#apache-arrow#visualization#python#bigdata#data-science#hdf5

FAQ

What is vaex?
Python DataFrame library for lazily exploring and visualizing billion-row tabular datasets without loading them fully into memory. Vaex is a Python library for working with large tabular datasets that do not fit in memory. It offers a Pandas-like DataFrame API, but keeps data on disk through memory mapping and evaluates expressions lazily, so filtering and transformations do not create copies of the data.
Is vaex free to use? What license does it have?
vaex is open source under the MIT license. Check the license terms before using it in commercial products.
Is vaex still maintained?
Actively maintained. The last commit to vaex was on Nov 6, 2026.
What can I use vaex for?
Exploring multi-gigabyte or billion-row tables on a single laptop Creating interactive density plots and histograms of large datasets Training scikit-learn models on very large tabular data Building interactive dashboards in Jupyter or Voila
How do I install vaex?
Install Vaex with pip: pip install vaex Or install Vaex with conda from conda-forge: conda install -c conda-forge vaex
Where can I get vaex?
The source code and installation instructions are on GitHub: https://github.com/vaexio/vaex

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects