Repolode
Submit
xqlsystems

xqlsystems/xarray-sql

A Python library that lets you query Xarray datasets with SQL by treating array data as tables, usable with DuckDB, Polars, DataFusion and ADBC engines.

View on GitHubWebsite
Stars
153
Forks
23
Open issues
104
Last commit
5 days ago

What is it?

xarray-sql is a SQL interface for array datasets. It pivots Xarray Datasets into table form so that SQL queries can run against them on the query engine of your choice. A dataset is registered as a table with from_dataset, queried with standard SQL, and the result can be converted back into an Xarray Dataset for further analysis or plotting.

The same registration works across several engines, including DuckDB, Polars and DataFusion through a PyArrow dataset interface. Any database with an ADBC driver such as PostgreSQL, SQLite, Snowflake or BigQuery can ingest the data, and xql.serve exposes datasets as an Arrow Flight SQL server. The project documents support for cloud-native Zarr stores, with column projection and dimension-based predicate pushdown, so queries over large datasets such as ARCO-ERA5 only read the needed variables and partitions.

The project is an experiment and is still early, with known limitations documented per engine. Optional extras add geospatial support through a proj UDF.

Key features

  • 01Registers Xarray Datasets as SQL tables with from_dataset or xql.register.
  • 02Converts query results back to Xarray Datasets with to_dataset.
  • 03Works with DuckDB, Polars, DataFusion and any ADBC-supported database.
  • 04Pushes column projection and dimension filters down to Zarr and partition pruning.
  • 05Serves datasets over Arrow Flight SQL for external clients.

Use cases

  • Running SQL aggregations such as monthly climatologies over gridded weather or climate data.
  • Joining tabular data with raster or gridded datasets using SQL.
  • Querying large cloud-hosted Zarr datasets without loading everything into memory.
  • Letting SQL users access scientific array data through familiar database tools.

Quick start

  1. 1

    Install the library from PyPI

    pip install xarray-sql
  2. 2

    Install optional dependencies for the tutorial example (pooch, netCDF4, matplotlib)

    pip install pooch netCDF4 matplotlib

Things to know

  • Early-stage project; the maintainers are looking for early users and it may change.
  • Some engine-specific and fundamental known issues are documented separately, so check them before relying on it.
  • Dense tensor operations are slower in SQL than in native Xarray.

In collections

Databases & SQL toolsData science courses & tools

Topics

#xarray#sql#data-analysis#zarr#duckdb#datafusion#arrow#geospatial

FAQ

What is xarray-sql?
A Python library that lets you query Xarray datasets with SQL by treating array data as tables, usable with DuckDB, Polars, DataFusion and ADBC engines. xarray-sql is a SQL interface for array datasets. It pivots Xarray Datasets into table form so that SQL queries can run against them on the query engine of your choice. A dataset is registered as a table with from_dataset, queried with standard SQL, and the result can be converted back into an Xarray Dataset for further analysis or plotting.
Is xarray-sql free to use? What license does it have?
xarray-sql is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is xarray-sql still maintained?
Actively maintained. The last commit to xarray-sql was on Nov 2, 2026.
What can I use xarray-sql for?
Running SQL aggregations such as monthly climatologies over gridded weather or climate data. Joining tabular data with raster or gridded datasets using SQL. Querying large cloud-hosted Zarr datasets without loading everything into memory. Letting SQL users access scientific array data through familiar database tools.
How do I install xarray-sql?
Install the library from PyPI: pip install xarray-sql Install optional dependencies for the tutorial example (pooch, netCDF4, matplotlib): pip install pooch netCDF4 matplotlib
Where can I get xarray-sql?
The source code and installation instructions are on GitHub: https://github.com/xqlsystems/xarray-sql

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects