Repolode
Submit
apache

apache/iceberg

A high-performance open table format for huge analytic datasets that lets engines like Spark, Trino, Flink and Hive safely share the same tables.

View on GitHubWebsite
Stars
9,412
Forks
3,669
Open issues
909
Last commit
22 hours ago

What is it?

Apache Iceberg is a table format for large analytic datasets. It brings SQL-table reliability and simplicity to data lakes, so that multiple processing engines can work on the same tables at the same time without corrupting them.

This repository holds the core Java library, which serves as the reference implementation of the Iceberg specification. It is split into modules: a public API, core implementation with Avro support, optional Parquet, ORC and Arrow modules, a Hive metastore backed catalog, and a data module for reading and writing tables directly from JVM applications. Engine integrations for Spark, Flink and Hive are included as well.

Other language implementations are maintained in separate repositories, including Go, Python (PyIceberg), Rust and C++. The project is built with Gradle on Java 17 or 21.

Key features

  • 01Provides SQL-table-like reliability for data lake tables.
  • 02Lets multiple engines such as Spark, Flink and Hive work on the same tables concurrently.
  • 03Supports Avro, Parquet and ORC data files through optional modules.
  • 04Includes a Hive metastore backed table implementation.
  • 05Offers a Java API for reading and writing tables from JVM applications.

Use cases

  • Building a data lake where Spark and Trino query the same analytic tables.
  • Streaming data into analytic tables with Flink while batch jobs read them.
  • Adding table-level reliability to Parquet or ORC datasets in a Hive-based stack.

Quick start

  1. 1

    Clone the repository

  2. 2

    Build the project and run tests with Gradle (requires Java 17 or 21)

    ./gradlew build

Things to know

  • Building and running tests requires Java 17 or 21 and Docker for the test suite.
  • This repository is the Java library; it is not a standalone database or end-user application.

Topics

#data-lake#table-format#big-data#spark#flink#java#analytics#apache#hacktoberfest#iceberg

FAQ

What is iceberg?
A high-performance open table format for huge analytic datasets that lets engines like Spark, Trino, Flink and Hive safely share the same tables. Apache Iceberg is a table format for large analytic datasets. It brings SQL-table reliability and simplicity to data lakes, so that multiple processing engines can work on the same tables at the same time without corrupting them.
Is iceberg free to use? What license does it have?
iceberg is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is iceberg still maintained?
Actively maintained. The last commit to iceberg was on Nov 6, 2026.
What can I use iceberg for?
Building a data lake where Spark and Trino query the same analytic tables. Streaming data into analytic tables with Flink while batch jobs read them. Adding table-level reliability to Parquet or ORC datasets in a Hive-based stack.
How do I install iceberg?
Clone the repository Build the project and run tests with Gradle (requires Java 17 or 21): ./gradlew build
Where can I get iceberg?
The source code and installation instructions are on GitHub: https://github.com/apache/iceberg

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects