Repolode
apache

apache/linkis

Computation middleware connecting applications to data engines like Spark, Hive and Flink via REST/WS/JDBC interfaces.

Stars
3,417
Forks
1,168
Open issues
188
Last commit
3 days ago

What is it?

Apache Linkis sits between upper-layer applications and underlying data engines as a computation middleware layer. Instead of wiring each application directly to engines such as MySQL, Spark, Hive, Presto or Flink, applications call Linkis over standard REST, WebSocket and JDBC interfaces. This decouples the application layer from the engine layer, simplifying network call relationships and reducing development and maintenance effort.

Beyond connectivity, Linkis handles intercommunication of user resources across engines: variables, scripts, UDFs, functions and resource files are set once and referenced automatically. It also provides computation governance such as task routing, load balancing, multi-tenancy, traffic control and resource control based on multi-level labels, plus public services including a materials library, context service and data source management. Supported engines include Spark, Hive, Python, Shell, JDBC, Flink, Pipeline, Sqoop, openLooKeng, Presto, ElasticSearch, Trino and SeaTunnel, with task types spanning offline batch, interactive query, real-time streaming and data lake workloads.

Key features

  • 01Supports many underlying engines: Spark, Hive, Python, Shell, Flink, JDBC, Pipeline, Sqoop, openLooKeng, Presto, ElasticSearch, Trino and SeaTunnel.
  • 02Accepts multiple languages: SparkSQL, HiveSQL, Python, Shell, Pyspark, Scala, JSON and Java.
  • 03Computation governance with task routing, load balancing, multi-tenancy, traffic control and resource control based on multi-level labels.
  • 04Unified context service to associate resource files (JAR, ZIP, Properties), result sets, parameter variables, functions and UDFs across users, systems and engines.
  • 05Unified materials management at system and user level, with sharing across users and systems.
  • 06Unified data source management for Hive, ElasticSearch, MySQL, Kafka, MongoDB and others, including CRUD, version control, connection tests and metadata queries.
  • 07Error codes with suggested solutions for common task errors so users can locate problems themselves.
  • 08Exposes REST/WS/JDBC interfaces to upper-layer applications.

Use cases

  • Serving as the unified entry point for the underlying compute and storage engines of a big data platform.
  • Applications that need to reach Spark, Hive, Presto, Flink and other engines through one consistent interface.
  • Multi-tenant platforms that need resource isolation, traffic control and task routing.
  • Sharing scripts, UDFs and resource files across users, systems and engines.
  • Running offline batch, interactive query, streaming and data lake tasks through a single task submission and management path.

Quick start

  1. 1

    Download a compiled distribution or source package from the Linkis releases page

  2. 2

    Install the parent POM before the first compile

    ./mvnw -N install
  3. 3

    Build the backend distribution package

    ./mvnw clean install -Dmaven.javadoc.skip=true -Dmaven.test.skip=true
  4. 4

    Follow the Quick Deployment guide on the official docs site to deploy

Things to know

  • Deployment is heavy: it is a multi-service Java platform that needs Maven builds or a full cluster setup.
  • The MySQL JDBC driver is not bundled in the official image for licensing reasons, so you must add it yourself.

In collections

Topics

#big-data#computation-middleware#spark#hive#flink#presto#sql#jdbc#data-governance#application-manager

FAQ

What is linkis?
Computation middleware connecting applications to data engines like Spark, Hive and Flink via REST/WS/JDBC interfaces. Apache Linkis sits between upper-layer applications and underlying data engines as a computation middleware layer. Instead of wiring each application directly to engines such as MySQL, Spark, Hive, Presto or Flink, applications call Linkis over standard REST, WebSocket and JDBC interfaces. This decouples the application layer from the engine layer, simplifying network call relationships and reducing development and maintenance effort.
Is linkis free to use? What license does it have?
linkis is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is linkis still maintained?
Actively maintained. The last commit to linkis was on Nov 4, 2026.
What can I use linkis for?
Serving as the unified entry point for the underlying compute and storage engines of a big data platform. Applications that need to reach Spark, Hive, Presto, Flink and other engines through one consistent interface. Multi-tenant platforms that need resource isolation, traffic control and task routing. Sharing scripts, UDFs and resource files across users, systems and engines. Running offline batch, interactive query, streaming and data lake tasks through a single task submission and management path.
How do I install linkis?
Download a compiled distribution or source package from the Linkis releases page Install the parent POM before the first compile: ./mvnw -N install Build the backend distribution package: ./mvnw clean install -Dmaven.javadoc.skip=true -Dmaven.test.skip=true Follow the Quick Deployment guide on the official docs site to deploy
Where can I get linkis?
The source code and installation instructions are on GitHub: https://github.com/apache/linkis

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects