Repolode
Submit
OpenSenseNova

OpenSenseNova/SenseNova-Vision

A unified multimodal model that handles computer vision tasks like detection, OCR, segmentation, depth and multi-view geometry through text and image generation.

View on GitHub
Stars
717
Forks
42
Open issues
5
Last commit
2 months ago

What is it?

SenseNova-Vision reframes computer vision tasks as unified multimodal generation. Instead of task-specific heads or decoders, a single model receives natural-language instructions and optional visual prompts, then responds with text, an image, or a mixed text-image output. Symbolic results such as boxes, points, OCR strings, keypoints and camera parameters are produced as text, while dense outputs like segmentation masks, depth maps, surface normals and multi-view point maps are produced as images.

The project includes a 7B MoT model released on Hugging Face and ModelScope, an inference entrypoint with single-image, interactive and Gradio web demo modes, a benchmark evaluation pipeline, and training code. The SenseNova-Vision Corpus, a 50M-sample instruction-response dataset, is also published, and the training data is built by converting existing CV annotations into this format.

The web demo was validated on a single 80GB GPU in BF16, while the full benchmark and training require multi-GPU setups.

Key features

  • 01Handles detection, OCR, keypoints, GUI grounding, segmentation, depth and normals within one model.
  • 02Accepts natural-language instructions and visual prompts to define task and output format.
  • 03Produces text, image, or mixed text-image responses depending on the task.
  • 04Ships inference scripts, a Gradio web demo, and a benchmark evaluation pipeline.
  • 05Provides training data preparation and training pipeline documentation.

Use cases

  • Running open-vocabulary detection or referring segmentation with a single model and text prompts.
  • Extracting text from documents or screenshots with bounding boxes.
  • Estimating depth and surface normals from single images.
  • Reconstructing multi-view point maps and camera poses for 3D research.

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/OpenSenseNova/SenseNova-Vision.git
  2. 2

    Enter the repository directory

    cd SenseNova-Vision
  3. 3

    Create the environment with setup script

    bash setup.sh sensenova-vision
  4. 4

    Run the curated example

    bash scripts/run_sensenova_vision.sh example

Things to know

  • The web demo was only validated on an 80GB GPU; GPUs with less memory are not yet validated.
  • Full benchmark needs at least one 8 x 80GB GPU machine, and training needs at least 2 such machines.
  • Model weights must be downloaded separately from Hugging Face or ModelScope.

Topics

#multimodal#computer-vision#vision-language-model#segmentation#depth-estimation#ocr#object-detection#mllm#mllm-for-cv#unified-cv

FAQ

What is SenseNova-Vision?
A unified multimodal model that handles computer vision tasks like detection, OCR, segmentation, depth and multi-view geometry through text and image generation. SenseNova-Vision reframes computer vision tasks as unified multimodal generation. Instead of task-specific heads or decoders, a single model receives natural-language instructions and optional visual prompts, then responds with text, an image, or a mixed text-image output. Symbolic results such as boxes, points, OCR strings, keypoints and camera parameters are produced as text, while dense outputs like segmentation masks, depth maps, surface normals and multi-view point maps are produced as images.
Is SenseNova-Vision free to use? What license does it have?
SenseNova-Vision is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is SenseNova-Vision still maintained?
Updated within 6 months. The last commit to SenseNova-Vision was on Sep 1, 2026.
What can I use SenseNova-Vision for?
Running open-vocabulary detection or referring segmentation with a single model and text prompts. Extracting text from documents or screenshots with bounding boxes. Estimating depth and surface normals from single images. Reconstructing multi-view point maps and camera poses for 3D research.
How do I install SenseNova-Vision?
Clone the repository: git clone https://github.com/OpenSenseNova/SenseNova-Vision.git Enter the repository directory: cd SenseNova-Vision Create the environment with setup script: bash setup.sh sensenova-vision Run the curated example: bash scripts/run_sensenova_vision.sh example
Where can I get SenseNova-Vision?
The source code and installation instructions are on GitHub: https://github.com/OpenSenseNova/SenseNova-Vision

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects