Repolode
Submit
NoizAI

NoizAI/HelixWorld

A real-time interactive world model that generates a navigable scene from an image and prompt, with picture and camera-aligned stereo sound updating together.

View on GitHubWebsite
Stars
626
Forks
32
Open issues
2
Last commit
yesterday

What is it?

HelixWorld takes a first-frame image and a text prompt, then generates a scene the user can move through. Walking forward or turning updates both the picture and the sound, and the spatial audio field rotates with the camera. The project positions audio as generated jointly with the visuals rather than added afterwards as a soundtrack.

The model injects continuous 6-DoF camera poses into the video branch using Projective Relative Positional Encoding, and uses cross-modal attention to link visual geometry with acoustic features. The repository provides Preview v1 inference code that runs on Linux with Python 3.11, CUDA 12.x and an NVIDIA GPU, with an 80 GB VRAM recommendation. Inputs include action sequences such as W/A/S/D and camera keys, and output is an MP4 file.

The project also releases HelixBench, a benchmark for checking whether generated sound fields track viewpoint motion, along with a browser demo and a technical report on arXiv.

Key features

  • 01Generates a navigable scene from a single image and text prompt.
  • 02Synchronizes picture and camera-aligned stereo sound during interaction.
  • 03Accepts keyboard action sequences such as W, A, S, D and camera turns.
  • 04Runs single-GPU inference with BF16 precision and exports MP4 output.
  • 05Ships the HelixBench benchmark for spatial-acoustic consistency.

Use cases

  • Researching audio-visual world models and camera-conditioned video generation.
  • Prototyping interactive scenes where sound must follow the viewer's movement.
  • Evaluating whether generated audio tracks viewpoint changes using HelixBench.

Quick start

  1. 1

    Clone the repository

    git clone https://github.com/NoizAI/HelixWorld.git
  2. 2

    Create the Python 3.11 conda environment

    conda create -n helixworld-preview python=3.11 -y
  3. 3

    Install dependencies and download the model weights

    python -m pip install -r requirements.txt && python download_models.py
  4. 4

    Run inference from a first-frame image and prompt file on one GPU

    CUDA_VISIBLE_DEVICES=0 ./run.sh --image /path/to/first_frame.png --prompt-file examples/prompt.json --actions "W:5,right:5,stop:5" --perspective first_person --num-frames 121 --output-dir outputs/demo

Things to know

  • Requires Linux, Python 3.11, CUDA 12.x and an NVIDIA GPU.
  • About 70 GB of VRAM is used in a single-GPU run, and 80 GB is recommended.
  • This is a preview release, so behavior may change.
  • The model weights have a separate license from the Apache 2.0 code license.

Topics

#world-model#audio-visual#generative-ai#spatial-audio#realtime#video-generation

FAQ

What is HelixWorld?
A real-time interactive world model that generates a navigable scene from an image and prompt, with picture and camera-aligned stereo sound updating together. HelixWorld takes a first-frame image and a text prompt, then generates a scene the user can move through. Walking forward or turning updates both the picture and the sound, and the spatial audio field rotates with the camera. The project positions audio as generated jointly with the visuals rather than added afterwards as a soundtrack.
Is HelixWorld free to use? What license does it have?
HelixWorld is open source under the Apache-2.0 license. Check the license terms before using it in commercial products.
Is HelixWorld still maintained?
Actively maintained. The last commit to HelixWorld was on Oct 19, 2026.
What can I use HelixWorld for?
Researching audio-visual world models and camera-conditioned video generation. Prototyping interactive scenes where sound must follow the viewer's movement. Evaluating whether generated audio tracks viewpoint changes using HelixBench.
How do I install HelixWorld?
Clone the repository: git clone https://github.com/NoizAI/HelixWorld.git Create the Python 3.11 conda environment: conda create -n helixworld-preview python=3.11 -y Install dependencies and download the model weights: python -m pip install -r requirements.txt && python download_models.py Run inference from a first-frame image and prompt file on one GPU: CUDA_VISIBLE_DEVICES=0 ./run.sh --image /path/to/first_frame.png --prompt-file examples/prompt.json --actions "W:5,right:5,stop:5" --perspective first_person --num-frames 121 --output-dir outputs/demo
Where can I get HelixWorld?
The source code and installation instructions are on GitHub: https://github.com/NoizAI/HelixWorld

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects