Repolode
Submit
web-infra-dev

web-infra-dev/midscene

Midscene.js is a vision-driven E2E testing framework that lets you write UI test steps and assertions in natural language for web, mobile, and desktop apps.

View on GitHubWebsite
Stars
15,331
Forks
1,201
Open issues
123
Last commit
7 hours ago

What is it?

Midscene.js brings AI vision to end-to-end testing. Tests describe UI steps and expected results in plain language, and a visual GUI agent locates elements from screenshots, clicks, types, scrolls, and checks what is visible on screen. This means icon-only buttons, custom controls, canvas content, and cross-origin iframes can be targeted without maintaining CSS or XPath selectors.

The project offers two ways in. Midscene Test lets you write cases in YAML and reuse TypeScript nodes for setup, API calls, and cleanup. Alternatively, the Agent APIs (aiAct, aiTap, aiInput, aiAssert, aiQuery, aiWaitFor) can be added to existing Playwright or Puppeteer tests. Platform support includes Web browsers, Android via ADB, iOS via WebDriverAgent, HarmonyOS via HDC, and desktop systems.

Model choice is flexible: the project works with several vision-capable models, including options that can be self-hosted. Test runs produce interactive HTML reports with screenshots, element locations, and action and assertion results.

Key features

  • 01Writes UI actions and assertions in natural language, including checks on colors, layout, and visual feedback.
  • 02Locates elements from screenshots, so no selectors are needed for icon buttons or canvas content.
  • 03Supports YAML test cases through Midscene Test with reusable TypeScript nodes.
  • 04Integrates with Playwright and Puppeteer through the Agent API.
  • 05Covers Web, Android, iOS, HarmonyOS, and desktop platforms.
  • 06Generates interactive HTML reports with screenshots and execution results.

Use cases

  • Writing end-to-end tests for web apps where selectors are brittle or unavailable.
  • Testing canvas-based or visually rendered interfaces that DOM assertions cannot inspect.
  • Adding visual checks to an existing Playwright or Puppeteer test suite.
  • Automating mobile app flows on Android or iOS devices and emulators.

Quick start

  1. 1

    Install the Web SDK package from npm

  2. 2

    Create a Midscene Test project (see the guide for the exact setup command)

  3. 3

    Configure a model and the Web runtime, then write YAML test cases and run them

Things to know

  • Needs access to a vision-capable AI model, which you must configure and pay for.
  • Reported benchmark results depend on specific models and run configurations.

Topics

#e2e-testing#gui-agent#browser-automation#playwright#visual-testing#natural-language#mobile-testing#browser-use#computer-use#gpt-operator

FAQ

What is midscene?
Midscene.js is a vision-driven E2E testing framework that lets you write UI test steps and assertions in natural language for web, mobile, and desktop apps. Midscene.js brings AI vision to end-to-end testing. Tests describe UI steps and expected results in plain language, and a visual GUI agent locates elements from screenshots, clicks, types, scrolls, and checks what is visible on screen. This means icon-only buttons, custom controls, canvas content, and cross-origin iframes can be targeted without maintaining CSS or XPath selectors.
Is midscene free to use? What license does it have?
midscene is open source under the MIT license. Check the license terms before using it in commercial products.
Is midscene still maintained?
Actively maintained. The last commit to midscene was on Oct 20, 2026.
What can I use midscene for?
Writing end-to-end tests for web apps where selectors are brittle or unavailable. Testing canvas-based or visually rendered interfaces that DOM assertions cannot inspect. Adding visual checks to an existing Playwright or Puppeteer test suite. Automating mobile app flows on Android or iOS devices and emulators.
How do I install midscene?
Install the Web SDK package from npm Create a Midscene Test project (see the guide for the exact setup command) Configure a model and the Web runtime, then write YAML test cases and run them
Where can I get midscene?
The source code and installation instructions are on GitHub: https://github.com/web-infra-dev/midscene

Summary generated by AI from the project's README. GitHub stats refresh daily.

Related projects