Aquila Voice Test Suite for Home Assistant

An extensive test suite for voice assistants within Home Assistant allowing you to test any LLM or conversation agent against a battery of tests varying in scope and complexity. See the ~/model_results/ directory for existing model results.

Leaderboard

Below shows the current leaderboard of conversation agents and their accuracy mapping messages to HA intents. This only currently measures the "family_home_medium_two_storey" test suite, as the others need to be manually vetted for hallucinations, improper config / sentences, etc.

Model Name	# Tests	Success	Failed	Duration	% Success
Sophia NLU	3715	3655	60	10.1s	98.4%
Qwen3 4B Instruct	3715	1338	2377	99m 13s	36.0%
HA Assist	3715	540	3175	1m 6s	14.5%

NOTE: Both Sophia NLU and built-in HA Assist are deterministic, meaning runnning the tests against them will always result in the exact same results shown above. LLMs are probabilistic, so the results above will differ slightly but will generally the be the same.

Setup

To run the test suite, you must be familiar with docker-compose. Open terminal and run the following:

git clone https://git.cicero.sh/aquila/ha-voice-test-suite.git
cd ha-voice-test-suite
git clone https://github.com/home-assistant/core
mkdir -p core/config
docker compose build
docker compose run --rm hass
bash /workspaces/setup.sh

You may immediately stop Home Assistant by pressing Ctrl+C after it starts. Only the initialization needs to be completed, but it doesn't need to be running.

Run Tests

Modify the config.yaml file within this directory and define the test suites to run, along with which adapter / LLM model to test against. Once defined, run the tests from within the docker file with:

pytest tests/run.py -s

Be patient as it runs through all test sentences. Once complete, the full results will be available in the ~/results/ sub-directory including all failed sentences.

Upload Model Results

Help expand the leader board and keep the HA community informed about the best models. After running tests simply copy the resulting ~/results/summary.md file into ~/model_results/MODEL_NAME.md along with the other model results. Then simply push a PR with the new model result file, or simply e-mail the summary.md file to matt@nlu.to.

Test Single Sentence

You may test a single sentence with the following command:

pytest tests/single.py --sentence "Turn on the kitchen lights" -s

Test Sophia NLU

To run Sophia NLU through the tests, follow these steps:

1. If you do not already have a copy, visit https://nlu.to/ha/ and grab your free trial.
2. Login to the account portal, download and unzip Sophia.
3. Copy the ./src/addons/sophia_nlu/sophia-amd64 file to the directory where this test suite reporesides.
4. Login to the Docker container with: docker compose run --rm hass
5. Start the Sophia daemon with: ./sophia-amd64 -d
6. Open your web browser to http://127.0.0.1:10520/, configure and activate the Sophia license as necessary. If asked for location of the config directory, use the ./core/config/ directory that now resides within the test suite.
7. Modify the confiy.yaml file and change the adapter to "sophia_nlu".
8. That's it. Now within the Docker container run the tests with the same command: pytest tests/run.sh -s

Custom Conversation Agents

This test suite utilizes an adapter design pattern, and you can view the existing adapters in the ./adapters. directory. If you have a custom conversation agent you would like to test, instructions for implementing yor own conversation agent can be found in ./adapters/CLAUDE.md. Feel free to send a PR with your adapter, it will be included in the main branch.

About

This test suite was developed by Aquila Labs who also developed Sophia NLU. An on device, low compute, deterministic natural language engine that acts as a conversation agent within Home Assistant.

For full details and demo please visit: https://nlu.to/ha/