Back to Archive

Safety System Products / Open Source

From a BLE Crash to Agentic HIL Testing

I have been building a cross-platform Flutter app at SSP for the past few months. It targets ~all platforms: Windows, Android, iOS, MacOS, Linux and Web. And to interface with the BLE of each platform, I am using this excellent "Universal BLE" Flutter package.

During early development, the Windows version was randomly crashing during BLE communication. Without throwing a dart-side exception or a trace of any sort. The app would just disappear. It was particularly frustrating because this app is meant to replace a troublesome legacy application and communicate reliably with a functional-safety device. Random crashes were hardly an improvement.

I had already used Universal BLE in my Smart Insole project, so I followed the problem into a library I knew and depended on. Moving the BLE device farther away from my laptop made the crash reproducible. I contributed a fix, kept testing, and found more bugs. At that point, I wanted a better way to reproduce these failures than walking around with a Bluetooth device and hoping to hit the right timing.

That grew into a hardware-in-the-loop (HIL) test system built around a programmable BLE peripheral. I used loop engineering to coordinate AI agents investigating bugs, writing fault-injection tests, and checking fixes on real hardware. My functional-safety background shaped the checks; the agents wrote changes and reran tests. The work has now identified and fixed 13+ bugs upstream, with a reusable test suite for subsequent releases.

Universal BLE contributor list with Usman Mehmood's profile circled in green

Project Brief

Context
Cross-platform Flutter application for a functional-safety device
My role
Test-system architecture, agent orchestration, BLE hardening, and verification
Tools
Flutter, Universal BLE, Zephyr, nRF52, agentic AI
Upstream
Merged fixes and reusable hardware tests for Universal BLE
8accepted PRs
13+bugs identified and fixed
85implemented HIL tests

1.From the Smart Insoles to an application at SSP

01 Learning Flutter from the device side

My Flutter work began with the Smart Insole Firmware project. I already knew BLE from embedded development, but building the companion application meant handling the other side of the connection too. The app needed to stream sensor data, synchronize stored sessions, configure the insoles, and recover when communication was interrupted.

That was where I discovered Universal BLE. It gave me a common Flutter API for Bluetooth across platforms. Many of the BLE features I developed for the insoles were later reimplemented in an upcoming application I am building at SSP for Windows, Android, and Web.

02 An application that needed to be dependable

The SSP application will replace a legacy app that had too many issues. I cannot disclose the application or device name yet, but it communicates with a functional-safety device. An application that randomly disappears during use would be a poor replacement.

The app itself is not a certified safety component. Still, my work as a functional-safety software engineer shaped what I expected from it: communication can fail, and the software needs to handle that failure and recover predictably.

2.The crash that started the investigation

01 Walking the device away from my laptop

During development, the Windows application would sometimes terminate without a trace. There was no Dart exception or useful stack trace, despite the exception handling in the app. Weak signals and reconnection attempts seemed to trigger it.

With the BLE device close to the laptop, it was intermittent. Once I started taking the device farther away, I could reproduce it reliably. The failure was in the native Windows implementation underneath Flutter, so my application never got a chance to handle it.

02 Fixing it raised a testing problem

My first two contributions were developed and checked manually against a physical BLE device, before I had a HIL suite. I investigated the crash, contributed the first fix, and continued testing connections and recovery. More bugs appeared. Universal BLE had already saved me considerable work, so contributing those improvements back made sense.

During the second PR, I started seeing how much more useful a programmable test peripheral could be. Walking a device around the room gave me very little control over when a failure happened. I wanted to repeat those failures deliberately and keep the tests for future changes, so I suggested the approach to the maintainer.

His response in the HIL discussion confirmed that this addressed a problem he was already facing: "One of the biggest pain points so far was reproducing obscure Windows issues so there is definitely value in it." He also offered to get a board and run the tests locally before releases.

3.Applying the verification practices I already knew

In functional-safety development, I am used to deliberately introducing faults and checking how software responds. It's called "Fault Injection Testing". I could apply that same approach here by controlling the BLE peripheral. Instead of waiting for a weak signal to interrupt an operation, the peripheral could disconnect at a chosen point or delay its response until the application timed out.

The tests needed to catch the bugs reliably and check what happened afterward. A failed connection should not leave the application unable to reconnect or complete another operation.

I decided to build a controllable hardware fixture and keep the reproduced failures as regression tests. Agentic AI helped me manage the changes to the application, library, and firmware needed to make that work.

4.Building hardware that could misbehave on command

A programmable peripheral

I architected the HIL system around an nRF52 development kit running purpose-built Zephyr firmware. A Flutter test uses Universal BLE to communicate with the board over a real Bluetooth connection. The test can arm a fault on the peripheral before starting the operation it wants to disturb, while USB serial provides firmware logs.

The tests communicate through the operating system and Bluetooth adapter over a physical radio link, just as the application does. The firmware can reject requests, delay responses, disconnect during an operation, or change the state the application expects to find.

Universal BLE hardware-in-the-loop architectureFlutter integration tests communicate through the Universal BLE Windows plugin and Windows Bluetooth stack with an nRF52 DK running Zephyr fault-injection firmware.APPLICATIONFlutter / DartIntegration testsUniversal BLE APINATIVE BOUNDARYWindows pluginPigeon + C++ / WinRTUniversal BLE pluginTRANSPORTWindows BLEBluetooth stackPhysical radio linkFAULT FIXTUREnRF52 DKZephyr GATT serverControlled fault plansObserve the failure, verify recovery, and repeat

Knowing whether the fault actually happened

A test can pass simply because the fault happened too early or too late to trigger the bug. I added replayable timing scenarios and peripheral counters so I could repeat the timing and confirm that the requested disruption actually happened.

Ordinary BLE tests established the baseline first. Fault-injection tests then checked the response to a disruption and the next operation afterward. That follow-up matters: reporting an error correctly is not much help if the connection remains unusable.

5.Giving the agents context from all three repositories

Three repositories in context

I provided context from the SSP application, the Universal BLE repository, and the fixture firmware. The application showed what real usage and recovery looked like. The library contained the behavior under investigation, and the firmware defined what the physical device could do to test it.

This was my practical use of context engineering. An agent working on a Flutter test needed to understand the firmware command it invoked and the library behavior it was checking. Keeping those pieces available together helped me avoid tests and firmware that looked reasonable separately but did not reproduce the intended failure together.

Dividing the work between agents

I split work between agents investigating the library, implementing Flutter tests, and developing the Zephyr fault-injection behavior. Independent tasks could run in parallel, with their results brought back into one coordinating thread. Separate verification work checked the proposed changes.

I defined the architecture, chose the failure scenarios, and decided what counted as correct recovery. The agents wrote much of the test code and firmware and automated the build, flash, and test cycle. I reviewed their results and directed the next iteration.

Where I needed help, and where I could judge the work

Flutter integration testing was new to me, so I used AI heavily on that side. The agents also implemented the initial fixture firmware, which I corrected and improved at various points. Embedded software, BLE, and Zephyr are my strongest areas, so I could check whether the peripheral behavior could actually occur during BLE communication and whether the proposed test made sense.

I coordinated the agents and reviewed how their changes worked together. When a test passed, I checked whether it had triggered the fault and confirmed the expected response.

6.Loop engineering: letting test results drive the next step

I based this workflow on loop engineering. In agentic AI, that means designing a feedback loop in which agents act, check the result, and use that evidence to decide what to do next. The checks and continuation rules let work progress without me writing a new prompt for every build failure or unsuccessful test.

Here, the feedback came from real hardware. The agents could adjust a fault scenario or a fix and repeat the build, flash, and test cycle. I defined the acceptance criteria and reviewed the changes. When the results were unclear, we investigated before accepting the work.

The hardware-backed agent loop

A hardware-backed loop with verification and retry branchesDefine a weakness and acceptance criteria. Agents implement or revise the test and fix. Build, flash, and run on real hardware. Check the evidence: the test must fail without protection and pass with it, including recovery. If checks are not satisfied, return the evidence to the next attempt. If they are, run regression tests and retain the scenario.Define the scenarioWeakness, context,and acceptance criteriaAgents make changesImplement or revisethe fault test and fixRun on hardwareBuild, flash, test;collect logs and resultsEvidencechecks out?Retry with evidencePassRetryRun regression suiteRetain the testAgentic AI+ real hardwareA hardware-backed loop with verification and retry branchesDefine a weakness and acceptance criteria. Agents implement or revise the test and fix. Build, flash, and run on real hardware. Check the evidence: the test must fail without protection and pass with it, including recovery. If checks are not satisfied, return the evidence to the next attempt. If they are, run regression tests and retain the scenario.Define the scenarioWeakness, context,and acceptance criteriaAgents make changesImplement or revisethe fault test and fixRun on hardwareBuild, flash, test;collect logs and resultsEvidencechecks out?RetryPassStartRun regression suiteRetain the test

Acceptance: fails without protection, passes with the fix, and recovers afterward.

I set the scope and acceptance criteria, review changes, and resolve ambiguous results. The agents carry out the repeated implementation and verification work.

1. Find a potential bug

The original crash gave us a starting point. From there, I used the agents to investigate related behavior and identify other places where an interruption or badly timed operation could leave the library in trouble. For each suspected bug, we needed a failure we could reproduce on the board.

2. Create the hardware scenario and host test

The agents implemented the matching peripheral behavior and Flutter test, built and flashed the firmware, then ran the scenario against the physical board. Logs and test results fed into the next iteration when the fault or test did not behave as intended.

3. Check that the test fails for the right reason

Before accepting a test, I needed it to expose the unprotected behavior. For existing fixes, this meant removing or bypassing the protection and running the scenario again. If the test stayed green, we investigated what it had missed. A build error or an unrelated connection failure did not count as evidence that it detected the bug.

4. Implement the fix and verify recovery

Once we could reproduce the bug, the agents implemented or refined the fix and reran the same test. For the protection-removal checks, we restored the hardening and verified that the result changed back to a pass. Follow-up reads, writes, or reconnections checked whether the library was usable after the injected failure.

5. Rerun the suite and retain the test

The corrected scenario joined the regression suite, and the remaining tests checked that the fix had not broken other BLE operations. Those tests remain available for subsequent library changes and releases, so the next round of work starts with the failures we already know about.

7.What grew out of that first crash

01 13+ bugs identified and fixed

The investigation and the test system built around it have now identified and fixed 13+ bugs in Universal BLE. The fixes cover connection recovery and other lifecycle problems on Windows, as well as improvements on Web.

The hardware suite started with 64 tests and has grown to 85: 25 baseline tests and 60 fault-injection tests. New scenarios continued to reveal bugs, which went back through the same failure, fix, and verification cycle.

02 A repeatable regression process

The suite gives me a way to rerun known failure scenarios against subsequent releases without recreating the investigation by hand. The maintainers also plan to run it before releases, addressing the Windows testing difficulty that came up during my first contribution.

I used the same fault-injection checks I use in functional-safety work, with AI agents helping me write and run the tests. The resulting firmware and host tests are part of the upstream contributions, so the maintainers can use them for their own release checks.

03 Contributing the work upstream

Eight contributions have merged into Universal BLE. The first fixes were #217 and #278. The hardware test system followed in #282, with further hardening in #284, #310, #311, and #319. #333 expanded the fixture and test coverage, alongside another fix exposed by that testing.

GitHub contributor activity from July to October 2026 showing usmanmehmood55 ranked second, behind fotiDim
My contributions alongside the maintainer's work, July to October 2026.

8.What the suite covers today

The automated hardware suite currently runs on Windows. It needs the physical nRF52 board and exclusive access to a Bluetooth adapter, so it does not run in ordinary pull-request CI yet. Automating that hardware-backed regression testing is ongoing work. Web changes have been manually tested; they do not yet have equivalent automated coverage.

The Zephyr peripheral can inject protocol errors, timing faults, disconnects, and state changes. It cannot generate arbitrary malformed link-layer packets. I can rerun the covered failures and compare the results, then add tests when I find something the suite misses.

My next step is to expand the fixture to two or three BLE peripherals operating at the same time. That will let me test concurrent connections and overlapping operations, including whether a failure on one device disrupts another. I want to use those scenarios to find and fix concurrency and thread-safety problems that may only appear when several devices are connected.