Letting agents test HDMI output

I’m bringing up HDMI on a small SG2000 board, and AI coding agents do much of the driver work. A session can change the driver, build it, load it on the board and start the display, but it can’t see the screen.

HDMI was putting out something, mostly visual gibberish, and every experiment waited for me to babysit a monitor and say what it showed. My eyes turned out to be easy to fool.

A USB capture device records the board’s real HDMI output, and a classifier the sessions wrote turns frames from each recording into a verdict a session can read: did the frame match the solid red that had been requested? The questions and the go-aheads stayed with me. The verdict that mattered most said no, to a hunch of mine.

Close the loop on the recorded picture

Driver modules are built on a separate machine, so the board only loads them. A session on the laptop drives the board over SSH.

Each experiment follows the same cycle: change, build, load, start, record, classify, propose. The cycle ends on a value the rig prints, never on a command’s exit status.

▶ What the capture device taught the scripts (expand)

A session wrote the requirements before I bought one:

• Accept 720p at 60 frames per second.
• Use USB Video Class (UVC), the standard that lets Linux and ffmpeg drive it without vendor software.
• Keep working as the signal drops and returns during driver tests.

I bought a cheap device that met the list. Its quirks shaped the scripts:

• No picture until a capture stream opens, then a few seconds to settle, so record a short clip and keep a late frame.
• Reopening too fast drops it off USB until replugged, so wait at least 20 seconds between streams and pause for a person if it disappears.

I tested the capture device by looping my laptop’s HDMI output back into its input. A recognizable desktop established that the capture path worked independently of the board.

A monitor is a forgiving judge

A monitor had already shown full-screen color bars from a community Linux image built on the chip vendor’s kernel, which I’ll call the vendor image, and the notes called it first pixels.

The main chip sends pixels to an HDMI bridge over MIPI DSI, a serial display link with four data lanes. An earlier session had ruled lane order out by looking at a monitor.

With the capture device, a session tried all 24 orders and read the bridge’s line counter over I2C, its two-wire control bus, after each one. For one order the bridge counted 750 lines per frame, the total line count of a 720p frame, and the capture showed a stripe. For the other 23 it read 1 and the capture was dark. The count is timing, not pixels.

The main chip’s test-pattern generator draws built-in images without reading frames from memory. The session requested solid red, then green, then blue, and hashed each capture. All three hashes were identical. A format or packing mistake would have scrambled each color differently. The recorded picture did not follow the request.

An exact-hash match needs a reference with the same bridge history, since the bridge keeps state across warm reboots. It doesn’t locate where the picture stopped.

The “color bars” celebrated on a monitor were a fixed pattern of one-pixel-wide colored columns, the vendor image’s own output, which I’ll call the stripe.

A week later a session asked me to look at a captured frame and confirm it before committing it as the classifier’s reference. I pushed back. Sessions had been telling black, white and stripe apart from captures, down to the byte. Putting my eyes back in the loop would have been a regression.

A verdict needs a no-signal answer and a narrow success class

An early classifier compared each frame against a stored stripe by pixel difference. A dead rig’s dark frame differed, so it scored as “the picture moved”. Losing the signal could look like progress. A stripe whose colors had shifted since the reference was captured also scored as progress. The fix was to recognize the stripe by its structure, whatever its colors, and to make success narrower than “different”.

A uniform frame near a named color is solid. A uniform frame in a color it doesn’t name is flat. The stripe is pinstripe. Anything else is hit, and a hit means “unclassified”, not “the picture arrived”. Black is a solid color and never counts as evidence that the link carries data. With no signal, this device sends a uniform dark frame, which the classifier reports as solid black.

Each request carries an expectation, such as solid:red, and the verdict says whether the frame matched it. The classifier’s exit status separates a measured mismatch from a broken input, so a session can’t confuse “red didn’t arrive” with “the capture failed”.

Test the instrument before trusting its no

A verdict that only ever says no proves nothing until a picture injected past the suspected fault comes through. To split the chain, a session asked the bridge to draw its own picture, bypassing the pixels arriving over DSI. One register write gave black; setting its fill bit gave white at 1280 by 720. Unlike the red, green and blue requests, the capture followed this one. That cleared the bridge’s HDMI output, the cable and the capture device. It said nothing about the main chip’s DSI output or the bridge’s DSI receiver.

When board captures went dark again, I asked for repeated recordings of the laptop’s HDMI output. Its desktop came through, so the capture path still worked.

A working capture path can still record the wrong source: a session recorded my desktop in what should have been a board recording. I had left the HDMI input connected to the laptop, so those recordings were excluded from the board test results.

Comparing driver changes meant keeping capture conditions consistent. Otherwise, a change in the recording setup could look like a change in the board’s output.

I pinned the stream at MJPEG, a stream of compressed JPEG frames, 1280 by 720, requesting 60 frames per second. Later, the scripts fixed a 15-second measurement window after two seconds of warm-up. No session could pick a flattering stretch afterward.

The timing check distinguishes a strict pass, with every frame arriving on time in the measured interval, from a diagnostic result that allows a startup hiccup with a caveat. That caveat never counts as a strict pass. The whole recording is kept, and the picture is checked separately, 13 samples once a second against a named reference, so faults between samples can be missed.

By the solid-red trial, the rig had judged more than two hundred one-second samples.

The agent proposes; a person grants

When a session asked to run the vendor’s old start-up recipe, which writes registers directly, I approved it as a supervised comparison against the card we knew worked. The runner that drives a trial labels what it saw and names a candidate follow-up. Its documentation is explicit: a suggestion is not authorization to run it.

What the loop said about solid red

A session’s summary said the board had been asked for color bars and had shown the stripe, and I asked whether color bars were really what we had meant to request. Two days later a cut-down driver reproduced the stripe, and I thought the bars might be partly coming through, so I asked for a different image: solid red.

A session changed only the pattern the main chip’s generator draws, from color bars to solid red, and confirmed from register snapshots that the new setting had been programmed. This earlier trial recorded 15 seconds in total. The verdict: 13 of 13 settled samples were the stripe, and none was red.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

{
  "settled_samples": 13,
  "red_matches": 0,
  "classes": ["pinstripe"],
  # ...
  "timing": {
    "strict_valid": false,
    "diagnostic_valid": true,
    "caveat": "startup timestamp anomaly within 2s; not strict acceptance"
  }
}

From the trial’s own result file, with the color averages and frame intervals left out. The test script itself exited 0: a process exit is not a verdict.

The verdict closed my hypothesis without my eyes. The file carries both verdicts: the recorded picture did not follow the request, and the timing was good enough for diagnosis but not for a strict pass.

This was one warm boot with diagnostic-only timing, in a manual window I had opened with the board-sharing service stopped. I make no claim about a cold start.

The requested picture came later, with different link settings.

The rule I’d carry to any rig: give the agent a verdict it can read. Keep the questions, and the limits on what it may do, with a person.