AI Workflow 02 · AI Quality Assurance · Declassified
Pip: an AI QA agent that tests like a 7-year-old and a teacher
Reports up · never shipsI built Stackadoo with AI coding agents. That's fast, but it makes bugs. Pip plays the game the way a real kid would, then again the way a picky teacher would.
ON THIS PAGE
01 Problem · 02 Setup · 03 Steps · 04 Tools
05 Results · 06 What you get · 07 Try it ▸ · 08 Talk
01 · The Problem
Kids don't file bug reports. They just stop playing.
Stackadoo is a kids' math and building game (stackadoo.com and the Android app). I built it with AI coding agents.
That's fast, but it creates bugs and UI clutter, and I couldn't replay every screen after every change.
When a kid gets stuck, they don't tell you. They close the app. I needed a tester who notices what a 7-year-old notices.
02 · The Setup
Two lenses, every session
Pip plays and reports, but doesn't ship code. Cobble turns findings into fix briefs and PRs and checks back with Pip. Riley routes the work. Cody hands bigger fixes to ChatGPT/Codex or Claude.
| Lens | Who Pip pretends to be | What it catches |
|---|---|---|
| Kid lens | A 7-year-old first-timer. Taps the shiny thing, skips long text, freezes if the next step isn't obvious. | Confusion, dead ends, scary or unclear moments |
| Pro lens | An adult QA tester and parent-teacher. | Grading bugs, layout breaks, settings that don't stick |
- Every finding is tagged kid-UX, adult-bug, or both.
- Focus areas: first-run scaffolding, house and stairs interiors, and pet-shop "buy everything" spam.
- Report up, don't ship.
- No faked metrics. Pip must never pad Google Play closed-test engagement.
- Hands off real players' data. My kids' account is look-only.
03 · The Steps
How it runs, step by step
- Wrote the dual-lens brief: personas, tags, and focus areas.
- Sep 2: a full dual-lens playtest of the web game. Setup, Math Meadow, Reading Town, building, shops, and quizzes, with screenshots of every step. 17 findings, ranked by severity.
- Top priorities: the quiz showing "Oops" on correct answers (8×8→64, 5+54→59, 34−15→19), the reward screen racing the next question, a scary "Gloomies" mode that was on by default and didn't stay off, and Quit hidden behind a hamburger menu a 7-year-old won't find.
- Cobble turned these into fix briefs. Fixes shipped as GitHub PRs #482–#486.
- Pip verified on the branch before merge, then again live. It caught one case (15÷3→5) still failing live, and Cobble hardened the fix. Final live batch: 80/80 correct answers graded Correct. Pip also confirmed the visible Leave button, a sticky Next button, and 16 starter blocks.
- Moved testing onto a real phone: DroidPilot (a phone-control MCP server + Android app) on a Samsung Galaxy S24 Ultra, reached over Tailscale. Pip read the on-screen UI tree and took screenshots.
- Sep 7 phone-only findings: the app ran inside a browser custom tab (kids could see browser chrome), a cluttered saved-game list pushed NEW GAME below the fold, a kid could land with 0 blocks and a build tip they couldn't act on, a truncated objective tip, and unclear purple floor cubes. No crashes.
- Sep 12 clutter passes at Pixel 7 (412×915), 768px, 1024px, plus iPad Mini: competing hero buttons, nav wrapping, no phone menu, clipped card rails. In-game build 287 had a crowded HUD, NPC bubbles over the paint palette, an intro bubble covering controls, a busy shop, and the canvas filling half the iPad screen.
- Routed each clutter brief to the right fixer: Claude for the HUD, Codex for landing-page items.
- Ran the Play Console routines during closed testing: daily install tracker, reminders, reciprocal cleanup.
- QA'd partner apps too, and found real bugs for other developers: a launch crash from a test API key, a country block, and missing US units.
04 · Tools
- AI agents: QA, dev, organizer, coding bridge
- Grok
- Claude
- ChatGPT / Codex
- DroidPilot + Tailscale
- Galaxy S24 Ultra (Android 16)
- Chrome DevTools device mode
- GitHub PRs
- GitHub Pages + Cloudflare
- Google Play Console
- Screenshots + JSON findings files
05 · Results
What changed
Findings in the first playtest
Fix PRs in one day
Correct answers graded correctly, live
Screen sizes checked, plus a real device
Time saved per pass
Parent / player feedback
Also: scary mode removed from kid setup, and the Leave button made visible.
06 · What a Client Gets
If you hire me for this
- A written tester persona and checklist fitted to your real users.
- Real-device Android testing plus phone and tablet layout checks.
- Screenshot-backed findings, ranked by severity.
- A handoff format your developer (or AI coding agent) can act on.
- A re-test after each fix.
- Guardrails so testing never fakes app-store metrics or touches real user data.
07 · Try It · Interactive Workflow Example
Two lenses
Same screen, two testers. Flip between the Kid lens and the Pro lens to see different findings. Select the ones you care about, then send them to dev to get a sample fix ticket.
Read each question carefully and then choose the answer you think is correct from the four choices below, then wait for the next question.
Kid lens · tap a number or a finding to select it
Mock screen rebuilt for this demo. The findings mirror the real Sep 2 playtest.
Want something like this for your business?
Bring one workflow that feels slower or more manual than it should. We'll look at it together on a free 15-minute fit call, and I'll give you an honest answer, even if that answer is "don't automate this."