Docs: add CLAUDE.md build/run/drive loop; fix Vita3K MCP Wayland env + frame storage
- New CLAUDE.md documenting the build -> install -> launch -> drive -> logs loop. - session_start must force SDL_VIDEODRIVER=wayland + empty DISPLAY, else Vita3K renders to the host X display (blank frames, dropped input). - Native build now targets build/ instead of build-native/. - Capture frames into agent-run/frames/ (wiped each session) and clear the /tmp/kwin-mcp-screenshots-* folder afterwards instead of leaving them in /tmp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+23
-11
@@ -68,18 +68,29 @@ tool works there. (Connecting to the live desktop with `session_connect` does
|
|||||||
the only reliable way to capture, and it is headless: the human cannot watch it
|
the only reliable way to capture, and it is headless: the human cannot watch it
|
||||||
live on their monitor.)
|
live on their monitor.)
|
||||||
|
|
||||||
- `session_start` with `app_command: "vita3k -r VIMM00001"` **and
|
- **Before starting, wipe the per-session frame folder** so each run is clean:
|
||||||
`keep_screenshots: true`** — starts an isolated Wayland session, launches the
|
`rm -rf agent-run/frames && mkdir -p agent-run/frames`. (`agent-run/` is
|
||||||
app, and keeps every captured frame on disk after `session_stop`. Give it ~12s
|
git-ignored; `frames/` is the throwaway capture folder — keepers live in
|
||||||
to boot.
|
`agent-run/` proper and are never touched.)
|
||||||
|
- `session_start` with `app_command: "vita3k -r VIMM00001"`,
|
||||||
|
`keep_screenshots: true`, **and `env: {"SDL_VIDEODRIVER": "wayland", "DISPLAY":
|
||||||
|
""}`** — starts an isolated Wayland session, launches the app, and keeps every
|
||||||
|
captured frame on disk after `session_stop`. Without that `env` the frames come
|
||||||
|
back blank white and input is dropped (host `DISPLAY=:1` sends SDL to X11). Give
|
||||||
|
it ~12s to boot.
|
||||||
- **`focus_window` with `app_name: "Vita3K"` BEFORE sending keys.** Without focus
|
- **`focus_window` with `app_name: "Vita3K"` BEFORE sending keys.** Without focus
|
||||||
the keypresses are dropped. Presses also drop intermittently — re-focus and
|
the keypresses are dropped. Presses also drop intermittently — re-focus and
|
||||||
re-press. Send one key per call; rapid bursts get coalesced/dropped.
|
re-press. Send one key per call; rapid bursts get coalesced/dropped.
|
||||||
- `keyboard_key` with `screenshot_after_ms: [ms]` after **every** command, so the
|
- `keyboard_key` with `screenshot_after_ms: [ms]` after **every** command, so the
|
||||||
frame folder is a complete visual record of the run for the human to scrub.
|
frame folder is a complete visual record of the run for the human to scrub.
|
||||||
- `session_stop` when done (and `pkill -9 -f vita3k`). Frames survive in
|
- `session_stop` when done (and `pkill -9 -f vita3k`). kwin-mcp always writes
|
||||||
`/tmp/kwin-mcp-screenshots-*/`; copy them to `./agent-run/` (git-ignored) and
|
frames to `/tmp/kwin-mcp-screenshots-*/` (that path is not configurable); don't
|
||||||
tell the human where they are.
|
leave them there — **move them into the repo and clear the tmp folder**:
|
||||||
|
```bash
|
||||||
|
cp /tmp/kwin-mcp-screenshots-*/*.png agent-run/frames/ 2>/dev/null
|
||||||
|
rm -rf /tmp/kwin-mcp-screenshots-*
|
||||||
|
```
|
||||||
|
Then tell the human the frames are in `agent-run/frames/`.
|
||||||
|
|
||||||
### Zero-context-cost capture (the standing rule)
|
### Zero-context-cost capture (the standing rule)
|
||||||
|
|
||||||
@@ -110,7 +121,8 @@ Vita3K maps host keys → Vita buttons. The ones this app uses:
|
|||||||
|
|
||||||
### Typical verify loop (capture everything, Read nothing)
|
### Typical verify loop (capture everything, Read nothing)
|
||||||
|
|
||||||
1. `session_start` (`keep_screenshots: true`) → wait ~12s.
|
1. `rm -rf agent-run/frames && mkdir -p agent-run/frames`, then `session_start`
|
||||||
|
(`keep_screenshots: true`, `env` forcing Wayland) → wait ~12s.
|
||||||
2. `focus_window "Vita3K"`.
|
2. `focus_window "Vita3K"`.
|
||||||
3. `keyboard_key "Right"` (with `screenshot_after_ms`) … stepping to the target
|
3. `keyboard_key "Right"` (with `screenshot_after_ms`) … stepping to the target
|
||||||
tile, one key per call.
|
tile, one key per call.
|
||||||
@@ -119,6 +131,6 @@ Vita3K maps host keys → Vita buttons. The ones this app uses:
|
|||||||
- `~/.local/share/Vita3K/Vita3K/ux0/data/vitaimmich/log.txt` (the app's own log
|
- `~/.local/share/Vita3K/Vita3K/ux0/data/vitaimmich/log.txt` (the app's own log
|
||||||
— records page fetches, video open/play, scans, errors)
|
— records page fetches, video open/play, scans, errors)
|
||||||
- MCP `read_app_log` with the launched PID, or `/tmp/vita3k-runtime.log`.
|
- MCP `read_app_log` with the launched PID, or `/tmp/vita3k-runtime.log`.
|
||||||
6. On `session_stop`, the captured frames remain in
|
6. On `session_stop`, move the frames out of `/tmp` and clear it:
|
||||||
`/tmp/kwin-mcp-screenshots-*/` — hand that folder (or a copy in `./agent-run/`)
|
`cp /tmp/kwin-mcp-screenshots-*/*.png agent-run/frames/ 2>/dev/null && rm -rf /tmp/kwin-mcp-screenshots-*`.
|
||||||
to the human as the visual record.
|
Hand `agent-run/frames/` to the human as the visual record.
|
||||||
|
|||||||
@@ -0,0 +1,160 @@
|
|||||||
|
# CLAUDE.md — vitaImmich
|
||||||
|
|
||||||
|
PS Vita Immich client (PoC). C sources in `src/` (`main.c`, vendored `jsmn.h`).
|
||||||
|
Built with the VitaSDK toolchain into a `.vpk`, run in the **Vita3K** emulator,
|
||||||
|
and driven headlessly through the **kwin-mcp** server. This file is the
|
||||||
|
end-to-end loop: **build → install → launch → drive → read logs.** For the
|
||||||
|
narrative version see `AgenticLoop.md`; full key map is in `controls.MD`.
|
||||||
|
|
||||||
|
App title id: **`VIMM00001`**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Build the changes → a `.vpk`
|
||||||
|
|
||||||
|
After editing anything in `src/` (or `CMakeLists.txt`), rebuild.
|
||||||
|
|
||||||
|
**Preferred — Docker (reproducible), writes `build/vitaImmich.vpk`:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./build.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
> `build/` ends up root-owned. If the Docker runtime is broken (containerd shim
|
||||||
|
> `readlink /proc/self/exe: no such file or directory`), use the native fallback.
|
||||||
|
|
||||||
|
**Fallback — native host SDK, also writes `build/vitaImmich.vpk`:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export VITASDK="$HOME/vitasdk"
|
||||||
|
export PATH="$VITASDK/bin:$PATH"
|
||||||
|
cmake -B build -DCMAKE_BUILD_TYPE=Release \
|
||||||
|
-DCMAKE_TOOLCHAIN_FILE="$VITASDK/share/vita.toolchain.cmake" .
|
||||||
|
cmake --build build --parallel "$(nproc)"
|
||||||
|
```
|
||||||
|
|
||||||
|
> If a prior Docker build left `build/` root-owned, clear it first
|
||||||
|
> (`sudo rm -rf build`) so the native build can write there.
|
||||||
|
|
||||||
|
Whichever path you use, the next step installs that `.vpk`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Install the rebuilt VPK into Vita3K
|
||||||
|
|
||||||
|
Installing = launch Vita3K once with the `.vpk` as an argument; it extracts and
|
||||||
|
auto-boots. `deploy.sh` does install-then-launch in one shot:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./deploy.sh # installs ./build/vitaImmich.vpk, then launches VIMM00001 detached
|
||||||
|
```
|
||||||
|
|
||||||
|
Gotchas (these are why it must run from a script, not inline):
|
||||||
|
|
||||||
|
- Background it from a **script file** with a `sleep`/`wait` then `pkill`. Inline
|
||||||
|
backgrounding from the agent's shell does not persist.
|
||||||
|
- Guard `pkill` with `|| true` — a no-match exit code aborts the chain.
|
||||||
|
- Vita3K preserves the **zip's internal mtime** on extract. To confirm a *fresh*
|
||||||
|
install, compare the installed `eboot.bin` mtime against the vpk entry, not
|
||||||
|
wall-clock:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
unzip -l build/vitaImmich.vpk | grep eboot
|
||||||
|
stat -c 'eboot: %y' ~/.local/share/Vita3K/Vita3K/ux0/app/VIMM00001/eboot.bin
|
||||||
|
```
|
||||||
|
|
||||||
|
- Wipe a stale install: `vita3k -d VIMM00001`.
|
||||||
|
|
||||||
|
> `deploy.sh` launches detached on the **live desktop**. For agent-driven
|
||||||
|
> verification with screenshots, **skip `deploy.sh`'s launch** and instead launch
|
||||||
|
> inside the MCP's isolated session (step 3) — only do the *install* half first
|
||||||
|
> (`vita3k ./build/vitaImmich.vpk` for ~4s, then `pkill`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Launch + drive via kwin-mcp (screenshots + keystrokes)
|
||||||
|
|
||||||
|
Launch the app **inside kwin-mcp's own isolated KWin session** — its screenshot
|
||||||
|
tool only works there. `session_connect` to the live desktop does **not** capture
|
||||||
|
(spectacle can't save non-interactively; KWin's `ScreenShot2` DBus rejects
|
||||||
|
unauthorized callers). The isolated session is headless: the human cannot watch
|
||||||
|
it live, which is exactly why we capture every frame to disk.
|
||||||
|
|
||||||
|
0. **Wipe the per-session frame folder first:**
|
||||||
|
`rm -rf agent-run/frames && mkdir -p agent-run/frames`. (`agent-run/` is
|
||||||
|
git-ignored; `frames/` is the throwaway capture folder — your curated keepers
|
||||||
|
stay in `agent-run/` proper and are never touched.)
|
||||||
|
1. **`session_start`** with `app_command: "vita3k -r VIMM00001"`,
|
||||||
|
`keep_screenshots: true`, **and `env: {"SDL_VIDEODRIVER": "wayland",
|
||||||
|
"DISPLAY": ""}`** — starts the isolated session, launches the app, and keeps
|
||||||
|
every captured frame after `session_stop`. Give it **~12s** to boot.
|
||||||
|
|
||||||
|
> **The `env` is mandatory.** The session inherits the host's `DISPLAY=:1`, so
|
||||||
|
> without it Vita3K's SDL backend picks X11 and renders to the *host* display —
|
||||||
|
> the captured frames come back **blank white**, `focus_window "Vita3K"` fails
|
||||||
|
> ("no application matching"), and keystrokes never reach the app. Forcing
|
||||||
|
> `SDL_VIDEODRIVER=wayland` and clearing `DISPLAY` makes Vita3K render into the
|
||||||
|
> MCP's Wayland session, where capture and input both work. Sanity check:
|
||||||
|
> `list_windows` should show a `Vita3K` window and frames should be ~300–500 KB,
|
||||||
|
> not ~8 KB.
|
||||||
|
2. **`focus_window` with `app_name: "Vita3K"` BEFORE every key.** Without focus,
|
||||||
|
keypresses are dropped. They also drop intermittently — re-focus and re-press.
|
||||||
|
**One key per call**; rapid bursts get coalesced/dropped.
|
||||||
|
3. **`keyboard_key`** with `screenshot_after_ms: <ms>` after **every** command,
|
||||||
|
so the frame folder is a complete visual record.
|
||||||
|
4. **`session_stop`** when done, then `pkill -9 -f vita3k`. kwin-mcp always writes
|
||||||
|
frames to `/tmp/kwin-mcp-screenshots-*/` (not configurable) — move them into the
|
||||||
|
repo and clear tmp, then tell the human where they are:
|
||||||
|
```bash
|
||||||
|
cp /tmp/kwin-mcp-screenshots-*/*.png agent-run/frames/ 2>/dev/null
|
||||||
|
rm -rf /tmp/kwin-mcp-screenshots-*
|
||||||
|
```
|
||||||
|
|
||||||
|
### Key map (this app's actions; full list in `controls.MD`)
|
||||||
|
|
||||||
|
| Vita button | host key (kwin-mcp) | in-app action |
|
||||||
|
|-------------|------------------------------|----------------------|
|
||||||
|
| D-pad | `Right` `Left` `Up` `Down` | move grid / browse |
|
||||||
|
| Cross (X) | `x` | view / play / replay |
|
||||||
|
| Circle (O) | `c` | back |
|
||||||
|
| Select | `Shift_R` | sync |
|
||||||
|
| Start | `Return` | exit |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Read the logs — verify from TEXT, not frames
|
||||||
|
|
||||||
|
**Capturing a frame is free** (PNGs on disk are not in context). **Reading a PNG
|
||||||
|
back into context is the expensive part.** So capture every command, but **do not
|
||||||
|
Read frames back** during the loop. Verify progress from the app's own **text**
|
||||||
|
log instead:
|
||||||
|
|
||||||
|
```
|
||||||
|
~/.local/share/Vita3K/Vita3K/ux0/data/vitaimmich/log.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
This is the app's own log — it records page fetches, video open/play, scans, and
|
||||||
|
errors. Also available:
|
||||||
|
|
||||||
|
- MCP **`read_app_log`** with the launched PID.
|
||||||
|
- `/tmp/vita3k-runtime.log` (Vita3K runtime stdout/stderr from `deploy.sh`).
|
||||||
|
- `/tmp/vita3k-install.log` (install output).
|
||||||
|
|
||||||
|
**Only Read a frame** when a screenshot is genuinely the only way to resolve an
|
||||||
|
ambiguity (rare). Reading every frame is what bloats context — that's what this
|
||||||
|
rule avoids.
|
||||||
|
|
||||||
|
### Typical verify loop
|
||||||
|
|
||||||
|
1. Build (step 1) → install half of step 2.
|
||||||
|
2. `rm -rf agent-run/frames && mkdir -p agent-run/frames`, then `session_start`
|
||||||
|
(`keep_screenshots: true`, `env` forcing Wayland) → wait ~12s.
|
||||||
|
3. `focus_window "Vita3K"`.
|
||||||
|
4. `keyboard_key "Right"` (+ `screenshot_after_ms`), stepping to the target tile,
|
||||||
|
one key per call; `x` to open, `x` again to play.
|
||||||
|
5. Confirm what happened from `log.txt` — **not** the frames.
|
||||||
|
6. `session_stop` + `pkill`; move frames into the repo and clear tmp
|
||||||
|
(`cp /tmp/kwin-mcp-screenshots-*/*.png agent-run/frames/ 2>/dev/null && rm -rf /tmp/kwin-mcp-screenshots-*`),
|
||||||
|
then hand `agent-run/frames/` to the human as the visual record.
|
||||||
|
|
||||||
|
---
|
||||||
Reference in New Issue
Block a user