From f145b404719fc60f5432c911b896030cd754f73f Mon Sep 17 00:00:00 2001 From: Sads Date: Sat, 13 Jun 2026 22:24:28 +0200 Subject: [PATCH] Docs: add CLAUDE.md build/run/drive loop; fix Vita3K MCP Wayland env + frame storage - New CLAUDE.md documenting the build -> install -> launch -> drive -> logs loop. - session_start must force SDL_VIDEODRIVER=wayland + empty DISPLAY, else Vita3K renders to the host X display (blank frames, dropped input). - Native build now targets build/ instead of build-native/. - Capture frames into agent-run/frames/ (wiped each session) and clear the /tmp/kwin-mcp-screenshots-* folder afterwards instead of leaving them in /tmp. Co-Authored-By: Claude Opus 4.8 --- AgenticLoop.md | 34 +++++++---- CLAUDE.md | 160 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 183 insertions(+), 11 deletions(-) create mode 100644 CLAUDE.md diff --git a/AgenticLoop.md b/AgenticLoop.md index 37f8de6..9dc5838 100644 --- a/AgenticLoop.md +++ b/AgenticLoop.md @@ -68,18 +68,29 @@ tool works there. (Connecting to the live desktop with `session_connect` does the only reliable way to capture, and it is headless: the human cannot watch it live on their monitor.) -- `session_start` with `app_command: "vita3k -r VIMM00001"` **and - `keep_screenshots: true`** — starts an isolated Wayland session, launches the - app, and keeps every captured frame on disk after `session_stop`. Give it ~12s - to boot. +- **Before starting, wipe the per-session frame folder** so each run is clean: + `rm -rf agent-run/frames && mkdir -p agent-run/frames`. (`agent-run/` is + git-ignored; `frames/` is the throwaway capture folder — keepers live in + `agent-run/` proper and are never touched.) +- `session_start` with `app_command: "vita3k -r VIMM00001"`, + `keep_screenshots: true`, **and `env: {"SDL_VIDEODRIVER": "wayland", "DISPLAY": + ""}`** — starts an isolated Wayland session, launches the app, and keeps every + captured frame on disk after `session_stop`. Without that `env` the frames come + back blank white and input is dropped (host `DISPLAY=:1` sends SDL to X11). Give + it ~12s to boot. - **`focus_window` with `app_name: "Vita3K"` BEFORE sending keys.** Without focus the keypresses are dropped. Presses also drop intermittently — re-focus and re-press. Send one key per call; rapid bursts get coalesced/dropped. - `keyboard_key` with `screenshot_after_ms: [ms]` after **every** command, so the frame folder is a complete visual record of the run for the human to scrub. -- `session_stop` when done (and `pkill -9 -f vita3k`). Frames survive in - `/tmp/kwin-mcp-screenshots-*/`; copy them to `./agent-run/` (git-ignored) and - tell the human where they are. +- `session_stop` when done (and `pkill -9 -f vita3k`). kwin-mcp always writes + frames to `/tmp/kwin-mcp-screenshots-*/` (that path is not configurable); don't + leave them there — **move them into the repo and clear the tmp folder**: + ```bash + cp /tmp/kwin-mcp-screenshots-*/*.png agent-run/frames/ 2>/dev/null + rm -rf /tmp/kwin-mcp-screenshots-* + ``` + Then tell the human the frames are in `agent-run/frames/`. ### Zero-context-cost capture (the standing rule) @@ -110,7 +121,8 @@ Vita3K maps host keys → Vita buttons. The ones this app uses: ### Typical verify loop (capture everything, Read nothing) -1. `session_start` (`keep_screenshots: true`) → wait ~12s. +1. `rm -rf agent-run/frames && mkdir -p agent-run/frames`, then `session_start` + (`keep_screenshots: true`, `env` forcing Wayland) → wait ~12s. 2. `focus_window "Vita3K"`. 3. `keyboard_key "Right"` (with `screenshot_after_ms`) … stepping to the target tile, one key per call. @@ -119,6 +131,6 @@ Vita3K maps host keys → Vita buttons. The ones this app uses: - `~/.local/share/Vita3K/Vita3K/ux0/data/vitaimmich/log.txt` (the app's own log — records page fetches, video open/play, scans, errors) - MCP `read_app_log` with the launched PID, or `/tmp/vita3k-runtime.log`. -6. On `session_stop`, the captured frames remain in - `/tmp/kwin-mcp-screenshots-*/` — hand that folder (or a copy in `./agent-run/`) - to the human as the visual record. +6. On `session_stop`, move the frames out of `/tmp` and clear it: + `cp /tmp/kwin-mcp-screenshots-*/*.png agent-run/frames/ 2>/dev/null && rm -rf /tmp/kwin-mcp-screenshots-*`. + Hand `agent-run/frames/` to the human as the visual record. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..2b45925 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,160 @@ +# CLAUDE.md — vitaImmich + +PS Vita Immich client (PoC). C sources in `src/` (`main.c`, vendored `jsmn.h`). +Built with the VitaSDK toolchain into a `.vpk`, run in the **Vita3K** emulator, +and driven headlessly through the **kwin-mcp** server. This file is the +end-to-end loop: **build → install → launch → drive → read logs.** For the +narrative version see `AgenticLoop.md`; full key map is in `controls.MD`. + +App title id: **`VIMM00001`**. + +--- + +## 1. Build the changes → a `.vpk` + +After editing anything in `src/` (or `CMakeLists.txt`), rebuild. + +**Preferred — Docker (reproducible), writes `build/vitaImmich.vpk`:** + +```bash +./build.sh +``` + +> `build/` ends up root-owned. If the Docker runtime is broken (containerd shim +> `readlink /proc/self/exe: no such file or directory`), use the native fallback. + +**Fallback — native host SDK, also writes `build/vitaImmich.vpk`:** + +```bash +export VITASDK="$HOME/vitasdk" +export PATH="$VITASDK/bin:$PATH" +cmake -B build -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_TOOLCHAIN_FILE="$VITASDK/share/vita.toolchain.cmake" . +cmake --build build --parallel "$(nproc)" +``` + +> If a prior Docker build left `build/` root-owned, clear it first +> (`sudo rm -rf build`) so the native build can write there. + +Whichever path you use, the next step installs that `.vpk`. + +--- + +## 2. Install the rebuilt VPK into Vita3K + +Installing = launch Vita3K once with the `.vpk` as an argument; it extracts and +auto-boots. `deploy.sh` does install-then-launch in one shot: + +```bash +./deploy.sh # installs ./build/vitaImmich.vpk, then launches VIMM00001 detached +``` + +Gotchas (these are why it must run from a script, not inline): + +- Background it from a **script file** with a `sleep`/`wait` then `pkill`. Inline + backgrounding from the agent's shell does not persist. +- Guard `pkill` with `|| true` — a no-match exit code aborts the chain. +- Vita3K preserves the **zip's internal mtime** on extract. To confirm a *fresh* + install, compare the installed `eboot.bin` mtime against the vpk entry, not + wall-clock: + + ```bash + unzip -l build/vitaImmich.vpk | grep eboot + stat -c 'eboot: %y' ~/.local/share/Vita3K/Vita3K/ux0/app/VIMM00001/eboot.bin + ``` + +- Wipe a stale install: `vita3k -d VIMM00001`. + +> `deploy.sh` launches detached on the **live desktop**. For agent-driven +> verification with screenshots, **skip `deploy.sh`'s launch** and instead launch +> inside the MCP's isolated session (step 3) — only do the *install* half first +> (`vita3k ./build/vitaImmich.vpk` for ~4s, then `pkill`). + +--- + +## 3. Launch + drive via kwin-mcp (screenshots + keystrokes) + +Launch the app **inside kwin-mcp's own isolated KWin session** — its screenshot +tool only works there. `session_connect` to the live desktop does **not** capture +(spectacle can't save non-interactively; KWin's `ScreenShot2` DBus rejects +unauthorized callers). The isolated session is headless: the human cannot watch +it live, which is exactly why we capture every frame to disk. + +0. **Wipe the per-session frame folder first:** + `rm -rf agent-run/frames && mkdir -p agent-run/frames`. (`agent-run/` is + git-ignored; `frames/` is the throwaway capture folder — your curated keepers + stay in `agent-run/` proper and are never touched.) +1. **`session_start`** with `app_command: "vita3k -r VIMM00001"`, + `keep_screenshots: true`, **and `env: {"SDL_VIDEODRIVER": "wayland", + "DISPLAY": ""}`** — starts the isolated session, launches the app, and keeps + every captured frame after `session_stop`. Give it **~12s** to boot. + + > **The `env` is mandatory.** The session inherits the host's `DISPLAY=:1`, so + > without it Vita3K's SDL backend picks X11 and renders to the *host* display — + > the captured frames come back **blank white**, `focus_window "Vita3K"` fails + > ("no application matching"), and keystrokes never reach the app. Forcing + > `SDL_VIDEODRIVER=wayland` and clearing `DISPLAY` makes Vita3K render into the + > MCP's Wayland session, where capture and input both work. Sanity check: + > `list_windows` should show a `Vita3K` window and frames should be ~300–500 KB, + > not ~8 KB. +2. **`focus_window` with `app_name: "Vita3K"` BEFORE every key.** Without focus, + keypresses are dropped. They also drop intermittently — re-focus and re-press. + **One key per call**; rapid bursts get coalesced/dropped. +3. **`keyboard_key`** with `screenshot_after_ms: ` after **every** command, + so the frame folder is a complete visual record. +4. **`session_stop`** when done, then `pkill -9 -f vita3k`. kwin-mcp always writes + frames to `/tmp/kwin-mcp-screenshots-*/` (not configurable) — move them into the + repo and clear tmp, then tell the human where they are: + ```bash + cp /tmp/kwin-mcp-screenshots-*/*.png agent-run/frames/ 2>/dev/null + rm -rf /tmp/kwin-mcp-screenshots-* + ``` + +### Key map (this app's actions; full list in `controls.MD`) + +| Vita button | host key (kwin-mcp) | in-app action | +|-------------|------------------------------|----------------------| +| D-pad | `Right` `Left` `Up` `Down` | move grid / browse | +| Cross (X) | `x` | view / play / replay | +| Circle (O) | `c` | back | +| Select | `Shift_R` | sync | +| Start | `Return` | exit | + +--- + +## 4. Read the logs — verify from TEXT, not frames + +**Capturing a frame is free** (PNGs on disk are not in context). **Reading a PNG +back into context is the expensive part.** So capture every command, but **do not +Read frames back** during the loop. Verify progress from the app's own **text** +log instead: + +``` +~/.local/share/Vita3K/Vita3K/ux0/data/vitaimmich/log.txt +``` + +This is the app's own log — it records page fetches, video open/play, scans, and +errors. Also available: + +- MCP **`read_app_log`** with the launched PID. +- `/tmp/vita3k-runtime.log` (Vita3K runtime stdout/stderr from `deploy.sh`). +- `/tmp/vita3k-install.log` (install output). + +**Only Read a frame** when a screenshot is genuinely the only way to resolve an +ambiguity (rare). Reading every frame is what bloats context — that's what this +rule avoids. + +### Typical verify loop + +1. Build (step 1) → install half of step 2. +2. `rm -rf agent-run/frames && mkdir -p agent-run/frames`, then `session_start` + (`keep_screenshots: true`, `env` forcing Wayland) → wait ~12s. +3. `focus_window "Vita3K"`. +4. `keyboard_key "Right"` (+ `screenshot_after_ms`), stepping to the target tile, + one key per call; `x` to open, `x` again to play. +5. Confirm what happened from `log.txt` — **not** the frames. +6. `session_stop` + `pkill`; move frames into the repo and clear tmp + (`cp /tmp/kwin-mcp-screenshots-*/*.png agent-run/frames/ 2>/dev/null && rm -rf /tmp/kwin-mcp-screenshots-*`), + then hand `agent-run/frames/` to the human as the visual record. + +---