Video: show real duration, stay on replay prompt at end

Fix the playback HUD showing "/ 0:00": duration is re-polled in the
render loop until sceAvPlayerGetStreamInfo reports it, instead of being
latched once before async parsing finishes.

When a video reaches its end, hold on the last frame with an
"X replay / O back" prompt instead of returning to the detail screen
(which, for server videos, forced a full re-download to watch again).
X replays from the still-local file via re-init; the decoder claim is
held across replays.

Also add the agentic loop tooling/docs: AgenticLoop.md runbook
(build, install, drive Vita3K via kwin-mcp, zero-context-cost frame
capture), controls.MD key map, deploy.sh, .mcp.json, and gitignore
build-native/ and agent-run/.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Sads
2026-06-12 18:13:03 +02:00
co-authored by Claude Opus 4.8
parent 1f6f913b49
commit 6c7cc4b322
6 changed files with 241 additions and 7 deletions
+2
View File
@@ -1 +1,3 @@
build/ build/
build-native/
agent-run/
+8
View File
@@ -0,0 +1,8 @@
{
"mcpServers": {
"kwin-mcp": {
"command": "uvx",
"args": ["kwin-mcp"]
}
}
}
+124
View File
@@ -0,0 +1,124 @@
# Agentic loop: build, run & drive vitaImmich in Vita3K
This is the loop an agent uses to make a change, run it in the Vita3K emulator,
and verify it by sending input + reading screenshots — all headless, on KDE
Plasma (Wayland).
## One-time setup
- VitaSDK (community): native toolchain at `~/vitasdk` (`arm-vita-eabi-gcc`,
`vita-pack-vpk`, toolchain at `share/vita.toolchain.cmake`).
- Vita3K installed and on `PATH` (`/usr/bin/vita3k`).
- The MCP that drives the GUI: `uv tool install kwin-mcp` (configured in
`.mcp.json` as the `kwin-mcp` server).
## 1. Build the VPK → `build/vitaImmich.vpk`
Preferred (Docker, reproducible):
```bash
./build.sh # writes build/vitaImmich.vpk (build/ ends up root-owned)
```
Fallback when the Docker runtime is broken (e.g. containerd shim error
`readlink /proc/self/exe: no such file or directory`), build natively with the
host SDK into `build-native/` (git-ignored):
```bash
export VITASDK="$HOME/vitasdk"
export PATH=$VITASDK/bin:$PATH
cmake -B build-native -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_TOOLCHAIN_FILE=$VITASDK/share/vita.toolchain.cmake .
cmake --build build-native --parallel "$(nproc)"
```
## 2. Install the VPK into Vita3K
Installing = launch Vita3K once with the `.vpk` as an argument; it extracts and
auto-boots. Gotchas:
- Run it **from a script file**, backgrounded with a wait then killed. Inline
backgrounding from the agent's shell does not persist.
- Guard `pkill` with `|| true` — a no-match exit code aborts the chain.
- Vita3K extracts files preserving the **zip's internal mtime**, so to confirm a
fresh install compare the installed `eboot.bin` mtime against
`unzip -l <vpk> | grep eboot`, not wall-clock time.
```bash
#!/bin/bash
cd "$(git rev-parse --show-toplevel)"
pkill -9 -f vita3k 2>/dev/null || true
sleep 1
vita3k ./build-native/vitaImmich.vpk >/tmp/vinst.log 2>&1 & # or ./build/...
sleep 8
pkill -9 -f vita3k 2>/dev/null || true
sleep 1
grep -i "successfully" /tmp/vinst.log | tail -1
stat -c 'eboot: %y' ~/.local/share/Vita3K/Vita3K/ux0/app/VIMM00001/eboot.bin
```
App title id: **`VIMM00001`**. To wipe a stale install: `vita3k -d VIMM00001`.
## 3. Launch + drive via kwin-mcp
Launch the app **inside the MCP's own isolated KWin session** — its screenshot
tool works there. (Connecting to the live desktop with `session_connect` does
*not* work for capture: `spectacle` can't save non-interactively, and KWin's
`ScreenShot2` DBus API rejects unauthorized callers — so the isolated session is
the only reliable way to capture, and it is headless: the human cannot watch it
live on their monitor.)
- `session_start` with `app_command: "vita3k -r VIMM00001"` **and
`keep_screenshots: true`** — starts an isolated Wayland session, launches the
app, and keeps every captured frame on disk after `session_stop`. Give it ~12s
to boot.
- **`focus_window` with `app_name: "Vita3K"` BEFORE sending keys.** Without focus
the keypresses are dropped. Presses also drop intermittently — re-focus and
re-press. Send one key per call; rapid bursts get coalesced/dropped.
- `keyboard_key` with `screenshot_after_ms: [ms]` after **every** command, so the
frame folder is a complete visual record of the run for the human to scrub.
- `session_stop` when done (and `pkill -9 -f vita3k`). Frames survive in
`/tmp/kwin-mcp-screenshots-*/`; copy them to `./agent-run/` (git-ignored) and
tell the human where they are.
### Zero-context-cost capture (the standing rule)
Capturing a frame (writing the PNG) costs the agent **nothing** — files on disk
are not in context. The cost is only incurred when the agent **Reads** a PNG
back. So:
- **Capture every command** (`screenshot_after_ms`) — free, and it gives the
human the full record.
- **Do not Read frames back** into context during the loop. Verify progress from
the app's own **text** log instead (cheap), not from images:
`~/.local/share/Vita3K/Vita3K/ux0/data/vitaimmich/log.txt`.
- Only Read a frame when a screenshot is genuinely the only way to resolve an
ambiguity (rare). Reading every frame is what bloats context and slows the
session — that is exactly what this rule avoids.
### Key mapping (full list in `controls.MD`)
Vita3K maps host keys → Vita buttons. The ones this app uses:
| Vita button | host key (kwin-mcp) | in-app action |
|-------------|---------------------|--------------------------|
| D-pad | `Right` `Left` `Up` `Down` | move grid / browse |
| Cross (X) | `x` | view / play / replay |
| Circle (O) | `c` | back |
| Select | `Shift_R` | sync |
| Start | `Return` | exit |
### Typical verify loop (capture everything, Read nothing)
1. `session_start` (`keep_screenshots: true`) → wait ~12s.
2. `focus_window "Vita3K"`.
3. `keyboard_key "Right"` (with `screenshot_after_ms`) … stepping to the target
tile, one key per call.
4. `keyboard_key "x"` to open, `x` again to play.
5. Confirm what happened from the **text** log, not the frames:
- `~/.local/share/Vita3K/Vita3K/ux0/data/vitaimmich/log.txt` (the app's own log
— records page fetches, video open/play, scans, errors)
- MCP `read_app_log` with the launched PID, or `/tmp/vita3k-runtime.log`.
6. On `session_stop`, the captured frames remain in
`/tmp/kwin-mcp-screenshots-*/` — hand that folder (or a copy in `./agent-run/`)
to the human as the visual record.
+29
View File
@@ -0,0 +1,29 @@
D-pad
up: ArrowUp
left : ArrowLeft
right : ArrowRight
down : ArrowDown
Left Stick
Up : W
Left : A
Right : D
Down : S
L1 : Q
L2 : E
Face Buttons:
Triangle : V
Square : Z
Circle : C
Cross : X
Right Stick:
Up : I
Left : J
Right : L
Down: K
PS Button : P
Select : Shift Right
Start : Enter
L2 : U
L3 : F
R3 : H
R2 : O
Executable
+25
View File
@@ -0,0 +1,25 @@
#!/bin/bash
set -euo pipefail
TITLE="VIMM00001"
cd "$(dirname "$0")"
pkill -9 -f vita3k 2>/dev/null || true
sleep 1
# Install
vita3k ./build/vitaImmich.vpk >/tmp/vita3k-install.log 2>&1 &
sleep 4
pkill -9 -f vita3k 2>/dev/null || true
sleep 1
# Launch detached
nohup vita3k -r "$TITLE" \
>/tmp/vita3k-runtime.log 2>&1 \
</dev/null &
PID=$!
echo "{\"status\":\"live\",\"pid\":$PID,\"title\":\"$TITLE\"}"
+53 -7
View File
@@ -1622,6 +1622,12 @@ static void play_video_file(const char *path)
for (int i = 0; i < 720 && g_poster_active; i++) for (int i = 0; i < 720 && g_poster_active; i++)
show_status("Starting video..."); show_status("Starting video...");
/* replay re-inits the player from the same (already-local) file, so the
* decoder claim above is taken once and held across replays. */
int want_replay;
replay:
want_replay = 0;
SceAvPlayerInitData init; SceAvPlayerInitData init;
memset(&init, 0, sizeof(init)); memset(&init, 0, sizeof(init));
init.memoryReplacement.allocate = av_alloc; init.memoryReplacement.allocate = av_alloc;
@@ -1673,13 +1679,10 @@ static void play_video_file(const char *path)
for (int i = 0; i < 300 && !(active = sceAvPlayerIsActive(g_avp)); i++) for (int i = 0; i < 300 && !(active = sceAvPlayerIsActive(g_avp)); i++)
show_status("Starting video..."); show_status("Starting video...");
/* AddSource parses asynchronously, so the stream duration is usually not
* available yet at this point; keep re-querying it in the loop below until
* it becomes known instead of latching the initial 0. */
uint64_t duration = 0; uint64_t duration = 0;
if (active) {
SceAvPlayerStreamInfo sinfo;
memset(&sinfo, 0, sizeof(sinfo));
if (sceAvPlayerGetStreamInfo(g_avp, 0, &sinfo) >= 0)
duration = sinfo.duration;
}
/* four rotating frame wrappers (matching numOutputVideoFrameBuffers): /* four rotating frame wrappers (matching numOutputVideoFrameBuffers):
* while the GPU still samples one frame the decoder fills another, and * while the GPU still samples one frame the decoder fills another, and
@@ -1694,6 +1697,7 @@ static void play_video_file(const char *path)
vita2d_texture *cur = NULL; vita2d_texture *cur = NULL;
int paused = 0; int paused = 0;
int ended_eos = 0;
unsigned int prev = 0xffffffff; /* swallow the X press that got us here */ unsigned int prev = 0xffffffff; /* swallow the X press that got us here */
while (active) { while (active) {
@@ -1721,8 +1725,10 @@ static void play_video_file(const char *path)
sceAvPlayerJumpToTime(g_avp, t); sceAvPlayerJumpToTime(g_avp, t);
} }
if (!sceAvPlayerIsActive(g_avp)) if (!sceAvPlayerIsActive(g_avp)) {
ended_eos = 1;
break; /* end of stream */ break; /* end of stream */
}
if (sceAvPlayerGetVideoData(g_avp, &vframe[buf_idx])) { if (sceAvPlayerGetVideoData(g_avp, &vframe[buf_idx])) {
sceGxmTextureInitLinear(&vtex[buf_idx].gxm_tex, sceGxmTextureInitLinear(&vtex[buf_idx].gxm_tex,
@@ -1740,6 +1746,13 @@ static void play_video_file(const char *path)
if (cur) if (cur)
draw_texture_fitted(cur, 0, 0, SCREEN_W, SCREEN_H); draw_texture_fitted(cur, 0, 0, SCREEN_W, SCREEN_H);
if (duration == 0) {
SceAvPlayerStreamInfo sinfo;
memset(&sinfo, 0, sizeof(sinfo));
if (sceAvPlayerGetStreamInfo(g_avp, 0, &sinfo) >= 0)
duration = sinfo.duration;
}
unsigned int cs = (unsigned int)(sceAvPlayerCurrentTime(g_avp) / 1000); unsigned int cs = (unsigned int)(sceAvPlayerCurrentTime(g_avp) / 1000);
unsigned int ds = (unsigned int)(duration / 1000); unsigned int ds = (unsigned int)(duration / 1000);
char hud[160]; char hud[160];
@@ -1761,6 +1774,36 @@ static void play_video_file(const char *path)
"check Immich transcoding settings."); "check Immich transcoding settings.");
} }
/* reached the end of the video: hold on the last frame with a replay
* prompt instead of returning (which, for a server video, drops the user
* back to the detail screen and would re-download to play it again). X
* replays from the still-local file; O leaves. The decoder buffers behind
* `cur` are still valid here — teardown happens below, after this loop. */
if (ended_eos) {
unsigned int eprev = 0xffffffff;
for (;;) {
SceCtrlData pad;
sceCtrlPeekBufferPositive(0, &pad, 1);
unsigned int pressed = pad.buttons & ~eprev;
eprev = pad.buttons;
if (pressed & SCE_CTRL_CIRCLE)
break;
if (pressed & SCE_CTRL_CROSS) {
want_replay = 1;
break;
}
vita2d_start_drawing();
vita2d_clear_screen();
if (cur)
draw_texture_fitted(cur, 0, 0, SCREEN_W, SCREEN_H);
draw_hud("ended X replay O back");
vita2d_end_drawing();
vita2d_swap_buffers();
}
}
g_av_audio_run = 0; g_av_audio_run = 0;
if (audio_thid >= 0) { if (audio_thid >= 0) {
sceKernelWaitThreadEnd(audio_thid, NULL, NULL); sceKernelWaitThreadEnd(audio_thid, NULL, NULL);
@@ -1771,6 +1814,9 @@ static void play_video_file(const char *path)
sceAvPlayerStop(g_avp); sceAvPlayerStop(g_avp);
sceAvPlayerClose(g_avp); sceAvPlayerClose(g_avp);
if (want_replay)
goto replay; /* re-init from the same file; no re-download */
/* release the decoder so the worker can resume poster extraction */ /* release the decoder so the worker can resume poster extraction */
g_player_active = 0; g_player_active = 0;
__sync_synchronize(); __sync_synchronize();