mpmux by Mighty Painting
~/mpmux/docs ❯ less troubleshooting/contents ↓

mpmux troubleshooting#

Start with the narrowest layer: package → daemon → transport → auth → client → host control. Never include raw tokens, pairing codes, full endpoint ids, or unredacted release hashes in shared logs.

Linux client does not launch#

Check dynamic libraries:

ldd ~/.local/bin/mpmux | grep 'not found' || true

The preview bundle expects a modern glibc desktop with Wayland or X11, libxkbcommon, Xcursor, ALSA/PulseAudio, and OpenSSL 3 libraries. Launch from a terminal once to retain diagnostics. For backend selection problems, record WAYLAND_DISPLAY, DISPLAY, and XDG_SESSION_TYPE without unrelated environment secrets.

mpmux-sessiond.service fails#

systemctl --user status mpmux-sessiond.service
journalctl --user -u mpmux-sessiond.service -n 100 --no-pager
systemctl --user cat mpmux-sessiond.service

Common causes:

  • installed prefix differs from the service's ExecStart;
  • an old override references a replaced binary or unsupported feature build;
  • auth store/token/key directories are not writable by the user;
  • another daemon owns the configured socket or TCP port;
  • --iroh-listen was requested from a binary built without iroh-transport.

The supported Linux bundle builds sessiond and CLI with Iroh enabled.

When the session host cannot start as configured, it exits with status 78 and systemd does not restart it (RestartPreventExitStatus=78): the status shows status=78/CONFIG once, and the journal says why. The usual one:

mpmux-sessiond: another process serves /run/user/1000/mpmux-sessiond.sock, perhaps a session host an mpmux app started while this one was restarting.

ss -xlp | grep mpmux-sessiond.sock names that process. End it (its sessions end with it), then systemctl --user start mpmux-sessiond.service. An app newer than 0.1.11 waits for a service that is starting or restarting instead of starting a session host of its own. A unit written by an older mpmux-cli service install lacks the line: run mpmux-cli service install again (with --remote if you pair devices) to rewrite it.

Local socket is missing or stale#

systemctl --user restart mpmux-sessiond.service
ls -l "$XDG_RUNTIME_DIR/mpmux-sessiond.sock"
mpmux-cli session \
  --sessiond-socket "$XDG_RUNTIME_DIR/mpmux-sessiond.sock" status

Do not blindly delete sockets belonging to a live process. mpmux probes stale Unix sockets and removes them only after identity-safe checks.

Remote clients cannot reconnect but the local socket works#

Symptom: mpmux-cli session --sessiond-socket ... status answers, but every Android or remote Iroh client fails with a connect timeout. This is an Iroh listener failure, not an authentication failure.

Sessiond supervises its Iroh listener. It re-binds the endpoint by itself, with bounded backoff, when:

  • the accept loop ends, or
  • the endpoint had a relay connection, lost it for 90 seconds, and no client connection is active.

The endpoint identity is preserved, so clients keep the same endpoint id and no re-pairing is needed. A relay-only endpoint is re-bound even when clients are still connected, because those connections die with the relay.

Check the journal:

journalctl --user -u mpmux-sessiond.service -n 100 --no-pager |
  grep -E 'Iroh listener|Iroh endpoint'

Expected messages during recovery:

  • Iroh endpoint had no relay connection for <n>s and no active client; re-binding the endpoint
  • Iroh listener re-bound as <id> via <relay>

An endpoint that never reached a relay (for example a blocked relay with direct-only clients) is left alone, so a blocked relay cannot cause a re-bind loop. If the relay itself is down, clients cannot connect until it returns; sessiond recovers automatically afterwards.

Verify that a hosted session survives a network change#

Use the physical flap test instead of judgement by impression. It cuts the tablet's radios for a configurable number of cycles and asserts continuity:

scripts/mpmux-android-network-flap-test.sh \
  --apk .tmp/android-release-v2026091712/mpmux-0.1.0-dev.20260917.12.apk \
  --outages 4 --outage-seconds 45 --gap-seconds 25

It requires a USB adb serial, because a network adb target would be cut off by the test itself. The app is driven through its host-control TCP listener, so no tapping is needed. Radios are restored on every exit path, and the script refuses to report success when it cannot confirm they are back on.

The verdict reports one line per assertion:

  • the PTY pid is unchanged, so the session was not recreated;
  • the pane's marker counter advanced without restarting, which catches both a restarted shell and input replay;
  • exactly one Iroh client endpoint was bound for the whole run, and no request discarded or replaced it;
  • the stream was reopened;
  • the sessiond reported no stream lease commit was rejected.

Evidence (logcat, app status, sessiond journal) is written to .tmp/network-flap-<timestamp>/. A failing verdict exits non-zero.

A single long outage exercises the other policy branch:

scripts/mpmux-android-network-flap-test.sh --outages 1 --outage-seconds 95

Expect a stream-recovery-plan with a waiting for a network reason rather than an immediate retry, and recovery within seconds of the radios returning.

Android cannot reach a direct-LAN host#

First test routing from the tablet network. Guest Wi-Fi and AP/client isolation are frequent causes. A tablet can reach the internet while being unable to reach a peer such as 192.168.x.x.

Check:

  1. Host firewall permits the chosen TCP port from the trusted LAN.
  2. Host and tablet are on a network that allows client-to-client traffic.
  3. The advertised address is a host LAN address, not 127.0.0.1, 0.0.0.0, a container-only address, or an obsolete DHCP address.
  4. Sessiond was deliberately started with --tcp-allow-non-loopback and auth.

Direct-LAN mpmux TCP is authenticated plaintext. Prefer Iroh, WireGuard, SSH, or another encrypted path.

Android works through relay but not direct Iroh#

This usually indicates routing, NAT, firewall, or AP isolation rather than an application protocol failure. Relay fallback is a valid encrypted transport. Keep it enabled while fixing network configuration; do not weaken auth or change endpoint identity merely to force a direct path.

ADB USB development path#

Keep TCP bound to host loopback and reverse it into Android:

adb devices
adb reverse tcp:4075 tcp:4075
adb reverse --list

Then use 127.0.0.1:4075 inside Android. If multiple devices are attached, add -s <serial> to every ADB command. Remove temporary forwarding when finished:

adb reverse --remove tcp:4075

ADB forwarding is a development path, not release pairing evidence.

Pairing fails#

mpmux-cli pair says "auth-unavailable: sessiond was started without --auth-store": the session host runs without device auth. A plain mpmux-cli service install and the service the Linux packages ship start it that way. Switch it on (this ends the host's shells):

mpmux-cli service install --remote --restart
mpmux-cli service status   # remote: on

Otherwise:

  • Ensure the invite has not expired or already been consumed.
  • Ensure desktop and Android artifacts come from compatible releases.
  • Check sessiond is using the intended auth store and endpoint key.
  • Create a fresh invite rather than reusing or editing an old payload.
  • Check the device was not revoked.
  • For short codes, preserve every character and use the configured rendezvous.

Pairing invites contain one-time material. Device tokens are returned once and stored hashed on the host. Lost-device recovery is revoke and re-pair.

Android reconnects repeatedly#

Collect bounded logs around one transition and check for:

  • protocol/capability mismatch;
  • invalid delta or cursor gap (client should request a full snapshot);
  • network path transition;
  • revoked/incorrect device credentials;
  • final terminal exit (the stream should close and must not reconnect).

After Ctrl-D, a final running=false state and stream EOF are expected. The pane should disappear, the PTY process should exit, and sessiond should remain at idle CPU. If not, record timestamps for input, final state, EOF, delete request, and pane removal.

Host-control commands time out#

Find the current host socket through the running app/session metadata rather than reusing an old path. Host sockets include process/session identity and old paths can remain visible after crashes.

mpmux-cli status
mpmux-cli show-control-state

For Android automation, host control is normally loopback-only and exposed through an explicit ADB/TCP bridge. It is not the terminal session transport and should not be published to a LAN.

"incompatible host-control protocol version"#

A script (the todos sidebar, the run monitor, mpmux-cli) and the mpmux it talks to speak different host-control versions, usually after only one of them was updated. The message says which side is older: update mpmux on the device that shows the session, or the script (mpmux-cli agents install refreshes the agent integrations; a package update refreshes mpmux-cli).

Fonts, emoji, or Kitty images look wrong#

Confirm the release bundle retained share resources and was installed with its helper rather than copying only the executable. On Android, verify APK assets with:

scripts/mpmux-android-build-smoke-verify.sh

Kitty image stores are bounded; eviction under a large workload is expected, but crashes, unbounded memory growth, or stale placements are not.

Workspace changeover looks wrong#

Active on another device means another client owns the workspace: the former owner keeps the pane processes but does not accept input. It never reclaims automatically; use the explicit Take Over action on the device that should own it.

The former owner's panes stay frozen on the last frame they received. This is expected: the daemon only admits a stream for the current owner, so there is no read-only live view. Each frozen pane shows a View only badge naming the owning device. A frozen pane that also fails to reattach is still the expected outcome until you take the workspace over.

  • Workspace ownership unknown — a takeover response was lost and the outcome could not be established. Check mpmux-cli session takeover-status --workspace-id <id> before retrying; do not retry blindly.
  • the session host does not support safe workspace takeover — the daemon is older than workspace-takeover-v1. Update mpmux-sessiond; do not fall back to manual detach/attach, which can split ownership.
  • workspace-ownership-conflict / workspace-membership-conflict — another transfer or a pane-membership change won. Re-read ownership and retry once.
  • A transfer that reports success but leaves a pane disconnected is a post-commit destination failure: the workspace is owned but offline. Keep the state visible and reconnect; do not delete the host session.
  • session restore incomplete: N of M panes have no snapshot (<sessions>) — the layout restored, but some sessions attached without producing a snapshot, so those panes are empty until their streams deliver state. This is a partial restore, not a failed one; the tab's status names the affected sessions.
  • session restore failed: hosted workspace references unknown session <id> — the host workspace refers to a session the daemon no longer has. The previous runtime is kept; re-read the workspace before retrying.

See docs/HANDOVER_VALIDATION.md for the hardware runbook and docs/SESSION_ATTACH_PROTOCOL.md section 14 for the contract.

The Viewer#

  • A page shows its source instead of the page (Linux desktop). The web engine is not installed: press Install web engine over the source, or run Install Web Engine from the command palette. It downloads about 320 MB once, to ~/.local/share/mpmux/web-engine. If the install fails (the Viewer says "The web engine install failed: …"), check the network and run it again; deleting that folder starts from scratch.
  • A link in a document says "… does not carry …". The file did not exist, or was outside the document's folder, when the document was shown. mpmux-cli media show names such links on standard error. Build the file (for the Wasm examples: cargo build --release --target wasm32-unknown-unknown --examples) and show the document again.
  • "The guest did not finish a frame in 2 s and was stopped". A Wasm canvas guest hangs or computes too much in one frame. Time it with cargo run --release -p mpmux-wasm-canvas --example measure -- guest.wasm and do less per frame. "The guest stopped: …" means it trapped (a bug in the guest); the message names the reason.
  • media show says the file is too large. An object is at most 8 MB, 640 KB from standard input or over TCP; a page or a document with its linked files at most 64 files and 24 MB.
  • No voice input on Ubuntu or Debian. The .deb has no Whistle (its engine needs a newer glibc than the package's base); the Arch package, the macOS app on Apple silicon and the Android app have it.

Safe diagnostic bundle#

Useful non-secret facts:

  • mpmux version/release manifest commit;
  • OS, architecture, display backend, Android model/API level;
  • transport class (Unix, loopback TCP/ADB, direct LAN, Iroh direct/relay);
  • service state, process/thread count, bounded CPU/RSS samples;
  • redacted error codes and transition timestamps.

Always redact:

  • admin/device tokens and token files;
  • pairing invites and short codes;
  • full Iroh endpoint ids/private keys;
  • clipboard/terminal contents;
  • signed manifest hashes when the release procedure marks them sensitive.