I Turned My Notes Into a 4D JARVIS

Your markdown notes as a 3D brain you fly through with your hands. Free, MIT, runs on a Claude subscription you already have.

Repo: https://github.com/austinpeechatt/vault-brain

---

What you’re building

Point a webcam at yourself. Pinch and drag — the brain spins. Pull your fists apart — you fly in. Aim a finger at any note and hold — it opens.

Then ask it a question out loud. A dry British butler answers from your actual notes, in voice, while the camera flies to the sources so you can check its work.

The hand tracking runs entirely in your browser. No cloud gesture service, no API bill.

---

Start here (2 minutes)

git clone https://github.com/austinpeechatt/vault-brain.git
cd vault-brain
npm install

Open brain.config.json and point it at your notes:

{ "id": "notes", "label": "My Notes", "path": "~/Documents/my-vault", "include": ["."], "color": "#386cda" }

Then:

npm run dev

That’s the 3D graph, live on localhost:5173.

Stop there and it’s already worth it. Everything below is optional and independently testable — that’s the whole design. Ship whatever’s done.

Needs: Node 20+, and a folder of markdown notes that use [[wikilinks]]. An Obsidian vault is ideal. Any .md folder works.

---

The part everyone skips (and it’s 80% of the result)

The graph is only as good as the linking. If your notes are a pile of daily journals, you get a hairball and you’ll quit.

What works:

• One concept per note. A person, a project, a tool, a decision, a lesson. Distilled pages, not diaries.
• 100–500 notes is the sweet spot. Below that it looks sparse. Above ~2000 you’ll want to revisit retrieval.
• Folders become categories — and colors in the graph. First folder segment under the root wins.
• Link liberally. Every [[other-note]] is an edge. The hubs that emerge are the insight — that’s the actual reason this beats a chat window.

If you keep raw transcripts next to distilled pages, parse only the distilled layer ("include": ["wiki"]) or the graph floods with noise.

---

The hand gestures

This is the part people actually want, so it gets its own doc in the repo — GESTURES.md — with the detection math and every tuning constant.

The vocabulary that survived real testing:

Gesture	Does
🤏 pinch + drag	orbit the brain
✊✊ fists apart / together	fly in / pull back
🤏🤏 two-hand pinch spread	precision zoom
☝️ point + hold	select the note you’re aiming at
🙏 open hands together	reset and fit everything

The four things that will make yours feel broken:

1. Slow feedback reads as broken. A gesture whose effect takes a second feels non-functional even when it works perfectly. The user assumes their hand wasn’t detected and tries again, which makes it worse. My first version mapped fist-spread to expanding the force layout — technically correct, felt completely dead. Remapping it to camera distance fixed it instantly. Every gesture must move something on the next frame.

2. Don’t bind actions to static poses that neighbor other poses. A thumbs-up is one flicker away from a fist. I bound the microphone to a thumbs-up and it kept opening itself mid-zoom. Motion gestures (drag, spread) are robust because they require sustained intent. Pose-holds need big dwells and cooldowns.

3. Interlocked fingers cannot be a gesture. Seems like an obvious “clasp to reset” — but interlacing occludes both hands and the tracker drops them entirely. No recipe fixes this. Hands-together-but-flat covers the same intent.

4. Use two thresholds, not one. Engage the pinch at 0.06, only release at 0.09. Without that hysteresis it chatters on and off at the boundary. Highest-value line of code in the whole gesture layer.

Point-to-select also needs three things or it’s useless in a dense graph: a visible reticle so you can see your aim, region-expanded aim (central 55% of the frame maps to the full screen), and a sticky lock so a neighboring node can’t steal your target mid-dwell.

Needs: any webcam. Tracking is off by default so it never fights your mouse.

---

Making it cost nothing

The brain runs through the Claude Code CLI on your existing subscription — $0 per question, not API credits.

The trick is spawning one persistent process and writing turns to its stdin, rather than shelling out per question:

claude -p --input-format stream-json --output-format stream-json \
  --include-partial-messages --verbose --model sonnet \
  --system-prompt "<your JARVIS prompt>"

That does two things at once: subscription auth means no per-question bill, and keeping the process warm kills the ~5s CLI boot you’d otherwise pay every single time. First answer lands in ~8–10s, follow-ups in ~4–6s.

Two gotchas that cost me real time:

• Run it from an empty working directory. Claude Code loads any CLAUDE.md it finds, and your coding-assistant persona will leak straight into JARVIS. Test it by asking “who are you?”
• Strip ANTHROPIC_API_KEY from the child process env or billing silently switches to API credits and you won’t notice until the invoice.

Voice out is pluggable behind one env var: ElevenLabs for the premium voice, or Kokoro — an 82M model that runs locally, sounds genuinely good, and is free forever. Voice in is whisper.cpp, fully local. Questions about your private notes should never transit a third party.

Subscription + Kokoro + whisper = the whole thing runs at zero.

---

Retrieval: skip embeddings

At a few hundred notes, keyword scoring beats a vector database and takes an afternoon instead of a week.

Score every note — whole-word title match strongest, then tags, then body occurrences — then let the top hits’ graph neighbors inherit 15% of their score. Feed the top ~7 full note bodies into the prompt.

Filter stopwords. Without it, “what is X” ranks every note containing the word “what” above X itself. That bug shipped, and I found it by asking the thing a question. Revisit embeddings around 2000 notes, not before.

---

One design rule worth stealing

Make errors loud. Every API failure surfaces the real message in the UI, in red.

The tempting alternative — silently falling back when something fails — produces answers that look completely plausible and are quietly wrong. That’s worse than a visible crash, because you trust it. Learned in production, not theory.

Same category: the parser is strictly read-only on your vault. It writes exactly one file, public/graph.json. Your notes are never touched. Build it that way from line one and you’ll never be nervous pointing it at real work.

---

Full build guide

Everything above in order, with the code: BUILD-GUIDE.md in the repo. Eleven sections, architecture through costs, and every mistake already made so you don’t repeat them.

Build order that works:

1. Prepare your vault — this is the 80%
2. Parser + 3D graph → first “whoa” moment, get here fast
3. Server + brain on the subscription trick → typed Q&A working
4. Voice out, then voice in
5. Gestures last — they’re the cherry, not the cake

If you build something with this, post it in here. I want to see it.