If you read my last piece about PRD-first development, you know I'm big on giving your AI a fixed point to work from. A document it can always return to. A north star.
But there's a problem the PRD doesn't solve.
Every time you start a new session, your AI wakes up with no memory of you. Doesn't know your projects. Doesn't know your stack preferences. Doesn't know that you spent three hours last Tuesday debugging a specific issue and landed on a specific fix. You either re-explain everything from scratch, or you paste in context and hope for the best.
For a while I just lived with it. Then it started costing me real time.
The amnesia problem
Here's what it looks like in practice. You build something. You make decisions, why you picked this library, why you structured the database this way, what the gotcha was with the audio pipeline. Good decisions, documented in your head.
Two weeks later you come back to the project. New session. Your AI is a stranger who's never met you or your code.
You can put notes in your CLAUDE.md, and you should. But that's project-specific. It doesn't know anything about you, your preferences, your infrastructure, your history across every other thing you've built. And it sure doesn't carry context between conversations.
I needed something that would.
The first piece: Hindsight
I found an open source project called Hindsight, built by Vectorize. It's an MIT-licensed, self-hostable memory server that connects to your AI via MCP, the same protocol Claude uses to talk to external tools. You run it locally, your AI connects to it, and suddenly it has a place to store and retrieve structured facts across sessions.
The three core operations are exactly what they sound like: retain (store something), recall (search for something), reflect (reason across what's stored). Each project gets its own memory bank. When I start a conversation about an app I'm building, the relevant bank loads in, stack decisions, port assignments, known gotchas, milestone history. The AI arrives already knowing the shape of what we're working on.
It solved the technical memory problem completely. But I realized pretty quickly that technical facts were only half of what I was missing.
(If you want to check it out: github.com/vectorize-io/hindsight)
The second piece: Ib
The other half is harder to describe. It's the relational stuff. How I like to work. What I'm currently focused on. Decisions I've made across projects and why. The context that makes an interaction feel continuous rather than like starting over every single time.
Hindsight wasn't designed for that, it's a structured fact store, and that's what makes it good at its job. So I built a second system called Ib to sit alongside it.
Ib is fuzzier and more personal. It accumulates context over time, surfaces what's actually relevant to the current conversation, and quietly lets things decay when they stop being useful. Less like a database, more like working with someone who's been paying attention. It runs its own curation pass using a local LLM, so nothing leaves my machine.
The two systems split the job cleanly:
- Hindsight, technical facts, by project. Structured, reliable, retrievable on demand.
- Ib, relational context, across everything. Soft, cumulative, personal.
What it feels like
Let me give you a specific example.
The other night I was more than halfway through building an app. Not planning it, actually building it. We'd been deep in it long enough that we'd even decided to rename the project mid-build, which happens. At some point I switched from the web interface over to Claude Code to keep going.
It already knew.
Not because I pasted anything in. Not because I summarized where we were. Claude Code picked up the project mid-stream, correct name and all, and just asked if we were continuing. That was it. Zero re-prompting, zero lost context.
That's the moment it stopped feeling like a trick and started feeling like infrastructure.
I don't re-explain my stack every session anymore. I don't re-litigate decisions I already made. When I reference a project by name, the AI already knows what it is, what it's built on, and what the open questions are. On the relational side, it's the difference between working with a tool and working with a collaborator.
The honest take
Most people building with AI are leaving a lot on the table by not solving the memory problem. Your workflow, your preferences, your hard-won project knowledge, it evaporates at the end of every session.
It doesn't have to.
Start with Hindsight if this sounds interesting, it's free, open source, and gets you most of the way there out of the box. The relational layer is a further step, but even just having a structured technical memory changes how this whole thing feels.
Less wrestling with a stateless chatbot. More like a build partner who actually shows up.
This is part of the infrastructure behind Axly's Customs, local-first apps, no telemetry, no subscriptions. More at axly.com/blog. And if you missed the PRD-first workflow piece that started this series, go read that one first, this builds on it.