Creating my second brain

The problem with memory in the cloud

My first reflex was the memory built into the chat apps. It helps for small things: my name, my tone, a preference. For real work, it fell short.

• I couldn't see it well. The app decides what to remember, in its own words. I can't read it like a document, check why something is there, or see what changed.
• It's a summary, not my decisions. It remembers that I like short copy. It doesn't keep the brief, the reason a direction was dropped, or the correction I gave last week.
• It belongs to one app. What one assistant learned, the other one doesn't know. Switching tools meant starting over.
• Getting it out is manual. When I wanted my context somewhere else, I exported it by hand, or copied it into a prompt. Memory I have to carry around isn't really mine.

The problem with many projects

Then came the second wall. I work on several things at once: an agency, client sites, training offers, personal projects. One shared memory mixed them up. Separate project spaces isolated them so well that I ended up copying the same brand rules into each one, then forgetting to update half of them.

What I needed was in between: projects that stay apart by default, and share context on purpose. My agency's brand project owns its identity and voice. The website, the pitch deck and the training offers each declare that they read it. When the voice changes, every project that depends on it picks it up, and I can always see why an agent knows something.

How that context moves between the brain, the code and the design files while I work is the subject of the next note, Context that travels.

A memory I own

So I moved the memory out of the apps and into plain text files, in a git repo, on my own machine. No database, no vector store, no app. Search is file names, a one-line summary at the top of each file, and grep. Boring on purpose: any agent can read it today, any tool will read it years from now, and every change has a history.

The first decision was what does not go in the brain:

• ~/second-brain: the thinking and the memory. Ideas, research, briefs, decisions, copy. Text only.
• ~/dev: the code. One repo per site or app, each pointing back to its project in the brain.
• ~/design: the design files. Exports, images, PDFs, diagrams.
• ~/data-sandbox: the raw material. PDFs, transcripts, crawled pages, untouched until I import what matters.

Owning it also means not depending on one vendor:

• One instruction file for every agent. Claude Code, Codex or the next one all read the same AGENTS.md and the same skills. I can change agents tomorrow and keep my memory.
• Tools through the command line, not connectors. A CLI works the same in every agent, where a connector belongs to one app. Note 2 explains why I avoid them as much as I can.
• Hosted where I choose. Because the brain is a git repo, it syncs to a private remote instead of being exported by hand. The next step is a server I control, a VPS, so agents can run around the clock on the same brain I use every day, with no copy to keep in sync.

Context in layers

Loading everything feels safe, but it buries the one rule that matters under all the ones that don't. So context is organised in levels, from what every session needs to what almost none does:

1. The global core: who I am, the rules that apply everywhere, the corrections that matter across projects.
2. The project essentials: what the project is, my brief and my rules, its lessons.
3. The decisions that match the task, in full, with their reasons. The others stay listed.
4. Everything else on demand: details, roadmap, research, sources.
5. History, never at start: old decisions and replaced text, kept for when a past choice needs explaining.

Rules follow the same logic. A correction becomes a short lesson at the narrowest level it applies to: one project, one skill, or every project. One real example: an explanation didn't land, so I asked for one concrete scenario instead. It became "When an explanation fails, use one concrete scenario." Every agent now applies it, without me repeating it. If I have to correct the same thing again, the lesson is reopened and the agent has to find out why it was missed.

Dream: a review before memory drifts

Memory saved session after session drifts. The same rule ends up in two places with different words, a summary goes stale, a correction never reaches the file the agent actually reads.

So I added a review step called dream, after what sleep does for memory: replay, keep what matters, make connections. I run it by hand, on one project or the whole brain. It looks for forgotten corrections first, then duplicates, contradictions, stale summaries and useful links between projects. It saves the originals before touching anything, and anything that changes a commitment comes to me as a proposal. Note 2 shows what it found on its first real run.

The rules that keep it trustworthy

A memory is only useful if I trust it. These came from things that went wrong first:

• A decision comes with its reason. Without the reason, the agent follows a rule blindly. With it, it can judge a new case.
• My parts are mine. Brief, rules, my drafts: agents read them and never rewrite them. They propose next to my text, never over it.
• Facts, guesses and decisions stay apart. An agent's hypothesis written in the tone of a decision becomes a decision two sessions later. Anything inferred carries a [CHECK] tag until I confirm it.
• History is never rewritten. When a decision changes, the original is saved before it's replaced.
• Strict guardrails, free defaults. No sending, deleting, publishing or spending without my OK. Everything else is a default an agent can depart from if it says why.
• Automation earns its place. I run a routine by hand until I trust it, before it gets a schedule.

Beyond one person: a team, or a company

Nothing in this is personal by design: it's folders, text and rules about who writes what. The same structure can serve a team, people and agents together.

Today, product holds the roadmap in one tool, design its decisions in Figma comments, marketing the voice in a slide deck, engineering the constraints in their heads. Every AI assistant on that team starts from zero, each with a different, partial version of the product.

A shared brain gives them one source. Product owns the brief and the decisions, design the direction, marketing the voice, engineering the build rules. Agents read all of it and write only where they're allowed. Anyone can read everything; each team writes in its own place and proposes changes elsewhere.

A team version needs a bit more: clear ownership per folder, review for shared decisions (a pull request, like code), access rules for what shouldn't reach everyone, and someone, or a dream-like review, to tend it. At company scale, swap teams for departments. The rules don't change, only the number of owners.

What it changed

Concretely, after a few weeks of daily use:

• No more blank page. Ask for the home page hero of a client's booking app without the brain, and you typically get "Book anything, anytime, seamlessly." With it, the agent knows the readers are clinic managers, that the brief bans "seamless" and invented claims, that their real pain is no-shows. The first draft sounds like the product, and my edits go into substance instead of tone.
• I say things once. A correction becomes a lesson every agent applies.
• I switch agents freely. The memory stays; only the tool changes.
• Projects share without copy-paste. One brand, read by every project that needs it.
• I can explain my choices. Every decision has its reason and its history.
• The memory is mine. Readable, versioned, hosted where I decide.

What it costs: discipline. A decision needs its reason written down, and the memory needs a review now and then, or it fills with noise. And it doesn't replace judgment. Context makes the output relevant. Deciding whether it's right is still my job.

Close

Most people try to get better results from AI with better prompts. I get them with better memory. A prompt lasts one conversation. The brain lasts the whole project, and the next one.

Next note: how that memory travels between the brain, the code and the design files while I work, and how it stays clean.