---
title: "Beyond the Karpathy Build: The Second Brain Evolution Guide"
author: "Coach Jae Hugh"
date: 2026-10-17
status: published-guide-source
audience: founders, coaches, consultants, solopreneurs, operators, AI assistants
live_url: https://secondbrain.johnhughesai.com
---

# Beyond the Karpathy Build

## The Second Brain Evolution Guide

The original Karpathy-style LLM wiki gave me a strong foundation: preserve raw sources, synthesize them into linked Markdown pages, and let an AI maintain the knowledge layer.

I kept building.

My second brain now handles personal memory, projects, recorded calls, AI conversations, source verification, agent context, task handoffs, review, and reusable system packaging.

The biggest change is simple. The brain no longer stores context and waits for me to retrieve it. It prepares the right context for the right AI, governs the work, records the proof, and reconciles the result into memory.

This guide explains each major upgrade, what it enabled, and how you or your AI can add the same capability.

## The base architecture

Start with two layers:

- `raw/` preserves original notes, transcripts, clips, files, and conversations.
- `wiki/` holds concise source notes, concepts, entities, syntheses, and open questions.

Add three control files:

- `AGENTS.md` tells every AI how to work inside the brain.
- `index.md` maps the important knowledge and operating areas.
- `log.md` records material changes in date order.

This foundation gives your AI persistent, traceable memory. The upgrades below turn that memory into an operating system.

## 1. Add a processed-source lifecycle

Create `raw/sources/`, `raw/processed/`, and `raw/assets/`. New material enters through `raw/sources/`. Completed items move to `raw/processed/`. Images and attachments live in `raw/assets/`.

**What this enabled:** My AI can tell which sources still need work, which ones have been processed, and where supporting files belong. The same file no longer gets ingested again by accident.

## 2. Add human operating areas

Keep the wiki layer, then add folders that match your life and work. Mine include Personal Life, Personal Brand, Divitiae Solutions, Projects, Templates, and Daily Notes.

**What this enabled:** My brain can support active work instead of answering questions from a library. Plans, decisions, assets, drafts, and current responsibilities have clear homes.

## 3. Install journal-grounded memory

Give journal entries their own folder and index. Tell your AI to read related entries before responding, ask focused follow-up questions, and extract durable patterns into the appropriate private pages.

**What this enabled:** Each reflection can build on prior context. My AI can recognize recurring patterns instead of treating every journal entry as a blank page.

## 4. Capture substantive AI conversations

Save important AI conversations twice: one raw source copy and one readable platform archive. Include the goal, decisions, corrections, resources, implementation details, and unresolved questions.

**What this enabled:** Valuable decisions stopped disappearing inside chat history. Future agents can recover the reasoning behind a system instead of seeing only the final file.

## 5. Audit conversation history

Run a duplicate-safe audit on a schedule. Search accessible conversation history, save missing substantive chats, repair incomplete archive pairs, and state which history could not be accessed.

**What this enabled:** I no longer have to remember which conversations deserve preservation. The system catches gaps without creating copies of records that already exist.

## 6. Require source-of-truth reorientation

Before substantive work, require the AI to read the index, recent log entries, project rules, and the smallest useful set of evidence files.

**What this enabled:** Current written decisions outrank the model's memory. An agent can resume work after weeks or months without asking me to repeat captured context.

## 7. Strengthen provenance and contradiction handling

Require backlinks from every material claim to its source note or raw evidence. Keep canonical URLs, dates, creator data, and final processed paths. When sources disagree, record the conflict and update the synthesis.

**What this enabled:** I can trace a claim back to evidence. New information can correct the brain without erasing the history of how the decision changed.

## 8. Build a creator intelligence layer

Create a profile for each meaningful creator, author, speaker, publisher, or channel. Track aliases, links, contact fields, observations, source notes, and a deduplicated ledger of every item added to the brain.

**What this enabled:** Source intake became relationship intelligence. I can see what I have learned from someone, what content is already captured, and where a future connection may make sense.

## 9. Add participant attribution

For interviews, podcasts, panels, and demonstrations, identify the host, guest, speaker, demonstrator, and framework owner. Link each material participant to the source and to one another.

**What this enabled:** The person who contributed the idea receives the credit. My AI can retrieve knowledge by the right expert instead of filing everything under the publisher.

## 10. Build visual dashboards

Create Obsidian canvases for projects and major operating areas. Put current decisions, source context, key files, and next actions where you can see them.

**What this enabled:** I gained a visual navigation layer. I can understand the shape and condition of an area without opening a dozen notes.

## 11. Standardize every project workspace

Give every major project four files:

- `AGENTS.md` for local rules and boundaries
- `README.md` for orientation and current status
- `TASKS.md` for the actionable queue
- `SESSION_LOG.md` for dated receipts

**What this enabled:** Any qualified AI can enter a project, understand the current gate, do bounded work, and leave the project easier to resume.

## 12. Add Git synchronization and recovery

Place the vault under version control. Run a scheduled sync that reviews changes, filters secrets and generated junk, commits valid work, and pushes it to a shared remote. Add recovery rules for unpublished history and oversized files.

**What this enabled:** The brain gained history, rollback, collaboration, and off-device continuity. Other approved assistants can work from the current canonical state.

## 13. Automate raw-source processing

Schedule an agent to inspect the intake folders, create source notes, update related pages, repair links, move completed sources, and append the log.

**What this enabled:** Captured material can become usable knowledge without waiting for a large manual cleanup session.

## 14. Create verified YouTube and web ingestion

Capture title, channel, URL, publication date, transcript status, downloads, social links, and useful description resources. Verify the exact video identity before removing it from an intake playlist.

**What this enabled:** A saved video became a traceable knowledge asset. Failed or ambiguous captures remain visible instead of producing false completion.

## 15. Connect recorded calls through NotebookLM

Use person-specific notebooks for recurring relationships, project notebooks for stable groups, and dated notebooks for one-off calls. Keep the local vault as the canonical home for decisions, assignments, and receipts.

**What this enabled:** A phone recording can become project context and completed follow-up during the same work cycle. Important commitments no longer remain trapped in audio.

## 16. Preserve externally shared artifacts

Save the exact document in the vault before or alongside emailing, attaching, or sharing it. Add its title to the index and record the delivery receipt.

**What this enabled:** My brain retains the version the other person received. Future agents can find the exact handoff without searching an inbox.

## 17. Give the brain an AI Chief of Staff

Assign one agent to own canonical context, cross-system reconciliation, contradiction handling, strategy, and final synthesis. In my system, that role is Samantha.

**What this enabled:** Notes and outputs from different assistants no longer become disconnected silos. One agent keeps the complete operating picture coherent.

## 18. Add proactive intelligence reviews

Create recurring jobs for strategic blind spots, revenue risks, source processing, conversation capture, and vault synchronization. Give each job a narrow purpose, evidence rules, and a defined destination.

**What this enabled:** The brain can surface risks and opportunities before I remember to ask. It supports decisions instead of waiting for questions.

## 19. Require a soul for each AI employee

Create a `soul.md` before or alongside an employee's first skill or schedule. Define identity, temperament, role, protections, relationships, boundaries, default ritual, and failure mode.

**What this enabled:** Task prompts no longer define the whole employee. Each AI has a stable operating identity that survives changing assignments.

## 20. Add proactive context brokerage

Before dispatch, prepare the smallest sufficient context packet. Include the employee's lane, assignment, acceptance criteria, sources, recent corrections, tool state, output destination, reviewer, prohibited actions, and freshness timestamp.

**What this enabled:** Agents receive the context before they fail. I do not have to repeat information that the brain has already captured.

## 21. Separate workforce management from final judgment

Give one role responsibility for queues, deadlines, handoffs, callbacks, and task health. Keep canonical reconciliation and high-risk review with the Chief of Staff.

**What this enabled:** Specialist work can move without sending every routine decision to me. Accountability remains clear at each stage.

## 22. Install agents in the right order

Use this sequence:

`Human owner -> second brain -> AI Chief of Staff -> work board -> workforce operator -> specialists`

Qualify each layer before adding the next one.

**What this enabled:** New specialists enter a working management system. They have context, assignments, review, and a place to return their work.

## 23. Build a live Task Command Center

Give the workforce a private operating surface with task rooms, briefs, acceptance criteria, checklists, comments, artifacts, blockers, reviewers, dispatch receipts, and status history.

**What this enabled:** Live work gained a canonical rail. The vault can remain durable memory while the Command Center handles assignments, collaboration, and review.

## 24. Add a privacy-scoped query layer

Generate a searchable manifest of approved vault records. Return grounded answers with exact source paths and freshness dates. Replace the private manifest with an empty or sanitized version in public releases.

**What this enabled:** I can ask the brain questions from a hosted interface without giving the hosted system unrestricted access to my filesystem.

## 25. Add a Brain DNA map

Turn the scoped manifest into an interactive hierarchy. Let the owner traverse domains, folders, and notes, inspect recent records, copy paths, and hand a branch into a grounded query.

**What this enabled:** The structure of the brain became visible. I can browse relationships and choose the right scope before asking a question.

## 26. Govern agent dispatch and review

Require dispatch acknowledgment, task-room progress, exact artifacts, a named reviewer, and a completion receipt. Do not count route availability as completed work. Block self-approval.

**What this enabled:** Agent activity became auditable production. A status label cannot hide missing work or an inaccessible artifact.

## 27. Enforce least-privilege access

Separate vault access, task-room access, connector permissions, public exports, and human approval. Give each agent the minimum scope needed for its lane.

**What this enabled:** I can expand the workforce without handing every agent the entire private brain or every connected account.

## 28. Create a correction-to-skill loop

When you correct an agent, update the relevant skill, SOP, soul, template, or operating rule during the same work cycle. Record the change in the log.

**What this enabled:** A mistake can improve future behavior across sessions. The correction no longer lives in one forgotten conversation.

## 29. Package reusable systems completely

For each repeatable system, include the SOP, intake, discovery guide, post-call prompt, assembly instructions, schema notes, quality rules, approval gates, and stop-loss conditions.

**What this enabled:** Another person or AI can run the system without the original conversation or the person who designed it.

## 30. Turn the brain into a portable installer

Package the architecture as an interview-first installer. Ask about the owner's real operating areas, sources, projects, access boundaries, and recurring jobs before creating the files.

**What this enabled:** The second brain can be installed for another person without copying my private details or forcing them into my folder structure.

## A practical installation order

Do not install all 30 upgrades at once. Use five stages.

### Stage 1: Trustworthy memory

Install the raw layer, wiki layer, source notes, index, log, and processed-source lifecycle.

### Stage 2: Personal operations

Add journals, human operating areas, conversation capture, project workspaces, and Git synchronization.

### Stage 3: Better intake

Add creator intelligence, participant attribution, YouTube ingestion, call intelligence, and artifact preservation.

### Stage 4: Proactive management

Add recurring reviews, an AI Chief of Staff, soul files, context brokerage, workforce operations, and governed review.

### Stage 5: Productization

Add the hosted query layer, visual map, reusable system standard, and portable installer.

## Copy-paste prompt for your AI

```text
Help me upgrade my current second brain into a source-grounded operating system.

Start by inspecting the existing structure. Do not delete, rename, or move files without my approval.

Create a written gap analysis against these capabilities:

1. Raw source intake and processed-source tracking
2. Source notes and traceable wiki synthesis
3. Human operating areas for life, business, and projects
4. Journal and AI conversation capture
5. Project orientation, task, and session records
6. Git synchronization and recovery
7. Creator and participant attribution
8. Recorded-call intelligence
9. AI Chief of Staff ownership
10. Soul files and proactive context brokerage
11. Governed task dispatch, review, and receipts
12. Correction-to-skill learning
13. Reusable system packaging
14. Privacy-scoped hosted search

For each gap, explain:

- what is missing
- what the change would enable
- the smallest safe implementation
- any privacy or approval risk
- how we will test it

Then propose a staged installation plan. Wait for my approval before making structural changes.
```

## Definition of done

Your second brain has become an operating system when it can:

- preserve original evidence;
- explain where a claim came from;
- recover decisions from conversations and calls;
- orient an AI inside any active project;
- deliver the right context before assigning work;
- collect proof and route review;
- learn from corrections;
- keep private knowledge separate from public systems; and
- continue operating without depending on one chat or one agent's memory.

Build the memory first. Add management second. Add more agents after both layers work.