Two weeks ago, Andrej Karpathy, a founding member of OpenAI and former head of AI at Tesla, published how he approaches knowledge bases with LLMs. I think his setup deserves attention from anyone running a law firm or legal function, because it points to something I keep coming back to: firms that build on commoditized legal AI are backing themselves into a corner, and this is one of the first concrete architectures I have seen that offers a way out.
Out of all professions I can think of, this should matter most to lawyers. The more they adopt legal AI, the more their output gets commoditized, and the more exposed they become to fee pressure. By building their technical infrastructure in a way that gets them differentiated outputs, they reduce that exposure.
To see why this is a real distinction and not just another tech fad, it helps to understand where legal AI sits today. Almost all legal AI is built on RAG. RAG stands for retrieval-augmented generation and uses your data together with the internal knowledge of the AI model to give you better results. In a typical setting, a RAG system would combine your firm’s matter data, connected databases like LexisNexis, and the model’s own knowledge to produce a result. The pitch sounds great in theory, but I think we have all seen those RAG systems struggle in practice. They struggle to figure out which documents are relevant, or which to prioritise, or sometimes forget to retrieve the right documents entirely. Semantic search is useful, but it is not all it is meant to be. And crucially, every query starts from zero: the model rediscovers your knowledge base from scratch each time you ask a question. Nothing accumulates.
Karpathy’s alternative looks quite different. At a high level, it has three layers: the raw source files, which never change; a separate wiki of markdown files that the LLM writes and maintains on top of those sources; and a schema document that tells the LLM how to ingest new material, update the wiki, and answer questions against it. The architecture matters less than the consequence, which is the important part: the wiki is a persistent, compounding artifact. Every new document you add, and every question you ask, updates the wiki and makes it richer. The synthesis is built once and then kept current, not re-derived on every query. In other words, your knowledge base actually gets smarter the more you use it, which is something RAG pipelines structurally cannot do.
Three features of this setup are particularly relevant to senior legal leaders. First, it is explicit. You can see exactly what the system knows about you, your matters, and your clients, because it all lives in markdown files you can open and read. No black-box memory, no opaque vendor store. Second, the data is yours. It sits on your infrastructure, in universal file formats, under your control. Third, because the whole thing works through your file system rather than inside a vendor’s walled garden, there is no lock-in: you can swap the underlying model at will, use Claude today and something else tomorrow, and the knowledge base keeps working. Any of these on their own should matter to a GC or managing partner; together they describe a categorically different relationship with AI than the one most firms are signing up for.
Applied to the legal industry, building on a knowledge base like this, rather than just plugging your data into an off-the-shelf RAG tool, gets you two things that I believe matter enormously. First, personalised results: an AI that is genuinely grounded in how your firm practises, not in how the average firm practises. Second, differentiated results: outputs your competitors, running the same vendor tools, structurally cannot replicate. I have argued before that firms buying the same AI tool as everyone else are setting themselves up to sell undifferentiated work at commoditized prices. The LLM knowledge base approach is the first concrete answer I have seen to that problem.
I have been using a similar setup myself, running Claude Code inside my Obsidian vault, since about last summer. For a long time, the models and context windows weren’t where they needed to be, and setting such a system up was quite complex for a non-technical audience. With recent improvements in the models and in the agents, particularly the combination of Claude and Obsidian, or any other markdown-based setup, you can now get a tremendous result in a very time-effective way. For a non-technical audience, building a cleaner user interface on top of it has also never been easier. It is an area where I think many law firms are significantly under-investing today, and I suspect the ones that move now will look, in a few years, like the ones that got AI right.
[1]: Karpathy’s original post on LLM Knowledge Bases https://x.com/karpathy/status/2039805659525644595?s=20
[2]: Repository in which Karpathy explains his approach and how to build such a knowledge base yourself in more detail: https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f