Andrej Karpathy Moves Beyond RAG, Builds LLM-Powered Personal Knowledge Bases

OpenAI co-founder Andrej Karpathy said he is increasingly using large language models (LLMs) to build structured personal knowledge bases, moving from writing code to organising and querying information.

In a post on X, Karpathy described a system where raw materials such as articles, research papers, repositories, datasets, and images are first collected and indexed. 

An LLM is then used to compile this data into a structured wiki made up of markdown files, including summaries, backlinks, and concept-level organisation.

“The wiki includes summaries of all the data in raw/, backlinks, and then it categorises data into concepts, writes articles for them, and links them all,” he wrote.

Karpathy said he uses the knowledge management app Obsidian as the interface to view both raw and processed data, while the LLM maintains the knowledge base. He added that he rarely edits the files manually.

As the knowledge base grows, the system supports complex querying. Instead of relying on retrieval-augmented generation (RAG), Karpathy said that LLMs can manage indexing and summaries internally at smaller scales. 

“I thought I had to reach for fancy RAG, but the LLM has been pretty good about auto-maintaining index files and brief summaries of all the documents, and it reads all the important related data fairly easily at this ~small scale,” he wrote.

Outputs from queries are generated as markdown documents, slide presentations, or visualisations, which are then added back into the knowledge base. “My own explorations and queries always ‘add up’ in the knowledge base,” he said.

Karpathy also described using LLMs for maintenance tasks such as identifying inconsistencies, filling missing information, and suggesting new areas of exploration. He said additional tools, including a custom search interface, are used to assist both users and LLM agents.

Looking ahead, he pointed to the potential for synthetic data generation and fine-tuning models so that knowledge can be embedded directly into model weights rather than relying only on context.

“I think there is room here for an incredible new product instead of a hacky collection of scripts,” Karpathy said.