Article complete

Andrej Karpathy Moves Beyond RAG, Builds LLM-Powered Personal Knowledge Bases

Instead of relying on RAG, Andrej Karpathy said that LLMs can manage indexing and summaries internally at smaller scales.

A man with short dark hair and a plaid shirt smiles against a graphic background featuring abstract shapes and patterns.

Key takeaways

  • Andrej Karpathy is using large language models to create structured personal knowledge bases.
  • LLMs can manage indexing and summarising data internally, reducing reliance on retrieval-augmented generation.
  • Karpathy's system compiles articles, datasets, and images into a structured wiki with summaries and backlinks.
  • The knowledge management app Obsidian serves as the interface for viewing both raw and processed data.
  • LLMs assist in maintenance tasks, including identifying inconsistencies and suggesting new areas of exploration.

Takeaways drafted with AI from the article text.

OpenAI co-founder Andrej Karpathy said he is increasingly using large language models (LLMs) to build structured personal knowledge bases, moving from writing code to organising and querying information.

In a post on X, Karpathy described a system where raw materials such as articles, research papers, repositories, datasets, and images are first collected and indexed. 

An LLM is then used to compile this data into a structured wiki made up of markdown files, including summaries, backlinks, and concept-level organisation.

“The wiki includes summaries of all the data in raw/, backlinks, and then it categorises data into concepts, writes articles for them, and links them all,” he wrote.

Karpathy said he uses the knowledge management app Obsidian as the interface to view both raw and processed data, while the LLM maintains the knowledge base. He added that he rarely edits the files manually.

As the knowledge base grows, the system supports complex querying. Instead of relying on retrieval-augmented generation (RAG), Karpathy said that LLMs can manage indexing and summaries internally at smaller scales. 

“I thought I had to reach for fancy RAG, but the LLM has been pretty good about auto-maintaining index files and brief summaries of all the documents, and it reads all the important related data fairly easily at this ~small scale,” he wrote.

Outputs from queries are generated as markdown documents, slide presentations, or visualisations, which are then added back into the knowledge base. “My own explorations and queries always ‘add up’ in the knowledge base,” he said.

Karpathy also described using LLMs for maintenance tasks such as identifying inconsistencies, filling missing information, and suggesting new areas of exploration. He said additional tools, including a custom search interface, are used to assist both users and LLM agents.

Looking ahead, he pointed to the potential for synthetic data generation and fine-tuning models so that knowledge can be embedded directly into model weights rather than relying only on context.

“I think there is room here for an incredible new product instead of a hacky collection of scripts,” Karpathy said.



Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.

More in this story

OpenAI · 5 stories
  1. 13 Best Research Papers on LLMs
  2. Copycats, Snake Oil Sellers, and Doomsayers of AI
  3. OpenAI Building New Voice Model Ahead of AI Device Launch: Report
  4. App Stores are Outdated in the Era of Vibe Coding: Andrej Karpathy
  5. You are here: Andrej Karpathy Moves Beyond RAG, Builds LLM-Powered Personal Knowledge Bases
About the author

Tech Journalist

Siddharth is a media graduate who loves to explore tech through journalism and putting forward ideas worth pondering about in the era of artificial intelligence.

All stories
Belamy · Daily

The day’s AI news, before your first meeting.