Field Notes

Simple Local RAGs

A practical experiment in turning a local markdown folder into a lightweight memory system with AnythingLLM, Ollama, and a little less workflow chaos.

Local AI AnythingLLM Ollama Markdown Memory RAG Experiments

A working field note on lightweight local retrieval, memory, and AI workflow behavior.

Finder view of the Micro RAG markdown folder used in the local AnythingLLM experiment.
The knowledge base stays intentionally small. That is part of the point.

1. Why I wanted this

I wanted local memory that felt useful, not precious

A lot of “personal knowledge” setups look impressive right up until you have to actually use them. Then you end up buried in categories, overthinking folders, or wondering why your AI assistant still cannot remember something simple like a birthday or a note you saved yesterday.

This experiment started with a pretty basic question: can a local AI setup read and write a small markdown knowledge base in a way that feels practical for normal life and normal work?

Not a giant second-brain system. Not an enterprise search platform. Just a simple local RAG that can help store notes, retrieve them later, and stay out of its own way.

2. What the setup actually is

The working version is surprisingly simple on paper

AnythingLLM Desktop

The main local app handling the workspace, agent skills, retrieval settings, and file access.

Ollama

The local model layer powering the workspace and agent behavior.

Markdown knowledge folder

A small folder with files like people.md, ideas.md, and learning_notes.md.

File System Access

The key permission layer that lets the agent actually read and update those markdown files.

The important part is that the setup is local end to end. The notes live in normal markdown files. The models run through Ollama. The workspace can retrieve across the folder and, if configured properly, write back to the right place.

Markdown file showing a saved local RAG blog post idea inside ideas.md.
A plain markdown file is enough for lightweight captured ideas.
Markdown file showing learning notes captured from the local RAG experiment.
The setup works best when the file structure stays boring and predictable.

3. What actually worked

Once the right model and prompt were in place, the system started acting like memory instead of a demo

Birthday retrieval

The system could read people.md and answer questions like when someone’s birthday is or how many birthday entries were stored.

Automatic routing

It could pull from the right file depending on the question, like ideas.md for a blog idea or learning_notes.md for a chunking note.

Write behavior

After prompt updates, it could append new entries to existing files instead of treating every note like a rewrite job.

Multi-file retrieval

It could answer a broader question about RAG or AI agents by surfacing notes from more than one markdown file.

That is the point where the experiment started feeling real. Not perfect, but real. Once it could save, preserve, and retrieve notes from a lightweight markdown folder, it stopped feeling like a toy and started feeling like something you might actually keep using.

Markdown note showing a stored blog idea that can be retrieved later by the local RAG setup.
One of the first useful wins was simple: save a note, come back later, and actually retrieve it.

4. Why it finally started working

The fix was not just “pick a local model”

The biggest lesson from this setup is that local RAG behavior depends on more than whether the model can answer questions.

The model has to support tool use well enough to read files, write files, and behave consistently around memory tasks. In this setup, gpt-oss:latest became the practical working choice because it could actually handle that loop.

The other big piece was the system prompt. The working prompt gives the assistant clear rules around:

  • retrieving automatically
  • only writing when the user clearly wants something saved
  • routing notes to the right markdown file
  • reading the current file before writing
  • appending instead of overwriting

That last one mattered more than I expected. Sometimes workflows turn into spaghetti. Sometimes models turn into overly confident file editors. It happens.

AnythingLLM system prompt settings for the Micro RAG workspace.
The rules matter. The working prompt does a lot of quiet operational heavy lifting.

5. Where it still breaks

This is usable now, but it is still an experiment

Tool support is not optional

Some models can chat just fine and still fall apart the moment file tools are involved.

Weak prompts create risky behavior

If append and preserve rules are vague, file writes can get weird fast.

Retrieval is only as clean as the setup

Chunking, file structure, and context settings still matter more than most people want them to.

Casual memory capture can get messy

If the agent stores everything too aggressively, you end up with noisy notes instead of useful memory.

So the current version is not “set it and forget it.” It is more like: set it up carefully, pressure test it, and give the assistant better operating rules than you think it needs.

AnythingLLM retrieval example showing a local RAG failure case during the experiment.
Useful systems still miss sometimes. That is why failure cases belong in the writeup too.

6. Why this experiment matters

I think there is a very practical middle ground here

A lot of AI memory tooling gets framed in extremes. Either it is a huge enterprise knowledge stack, or it is a messy little demo that never survives real use.

What interests me more is the middle ground: small local systems that help normal people remember useful things, store rough notes, and retrieve them later without needing a giant app stack to make it work.

That is what makes this experiment worth documenting. It is a very ordinary use case, which is exactly why it matters.

If this keeps improving, it could feed future work around:

  • local AI beginner guides
  • personal workflow templates
  • RAG experiments for non-developers
  • practical memory systems that stay lightweight
Back to Field Notes Explore GetPrompting