275 lines
4.2 KiB
Markdown
275 lines
4.2 KiB
Markdown
# TMI RAG (rag-tmi)
|
|
|
|
A lightweight **local Retrieval-Augmented Generation (RAG) system** used to index and search technical documentation and source code across the Terra-Mechanics development workspace.
|
|
|
|
This tool allows Claude Code, Codex, or any LLM assistant to retrieve relevant context from engineering documentation before answering questions.
|
|
|
|
The goal is to create a **searchable semantic memory layer** for projects such as:
|
|
|
|
- Terra-View
|
|
- Seismo Relay
|
|
- Series 3 / Minimate protocol research
|
|
- Modem / SLM documentation
|
|
- Reverse engineering notes
|
|
- Parser documentation
|
|
|
|
Instead of manually searching repos or notes, the system retrieves relevant information using **vector similarity search**.
|
|
|
|
---
|
|
|
|
# How It Works
|
|
|
|
The system follows a standard RAG architecture.
|
|
|
|
### 1. Indexing
|
|
|
|
`ingest.py`:
|
|
|
|
1. Reads directories listed in `sources.yaml`
|
|
2. Recursively scans for supported files (`.md`, `.txt`, `.py`)
|
|
3. Splits content into chunks
|
|
4. Generates embeddings using OpenAI
|
|
5. Stores vectors in a FAISS index
|
|
|
|
Result:
|
|
|
|
```
|
|
index/
|
|
index.faiss
|
|
meta.pkl
|
|
```
|
|
|
|
This becomes the **semantic database**.
|
|
|
|
---
|
|
|
|
### 2. Querying
|
|
|
|
`query.py`:
|
|
|
|
1. Embeds the query text
|
|
2. Searches the FAISS vector index
|
|
3. Returns the most relevant chunks of text
|
|
|
|
These results can be pasted into an LLM chat (Claude Code, Codex, etc.) as context.
|
|
|
|
Example:
|
|
|
|
```
|
|
python query.py "checksum algorithm"
|
|
```
|
|
|
|
---
|
|
|
|
# Repository Structure
|
|
|
|
```
|
|
rag-tmi/
|
|
│
|
|
├─ ingest.py
|
|
├─ query.py
|
|
├─ sources.yaml
|
|
├─ requirements.txt
|
|
├─ README.md
|
|
│
|
|
├─ index/ # Generated FAISS index (not committed)
|
|
│
|
|
└─ .venv/ # Local Python environment (not committed)
|
|
```
|
|
|
|
---
|
|
|
|
# Installation
|
|
|
|
Create a virtual environment:
|
|
|
|
```
|
|
python -m venv .venv
|
|
```
|
|
|
|
Activate it.
|
|
|
|
Windows (PowerShell):
|
|
|
|
```
|
|
.venv\Scripts\activate
|
|
```
|
|
|
|
Install dependencies:
|
|
|
|
```
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
---
|
|
|
|
# Environment Variables
|
|
|
|
Create a `.env` file in the project root.
|
|
|
|
```
|
|
OPENAI_API_KEY=your_key_here
|
|
```
|
|
|
|
The key is loaded automatically by:
|
|
|
|
```
|
|
python-dotenv
|
|
```
|
|
|
|
---
|
|
|
|
# Running the Indexer
|
|
|
|
Build the vector database:
|
|
|
|
```
|
|
python ingest.py
|
|
```
|
|
|
|
The script will:
|
|
|
|
- scan all configured sources
|
|
- generate embeddings
|
|
- build a FAISS index
|
|
|
|
You should see progress like:
|
|
|
|
```
|
|
Embedding 72 chunks
|
|
```
|
|
|
|
The index will be written to:
|
|
|
|
```
|
|
index/index.faiss
|
|
index/meta.pkl
|
|
```
|
|
|
|
---
|
|
|
|
# Querying the Database
|
|
|
|
Run:
|
|
|
|
```
|
|
python query.py "frame checksum"
|
|
```
|
|
|
|
The system will return the most relevant chunks from indexed documents.
|
|
|
|
These chunks can be copied into Claude or Codex to provide **accurate context** for responses.
|
|
|
|
---
|
|
|
|
# sources.yaml
|
|
|
|
This file defines which directories are indexed.
|
|
|
|
Example:
|
|
|
|
```yaml
|
|
sources:
|
|
- ../terra-view
|
|
- ../seismo-relay
|
|
- ../series3-agent
|
|
- ../protocol-docs
|
|
```
|
|
|
|
Paths are relative to the `rag-tmi` directory.
|
|
|
|
---
|
|
|
|
# Supported File Types
|
|
|
|
Currently indexed:
|
|
|
|
```
|
|
.md
|
|
.txt
|
|
.py
|
|
```
|
|
|
|
Additional types can be added inside `ingest.py`.
|
|
|
|
---
|
|
|
|
# Git Notes
|
|
|
|
The following files should **not be committed**:
|
|
|
|
```
|
|
.venv/
|
|
.env
|
|
index/
|
|
__pycache__/
|
|
```
|
|
|
|
Example `.gitignore` entries:
|
|
|
|
```
|
|
.venv/
|
|
.env
|
|
index/
|
|
__pycache__/
|
|
```
|
|
|
|
---
|
|
|
|
# Why This Exists
|
|
|
|
Large engineering projects accumulate knowledge across:
|
|
|
|
- source code
|
|
- documentation
|
|
- research notes
|
|
- reverse engineering logs
|
|
|
|
Traditional search tools rely on **exact text matches**.
|
|
|
|
RAG enables **semantic search**, meaning queries like:
|
|
|
|
```
|
|
"frame checksum"
|
|
```
|
|
|
|
can retrieve documentation that contains phrases like:
|
|
|
|
```
|
|
payload CRC
|
|
frame validation
|
|
checksum calculation
|
|
```
|
|
|
|
even if the exact words do not match.
|
|
|
|
---
|
|
|
|
# Future Improvements
|
|
|
|
Possible upgrades:
|
|
|
|
- CLI command wrapper (`rag "query"`)
|
|
- automatic repo indexing
|
|
- incremental indexing
|
|
- MCP server for AI tool access
|
|
- reranking model
|
|
- code-aware chunking
|
|
- web UI for search
|
|
|
|
---
|
|
|
|
# Philosophy
|
|
|
|
This tool acts as a **local memory layer** for engineering work.
|
|
|
|
Instead of searching through repos manually, developers and AI assistants can retrieve the most relevant information instantly.
|
|
|
|
The system is intentionally simple:
|
|
|
|
- local
|
|
- transparent
|
|
- easy to rebuild
|
|
- no heavy frameworks
|
|
|
|
It follows the principle that **small tools that solve real problems are better than complex systems that are hard to maintain.** |