# TMI RAG (rag-tmi)

A lightweight **local Retrieval-Augmented Generation (RAG) system** used to index and search technical documentation and source code across the Terra-Mechanics development workspace.

This tool allows Claude Code, Codex, or any LLM assistant to retrieve relevant context from engineering documentation before answering questions.

The goal is to create a **searchable semantic memory layer** for projects such as:

- Terra-View
- Seismo Relay
- Series 3 / Minimate protocol research
- Modem / SLM documentation
- Reverse engineering notes
- Parser documentation

Instead of manually searching repos or notes, the system retrieves relevant information using **vector similarity search**.

---

# How It Works

The system follows a standard RAG architecture.

### 1. Indexing

`ingest.py`:

1. Reads directories listed in `sources.yaml`
2. Recursively scans for supported files (`.md`, `.txt`, `.py`)
3. Splits content into chunks
4. Generates embeddings using OpenAI
5. Stores vectors in a FAISS index

Result:

```
index/
  index.faiss
  meta.pkl
```

This becomes the **semantic database**.

---

### 2. Querying

`query.py`:

1. Embeds the query text
2. Searches the FAISS vector index
3. Returns the most relevant chunks of text

These results can be pasted into an LLM chat (Claude Code, Codex, etc.) as context.

Example:

```
python query.py "checksum algorithm"
```

---

# Repository Structure

```
rag-tmi/
│
├─ ingest.py
├─ query.py
├─ sources.yaml
├─ requirements.txt
├─ README.md
│
├─ index/            # Generated FAISS index (not committed)
│
└─ .venv/            # Local Python environment (not committed)
```

---

# Installation

Create a virtual environment:

```
python -m venv .venv
```

Activate it.

Windows (PowerShell):

```
.venv\Scripts\activate
```

Install dependencies:

```
pip install -r requirements.txt
```

---

# Environment Variables

Create a `.env` file in the project root.

```
OPENAI_API_KEY=your_key_here
```

The key is loaded automatically by:

```
python-dotenv
```

---

# Running the Indexer

Build the vector database:

```
python ingest.py
```

The script will:

- scan all configured sources
- generate embeddings
- build a FAISS index

You should see progress like:

```
Embedding 72 chunks
```

The index will be written to:

```
index/index.faiss
index/meta.pkl
```

---

# Querying the Database

Run:

```
python query.py "frame checksum"
```

The system will return the most relevant chunks from indexed documents.

These chunks can be copied into Claude or Codex to provide **accurate context** for responses.

---

# sources.yaml

This file defines which directories are indexed.

Example:

```yaml
sources:
  - ../terra-view
  - ../seismo-relay
  - ../series3-agent
  - ../protocol-docs
```

Paths are relative to the `rag-tmi` directory.

---

# Supported File Types

Currently indexed:

```
.md
.txt
.py
```

Additional types can be added inside `ingest.py`.

---

# Git Notes

The following files should **not be committed**:

```
.venv/
.env
index/
__pycache__/
```

Example `.gitignore` entries:

```
.venv/
.env
index/
__pycache__/
```

---

# Why This Exists

Large engineering projects accumulate knowledge across:

- source code
- documentation
- research notes
- reverse engineering logs

Traditional search tools rely on **exact text matches**.

RAG enables **semantic search**, meaning queries like:

```
"frame checksum"
```

can retrieve documentation that contains phrases like:

```
payload CRC
frame validation
checksum calculation
```

even if the exact words do not match.

---

# Future Improvements

Possible upgrades:

- CLI command wrapper (`rag "query"`)
- automatic repo indexing
- incremental indexing
- MCP server for AI tool access
- reranking model
- code-aware chunking
- web UI for search

---

# Philosophy

This tool acts as a **local memory layer** for engineering work.

Instead of searching through repos manually, developers and AI assistants can retrieve the most relevant information instantly.

The system is intentionally simple:

- local
- transparent
- easy to rebuild
- no heavy frameworks

It follows the principle that **small tools that solve real problems are better than complex systems that are hard to maintain.**