thor-watcher/README.md

Series 4 Ingest Agent

Version: 0.1.3

Micromate (Series 4) ingest agent for Seismo Fleet Manager (SFM).

Overview

Series 4 Ingest Agent is a Python-based monitoring tool that scans for Micromate unit activity logs and reports their status. It monitors MLG (Micromate Log) files in the THORDATA directory structure, determines unit health based on last activity, and optionally transmits heartbeat data to a Seismo Fleet Manager backend.

Features

• Automatic MLG File Discovery: Scans C:\THORDATA\<Project>\<UM####>\*.MLG for Micromate units
• Intelligent Unit Tracking: Automatically identifies the newest MLG file per unit based on timestamp
• Status Classification: Categorizes units as OK, LATE, or STALE based on configurable age thresholds
• Console Heartbeat Display: Real-time status dashboard with formatted output
• SFM Backend Integration: Optional HTTP POST of JSON telemetry to Fleet Manager
• External Configuration: JSON-based configuration file for easy customization
• Robust Error Handling: Graceful degradation with informative warnings
• No External Dependencies: Uses Python standard library only (urllib, json, datetime, etc.)

Requirements

• Python 3.6 or higher
• Windows OS (designed for THOR PC/VM environments)
• Access to THORDATA directory structure

Installation

1. Download and run the MSI installer (Windows, admin required)
2. Edit config at C:\ProgramData\ThorIngest\config.json
3. The service starts automatically and runs on boot

No command line is required for normal use. The installer sets up the Windows Service.

For development or manual runs:

1. Ensure Python 3.6+ is installed
2. Configure config.json (see Configuration section)
3. Run the agent:

python series4_ingest.py

Configuration

All configuration is managed through config.json in:

C:\ProgramData\ThorIngest\config.json

config.json Structure

{
  "thordata_path": "C:\\THORDATA",
  "scan_interval": 60,
  "late_days": 2,
  "stale_days": 60,
  "sfm_endpoint": "http://<sfm backend ip>:8001/api/series4/heartbeat"
  "sfm_timeout": 5,
  "debug": true
}

Configuration Options

Parameter	Type	Default	Description
thordata_path	string	C:\THORDATA	Root directory containing project/unit MLG files
scan_interval	integer	60	Time in seconds between scans
late_days	integer	2	Days before a unit is marked as LATE
stale_days	integer	60	Days before a unit is marked as STALE
sfm_endpoint	string	""	SFM backend URL (leave empty to disable HTTP)
sfm_timeout	integer	5	HTTP request timeout in seconds
debug	boolean	true	Enable/disable debug logging

Fallback Behavior

If config.json is missing or malformed, the agent will:

• Display a warning message
• Use built-in default values
• Continue running normally

Usage

Start the Agent (manual)

python series4_ingest.py

Common Flags

• --config <path>: Override config file path
• --log-dir <path>: Override log directory
• --once: Run one scan and exit
• --version: Print version and exit

Run Once (manual)

python series4_ingest.py --once

Show Version

python series4_ingest.py --version

Console Output Example

Series 4 Ingest Agent — Micromate Heartbeat (v0.1.3)
THORDATA root: C:\THORDATA
Now: 2025-12-08 14:30:00
--------------------------------------------------------------------------------
UM11719  OK     Age: 1h 12m  Last: 2025-12-08 13:18:00  Project: Clearwater - ECMS 57940
UM12345  LATE   Age: 3d 5h   Last: 2025-12-05 09:15:30  Project: Site Alpha
UM98765  STALE  Age: 65d     Last: 2025-10-04 08:22:15  Project: Legacy Site
--------------------------------------------------------------------------------
Total units: 3
Next scan in 60 seconds...

Stop the Agent (manual)

Press Ctrl+C to gracefully stop the agent.

Windows Service Installation

The MSI installs a Windows Service named ThorIngest using NSSM. It runs at startup.

Check status

sc query ThorIngest

Start / stop

sc start ThorIngest
sc stop ThorIngest

Service Logs

Logs are written to:

C:\ProgramData\ThorIngest\logs\thor_ingest.log

Updating

1. Download the newer MSI from the internal file share/URL.
2. Run the MSI to upgrade in place.
3. Your existing config and logs are preserved.

Building the EXE (PyInstaller)

pyinstaller --onefile --name thor-ingest-agent series4_ingest.py

The output EXE will be in dist/.

Status Classification

Units are classified based on the age of their last MLG file:

• OK: Last activity within late_days threshold (default: < 2 days)
• LATE: Last activity between late_days and stale_days (default: 2-60 days)
• STALE: Last activity exceeds stale_days threshold (default: ≥ 60 days)

SFM Backend Integration

When sfm_endpoint is configured, the agent sends JSON payloads to the Seismo Fleet Manager:

Payload Structure

{
  "source": "series4_ingest",
  "generated_at": "2025-12-08T14:30:00",
  "units": [
    {
      "unit_id": "UM11719",
      "type": "micromate",
      "project_hint": "Clearwater - ECMS 57940",
      "last_call": "2025-12-08T13:18:00",
      "status": "OK",
      "age_days": 0.05,
      "age_hours": 1.2,
      "mlg_path": "C:\\THORDATA\\Clearwater - ECMS 57940\\UM11719\\UM11719_20251208131800.MLG"
    }
  ]
}

Disabling SFM Integration

Set sfm_endpoint to an empty string in config.json:

{
  "sfm_endpoint": ""
}

MLG File Format

The agent expects MLG files to follow this naming convention:

UM<unit_number>_YYYYMMDDHHMMSS.MLG

Example: UM11719_20251208131800.MLG

• UM11719: Unit ID
• 20251208131800: Timestamp (2025-12-08 13:18:00)
• .MLG: Micromate Log file extension

Directory Structure

Expected THORDATA folder structure:

C:\THORDATA\
├── Project A\
│   ├── UM11719\
│   │   ├── UM11719_20251208131800.MLG
│   │   ├── UM11719_20251207095430.MLG
│   │   └── ...
│   └── UM12345\
│       └── UM12345_20251205091530.MLG
└── Project B\
    └── UM98765\
        └── UM98765_20251004082215.MLG

Troubleshooting

Config file not found

If you see [WARN] Config file not found, create config.json in C:\ProgramData\ThorIngest\config.json (or pass --config).

THORDATA path doesn't exist

Verify the thordata_path in config.json points to the correct directory.

No units found

Ensure MLG files exist in the expected directory structure and follow the naming convention.

SFM POST failures

Check that:

• sfm_endpoint URL is correct and accessible
• Network connectivity is available
• SFM backend is running and accepting requests

License

Proprietary - Internal use only

Version History

See CHANGELOG.md for detailed version history.

Support

For issues or questions, contact the Seismo development team.