273 lines
7.0 KiB
Markdown
273 lines
7.0 KiB
Markdown
# Series 4 Ingest Agent
|
|
|
|
**Version:** 0.1.3
|
|
|
|
Micromate (Series 4) ingest agent for Seismo Fleet Manager (SFM).
|
|
|
|
## Overview
|
|
|
|
Series 4 Ingest Agent is a Python-based monitoring tool that scans for Micromate unit activity logs and reports their status. It monitors MLG (Micromate Log) files in the THORDATA directory structure, determines unit health based on last activity, and optionally transmits heartbeat data to a Seismo Fleet Manager backend.
|
|
|
|
## Features
|
|
|
|
- **Automatic MLG File Discovery**: Scans `C:\THORDATA\<Project>\<UM####>\*.MLG` for Micromate units
|
|
- **Intelligent Unit Tracking**: Automatically identifies the newest MLG file per unit based on timestamp
|
|
- **Status Classification**: Categorizes units as OK, LATE, or STALE based on configurable age thresholds
|
|
- **Console Heartbeat Display**: Real-time status dashboard with formatted output
|
|
- **SFM Backend Integration**: Optional HTTP POST of JSON telemetry to Fleet Manager
|
|
- **External Configuration**: JSON-based configuration file for easy customization
|
|
- **Robust Error Handling**: Graceful degradation with informative warnings
|
|
- **No External Dependencies**: Uses Python standard library only (urllib, json, datetime, etc.)
|
|
|
|
## Requirements
|
|
|
|
- Python 3.6 or higher
|
|
- Windows OS (designed for THOR PC/VM environments)
|
|
- Access to THORDATA directory structure
|
|
|
|
## Installation
|
|
|
|
1. Download and run the MSI installer (Windows, admin required)
|
|
2. Edit config at `C:\ProgramData\ThorIngest\config.json`
|
|
3. The service starts automatically and runs on boot
|
|
|
|
No command line is required for normal use. The installer sets up the Windows Service.
|
|
|
|
For development or manual runs:
|
|
1. Ensure Python 3.6+ is installed
|
|
2. Configure `config.json` (see Configuration section)
|
|
3. Run the agent:
|
|
|
|
```bash
|
|
python series4_ingest.py
|
|
```
|
|
|
|
## Configuration
|
|
|
|
All configuration is managed through `config.json` in:
|
|
|
|
```
|
|
C:\ProgramData\ThorIngest\config.json
|
|
```
|
|
|
|
### config.json Structure
|
|
|
|
```json
|
|
{
|
|
"thordata_path": "C:\\THORDATA",
|
|
"scan_interval": 60,
|
|
"late_days": 2,
|
|
"stale_days": 60,
|
|
"sfm_endpoint": "http://<sfm backend ip>:8001/api/series4/heartbeat"
|
|
"sfm_timeout": 5,
|
|
"debug": true
|
|
}
|
|
```
|
|
|
|
### Configuration Options
|
|
|
|
| Parameter | Type | Default | Description |
|
|
|-----------|------|---------|-------------|
|
|
| `thordata_path` | string | `C:\THORDATA` | Root directory containing project/unit MLG files |
|
|
| `scan_interval` | integer | `60` | Time in seconds between scans |
|
|
| `late_days` | integer | `2` | Days before a unit is marked as LATE |
|
|
| `stale_days` | integer | `60` | Days before a unit is marked as STALE |
|
|
| `sfm_endpoint` | string | `""` | SFM backend URL (leave empty to disable HTTP) |
|
|
| `sfm_timeout` | integer | `5` | HTTP request timeout in seconds |
|
|
| `debug` | boolean | `true` | Enable/disable debug logging |
|
|
|
|
### Fallback Behavior
|
|
|
|
If `config.json` is missing or malformed, the agent will:
|
|
- Display a warning message
|
|
- Use built-in default values
|
|
- Continue running normally
|
|
|
|
## Usage
|
|
|
|
### Start the Agent (manual)
|
|
|
|
```bash
|
|
python series4_ingest.py
|
|
```
|
|
|
|
### Common Flags
|
|
|
|
- `--config <path>`: Override config file path
|
|
- `--log-dir <path>`: Override log directory
|
|
- `--once`: Run one scan and exit
|
|
- `--version`: Print version and exit
|
|
|
|
### Run Once (manual)
|
|
|
|
```bash
|
|
python series4_ingest.py --once
|
|
```
|
|
|
|
### Show Version
|
|
|
|
```bash
|
|
python series4_ingest.py --version
|
|
```
|
|
|
|
### Console Output Example
|
|
|
|
```
|
|
Series 4 Ingest Agent — Micromate Heartbeat (v0.1.3)
|
|
THORDATA root: C:\THORDATA
|
|
Now: 2025-12-08 14:30:00
|
|
--------------------------------------------------------------------------------
|
|
UM11719 OK Age: 1h 12m Last: 2025-12-08 13:18:00 Project: Clearwater - ECMS 57940
|
|
UM12345 LATE Age: 3d 5h Last: 2025-12-05 09:15:30 Project: Site Alpha
|
|
UM98765 STALE Age: 65d Last: 2025-10-04 08:22:15 Project: Legacy Site
|
|
--------------------------------------------------------------------------------
|
|
Total units: 3
|
|
Next scan in 60 seconds...
|
|
```
|
|
|
|
### Stop the Agent (manual)
|
|
|
|
Press `Ctrl+C` to gracefully stop the agent.
|
|
|
|
## Windows Service Installation
|
|
|
|
The MSI installs a Windows Service named `ThorIngest` using NSSM. It runs at startup.
|
|
|
|
**Check status**
|
|
|
|
```
|
|
sc query ThorIngest
|
|
```
|
|
|
|
**Start / stop**
|
|
|
|
```
|
|
sc start ThorIngest
|
|
sc stop ThorIngest
|
|
```
|
|
|
|
**Service Logs**
|
|
|
|
Logs are written to:
|
|
```
|
|
C:\ProgramData\ThorIngest\logs\thor_ingest.log
|
|
```
|
|
|
|
## Updating
|
|
|
|
1. Download the newer MSI from the internal file share/URL.
|
|
2. Run the MSI to upgrade in place.
|
|
3. Your existing config and logs are preserved.
|
|
|
|
## Building the EXE (PyInstaller)
|
|
|
|
```bash
|
|
pyinstaller --onefile --name thor-ingest-agent series4_ingest.py
|
|
```
|
|
|
|
The output EXE will be in `dist/`.
|
|
|
|
## Status Classification
|
|
|
|
Units are classified based on the age of their last MLG file:
|
|
|
|
- **OK**: Last activity within `late_days` threshold (default: < 2 days)
|
|
- **LATE**: Last activity between `late_days` and `stale_days` (default: 2-60 days)
|
|
- **STALE**: Last activity exceeds `stale_days` threshold (default: ≥ 60 days)
|
|
|
|
## SFM Backend Integration
|
|
|
|
When `sfm_endpoint` is configured, the agent sends JSON payloads to the Seismo Fleet Manager:
|
|
|
|
### Payload Structure
|
|
|
|
```json
|
|
{
|
|
"source": "series4_ingest",
|
|
"generated_at": "2025-12-08T14:30:00",
|
|
"units": [
|
|
{
|
|
"unit_id": "UM11719",
|
|
"type": "micromate",
|
|
"project_hint": "Clearwater - ECMS 57940",
|
|
"last_call": "2025-12-08T13:18:00",
|
|
"status": "OK",
|
|
"age_days": 0.05,
|
|
"age_hours": 1.2,
|
|
"mlg_path": "C:\\THORDATA\\Clearwater - ECMS 57940\\UM11719\\UM11719_20251208131800.MLG"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### Disabling SFM Integration
|
|
|
|
Set `sfm_endpoint` to an empty string in `config.json`:
|
|
|
|
```json
|
|
{
|
|
"sfm_endpoint": ""
|
|
}
|
|
```
|
|
|
|
## MLG File Format
|
|
|
|
The agent expects MLG files to follow this naming convention:
|
|
|
|
```
|
|
UM<unit_number>_YYYYMMDDHHMMSS.MLG
|
|
```
|
|
|
|
Example: `UM11719_20251208131800.MLG`
|
|
|
|
- `UM11719`: Unit ID
|
|
- `20251208131800`: Timestamp (2025-12-08 13:18:00)
|
|
- `.MLG`: Micromate Log file extension
|
|
|
|
## Directory Structure
|
|
|
|
Expected THORDATA folder structure:
|
|
|
|
```
|
|
C:\THORDATA\
|
|
├── Project A\
|
|
│ ├── UM11719\
|
|
│ │ ├── UM11719_20251208131800.MLG
|
|
│ │ ├── UM11719_20251207095430.MLG
|
|
│ │ └── ...
|
|
│ └── UM12345\
|
|
│ └── UM12345_20251205091530.MLG
|
|
└── Project B\
|
|
└── UM98765\
|
|
└── UM98765_20251004082215.MLG
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Config file not found
|
|
If you see `[WARN] Config file not found`, create `config.json` in `C:\ProgramData\ThorIngest\config.json` (or pass `--config`).
|
|
|
|
### THORDATA path doesn't exist
|
|
Verify the `thordata_path` in `config.json` points to the correct directory.
|
|
|
|
### No units found
|
|
Ensure MLG files exist in the expected directory structure and follow the naming convention.
|
|
|
|
### SFM POST failures
|
|
Check that:
|
|
- `sfm_endpoint` URL is correct and accessible
|
|
- Network connectivity is available
|
|
- SFM backend is running and accepting requests
|
|
|
|
## License
|
|
|
|
Proprietary - Internal use only
|
|
|
|
## Version History
|
|
|
|
See [CHANGELOG.md](CHANGELOG.md) for detailed version history.
|
|
|
|
## Support
|
|
|
|
For issues or questions, contact the Seismo development team.
|