feat: update to version 0.1.3 with new CLI flags, logging enhancements, and Windows MSI installer
This commit is contained in:
BIN
.gitignore
vendored
BIN
.gitignore
vendored
Binary file not shown.
13
CHANGELOG.md
13
CHANGELOG.md
@@ -5,6 +5,19 @@ All notable changes to this project will be documented in this file.
|
|||||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
|
|
||||||
|
## [0.1.3] - 2026-02-09
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- CLI flags: `--config`, `--log-dir`, `--once`, `--version`
|
||||||
|
- File logging with rotation (default in `C:\ProgramData\ThorIngest\logs`)
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- Default config location to `C:\ProgramData\ThorIngest\config.json` with fallback to local `config.json`
|
||||||
|
- Updated console output to use structured logging
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Windows MSI installer scaffolding under `installer/` (WiX + NSSM setup)
|
||||||
|
|
||||||
## [0.1.1] - 2025-12-08
|
## [0.1.1] - 2025-12-08
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|||||||
87
README.md
87
README.md
@@ -1,6 +1,6 @@
|
|||||||
# Series 4 Ingest Agent
|
# Series 4 Ingest Agent
|
||||||
|
|
||||||
**Version:** 0.1.2
|
**Version:** 0.1.3
|
||||||
|
|
||||||
Micromate (Series 4) ingest agent for Seismo Fleet Manager (SFM).
|
Micromate (Series 4) ingest agent for Seismo Fleet Manager (SFM).
|
||||||
|
|
||||||
@@ -27,10 +27,16 @@ Series 4 Ingest Agent is a Python-based monitoring tool that scans for Micromate
|
|||||||
|
|
||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
1. Clone or copy this repository to your THOR PC/VM
|
1. Download and run the MSI installer (Windows, admin required)
|
||||||
2. Ensure Python 3.6+ is installed
|
2. Edit config at `C:\ProgramData\ThorIngest\config.json`
|
||||||
3. Configure `config.json` (see Configuration section)
|
3. The service starts automatically and runs on boot
|
||||||
4. Run the agent:
|
|
||||||
|
No command line is required for normal use. The installer sets up the Windows Service.
|
||||||
|
|
||||||
|
For development or manual runs:
|
||||||
|
1. Ensure Python 3.6+ is installed
|
||||||
|
2. Configure `config.json` (see Configuration section)
|
||||||
|
3. Run the agent:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python series4_ingest.py
|
python series4_ingest.py
|
||||||
@@ -38,7 +44,11 @@ python series4_ingest.py
|
|||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
All configuration is managed through `config.json` in the same directory as the script.
|
All configuration is managed through `config.json` in:
|
||||||
|
|
||||||
|
```
|
||||||
|
C:\ProgramData\ThorIngest\config.json
|
||||||
|
```
|
||||||
|
|
||||||
### config.json Structure
|
### config.json Structure
|
||||||
|
|
||||||
@@ -75,16 +85,35 @@ If `config.json` is missing or malformed, the agent will:
|
|||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
### Start the Agent
|
### Start the Agent (manual)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python series4_ingest.py
|
python series4_ingest.py
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Common Flags
|
||||||
|
|
||||||
|
- `--config <path>`: Override config file path
|
||||||
|
- `--log-dir <path>`: Override log directory
|
||||||
|
- `--once`: Run one scan and exit
|
||||||
|
- `--version`: Print version and exit
|
||||||
|
|
||||||
|
### Run Once (manual)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python series4_ingest.py --once
|
||||||
|
```
|
||||||
|
|
||||||
|
### Show Version
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python series4_ingest.py --version
|
||||||
|
```
|
||||||
|
|
||||||
### Console Output Example
|
### Console Output Example
|
||||||
|
|
||||||
```
|
```
|
||||||
Series 4 Ingest Agent — Micromate Heartbeat (v0.1.2)
|
Series 4 Ingest Agent — Micromate Heartbeat (v0.1.3)
|
||||||
THORDATA root: C:\THORDATA
|
THORDATA root: C:\THORDATA
|
||||||
Now: 2025-12-08 14:30:00
|
Now: 2025-12-08 14:30:00
|
||||||
--------------------------------------------------------------------------------
|
--------------------------------------------------------------------------------
|
||||||
@@ -96,10 +125,48 @@ Total units: 3
|
|||||||
Next scan in 60 seconds...
|
Next scan in 60 seconds...
|
||||||
```
|
```
|
||||||
|
|
||||||
### Stop the Agent
|
### Stop the Agent (manual)
|
||||||
|
|
||||||
Press `Ctrl+C` to gracefully stop the agent.
|
Press `Ctrl+C` to gracefully stop the agent.
|
||||||
|
|
||||||
|
## Windows Service Installation
|
||||||
|
|
||||||
|
The MSI installs a Windows Service named `ThorIngest` using NSSM. It runs at startup.
|
||||||
|
|
||||||
|
**Check status**
|
||||||
|
|
||||||
|
```
|
||||||
|
sc query ThorIngest
|
||||||
|
```
|
||||||
|
|
||||||
|
**Start / stop**
|
||||||
|
|
||||||
|
```
|
||||||
|
sc start ThorIngest
|
||||||
|
sc stop ThorIngest
|
||||||
|
```
|
||||||
|
|
||||||
|
**Service Logs**
|
||||||
|
|
||||||
|
Logs are written to:
|
||||||
|
```
|
||||||
|
C:\ProgramData\ThorIngest\logs\thor_ingest.log
|
||||||
|
```
|
||||||
|
|
||||||
|
## Updating
|
||||||
|
|
||||||
|
1. Download the newer MSI from the internal file share/URL.
|
||||||
|
2. Run the MSI to upgrade in place.
|
||||||
|
3. Your existing config and logs are preserved.
|
||||||
|
|
||||||
|
## Building the EXE (PyInstaller)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pyinstaller --onefile --name thor-ingest-agent series4_ingest.py
|
||||||
|
```
|
||||||
|
|
||||||
|
The output EXE will be in `dist/`.
|
||||||
|
|
||||||
## Status Classification
|
## Status Classification
|
||||||
|
|
||||||
Units are classified based on the age of their last MLG file:
|
Units are classified based on the age of their last MLG file:
|
||||||
@@ -178,7 +245,7 @@ C:\THORDATA\
|
|||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
### Config file not found
|
### Config file not found
|
||||||
If you see `[WARN] Config file not found`, create `config.json` in the same directory as `series4_ingest.py`.
|
If you see `[WARN] Config file not found`, create `config.json` in `C:\ProgramData\ThorIngest\config.json` (or pass `--config`).
|
||||||
|
|
||||||
### THORDATA path doesn't exist
|
### THORDATA path doesn't exist
|
||||||
Verify the `thordata_path` in `config.json` points to the correct directory.
|
Verify the `thordata_path` in `config.json` points to the correct directory.
|
||||||
|
|||||||
39
RELEASE.md
Normal file
39
RELEASE.md
Normal file
@@ -0,0 +1,39 @@
|
|||||||
|
# Release Packaging (Windows)
|
||||||
|
|
||||||
|
Goal: Users download a release artifact and install without using the command line.
|
||||||
|
|
||||||
|
## Deliverables
|
||||||
|
|
||||||
|
- `ThorIngest.msi` (primary installer)
|
||||||
|
- Optional: `thor-ingest-agent.exe` (for internal diagnostics)
|
||||||
|
|
||||||
|
## Build Steps (internal)
|
||||||
|
|
||||||
|
1. Build EXE:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pyinstaller --onefile --name thor-ingest-agent series4_ingest.py
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Place NSSM:
|
||||||
|
|
||||||
|
```
|
||||||
|
thor-ingest-agent/installer/tools/nssm.exe
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Build MSI:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
./installer/build.ps1
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Publish:
|
||||||
|
|
||||||
|
- Upload `installer/ThorIngest.msi` to the internal release URL.
|
||||||
|
- Provide a short “Download and run the installer” instruction to users.
|
||||||
|
|
||||||
|
## Install UX
|
||||||
|
|
||||||
|
- Double‑click the MSI
|
||||||
|
- Minimal UI (Next/Install/Finish)
|
||||||
|
- Service installed and started automatically
|
||||||
114
installer/Product.wxs
Normal file
114
installer/Product.wxs
Normal file
@@ -0,0 +1,114 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
|
||||||
|
<Product
|
||||||
|
Id="*"
|
||||||
|
Name="Thor Ingest Agent"
|
||||||
|
Language="1033"
|
||||||
|
Version="0.1.3"
|
||||||
|
Manufacturer="Seismo"
|
||||||
|
UpgradeCode="0E2EFA7E-2A75-4A2E-A5D5-9E9C6B99C0E1">
|
||||||
|
|
||||||
|
<Package InstallerVersion="500" Compressed="yes" InstallScope="perMachine" Platform="x64" />
|
||||||
|
<MajorUpgrade DowngradeErrorMessage="A newer version of [ProductName] is already installed." />
|
||||||
|
<MediaTemplate />
|
||||||
|
|
||||||
|
<Property Id="WIXUI_INSTALLDIR" Value="INSTALLFOLDER" />
|
||||||
|
|
||||||
|
<Directory Id="TARGETDIR" Name="SourceDir">
|
||||||
|
<Directory Id="ProgramFiles64Folder">
|
||||||
|
<Directory Id="INSTALLFOLDER" Name="ThorIngest">
|
||||||
|
<Component Id="MainExecutable" Guid="9A9F3E64-93A0-4D9F-8D2E-1C4B8B9E5E7A">
|
||||||
|
<File Id="ThorIngestExe" Source="..\dist\thor-ingest-agent.exe" KeyPath="yes" />
|
||||||
|
</Component>
|
||||||
|
<Component Id="ConfigExample" Guid="A6B1877F-3C8C-4CE1-9C30-70C8E2D6CB8F">
|
||||||
|
<File Id="ConfigExampleFile" Source="..\config.example.json" KeyPath="yes" />
|
||||||
|
</Component>
|
||||||
|
<Component Id="NssmBinary" Guid="F9E2D403-4F1C-4E55-8A34-0D7CBA74F2BC">
|
||||||
|
<File Id="NssmExe" Source="tools\nssm.exe" KeyPath="yes" />
|
||||||
|
</Component>
|
||||||
|
</Directory>
|
||||||
|
</Directory>
|
||||||
|
<Directory Id="CommonAppDataFolder">
|
||||||
|
<Directory Id="ProgramDataThorIngest" Name="ThorIngest">
|
||||||
|
<Directory Id="ProgramDataLogs" Name="logs" />
|
||||||
|
</Directory>
|
||||||
|
</Directory>
|
||||||
|
</Directory>
|
||||||
|
|
||||||
|
<Feature Id="MainFeature" Title="Thor Ingest Agent" Level="1">
|
||||||
|
<ComponentRef Id="MainExecutable" />
|
||||||
|
<ComponentRef Id="ConfigExample" />
|
||||||
|
<ComponentRef Id="NssmBinary" />
|
||||||
|
</Feature>
|
||||||
|
|
||||||
|
<CustomAction
|
||||||
|
Id="CreateProgramDataDirs"
|
||||||
|
Directory="TARGETDIR"
|
||||||
|
ExeCommand="cmd.exe /c if not exist "[CommonAppDataFolder]ThorIngest\" mkdir "[CommonAppDataFolder]ThorIngest\" & if not exist "[CommonAppDataFolder]ThorIngest\logs\" mkdir "[CommonAppDataFolder]ThorIngest\logs\""
|
||||||
|
Execute="deferred"
|
||||||
|
Impersonate="no"
|
||||||
|
Return="check" />
|
||||||
|
|
||||||
|
<CustomAction
|
||||||
|
Id="CopyDefaultConfig"
|
||||||
|
Directory="TARGETDIR"
|
||||||
|
ExeCommand="cmd.exe /c if not exist "[CommonAppDataFolder]ThorIngest\config.json" copy "[INSTALLFOLDER]config.example.json" "[CommonAppDataFolder]ThorIngest\config.json""
|
||||||
|
Execute="deferred"
|
||||||
|
Impersonate="no"
|
||||||
|
Return="check" />
|
||||||
|
|
||||||
|
<CustomAction
|
||||||
|
Id="InstallService"
|
||||||
|
Directory="INSTALLFOLDER"
|
||||||
|
ExeCommand=""[INSTALLFOLDER]nssm.exe" install ThorIngest "[INSTALLFOLDER]thor-ingest-agent.exe" --config "[CommonAppDataFolder]ThorIngest\config.json" --log-dir "[CommonAppDataFolder]ThorIngest\logs""
|
||||||
|
Execute="deferred"
|
||||||
|
Impersonate="no"
|
||||||
|
Return="check" />
|
||||||
|
|
||||||
|
<CustomAction
|
||||||
|
Id="ConfigureServiceStart"
|
||||||
|
Directory="INSTALLFOLDER"
|
||||||
|
ExeCommand=""[INSTALLFOLDER]nssm.exe" set ThorIngest Start SERVICE_AUTO_START"
|
||||||
|
Execute="deferred"
|
||||||
|
Impersonate="no"
|
||||||
|
Return="check" />
|
||||||
|
|
||||||
|
<CustomAction
|
||||||
|
Id="ConfigureServiceRestart"
|
||||||
|
Directory="INSTALLFOLDER"
|
||||||
|
ExeCommand=""[INSTALLFOLDER]nssm.exe" set ThorIngest AppExit Default Restart"
|
||||||
|
Execute="deferred"
|
||||||
|
Impersonate="no"
|
||||||
|
Return="check" />
|
||||||
|
|
||||||
|
<CustomAction
|
||||||
|
Id="StartService"
|
||||||
|
Directory="INSTALLFOLDER"
|
||||||
|
ExeCommand=""[INSTALLFOLDER]nssm.exe" start ThorIngest"
|
||||||
|
Execute="deferred"
|
||||||
|
Impersonate="no"
|
||||||
|
Return="check" />
|
||||||
|
|
||||||
|
<CustomAction
|
||||||
|
Id="RemoveService"
|
||||||
|
Directory="INSTALLFOLDER"
|
||||||
|
ExeCommand=""[INSTALLFOLDER]nssm.exe" remove ThorIngest confirm"
|
||||||
|
Execute="deferred"
|
||||||
|
Impersonate="no"
|
||||||
|
Return="check" />
|
||||||
|
|
||||||
|
<InstallExecuteSequence>
|
||||||
|
<Custom Action="CreateProgramDataDirs" After="InstallFiles">NOT Installed</Custom>
|
||||||
|
<Custom Action="CopyDefaultConfig" After="CreateProgramDataDirs">NOT Installed</Custom>
|
||||||
|
<Custom Action="InstallService" After="CopyDefaultConfig">NOT Installed</Custom>
|
||||||
|
<Custom Action="ConfigureServiceStart" After="InstallService">NOT Installed</Custom>
|
||||||
|
<Custom Action="ConfigureServiceRestart" After="ConfigureServiceStart">NOT Installed</Custom>
|
||||||
|
<Custom Action="StartService" After="ConfigureServiceRestart">NOT Installed</Custom>
|
||||||
|
<Custom Action="RemoveService" Before="RemoveFiles">REMOVE="ALL"</Custom>
|
||||||
|
</InstallExecuteSequence>
|
||||||
|
|
||||||
|
<UIRef Id="WixUI_Minimal" />
|
||||||
|
<UIRef Id="WixUI_ErrorProgressText" />
|
||||||
|
|
||||||
|
</Product>
|
||||||
|
</Wix>
|
||||||
21
installer/README.md
Normal file
21
installer/README.md
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
# Thor Ingest MSI Installer
|
||||||
|
|
||||||
|
This folder contains the WiX MSI definition and build script.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- WiX Toolset v3.11+ (set `WIX` env var to the WiX `bin` path)
|
||||||
|
- `nssm.exe` placed at `installer/tools/nssm.exe`
|
||||||
|
- `thor-ingest-agent.exe` built into `thor-ingest-agent/dist/`
|
||||||
|
|
||||||
|
## Build
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
./build.ps1
|
||||||
|
```
|
||||||
|
|
||||||
|
The MSI output will be:
|
||||||
|
|
||||||
|
```
|
||||||
|
installer/ThorIngest.msi
|
||||||
|
```
|
||||||
30
installer/build.ps1
Normal file
30
installer/build.ps1
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
$ErrorActionPreference = "Stop"
|
||||||
|
|
||||||
|
$WixBin = $env:WIX
|
||||||
|
if (-not $WixBin) {
|
||||||
|
Write-Error "WIX environment variable not set. Set WIX to your WiX Toolset bin path (e.g., C:\Program Files (x86)\WiX Toolset v3.11\bin)."
|
||||||
|
}
|
||||||
|
|
||||||
|
$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
|
||||||
|
$InstallerDir = $ScriptDir
|
||||||
|
$RootDir = Resolve-Path (Join-Path $InstallerDir "..")
|
||||||
|
|
||||||
|
$ProductWxs = Join-Path $InstallerDir "Product.wxs"
|
||||||
|
$DistDir = Join-Path $RootDir "dist"
|
||||||
|
$NssmPath = Join-Path $InstallerDir "tools\nssm.exe"
|
||||||
|
|
||||||
|
if (-not (Test-Path $DistDir)) {
|
||||||
|
Write-Error "dist folder not found. Build the EXE with PyInstaller first."
|
||||||
|
}
|
||||||
|
|
||||||
|
if (-not (Test-Path $NssmPath)) {
|
||||||
|
Write-Error "Missing nssm.exe at $NssmPath. Place the NSSM binary there before building."
|
||||||
|
}
|
||||||
|
|
||||||
|
$candle = Join-Path $WixBin "candle.exe"
|
||||||
|
$light = Join-Path $WixBin "light.exe"
|
||||||
|
|
||||||
|
& $candle -nologo $ProductWxs -out (Join-Path $InstallerDir "Product.wixobj")
|
||||||
|
& $light -nologo -ext WixUIExtension -ext WixUtilExtension -out (Join-Path $InstallerDir "ThorIngest.msi") (Join-Path $InstallerDir "Product.wixobj")
|
||||||
|
|
||||||
|
Write-Host "Built MSI: $InstallerDir\ThorIngest.msi"
|
||||||
1
installer/tools/README.txt
Normal file
1
installer/tools/README.txt
Normal file
@@ -0,0 +1 @@
|
|||||||
|
Place nssm.exe in this folder before building the MSI.
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
"""
|
"""
|
||||||
Series 4 Ingest Agent — v0.1.2
|
Series 4 Ingest Agent — v0.1.3
|
||||||
|
|
||||||
Micromate (Series 4) ingest agent for Seismo Fleet Manager (SFM).
|
Micromate (Series 4) ingest agent for Seismo Fleet Manager (SFM).
|
||||||
|
|
||||||
@@ -14,6 +14,9 @@ Behavior:
|
|||||||
No roster. SFM backend decides what to do with each unit.
|
No roster. SFM backend decides what to do with each unit.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import logging
|
||||||
|
from logging.handlers import RotatingFileHandler
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
import time
|
import time
|
||||||
@@ -30,9 +33,14 @@ try:
|
|||||||
except ImportError:
|
except ImportError:
|
||||||
urllib = None # type: ignore
|
urllib = None # type: ignore
|
||||||
|
|
||||||
|
__version__ = "0.1.3"
|
||||||
|
|
||||||
# ---------------- Config ----------------
|
# ---------------- Config ----------------
|
||||||
|
|
||||||
def load_config(config_path: str = "config.json") -> Dict[str, Any]:
|
DEFAULT_CONFIG_PATH = r"C:\ProgramData\ThorIngest\config.json"
|
||||||
|
DEFAULT_LOG_DIR = r"C:\ProgramData\ThorIngest\logs"
|
||||||
|
|
||||||
|
def load_config(config_path: Optional[str] = None) -> Dict[str, Any]:
|
||||||
"""
|
"""
|
||||||
Load configuration from a JSON file.
|
Load configuration from a JSON file.
|
||||||
|
|
||||||
@@ -48,12 +56,28 @@ def load_config(config_path: str = "config.json") -> Dict[str, Any]:
|
|||||||
"debug": True
|
"debug": True
|
||||||
}
|
}
|
||||||
|
|
||||||
# Try to find config file relative to script location
|
# Resolve config path order:
|
||||||
|
# 1) explicit config_path (if provided)
|
||||||
|
# 2) ProgramData default
|
||||||
|
# 3) local directory config.json (script dir)
|
||||||
script_dir = os.path.dirname(os.path.abspath(__file__))
|
script_dir = os.path.dirname(os.path.abspath(__file__))
|
||||||
full_config_path = os.path.join(script_dir, config_path)
|
candidates = []
|
||||||
|
if config_path:
|
||||||
|
candidates.append(config_path)
|
||||||
|
candidates.append(DEFAULT_CONFIG_PATH)
|
||||||
|
candidates.append(os.path.join(script_dir, "config.json"))
|
||||||
|
|
||||||
if not os.path.exists(full_config_path):
|
full_config_path = None
|
||||||
print(f"[WARN] Config file not found at {full_config_path}, using defaults", file=sys.stderr)
|
for path in candidates:
|
||||||
|
if os.path.exists(path):
|
||||||
|
full_config_path = path
|
||||||
|
break
|
||||||
|
|
||||||
|
if full_config_path is None:
|
||||||
|
print(
|
||||||
|
f"[WARN] Config file not found in: {', '.join(candidates)}, using defaults",
|
||||||
|
file=sys.stderr
|
||||||
|
)
|
||||||
return defaults
|
return defaults
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -68,16 +92,15 @@ def load_config(config_path: str = "config.json") -> Dict[str, Any]:
|
|||||||
print(f"[WARN] Error loading config file: {e}, using defaults", file=sys.stderr)
|
print(f"[WARN] Error loading config file: {e}, using defaults", file=sys.stderr)
|
||||||
return defaults
|
return defaults
|
||||||
|
|
||||||
# Load configuration
|
THORDATA_PATH = r"C:\THORDATA"
|
||||||
config = load_config()
|
SCAN_INTERVAL = 60
|
||||||
|
LATE_DAYS = 2
|
||||||
|
STALE_DAYS = 60
|
||||||
|
SFM_ENDPOINT = ""
|
||||||
|
SFM_TIMEOUT = 5
|
||||||
|
DEBUG = True
|
||||||
|
|
||||||
THORDATA_PATH = config["thordata_path"]
|
logger = logging.getLogger("thor_ingest")
|
||||||
SCAN_INTERVAL = config["scan_interval"]
|
|
||||||
LATE_DAYS = config["late_days"]
|
|
||||||
STALE_DAYS = config["stale_days"]
|
|
||||||
SFM_ENDPOINT = config["sfm_endpoint"]
|
|
||||||
SFM_TIMEOUT = config["sfm_timeout"]
|
|
||||||
DEBUG = config["debug"]
|
|
||||||
|
|
||||||
# Regex: UM12345_YYYYMMDDHHMMSS.MLG
|
# Regex: UM12345_YYYYMMDDHHMMSS.MLG
|
||||||
MLG_PATTERN = re.compile(r"^(UM\d+)_([0-9]{14})\.MLG$", re.IGNORECASE)
|
MLG_PATTERN = re.compile(r"^(UM\d+)_([0-9]{14})\.MLG$", re.IGNORECASE)
|
||||||
@@ -88,7 +111,7 @@ MLG_PATTERN = re.compile(r"^(UM\d+)_([0-9]{14})\.MLG$", re.IGNORECASE)
|
|||||||
|
|
||||||
def debug(msg: str) -> None:
|
def debug(msg: str) -> None:
|
||||||
if DEBUG:
|
if DEBUG:
|
||||||
print(f"[DEBUG] {msg}", file=sys.stderr, flush=True)
|
logger.debug(msg)
|
||||||
|
|
||||||
|
|
||||||
def parse_mlg_filename(name: str) -> Optional[Tuple[str, datetime]]:
|
def parse_mlg_filename(name: str) -> Optional[Tuple[str, datetime]]:
|
||||||
@@ -151,8 +174,8 @@ def scan_thordata(root: str) -> Dict[str, Dict[str, Any]]:
|
|||||||
try:
|
try:
|
||||||
unit_dirs = os.listdir(project_path)
|
unit_dirs = os.listdir(project_path)
|
||||||
except OSError as e:
|
except OSError as e:
|
||||||
debug(f"Failed to list project '{project_path}': {e}")
|
debug(f"Failed to list project '{project_path}': {e}")
|
||||||
continue
|
continue
|
||||||
|
|
||||||
for unit_name in unit_dirs:
|
for unit_name in unit_dirs:
|
||||||
unit_path = os.path.join(project_path, unit_name)
|
unit_path = os.path.join(project_path, unit_name)
|
||||||
@@ -249,6 +272,8 @@ def format_age(td: timedelta) -> str:
|
|||||||
|
|
||||||
def clear_console() -> None:
|
def clear_console() -> None:
|
||||||
"""Clear the console screen (Windows / *nix)."""
|
"""Clear the console screen (Windows / *nix)."""
|
||||||
|
if not sys.stdout.isatty():
|
||||||
|
return
|
||||||
if os.name == "nt":
|
if os.name == "nt":
|
||||||
os.system("cls")
|
os.system("cls")
|
||||||
else:
|
else:
|
||||||
@@ -265,13 +290,13 @@ def print_heartbeat(unit_map: Dict[str, Dict[str, Any]]) -> None:
|
|||||||
now = datetime.now()
|
now = datetime.now()
|
||||||
|
|
||||||
clear_console()
|
clear_console()
|
||||||
print("Series 4 Ingest Agent — Micromate Heartbeat (v0.1.2)")
|
logger.info("Series 4 Ingest Agent — Micromate Heartbeat (v%s)", __version__)
|
||||||
print(f"THORDATA root: {THORDATA_PATH}")
|
logger.info("THORDATA root: %s", THORDATA_PATH)
|
||||||
print(f"Now: {now.strftime('%Y-%m-%d %H:%M:%S')}")
|
logger.info("Now: %s", now.strftime('%Y-%m-%d %H:%M:%S'))
|
||||||
print("-" * 80)
|
logger.info("-" * 80)
|
||||||
|
|
||||||
if not unit_map:
|
if not unit_map:
|
||||||
print("No units found (no .MLG files detected).")
|
logger.info("No units found (no .MLG files detected).")
|
||||||
return
|
return
|
||||||
|
|
||||||
# Sort by unit_id for stable output
|
# Sort by unit_id for stable output
|
||||||
@@ -285,15 +310,18 @@ def print_heartbeat(unit_map: Dict[str, Dict[str, Any]]) -> None:
|
|||||||
age_str = format_age(age_td)
|
age_str = format_age(age_td)
|
||||||
last_str = last_call.strftime("%Y-%m-%d %H:%M:%S")
|
last_str = last_call.strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
|
||||||
print(
|
logger.info(
|
||||||
f"{unit_id:<8} {status:<6} Age: {age_str:<8} "
|
"%-8s %-6s Age: %-8s Last: %s Project: %s",
|
||||||
f"Last: {last_str} Project: {project}"
|
unit_id,
|
||||||
|
status,
|
||||||
|
age_str,
|
||||||
|
last_str,
|
||||||
|
project,
|
||||||
)
|
)
|
||||||
|
|
||||||
print("-" * 80)
|
logger.info("-" * 80)
|
||||||
print(f"Total units: {len(unit_map)}")
|
logger.info("Total units: %s", len(unit_map))
|
||||||
print(f"Next scan in {SCAN_INTERVAL} seconds...")
|
logger.info("Next scan in %s seconds...", SCAN_INTERVAL)
|
||||||
sys.stdout.flush()
|
|
||||||
|
|
||||||
|
|
||||||
def build_sfm_payload(unit_map: Dict[str, Dict[str, Any]]) -> Dict[str, Any]:
|
def build_sfm_payload(unit_map: Dict[str, Dict[str, Any]]) -> Dict[str, Any]:
|
||||||
@@ -372,10 +400,9 @@ def emit_sfm_payload(unit_map: Dict[str, Dict[str, Any]]) -> None:
|
|||||||
return
|
return
|
||||||
|
|
||||||
if urllib is None:
|
if urllib is None:
|
||||||
print(
|
logger.warning(
|
||||||
"[WARN] urllib not available; cannot POST to SFM. "
|
"urllib not available; cannot POST to SFM. "
|
||||||
"Install standard Python or disable SFM_ENDPOINT.",
|
"Install standard Python or disable SFM_ENDPOINT."
|
||||||
file=sys.stderr,
|
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -394,44 +421,99 @@ def emit_sfm_payload(unit_map: Dict[str, Dict[str, Any]]) -> None:
|
|||||||
_ = resp.read() # we don't care about the body for now
|
_ = resp.read() # we don't care about the body for now
|
||||||
debug(f"SFM POST OK: HTTP {resp.status}")
|
debug(f"SFM POST OK: HTTP {resp.status}")
|
||||||
except urllib.error.URLError as e:
|
except urllib.error.URLError as e:
|
||||||
print(f"[WARN] Failed to POST to SFM: {e}", file=sys.stderr)
|
logger.warning("Failed to POST to SFM: %s", e)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"[WARN] Unexpected error during SFM POST: {e}", file=sys.stderr)
|
logger.warning("Unexpected error during SFM POST: %s", e)
|
||||||
|
|
||||||
|
|
||||||
|
def setup_logging(log_dir: str) -> None:
|
||||||
|
os.makedirs(log_dir, exist_ok=True)
|
||||||
|
|
||||||
|
logger.setLevel(logging.DEBUG if DEBUG else logging.INFO)
|
||||||
|
|
||||||
|
# File handler with rotation
|
||||||
|
file_handler = RotatingFileHandler(
|
||||||
|
os.path.join(log_dir, "thor_ingest.log"),
|
||||||
|
maxBytes=5 * 1024 * 1024,
|
||||||
|
backupCount=5,
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
file_handler.setLevel(logging.DEBUG if DEBUG else logging.INFO)
|
||||||
|
file_format = logging.Formatter("%(asctime)s [%(levelname)s] %(message)s")
|
||||||
|
file_handler.setFormatter(file_format)
|
||||||
|
|
||||||
|
# Console handler (clean output)
|
||||||
|
console_handler = logging.StreamHandler(sys.stdout)
|
||||||
|
console_handler.setLevel(logging.DEBUG if DEBUG else logging.INFO)
|
||||||
|
console_format = logging.Formatter("%(message)s")
|
||||||
|
console_handler.setFormatter(console_format)
|
||||||
|
|
||||||
|
# Avoid duplicate handlers if main is re-entered
|
||||||
|
if not logger.handlers:
|
||||||
|
logger.addHandler(file_handler)
|
||||||
|
logger.addHandler(console_handler)
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
print("Starting Series 4 Ingest Agent (Micromate) v0.1.2")
|
parser = argparse.ArgumentParser(description="Series 4 Ingest Agent (Micromate)")
|
||||||
print(f"THORDATA_PATH = {THORDATA_PATH}")
|
parser.add_argument("--config", help="Path to config.json")
|
||||||
print(f"SCAN_INTERVAL = {SCAN_INTERVAL} seconds")
|
parser.add_argument("--log-dir", help="Directory for log output")
|
||||||
print(f"LATE_DAYS = {LATE_DAYS}, STALE_DAYS = {STALE_DAYS}")
|
parser.add_argument("--once", action="store_true", help="Run one scan and exit")
|
||||||
|
parser.add_argument("--version", action="store_true", help="Print version and exit")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
if args.version:
|
||||||
|
print(__version__)
|
||||||
|
return
|
||||||
|
|
||||||
|
config = load_config(args.config)
|
||||||
|
|
||||||
|
global THORDATA_PATH, SCAN_INTERVAL, LATE_DAYS, STALE_DAYS, SFM_ENDPOINT, SFM_TIMEOUT, DEBUG
|
||||||
|
THORDATA_PATH = config["thordata_path"]
|
||||||
|
SCAN_INTERVAL = config["scan_interval"]
|
||||||
|
LATE_DAYS = config["late_days"]
|
||||||
|
STALE_DAYS = config["stale_days"]
|
||||||
|
SFM_ENDPOINT = config["sfm_endpoint"]
|
||||||
|
SFM_TIMEOUT = config["sfm_timeout"]
|
||||||
|
DEBUG = config["debug"]
|
||||||
|
|
||||||
|
log_dir = args.log_dir or DEFAULT_LOG_DIR
|
||||||
|
setup_logging(log_dir)
|
||||||
|
|
||||||
|
logger.info("Starting Series 4 Ingest Agent (Micromate) v%s", __version__)
|
||||||
|
logger.info("THORDATA_PATH = %s", THORDATA_PATH)
|
||||||
|
logger.info("SCAN_INTERVAL = %s seconds", SCAN_INTERVAL)
|
||||||
|
logger.info("LATE_DAYS = %s, STALE_DAYS = %s", LATE_DAYS, STALE_DAYS)
|
||||||
if not os.path.isdir(THORDATA_PATH):
|
if not os.path.isdir(THORDATA_PATH):
|
||||||
print(f"[WARN] THORDATA_PATH does not exist: {THORDATA_PATH}", file=sys.stderr)
|
logger.warning("THORDATA_PATH does not exist: %s", THORDATA_PATH)
|
||||||
|
|
||||||
loop_counter = 0
|
loop_counter = 0
|
||||||
|
|
||||||
try:
|
try:
|
||||||
while True:
|
while True:
|
||||||
loop_counter += 1
|
loop_counter += 1
|
||||||
print(f"\n[LOOP] Iteration {loop_counter} starting...", flush=True)
|
logger.info("\n[LOOP] Iteration %s starting...", loop_counter)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
unit_map = scan_thordata(THORDATA_PATH)
|
unit_map = scan_thordata(THORDATA_PATH)
|
||||||
debug(f"scan_thordata found {len(unit_map)} units")
|
debug(f"scan_thordata found {len(unit_map)} units")
|
||||||
print_heartbeat(unit_map)
|
print_heartbeat(unit_map)
|
||||||
emit_sfm_payload(unit_map)
|
emit_sfm_payload(unit_map)
|
||||||
print("[LOOP] Iteration complete, entering sleep...", flush=True)
|
logger.info("[LOOP] Iteration complete, entering sleep...")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
# Catch-all so a single error doesn't kill the loop
|
# Catch-all so a single error doesn't kill the loop
|
||||||
print(f"[ERROR] Exception in main loop: {e}", file=sys.stderr)
|
logger.error("Exception in main loop: %s", e)
|
||||||
sys.stderr.flush()
|
|
||||||
|
if args.once:
|
||||||
|
break
|
||||||
|
|
||||||
# Sleep in 1-second chunks to avoid VM time drift weirdness
|
# Sleep in 1-second chunks to avoid VM time drift weirdness
|
||||||
for i in range(SCAN_INTERVAL):
|
for i in range(SCAN_INTERVAL):
|
||||||
time.sleep(1)
|
time.sleep(1)
|
||||||
print("[LOOP] Woke up for next scan", flush=True)
|
logger.info("[LOOP] Woke up for next scan")
|
||||||
|
|
||||||
except KeyboardInterrupt:
|
except KeyboardInterrupt:
|
||||||
print("\nSeries 4 Ingest Agent stopped by user.")
|
logger.info("\nSeries 4 Ingest Agent stopped by user.")
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user