Imported from hemapriyan-rk/vimes-industrial-intrusion-detection (AGENTS.md). Install upstream with npx skills add hemapriyan-rk/vimes-industrial-intrusion-detection. Copyright stays with the author.

VIMES — Dual & Multi-Camera Vision Safety System: Agent Guide

This document provides complete instructions, architecture documentation, configuration specifications, and operational workflows for any coding agent working on the VIMES (Vision-based Industrial Monitoring & Emergency Safety System) repository.

1. System Overview & Architecture

VIMES is an industrial-grade edge vision and safety monitoring platform designed to prevent worker-machine collisions and enforce perimeter safety in real-time.

┌────────────────────────────────────────────────────────────────────────┐
│                        VIMES DUAL-CAMERA SYSTEM                        │
└────────────────────────────────────────────────────────────────────────┘
            │                                           │
    [CAMERA 1: ALPHA]                           [CAMERA 2: BRAVO]
 (e.g. 10.169.247.7:8080)                   (e.g. 10.169.247.187:8080)
            │                                           │
            ▼                                           ▼
 ┌─────────────────────┐                     ┌─────────────────────┐
 │  YOLO11 Detection   │                     │  YOLO11 Detection   │
 │  ByteTrack Tracking │                     │  ByteTrack Tracking │
 └──────────┬──────────┘                     └──────────┬──────────┘
            │ Local Tracks                              │ Local Tracks
            └─────────────────────┬─────────────────────┘
                                  │
                                  ▼
                ┌───────────────────────────────────┐
                │   CROSS-CAMERA RE-ID ENGINE       │
                │  - MobileNetV2 128-d Embeddings   │
                │  - Cosine Distance Metric         │
                │  - Spatial-Temporal Gating Matrix │
                │  - Persistent Global ID Registry  │
                └─────────────────┬─────────────────┘
                                  │ Global Tracks
                                  ▼
                ┌───────────────────────────────────┐
                │ PER-CAMERA ANALYTICS PIPELINE     │
                │  - Ground-Plane Homography (m)    │
                │  - Metric Kinematics (v, a, deg)  │
                │  - Polygon Zone Classification    │
                │  - Proximity & TTC Analysis       │
                │  - Deterministic Risk Engine      │
                └─────────────────┬─────────────────┘
                                  │
                                  ▼
                ┌───────────────────────────────────┐
                │   UNIFIED GRID HUD / DASHBOARD    │
                │  - Side-by-Side Live Video Grid   │
                │  - Interactive Hazard Zone Editor │
                │  - Real-Time Risk Alarm Banner    │
                │  - Structured Event Logger        │
                └───────────────────────────────────┘

2. Directory Layout

d:\VIMES\
├── configs/                          # Edge server global configurations
│   ├── hardware.yaml                 # ESP32 serial & sensor thresholds
│   ├── safety.yaml                   # Risk engine safety & TTC thresholds
│   └── zones.yaml                    # Global edge-server zones
├── hardware/                         # ESP32 acquisition firmware & pinouts
│   ├── esp32/firmware/               # FreeRTOS firmware (temp, vibration, PIR)
│   ├── schematics/                   # Hardware wiring diagrams
│   └── wiring/
├── software/
│   ├── analytics/                    # Kinematics, trajectory prediction, proximity
│   ├── app/                          # Edge application orchestrator
│   ├── fusion/                       # Multi-modal sensor fusion engine
│   ├── hardware_interface/           # Serial parser & telemetry models
│   ├── perception/                   # Deep learning detectors & ByteTrack
│   ├── safety/                       # Deterministic risk engine & reason codes
│   ├── tests/                        # Automated unit & integration tests
│   ├── visualization/                # Web Command Center & OpenCV renderers
│   └── vision/                       # Dual & Multi-Camera Vision Engine
│       ├── camera_config.py          # Camera loader & schema parser
│       ├── main.py                   # Single-camera vision runner
│       ├── multi_camera_main.py      # Dual/Multi-camera vision runner
│       ├── config/
│       │   ├── cameras.json          # Dual camera network definitions
│       │   ├── calibration.json      # Homography 4-point ground plane
│       │   ├── zones.json            # Default hazard zones
│       │   ├── zones_cam1.json       # Sector Alpha custom hazard zones
│       │   └── zones_cam2.json       # Sector Bravo custom hazard zones
│       ├── core/
│       │   ├── detector.py           # YOLO11 inference wrapper
│       │   ├── homography.py         # Perspective Ground-Plane Transformer
│       │   ├── kinematics.py         # Velocity, acceleration, and bearing
│       │   ├── logger.py             # Structured JSONL event logging
│       │   ├── proximity.py          # Human-machine pairwise distance & TTC
│       │   ├── reid.py               # MobileNetV2 Cross-Camera Re-ID
│       │   ├── risk_engine.py        # Score (0-100) and Level (SAFE..CRITICAL)
│       │   ├── tracker.py            # ByteTrack multi-object tracker
│       │   ├── visualizer.py         # Real-time OpenCV HUD renderer
│       │   └── zones.py              # Polygon point-in-polygon classifier
│       └── models/                   # YOLO weights (.pt)
└── AGENTS.md                         # This specification guide

3. Dual-Camera Configuration (`cameras.json`)

The primary configuration for network IP cameras is located at: software/vision/config/cameras.json

{
    "_doc": "Dual camera configuration for VIMES industrial monitoring.",
    "layout": {
        "columns": 2
    },
    "cameras": [
        {
            "name": "CAM_01_ALPHA",
            "ip": "10.169.247.7",
            "port": 8080,
            "endpoint": "/video",
            "source": "http://10.169.247.7:8080/video",
            "zones_file": "config/zones_cam1.json",
            "enabled": true,
            "description": "Primary Sector Alpha Camera"
        },
        {
            "name": "CAM_02_BRAVO",
            "ip": "10.169.247.187",
            "port": 8080,
            "endpoint": "/video",
            "source": "http://10.169.247.187:8080/video",
            "zones_file": "config/zones_cam2.json",
            "enabled": true,
            "description": "Secondary Sector Bravo Camera"
        },
        {
            "name": "CAM_03_CHARLIE",
            "source": "synthetic",
            "zones_file": "config/zones_cam3.json",
            "enabled": false,
            "description": "Synthetic test feed (Disabled)"
        }
    ]
}

Camera Parameters

name: Unique display string shown in HUD status badges and event logs.
source: Can be:
- Full HTTP stream URL: http://10.169.247.7:8080/video
- Full RTSP stream URL: rtsp://username:password@10.169.247.7:554/live
- Local webcam device index: "0", "1"
- Video file: "software/data/test_videos/forklift.mp4"
- Synthetic simulation generator: "synthetic"
zones_file: Path to camera-specific hazard zones JSON (relative to software/vision).
enabled: Boolean (true / false) controlling whether the pipeline captures and renders the camera.

4. Execution Commands for Agents & Operators

A. Run Dual/Multi-Camera Pipeline with ESP32 Hardware (Live Network IP Feeds)

python software/vision/multi_camera_main.py --config software/vision/config/cameras.json --serial-port COM5

Integrates IP streams configured in cameras.json with physical ESP32 sensor telemetry on COM5.

B. Run Single-Camera Pipeline (Primary IP Camera / Webcam + ESP32 Hardware)

python software/vision/main.py --serial-port COM5

(Automatically connects to the first enabled camera from cameras.json and ESP32 on COM5)

C. Run Synthetic Dual-Camera Test with ESP32 Hardware (Offline / CI Verification)

To test cross-camera Re-ID, grid HUD, and live physical sensor telemetry without physical IP cameras:

python software/vision/multi_camera_main.py --sources synthetic,synthetic --serial-port COM5

5. Interactive Keyboard & Dashboard Controls

When running the OpenCV visualizer window:

A. View Mode Switching (When NOT in Zone Editor)

Key	View Name	Description
W	Toggle Side-by-Side Dual Windows	Toggles between Single Combined Window and Dual Synchronized Windows (Window 1: Cameras Grid, Window 2: Hardware I/O & Manipulator Cockpit)
1	Master Unified Dashboard	Side-by-side video feeds + Top HUD banner + Bottom Worker Risk Cards ($P_1, P_2$) & Actuator Bar
2	Cameras Grid Only	Full-screen multi-camera video grid with 3D hazard zones
3	Hardware I/O & Decision Matrix	Dedicated instrument deck (Temp gauge with $dT/dt$, Vibration bar, PIR beacon, Machine state, Buzzer/Relay status, Worker Risk breakdown table)
4	Hardware Input Manipulator Deck	Interactive test bench: Override toggle, manual sensor injection controls, 5 scenario presets, and live actuator verification

B. Hardware Input Manipulator & Actuator Controls

Key	Action	Description
M	Toggle Manipulator Mode	Switch between Live ESP32 stream on COM5 and Custom Injection Mode
T	Temp +5°C	Increase manipulated temperature by $+5^\circ\text{C}$
t	Temp -5°C	Decrease manipulated temperature by $-5^\circ\text{C}$
V	Cycle Vibration	Cycle vibration intensity (`0.05 G` $\to$ `0.50 G` $\to$ `0.88 G`)
P	Toggle PIR Motion	Toggle PIR motion presence (`ACTIVE` $\leftrightarrow$ `CLEAR`)
O	Cycle Scenario Preset	Cycle Presets 1–5 (`NORMAL_IDLE`, `MACHINE_RUNNING`, `MACHINE_OVERHEAT`, `VIBRATION_FAULT`, `EMERGENCY_CRITICAL`)
R	Reset Emergency Stop	Manually reset latched Emergency Stop in Safety Engine and unlatch relay

C. Multi-Camera Focus & Interactive Zone Editor

Key	Action	Description
Tab	Cycle Camera Focus	Switches active camera focus between CAM 1 and CAM 2 in the grid
Click	Focus Camera	Left-clicking any camera pane sets it as the active editing target
Z	Enter Zone Editor	Starts interactive polygon hazard zone definition for the active camera
1	Select SAFE Zone	Green pedestrian safe corridor (in zone editor)
2	Select MODERATE Zone	Yellow caution / approach lane (in zone editor)
3	Select HAZARD Zone	Red machine exclusion zone (in zone editor)
Left Click	Add Vertex	Places a vertex point for the polygon on the live video
Right Click	Undo Vertex	Removes the last placed vertex point
Enter	Save Zone	Saves polygon to the active camera's `zones_camX.json`
Esc	Cancel Editor	Discards in-progress polygon edits
C	Clear Zones	Deletes all zones for the active camera
S	Save Screenshot	Captures timestamped frame to `data/events/`
Q	Quit Application	Gracefully releases camera captures and closes windows

6. Cross-Camera Re-ID & Multi-Object Tracking

How Cross-Camera Association Works (`software/vision/core/reid.py`)

Local Tracking: ByteTrack assigns local track IDs within each camera stream.
Feature Extraction: Worker crops ($128 \times 256$) are passed through a MobileNetV2 feature extractor generating 128-dimensional $L_2$-normalized visual embeddings.
Similarity Gating: Cosine distance matrix is evaluated against active global identities: $$\text{Similarity}(u, v) = \frac{u \cdot v}{|u|_2 |v|_2}$$
Spatial-Temporal Constraints: A worker cannot jump across non-adjacent cameras in sub-second intervals. Spatial-temporal gating prevents false cross-camera identity switches.
Global Identity Assignment: If similarity $\ge 0.72$ within spatial-temporal bounds, the existing Global_ID is maintained across camera views.

7. Testing & Quality Assurance

To execute the test suite across all vision, kinematics, Re-ID, hardware manipulator, and safety modules:

python -m pytest software/tests -v

Key Test Modules:

software/tests/safety/test_fusion_decision_engine.py: Tests full 20-Case Sensor-Vision Decision Matrix and temporal hysteresis.
software/tests/unit/test_sensor_interpreter.py: Tests thermal derivative $dT/dt$, vibration hysteresis bands, and machine operational states.
software/tests/unit/test_hardware_manipulator.py: Tests live serial multiplexing, custom sensor injection, and actuator dispatch.
software/tests/vision/test_multi_dashboard_views.py: Tests rendering across all 4 Dashboard Views.
software/tests/vision/test_multicam_config.py: Validates camera configuration loading and fallback resolution.
software/tests/vision/test_reid_multicam.py: Tests cross-camera feature extraction and global ID matching.
software/tests/vision/test_reid_optimizations.py: Tests spatial-temporal gating and embedding caches.
software/tests/vision/test_kinematics_homography.py: Tests ground plane perspective transforms and metric velocities.
software/tests/vision/test_zones_proximity.py: Tests polygon intersection and TTC calculations.
software/tests/vision/test_risk_engine.py: Validates deterministic safety scoring (0–100).

vimes-industrial-intrusion-detection

About

VIMES — Dual & Multi-Camera Vision Safety System: Agent Guide

1. System Overview & Architecture

2. Directory Layout

3. Dual-Camera Configuration (`cameras.json`)

Camera Parameters

4. Execution Commands for Agents & Operators

A. Run Dual/Multi-Camera Pipeline with ESP32 Hardware (Live Network IP Feeds)

B. Run Single-Camera Pipeline (Primary IP Camera / Webcam + ESP32 Hardware)

C. Run Synthetic Dual-Camera Test with ESP32 Hardware (Offline / CI Verification)

5. Interactive Keyboard & Dashboard Controls

A. View Mode Switching (When NOT in Zone Editor)

B. Hardware Input Manipulator & Actuator Controls

C. Multi-Camera Focus & Interactive Zone Editor

6. Cross-Camera Re-ID & Multi-Object Tracking

How Cross-Camera Association Works (`software/vision/core/reid.py`)

7. Testing & Quality Assurance

Key Test Modules:

Use it

Manifest

Reviews

About

VIMES — Dual & Multi-Camera Vision Safety System: Agent Guide

1. System Overview & Architecture

2. Directory Layout

3. Dual-Camera Configuration (cameras.json)

Camera Parameters

4. Execution Commands for Agents & Operators

A. Run Dual/Multi-Camera Pipeline with ESP32 Hardware (Live Network IP Feeds)

B. Run Single-Camera Pipeline (Primary IP Camera / Webcam + ESP32 Hardware)

C. Run Synthetic Dual-Camera Test with ESP32 Hardware (Offline / CI Verification)

5. Interactive Keyboard & Dashboard Controls

A. View Mode Switching (When NOT in Zone Editor)

B. Hardware Input Manipulator & Actuator Controls

C. Multi-Camera Focus & Interactive Zone Editor

6. Cross-Camera Re-ID & Multi-Object Tracking

How Cross-Camera Association Works (software/vision/core/reid.py)

7. Testing & Quality Assurance

Key Test Modules:

Use it

Manifest

Reviews

3. Dual-Camera Configuration (`cameras.json`)

How Cross-Camera Association Works (`software/vision/core/reid.py`)