# Arjuna for TASRA Mines — Real-Time Data Pipeline and Failure Handling

**Scope.** How camera video, events and telemetry reach the Arjuna Command Centre in real time, what the system does automatically when any part of the chain fails, and how data is recovered afterwards. Companion to the architecture diagram (Full architecture, Remote & vehicle flow and Resilience & fallback views).

**Status.** Proposal, September 2026. Figures are design targets sized for a mine-site deployment; they are confirmed during site survey and factory acceptance testing. Everything in this document runs on the TASRA intranet — no camera, NVR or edge node ever needs Internet access.

---

## 1. The one-page version

1. **Every camera is recorded as close to itself as possible.** Fixed cameras record on the zone's Local NVR; remote cameras record on their SD card and edge node; vehicles record on the Mobile NVR. The central site is never a precondition for recording.
2. **Two streams per camera.** The main stream (full resolution) is the evidence copy and stays on the NVR. The sub-stream (640p, 5–8 fps sampled) is what Arjuna AI analyses and what the live wall shows. Alerts pull the high-resolution clip on demand.
3. **Edge AI keeps working without the core.** The Edge Arjuna Node runs the same detectors locally, buffers events and clips for 72 hours, and replays them to the central Event Engine when the link returns.
4. **Every link has a fallback and every buffer has a size.** Fibre → PtP wireless → private LTE → secured 4G/5G VPN. NVR 30 days, edge 72 h, vehicle 72 h, camera SD 7 days.
5. **The operator always knows.** Camera Health turns amber within 30 s of a problem and red within 90 s, with an automatic ticket. Nothing fails silently.

---

## 2. End-to-end real-time path

### 2.1 Stage by stage

| # | Stage | What happens | Protocol / mechanism | Runs where |
|---|-------|--------------|----------------------|-----------|
| 1 | Capture | Camera encodes main stream (1080p–4 MP @ 15 fps, H.265) and sub-stream (640p @ 8 fps) | ONVIF Profile S/T, RTSP over TCP | Camera |
| 2 | Zone aggregation | Industrial PoE switch powers cameras and forwards both streams | 802.3bt PoE, VLAN 10, ERPS ring uplink | Zone cabinet |
| 3 | Local recording | Local NVR records the main stream continuously (Profile G), keeps 30 days on RAID 5 | RTSP pull, ONVIF Profile G playback | Zone / site NVR |
| 4 | Discovery and normalisation | Edge Gateway discovers cameras, proxies the sub-stream, publishes a health beacon every 10 s | ONVIF discovery, RTSP proxy, MQTT beacon | Edge node |
| 5 | Edge inference (optional) | Edge Arjuna Node runs PPE / intrusion / vehicle detectors locally on 8–16 streams, cuts ±10 s clips on events | GPU inference, local clip buffer 72 h | Edge node |
| 6 | Transport | Streams and events cross the mine network — fibre first, wireless for remote and moving assets | 10G fibre ring, PtP / PtMP 5 GHz, Wi-Fi 6, private LTE, VPN fallback | Mine connectivity |
| 7 | Segmentation | Traffic is confined to surveillance VLANs; OT firewall allows only gateway → Arjuna segment | VLAN 10–60, allow-list rules, IDS | Core network |
| 8 | Adaptation | ONVIF / RTSP / SDK / NVR-channel / dashcam adapters convert every vendor to one schema | Adapter per source type, auto-reconnect 5 s | Arjuna segment |
| 9 | Unified Stream Gateway | Assigns stream-id, exposes sub-stream + metadata, applies backpressure and retry | Internal gRPC / RTSP re-publish | Arjuna segment |
| 10 | Stream Manager | Hardware decode (NVDEC), samples 5–8 fps per stream for AI | GPU decode | Arjuna core |
| 11 | AI Scheduler + Inference | Assigns streams to N+1 GPU workers by zone priority; detectors run < 150 ms per frame | Batch 8–16 | Arjuna core |
| 12 | Tracking + Rules | Multi-object tracking, zone / line-cross / dwell / direction / speed rules | Rule engine | Arjuna core |
| 13 | Event Engine | De-duplicates, scores severity, pulls the high-res clip from NVR or edge, writes metadata | HTTPS clip pull, metadata DB | Arjuna core |
| 14 | Message Bus | Publishes events (MQTT QoS 1 for UI push, Kafka topic with 7-day retention for replay) | MQTT + Kafka | Arjuna core |
| 15 | Storage | Clips and metadata land in tiered storage: hot 30 d SSD, warm 90 d, cold 365 d, object-lock for evidence | NAS / SAN | Arjuna core |
| 16 | Command Centre | Live map, multi-camera wall (WebRTC), alerts, search, health, reports, SSO | HTTPS, WebRTC | Operator |

### 2.2 Data types and where each one flows

| Data | Source | Path | Rate (per camera) | Buffered at |
|------|--------|------|-------------------|-------------|
| Main stream (evidence) | Camera | Camera → PoE → Local NVR. Never streamed to the core unless an operator or event requests it | 4–8 Mbps | NVR (30 d) |
| Sub-stream (AI + live) | Camera | Camera → PoE → Edge Gateway → network → Stream Gateway → Stream Manager | 0.5–1 Mbps | Not buffered; regenerated live |
| Edge events + clips | Edge Arjuna Node | Edge → network → Event Engine (bypasses central inference) | Bursty, ~2–10 MB per event | Edge (72 h) |
| Vehicle clips + GPS | Mobile NVR | Vehicle → Wi-Fi / LTE → Dashcam Adapter → Stream Gateway → Storage | 1 Hz GPS, clips on event | Vehicle (72 h) |
| Health beacons | Edge Gateway, cameras | MQTT every 10 s → Camera Health | ~1 KB / 10 s | Not buffered |
| Alerts | Event Engine | Message Bus → Command Centre (MQTT push) | On event | Kafka (7 d) |

### 2.3 Latency budget (glass to glass)

| Segment | Target | Notes |
|---------|--------|-------|
| Camera encode + packetise | 150–250 ms | Depends on GOP; sub-stream uses 1 s GOP |
| Zone switch → core (fibre) | < 2 ms | Wireless adds 5–20 ms; LTE adds 30–80 ms |
| Stream Gateway + decode | 100–200 ms | Hardware decode |
| Live wall (WebRTC) | 300–500 ms | Includes jitter buffer |
| **Live view total** | **< 2 s** | Design target; typical 0.8–1.2 s on fibre |
| Inference + tracking + rules | 200–400 ms | Sampled at 5–8 fps |
| Event Engine + clip cut | 1–2 s | Clip pull runs in parallel with the alert |
| Message Bus → operator screen | < 200 ms | MQTT QoS 1 |
| **Alert total** | **< 5 s** | From the frame where the violation appears to the operator's screen |
| Clip attached to alert | < 30 s | Pulled from NVR / edge, hashed, stored |

---

## 3. What fails, what the system does, how data comes back

Each scenario lists: how it is detected, what happens automatically (no operator action), what data is or isn't lost, how the gap is recovered, what the operator sees, and the recovery objectives. **RTO** is how long until the function is restored; **RPO** is how much data can be lost.

### 3.1 Camera and edge

#### F1 · A camera goes offline (power, cable, hardware)

- **Detected:** Edge Gateway misses the RTSP keep-alive for 3 consecutive attempts (≈ 15 s). Health beacon marks the camera amber at 30 s, red at 90 s.
- **Automatic:** Adapter enters reconnect loop (5 s, backing off to 60 s). AI Scheduler releases the GPU slot. Rules bound to that camera pause instead of raising false "zone clear" events.
- **Data:** Live and AI for that camera stop. Nothing already recorded is lost.
- **Recovery:** On reconnect the adapter resumes the stream automatically and the NVR resumes recording. No backfill is possible for the gap — the camera captured nothing.
- **Operator sees:** Camera turns red on the Live Mine Map and in Camera Health; an auto-ticket is raised with cabinet, switch port and last-seen time.
- **RTO:** automatic on power/cable restore. **RPO:** the outage window for that camera only.

#### F2 · Industrial PoE switch or zone cabinet fails

- **Detected:** All cameras on that switch go red together; switch stops answering SNMP; ERPS ring reports a topology change.
- **Automatic:** ERPS ring re-converges in < 50 ms if only the uplink failed. If the switch itself is dead, its cameras are unpowered — nothing can substitute.
- **Data:** Recording stops for that zone's cameras.
- **Recovery:** Dual-PSU switches survive a single supply failure. Spare switch swap is a 30-minute field task; configuration is pushed from the core via template.
- **Operator sees:** A grouped "zone cabinet down" alert instead of N separate camera alerts, with the cabinet ID.
- **RTO:** ≤ 30 min (hardware swap). **RPO:** outage window for that zone.

#### F3 · Local NVR disk or unit fails

- **Detected:** RAID controller SNMP trap (disk) or NVR health beacon missing (unit). Profile G playback test runs hourly from the core.
- **Automatic:** RAID 5 survives one disk. On unit failure, the Edge Gateway continues proxying sub-streams so live view and AI keep working; the Edge Arjuna Node continues cutting event clips into its own 72 h buffer, so alert evidence is still captured.
- **Data:** Continuous main-stream recording is lost for the outage window. Event clips are not.
- **Recovery:** Replace disk (hot-swap) or unit. For a unit swap, camera assignments are re-pushed from the core.
- **Operator sees:** "Recording degraded" banner on the affected cameras; alerts continue.
- **RTO:** disk hot-swap immediate; unit ≤ 4 h. **RPO:** continuous recording gap only; events preserved.

#### F4 · Edge Gateway crashes or is rebooted

- **Detected:** Health beacon missing for 30 s.
- **Automatic:** Watchdog restarts the gateway service in < 20 s. Cameras keep recording to the NVR regardless — the gateway is not in the recording path. Stream Gateway adapters retry every 5 s and re-attach when it returns.
- **Data:** Live view and central AI pause for that site for the restart window. Nothing is lost on the NVR.
- **Recovery:** Automatic.
- **Operator sees:** Site flashes amber, returns to green. Ticket only if it recurs 3× in an hour.
- **RTO:** < 1 min. **RPO:** none for recordings; live/AI gap < 1 min.

#### F5 · Edge Arjuna Node (GPU) fails

- **Detected:** Beacon missing; inference heartbeat stops.
- **Automatic:** AI Scheduler at the core takes over the site's sub-streams (this is why the core is sized N+1). Alerts continue from central inference with the same rules.
- **Data:** Events already in the edge buffer that were not yet forwarded remain on its disk and are replayed when it returns (buffer is persistent, not RAM).
- **Recovery:** Repair or swap node; buffer replays automatically on boot.
- **Operator sees:** Alert latency for that site may rise from < 5 s to ~6–8 s while running centrally over wireless; a note in Camera Health says "edge AI degraded, running central".
- **RTO:** failover automatic; hardware ≤ 1 business day. **RPO:** none.

### 3.2 Connectivity

#### F6 · Fibre cut between a zone and the core

- **Detected:** ERPS topology change; link-down SNMP trap; latency probe.
- **Automatic:** Ring topology re-routes traffic the other way round in < 50 ms — no visible effect. If both ring directions are cut (an isolated zone), the Edge Gateway raises "site isolated" and switches to store-and-forward mode: the NVR keeps recording, the Edge Arjuna Node keeps running AI locally and buffering events, and any PtP wireless backup on that zone carries sub-streams for the highest-priority cameras only (QoS list configured per zone).
- **Data:** Nothing is lost at the edge. Central live view drops to the priority cameras or none, depending on the backup link.
- **Recovery:** When fibre returns, (1) the edge event buffer replays to the Event Engine in order, oldest first, rate-limited so live traffic keeps priority; (2) the Event Engine pulls the clips it was missing from the NVR; (3) the Kafka consumer offset guarantees no duplicate alerts are shown.
- **Operator sees:** "Site isolated — recording locally" banner with time since isolation and buffer fill %. When restored, a summary "N events replayed from zone X" appears.
- **RTO:** ring < 50 ms; isolated zone depends on repair. **RPO:** none for recordings and events; live view only.

#### F7 · PtP / PtMP wireless link degrades or drops (remote pit)

- **Detected:** Link RSSI / modulation drop reported by the radio; packet-loss probe > 2 %; RTSP stalls.
- **Automatic:** QoS drops lower-priority sub-streams first (configured order: intrusion-critical cameras last). Remote cameras record to on-camera SD (7 days) and to the remote node's NVR. Edge AI keeps running locally.
- **Data:** Live view degrades gracefully (fewer cameras, then none). Events and recordings safe at the edge.
- **Recovery:** ONVIF Profile G is used to fetch the SD / NVR recordings the core did not see, only for time ranges flagged by events or by an operator request — the full main stream is not backhauled over wireless.
- **Operator sees:** Cameras on that link marked "wireless degraded" (amber), with link quality shown.
- **RTO:** automatic. **RPO:** none.

#### F8 · Vehicle leaves Wi-Fi / LTE coverage

- **Detected:** Mobile NVR's last upload time and GPS trail; Dashcam Adapter marks vehicle "out of coverage" after 60 s without a beacon.
- **Automatic:** Mobile NVR records all channels locally (72 h SSD) with GPS and G-sensor data. Event clips (harsh braking, collision, PPE if edge-capable) are queued. When the vehicle re-enters coverage, upload resumes from the last acknowledged chunk (resumable HTTPS upload, de-duplicated by clip hash). Wi-Fi is preferred for bulk upload; LTE carries live sub-stream and telemetry only.
- **Data:** Nothing lost within 72 h. If a vehicle stays out of coverage longer, the oldest non-event footage is overwritten first; event clips are protected until uploaded.
- **Recovery:** Automatic on coverage. Fleet report shows each vehicle's last sync.
- **Operator sees:** Vehicle icon greyed on the Live Mine Map with last GPS position and "last seen" time; turns live on return.
- **RTO:** automatic. **RPO:** none within 72 h.

#### F9 · Private LTE core or Wi-Fi controller fails

- **Detected:** Every vehicle goes "out of coverage" simultaneously; controller health check fails.
- **Automatic:** Vehicles behave exactly as F8. Mobile NVRs with a secured 4G/5G SIM fall back to the VPN path for events and telemetry only (metered; no live video).
- **Recovery:** Controller restart / HA pair; bulk upload catches up over Wi-Fi.
- **RTO:** HA failover < 1 min; single controller ≤ 4 h. **RPO:** none within 72 h.

#### F10 · Public 4G / 5G VPN fallback is unavailable

- **Detected:** IPsec / WireGuard tunnel down.
- **Automatic:** This path only ever carries events and clips as a last resort; its loss means those wait in the edge / vehicle buffer.
- **Operator sees:** Nothing unless a site is *also* isolated from fibre and wireless, in which case the "site isolated" banner already applies.
- **RTO / RPO:** as F6 / F8.

### 3.3 Core network and security

#### F11 · Core or distribution switch fails

- **Detected:** VRRP / MLAG failover event; SNMP.
- **Automatic:** Dual-core with VRRP / MLAG: the surviving core carries all traffic. Uplinks are 40G so a single core has the headroom.
- **Data:** Sub-second blip in live view; no loss of recordings or events.
- **RTO:** < 3 s. **RPO:** none.

#### F12 · OT firewall fails or a rule change blocks surveillance traffic

- **Detected:** Stream Gateway sees all adapters disconnect at once while edge health beacons (which take the same path) also stop — this pattern is classified as "core path blocked", not "N cameras down".
- **Automatic:** Firewall HA pair fails over. Edge sites behave as isolated (F6): they record and buffer.
- **Recovery:** Rule changes are versioned; a bad change is rolled back from the previous version. Edge buffers replay.
- **Operator sees:** One "core path blocked" incident, not hundreds of camera alerts.
- **RTO:** HA < 10 s; rollback ≤ 15 min. **RPO:** none.

### 3.4 Arjuna core

#### F13 · Unified Stream Gateway or an adapter crashes

- **Detected:** Process supervisor; adapter heartbeat.
- **Automatic:** Adapters run as independent processes and restart in < 10 s. Backpressure means a slow adapter cannot stall the others. Reconnect to every source is automatic.
- **Data:** Live/AI gap < 10 s. Recordings unaffected (they are at the edge).
- **RTO:** < 10 s. **RPO:** none.

#### F14 · GPU worker fails (central inference)

- **Detected:** Worker heartbeat missing for 15 s.
- **Automatic:** AI Scheduler re-assigns that worker's streams to the spare (N+1). Zones are prioritised: safety-critical zones are re-assigned first. If more than one worker is lost, lower-priority cameras are analysed at a lower frame rate rather than dropped.
- **Data:** Up to 15 s of inference missed on the affected streams; recording unaffected.
- **Operator sees:** "AI capacity reduced" banner with the affected zones if degraded mode is entered.
- **RTO:** < 30 s. **RPO:** ≤ 15 s of detections.

#### F15 · Message Bus (MQTT / Kafka) is unavailable

- **Detected:** Producer errors at the Event Engine; consumer lag alarm.
- **Automatic:** Event Engine persists events to its local database first and publishes second, so nothing is lost; it retries publishing with the stored events. Kafka retains 7 days, so a UI that was disconnected reads back from its last offset. MQTT QoS 1 guarantees at-least-once delivery to the Command Centre; the UI de-duplicates by event ID.
- **Operator sees:** Alerts pause, then arrive in order once the bus returns. A "delayed alerts" badge shows the oldest pending event time.
- **RTO:** bus HA < 1 min. **RPO:** none.

#### F16 · Central storage full or unreachable

- **Detected:** Capacity alarm at 80 % and 90 %; write errors.
- **Automatic:** Tiering moves hot → warm → cold on schedule; at 90 % the oldest non-evidence clips in the warm tier are expired early. Evidence (object-locked) is never expired automatically. If storage is unreachable, the Event Engine keeps event metadata in its database and marks clips "pending"; clip pulls are retried, and the NVR / edge still hold the source clips (30 d / 72 h), so the retry window is generous.
- **Operator sees:** Alerts still arrive with metadata; clip thumbnails show "pending" until storage recovers.
- **RTO:** depends on cause. **RPO:** none as long as storage returns within the edge buffer windows.

#### F17 · Whole central site down (power, DC failure, planned maintenance)

- **Detected:** Everything at once; the site UPS reports on battery.
- **Automatic:** This is the case the whole design is built for. Every edge site and vehicle carries on recording and running edge AI. Nothing at the edge depends on the core. UPS covers the core for ≥ 30 min for a clean shutdown; a standby VMS / Arjuna instance (warm standby, replicated metadata) can be brought up in the DR location.
- **Data:** No recordings or edge events are lost within the buffer windows (NVR 30 d, edge 72 h, vehicle 72 h). Central live view and central alerts are unavailable until the core or standby is up.
- **Recovery:** On restore, edge buffers replay in parallel, rate-limited per site; the Event Engine reconciles by event ID and time so nothing is duplicated; Camera Health rebuilds its state from the first beacons.
- **Operator sees:** Command Centre unavailable → standby URL. After recovery, a "replay in progress" progress bar per site.
- **RTO:** warm standby ≤ 1 h; primary as per power restore. **RPO:** none for edge data.

#### F18 · Identity provider (AD / LDAP) unreachable

- **Detected:** SSO login failures.
- **Automatic:** Existing sessions continue to their timeout. A break-glass local operator account (MFA, audited) allows the control room to keep working.
- **RTO:** automatic. **RPO:** none.

### 3.5 Summary table

| # | Failure | Live view | Alerts | Recording | Recovery | RPO |
|---|---------|-----------|--------|-----------|----------|-----|
| F1 | Camera offline | Lost (that camera) | Paused (that camera) | Lost (that camera) | Auto on return | Outage window |
| F2 | PoE switch / cabinet | Lost (zone) | Paused (zone) | Lost (zone) | Ring < 50 ms or swap ≤ 30 min | Outage window |
| F3 | Local NVR | OK | OK (edge clips) | Continuous lost, events kept | Disk hot-swap / unit ≤ 4 h | Continuous only |
| F4 | Edge Gateway | Gap < 1 min | Gap < 1 min | OK | Auto | None |
| F5 | Edge AI node | OK | Central takes over | OK | Auto | None |
| F6 | Fibre cut | Ring: none. Isolated: priority cams only | Edge-local, replayed | OK | Auto replay | None |
| F7 | Wireless degraded | Graceful reduction | Edge-local | OK (SD + NVR) | Auto | None |
| F8 | Vehicle out of coverage | Lost until return | Queued | OK (72 h) | Resumable upload | None ≤ 72 h |
| F9 | LTE / Wi-Fi controller | Lost (all vehicles) | Via VPN | OK (72 h) | HA < 1 min | None ≤ 72 h |
| F11 | Core switch | Blip < 3 s | OK | OK | Auto | None |
| F12 | Firewall | Sites isolated | Edge-local, replayed | OK | HA < 10 s | None |
| F13 | Stream Gateway | Gap < 10 s | Gap < 10 s | OK | Auto | None |
| F14 | GPU worker | OK | Re-assigned < 30 s | OK | Auto | ≤ 15 s |
| F15 | Message Bus | OK | Delayed, in order | OK | HA < 1 min | None |
| F16 | Storage | OK | Metadata now, clips pending | OK | Retry from edge | None |
| F17 | Central site down | Lost until standby | Edge-local, replayed | OK everywhere | Standby ≤ 1 h | None |

---

## 4. Store-and-forward: how replay works

1. **Everything that leaves the edge has an ID and a timestamp.** Events carry `site / camera / event-id / t0 / t1`; clips carry a SHA-256 hash.
2. **Buffers are on disk, not in memory.** Edge Arjuna Node: 72 h of event clips + metadata. Mobile NVR: 72 h of all channels; event clips protected. Camera SD: 7 days. NVR: 30 days.
3. **Replay order is oldest first, per site, rate-limited.** Replay uses at most 30 % of the link so live traffic keeps its budget; a site with a 72 h backlog over a 200 Mbps wireless link drains in well under an hour because only events and clips are replayed, never continuous footage.
4. **Idempotent ingest.** The Event Engine upserts by event ID; the storage layer de-duplicates clips by hash. Replaying twice is harmless.
5. **Continuous footage is fetched on demand only.** If an investigation needs main-stream video from an isolation window, the operator selects the range and the core pulls it via ONVIF Profile G from the NVR — the same mechanism used every day for evidence export.
6. **Operators see the backlog.** Camera Health shows per-site buffer fill %, oldest pending event, and replay progress. A buffer above 60 % raises a ticket so the link problem is fixed before the buffer overflows.

---

## 5. Detection: the monitoring that makes the above real

| Signal | Source | Interval | Amber | Red | Action |
|--------|--------|----------|-------|-----|--------|
| Camera RTSP keep-alive | Adapter / Edge Gateway | 5 s | 30 s missed | 90 s missed | Reconnect loop; ticket at red |
| Frame rate per stream | Stream Manager | 10 s | < 70 % of expected | < 30 % | Mark degraded; QoS check |
| Packet loss / latency probe | Edge Gateway → core | 10 s | > 1 % / > 50 ms | > 3 % / > 150 ms | Wireless degraded state |
| Edge health beacon | Edge Gateway, Edge AI, Mobile NVR | 10 s | 30 s missed | 60 s missed | Site isolated / vehicle out of coverage |
| NVR recording check | Core → NVR (Profile G) | 1 h | Gap in last hour | Gap > 15 min | Ticket |
| RAID / disk SMART | NVR, storage | SNMP trap | Predictive failure | Disk failed | Ticket |
| GPU worker heartbeat | AI Scheduler | 5 s | 15 s missed | — | Re-assign streams |
| Kafka consumer lag | Message Bus | 30 s | > 30 s | > 5 min | Bus HA check |
| Storage capacity | Storage | 5 min | 80 % | 90 % | Early expiry of non-evidence |
| Buffer fill % | Edge AI, Mobile NVR | 1 min | 60 % | 85 % | Ticket to fix the link |
| UPS state | Core, cabinets | Event | On battery | < 10 min left | Clean shutdown sequence |

Every red state creates a ticket automatically (Reports + APIs → ITSM webhook) with the component, location, first-seen time and the runbook link.

---

## 6. Assumptions to confirm during site survey

- Number of cameras per zone and per vehicle; this drives NVR and edge sizing.
- Existing NVR models and whether they expose ONVIF Profile G playback (otherwise vendor SDK, marked dashed on the diagram).
- Fibre ring topology and whether every zone cabinet is dual-homed.
- Available spectrum and line-of-sight for PtP links to remote pits.
- Private LTE availability vs Wi-Fi mesh coverage along haul roads.
- Retention policy per zone (30 / 90 / 365 days) and which zones require object-locked evidence.
- Standby / DR location for the central site.
- Whether any external network (SAIL WAN) is permitted for the VPN fallback path, or whether it must be a dedicated mobile APN.