Arjuna for TASRA Mines — Real-Time Data Pipeline and Failure Handling

Scope. How camera video, events and telemetry reach the Arjuna Command Centre in real time, what the system does automatically when any part of the chain fails, and how data is recovered afterwards. Companion to the architecture diagram (Full architecture, Remote & vehicle flow and Resilience & fallback views).

Status. Proposal, September 2026. Figures are design targets sized for a mine-site deployment; they are confirmed during site survey and factory acceptance testing. Everything in this document runs on the TASRA intranet — no camera, NVR or edge node ever needs Internet access.

1. The one-page version

1. Every camera is recorded as close to itself as possible. Fixed cameras record on the zone's Local NVR; remote cameras record on their SD card and edge node; vehicles record on the Mobile NVR. The central site is never a precondition for recording.
2. Two streams per camera. The main stream (full resolution) is the evidence copy and stays on the NVR. The sub-stream (640p, 5–8 fps sampled) is what Arjuna AI analyses and what the live wall shows. Alerts pull the high-resolution clip on demand.
3. Edge AI keeps working without the core. The Edge Arjuna Node runs the same detectors locally, buffers events and clips for 72 hours, and replays them to the central Event Engine when the link returns.
4. Every link has a fallback and every buffer has a size. Fibre → PtP wireless → private LTE → secured 4G/5G VPN. NVR 30 days, edge 72 h, vehicle 72 h, camera SD 7 days.
5. The operator always knows. Camera Health turns amber within 30 s of a problem and red within 90 s, with an automatic ticket. Nothing fails silently.

2. End-to-end real-time path

2.1 Stage by stage

2.2 Data types and where each one flows

2.3 Latency budget (glass to glass)

3. What fails, what the system does, how data comes back

Each scenario lists: how it is detected, what happens automatically (no operator action), what data is or isn't lost, how the gap is recovered, what the operator sees, and the recovery objectives. RTO is how long until the function is restored; RPO is how much data can be lost.

3.1 Camera and edge

F1 · A camera goes offline (power, cable, hardware)

• Detected: Edge Gateway misses the RTSP keep-alive for 3 consecutive attempts (≈ 15 s). Health beacon marks the camera amber at 30 s, red at 90 s.
• Automatic: Adapter enters reconnect loop (5 s, backing off to 60 s). AI Scheduler releases the GPU slot. Rules bound to that camera pause instead of raising false "zone clear" events.
• Data: Live and AI for that camera stop. Nothing already recorded is lost.
• Recovery: On reconnect the adapter resumes the stream automatically and the NVR resumes recording. No backfill is possible for the gap — the camera captured nothing.
• Operator sees: Camera turns red on the Live Mine Map and in Camera Health; an auto-ticket is raised with cabinet, switch port and last-seen time.
• RTO: automatic on power/cable restore. RPO: the outage window for that camera only.

F2 · Industrial PoE switch or zone cabinet fails

• Detected: All cameras on that switch go red together; switch stops answering SNMP; ERPS ring reports a topology change.
• Automatic: ERPS ring re-converges in < 50 ms if only the uplink failed. If the switch itself is dead, its cameras are unpowered — nothing can substitute.
• Data: Recording stops for that zone's cameras.
• Recovery: Dual-PSU switches survive a single supply failure. Spare switch swap is a 30-minute field task; configuration is pushed from the core via template.
• Operator sees: A grouped "zone cabinet down" alert instead of N separate camera alerts, with the cabinet ID.
• RTO: ≤ 30 min (hardware swap). RPO: outage window for that zone.

F3 · Local NVR disk or unit fails

• Detected: RAID controller SNMP trap (disk) or NVR health beacon missing (unit). Profile G playback test runs hourly from the core.
• Automatic: RAID 5 survives one disk. On unit failure, the Edge Gateway continues proxying sub-streams so live view and AI keep working; the Edge Arjuna Node continues cutting event clips into its own 72 h buffer, so alert evidence is still captured.
• Data: Continuous main-stream recording is lost for the outage window. Event clips are not.
• Recovery: Replace disk (hot-swap) or unit. For a unit swap, camera assignments are re-pushed from the core.
• Operator sees: "Recording degraded" banner on the affected cameras; alerts continue.
• RTO: disk hot-swap immediate; unit ≤ 4 h. RPO: continuous recording gap only; events preserved.

F4 · Edge Gateway crashes or is rebooted

• Detected: Health beacon missing for 30 s.
• Automatic: Watchdog restarts the gateway service in < 20 s. Cameras keep recording to the NVR regardless — the gateway is not in the recording path. Stream Gateway adapters retry every 5 s and re-attach when it returns.
• Data: Live view and central AI pause for that site for the restart window. Nothing is lost on the NVR.
• Recovery: Automatic.
• Operator sees: Site flashes amber, returns to green. Ticket only if it recurs 3× in an hour.
• RTO: < 1 min. RPO: none for recordings; live/AI gap < 1 min.

F5 · Edge Arjuna Node (GPU) fails

• Detected: Beacon missing; inference heartbeat stops.
• Automatic: AI Scheduler at the core takes over the site's sub-streams (this is why the core is sized N+1). Alerts continue from central inference with the same rules.
• Data: Events already in the edge buffer that were not yet forwarded remain on its disk and are replayed when it returns (buffer is persistent, not RAM).
• Recovery: Repair or swap node; buffer replays automatically on boot.
• Operator sees: Alert latency for that site may rise from < 5 s to ~6–8 s while running centrally over wireless; a note in Camera Health says "edge AI degraded, running central".
• RTO: failover automatic; hardware ≤ 1 business day. RPO: none.

3.2 Connectivity

F6 · Fibre cut between a zone and the core

• Detected: ERPS topology change; link-down SNMP trap; latency probe.
• Automatic: Ring topology re-routes traffic the other way round in < 50 ms — no visible effect. If both ring directions are cut (an isolated zone), the Edge Gateway raises "site isolated" and switches to store-and-forward mode: the NVR keeps recording, the Edge Arjuna Node keeps running AI locally and buffering events, and any PtP wireless backup on that zone carries sub-streams for the highest-priority cameras only (QoS list configured per zone).
• Data: Nothing is lost at the edge. Central live view drops to the priority cameras or none, depending on the backup link.
• Recovery: When fibre returns, (1) the edge event buffer replays to the Event Engine in order, oldest first, rate-limited so live traffic keeps priority; (2) the Event Engine pulls the clips it was missing from the NVR; (3) the Kafka consumer offset guarantees no duplicate alerts are shown.
• Operator sees: "Site isolated — recording locally" banner with time since isolation and buffer fill %. When restored, a summary "N events replayed from zone X" appears.
• RTO: ring < 50 ms; isolated zone depends on repair. RPO: none for recordings and events; live view only.

F7 · PtP / PtMP wireless link degrades or drops (remote pit)

• Detected: Link RSSI / modulation drop reported by the radio; packet-loss probe > 2 %; RTSP stalls.
• Automatic: QoS drops lower-priority sub-streams first (configured order: intrusion-critical cameras last). Remote cameras record to on-camera SD (7 days) and to the remote node's NVR. Edge AI keeps running locally.
• Data: Live view degrades gracefully (fewer cameras, then none). Events and recordings safe at the edge.
• Recovery: ONVIF Profile G is used to fetch the SD / NVR recordings the core did not see, only for time ranges flagged by events or by an operator request — the full main stream is not backhauled over wireless.
• Operator sees: Cameras on that link marked "wireless degraded" (amber), with link quality shown.
• RTO: automatic. RPO: none.

F8 · Vehicle leaves Wi-Fi / LTE coverage

• Detected: Mobile NVR's last upload time and GPS trail; Dashcam Adapter marks vehicle "out of coverage" after 60 s without a beacon.
• Automatic: Mobile NVR records all channels locally (72 h SSD) with GPS and G-sensor data. Event clips (harsh braking, collision, PPE if edge-capable) are queued. When the vehicle re-enters coverage, upload resumes from the last acknowledged chunk (resumable HTTPS upload, de-duplicated by clip hash). Wi-Fi is preferred for bulk upload; LTE carries live sub-stream and telemetry only.
• Data: Nothing lost within 72 h. If a vehicle stays out of coverage longer, the oldest non-event footage is overwritten first; event clips are protected until uploaded.
• Recovery: Automatic on coverage. Fleet report shows each vehicle's last sync.
• Operator sees: Vehicle icon greyed on the Live Mine Map with last GPS position and "last seen" time; turns live on return.
• RTO: automatic. RPO: none within 72 h.

F9 · Private LTE core or Wi-Fi controller fails

• Detected: Every vehicle goes "out of coverage" simultaneously; controller health check fails.
• Automatic: Vehicles behave exactly as F8. Mobile NVRs with a secured 4G/5G SIM fall back to the VPN path for events and telemetry only (metered; no live video).
• Recovery: Controller restart / HA pair; bulk upload catches up over Wi-Fi.
• RTO: HA failover < 1 min; single controller ≤ 4 h. RPO: none within 72 h.

F10 · Public 4G / 5G VPN fallback is unavailable

• Detected: IPsec / WireGuard tunnel down.
• Automatic: This path only ever carries events and clips as a last resort; its loss means those wait in the edge / vehicle buffer.
• Operator sees: Nothing unless a site is *also* isolated from fibre and wireless, in which case the "site isolated" banner already applies.
• RTO / RPO: as F6 / F8.

3.3 Core network and security

F11 · Core or distribution switch fails

• Detected: VRRP / MLAG failover event; SNMP.
• Automatic: Dual-core with VRRP / MLAG: the surviving core carries all traffic. Uplinks are 40G so a single core has the headroom.
• Data: Sub-second blip in live view; no loss of recordings or events.
• RTO: < 3 s. RPO: none.

F12 · OT firewall fails or a rule change blocks surveillance traffic

• Detected: Stream Gateway sees all adapters disconnect at once while edge health beacons (which take the same path) also stop — this pattern is classified as "core path blocked", not "N cameras down".
• Automatic: Firewall HA pair fails over. Edge sites behave as isolated (F6): they record and buffer.
• Recovery: Rule changes are versioned; a bad change is rolled back from the previous version. Edge buffers replay.
• Operator sees: One "core path blocked" incident, not hundreds of camera alerts.
• RTO: HA < 10 s; rollback ≤ 15 min. RPO: none.

3.4 Arjuna core

F13 · Unified Stream Gateway or an adapter crashes

• Detected: Process supervisor; adapter heartbeat.
• Automatic: Adapters run as independent processes and restart in < 10 s. Backpressure means a slow adapter cannot stall the others. Reconnect to every source is automatic.
• Data: Live/AI gap < 10 s. Recordings unaffected (they are at the edge).
• RTO: < 10 s. RPO: none.

F14 · GPU worker fails (central inference)

• Detected: Worker heartbeat missing for 15 s.
• Automatic: AI Scheduler re-assigns that worker's streams to the spare (N+1). Zones are prioritised: safety-critical zones are re-assigned first. If more than one worker is lost, lower-priority cameras are analysed at a lower frame rate rather than dropped.
• Data: Up to 15 s of inference missed on the affected streams; recording unaffected.
• Operator sees: "AI capacity reduced" banner with the affected zones if degraded mode is entered.
• RTO: < 30 s. RPO: ≤ 15 s of detections.

F15 · Message Bus (MQTT / Kafka) is unavailable

• Detected: Producer errors at the Event Engine; consumer lag alarm.
• Automatic: Event Engine persists events to its local database first and publishes second, so nothing is lost; it retries publishing with the stored events. Kafka retains 7 days, so a UI that was disconnected reads back from its last offset. MQTT QoS 1 guarantees at-least-once delivery to the Command Centre; the UI de-duplicates by event ID.
• Operator sees: Alerts pause, then arrive in order once the bus returns. A "delayed alerts" badge shows the oldest pending event time.
• RTO: bus HA < 1 min. RPO: none.

F16 · Central storage full or unreachable

• Detected: Capacity alarm at 80 % and 90 %; write errors.
• Automatic: Tiering moves hot → warm → cold on schedule; at 90 % the oldest non-evidence clips in the warm tier are expired early. Evidence (object-locked) is never expired automatically. If storage is unreachable, the Event Engine keeps event metadata in its database and marks clips "pending"; clip pulls are retried, and the NVR / edge still hold the source clips (30 d / 72 h), so the retry window is generous.
• Operator sees: Alerts still arrive with metadata; clip thumbnails show "pending" until storage recovers.
• RTO: depends on cause. RPO: none as long as storage returns within the edge buffer windows.

F17 · Whole central site down (power, DC failure, planned maintenance)

• Detected: Everything at once; the site UPS reports on battery.
• Automatic: This is the case the whole design is built for. Every edge site and vehicle carries on recording and running edge AI. Nothing at the edge depends on the core. UPS covers the core for ≥ 30 min for a clean shutdown; a standby VMS / Arjuna instance (warm standby, replicated metadata) can be brought up in the DR location.
• Data: No recordings or edge events are lost within the buffer windows (NVR 30 d, edge 72 h, vehicle 72 h). Central live view and central alerts are unavailable until the core or standby is up.
• Recovery: On restore, edge buffers replay in parallel, rate-limited per site; the Event Engine reconciles by event ID and time so nothing is duplicated; Camera Health rebuilds its state from the first beacons.
• Operator sees: Command Centre unavailable → standby URL. After recovery, a "replay in progress" progress bar per site.
• RTO: warm standby ≤ 1 h; primary as per power restore. RPO: none for edge data.

F18 · Identity provider (AD / LDAP) unreachable

• Detected: SSO login failures.
• Automatic: Existing sessions continue to their timeout. A break-glass local operator account (MFA, audited) allows the control room to keep working.
• RTO: automatic. RPO: none.

3.5 Summary table

4. Store-and-forward: how replay works

1. Everything that leaves the edge has an ID and a timestamp. Events carry site / camera / event-id / t0 / t1; clips carry a SHA-256 hash.
2. Buffers are on disk, not in memory. Edge Arjuna Node: 72 h of event clips + metadata. Mobile NVR: 72 h of all channels; event clips protected. Camera SD: 7 days. NVR: 30 days.
3. Replay order is oldest first, per site, rate-limited. Replay uses at most 30 % of the link so live traffic keeps its budget; a site with a 72 h backlog over a 200 Mbps wireless link drains in well under an hour because only events and clips are replayed, never continuous footage.
4. Idempotent ingest. The Event Engine upserts by event ID; the storage layer de-duplicates clips by hash. Replaying twice is harmless.
5. Continuous footage is fetched on demand only. If an investigation needs main-stream video from an isolation window, the operator selects the range and the core pulls it via ONVIF Profile G from the NVR — the same mechanism used every day for evidence export.
6. Operators see the backlog. Camera Health shows per-site buffer fill %, oldest pending event, and replay progress. A buffer above 60 % raises a ticket so the link problem is fixed before the buffer overflows.

5. Detection: the monitoring that makes the above real

Every red state creates a ticket automatically (Reports + APIs → ITSM webhook) with the component, location, first-seen time and the runbook link.

6. Assumptions to confirm during site survey

• Number of cameras per zone and per vehicle; this drives NVR and edge sizing.
• Existing NVR models and whether they expose ONVIF Profile G playback (otherwise vendor SDK, marked dashed on the diagram).
• Fibre ring topology and whether every zone cabinet is dual-homed.
• Available spectrum and line-of-sight for PtP links to remote pits.
• Private LTE availability vs Wi-Fi mesh coverage along haul roads.
• Retention policy per zone (30 / 90 / 365 days) and which zones require object-locked evidence.
• Standby / DR location for the central site.
• Whether any external network (SAIL WAN) is permitted for the VPN fallback path, or whether it must be a dedicated mobile APN.