Node::Isolated Realtime RFID Telemetry Mesh

01. Executive Summary

A containerized telemetry pipeline designed to process high-frequency distributed radio-frequency identification (RFID) data streams without public internet ingress dependency. Evolving over 1.5 years of active production engineering, the middleware platform supports multi-warehouse deployments featuring more than six concurrent reader gates operating simultaneously.

The system decouples incoming edge sensor streams into an isolated local messaging ring across 4 specialized .NET 8 microservices. It features parallel gate-reading pipelines, zero-allocation serialization using Span<T>, and AI-camera verification filters to match spatial entries. Real-time dashboards are updated at 60Hz via a local SignalR mesh.

02. Problem Statement

Legacy RFID middleware built on standard MVC patterns suffers from severe throughput choke points. When six or more industrial gates scan items simultaneously, they generate bursts of 12,000+ scans per second.

1. Gen 2 GC Pauses: Deserializing XML/JSON text strings on the HTTP heap creates millions of short-lived objects. This triggers frequent Gen 2 Garbage Collection sweeps, freezing the pipeline for 50ms - 200ms and causing socket overflows.
2. Blocking Database Calls: Synchronous Entity Framework queries on the ingestion path create database thread starvation under peak loads.
3. Sequential Bottlenecks: Single-threaded processing loops fail to handle gate reads concurrently, leading to sequence drifts.

The system required a redesign capable of processing continuous high-frequency streams with a loopback latency below 15ms, zero heap-allocation spikes, and strict offline durability.

03. System Architecture

The middleware isolates edge sensor ingestion from relational persistence. Edge streams are buffered immediately into an in-memory queue, analyzed via a Python-based telemetry analyzer embedded inside the service stack, and dispatched to operator UI screens.

[ RFID Gate Reader Array ]      [ AI Camera Verification Node ]
    (6+ Active Gates)                       │
            │ (Raw TCP Packets)             │ (REST Validation)
            ▼                               ▼
     [ DeviceService ] <───────────> [ EF Core / DbContext ]
            │
            │ (WolverineFx Outbox)
            ▼
   [ RabbitMQ Broker ]
    /               \
   ▼                 ▼
[ TelemetryIntelligenceService ] <──► [ Redis Cloud L1/L2 Cache ]
   │ (Anomaly Check)
   ▼
[ NotificationService ]
   │ (SignalR Broadcast)
   ▼
[ TelemetryDashboard.Blazor ] (60Hz Viewport)

Microservice Topology

Cross-service messaging is governed by strict event contracts (DeviceEventCreated, MetricsUpdated, TelemetryAnomalyDetected, and TelemetryAnalysisCompleted). A shared Shared.Infrastructure library provides base implementations of the ICacheService/CacheService consumed by all microservices.

04. Engineering Decisions

WolverineFx + Transactional Outbox for Spikes

To prevent data loss during network disconnects or database load spikes, the system rejects global distributed transactions. Instead, the DeviceService writes events directly to a local PostgreSQL ledger and routes them via the WolverineFx Outbox pattern. Wolverine automatically retries RabbitMQ dispatches in the background, maintaining database-event consistency.

Blazor WebAssembly Direct Streaming

To reduce server memory consumption, the Blazor dashboard is compiled to WASM and runs client-side. The WASM container establishes a direct WebSocket connection via SignalR to the NotificationService. Ticks are rendered directly on the client, minimizing backend rendering load.

05. Scaling Strategy

Parallel Processing for Gate Reads

Rather than processing all gate reads sequentially, the DeviceService implements a multi-channel reading pipeline. Incoming socket streams are split across isolated System.Threading.Channels:

• Each gate has its own buffer queue.
• Under peak traffic, locks are isolated to individual gate channels.
• Parallel tasks process each channel independently, ensuring single-gate traffic surges never starve adjacent gates.

Dual-Tier Redis Caching Topology (L1/L2)

A dual-tier cache structure is implemented:

1. L1 Cache (In-Memory Dictionary): Stores active reader statuses directly in microservice memory for microsecond reads.
2. L2 Cache (Redis Cloud): Shared distributed cache containing warehouse configurations and active session states. If the L2 cache has latency spikes, the L1 local cache serves queries, protecting the fast ingestion path.

06. Performance Optimizations

Zero-Allocation Parsing via Span<T>

To eliminate Gen 2 GC pauses, incoming raw hex packets from the RFID readers are parsed using .NET 8 ReadOnlySpan<char> and ReadOnlySpan<byte>:

• Packets are sliced directly in the socket buffer without allocating heap strings.
• Numeric identifiers are read using Utf8Parser.TryParse.
• Memory utilization fell by 84%, and GC execution was eliminated from the execution critical path.

EF Core Query Hardening

Entity Framework query patterns were optimized:

• All telemetry logging queries use .AsNoTracking() to avoid tracking state overhead in EF Core.
• Multi-row warehouse updates are batched using .ExecuteUpdateAsync() and .ExecuteDeleteAsync(), which compile directly to native SQL queries and bypass object mapping entirely.

07. Challenges Solved

Hangfire Background Retry Workflows

If a warehouse gate reader loses local power or connection, offline states must be synchronized without halting execution. The solution was implementing Hangfire background jobs:

1. When a gate heartbeat fails, Hangfire registers a scheduled retry job.
2. Retries are executed with exponential backoff.
3. If the gate remains offline, a ticket is logged, and adjacent gates automatically route inventory checks through their channels.

AI Camera Verification

RFID signals suffer from “ghost reads” — detecting tags passing near a gate without actually entering. We integrated AI cameras at each gate:

• When a tag is read, the system cross-references the camera timestamp.
• The camera detects motion direction at the gate.
• If no matching motion event is recorded within a ±500ms window, the RFID read is flagged as a potential anomaly and filtered out.

08. Lessons Learned

1. Avoid ORM Tracking in Pipelines: Entity Framework Core is highly efficient for CRUD applications but creates performance bottlenecks in high-frequency streams. Decoupling the ingestion path (direct ADO.NET / Npgsql commands) from the configuration panels (EF Core) was critical to achieving 12,402 frames/sec.
2. Connection Recovery: In highly containerized environments, databases occasionally restart. Automatic EF Core execution strategies (EnableRetryOnFailure) are required to prevent connection drops from crashing the ingestion background threads.

09. Infrastructure Diagram

Below is the event pipeline and service topology:

flowchart TD
    subgraph Edge ["Warehouse Edge (Hardware Layer)"]
        R1["RFID Gate Readers (6+)"] -->|Raw TCP Hex| Ingest["DeviceService Ingestion Socket"]
        Cam["AI Cameras"] -->|REST Validation| Ingest
    end

    subgraph ServiceMesh ["Distributed Middleware Ring (Docker)"]
        Ingest -->|Write Ledger| PG[(PostgreSQL)]
        Ingest -->|Outbox Pattern| RMQ[RabbitMQ Event Bus]
        
        RMQ -->|Wolverine Mediated| Intel["TelemetryIntelligenceService"]
        Intel <-->|L1/L2 Check| Red[(Redis Cloud)]
        
        Intel -->|TelemetryAnomalyDetected| RMQ
        RMQ -->|Push Alerts| Notify["NotificationService"]
        Notify -->|SignalR Loopback Hub| Blazor["TelemetryDashboard.Blazor"]
    end

    Blazor -->|60Hz Stream| Monitor["SOC Dashboard Screen"]

10. Technology Stack

• Runtime: .NET 8 (C#)
• Ingestion Middleware: ASP.NET Core Device Controllers
• Database Driver: Npgsql
• Object-Relational Mapper: Entity Framework Core 8
• Distributed Mediator: WolverineFx 3.5.0
• Message Broker: RabbitMQ
• Shared Storage: PostgreSQL 16
• Volatile Cache: Redis Cloud (L2 Cache)
• Background Scheduler: Hangfire Core 1.8
• Diagnostic Specifications: Swashbuckle OpenAPI 6.9.0
• Operator Console: Blazor WebAssembly + ASP.NET Core SignalR

11. Future Improvements

• Edge Gateway Buffering: Developing a lightweight SQLite-based edge daemon to store records locally if the main docker bridge network goes offline.
• MQTT Migration: Transitioning the raw TCP socket listener to MQTT to support standardized industrial IoT message formats out of the box.

12. Architecture Decision Records (ADR)

[ADR-01] SignalR for Telemetry Broadcasts

• Context: Delivering real-time warehouse gate reads to multiple client dashboards at 60Hz without performance degradation.
• Decision: Selected ASP.NET Core SignalR over standard HTTP polling or raw WebSockets.
• Rationale: Standard HTTP polling introduces heavy socket opening overhead and floods database connection pools under continuous tag surges. While raw WebSockets work, SignalR provides built-in fallback protocols (like Server-Sent Events) and automatic connection recovery, allowing client UI modules to re-establish feeds when walking through warehouses.

[ADR-02] Dual-Tier Redis L1/L2 Cache Topology

• Context: Managing active RFID tag sessions and configuration settings without hitting database tables.
• Decision: Implemented an in-memory L1 cache dictionary backed by a Redis Cloud L2 distributed cache.
• Rationale: Querying a remote Redis instance for every single tag read (12,000+ per second) would introduce latency and network bottleneck risks. Local L1 caching services read queries in microseconds, while Redis Cloud synchronizes state updates across separated worker nodes.

[ADR-03] Hangfire for Asynchronous Event Retries

• Context: Handling connection drops to physical reader gates without stalling the ingestion thread.
• Decision: Selected Hangfire over custom in-memory threads or cron daemons.
• Rationale: If a gate fails to ping back, retrying inline would block the main processing thread. Spawning custom threads in-memory loses reliability during container crashes. Hangfire persists job states in PostgreSQL, enabling reliable retries with exponential backoffs.

[ADR-04] Span<T>-Based Byte Ingestion

• Context: Parsing continuous high-frequency hex payloads from TCP socket listeners.
• Decision: Implemented allocation-free parsing using ReadOnlySpan<char> and ReadOnlySpan<byte>.
• Rationale: Converting raw byte strings to standard JSON/XML objects allocations short-lived heap references, which triggers frequent Gen 2 Garbage Collection pauses that stall the thread loop. Span<T> slices buffers directly in-place, eliminating GC execution overhead entirely.