Incident Ledger & RCA
Post-mortem logs documenting actual production bottlenecks, memory profiling diagnostic steps, and architectural refactoring mitigations.
// SignalR WebSocket Ingestion Bottleneck
Continuous derivative market ticks (exceeding 20,000 events/sec) flooded outbound SignalR WebSocket channels, triggering high thread-pool context switching and freezing mobile UI render threads.
Utilized dotnet-counters to observe ThreadPool starvation, revealing that the connection dispatch queues were backing up faster than the TCP sockets could drain.
Aggregated ticks into 100ms batches in-memory, compressed messages into binary Protocol Buffers (MessagePack) schemas, and restricted maximum connections per client session.
// Gen2 Garbage Collection Spikes
Converting high-frequency raw RFID tag socket bytes into JSON/String representations created millions of short-lived Heap references, forcing continuous Gen 2 GC pauses.
Analyzed memory dumps in JetBrains dotMemory, revealing that System.String objects from RFID parsers accounted for 84% of total heap allocations.
Refactored parsing loops to utilize ReadOnlySpan<char> to slice buffers in-place, eliminating string allocations on tag reads.
// Sequential Ingestion Locking
Processing concurrent tag reads from multiple antennas sequentially created blocking write locks on database transaction tables, causing dropped read frames.
Observed thread synchronization metrics and database log errors showing lock escalations and transaction timeouts on PostgreSQL tags ledger table.
Isolated antenna listeners into parallel thread pools, feeding data to in-memory buffers that bulk-write transactions via WolverineFx mediators.
// Container Memory Exhaustion (OOM)
Running 12 microservices inside Docker on a single 1GB host triggered Linux kernel OOM kills when the .NET runtime pre-allocated massive heap zones.
Checked kernel logs using dmesg -T | grep -i oom, revealing that the host ran out of physical memory and swap allocation partitions.
Overrode runtime settings to use Workstation GC (ServerGarbageCollection = false), capped container limits in Docker Compose, and configured Redis LRU caching policies.
// Emulation Catch-Up Budget Overruns
High peer network packet jitter (150ms+) triggered massive 15-frame rollback catch-up sequences, exceeding the cycle-accurate 16.6ms NTSC rendering window.
FFI console logs reported frame processing metrics, showing rollback catch-up loops consuming up to 24.2ms of processor execution time.
Serialized snapshot files using raw `bincode` encodings, optimized fast-forward state loads, and muted active APU sound generation during rollback catching.