🛠️ System Architecture Blueprint — Full Engineering Deep-Dive
An institutional-grade, real-time Indian index derivatives intelligence platform engineered to ingest market data via WebSocket from the Angel One broker and stream actionable trading signals to a Flutter mobile application in under 100ms. The entire system runs on commodity Intel Celeron-class hardware with a total container memory footprint of approximately 531 MB across 3 infrastructure containers.
Core Design Principle: Fail-Closed Deterministic Fast Path
Execution-critical decisions are deterministic and observable. AI components (Nim worker, Python sentiment) enrich and calibrate signals, but the core fast path — ingestion → validation → signal → execution — functions independently of AI, Python, or external API availability. This guarantees the system never deadlocks on a research task.
🧩 Service Topology (12 .NET 8 Microservices)
| Service | .csproj | Role |
|---|---|---|
| TradingGatewayService | src/TradingGatewayService/ | REST API, SignalR hubs, auth, session management, diagnostics |
| TradingIntelligenceService | src/TradingIntelligenceService/ | Market feed ingestion, signal generation, Angel One broker integration |
| TradingBot.Core | src/TradingBot.Core/ | Domain entities, validation runners, core business logic |
| TradingBot.Execution | src/TradingBot.Execution/ | Order routing, fill confirmation, position management |
| TradingBot.MarketFeed | src/TradingBot.MarketFeed/ | Real-time market data feed processing, sequence validation |
| TradingBot.OptionFlow | src/TradingBot.OptionFlow/ | Option chain construction, Greeks computation |
| TradingBot.Lifecycle | src/TradingBot.Lifecycle/ | Application lifecycle, health monitoring, circuit breakers |
| TradingBot.Validation | src/TradingBot.Validation/ | End-to-end system validation suite, report generation |
| TradingNimWorker | src/TradingNimWorker/ | Async slow-path AI/ML model inference (Nim runtime) |
| CommunicationNode | src/CommunicationNode/ | Internal service-to-service messaging bridge |
| NotificationWorker | src/NotificationWorker/ | Alert dispatch, notification workflows |
| telegram-sentiment-worker | python/telegram_sentiment_worker/ | Python-based Telegram crowd psychology aggregation |
🔄 Data Flow Matrix — 26-Stage Intelligence Pipeline
The core of the platform is a multi-stage signal processing pipeline:
| Stage | Component | Description |
|---|---|---|
| 1 | Market Data Ingestion | WebSocket feed from Angel One broker |
| 2 | Protocol Decoding | Binary market data deserialization |
| 3 | Normalization | Canonical market event schema mapping |
| 4 | Validation | Sequence gap detection, duplicate suppression |
| 5 | Enrichment | Derived instrument metadata injection |
| 6 | Primary Cache | Redis hot-key storage for active instruments |
| 7 | Option Chain Warehouse | Full options chain construction via python/option_chain_warehouse/warehouse.py |
| 8 | Volatility Calculation | Real-time implied volatility surface |
| 9 | Greeks Computation | Delta, Gamma, Theta, Vega, Rho per contract |
| 10 | Technical Indicators | Moving averages, RSI, Bollinger Bands |
| 11 | Pattern Recognition | Candlestick pattern detection |
| 12 | Market Microstructure | Order flow imbalance, spread analysis |
| 13 | Signal Generation | Composite signal builder from multiple models |
| 14 | Risk Check | Position sizing, max drawdown, exposure limits |
| 15 | Broker Session Watchdog | Connection health, deterministic reconnection with fallback |
| 16 | Outbox Pattern | WolverineFx durable message delivery |
| 17 | SignalR Broadcast | Real-time push to Flutter mobile clients |
| 18 | Research Pipeline | Async slow path for ML model inference (TradingNimWorker) |
| 19 | Sentiment Analysis | Telegram crowd psychology aggregation (telegram-sentiment-worker) |
| 20 | Backtesting Validation | Historical signal replay against stored market data |
| 21 | Performance Attribution | P&L decomposition by signal source and strategy |
| 22 | Log Aggregation | Structured logging pipeline with centralized collection |
| 23 | Metrics Export | Prometheus metric collection (custom trading gauges) |
| 24 | Grafana Dashboarding | Real-time visualization (P&L, latency, order flow, system health) |
| 25 | Health Checks | Service liveness probes, readiness probes, dependency checks |
| 26 | Circuit Breakers | Automatic service degradation under fault conditions |
📡 Communication Protocol Matrix
| Channel | Purpose | Technology |
|---|---|---|
| Client → Gateway | REST API, SignalR subscriptions | HTTP/1.1, WebSocket |
| Gateway → Mobile | Real-time market data, signals, positions | SignalR + Server-Sent Events (SSE) for AI streaming |
| Internal Services | Service-to-service communication | gRPC |
| Async Messaging | Event-driven data transport | RabbitMQ via WolverineFx |
| Broker Feed | Market data ingestion | Angel One WebSocket API |
🐍 Python In-Process Analytics (CSnakes)
Using CSnakes, Python analytics run directly in-process with the .NET services, eliminating IPC overhead. Key Python modules verified from source:
| Script | Purpose |
|---|---|
python/daily_calibration_reporter.py | Daily calibration and performance reporting |
python/option_chain_warehouse/warehouse.py | Full options chain storage and retrieval |
python/angelone_validate.py | Broker feed validation and reconciliation |
python/generate_final_reports.py | Automated validation and production readiness reports |
python/deep_validate.py | End-to-end system validation suite |
python/remote_exec.py | Remote deployment execution scripts |
python/install-python-dependencies.sh | Python dependency installer |
📚 Validation & Documentation Repository
Verified from actual markdown files in the project root — comprehensive documentation suite:
Architecture: ARCHITECTURE.md, ARCHITECTURE_OVERVIEW.md, ARCHITECTURE_DIFF.md, ARCHITECTURE_GAPS.md, ARCHITECTURE_HEALTH.md, ARCHITECTURE_VALIDATION.md
Infrastructure Reports: DOCKER_INFRASTRUCTURE_REPORT.md, POSTGRES_REPORT.md, RABBITMQ_REPORT.md, REDIS_REPORT.md, BUILDKIT_CACHE_REPORT.md, COMMUNICATION_NODE_REPORT.md
Performance & Readiness: PERFORMANCE_REVIEW_REPORT.md, PRODUCTION_READINESS_REPORT.md, FINAL_PRODUCTION_READINESS_REPORT.md, CODEBASE_MEMORY_REPORT.md
API Validation: ANGELONE_API_VALIDATION.md, ANGELONE_REPORT.md, API_VALIDATION_REPORT.md, API_CONNECTIVITY_REPORT.md, API_GATEWAY_REPORT.md, API_USAGE_REPORT.md
System Validation: CORE_LIVE_VALIDATION_REPORT.md, ALPHA_ZOO_VALIDATION.md, BRIDGE_VALIDATION_REPORT.md, CSNAKES_BUILD_FIX_VALIDATION.md, DEEP_VALIDATION_REPORT.md, AUTOMATION_PIPELINE_REPORT.md
AI & Research: AI_PROVIDER_VALIDATION.md, AI_ORCHESTRATION.md, AI_SAFETY_GUARDS.md, AGENTIC_AI_ORCHESTRATION.md, BACKTEST_ENGINE.md, CONFIDENCE_DRIFT_REPORT.md
⚡ Infrastructure Configuration (Verified from docker-compose.infra.yml)
Designed for low-cost commodity hardware with strict resource boundaries:
| Service | Image | Port | Memory | Storage | Health Check |
|---|---|---|---|---|---|
| PostgreSQL | pgvector/pgvector:pg15 | 5432:5432 | 100 MB | postgres-data volume + init SQL at migrations/00-init-vector.sql:ro | pg_isready -U postgres (10s interval, 5s timeout, 5 retries) |
| Redis | redis:7-alpine | 6379:6379 | 80 MB | redis-data:/data, 64 MB maxmemory allkeys-lru | Redis ping health check |
| RabbitMQ | rabbitmq:3.13-alpine | 5672:5672 | 100 MB | rabbitmq-data volume, custom rabbitmq.conf | RabbitMQ health check |
All services share a trading-network bridge network with restart: unless-stopped policy.
🏗️ Additional Infrastructure & Tooling
- Docker Compose: Multi-file compose (
docker-compose.yml+docker-compose.infra.yml) with separate infrastructure and application layers - .env Configuration: Multiple environment profiles —
.env,.env.development,.env.docker,.env.template - Deployment Scripts:
build_and_deploy.ps1,remote_deploy.sh,configure-ip.ps1,start_tis.ps1,linux_deploy/directory - Monitoring: Prometheus metrics + Grafana dashboards for P&L, latency, order flow, system health, memory pressure
- Mobile Client: Flutter mobile app consuming REST, SignalR (real-time), and SSE (AI streaming)
⚡ Use-Case Engineering Rationale
- Fail-closed deterministic fast path: Execution-critical decisions function independently of AI, Python, or external API availability. AI enriches signals but never controls the execution gate.
- Comprehensive validation culture: 30+ markdown validation documents covering every subsystem — broker integration, API connectivity, memory profiling, production readiness, infrastructure stress testing, and AI safety guards.
- Lowest-cost deployment: Designed for Intel Celeron-class hardware with ~531 MB total container RAM, making it viable for VPS instances under $10/month.