Semantic Blindness: 500,000 Sensors Confused an LLM - Raahul Singh & Vanč Levstik, Phaidra

Phaidra's engineers had to make an LLM reason over 500,000+ sensor names in gigawatt-scale data centers, and every naive approach — vector RAG, pure LLM, parallel sharding — failed on recall or hallucinated phantom equipment. Their fix is a clean architectural pattern: the LLM plans and judges, deterministic code does the lookup and set operations, and cost scales with the hierarchy's depth rather than the number of instances. The production numbers are the best argument: from 116M tokens and 30% accuracy down to 390K tokens and 100% accuracy across 66 real cases.

• Design the system to scale with the depth of the hierarchy (data center → hall → rack → GPU), not the number of instances — their path linearizer produces the same-size summary for 64 GPUs or 460,000.
• The LLM outputs structured plans (what to collect, scope, filters) instead of processing raw names, holding cost constant at ~9,000 tokens per query regardless of system size.
• Their old approach degraded from 80% accuracy at 64 GPUs to 30% at 460,000; the hybrid approach hit 100% with 300x fewer tokens.
• A useful inversion of Karpathy's framing: AI-native systems should start at Software 3.0 for fast demos, then mature toward 1.0 determinism for production.

Part of Issue Nº 001: LLMs for judgment, code for counting: scaling to 500K sensors on 300x fewer tokens