The development
What happened
OpenAI’s 23 October engineering report explains how Habitat balances shared Azure Cosmos DB capacity with workload isolation. The company reports that selected replication, nearby caching and a Rust service rewrite together reduced global median read latency from roughly 120 to 10 milliseconds.
Key facts
Habitat returned to shared containers after dedicated containers made launch capacity planning burdensome. Hot-key throttling and request priorities protect interactive workloads.
Regional capacity constrains data placement. Valkey caches use Cosmos DB change feeds to propagate invalidations globally within seconds.
The reported median improvement combines multiple changes; uncached slow requests still need underlying storage. It is not a Rust-only benchmark.
A separate RocksDB Cloud/Raft storage system remains under development, rather than a completed replacement.
The context
Why it matters
The account illustrates how latency, isolation and capacity planning interact at large scale. Its lesson is operational rather than a transferable speed guarantee: shared capacity needs workload protections, and caches do not eliminate distant storage on misses.