The development

What happened

OpenAI’s 23 October engineering report explains how Habitat balances shared Azure Cosmos DB capacity with workload isolation. The company reports that selected replication, nearby caching and a Rust service rewrite together reduced global median read latency from roughly 120 to 10 milliseconds.

Key facts

  • Habitat returned to shared containers after dedicated containers made launch capacity planning burdensome. Hot-key throttling and request priorities protect interactive workloads.

  • Regional capacity constrains data placement. Valkey caches use Cosmos DB change feeds to propagate invalidations globally within seconds.

  • The reported median improvement combines multiple changes; uncached slow requests still need underlying storage. It is not a Rust-only benchmark.

  • A separate RocksDB Cloud/Raft storage system remains under development, rather than a completed replacement.

The context

Why it matters

The account illustrates how latency, isolation and capacity planning interact at large scale. Its lesson is operational rather than a transferable speed guarantee: shared capacity needs workload protections, and caches do not eliminate distant storage on misses.