The development

What happened

A joint Microsoft and Hugging Face post released ThinkingBox-Training, a stack that turns executable checks of an agent’s final database state into reinforcement-learning rewards. The authors report that after 50 GRPO steps on Qwen3.8-27B, average success across 507 benchmark tasks increased from 51.70% to 60.78%. They also report more tasks passing all 20 recorded attempts, but slightly fewer tasks solved at least once. The benchmark was held out from training, though it shares framework components and task characteristics with the training setup.

Key facts

  • The authors say they ran 50 GRPO steps on Qwen3.8-27B using 187 training tasks outside ThinkingBox-Bench.

  • Across 507 benchmark tasks tested 20 times each, reported average success rose from 51.70% to 60.78%.

  • Tasks passing all 20 recorded attempts increased from 38 (7.50%) to 107 (21.10%), while tasks solved at least once fell from 453 to 446.

  • The released stack includes task selection, rollouts, rewards and Verl integration; its public sample is not the private 187-task training pool.

  • The benchmark and training setups share framework components, some tools and task style, limiting how far the reported results establish transfer to other environments.

The context

Why it matters

Using checks of final database state as rewards offers a concrete way to train agents on tasks with verifiable outcomes. The reported improvement is encouraging, but the benchmark’s overlap in framework and task characteristics limits how far it supports claims about generalization; it also does not establish production reliability or guarantee correct behavior.