Mathematical bounds for LLM alignment discovered
Researchers have established the theoretical maximum reward gain possible during language model alignment under fixed KL-divergence constraints. The findings demonstrate that 'best-of-N' sampling effectively reaches these theoretical bounds, whereas common reinforcement learning approaches like PPO and GRPO currently fall short of optimal performance.