| Parameter | Before | Now |
|---|---|---|
| Architecture | Remote LLM-based classification pipeline | Two-stage pipeline: regex candidate generation + fine-tuned ModernBERT classifier (ONNX, 8-bit quantized) |
| Precision | Baseline (previous LLM pipeline) | 13% higher (offline evaluation) |
| Recall | Baseline (previous LLM pipeline) | 19% higher (offline evaluation) |
| Owner-weighted acceptance | Baseline (previous LLM pipeline) | 11.5% higher (online A/B) |
| Median post-push scan execution time | Baseline (previous LLM pipeline) | 86% lower |
| Serving cost | Baseline (previous LLM pipeline) | More than 98% lower |
GitHub replaced its LLM-based pipeline for detecting generic secrets (like passwords and API keys) with a fine-tuned ModernBERT classifier combined with regex candidate generation. The new two-stage system improved detection accuracy while cutting scan time and serving costs dramatically, and it's now fast enough to run during push protection, not just after code is committed.