2 hours ago
Model upgrades always alter the established behavior of an agent since the underlying weights shift around so much. The instructions that gave perfect outputs yesterday suddenly trigger complete nonsense on the new architecture. Running side-by-side comparisons between versions is just the standard procedure for any production environment now. A lot of developers spin up local testing frameworks using Promptfoo to track those behavioral changes. That open-source tool lets devs run automated evaluations and catch regressions before users see them.