Smart alerts

Re-run a test or workflow on a schedule, and get told when something actually changed.

Two kinds of alert

A smart alert watches a single test. A workflow alert watches a whole saved workflow - a scheduled check is just an ordinary workflow run, so its history and your alert history are the same thing. A workflow can have at most one alert, since there's only one meaningful schedule to keep for it.

What gets checked

Each scheduled check re-runs the test (or workflow) and compares the new result against your original baseline. It looks for:

• Cost - the model got cheaper or more expensive.
• Speed - noticeably slower than before.
• Answer - the response looks meaningfully different from the original.

Cost changes are reported right away. Speed and answer changes have to show up on two checks in a row before they're reported - a single run can be noisy, and a real pattern is worth more than one data point. For a workflow alert, only a genuinely new warning (or new good news) triggers an email - a check that still agrees with the last one stays quiet.

This tells you something is different, not that it's worse - judging whether an answer still meets the bar is what Quality check is for.

Spend protection

Each alert can have its own cap on credits per check. A check estimated above that cap is skipped and recorded rather than run, and you'll see that it was skipped. For a workflow alert, that cap applies to the whole scheduled run, checked before anything is billed.

You'll also get a heads-up email whenever your balance is already too low to cover a check before it even runs - sent every time that's true, not just once, since a low balance that keeps being low is worth repeating until you top up. If a check is ever actually refused outright for lack of credits, that's reported separately too.