Stop a Deploy When an AI Model Your App Calls Is Retired

Picture a support app that calls gpt-4.1 for its answer drafts and o4-mini for tagging tickets. The model names sit in a config file and in a couple of prompts. Nobody on the team reads the provider’s deprecation page, and nobody should have to.

The cheapest place to notice a model going away is the deploy pipeline. It already runs on every change, it already fails loudly, and the people who can fix a model name are the ones watching it. So RadarModel AI goes there, as one more step before the deploy.

Four steps, from watching the models the app calls to failing the deploy job, next to a report where gpt-4.1 first warns, then turns breaking and stops the job with exit 1.

Warn once, then stop the deploy.

The setup

The watch list holds the two models the app calls, nothing else. The other models OpenAI lists can come and go without anyone hearing about it.

{
  "providers": [{"type": "openai"}],
  "watch": ["openai:gpt-4.1", "openai:o4-mini"]
}

In GitHub Actions, the snapshot folder lives in the cache, so each run compares with the one before:

- uses: actions/cache@v4
  with:
    path: .radarmodel
    key: radarmodel-${{ github.run_id }}
    restore-keys: radarmodel-
- run: radarmodel-ai check
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

What happens when the model goes

The run below is the released binary against a saved copy of OpenAI’s list, edited to take gpt-4.1 out. The first deploy after it disappears gets a warning and goes through:

$ radarmodel-ai check
RadarModel AI: 1 warning
openai  WARNING  gpt-4.1 *  gone (missing from this scan; breaking if the next scan misses it too)
openai           1 more below warning

One missing scan isn’t enough to stop anything, because lists sometimes come back short for a minute. The next deploy finds it still gone, calls it breaking, and the step exits 1:

$ radarmodel-ai check
RadarModel AI: 1 breaking
openai  BREAKING  gpt-4.1 *  gone (confirmed by a second scan)
$ echo $?
1

The job fails with the model’s name in the log. Someone switches the config to the replacement, the next deploy runs green, and users never saw an error. The “1 more below warning” was gpt-5 turning up in the list, which -min info would show.

If the provider has an outage instead, the step exits 2 and says the scan failed. Whether a failed scan should block the deploy is a team’s call; radarmodel-ai check || [ $? -eq 2 ] lets it through. The quick start has the rest of the setup, and why models drift has the background.