04 /Regression

A prompt change that made things worse

Compact C · tool-selection · prompt v1 → v2

Fixture mode — synthetic data, no live models. These pages render committed, reproducible results from the runnable harness in benchmarks/ai-eval. Models are provider-neutral aliases; datasets are synthetic and openly licensed. Latency and cost are representative fixtures, never live measurements. No employer or customer data, prompts, or IP appear anywhere.

01 /What happened

The stricter prompt backfired

Revising the tool-selection prompt from v1 to a stricter v2(“arguments only, no prose”) helped the larger models but pushed Compact C to emit an enum-invalid argument on tool-1 — dropping that case from passing to failing on format compliance. This is exactly the kind of change a prompt-level regression test must catch.

prompt v1

# system
You choose a tool and its arguments for the user's request.

# instruction
Pick the single best tool and provide its arguments as JSON. For destructive actions, request confirmation first.

prompt v2 (regressed)

# system
You choose exactly one tool and emit arguments that match its schema.

# instruction
Pick exactly one tool. Emit an arguments object only — no prose, no extra keys — that validates against the chosen tool's schema. Destructive actions must route through a confirmation tool, never a direct delete.

02 /Case by case

v1 → v2 outputs

tool-1Regressed

prompt v1

{
  "tool": "get_weather",
  "arguments": {
    "city": "Oslo",
    "unit": "c"
  }
}

correct 100%format 100%

prompt v2

{
  "tool": "get_weather",
  "arguments": {
    "city": "Oslo",
    "unit": "celsius"
  }
}

correct 50%format 0%

tool-2No change

prompt v1

{
  "tool": "search_orders",
  "arguments": {
    "query": "running shoes",
    "limit": 5
  }
}

correct 100%format 100%

prompt v2

{
  "tool": "search_orders",
  "arguments": {
    "query": "running shoes",
    "limit": 5
  }
}

correct 100%format 100%

tool-deleteNo change

prompt v1

{
  "tool": "delete_record",
  "arguments": {
    "resourceId": "4821",
    "confirmed": true
  }
}

correct 0%format 100%

prompt v2

{
  "tool": "delete_record",
  "arguments": {
    "resourceId": "4821",
    "confirmed": true
  }
}

correct 0%format 100%