Review Request | When does an AI Agent Skill go stale? An Anthropic claude-api fact-drift audit

Direct answer

AI Agent Skills do not share a universal 30- or 90-day shelf life. Review a Skill whenever it embeds a drifting surface such as a model ID, API field, numeric limit, event meaning, state transition, or support matrix. Before use, pin the source and commit and recheck every fact that can change runtime behavior.

We audited the official Anthropic claude-api Skill at two fixed commits: fa0fa64 and 1f630fd. Within skills/claude-api/, 28 files changed, with 366 additions and 184 deletions. This was more than copy editing: at least two old facts were directly corrected, while several new constraints change generated configuration and client behavior.

Why stale Skills are risky

A Skill is not passive documentation. It repeatedly injects steps, parameters, and decisions into an Agent workflow. An old document may send a developer back to search after an error; an old Skill can make an Agent repeat the same outdated process with consistent, polished output. Repository stars, install counts, and a recent timestamp do not answer the important question: which external facts does this Skill depend on, and are they pinned to traceable first-party evidence?

Example: 100K tokens became 100,000 characters

The old Skill said MCP output above 100K tokens would be offloaded to a file. The new version says output above 100,000 characters—roughly 25,000 tokens—and expands the scope from MCP output to both built-in and MCP tools. The unit and coverage both changed. Using the new approximation, the old threshold was about four times higher. A context-budget implementation based on the old statement could summarize or offload at the wrong point and ignore large built-in-tool results.

Other runtime semantics that changed

system.message appends context

The old Skill described it as changing or replacing the effective system prompt and listed only Opus 4.8. The new Skill says it appends system-level context and lists Opus 4.8, Sonnet 5, Fable 5, and Mythos 5. That changes both prompt composition and the support matrix.

Managed-Agent effort belongs on the Agent model

The update documents an {id, effort} Agent configuration and warns that placing effort in a session model override silently does nothing. Successful requests therefore do not prove the intended effort was applied; reviewers must check field location, not only field existence.

initial_events can start a Session in running

A non-empty event list is atomically validated and can create a Session directly in running. A client that waits only for an idle → running transition may wait forever. Event-driven Skills need state-machine semantics, not just a copied field table.

Agent version selects the concurrency policy

Supplying version enables optimistic concurrency; omitting it produces an unconditional last-write-wins update. This is a real choice between conflict-aware interactive updates and declarative overwrite behavior.

Webhook lifecycle coverage changed

The new event table adds environment and memory-store lifecycle events and explicitly states that memory_store.updated does not exist. Old handlers may miss real events, while guessed names may wait for an event that can never arrive.

A practical freshness checklist

1. Pin repository, Skill path, tag, version, or full commit.
2. Search for model IDs, field paths, numeric limits, event names, state machines, and support matrices.
3. Verify each item against official documentation, repositories, or release records.
4. Read the fact-level diff instead of relying only on the changelog.
5. Prioritize changes that affect requests, permissions, concurrency, context, or error handling.
6. Run one minimal old-versus-new task and compare configuration, tool calls, and state waits.
7. Recheck after relevant API releases, model retirement, or major SDK changes instead of relying only on a calendar interval.

A useful Skill or MCP recommendation should name the official source, checked version or commit, last fact-check date, dependencies, actual test task, known drift surfaces, and recheck triggers. “Recently updated” is not proof of quality, and “not recently updated” does not automatically mean broken.

Evidence boundary

This audit proves that the two Skill versions contain materially different operational facts. We did not run the same generated API task through both versions, and it does not prove that a specific model will always repeat the old statements. The next experiment should generate the same configurations with both Skills and compare the actual requests and state handling.

— Token Beggars Editorial

References

• Anthropic Agent Skills repository