An AI skill in OpenAI's official repository directed production apps to an unregistered domain for months. When researchers claimed it, their sinkhole collected live tokens and API calls. The domain was finally transferred to safe holding in mid-August, but the question stands: what if attackers had registered it first?
This incident exposes a critical gap in how your team treats AI components. You vet SaaS vendors. You review open-source dependencies. But what about the instruction files that tell your AI agents where to send credentials?
What This Checklist Covers
This checklist helps you establish vetting and monitoring controls for AI skills, plugins, and instruction files entering your environment. It assumes you're already running third-party risk management (TPRM) workflows for traditional software vendors and need to extend those controls to AI supply chain components.
Prerequisites
Before you start:
- Inventory your AI agent deployments. You can't vet what you don't know exists. Document which teams are using AI agents, which platforms host them, and which skills or plugins they've installed.
- Assign ownership. Clarify whether AI skills fall under software procurement, TPRM, or application security. Don't let them fall into a governance gap.
- Define your risk appetite. Decide which AI use cases justify skill installation, such as internal automation versus customer-facing agents handling personally identifiable information (PII).
Good looks like: A living register of AI agents, updated quarterly, with named risk owners for each deployment.
Checklist Items
1. Verify the Skill's Source Repository and Contributor Status
Check whether the skill comes from an official vendor repository, a community contribution, or an unverified third party. The Render skill sat in OpenAI's official repo but still contained a hallucinated domain introduced in January.
Good looks like: Documentation showing the skill's provenance (official vendor, verified contributor, or community-submitted) and whether the vendor has a security contact listed.
2. Review All External Domains and Endpoints Referenced in the Skill
Extract every URL, API endpoint, and domain the skill instructs your agent to contact. Confirm each domain is registered to the claimed vendor and serves the expected function. The render-internal.com domain was never registered by Render; it was a placeholder that became a liability.
Good looks like: A line-by-line audit log of domains, with DNS/WHOIS confirmation that each is controlled by the legitimate vendor.
3. Test the Skill in a Sandboxed Environment Before Production Deployment
Run the skill in an isolated environment with dummy credentials. Monitor outbound connections. When AIR Security tested the Render skill with Codex, it confirmed the hallucinated domain made its way into real application code as the default option.
Good looks like: Test logs showing which endpoints the skill contacted, what data it attempted to send, and whether any unexpected domains appeared.
4. Apply the Same Due Diligence Standards You Use for Software Vendors
If the skill integrates a third-party service, treat that service as a vendor. Request SOC 2 reports, review sub-outsourcing clauses if the skill calls nested APIs, and confirm the vendor has incident escalation procedures.
Good looks like: A completed SIG Core assessment for any third-party service the skill depends on, with documented right to audit provisions in the contract.
5. Establish Continuous Monitoring of Active Arrangements for Installed Skills
Don't assume a vetted skill stays safe. Monitor for commits to the skill's source repository, domain registration changes for referenced endpoints, and CVE disclosures. The Render issue was introduced with a January commit and sat undetected until researchers found it months later.
Good looks like: Automated alerts when a skill's source code changes or when a referenced domain's WHOIS record updates.
6. Define Notification Timelines for Skill-Related Incidents
Require vendors to notify you within a defined window if a skill is found to contain a vulnerability. OpenAI and Render resolved the issue after AIR Security's disclosure, but you need contractual language ensuring you're informed, not left to discover breaches passively.
Good looks like: Contract clauses requiring 24-hour notification of major ICT-related incidents affecting skills you've deployed, with escalation paths to your TPRM team.
7. Maintain a Substitutability Plan for Critical AI Skills
If a skill is revoked or found vulnerable, can you replace it without disrupting operations? OpenAI purged the Render plugin from openai/plugins entirely. You need a wind-down plan.
Good looks like: Documentation showing alternative skills or manual processes that can replace each critical AI skill within 48 hours.
8. Review Skills for Typosquatting and Namespace Confusion Risks
A recent Trojanized AI skill typosquatted popular services on skills.sh and garnered 1.7 million installations. Verify the skill name matches the official vendor namespace exactly. Check for homoglyphs, extra hyphens, or similar-sounding names.
Good looks like: A blocklist of known typosquatted skill names and a manual review step confirming the skill's namespace before installation.
Common Mistakes
Treating official repositories as inherently safe. The Render skill sat in OpenAI's official repo. Vendor-published skills can still contain mistakes, unsafe assumptions, or hallucinated dependencies.
Skipping pre-deployment testing. If AIR Security hadn't tested the skill in a controlled environment, the hallucinated domain would have stayed hidden. Your sandbox is your first line of defense.
Ignoring fourth-party dependencies. A skill that calls a third-party API introduces fourth and nth party management risk. If that API has its own sub-processors, you're managing a chain you didn't vet.
Assuming skills are static. Skills receive commits, updates, and patches. A skill that was safe in January may introduce a vulnerability in February. Continuous monitoring isn't optional.
No revocation process. When a skill is found vulnerable, you need a documented process to disable it across all deployments. Don't rely on manual memory.
Next Steps
- Map your current AI agent deployments to this checklist. Identify gaps where skills were installed without vetting.
- Extend your vendor intake workflow to include AI skills. Add a step requiring sandbox testing and domain verification before production use.
- Draft contract language for AI platform vendors requiring disclosure of skill vulnerabilities and notification timelines.
- Assign a risk owner for AI supply chain monitoring. This can't be an afterthought buried in DevOps.
The Render skill flaw was resolved before attackers exploited it. The next one might not be. Treat AI skills like any other supply chain component: vet them, monitor them, and plan for their failure.
