Back to Glossary
Agent Commerce
ClaudeBot
Anthropic's crawler for Claude. As with OpenAI, separate user agents cover different purposes, and robots.txt directives are honoured per agent.
Why it matters
Same reasoning as the OpenAI bots and usually the same mistake: a blanket AI-bot block that was never a deliberate decision about any specific assistant.
In practice
Audit every AI user agent in your robots.txt individually and record why each is allowed or blocked. A rule nobody can explain is a rule that will be wrong.
Related terms