# Create Web Page — crawl policy. # Public marketing, docs, legal, and published /sites/ pages are open to search # engines AND to AI answer engines (OpenAI, Anthropic, Perplexity, Google AI, # Apple, DuckDuckGo, Meta, Amazon, Mistral, Common Crawl). Private workflow # paths are disallowed. Machine-readable product summary: /llms.txt and # /llms-full.txt. MCP endpoint: https://create-web-page.com/mcp (docs: /docs/mcp). User-Agent: * Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: Googlebot Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: Bingbot Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: Applebot Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: DuckDuckBot Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: OAI-SearchBot Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: ChatGPT-User Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: GPTBot Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: Claude-SearchBot Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: Claude-User Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: ClaudeBot Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: PerplexityBot Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: Perplexity-User Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: Google-Extended Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: Applebot-Extended Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: DuckAssistBot Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: Meta-ExternalAgent Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: Amazonbot Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: MistralAI-User Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ User-Agent: CCBot Allow: / Allow: /abuse Allow: /cookies Allow: /docs/mcp Allow: /docs/mcp/claude Allow: /legal-notice Allow: /llms.txt Allow: /llms-full.txt Allow: /mcp Allow: /privacy Allow: /security Allow: /sites/ Allow: /terms Allow: /.well-known/llms.txt Allow: /.well-known/oauth-authorization-server Allow: /.well-known/oauth-protected-resource Disallow: /api/ Disallow: /claim/ Disallow: /media/upload/ Disallow: /oauth/ Disallow: /ops/ Disallow: /preview/ Sitemap: https://create-web-page.com/sitemap.xml Sitemap: https://create-web-page.com/sitemap-pages.xml Host: https://create-web-page.com