AI and crawlers
Language Model Access
UniConnectly guides and supports access to platform content by language models, AI systems and automated crawlers.
Platform summary
llms.txt
A short summary that helps language models understand UniConnectly: what the platform is, main pages, the list of blog posts and community pages.
Open file llms.txt (opens in a new tab)Full content
llms-full.txt
The full text of published blog posts in a single file, updated with every new post.
Open file llms-full.txt (opens in a new tab)Access rules
robots.txt
Rules that define which parts of the site search engines and AI crawlers can access.
Open file robots.txt (opens in a new tab)What these files do not include
Personal data, member lists, contact details and pages that require sign-in are not included. The files are generated from public pages only.
Recognized AI crawlers
41 AI crawlers named in the access rules:
- GPTBot
- OAI-SearchBot
- ChatGPT-User
- ClaudeBot
- Claude-Web
- Claude-SearchBot
- anthropic-ai
- Google-Extended
- GoogleOther
- Google-CloudVertexBot
- PerplexityBot
- Perplexity-User
- meta-externalagent
- FacebookBot
- Applebot-Extended
- DeepSeekBot
- Kimi-Bot
- MoonshotAI
- Moonshot-AI
- Qwenbot
- Alibaba-AI
- xAI-Bot
- Grokbot
- Doubao
- PanguBot
- CCBot
- Bytespider
- Amazonbot
- cohere-ai
- cohere-training-data-crawler
- DuckAssistBot
- MistralAI-User
- YouBot
- PetalBot
- Diffbot
- ImagesiftBot
- Timpibot
- omgili
- AI2Bot
- Webzio-Extended
- TikTokSpider