Robots.txt editor
Edit the site's robots.txt in a code editor; it is served at /robots.txt.
SEO Manager > Robots.txt has a plain-text code editor for the site's robots.txt. Whatever is saved is served as text/plain at /robots.txt. When nothing is saved, the editor pre-fills a suggested file (allow all plus a Sitemap line), but the live /robots.txt only returns 'User-agent: *' until you click Save.
How it works
- Empty setting: output is the single line 'User-agent: *'.
When to use it
- Block crawling of a staging section
- Point crawlers to the sitemap
Limitations
- No validation of robots.txt syntax
- The default shown in the editor is not live until saved
Availability
- Works in Designer and MCP.
- Included in every plan.
- Limits: MCP: 20,000 characters
How to use it.
- Open SEO Manager > Robots.txt
- Edit the text
- Click Save
Related features.
XML sitemap
Automatic /sitemap.xml with all indexable pages, CMS items per collection, every language, and room for custom URLs.
Per-page indexing controls (meta robots)
Choose per page whether search engines may index it and follow its links.
llms.txt and llms-full.txt
Publish /llms.txt and /llms-full.txt so AI assistants can read a clean summary of your site, written by hand or by AI.
Browse other features
Designer
Bring your ideas to life with our visual website builder. Drag and drop with ease, or dive into coding when needed.
→AI Studio
Describe it and AI builds it. Every change lands in the same project you can open in the Designer.
→MCP
Connect Claude Code, Cursor or any MCP client and build whole sites with your own agent.
→CMS
Create, organize, and manage your dynamic content with the maximum freedom and flexibility of our visual CMS.
→SEO
Divhunt delivers ultra-fast & structured sites that are good for ranking by default, while offering complete control for customization.
→REST API
Easily link to any REST API endpoint with instant preview and expand your site’s functionality smoother than ever before.
→Plugins
Divhunt offers an open platform for creating plugins to upgrade features, add new capabilities, and simplify building better sites.
→Localization
Localize everything from static pages and CMS content to URLs and metadata, in up to 15 languages with no add-on.
→Interactions
Visually create animations and interactions with a simple, intuitive timeline. Build smooth, engaging experiences without code.
→Custom Files
Manage JS and CSS files with a structure developers love, combining folders and clean organization.
→Memberships
Create plans, decide who sees what and connect it all to your CMS. Build courses, communities or client portals.
→