Qudoc allows you to hide your part or all of your website and supports the robots.txt files and the the noindex and nofollow directives.
We support the following directives in a robots.txt file:
| Directive | Description | Example |
|---|---|---|
| User-agent | We obey all rules for the agents * and qubot. Our agent runs as qubot on IP 58.108.186.149. Rules for qubot will override rules for *. | User-agent: qubot |
| Crawl-delay | Qubot accepts a crawl delay in seconds, with millisecond resolution. By default it has a delay of 100ms. | crawl-delay: 1.5 |
| Disallow |
Paths matching the expression will not be crawled. The most basic form
matches any path starting with the expression. Qubot also
accepts the wild cards * and $. * will match any sequence of zero (0) or more characters. $ will force match the whole path. Where a Disallow and an Allow directive both match a path, the longest will be used. |
Disallow: /cgi-bin/ Disallow: *.jpg$ |
| Allow |
Paths matching the expression will be crawled. This is used to
override Disallow commands. Qubot also
accepts the wild cards * and $. * will match any sequence of zero (0) or more characters. $ will force match the whole path. Where a Disallow and an Allow directive both match a path, the longest will be used. |
Allow: /documents/ Allow: *.doc$ |
| Sitemap | There can be any number of sitemap or siteindex entries. These must specify the full URL to the sitemap or siteindex entry. User-agent does not effect sitemaps. | Sitemap: http://www.example.com/sitemap.xml |
Qubot supports the following HTML metadata directives:
| Directive | Description | Example |
|---|---|---|
| noindex | Qubot will not index pages with this metadata value. | <meta name="qubot" content="noindex" /> |
| nofollow | Qubot will not follow links on a page with this metadata value. This can also be used for single links in anchor tags. |
<meta name="robots" content="nofollow" /> <a href="http://www.example.com/" rel="nofollow">Example link</a> |
