SkillokBot — the SKILLOK page fetcher

What it is#

SkillokBot fetches pages when a person asks the SKILLOK agent to find or read something on the internet. It is not a search crawler: the bot opens only the addresses needed for a specific question and does not follow links further.

It introduces itself in the User-Agent header like this:

SkillokBot/0.1 (+https://skillok.org/docs/account/skillok-bot)

What the bot respects#

• robots.txt — per the RFC 9309 standard. The bot looks for a User-agent: SkillokBot group and, if there is none, the * group. If robots.txt does not answer (a server or network error), the bot fetches nothing from the site until the file becomes available.
• It does not get around bot protection. A captcha, a browser check, a «no bots» refusal — the bot accepts the refusal and tells the agent that the site is closed to bots.
• Size and time. A page is at most 5 MB, a fetch takes at most 15 seconds.

How to close your site#

Add to robots.txt:

User-agent: SkillokBot
Disallow: /

To close only part of the site, give a path, for example Disallow: /private/.

Was this helpful?

Yes No