# SkillokBot — the SKILLOK page fetcher

How the SKILLOK agent reads pages on the internet, what it respects and how to close your site to it.

## What it is

`SkillokBot` fetches pages when a person asks the SKILLOK agent to find or read something on the
internet. It is not a search crawler: the bot opens only the addresses needed for a specific question
and does not follow links further.

It introduces itself in the `User-Agent` header like this:

```
SkillokBot/0.1 (+https://skillok.org/docs/account/skillok-bot)
```

## What the bot respects

* **robots.txt** — per the RFC 9309 standard. The bot looks for a `User-agent: SkillokBot` group and,
  if there is none, the `*` group. If `robots.txt` does not answer (a server or network error), the bot
  fetches nothing from the site until the file becomes available.
* **It does not get around bot protection.** A captcha, a browser check, a «no bots» refusal — the bot
  accepts the refusal and tells the agent that the site is closed to bots.
* **Size and time.** A page is at most 5 MB, a fetch takes at most 15 seconds.

## How to close your site

Add to `robots.txt`:

```
User-agent: SkillokBot
Disallow: /
```

To close only part of the site, give a path, for example `Disallow: /private/`.

---

https://skillok.org/docs/en/account/skillok-bot