Crawler
The Truscan crawler
You are probably here because you found truscan-search in your server logs. Here is exactly what it is and how to stop it.
How to identify it
Every request sends this user agent:
truscan-search/1.0 (+https://truscan.co/bot)Why it fetched your page
It is triggered by a person, not a schedule
Truscan does not run a background crawl and does not build an index of the web. A fetch happens only when someone runs a search, your page is among the results, and they asked for the page contents.
One request, then it stops
A single GET for the page a user asked about. It does not follow your links, it does not walk your site, and it does not come back on its own.
It only reads the visible text
The HTML is parsed for readable article text, which is returned to the person who searched. Scripts, styles and navigation are discarded.
It gives up quickly
Requests time out in seconds and redirects are capped. If your server is slow or returns an error, the page is skipped and the search continues without it.
How to block it
Because a fetch is made on behalf of a person who is already looking at your page in their results, the crawler does not read robots.txt. It behaves like a reader following a link rather than a bot indexing a site. If you would rather it did not fetch you at all, block the user agent at your edge:
if ($http_user_agent ~* "truscan-search") { return 403;}A 403 or 401 is understood and the page is dropped from the result. Nothing is retried and nothing is cached from a blocked response.
Ask us instead
If you would prefer not to configure anything, write to us with the domain and we will stop fetching it. We do not ask for a reason.