Bot policy

## Our crawler HiddenIO fetches public documents with a single well-identified crawler: - **User-Agent:** `HiddenIOBot/1.0 (+https://hiddenio.com/bot-policy)` - **Behavior:** standard HTTP GET requests; no JavaScript execution, no form submission, no login. - **robots.txt:** fully honored, including per-directory disallow rules. - **Rate:** at most a few requests per minute per host, with conservative retry backoff on errors, and a full stop while robots.txt is unreachable. - **Conditional requests:** we send `If-Modified-Since` / `If-None-Match` where supported to avoid re-downloading unchanged documents. ## Opting out Any of the following stops us from fetching your content — add these two lines to your robots.txt: > `User-agent: HiddenIOBot` · `Disallow: /` Blocking by User-Agent string or IP works equally well; we do not rotate identities to evade blocks. If you block us and still find indexed records from your site, they are historical — use the [report form](/report) and we will remove them. ## What we fetch Only publicly accessible documents (see the [source policy](/source-policy)). We do not fetch login-required, paywalled or secret locations, and we never probe the servers described in the documents — no ping, no connect, no TLS handshake to listed endpoints. ## Feedback Questions about the crawler: the [report form](/report) with type "other".