Public pages only
The crawler requests pages that are already public. It does not sign in, submit forms, or try to get past a paywall or access control.
What a fetch is for
ScaleMule reads public pages while gathering news and shopping information for local publications. A saved page is a record for a person to review. Requesting a page does not publish it.
robots.txt decides the paths
Each fetch follows the robots.txt group that applies to every crawler, User-agent: *. A Disallow on a path means that path is not requested.
A refusal stops the site
A block page, an access challenge, or a rate-limit response stops further fetches of that site. Addresses are not changed in order to continue.
How to opt out
The fastest opt-out is robots.txt. The crawler reads the group for every agent. A group that names only ScaleMuleBot does not apply, because that name is no longer sent.
User-agent: *
Disallow: /A narrower Disallow skips only that path. To skip one site without a robots change, email legal@scalemule.com and include the site name.
What a request contains
Earlier fetches identified the crawler as ScaleMuleBot and linked to this page. That identifier is retired. A current request does not include the ScaleMule name, the name of a publication, or the address of this page.
Fetches are spaced apart, limited per site each day, and made one at a time for a given site. If the site tells the crawler to slow down, that site is paused.