Shopvisly-Audit/1.0Shopvisly-Audit is the crawler behind Shopvisly’s Site Audit. It only runs when a Shopvisly user starts an audit of a site, and it checks pages for technical SEO issues such as broken links, redirects, missing titles and descriptions, canonical tags, structured data and noindex directives.
User agent
Every page request sends this exact header:
User-Agent: Shopvisly-Audit/1.0
Match on Shopvisly-Audit in firewall and rate-limit rules, so the rule keeps working when the version number changes.
robots.txt
- Before crawling, the crawler fetches
/robots.txtfrom the site’s origin and keeps those rules for the whole audit. - It obeys the group for
User-agent: Shopvisly-Auditif there is one, and otherwise the*group. Disallowed URLs are never requested, including the start URL. - If robots.txt is missing or cannot be reached, everything is allowed. If it answers 429, the crawler waits and tries once more.
Crawl-delayis not used; see crawl rate.
To keep the crawler out completely:
User-agent: Shopvisly-Audit
Disallow: /
What it crawls
- The start URL, the sitemaps listed in robots.txt, and
/sitemap.xml. Sitemap indexes are followed up to three levels deep, up to 300 sitemap files. Files larger than 10 MB are skipped. - Links found on crawled pages, but only on the same origin as the start URL.
- Up to the page limit set for the audit: between 10 and 10,000 pages, 50 by default.
Crawl rate
- The crawler starts at most one page request per second per audit. During discovery it reads up to five sitemap files at a time.
- On a 429 Too Many Requests, it pauses the whole crawl for the time in your
Retry-Afterheader. Without that header it waits 30 seconds, and longer after each refused URL. It also halves its pace, down to one request every 10 seconds, and returns to normal after 20 pages in a row are served. - A refused URL is tried once more after the pause, then reported as rate limited.
- The crawl stops early if three URLs in a row are refused, or after two hours of waiting in total. The report then says it is incomplete.
Allow the crawler through a firewall
If an audit reports “Crawler was blocked” (a bot challenge or 403) or many 429 pages, let the crawler through on your own site:
- Cloudflare: add a WAF custom rule that skips bot protection when the user agent contains
Shopvisly-Audit, and a rate-limiting rule exception for the same match. Some free plans may need bot protection relaxed instead. - Other firewalls and CDNs: add an allow or rate-limit exception for user agents containing
Shopvisly-Audit. - Then run the audit again. If your limit is strict, an audit with fewer pages also helps.
A user agent can be copied by anyone. Only allowlist it on sites you own, and scope the rule as narrowly as your firewall allows.
Shopify stores: crawler access
Shopify can rate-limit or block crawlers it has not authorized. Instead of an allowlist, create a crawler access signature in Shopify admin under Online Store → Preferences → Crawler access (see Shopify’s guide), and paste it into Shopvisly under Settings → Crawler access. Shopvisly sends those signature headers only to the store’s own domain, and stops using them when they expire.
JavaScript rendering and Lighthouse
- Render JavaScript is optional. When it is on, pages are loaded in a headless browser through a rendering provider, so those requests can come from the provider’s infrastructure. Rendered audits stop crawling after 22 hours. Rendering uses usage credits for the pages actually rendered.
- Lighthouse performance tests, when included, run on up to 10 pages (the homepage and one page per URL pattern) through a third-party Lighthouse service.
Questions or abuse reports
If you see unwanted traffic from Shopvisly-Audit, email support@shopvisly.com with the time, the URLs and your site’s domain.
FAQ
How do I block the Shopvisly crawler?
Add a group for Shopvisly-Audit to your robots.txt with Disallow: /. The crawler reads robots.txt before it fetches any page.
Does the crawler follow Crawl-delay?
No. It paces itself instead: at most one new page request per second, and slower when your server answers 429 Too Many Requests.
Does it crawl other websites?
No. It only follows links on the same origin as the start URL (same scheme, host and port).