Skip to content
CiteMagnet

For website owners

CiteMagnetBot

CiteMagnetBot is the crawler behind CiteMagnet's site audits. It reads public pages the way a search engine does, so business owners can find and fix problems on their own websites. The same crawler serves every CiteMagnet product (CiteMagnet, FoodCited, ShopCited, StayCited, SalonCited, TradeCited, FitCited, ClinicCited, and FirmCited).

How to recognise it

Every request carries this user agent:

Mozilla/5.0 (compatible; CiteMagnetBot/1.0; +https://citemagnet.com/bot)

How it behaves

  • Only sites our customers add

    CiteMagnetBot visits a website only after someone adds it to their CiteMagnet account to audit it, or runs one of our free checks on that address. It does not crawl the open web.

  • Respects robots.txt

    Before crawling, the bot reads /robots.txt. It follows rules for the CiteMagnetBot user agent when present, otherwise the * group, and skips every disallowed URL. Pages it skipped are listed in the audit so the site owner can see why.

  • Gentle by design

    At most four pages are fetched at a time, each with a timeout, up to the page limit of the owner's plan. Carts, admin areas and search results are skipped, and the bot backs off once a site starts answering “too many requests”.

  • No bypassing protection

    If a firewall, CAPTCHA or password blocks the bot, the audit stops and explains how to allowlist CiteMagnetBot. We never try to get around bot protection.

Allowing or blocking CiteMagnetBot

To keep the bot out of part of your site, or all of it, add a group to your robots.txt:

User-agent: CiteMagnetBot
Disallow: /private/

If you use CiteMagnet on your own site and your firewall (Cloudflare, AWS WAF, Sucuri, Wordfence…) blocks the audit, allow requests whose user agent contains CiteMagnetBot, then run the audit again.