NetfoldBot

What it is

NetfoldBot is the crawler behind Netfold. It reads the public feeds and blog index pages of writers that Netfold readers follow, and writers’ own pages to find a profile image. It stores titles, links, short excerpts and dates. Full article text stays on the publisher’s site.

How often it visits

Feeds are checked about once an hour, and less often, up to every 6 hours, when a feed has not changed. Requests are conditional (If-None-Match and If-Modified-Since), so an unchanged feed costs a 304 and no body. NetfoldBot honors Cache-Control max-age, up to a day. Blog index pages are checked about once a day. Feed and blog checks wait at least 3 seconds between requests to one host, or the site’s robots.txt Crawl-delay when that is longer. Reading robots.txt doesn’t wait, and a redirect back to a host already visited in that request waits only 3 seconds. Profile-image and link checks read a few pages once, when a writer is added or an admin asks.

How it behaves

NetfoldBot identifies itself with this User-Agent. It follows robots.txt: rules for NetfoldBot apply to every feed and page it reads, including redirect targets. When a site has no NetfoldBot rules, the rules for all bots (*) apply instead. Feed and blog checks honor Retry-After, and they back off for days when a site refuses them. It never solves challenges or CAPTCHAs, and it never disguises itself as a browser.

NetfoldBot/0.1 (+https://netfold.app/bot)

Where it comes from

NetfoldBot sends every request from one IP address. Its reverse DNS is crawl.netfold.app. The current address list is at https://netfold.app/bot/ips.json.

Block it or contact us

To block NetfoldBot, add these two lines to your robots.txt. To be removed from Netfold, or to allow us after you have blocked us, email [email protected] and tell us the site.

User-agent: NetfoldBot
Disallow: /