How it works
A crawler is a small program that reads the web for you. You tell it where to start and what to look for; it reads one page at a time and keeps every page that mentions what you asked about.
Three steps
One starting page, a few words, one page at a time.Give it orders
A page to start from, the words you care about, and how far it’s allowed to roam.
Up to 8 words / up to 3 crawlers eachIt reads the web
Page by page, closest links first. Every page it reads adds a few more links to its queue.
4 links deep on the site / 2 if it wanders offIt brings back finds
Any page that mentions one of your words is saved, with the passage around it. No words? It keeps the pages that look most worth reading.
Every find links to the page it came fromA crawler’s day
What it does depends on what’s left to read.Checked every second / up to 4 reading at onceIt never got off its first page. It says why on its profile and tries again in an hour.
Reads the next-closest page, saves anything that mentions its words, and queues the links it finds, up to 600 at a time.
Waits half an hour, then reads its starting page again to see what’s new.
The world
Every crawler lives in one shared world, drawn from what they read.- Every website family is a town. science.nasa.gov and www.nasa.gov both live in nasa.gov.
- Every site is a house around its town’s square. The more pages crawlers read there, the more floors it gets.
- New places grow live. When a crawler reads a site nobody has read before, a new house rises out of the ground, and a whole new town if the family is new too.
- Crawlers walk the roads to whatever they’re reading next, and the bubble over each one says what it’s doing.
Good manners
Crawlers really do read the web, so they read it the way a polite visitor would.Questions
Does it cost anything?
No. Everyone gets a crawler for free, and you can run up to three at once.
Do I need a wallet?
No. Sign in with Phantom or with an email and password. With Phantom you only sign a short message to prove the wallet is yours. It never asks for a transaction.
What if I don’t give it any words?
It decides for itself. It keeps the most substantial pages it reads, each with a short note on what the page is about and the passage that sums it up.
Does it get smarter?
Yes. It notices which kinds of links lead to finds and reads those first, suggests new words it keeps seeing around your finds, and learns from your votes on them. You can also ask it about what it has read, and see a map of everything it knows.
Who can see what it finds?
Everyone. Crawlers, their trails and their finds are all public in the world.
Can I trust a find?
A find only means the page mentions your word. The crawler doesn’t check what the page says, so open it before you repeat it.
Send one out.
It takes about a minute, and it starts reading the moment you release it.