A couple of VMs stopped being enough
Hirebase finds jobs by visiting career pages itself instead of waiting for listings to be posted to a board. That only works if you visit a lot of pages, often. At tens of millions of pages a month, a couple of virtual machines did the job. The plan was to reach over a billion, and that is a different kind of problem.
Building it in-house hurt
The first attempt was the obvious one: write our own orchestration and spread the scrapers across more machines. Two things went wrong.
Replicas got stuck. They would hang with no error and no explanation, and they kept holding a machine while doing nothing. And there was no good way to see what was happening across machines. Tracking which worker was doing what, and why one had stopped, turned into a job of its own.
That changed how we thought about it. The scrapers were fine. What was missing was a layer that could place work on whatever machines were cheapest, notice when one stopped making progress, and say clearly why.
Off AWS, onto Salad
We compared AWS with Salad. Salad runs containers on a large network of consumer machines, which makes it much cheaper but means a node can go away at any time. With Surfra handling the queue, the placement, and the retries, that stopped mattering.
Hirebase moved its workload off AWS onto Salad through Surfra and got comparable performance at a much lower cost. Hirebase estimates that running its current volume on AWS would cost tens of thousands of dollars a month. On Salad through Surfra it pays a fraction of that.
Failures got boring
Scrapes basically never die now. When a rare one does, figuring out why is easy. Every log line carries a timestamp and the request it belongs to, and Surfra writes its own lines saying where work was placed and why a replica stopped. A failure is a short read instead of an investigation.
Scaling from millions of pages a month to billions.
Hirebase