Surfrasurfra
Use case · Web scraping

Scrape billions of pages without babysitting machines.

Big crawls fail in boring ways: stuck workers, silent hangs, and a bill that grows faster than the page count. Surfra queues every page, spreads workers across the cheapest clouds you connect, and tells you exactly why a request failed.

Request access Read the docs
The shape of it

Every page is a request.

A crawl at tens of millions of pages can live on a couple of virtual machines. Past that, the failures stop looking like bugs in the scraper. A worker hangs with no error and keeps the machine. A cheap node disappears halfway through a page. The queue looks healthy while a slice of URLs never comes back.

Surfra treats every URL as one request in one queue. Replicas send heartbeats. A replica that stops reporting is replaced, and the request it was holding goes back on the queue for another worker. Salad machines are other people's computers, so a node going away is normal. You are not SSH-ing across a fleet to find the stuck one.

The worker is a handler

Surfra is not a crawler. It does not discover links, store pages, or decide what robots.txt allows. You bring that code. The Hello example is the shape of the worker: load once, then answer one request. Put your fetch where the greeting is.

app.pyHello example
import surfra

greeting = None

@surfra.warmup
def load():
    global greeting
    greeting = "hello"

@surfra.handler
def predict(request: surfra.Request):
    return {"text": f"{greeting} {request.input.get('name', 'world')}"}

Deploy that on a CPU. Two cores and 4 GB on Salad is the example. Put a second cloud behind it for when Salad is short, or a machine you already own. Salad, RunPod, Vast.ai, and DigitalOcean all have CPU machines. Yotta does not. Each cloud bills you at its own price, and Surfra adds no markup.

deploy.pyHello example
import surfra

app = surfra.App("hello")
app.deploy(
    image="ghcr.io/you/hello:1",
    entrypoint="app.py",
    runtime={"engine": "python"},
    autoscale={
        "max_replicas": 1,
        "providers": [{"provider": "salad", "gpu": "cpu", "vcpu": 2, "ram_gb": 4, "priority": 1}],
    },
)

print(app.run({"name": "Surfra"}, timeout=600))

Then send the URLs

Each log line carries a timestamp and, when Surfra can tell, the request it belongs to, so one failed page has one short trail. App logs are kept for 24 hours. To push a list through the worker, enqueue one call per item and collect the results. The calling guide shows it with prompts. A URL is the same call.

fanout.pyCall an app
calls = [app.enqueue({"prompt": p}) for p in prompts]
results = [call.result(timeout=900) for call in calls]

Fetch on a CPU. Read the page on a GPU

Fetching HTML should not hold a 4090 while a site is slow. Split them into two apps. A workflow runs the first, passes its output to the second, and each one scales on its own. The docs example is a transcript and then a summary. A crawl is a fetch and then an extract, with the same two stages.

pipeline.pyWorkflows
import surfra

@surfra.stage(app="transcribe")
def transcribe(data):
    ...

@surfra.stage(app="summarize", depends=("transcribe",))
def summarize(data):
    ...

surfra.Workflow("meeting-notes").deploy()

This is the shape Hirebase runs. It went from tens of millions of pages a month on a couple of AWS VMs to over a billion on Salad through Surfra, at a fraction of what it estimates the same volume would cost on AWS. Read how that move went.

Related

Keep going.

FAQ

Common questions

Can Surfra run CPU-only scraping workers?

Yes. Salad, RunPod, Vast.ai, and DigitalOcean all offer CPU machines on Surfra, and so do your own registered machines.

How do retries work for scraping?

If the worker running a request fails, the request goes back to the queue and is retried on another worker.

Point your crawler at the queue.

Surfra is invite only while we test it. Tell us what you want to run and we will send an invite.

Request access