Use case · Batch

Batch work that finishes, then stops costing money.

Embeddings, transcription, captioning, nightly reports. Queue the items, let replicas come up across your clouds, and watch them scale back to zero when the batch is done.

Request access Read the docs

  items 1..1,000,000
     │  map(concurrency=16)
     ▼
   queue ──▶ embedder x8 ──▶ done
                         └─▶ scale to zero

The shape of it

A batch that stays up is just a server.

Embeddings, transcription, captioning, nightly reports. There is a pile of items, each one is independent, and when the pile is done the GPUs should go away. A server left running after the corpus is embedded is how a batch turns into a standing bill. Surfra starts replicas when the queue has work and stops them after the idle timeout, so the cost follows the batch instead of the calendar.

Load the model once

The Batch example keeps the weights in a warmup, then takes up to eight queued prompts in one forward pass. If fewer are waiting, it waits 300 milliseconds for more before it starts. That is the difference between a GPU doing one prompt at a time and a GPU that stays busy.

app.pyBatch example

import surfra
from transformers import pipeline

generator = None

@surfra.warmup
def load():
    global generator
    generator = pipeline("text-generation", model="Qwen/Qwen3-0.6B", device=0)

@surfra.handler(batch_size=surfra.batch_size(1, 8), batch_wait=300)
def generate(requests: list[surfra.Request]):
    prompts = [r.input["prompt"] for r in requests]
    outputs = generator(prompts, max_new_tokens=64, batch_size=len(prompts))
    return [{"text": out[0]["generated_text"]} for out in outputs]

Deploy it on the cheapest GPU first, with a faster card behind it, up to four replicas. Then enqueue one call per item. Results come back as each one finishes, in this case three haikus. A million documents is the same loop with a longer list. When the last item returns and the idle timeout passes, the replicas stop and the cloud stops billing them.

deploy.pyBatch example

import surfra

app = surfra.App("qwen-batch")
app.deploy(
    image="ghcr.io/you/qwen-batch:1",
    entrypoint="app.py",
    runtime={"engine": "python"},
    autoscale={
        "max_replicas": 4,
        "providers": [
            {"provider": "salad", "gpu": "rtx3090", "priority": 1},
            {"provider": "runpod", "gpu": "rtx4090", "priority": 2},
        ],
    },
)

calls = [app.enqueue({"prompt": f"Write a haiku about {topic}"}) for topic in ["rain", "tea", "trains"]]
for call in calls:
    print(call.result(timeout=900))

A corpus, not three haikus

map() is the version you want for a real pile. It calls an app you already deployed, once per item, keeps a cap on how many calls are waiting, and yields the outputs in order. A job can hold a key that works only for the apps you name, and only while the job runs. When the job exits, the key stops working. The snippet is from the Jobs guide.

map.pyJobs guide

import surfra

for vector in surfra.SurfraClient().map("embedder", ({"text": t} for t in texts), concurrency=16):
    ...

A lot of batches are two steps pretending to be one. A model writes something, and a cheaper step cleans it up or writes it out. A workflow passes one app's output to the next, and the GPU app and the CPU app scale on their own, so you are not paying for a GPU to rename files.

Each cloud bills you for the machines Surfra started in your account, at that cloud's price. There is no markup. A batch that runs for two hours on eight machines costs those eight machines for about two hours, plus startup, and then nothing. Put the cheaper cloud first and a faster one behind it when the batch has a deadline and the cheap pool runs out.

Keep a live chat app off this queue. A million items will occupy every replica, and someone waiting on a response waits behind them. Serve people on an inference app, even if both load the same weights.

Keep going.

FAQ

Common questions

Can a batch job call a deployed model?

Yes. Allow the job to call that app and Surfra gives it a key that works only for the apps you listed, only while the job runs.

Run it on your own clouds.

Surfra is invite only while we test it. Tell us what you want to run and we will send an invite.

Request access