Surfrasurfra
Use case · Training

Fine-tunes and training runs, on whichever GPU is free.

Try your own workstation first and fall back to a cloud GPU. Surfra starts the run, streams the logs, stops it at your deadline with time to checkpoint, and releases the machine when it ends.

Request access Read the docs
The shape of it

A training run should happen once.

A fine-tune does not belong on the request queue. It runs for hours, it should happen once, and retrying the whole command is a second bill for the same experiment. It can also overwrite the checkpoint you just wrote. Surfra Jobs run one command, once, on one machine, and finish when the command exits. They are not retried. If you want another attempt, you submit one.

Register the workstation you already own and put it first. If it is busy, the next row can be a cloud GPU, an A100 or an H100 on the account you pick. The same spec runs in both places. A 4090 is enough for a small adapter on a 7B or 8B model. An A100 80GB or an H100 is what you want when the weights, the optimizer state, and the batch no longer fit. Surfra will not move a job onto a larger card halfway through, so the list you submit is the decision.

Save the spec, then change the flags

The Jobs guide keeps the image, the machine, the env file, and a 12 hour deadline under a name. The next submit only adds the learning rate. Words after -- are appended to the spec's command.

shellJobs guide
surfra job spec save train-llm --image ghcr.io/acme/train:3 --runner desk-4090 \
  --env-file train.env --deadline 12h -- python train.py
surfra job submit --spec train-llm -- --lr 3e-4

From Python it is the same run, and you can follow the output until the command exits. Standard output and errors also show up in the dashboard. Job logs are kept for 7 days, longer than the 24 hours an app's logs are kept, because you usually read a training log after the run is over.

train.pyJobs guide
import surfra

client = surfra.SurfraClient()
job = client.submit_job(
    "bash train.sh",
    args=["--epochs", "10"],
    image="ghcr.io/acme/train:3",
    providers=[{"provider": "runner", "runner": "desk-4090"}],
)
for line in job.logs(follow=True):
    print(line["m"])
print(job.wait()["status"])

The spec can be saved from Python too, with save_job_spec, and a later submit_job passes only the new arguments. That form is in the same guide.

Stopping without losing the checkpoint

At the deadline, or when you stop the job, the command gets SIGTERM and then a grace period, 30 seconds unless you change it, before SIGKILL. Write the checkpoint on SIGTERM, to storage that is not the machine. Spot GPUs, where the cloud offers them, can be taken back. Surfra warns you when you submit onto a spot row. Because the job is not retried, a checkpoint outside the box is the only way that run continues. If the machine is lost, Surfra releases it so it stops costing money, and the command does not start again on its own.

Related

Keep going.

FAQ

Common questions

Can I train on spot GPUs?

Yes, where the cloud offers them. Spot machines can be taken back, and jobs are not retried, so save checkpoints outside the machine.

Where do training logs go?

Standard output and errors stream to the command line and the dashboard, and are kept for 7 days.

Run it on your own clouds.

Surfra is invite only while we test it. Tell us what you want to run and we will send an invite.

Request access