1. Package a small, self-contained code ZIP

Place train.py at the ZIP root. Include your model modules and configuration, plus small local test data if needed. The ZIP limit is 16 MiB, expanding to at most 64 MiB.

train.py
model.py
config.yaml
small-test-data.jsonl  # optional; excluded from the code archive

The standard environment supplies PyTorch and common libraries. A requirements file documents dependencies; it is not installed automatically inside the job. Additional dependencies require a separately prepared environment.

Never bundle keys, passwords, virtual environments, caches, large datasets, or downloaded model weights. Importing train.py must only define functions/classes and lightweight configuration: do not start training, read the dataset, download files, or allocate the model at import time.

2. Implement the automatic batching contract

Declare a literal global effective batch size: the number of examples or sequences per optimizer update across all GPUs. RunFab chooses microbatch and gradient accumulation to preserve it.

EFFECTIVE_BATCH_SIZE = 32

def probe_batch(microbatch):
    # Construct the actual model and optimizer here.
    # Test representative worst-case inputs and optimizer steps.
    # Implement this hook; an empty probe cannot measure memory.
    raise NotImplementedError("Implement a real memory probe")

def train(microbatch, accumulation):
    # Construct a fresh model and train with these selected values.
    raise NotImplementedError("Implement your training loop")

This shows the required interface, not a complete trainer. The project includes a single-GPU example at examples/streaming-lm/train.py.

microbatch × accumulation × GPU count = EFFECTIVE_BATCH_SIZE. The effective batch must be an integer from 1 to 65536 and divisible by the GPU count. Do not overwrite the selected values with fixed batch settings.

3. Make the memory probe representative

Use the same architecture, maximum sequence length or image size, precision, and activation-checkpointing settings as training. Exercise forward, backward, and optimizer steps so lazy optimizer state is allocated. Test another iteration with that state present.

Each candidate runs in a fresh subprocess, with a 120-second limit. Probe every assigned GPU. Do not consume stdin, write checkpoints, emit training metrics, or leave child processes running. Let CUDA out-of-memory errors propagate so RunFab can reject the batch; do not catch them and return success.

RunFab stops at the first batch that fails to fit and uses the largest tested fit. Later OOMs still fail the run; there is no automatic checkpoint recovery. Custom CUDA allocators and spawned distributed processes must apply their own matching memory limits.

4. Stream data and keep memory bounded

Read records as an iterator, tokenize a bounded batch, train, then discard it. Use bounded packing/shuffle buffers and explicit sequence-length limits. Never materialize the entire stream with list(...) or build a full-dataset token cache.

import json
import sys

for line in sys.stdin:
    record = json.loads(line)
    # Validate, encode, and add to a bounded batch.
    # Train and release each batch as it becomes ready.

Attach streaming data with your code. A root-level runfabric.dataset.json selects a Hugging Face YOLO ZIP repository, pinned revision, split and number of passes. Upload and preview attach it automatically. The worker reads remote ZIP ranges and streams JSON records with image_b64, labels (class/cx/cy/w/h), boxes (cx/cy/w/h), class_ids, id, split and epoch into stdin. Each record is at most 1 MiB; no dataset archive is saved. Nodes must support this adapter and approve huggingface.co. Stdin reaches EOF when no source is configured. Advanced submissions can also use an approved HTTPS NDJSON source. For small local inputs, open a file from your uploaded bundle.

Training containers have no general network access. Dataset download calls, remote pretrained-model downloads, and dependency installation will not work. A configuration flag named “streaming” alone does not supply network access.

5. Keep the training loop correct

For equally weighted, equal-sized microbatches, divide each loss by accumulation before backpropagation. Step the optimizer and zero gradients only after the selected number of microbatches. Variable-length or masked objectives need normalization by the correct valid-token/example count instead.

# Illustration: batches must contain equal numbers of examples.
optimizer.zero_grad(set_to_none=True)
for index, batch in enumerate(batches, start=1):
    loss = compute_loss(model, batch)
    (loss / accumulation).backward()
    if index % accumulation == 0:
        optimizer.step()
        optimizer.zero_grad(set_to_none=True)

Define what happens to the final incomplete update: discard it explicitly or normalize it correctly. Avoid retaining GPU tensors in metric lists; record scalar values. Bound data-loader workers and prefetching to fit 2 CPU cores and 8 GiB RAM. Scratch and output also consume this RAM budget.

Set and record seeds, architecture settings, tokenizer settings, precision, optimizer parameters, and data revision where available. A seed alone does not guarantee reproducibility across different GPUs.

6. Report loss and throughput for the graphs

Print one JSON record per line to stdout with flush=True. Report about every 5–10 seconds and aggregate loss consistently over the reporting interval.

import json

print(json.dumps({
    "step": 100,
    "elapsed_seconds": 52.4,
    "loss": 2.31,
    "tokens_per_second": 1840.5,
}), flush=True)

Replace these example numbers with measurements. Throughput is completed work divided by wall-clock seconds over the interval. Synchronize GPU work when measuring asynchronous execution. You can report samples_per_second or steps_per_second instead of, or alongside, token throughput.

Elapsed time starts when actual training begins, excluding probes. Use finite numeric values and nonnegative throughput. The GUI refreshes every 5 seconds and keeps the latest 5,000 records. Without elapsed time, it plots coordinator receipt time. Keep each line below 8,192 characters; excessive logging can drop records.

7. Separate final models from disposable files

/workspace/           # read-only uploaded code
/tmp/                 # temporary cache, up to 256 MiB
/output/
  model/              # FINAL deliverables: retained
    model.safetensors
    config.json
    tokenizer.json
  checkpoints/        # intermediate snapshots: disposable
  data/               # temporary data: disposable

Put everything needed for inference in /output/model/, including architecture configuration and tokenizer files. Use RUNFABRIC_OUTPUT as the output root. Keep intermediate checkpoints and data outside the model folder. Prefer tensor-only serialization such as safetensors where suitable; do not load untrusted pickle files.

All output together must fit within 3 GiB; it uses RAM-backed storage. Choose a task-appropriate stopping condition, write a complete final model, and exit successfully. Cancellation, forced stops, and host failures do not provide checkpoint-aware recovery.

A top-level model.safetensors, model.pt, model.pth, or model.bin is also recognized, but use the model directory when additional files are needed.

8. Understand saving and cleanup

After completed output is transferred, RunFab automatically saves the final model to D:\RunFabric\<job-id>. It verifies the saved files before deleting the original output archive, intermediate checkpoints/data, and logs.

The archive keeps the final model, training-code.zip, graphs.json, run.json, and a checksum manifest. Run history and graphs also stay in the GUI. Code preservation includes common source/config files; it does not retain every file in the input ZIP. Keep essential inference assets inside the final model folder.

Shared input ZIPs remain while another unarchived job needs them. Original datasets outside RunFab are never deleted. If no final model is found, a path is unsafe, or saving/verification fails, the original output is retained and the run shows an error with a retry option.

9. Add multiple GPUs only when your code supports them

All GPUs for a run come from the same node. Your trainer must implement data parallelism, input sharding, gradient synchronization, device placement, and orderly shutdown. Moving the slider does not add these features to single-GPU code.

The automatic tuner assumes ordinary data parallel training and picks one microbatch that fits every selected GPU. It does not pool VRAM or support tensor/pipeline-parallel batch semantics. Mixed GPUs can bottleneck on the slowest device. Use local CUDA indices for the GPUs exposed to the job, not host GPU IDs.

Report global metrics once from the main rank, and save the final model once. Independently spawned processes need matching allocator limits. Start with one GPU, verify a complete small run, then test multiple GPUs.

Before uploading

• train.py is at the ZIP root, imports cleanly, and defines both real hooks.
• The probe covers worst-case training memory and does not consume training data.
• The trainer honors microbatch/accumulation, streams inputs, and handles EOF clearly.
• Metrics contain real measurements with explicit throughput units.
• Final inference files are under /output/model/; checkpoints/data are outside it.
• A short smoke run completes within RAM, scratch, output, and time limits.

For the detailed project contract, see docs/ADAPTERS.md in the repository. RunFab checks the adapter structure, but cannot prove that custom training code implements its semantics correctly.