> ## Documentation Index > Fetch the complete documentation index at: https://fireworks.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Overview > The Fireworks Training API lets you write custom Python training loops with full control over objectives while Fireworks manages distributed GPU infrastructure. > [!NOTE] > > Serverless Training is generally available. Graduate to Dedicated Training for larger workloads, more model selection, or full-parameter training. [Request access here](https://fireworks.ai/contact-training). > [!TIP] > > **Using a coding agent?** Install the [Fireworks training skill](https://fireworks.ai/docs/fine-tuning/agent/use-with-coding-agents.md) to help configure, run, and troubleshoot training jobs using current Fireworks best practices. ## What is the Training API? Fireworks Training API lets you write training logic in plain Python on your local machine while model computation runs on remote GPUs managed by Fireworks. Most users should start from [Cookbook recipes](https://fireworks.ai/docs/fine-tuning/training-api/cookbook/overview.md), the recommended entry point for standard SFT, DPO, GRPO-style training, and experimental async RL loops for agentic RL. Recipes use the Python SDK and can be run directly or through your agent. Use the Python SDK directly when you need full control over Training API behavior. | Starting point | Best for | How you use it | | - | - | - | | **Cookbook recipe** | Adapting a working SFT, DPO, GRPO-style, or experimental async RL loop | Run it with the Python SDK directly or through your agent | | **Python SDK** | Full control over training behavior | Write the training flow in Python while Fireworks runs GPUs | ## Choose infrastructure After choosing the Training API, decide how compute is provided: * [**Serverless Training**](https://fireworks.ai/docs/fine-tuning/training-api/serverless.md): shared pooled trainer for LoRA SFT, DPO, ORPO, RL, distillation, and custom loops on supported models, with no provisioning and per-token billing. * [**Dedicated Training**](https://fireworks.ai/docs/fine-tuning/training-api/dedicated.md): provisioned trainer and deployment resources, broader model and parameter-mode support, explicit checkpoint/resume/deployment control. > [card] [Serverless Training](https://fireworks.ai/docs/fine-tuning/training-api/serverless.md) > > Attach to a shared pooled trainer. There is no trainer or rollout deployment to provision. > [card] [Dedicated Training](https://fireworks.ai/docs/fine-tuning/training-api/dedicated.md) > > Provision trainer and deployment resources for your run, with broader model and parameter-mode support. ### Quick decision **Start with Serverless Training** The model is supported, LoRA covers the task, and you want pooled compute with per-token billing. **Choose Dedicated Training** You need full-parameter training, a model outside the serverless pool, explicit resume or deployment control, or sustained provisioned compute. ### Comparison | Dimension | Serverless | Dedicated | | - | - | - | | Provisioning | Shared pooled trainer; no trainer or sampler deployment creation | SDK provisions trainer and deployment resources | | Billing | Per token; no idle GPU charge | Time-based trainer and deployment billing | | Parameter mode | LoRA only | LoRA and full-parameter | | Methods | SFT, DPO, ORPO, RL, distillation, and custom loops | SFT, DPO, ORPO, RL, distillation, and custom loops | | Models | Current serverless model list | Models with an enabled dedicated training shape | | Capacity | Shared pool and per-account limits | Resources allocated to the run, subject to account quota and platform availability | | Checkpoint resume | In-run snapshots; cross-run resume is limited | Explicit checkpoint, reconnect, promotion, and deployment lifecycle | | Sampling | Sampler on the shared pool, no deployment to create; no training client needed | SDK-managed rollout or evaluation deployment | | Teardown | Session lifecycle is managed by the service | You must close trainers and delete or scale down deployments | | Best fit | LoRA workloads on supported models without provisioning | Full-parameter work, models outside the serverless pool, sustained workloads, and explicit lifecycle control | Always verify current models, limits, prices, and feature status in the [Serverless Training](https://fireworks.ai/docs/fine-tuning/training-api/serverless.md) and [Dedicated Training](https://fireworks.ai/docs/fine-tuning/training-api/dedicated.md) pages before launch. ## Who does what | Fireworks handles | Cookbook recipes handle | Python SDK users implement | | - | - | - | | GPU provisioning and cluster management | Training loop structure for supported recipes | Training loop logic (`forward_backward_custom` + `optim_step`) | | Service-mode trainer lifecycle (create, health-check, reconnect, delete) | Resource setup, health checks, reconnect, and cleanup | Managed service setup with `FiretitanServiceClient.from_firetitan_config(...)` | | Distributed forward pass, backward pass, optimizer execution | Common losses and reward/evaluation plumbing | Loss function and batch construction | | Checkpoint storage and export | Checkpoint save, resume, promotion, and sampler refresh | Checkpoint calls (`save_weights_for_sampler`, DCP snapshots) | | Inference deployments and weight sync | Deployment sampling and serving-integrated evaluation for RL recipes | Custom rollout, sampling, and evaluation logic through the managed service | | Preemption recovery and job resume | Resume logic for supported recipe checkpoints | Resume policy and state restoration calls | | Distributed training (multi-node, sharding, FSDP) | Config surfaces for learning rate, grad accumulation, context length, W\&B | Hyperparameter schedules, data pipeline, and experiment tracking | ## System architecture 1 · Your laptop **Python loop** Loads data, builds batches, computes rewards, and controls the experiment. 2 · Fireworks API **Control plane** Authenticates requests, routes operations, and manages the selected infrastructure. 3 · Remote compute **GPU trainer** Runs forward passes, backward passes, and optimizer steps. 4 · Separate compute **Sampling** Serves saved weights for rollouts and evaluation outside the trainer. 5 · Persistent output **Artifacts** Stores sampler snapshots and resumable training state for later use. Your Python process stays on your laptop throughout the run. It sends model operations to Fireworks and receives metrics or completions back. Sampling uses separate inference infrastructure rather than the trainer itself. ## How service-mode training works > [!WARNING] > > **Most common gotchas** > > * Remote operations such as `forward`, `forward_backward`, `optim_step`, sampling, and checkpoint saves return future-like results. Call `.result()` on operations that return one. > * `token_weights=0` means prompt/no-loss tokens, `token_weights=1` means response/learned tokens. > * `forward_backward_custom` computes gradients only; you still need `optim_step` to apply updates. ### Minimal training step lifecycle The shape of the loop is the same on both infrastructures. Toggle between them to see what changes at each stage: > [card] Training infrastructure Runs on Fireworks GPUs Train and sample repeat every training step > [card] > [card] `forward_backward()` runs a built-in trainer-side loss and is what most loops use. `forward_backward_custom()` is only for a custom Python loss function. A trained LoRA deploys to an on-demand deployment on either path. Serverless per-token serving of your own adapter is not available. Full walkthrough: ### Datums A **Datum** is the unit of training data sent to the remote GPU. It wraps tokenized input and per-token weights that your loss function needs. For SFT, token weight `0.0` marks prompt tokens and `1.0` marks response tokens. Cookbook renderers construct these weights from chat messages. ### Logprobs and forward\_backward\_custom When you call `forward_backward_custom`, the GPU runs a forward pass and returns **per-token log-probabilities** as PyTorch tensors with `requires_grad=True`. Your loss function computes a scalar loss, the API calls `loss.backward()`, and gradients are sent back to the GPU for the model backward pass. After accumulating gradients, call `optim_step` to apply the update. See the [Dedicated Training Quickstart](https://fireworks.ai/docs/fine-tuning/training-api/dedicated.md#quickstart) for one complete runnable Datum, loss, and optimizer loop. ### Futures Remote training operations such as `forward`, `forward_backward`, `optim_step`, and checkpoint saves return **future-like results**. Call `.result()` on operations that return one so failures surface. ### Checkpointing and weight sync After training, you export checkpoints for serving: * **Base snapshot:** a complete chain anchor for the trainable state. For LoRA this is the adapter; for full-parameter training it is model weights. * **Delta snapshot:** a change relative to a prior full-parameter base snapshot. The SDK selects base versus delta automatically unless the recipe overrides it. Checkpoint-to-sampler behavior depends on the infrastructure: * **Serverless:** save a snapshot and bind a sampling client to that snapshot, from the training process or any other. There is no deployment weight sync. See [Serverless Training](https://fireworks.ai/docs/fine-tuning/training-api/serverless.md). * **Dedicated:** save a snapshot and refresh an SDK-managed deployment sampler, which syncs weights onto the deployment. See [Dedicated Training and Sampling](https://fireworks.ai/docs/fine-tuning/training-api/dedicated.md#training-and-sampling). For dedicated RL rollouts that continue across weight sync, see [KV cache behavior for RL rollouts](https://fireworks.ai/docs/guides/rollout-inference.md#kv-cache-behavior-for-rl-rollouts). ### Dedicated RL rollout transition mode When a dedicated RL recipe provisions a hot-load rollout deployment, you can set `hot_load_transition_type` to `ASYNC` or `SYNC` in the SDK provisioning config or the cookbook rollout deployment config. Leave it unset to keep the recommended `ASYNC` default; set `SYNC` when a rollout must not span a weight transition. For the tradeoffs and KV-cache behavior, see [Async transition (recommended, default for RL)](https://fireworks.ai/docs/fine-tuning/rl-rollout-integration.md#async-transition-recommended-default-for-rl). ## Key APIs | API | Purpose | | - | - | | [`FiretitanServiceClient`](https://fireworks.ai/docs/fine-tuning/training-api/reference/service-client) | Recommended direct SDK entry point. Creates or reattaches trainers/deployments and returns training, reference, and sampling clients. | | [`FiretitanTrainingClient`](https://fireworks.ai/docs/fine-tuning/training-api/reference/service-client) | Tinker-compatible training client: `forward_backward_custom`, `optim_step`, `save_weights_for_sampler`, `save_state`, and load methods. | | [`DeploymentSampler`](https://fireworks.ai/docs/fine-tuning/training-api/reference/deployment-sampler) | FireTitan-native sampler for tokenized rollout/evaluation from SDK-managed deployments. | | [`FireworksClient`](https://fireworks.ai/docs/fine-tuning/training-api/reference/fireworks-client) | Standalone checkpoint operations such as listing checkpoints or promoting a model without a live training instance. | | [`TrainerJobManager`](https://fireworks.ai/docs/fine-tuning/training-api/reference/trainer-job-manager) | Legacy/compatibility lifecycle manager. Documented for existing SDK users and advanced debugging; not the recommended user-facing path. | | [`DeploymentManager`](https://fireworks.ai/docs/fine-tuning/training-api/reference/deployment-manager) | Legacy/compatibility deployment manager. Documented for existing SDK users and advanced debugging; normal code uses `FiretitanServiceClient`. | ## Renderers Chat-template formatting, stop-token handling, and loss-weight masking for SFT/DPO datasets are handled by **renderers** — pluggable per-model classes that turn raw conversations into the trainer's `Datum` shape. Most users never touch a renderer directly; cookbook recipes pick the right one for the `base_model` you set. If you need to author a new one or debug parity against HuggingFace, use the canonical training skill's [renderer implementation reference](https://github.com/fw-ai-external/cookbook/blob/main/skills/fireworks-training/references/renderer.md) and [verification reference](https://github.com/fw-ai-external/cookbook/blob/main/skills/fireworks-training/references/renderer-verification.md). ## Comparing Training API pricing vs DIY bare metal When comparing a managed training platform with a self-managed bare-metal stack, optimize for **cost per successful iteration**, not just headline `$ / GPU-hour`. ### What to compare * **Time to first deployed model**: include environment setup, training orchestration, checkpoint handoff, and serving integration. * **Iteration cycle time** (`train -> eval -> deploy -> repeat`): include all retrain/redeploy plumbing, not just GPU runtime. * **Infra engineering overhead**: include one-time setup and recurring maintenance for containers, runtimes, deployment workflows, and compatibility fixes. * **Effective `$ / GPU-hour` at real utilization**: include idle capacity, reservation constraints, and burst/overflow behavior. * **Train/serve parity risk**: account for potential quality drift when training and inference runtimes diverge. * **Parallel experiment capacity**: compare fixed-reservation throughput against elastic capacity for sweeps and multi-seed runs. ### Useful formulas ```text theme={null} iterations_per_month = available_working_days / cycle_time_days effective_cost_per_gpu_hour = total_monthly_spend / gpu_hours_consumed multi_turn_success ~= (single_turn_success)^turn_count ``` ### Keep assumptions explicit Document assumptions so readers can adjust them for their own workload: * team size and fully-loaded engineering cost * average cycle duration in each setup * expected utilization and burst profile * average turn count for production agent workflows * required concurrent experiment count ## FAQ ### Why is my training run "doing nothing" even though code executed? Usually because `.result()` was not called on futures, so failures were never surfaced. ### What's the difference between base and delta checkpoints, and when should I use each? Let the SDK select automatically. LoRA snapshots contain the full adapter; full-parameter delta snapshots can accelerate synchronization but are not promotable. See [Saving and Loading](https://fireworks.ai/docs/fine-tuning/training-api/dedicated.md#saving-and-loading). ### Do I need to manage distributed training infra? No. You implement training logic while Fireworks manages GPU provisioning and distributed infrastructure. ### Should I start with a Cookbook recipe or the Python SDK? Start with a Cookbook recipe for most SFT, DPO, or GRPO adaptations. Use the Python SDK directly when you need custom loop semantics and full control. ### Can I evaluate serving behavior during training? Yes. On serverless, save a snapshot and sample from it in the same session. On dedicated infrastructure, sync a snapshot to the SDK-managed deployment sampler and evaluate there. ### How should I compare Training API pricing vs a DIY bare-metal setup? Use the framework in [Comparing Training API pricing vs DIY bare metal](#comparing-training-api-pricing-vs-diy-bare-metal). Focus on total iteration economics (cycle time, engineering overhead, utilization-adjusted cost, and quality-parity risk), then plug in your own assumptions. ### How can I compare rollout cost vs other providers? See the [Price comparison vs Tinker](https://fireworks.ai/docs/fine-tuning/multi-turn-cost-comparison.md) calculator to estimate scenario-based costs on Fireworks Dedicated against Tinker's per-token pricing. ## Next steps * [Dedicated quickstart](https://fireworks.ai/docs/fine-tuning/training-api/dedicated.md#quickstart) — run a minimal dedicated custom loop * [Choose infrastructure](https://fireworks.ai/docs/fine-tuning/training-api/introduction.md#infrastructure) — compare serverless and dedicated training * [Serverless Training](https://fireworks.ai/docs/fine-tuning/training-api/serverless.md) — shared pooled LoRA training * [Dedicated Training](https://fireworks.ai/docs/fine-tuning/training-api/dedicated.md) — provisioned trainer and deployment lifecycle * [Dedicated Training and Sampling](https://fireworks.ai/docs/fine-tuning/training-api/dedicated.md#training-and-sampling) — deployment-sampling lifecycle * [Loss Functions](https://fireworks.ai/docs/fine-tuning/training-api/dedicated.md#loss-functions) — built-in and custom loss functions * [Vision Inputs](https://fireworks.ai/docs/fine-tuning/models.md) — train vision-language models with image and text data * [The Cookbook](https://fireworks.ai/docs/fine-tuning/training-api/cookbook/overview.md) — ready-to-run recipes for SFT, DPO, ORPO, GRPO/IGPO, and async RL (experimental) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.