> ## Documentation Index > Fetch the complete documentation index at: https://fireworks.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Deployments > Create and manage on-demand Fireworks deployments on dedicated GPUs: choose accelerator shapes, autoscaling, replicas, and lifecycle controls for production. > [!NOTE] > > **New to deployments?** Start with our [Deployments Quickstart](https://fireworks.ai/docs/getting-started/ondemand-quickstart.md) to deploy and query your first model in minutes, then return here to learn about configuration options. On-demand deployments give you dedicated GPUs for your models, providing several advantages over serverless: * **Better performance** – Lower latency, higher throughput, and predictable performance unaffected by other users * **No hard rate limits** – Only limited by your deployment's capacity * **Cost-effective at scale** – Cheaper under high utilization. Unlike serverless models (billed per token), on-demand deployments are [billed by GPU-second](https://fireworks.ai/pricing). * **Broader model selection** – Access models not available on serverless * **Custom models** – Upload your own models (for supported architectures) from Hugging Face or elsewhere Need higher GPU quotas or want to reserve capacity? [Contact us](https://fireworks.ai/contact). ## Creating & querying deployments **Create a deployment:** ```bash theme={null} # This command returns your accounts//deployments/ - save it for querying firectl deployment create accounts/fireworks/models/ \ --deployment-shape default \ --wait ``` `--deployment-shape default` has Fireworks pick a validated shape for the model — see [Letting Fireworks pick a shape](#letting-fireworks-pick-a-shape). To choose a shape yourself, run `firectl deployment-shape-version match --model ` first and pass a shape's name to `--deployment-shape`. > [!WARNING] > > **Deployment placement (`--region`) must be set at creation time and cannot be changed in place.** > > If you do not specify `--region`, the deployment is pinned to a single datacenter at creation time and will not be automatically migrated later. > > For production workloads that need geographic availability or capacity failover, always set `--region` explicitly: > > ```bash theme={null} > firectl deployment create accounts/fireworks/models/ \ > --deployment-shape default \ > --region GLOBAL # recommended default > firectl deployment create accounts/fireworks/models/ \ > --deployment-shape default \ > --region US > firectl deployment create accounts/fireworks/models/ \ > --deployment-shape default \ > --region CANADA > firectl deployment create accounts/fireworks/models/ \ > --deployment-shape default \ > --region EUROPE > firectl deployment create accounts/fireworks/models/ \ > --deployment-shape default \ > --region APAC > ``` > > Only **GLOBAL** is available by default. Quota for **US**, **CANADA**, **EUROPE**, **APAC**, and single regions must be granted by Fireworks — [contact us](https://fireworks.ai/contact). Deploying with a region you have no quota for is rejected at creation. See [Regions](https://fireworks.ai/docs/deployments/regions.md#quotas) for details. ### Check current placement ```bash theme={null} firectl deployment get ``` The deployment metadata shows where the deployment is currently allowed to schedule replicas (placement / region configuration). ### Change placement There is no supported command to change region placement on an existing deployment. To change placement, recreate the deployment: ```bash theme={null} # 1. Create replacement with correct region firectl deployment create accounts/fireworks/models/ \ --deployment-shape default \ --region GLOBAL \ --min-replica-count 1 # 2. Verify it's healthy, then point your app at the new endpoint # 3. Delete old deployment firectl deployment delete ``` See [Regions](https://fireworks.ai/docs/deployments/regions.md) for mega-regions and hardware availability. See [Deployment shapes](#deployment-shapes) below to optimize for speed, throughput, or cost. **Query your deployment:** After creating a deployment, query it using this format: ``` accounts//deployments/ ``` > [!TIP] > > You can find your deployment name anytime with `firectl deployment list` and `firectl deployment get `. **Example:** ``` accounts/alice/deployments/12345678 ``` ### Code examples > [tab] Python (Fireworks SDK) > > ```python theme={null} > from fireworks import Fireworks > > client = Fireworks() > > response = client.chat.completions.create( > model="accounts/fireworks/models/gpt-oss-120b#", > messages=[{"role": "user", "content": "Explain quantum computing in simple terms"}] > ) > > print(response.choices[0].message.content) > ``` > [tab] Python (OpenAI SDK) > > ```python theme={null} > import os > from openai import OpenAI > > client = OpenAI( > api_key=os.environ.get("FIREWORKS_API_KEY"), > base_url="https://api.fireworks.ai/inference/v1" > ) > > response = client.chat.completions.create( > model="accounts//deployments/", > messages=[{"role": "user", "content": "Explain quantum computing in simple terms"}] > ) > > print(response.choices[0].message.content) > ``` > [tab] JavaScript > > ```javascript theme={null} > import OpenAI from "openai"; > > const client = new OpenAI({ > apiKey: process.env.FIREWORKS_API_KEY, > baseURL: "https://api.fireworks.ai/inference/v1", > }); > > const response = await client.chat.completions.create({ > model: "accounts//deployments/", > messages: [ > { > role: "user", > content: "Explain quantum computing in simple terms", > }, > ], > }); > > console.log(response.choices[0].message.content); > ``` > [tab] curl > > ```bash theme={null} > curl https://api.fireworks.ai/inference/v1/chat/completions \ > -H "Content-Type: application/json" \ > -H "Authorization: Bearer $FIREWORKS_API_KEY" \ > -d '{ > "model": "accounts//deployments/", > "messages": [ > { > "role": "user", > "content": "Explain quantum computing in simple terms" > } > ] > }' > ``` ### Deployment status states Deployment states from the Gateway API spec: * `CREATING` - still being created * `READY` - ready to be used * `UPDATING` - in-progress updates happening * `DELETING` - being deleted * `DELETED` - soft-deleted * `FAILED` - creation failed (see status for details) UI-only states are display labels derived from deployment fields: * `Inactive`: `state == READY && max_replica_count == 0 && ready_replica_count == 0` * `Scaled to 0`: `state == READY && min_replica_count == 0 && max_replica_count > 0 && desired_replica_count == 0 && ready_replica_count == 0` These are display labels computed from deployment fields; they are not new backend `Deployment.State` enum values. ## Deployment shapes Deployment shapes are the primary way to configure deployments. They're pre-configured templates optimized for speed, cost, or efficiency, including hardware, quantization, and other [performance factors](https://fireworks.ai/docs/deployments/benchmarking.md). Every shape is validated by Fireworks, so a deployment created from a shape uses a known-good configuration of GPU count, precision, and serving parameters. > [!WARNING] > > **Do not create deployments without a shape.** Always pass `--deployment-shape` — a concrete shape, or `default` to have Fireworks pick one. Deployments created without a shape skip these validation checks — mistakes like a GPU count that can't fit the model surface only at creation, where they cause failures. Most failed deployment creations on Fireworks are deployments without a shape. Enforcement is coming soon: shapeless creation will then require an explicit opt-in — see [Explicitly creating a deployment without a shape](#explicitly-creating-a-deployment-without-a-shape-advanced-users-only). If no shape fits your workload, [contact us](https://fireworks.ai/contact) and we'll help you find or add one. Over the API, `deploymentShape` also accepts the special value `default`: the server picks a validated shape for the model for you — pass `--deployment-shape default` with firectl. For details, see [Letting Fireworks pick a shape](#letting-fireworks-pick-a-shape). * **Fast** – Low latency for interactive workloads * **Throughput** – Cost-per-token at scale for high-volume workloads * **Minimal** – Lowest cost for testing or light workloads **Usage:** ```bash theme={null} # Let Fireworks pick a validated shape firectl deployment create accounts/fireworks/models/deepseek-v3 --deployment-shape default # Or match available shapes (the deployable set for your account) and choose one firectl deployment-shape-version match --model # Create with a shape (shorthand) firectl deployment create accounts/fireworks/models/deepseek-v3 --deployment-shape throughput ``` > [!NOTE] > > `match` reports the deployable set. Shapes your account cannot deploy on are dropped, exact base-model matches are preferred over same-model-type and parameter-count siblings, and LoRA addon and live-merge models are matched against their base model. Over the API, the equivalent is [Match Deployment Shape Versions](https://fireworks.ai/docs/api-reference/match-deployment-shape-versions.md). To see every validated shape matching a base model instead (including ones your account cannot use), use `firectl deployment-shape-version list --base-model ` — base models only, it fails on LoRA addon and live-merge models. `match` does not take addons into account. For a deployment with `--enable-addons`, use the [Match Deployment Shape Versions](https://fireworks.ai/docs/api-reference/match-deployment-shape-versions.md) API with `enableAddons` set. ```bash theme={null} # Create with full shape ID firectl deployment create accounts/fireworks/models/llama-v3p3-70b-instruct \ --deployment-shape accounts/fireworks/deploymentShapes/llama-v3p3-70b-instruct-fast # View shape details firectl deployment-shape-version get ``` ### Letting Fireworks pick a shape Every client surface accepts a special value that has the server pick a validated shape for the model and apply it, the same as passing that shape explicitly: > [tab] firectl > > ```bash theme={null} > firectl deployment create accounts/fireworks/models/gpt-oss-120b \ > --deployment-shape default > ``` > [tab] REST API > > Pass `"default"` as `deploymentShape`: > > ```bash theme={null} > curl -X POST "https://api.fireworks.ai/v1/accounts/YOUR_ACCOUNT_ID/deployments" \ > -H "Authorization: Bearer $FIREWORKS_API_KEY" \ > -H "Content-Type: application/json" \ > -d '{ > "baseModel": "accounts/fireworks/models/gpt-oss-120b", > "deploymentShape": "default" > }' > ``` > [tab] Python SDK > > ```python theme={null} > from fireworks import Fireworks > > client = Fireworks() > client.deployments.create( > account_id="YOUR_ACCOUNT_ID", > base_model="accounts/fireworks/models/gpt-oss-120b", > deployment_shape="default", > ) > ``` The pick never silently overrides fields you set. If every compatible shape conflicts with fields in your request (for example an `acceleratorType` no shape validates), the request fails with an error naming the conflicting fields, the compatible shapes, and how to create the deployment without a shape. If no shape is compatible with the model at all, the request also fails — in that case, [contact us](https://fireworks.ai/contact) and we'll help you find or add one. ### Explicitly creating a deployment without a shape (advanced users only) Every surface accepts an explicit opt-out that creates the deployment without a shape. Setting it together with a shape is an error — a deployment is either shaped or explicitly shapeless. > [tab] firectl > > ```bash theme={null} > firectl deployment create accounts/fireworks/models/gpt-oss-120b \ > --accept-shapeless-risk > ``` > [tab] REST API > > Pass `acceptShapelessRisk=true` as a query parameter: > > ```bash theme={null} > curl -X POST "https://api.fireworks.ai/v1/accounts/YOUR_ACCOUNT_ID/deployments?acceptShapelessRisk=true" \ > -H "Authorization: Bearer $FIREWORKS_API_KEY" \ > -H "Content-Type: application/json" \ > -d '{ > "baseModel": "accounts/fireworks/models/gpt-oss-120b" > }' > ``` > [tab] Python SDK > > ```python theme={null} > from fireworks import Fireworks > > client = Fireworks() > client.deployments.create( > account_id="YOUR_ACCOUNT_ID", > base_model="accounts/fireworks/models/gpt-oss-120b", > accept_shapeless_risk=True, > ) > ``` > [!WARNING] > > Deployments created this way skip shape validation and are far more likely to fail at creation — most failed deployment creations on Fireworks are shapeless deployments. Only use the opt-out when you have a specific reason no shape covers, and [contact us](https://fireworks.ai/contact) so we can help. Enforcement is coming soon: shapeless creation will then *require* this flag, so use a shape (or `default`) now unless you deliberately need this path. > [!TIP] > > Need even better performance with tailored optimizations? [Contact our team](https://fireworks.ai/contact). ## Managing & configuring deployments ### Basic management ```bash theme={null} # List all deployments firectl deployment list # Check deployment status firectl deployment get # Delete a deployment firectl deployment delete ``` Use [deployment tags](https://fireworks.ai/docs/deployments/deployment-tags.md) to attach customer-defined metadata such as an environment, team, or workload. > [!NOTE] > > By default, deployments scale to zero if unused for 1 hour. Deployments with min replicas set to 0 are automatically deleted after 7 days of no traffic. > [!WARNING] > > When a deployment is scaled to zero, requests return a `503` error immediately while the deployment scales up. Your application should implement retry logic to handle this. See [Scaling from zero behavior](https://fireworks.ai/docs/deployments/autoscaling.md#scaling-from-zero-behavior) for implementation details. ### GPU hardware > [!WARNING] > > **Do not create deployments without a shape.** If a [deployment shape](#deployment-shapes) fits your workload, use it — this section only applies when no shape fits. Creating a deployment without a shape (not passing `--deployment-shape`) skips validation: most failed deployment creations on Fireworks are deployments without a shape, and shapeless creation will soon require an explicit opt-in. If you need a configuration no shape covers, [contact us](https://fireworks.ai/contact) and we'll help you find or add one. Choose GPU type with `--accelerator-type`: * `NVIDIA_A100_80GB` * `NVIDIA_H100_80GB` * `NVIDIA_H200_141GB` * `NVIDIA_B200_180GB` * `NVIDIA_B300_288GB` * `AMD_MI325X_256GB` * `AMD_MI350X_288GB` GPU availability varies by [region](https://fireworks.ai/docs/deployments/regions.md). See the [Create Deployment API reference](https://fireworks.ai/docs/api-reference/create-deployment.md#body-accelerator-type) for the authoritative list of supported accelerator types. > [!NOTE] > > **Not every model runs on every GPU.** Each model is validated only on specific accelerator type, GPU count, and precision combinations — many models support just one or two entries from the list above. To see what a model supports, match its deployment shapes: > > ```bash theme={null} > firectl deployment-shape-version match --model accounts/fireworks/models/ > ``` > > The `GPUS`, `ACCELERATOR`, and `PRECISION` columns are the validated combinations; the model's page in the [Fireworks console](https://app.fireworks.ai) shows the same options under dedicated deployments. Setting `--accelerator-type` or `--accelerator-count` to a combination no shape validates will typically fail at creation — often with a generic `Internal error occurred` message that does not name the hardware mismatch. If you need hardware no shape covers, [contact us](https://fireworks.ai/contact). ### Autoscaling Control replica counts, scale timing, and load targets for your deployment. See the [Autoscaling guide](https://fireworks.ai/docs/deployments/autoscaling.md) for configuration options. ### Multiple GPUs per replica Use multiple GPUs to improve latency and throughput. First check whether a multi-GPU [deployment shape](#deployment-shapes) already exists for your model: ```bash theme={null} firectl deployment-shape-version match --model ``` If no multi-GPU shape fits, start from the closest shape and override the GPU count with `--accelerator-count` — see the warning under [GPU hardware](#gpu-hardware) for why omitting `--deployment-shape` entirely should be a last resort: ```bash theme={null} firectl deployment create \ --deployment-shape \ --accelerator-count 2 ``` More GPUs = faster generation. Note that scaling is sub-linear (2x GPUs ≠ 2x performance). ## Preemptible deployments (eval / batch) For the training eval workflow, see [Evaluating Trained Models](https://fireworks.ai/docs/fine-tuning/evaluating-fine-tuned-models.md). Preemptible deployments let you run **evaluation and batch workloads without holding dedicated on-demand capacity**. Passing `--preemptible` at creation opts the deployment into *borrowing idle reserved GPUs* instead of reserving GPUs exclusively for you, so you do not pay to hold dedicated capacity for an eval that only runs occasionally. Because the capacity is borrowed, it can be **reclaimed (preempted) at any time** and is not guaranteed. In practice a preemptible deployment usually stays up long enough to run an eval end to end, but you must treat disappearance as a normal outcome, not an error. > [!WARNING] > > **Eval and batch workloads only.** A preemptible deployment can be preempted mid-request and disappear with no warning. **Never** point production or latency-sensitive traffic at a preemptible deployment. ### Guarantees vs. risks | | Preemptible (`--preemptible`) | Standard on-demand | | - | - | - | | **Capacity model** | Borrows idle reserved GPUs | Reserves dedicated GPUs for you | | **Cost** | No charge to hold dedicated capacity | Billed by GPU-second while replicas are active | | **Can be reclaimed mid-request?** | Yes, at any time, without warning | No | | **Availability guarantee** | None (treat as best-effort) | Held for you once scheduled | | **Safe for production / latency-sensitive traffic?** | No | Yes | | **Intended for** | Evaluation, batch, one-off runs | Production and interactive serving | > [!NOTE] > > **No unique availability risk.** Preemptible does not make capacity *harder* to get. If there is no capacity for a preemptible deployment, there is no capacity for an on-demand one either. The only difference is that a preemptible deployment can also be taken back after it starts. ### Requirements * **`firectl` >= 1.7.26.** `--preemptible` is a newer flag and is **silently ignored** on older builds (no error, and the deployment is created as standard on-demand). Check with `firectl version` and upgrade if you are below this version. ### Create a preemptible deployment ```bash theme={null} firectl deployment create accounts/fireworks/models/ \ -a \ --deployment-id --display-name \ --deployment-shape default \ --min-replica-count 1 --max-replica-count 1 \ --preemptible --wait ``` `--wait` blocks until the deployment is ready (1 hour default; tune with `--wait-timeout`). > [!WARNING] > > **`--preemptible` is immutable.** It is set at creation time and cannot be toggled on or off later. To change it, delete the deployment and recreate it. Check status and then send eval requests as you would to any deployment: ```bash theme={null} firectl deployment get -a ``` When the eval is done, delete the deployment to release the borrowed capacity: ```bash theme={null} firectl deployment delete -a ``` ## Advanced * **[Speculative decoding](https://fireworks.ai/docs/deployments/speculative-decoding.md)** - Speed up text generation using draft models or n-gram speculation * **[Quantization](https://fireworks.ai/docs/models/quantization.md)** - Reduce model precision (e.g., FP16 to FP8) to improve speeds and reduce costs by 30-50% * **[Performance benchmarking](https://fireworks.ai/docs/deployments/benchmarking.md)** - Measure and optimize your deployment's performance with load testing * **[Managing default deployments](https://fireworks.ai/docs/deployments/managing-default-deployments)** - Control which deployment handles queries when using just the model name * **[Publishing deployments](https://fireworks.ai/docs/deployments/publishing-deployments)** - Make your deployment accessible to other Fireworks users ## Next steps > [card] [Autoscaling](https://fireworks.ai/docs/deployments/autoscaling.md) > > Configure autoscaling for optimal cost and performance > [card] [Upload custom models](https://fireworks.ai/docs/models/uploading-custom-models.md) > > Deploy your own models from Hugging Face > [card] [Quantization](https://fireworks.ai/docs/models/quantization.md) > > Reduce costs with model quantization > [card] [Regions](https://fireworks.ai/docs/deployments/regions.md) > > Choose deployment regions for optimal latency > [card] [Reserved capacity](https://fireworks.ai/docs/deployments/reservations.md) > > Purchase reserved GPUs for guaranteed capacity > [card] [Training](https://fireworks.ai/docs/fine-tuning/finetuning-intro.md) > > Train models for your specific use case This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.