> ## Documentation Index
> Fetch the complete documentation index at: https://fireworks.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Quantization

> Reduce Fireworks model precision to FP16, FP8, BF16, or INT8 to cut cost by 30-50% and speed up inference, with firectl prepare-model examples.

Quantization reduces the number of bits used to serve a model, improving performance and reducing cost by 30-50%. However, this can change model numerics which may introduce small changes to the output.

> [!TIP]
>
> Read our [blog post](https://fireworks.ai/blog/fireworks-quantization) for a detailed treatment of how quantization affects model quality.

## Checking available precisions

Models may support different numerical precisions like FP16, FP8, BF16, or INT8, which affect memory usage and inference speed.

**Check default precision:**

```bash theme={null}
firectl model get accounts/fireworks/models/llama-v3p1-8b-instruct | grep "Default Precision"
```

**Check supported precisions:**

```bash theme={null}
firectl model get accounts/fireworks/models/llama-v3p1-8b-instruct | grep -E "(Supported Precisions|Supported Precisions With Calibration)"
```

The `Precisions` field indicates what precisions the model has been prepared for.

## Quantizing a model

A model can be quantized to 8-bit floating-point (FP8) precision.

> [tab] firectl
>
> ```bash theme={null}
> firectl prepare-model <MODEL_ID>
> ```

> [tab] Python (REST API)
>
> ```python theme={null}
> import os
> import requests
>
> ACCOUNT_ID = os.environ.get("FIREWORKS_ACCOUNT_ID")
> API_KEY = os.environ.get("FIREWORKS_API_KEY")
> MODEL_ID = "<YOUR_MODEL_ID>" # The ID of the model you want to prepare
>
> response = requests.post(
>   f"https://api.fireworks.ai/v1/accounts/{ACCOUNT_ID}/models/{MODEL_ID}:prepare",
>   headers={
>     "Authorization": f"Bearer {API_KEY}",
>     "Content-Type": "application/json"
>   },
>   json={
>     "precision": "FP8"
>   }
> )
>
> print(response.json())
> ```

> [!NOTE]
>
> This is an additive process that enables creating deployments with additional precisions. The original FP16 checkpoint is still available for use.

You can check on the status of preparation by running:

> [tab] firectl
>
> ```bash theme={null}
> firectl model get <MODEL_ID>
> ```

> [tab] Python (REST API)
>
> ```python theme={null}
> import os
> import requests
>
> ACCOUNT_ID = os.environ.get("FIREWORKS_ACCOUNT_ID")
> API_KEY = os.environ.get("FIREWORKS_API_KEY")
> MODEL_ID = "<YOUR_MODEL_ID>" # The ID of the model you want to get
>
> response = requests.get(
>   f"https://api.fireworks.ai/v1/accounts/{ACCOUNT_ID}/models/{MODEL_ID}",
>   headers={
>     "Authorization": f"Bearer {API_KEY}"
>   }
> )
>
> print(response.json())
> ```

and checking if the state is still in `PREPARING`. A successfully prepared model will have the desired precision added
to the `Precisions` list.

## Creating an FP8 deployment

By default, creating a deployment uses the FP16 checkpoint. To use a quantized FP8 checkpoint, first ensure the model has been prepared for FP8 (see [Checking available precisions](#checking-available-precisions) above).

Do not deploy without a [deployment shape](https://fireworks.ai/docs/guides/ondemand-deployments.md#deployment-shapes) — a shape that includes FP8 precision is the only validated path. Match the model's shapes with `firectl deployment-shape-version match --model <MODEL>` and look for one with the precision you want in the `PRECISION` column.

If no FP8 shape exists for your model, start from the closest shape and override the precision with `--precision`. Do not create the deployment without a shape entirely: it skips validation, is the most common cause of failed deployment creations, and shapeless creation will soon require an explicit opt-in:

> [tab] firectl
>
> ```bash theme={null}
> firectl deployment create <MODEL> --deployment-shape <shape> --precision FP8
> ```

> [tab] Python (REST API)
>
> ```python theme={null}
> import os
> import requests
>
> ACCOUNT_ID = os.environ.get("FIREWORKS_ACCOUNT_ID")
> API_KEY = os.environ.get("FIREWORKS_API_KEY")
> # The ID of the model you want to deploy.
> # The model must be prepared for FP8 precision.
> MODEL_ID = "<YOUR_MODEL_ID>"
> DEPLOYMENT_NAME = "My FP8 Deployment"
>
> response = requests.post(
>   f"https://api.fireworks.ai/v1/accounts/{ACCOUNT_ID}/deployments",
>   headers={
>     "Authorization": f"Bearer {API_KEY}",
>     "Content-Type": "application/json"
>   },
>   json={
>     "displayName": DEPLOYMENT_NAME,
>     "baseModel": MODEL_ID,
>     "deploymentShape": "<SHAPE_ID>",
>     "precision": "FP8",
>   }
> )
>
> print(response.json())
> ```

> [!NOTE]
>
> Quantized deployments can only be served using H100 GPUs.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.