> ## Documentation Index > Fetch the complete documentation index at: https://fireworks.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Speculative Decoding > Speed up token generation on Fireworks dedicated deployments with speculative decoding using draft models, n-gram speculation, and per-request configuration. Speculative decoding reduces generation latency by proposing multiple tokens and letting the target model verify them in parallel. The target model still verifies every accepted token; the drafter does not replace the target model. The benefit depends on both the cost of producing draft tokens and how often the target model accepts them. A poorly matched drafter can make generation slower, so benchmark with representative traffic before overriding Fireworks defaults. > [!NOTE] > > The deployment flags on this page apply to [dedicated > deployments](https://fireworks.ai/docs/guides/ondemand-deployments.md). Fireworks manages the serving > configuration for Serverless models. ## Start with the default > [!TIP] > > **For most supported models, a default drafter and draft-token count are already > configured.** A new deployment inherits those settings, so you usually do not > need to pass any speculative-decoding flags. > > Create the deployment normally, then benchmark it before changing the drafter: > > ```bash theme={null} > firectl deployment create accounts/fireworks/models/ \ > --deployment-shape default \ > --wait > ``` If the base model does not define a default drafter, the deployment runs without model-based speculative decoding. To explicitly disable an inherited default when creating a comparison deployment, use: ```bash theme={null} firectl deployment create accounts/fireworks/models/ \ --deployment-shape default \ --disable-speculative-decoding \ --wait ``` ## Choose a method | Method | Best starting point | Configuration | | - | - | - | | Default model-based speculation | General chat, reasoning, and coding traffic | No flags; inherit the model's default | | Custom draft model | A validated drafter for your model or traffic distribution | `--draft-model` and `--draft-token-count` | | N-gram speculation | Repetitive output, code editing, and structured generation where output often repeats the prompt or prior context | `--ngram-speculation-length` and `--draft-token-count` | | [Predicted Outputs](https://fireworks.ai/docs/guides/predicted-outputs.md) | The caller already knows most of the expected response, such as regenerating a file with a small edit | Request-level `prediction` or `speculation` input | Predicted Outputs can be used in addition to a deployment's model-based speculative decoding. ## Configuration options | Flag | Description | | - | - | | `--draft-model` | Resource name of a Fireworks or custom draft model. If omitted, the deployment inherits the base model's default drafter. | | `--draft-token-count` | Number of candidate tokens proposed per step. It is required with an explicitly selected draft model or N-gram speculation. Start with `4`, then benchmark nearby values. | | `--ngram-speculation-length` | Length of the previous input sequence used for N-gram matching. This does not require a separate draft model. | | `--disable-speculative-decoding` | Disables inherited speculative-decoding settings when creating a deployment. | > [!NOTE] > > `--draft-model` and `--ngram-speculation-length` are alternative deployment > strategies and cannot be used together. ## Custom draft models For self-service configuration, use a small base model that is compatible with the target model. In practice, this means using the same model family and tokenizer. A model that is merely smaller is not necessarily a useful drafter; its acceptance rate and execution cost both matter. ### Fallback draft models If the target model has no default drafter, the following small base models are reasonable starting points for an experiment. A purpose-built drafter generally performs better. | Draft model | Use with | | - | - | | `accounts/fireworks/models/llama-v3p2-1b-instruct` | All Llama models > 3B | | `accounts/fireworks/models/qwen2p5-0p5b-instruct` | All Qwen models > 3B | Fireworks also supports compatible EAGLE, DFlash, DSpark, and Medusa draft addons. These formats are architecture-specific and require a checkpoint and configuration prepared for the exact target model; they are not drop-in replacements for a small base-model drafter. [Contact Fireworks](https://fireworks.ai/company/contact-us) to validate an existing checkpoint or discuss a drafter adapted to your traffic. ## Examples > [tab] Draft model > > Create a deployment with an explicit small base-model drafter: > > ```bash theme={null} > firectl deployment create accounts/fireworks/models/llama-v3p3-70b-instruct \ > --deployment-shape default \ > --draft-model="accounts/fireworks/models/llama-v3p2-1b-instruct" \ > --draft-token-count=4 > ``` > [tab] N-gram speculation > > Use N-gram speculation without a separate draft model: > > ```bash theme={null} > firectl deployment create accounts/fireworks/models/llama-v3p3-70b-instruct \ > --deployment-shape default \ > --ngram-speculation-length=3 \ > --draft-token-count=4 > ``` You can change the explicit drafter and draft-token count on an existing deployment: ```bash theme={null} firectl deployment update \ --draft-model="accounts//models/" \ --draft-token-count=4 ``` ## Benchmark and tune Compare at least three configurations on the same target model and deployment shape: 1. The inherited Fireworks default. 2. Your candidate drafter or N-gram settings. 3. A deployment created with `--disable-speculative-decoding`. Use production-like prompts, output lengths, sampling parameters, and concurrency. Measure time to first token, inter-token latency, p50/p95 request latency, and maximum sustainable throughput. A high acceptance rate alone does not guarantee a speedup because the drafter also consumes compute. To inspect per-request metrics, set `perf_metrics_in_response` to `true` in the completion request. For dedicated deployments, the final response or final streaming chunk includes: * `speculation-generated-tokens`: number of tokens generated through speculation * `speculation-acceptance`: acceptance rate by proposed-token position Acceptance normally falls at later positions. Increase `--draft-token-count` only while the additional accepted tokens outweigh the extra drafting and verification work. Re-run the benchmark when the traffic mix, prompt format, model, quantization, or deployment shape changes. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.