> ## Documentation Index
> Fetch the complete documentation index at: https://fireworks.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Preference Optimization with DPO or ORPO

> Train Fireworks-hosted models on preferred and non-preferred response pairs using managed DPO or ORPO preference optimization, with LoRA or full-parameter.

Preference optimization trains models on pairs of preferred and non-preferred responses to the same prompt. Managed jobs support two objectives:

* **DPO** compares the policy against a reference model.
* **ORPO** combines supervised and preference objectives without a separate reference model.

Use either method for:

* Aligning model outputs with brand voice, tone, or style guidelines
* Reducing hallucinations or incorrect reasoning patterns
* Improving response quality where there's no single "correct" answer
* Teaching models to follow specific formatting or structural preferences

## Training with DPO or ORPO

**1. Prepare dataset**

Datasets must adhere strictly to the JSONL format, where each line represents a complete JSON-formatted training example.

**Minimum Requirements:**

* **Minimum examples needed:** 3
* **Maximum examples:** Up to 3 million examples per dataset
* **File format:** JSONL (each line is a valid JSON object)
* **Dataset Schema:** Each training sample must include the following fields:
  * An `input` field containing a `messages` array, where each message is an object with two fields:
    * `role`: one of `system`, `user`, or `assistant`
    * `content`: a string representing the message content
  * A `preferred_output` field containing an assistant message with an ideal response
  * A `non_preferred_output` field containing an assistant message with a suboptimal response

Here’s an example conversation dataset (one training example):

```json einstein_dpo.jsonl theme={null}
{
  "input": {
    "messages": [
      {
        "role": "user",
        "content": "What is Einstein famous for?"
      }
    ],
    "tools": []
  },
  "preferred_output": [
    {
      "role": "assistant",
      "content": "Einstein is renowned for his theory of relativity, especially the equation E=mc²."
    }
  ],
  "non_preferred_output": [
    {
      "role": "assistant",
      "content": "He was a famous scientist."
    }
  ]
}
```

> [!WARNING]
>
> We currently only support one-turn conversations for each example, where the preferred and non-preferred messages need to be the last assistant message.

Save this dataset as jsonl file locally, for example `einstein_dpo.jsonl`.

**2. Upload the dataset**

Upload the dataset with the Fireworks UI, `firectl`, or the REST API.

> [tab] UI
>
> * You can simply navigate to the dataset tab, click `Create Dataset` and follow the wizard.
>
>   ![Dataset Pn](https://mintcdn.com/fireworksai-docs/8SoVRbdR43snsimO/images/fine-tuning/dataset.png?fit=max\&auto=format\&n=8SoVRbdR43snsimO\&q=85\&s=b24281cc8a13417f6397acd0b1f1aac5)

> [tab] firectl
>
> * Upload dataset using `firectl`
>
> ```bash theme={null}
> firectl dataset create <dataset-id> /path/to/file.jsonl
> ```

> [tab] Restful API
>
> You need to make two separate HTTP requests. One for creating the dataset entry and one for uploading the dataset. Full reference here: [Create dataset](https://fireworks.ai/docs/api-reference/create-dataset.md). Note that the `exampleCount` parameter needs to be provided by the client.
>
> ```jsx theme={null}
> // Create Dataset Entry
> const createDatasetPayload = {
>   datasetId: "trader-poe-sample-data",
>   dataset: { userUploaded: {} }
>   // Additional params such as exampleCount
> };
> const urlCreateDataset = `${BASE_URL}/datasets`;
> const response = await fetch(urlCreateDataset, {
>   method: "POST",
>   headers: HEADERS_WITH_CONTENT_TYPE,
>   body: JSON.stringify(createDatasetPayload)
> });
> ```
>
> ```jsx theme={null}
> // Upload JSONL file
> const urlUpload = `${BASE_URL}/datasets/${DATASET_ID}:upload`;
> const files = new FormData();
> files.append("file", localFileInput.files[0]);
>
> const uploadResponse = await fetch(urlUpload, {
>   method: "POST",
>   headers: HEADERS,
>   body: files
> });
> ```

While all of the above approaches should work, `UI` is more suitable for smaller datasets `< 500MB` while `firectl` might work better for bigger datasets.

Ensure the dataset ID conforms to the [resource id restrictions](https://fireworks.ai/docs/getting-started/concepts.md#resource-names-and-ids).

**3. Create a DPO or ORPO Job**

> [tab] UI
>
> In the `Training` tab, click `New job` and choose **Direct Preference**, which is how the UI labels DPO. Select your base model and preference dataset, then follow the wizard.

> [tab] firectl
>
> ```bash theme={null}
> firectl dpo-job create \
>   --loss-method DPO \
>   --base-model accounts/account-id/models/base-model-id \
>   --dataset accounts/my-account-id/datasets/my-dataset-id \
>   --output-model new-model-id
> ```
>
> For ORPO, use the same preference dataset and select the ORPO objective:
>
> ```bash theme={null}
> firectl dpo-job create \
>   --loss-method ORPO \
>   --orpo-lambda <approved-value> \
>   --base-model accounts/account-id/models/base-model-id \
>   --dataset accounts/my-account-id/datasets/my-dataset-id \
>   --output-model new-model-id
> ```

> [tab] REST API
>
> ```bash theme={null}
> curl -X POST "https://api.fireworks.ai/v1/accounts/<ACCOUNT_ID>/dpoJobs" \
>   -H "Authorization: Bearer $FIREWORKS_API_KEY" \
>   -H "Content-Type: application/json" \
>   -d '{
>     "dataset": "accounts/<ACCOUNT_ID>/datasets/<DATASET_ID>",
>     "trainingConfig": {
>       "baseModel": "accounts/fireworks/models/<BASE_MODEL_ID>",
>       "outputModel": "accounts/<ACCOUNT_ID>/models/<OUTPUT_MODEL_ID>",
>       "loraRank": 8
>     },
>     "lossConfig": {"method": "DPO"}
>   }'
> ```
>
> For ORPO, set `"method": "ORPO"`. If `lossConfig` is omitted, the job uses DPO.

> [tab] Python SDK
>
> ```python theme={null}
> from fireworks import Fireworks
>
> client = Fireworks()  # reads FIREWORKS_API_KEY and FIREWORKS_ACCOUNT_ID
>
> job = client.dpo_jobs.create(
>     dataset="accounts/<ACCOUNT_ID>/datasets/<DATASET_ID>",
>     training_config={
>         "base_model": "accounts/fireworks/models/<BASE_MODEL_ID>",
>         "output_model": "accounts/<ACCOUNT_ID>/models/<OUTPUT_MODEL_ID>",
>         "lora_rank": 8,
>     },
>     loss_config={"method": "DPO"},  # "ORPO" for ORPO
> )
> print(job.name)
> ```

`firectl` trains a LoRA adapter with rank 8 by default. With the REST API or the Python SDK, always set the rank to train a LoRA adapter: `trainingConfig.loraRank` in the job body, or `training_config={"lora_rank": 8}` in `dpo_jobs.create(...)`. An omitted rank is sent as 0, which requests full-parameter training.

Reservation placement is off unless you request it: add `--use-reservation` with firectl, set `useReservation: true` in the REST body, or pass `use_reservation=True` to `dpo_jobs.create(...)` in the Python SDK. With a matching active reservation, the job runs only on that reservation, not on shared capacity; without one, the setting is ignored. For full-parameter DPO, policy and dedicated reference trainers try independently.

Choose a base model that [Models](https://fireworks.ai/docs/fine-tuning/models.md) marks as DPO-enabled, and a shape published for it.

To run DPO or ORPO from your own Python training loop instead of a managed job, start from the [cookbook DPO recipe](https://fireworks.ai/docs/fine-tuning/training-api/cookbook/dpo.md).

**4. Monitor the preference job**

> [tab] firectl
>
> ```bash theme={null}
> firectl dpo-job get dpo-job-id
> ```

> [tab] REST API
>
> ```bash theme={null}
> curl "https://api.fireworks.ai/v1/accounts/<ACCOUNT_ID>/dpoJobs/<DPO_JOB_ID>" \
>   -H "Authorization: Bearer $FIREWORKS_API_KEY"
> ```

> [tab] Python SDK
>
> ```python theme={null}
> job = client.dpo_jobs.get("<DPO_JOB_ID>")
> print(job.state)
> ```

Once the job is complete, the state is `JOB_STATE_COMPLETED`, and the trained model can be deployed.

**5. Deploy the DPO trained model**

Once training completes, you can create a deployment to interact with the trained model. Refer to [deploying a trained model](https://fireworks.ai/docs/fine-tuning/fine-tuning-models.md#deploying-a-trained-model) for more details. Deployments scale to zero when idle, and the first request after that returns 503 while the deployment scales back up; see [Scaling from zero behavior](https://fireworks.ai/docs/deployments/autoscaling.md#scaling-from-zero-behavior).

## Next Steps

Explore other training methods to improve model output for different use cases.

> [card] [Supervised Fine-Tuning - Text](https://fireworks.ai/docs/fine-tuning/fine-tuning-models.md)
>
> Train models on input-output examples to improve task-specific performance.

> [card] [Reinforcement Learning](https://fireworks.ai/docs/fine-tuning/training-api/cookbook/rl.md)
>
> Optimize models against a reward function on the Training API.

> [card] [Supervised Fine-Tuning - Vision](https://fireworks.ai/docs/fine-tuning/fine-tuning-models.md#vision-training)
>
> Train vision-language models to understand both images and text.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.