The ability to assess model performance in a production environment is critical for iterative improvement. Together AI provides a system for running A/B experiments, allowing for the comparison of candidate models against existing ones. The system supports up to 20 variant members, each with a percentage split of traffic, and a single control deployment. Traffic is re-sampled among the arms based on these percentages, ensuring accurate measurement of performance metrics.
The platform’s endpoint router operates by subdividing the control’s share of traffic, distributing it to the variants according to their assigned percentages. This approach ensures that the experiment’s traffic split remains constant, independent of replica counts and autoscaling. The system prevents incorrect measurements by ensuring that only the control resides in the base traffic split, and variants carry zero weight. This design mirrors a shadow deployment, facilitating a controlled comparison.
Updates to the experiment configuration are managed through an API, utilizing an etag to prevent conflicts from concurrent ramps. This mechanism simplifies the process of increasing or decreasing traffic to a variant, providing a clear and reviewable change. The system supports multi-way tests with up to 20 variant members, allowing for comprehensive evaluation of multiple model versions simultaneously. The experiment’s percentage setting dictates the traffic share, ensuring consistent measurement.
Clients remain unaffected by the A/B experiment, as the endpoint name, API, and keys remain consistent. The backend routing handles the traffic split, directing 5% of requests to the variant candidate. The platform integrates with existing metrics, allowing for the monitoring of latency, errors, and throughput per deployment, providing a comprehensive view of model performance. Source: https://www.together.ai/blog/a-b-test-models-in-production
