amazon-sagemaker - A/B Test feature in SageMaker: variant assignment is random?

Question

A/B test feature in SageMaker sounds so intriguing but the more I looked into, the more I am confused whether this is a useful feature. For this to be useful, you need to get the variant assignment data back and join with some internal data to figure out the best performing variant.

How is this assignment done? Is it purely random? Or am I supposed to pass some kind of ID (or hashed ID) which can indicate a person or a browser so that the same model is picked up for the same person.

标签： amazon-sagemaker

Answer 1

For this to be useful, you need to get the variant assignment data back and join with some internal data to figure out the best performing variant.

The InvokeEndpoint response includes the "InvokedProductionVariant", in order to support the kind of analysis you describe. Details can be found in the API documentation: https://docs.aws.amazon.com/sagemaker/latest/dg/API_runtime_InvokeEndpoint.html#API_runtime_InvokeEndpoint_ResponseSyntax

How is this assignment done? Is it purely random?

Traffic is distributed randomly while remaining proportional to the weight of the production variant.