[!INCLUDE dev v2]

In this article, you learn to deploy your model to an online endpoint for use in real-time inferencing. You begin by deploying a model on your local machine to debug any errors. Then, you deploy and test the model in Azure, view the deployment logs, and monitor the service-level agreement (SLA). By the end of this article, you'll have a scalable HTTPS/REST endpoint that you can use for real-time inference.

Online endpoints are endpoints that are used for real-time inferencing. There are two types of online endpoints: managed online endpoints and Kubernetes online endpoints. For more information on endpoints and differences between managed online endpoints and Kubernetes online endpoints, see What are Azure Machine Learning endpoints?

Managed online endpoints help to deploy your machine learning models in a turnkey manner. Managed online endpoints work with powerful CPU and GPU machines in Azure in a scalable, fully managed way. Managed online endpoints take care of serving, scaling, securing, and monitoring your models, freeing you from the overhead of setting up and managing the underlying infrastructure.

The main example in this doc uses managed online endpoints for deployment. To use Kubernetes instead, see the notes in this document that are inline with the managed online endpoint discussion.

[!INCLUDE cli v2]

[!INCLUDE basic prereqs cli]

• Azure role-based access controls (Azure RBAC) are used to grant access to operations in Azure Machine Learning. To perform the steps in this article, your user account must be assigned the owner or contributor role for the Azure Machine Learning workspace, or a custom role allowing Microsoft.MachineLearningServices/workspaces/onlineEndpoints/*. If you use the studio to create/manage online endpoints/deployments, you'll need an extra permission "Microsoft.Resources/deployments/write" from the resource group owner. For more information, see Manage access to an Azure Machine Learning workspace.

• (Optional) To deploy locally, you must install Docker Engine on your local computer. We highly recommend this option, so it's easier to debug issues.

[!INCLUDE sdk v2]

[!INCLUDE basic prereqs sdk]

• Azure role-based access controls (Azure RBAC) are used to grant access to operations in Azure Machine Learning. To perform the steps in this article, your user account must be assigned the owner or contributor role for the Azure Machine Learning workspace, or a custom role allowing Microsoft.MachineLearningServices/workspaces/onlineEndpoints/*. For more information, see Manage access to an Azure Machine Learning workspace.

• (Optional) To deploy locally, you must install Docker Engine on your local computer. We highly recommend this option, so it's easier to debug issues.

Before following the steps in this article, make sure you have the following prerequisites:

• An Azure subscription. If you don't have an Azure subscription, create a free account before you begin. Try the free or paid version of Azure Machine Learning.

• An Azure Machine Learning workspace and a compute instance. If you don't have these resources and want to create them, use the steps in the Quickstart: Create workspace resources article.

• Azure role-based access controls (Azure RBAC) are used to grant access to operations in Azure Machine Learning. To perform the steps in this article, your user account must be assigned the owner or contributor role for the Azure Machine Learning workspace, or a custom role allowing Microsoft.MachineLearningServices/workspaces/onlineEndpoints/*. For more information, see Manage access to an Azure Machine Learning workspace.

[!INCLUDE basic prereqs cli]

• Azure role-based access controls (Azure RBAC) are used to grant access to operations in Azure Machine Learning. To perform the steps in this article, your user account must be assigned the owner or contributor role for the Azure Machine Learning workspace, or a custom role allowing Microsoft.MachineLearningServices/workspaces/onlineEndpoints/*. For more information, see Manage access to an Azure Machine Learning workspace.

• Ensure that you have enough virtual machine (VM) quota allocated for deployment. Azure Machine Learning reserves 20% of your compute resources for performing upgrades on some VM SKUs. For example, if you request 10 instances in a deployment, you must have a quota for 12 for each number of cores for the VM SKU. Failure to account for the extra compute resources results in an error. There are some VM SKUs that are exempt from the extra quota reservation. For more information on quota allocation, see virtual machine quota allocation for deployment.

• Alternatively, you could use quota from Azure Machine Learning's shared quota pool for a limited time. Users can access quota from this pool to perform testing for a limited time. When you use the studio to deploy Llama-2, Phi, Nemotron, Mistral, Dolly, and Deci-DeciLM models from the model catalog to a managed online endpoint, Azure Machine Learning allows you to access its shared quota pool for a short time so that you can perform testing. For more information on the shared quota pool, see Azure Machine Learning shared quota.

If you haven't already set the defaults for the Azure CLI, save your default settings. To avoid passing in the values for your subscription, workspace, and resource group multiple times, run this code:

az account set --subscription <subscription ID>
az configure --defaults workspace=<Azure Machine Learning workspace name> group=<resource group>

To follow along with this article, first clone the examples repository (azureml-examples). Then, run the following code to go to the repository's cli/ directory:

git clone --depth 1 https://github.com/Azure/azureml-examples
cd azureml-examples
cd cli

The commands in this tutorial are in the files deploy-local-endpoint.sh and deploy-managed-online-endpoint.sh in the cli directory, and the YAML configuration files are in the endpoints/online/managed/sample/ subdirectory.

To run the training examples, first clone the examples repository (azureml-examples) and change into the azureml-examples/sdk/python/endpoints/online/managed directory:

git clone --depth 1 https://github.com/Azure/azureml-examples
cd azureml-examples/sdk/python/endpoints/online/managed

The information in this article is based on the online-endpoints-simple-deployment.ipynb notebook. It contains the same content as this article, although the order of the codes is slightly different.

The workspace is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section, you connect to the workspace in which you'll perform deployment tasks. To follow along, open your online-endpoints-simple-deployment.ipynb notebook.

1. Import the required libraries:

   # import required libraries
   from azure.ai.ml import MLClient
   from azure.ai.ml.entities import (
       ManagedOnlineEndpoint,
       ManagedOnlineDeployment,
       Model,
       Environment,
       CodeConfiguration,
   )
   from azure.identity import DefaultAzureCredential

   [!NOTE] If you're using the Kubernetes online endpoint, import the KubernetesOnlineEndpoint and KubernetesOnlineDeployment class from the azure.ai.ml.entities library.

2. Configure workspace details and get a handle to the workspace:

   To connect to a workspace, you need identifier parameters - a subscription, resource group, and workspace name. You use these details in the MLClient from azure.ai.ml to get a handle to the required Azure Machine Learning workspace. This example uses the default Azure authentication.

   # enter details of your Azure Machine Learning workspace
   subscription_id = "<SUBSCRIPTION_ID>"
   resource_group = "<RESOURCE_GROUP>"
   workspace = "<AZUREML_WORKSPACE_NAME>"

   # get a handle to the workspace
   ml_client = MLClient(
       DefaultAzureCredential(), subscription_id, resource_group, workspace
   )

If you have Git installed on your local machine, you can follow the instructions to clone the examples repository. Otherwise, follow the instructions to download files from the examples repository.

To follow along with this article, first clone the examples repository (azureml-examples) and then change into the azureml-examples/cli/endpoints/online/model-1 directory.

git clone --depth 1 https://github.com/Azure/azureml-examples
cd azureml-examples/cli/endpoints/online/model-1

If you cloned the examples repo, your local machine already has copies of the files for this example, and you can skip to the next section. If you didn't clone the repo, you can download it to your local machine.

1. Go to https://github.com/Azure/azureml-examples/.
2. Go to the <> Code button on the page, and then select Download ZIP from the Local tab.
3. Locate the folder /cli/endpoints/online/model-1/model and the file /cli/endpoints/online/model-1/onlinescoring/score.py.

Set the following environment variables, as they're used in the examples in this article. Replace the values with your Azure subscription ID, the Azure region where your workspace is located, the resource group that contains the workspace, and the workspace name:

export SUBSCRIPTION_ID="your Azure subscription ID"
export LOCATION="Azure region where your workspace is located"
export RESOURCE_GROUP="Azure resource group that contains your workspace"
export WORKSPACE="Azure Machine Learning workspace name"

A couple of the template examples require you to upload files to the Azure Blob store for your workspace. The following steps query the workspace and store this information in environment variables used in the examples:

1. Get an access token:

   :::code language="azurecli" source="~/azureml-examples-main/deploy-arm-templates-az-cli.sh" id="get_access_token":::

2. Set the REST API version:

   :::code language="azurecli" source="~/azureml-examples-main/deploy-arm-templates-az-cli.sh" id="api_version":::

3. Get the storage information:

   :::code language="azurecli" source="~/azureml-examples-main/deploy-arm-templates-az-cli.sh" id="get_storage_details":::

To follow along with this article, first clone the examples repository (azureml-examples). Then, run the following code to go to the examples directory:

git clone --depth 1 https://github.com/Azure/azureml-examples
cd azureml-examples

To define an online endpoint, specify the endpoint name and authentication mode. For more information on managed online endpoints, see Online endpoints.

To set your endpoint name, run the following command. Replace YOUR_ENDPOINT_NAME with a name that's unique in the Azure region. For more information on the naming rules, see endpoint limits.

For Linux, run this command:

:::code language="azurecli" source="~/azureml-examples-main/cli/deploy-local-endpoint.sh" ID="set_endpoint_name":::

The following snippet shows the endpoints/online/managed/sample/endpoint.yml file:

:::code language="yaml" source="~/azureml-examples-main/cli/endpoints/online/managed/sample/endpoint.yml":::

The reference for the endpoint YAML format is described in the following table. To learn how to specify these attributes, see the online endpoint YAML reference. For information about limits related to managed endpoints, see limits for online endpoints.

First define the name of the online endpoint, then configure the endpoint.

Your endpoint name must be unique in the Azure region. For more information on the naming rules, see endpoint limits.

# Define an endpoint name
endpoint_name = "my-endpoint"

# Example way to define a random name
import datetime

endpoint_name = "endpt-" + datetime.datetime.now().strftime("%m%d%H%M%f")

# create an online endpoint
endpoint = ManagedOnlineEndpoint(
    name = endpoint_name, 
    description="this is a sample endpoint",
    auth_mode="key"
)

The previous code uses key for key-based authentication. To use Azure Machine Learning token-based authentication, use aml_token. To use Microsoft Entra token-based authentication (preview), use aad_token. For more information on authenticating, see Authenticate clients for online endpoints.

When you deploy to Azure from the studio, you'll create an endpoint and a deployment to add to it. At that time, you'll be prompted to provide names for the endpoint and deployment.

To set your endpoint name, run the following command. Replace YOUR_ENDPOINT_NAME with a name that's unique in the Azure region. For more information on the naming rules, see endpoint limits.

For Linux, run this command:

:::code language="azurecli" source="~/azureml-examples-main/deploy-arm-templates-az-cli.sh" ID="set_endpoint_name":::

To define the endpoint and deployment, this article uses the Azure Resource Manager templates online-endpoint.json and online-endpoint-deployment.json. To use the templates for defining an online endpoint and deployment, see the Deploy to Azure section.

A deployment is a set of resources required for hosting the model that does the actual inferencing. For this example, you deploy a scikit-learn model that does regression and use a scoring script score.py to execute the model upon a given input request.

To learn about the key attributes of a deployment, see Online deployments.

Your deployment configuration uses the location of the model that you wish to deploy.

The following snippet shows the endpoints/online/managed/sample/blue-deployment.yml file, with all the required inputs to configure a deployment:

blue-deployment.yml :::code language="yaml" source="~/azureml-examples-main/cli/endpoints/online/managed/sample/blue-deployment.yml":::

The blue-deployment.yml file specifies the following deployment attributes:

• model - specifies the model properties inline, using the path (where to upload files from). The CLI automatically uploads the model files and registers the model with an autogenerated name.
• environment - using inline definitions that include where to upload files from, the CLI automatically uploads the conda.yaml file and registers the environment. Later, to build the environment, the deployment uses the image (in this example, it's mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest) for the base image, and the conda_file dependencies are installed on top of the base image.
• code_configuration - during deployment, the local files such as the Python source for the scoring model, are uploaded from the development environment.

For more information about the YAML schema, see the online endpoint YAML reference.

To configure a deployment:

model = Model(path="../model-1/model/sklearn_regression_model.pkl")
env = Environment(
    conda_file="../model-1/environment/conda.yaml",
    image="mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest",
)

blue_deployment = ManagedOnlineDeployment(
    name="blue",
    endpoint_name=endpoint_name,
    model=model,
    environment=env,
    code_configuration=CodeConfiguration(
        code="../model-1/onlinescoring", scoring_script="score.py"
    ),
    instance_type="Standard_DS3_v2",
    instance_count=1,
)

• Model - specifies the model properties inline, using the path (where to upload files from). The SDK automatically uploads the model files and registers the model with an autogenerated name.
• Environment - using inline definitions that include where to upload files from, the SDK automatically uploads the conda.yaml file and registers the environment. Later, to build the environment, the deployment uses the image (in this example, it's mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest) for the base image, and the conda_file dependencies are installed on top of the base image.
• CodeConfiguration - during deployment, the local files such as the Python source for the scoring model, are uploaded from the development environment.

For more information about online deployment definition, see OnlineDeployment Class.

When you deploy to Azure, you'll create an endpoint and a deployment to add to it. At that time, you'll be prompted to provide names for the endpoint and deployment.

To define the endpoint and deployment, this article uses the Azure Resource Manager templates online-endpoint.json and online-endpoint-deployment.json. To use the templates for defining an online endpoint and deployment, see the Deploy to Azure section.

The scoring script specified in code_configuration.scoring_script must have an init() function and a run() function.

The scoring script must have an init() function and a run() function.

The scoring script must have an init() function and a run() function.

The scoring script must have an init() function and a run() function. This article uses the score.py file.

When using a template for deployment, you must first upload the scoring file(s) to an Azure Blob store, and then register it:

1. The following code uses the Azure CLI command az storage blob upload-batch to upload the scoring file(s):

   :::code language="azurecli" source="~/azureml-examples-main/deploy-arm-templates-az-cli.sh" id="upload_code":::

2. The following code registers the code, using a template:

   :::code language="azurecli" source="~/azureml-examples-main/deploy-arm-templates-az-cli.sh" id="create_code":::

This example uses the score.py file: score.py :::code language="python" source="~/azureml-examples-main/cli/endpoints/online/model-1/onlinescoring/score.py" :::

The init() function is called when the container is initialized or started. Initialization typically occurs shortly after the deployment is created or updated. The init function is the place to write logic for global initialization operations like caching the model in memory (as shown in this score.py file).

The run() function is called every time the endpoint is invoked, and it does the actual scoring and prediction. In this score.py file, the run() function extracts data from a JSON input, calls the scikit-learn model's predict() method, and then returns the prediction result.

We highly recommend that you test-run your endpoint locally to validate and debug your code and configuration before you deploy to Azure. Azure CLI and Python SDK support local endpoints and deployments, while Azure Machine Learning studio and ARM template don't.

To deploy locally, Docker Engine must be installed and running. Docker Engine typically starts when the computer starts. If it doesn't, you can troubleshoot Docker Engine.

For more information on debugging online endpoints locally before deploying to Azure, see Online endpoint debugging.

First create an endpoint. Optionally, for a local endpoint, you can skip this step and directly create the deployment (next step), which will, in turn, create the required metadata. Deploying models locally is useful for development and testing purposes.

:::code language="azurecli" source="~/azureml-examples-main/cli/deploy-local-endpoint.sh" ID="create_endpoint":::

ml_client.online_endpoints.begin_create_or_update(endpoint, local=True)

The studio doesn't support local endpoints. See the Azure CLI or Python tabs for steps to test the endpoint locally.

The template doesn't support local endpoints. See the Azure CLI or Python tabs for steps to test the endpoint locally.

Now, create a deployment named blue under the endpoint.

:::code language="azurecli" source="~/azureml-examples-main/cli/deploy-local-endpoint.sh" ID="create_deployment":::

The --local flag directs the CLI to deploy the endpoint in the Docker environment.

ml_client.online_deployments.begin_create_or_update(
    deployment=blue_deployment, local=True
)

The local=True flag directs the SDK to deploy the endpoint in the Docker environment.

The studio doesn't support local endpoints. See the Azure CLI or Python tabs for steps to test the endpoint locally.

The template doesn't support local endpoints. See the Azure CLI or Python tabs for steps to test the endpoint locally.

Check the deployment status to see whether the model was deployed without error:

:::code language="azurecli" source="~/azureml-examples-main/cli/deploy-local-endpoint.sh" ID="get_status":::

The output should appear similar to the following JSON. The provisioning_state is Succeeded.

{
  "auth_mode": "key",
  "location": "local",
  "name": "docs-endpoint",
  "properties": {},
  "provisioning_state": "Succeeded",
  "scoring_uri": "http://localhost:49158/score",
  "tags": {},
  "traffic": {}
}

ml_client.online_endpoints.get(name=endpoint_name, local=True)

The method returns ManagedOnlineEndpoint entity. The provisioning_state is Succeeded.

ManagedOnlineEndpoint({'public_network_access': None, 'provisioning_state': 'Succeeded', 'scoring_uri': 'http://localhost:49158/score', 'swagger_uri': None, 'name': 'endpt-10061534497697', 'description': 'this is a sample endpoint', 'tags': {}, 'properties': {}, 'id': None, 'Resource__source_path': None, 'base_path': '/path/to/your/working/directory', 'creation_context': None, 'serialize': <msrest.serialization.Serializer object at 0x7ffb781bccd0>, 'auth_mode': 'key', 'location': 'local', 'identity': None, 'traffic': {}, 'mirror_traffic': {}, 'kind': None})

The studio doesn't support local endpoints. See the Azure CLI or Python tabs for steps to test the endpoint locally.

The template doesn't support local endpoints. See the Azure CLI or Python tabs for steps to test the endpoint locally.

The following table contains the possible values for provisioning_state:

Invoke the endpoint to score the model by using the invoke command and passing query parameters that are stored in a JSON file:

:::code language="azurecli" source="~/azureml-examples-main/cli/deploy-local-endpoint.sh" ID="test_endpoint":::

If you want to use a REST client (like curl), you must have the scoring URI. To get the scoring URI, run az ml online-endpoint show --local -n $ENDPOINT_NAME. In the returned data, find the scoring_uri attribute.

Invoke the endpoint to score the model by using the invoke command and passing query parameters that are stored in a JSON file.

ml_client.online_endpoints.invoke(
    endpoint_name=endpoint_name,
    request_file="../model-1/sample-request.json",
    local=True,
)

If you want to use a REST client (like curl), you must have the scoring URI. To get the scoring URI, run the following code. In the returned data, find the scoring_uri attribute.

endpoint = ml_client.online_endpoints.get(endpoint_name, local=True)
scoring_uri = endpoint.scoring_uri

The studio doesn't support local endpoints. See the Azure CLI or Python tabs for steps to test the endpoint locally.

The template doesn't support local endpoints. See the Azure CLI or Python tabs for steps to test the endpoint locally.

In the example score.py file, the run() method logs some output to the console.

You can view this output by using the get-logs command:

:::code language="azurecli" source="~/azureml-examples-main/cli/deploy-local-endpoint.sh" ID="get_logs":::

You can view this output by using the get_logs method:

ml_client.online_deployments.get_logs(
    name="blue", endpoint_name=endpoint_name, local=True, lines=50
)

The studio doesn't support local endpoints. See the Azure CLI or Python tabs for steps to test the endpoint locally.

The template doesn't support local endpoints. See the Azure CLI or Python tabs for steps to test the endpoint locally.

Next, deploy your online endpoint to Azure. As a best practice for production, we recommend that you register the model and environment that you'll use in your deployment.

We recommend that you register your model and environment before deployment to Azure so that you can specify their registered names and versions during deployment. Registering your assets allows you to reuse them without the need to upload them every time you create deployments, thereby increasing reproducibility and traceability.

To register the model and environment, use the form model: azureml:my-model:1 or environment: azureml:my-env:1. For registration, you can extract the YAML definitions of model and environment into separate YAML files and use the commands az ml model create and az ml environment create. To learn more about these commands, run az ml model create -h and az ml environment create -h.

1. Create a YAML definition for the model:

   $schema: https://azuremlschemas.azureedge.net/latest/model.schema.json
   name: my-model
   path: ../../model-1/model/

2. Register the model:

   az ml model create -n my-model -v 1 -f ./model.yaml
   

3. Create a YAML definition for the environment:

   $schema: https://azuremlschemas.azureedge.net/latest/environment.schema.json
   name: my-env
   image: mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest
   conda_file: ../../model-1/environment/conda.yaml

4. Register the environment:

   az ml environment create -n my-env -v 1 -f ./environment.yaml
   

For more information on registering your model as an asset, see Register your model as an asset in Machine Learning by using the CLI. For more information on creating an environment, see Manage Azure Machine Learning environments with the CLI & SDK (v2).

1. Register a model

   from azure.ai.ml.entities import Model
   from azure.ai.ml.constants import AssetTypes
   
   file_model = Model(
       path="../../model-1/model/",
       type=AssetTypes.CUSTOM_MODEL,
       name="my-model",
       description="Model created from local file.",
   )
   ml_client.models.create_or_update(file_model)

2. Register the environment:

   from azure.ai.ml.entities import Environment
   
   env_docker_conda = Environment(
       image="mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04",
       conda_file="../../model-1/environment/conda.yaml",
       name="my-env",
       description="Environment created from a Docker image plus Conda environment.",
   )
   ml_client.environments.create_or_update(env_docker_conda)

To learn how to register your model as an asset so that you can specify its registered name and version during deployment, see Register your model as an asset in Machine Learning by using the SDK.

For more information on creating an environment, see Manage Azure Machine Learning environments with the CLI & SDK (v2).

A model registration is a logical entity in the workspace that can contain a single model file or a directory of multiple files. As a best practice for production, you should register the model and environment. Before creating the endpoint and deployment in this article, you should register the model folder that contains the model.

To register the example model, follow these steps:

1. Go to the Azure Machine Learning studio.

2. In the left navigation bar, select the Models page.

3. Select Register, and then choose From local files.

4. Select Unspecified type for the Model type.

5. Select Browse, and choose Browse folder.

   :::image type="content" source="media/how-to-deploy-online-endpoints/register-model-folder.png" alt-text="A screenshot of the browse folder option." lightbox="media/how-to-deploy-online-endpoints/register-model-folder.png":::

6. Select the \azureml-examples\cli\endpoints\online\model-1\model folder from the local copy of the repo you cloned or downloaded earlier. When prompted, select Upload and wait for the upload to complete.

7. Select Next after the folder upload is completed.

8. Enter a friendly Name for the model. The steps in this article assume the model is named model-1.

9. Select Next, and then Register to complete registration.

For more information on working with registered models, see Register and work with models.

1. In the left navigation bar, select the Environments page.

2. Select Create.

3. On the "Settings" page, provide a name, such as my-env for the environment.

4. For "Select environment source" choose Use existing docker image with optional conda source.

   :::image type="content" source="media/how-to-deploy-online-endpoints/create-environment.png" alt-text="A screenshot showing how to create a custom environment." lightbox="media/how-to-deploy-online-endpoints/create-environment.png":::

5. Select Next to go to the "Customize" page.

6. Copy the contents of the \azureml-examples\cli\endpoints\online\model-1\environment\conda.yaml file from the local copy of the repo you cloned or downloaded earlier.

7. Paste the contents into the text box.

   :::image type="content" source="media/how-to-deploy-online-endpoints/customize-environment-with-conda-file.png" alt-text="A screenshot showing how to customize the environment, using a conda file." lightbox="media/how-to-deploy-online-endpoints/customize-environment-with-conda-file.png":::

8. Select Next until you get to the "Review" page.

9. select Create.

For more information on creating an environment in the studio, see Create an environment.

1. To register the model using a template, you must first upload the model file to an Azure Blob store. The following example uses the az storage blob upload-batch command to upload a file to the default storage for your workspace:

   :::code language="{language}" source="~/azureml-examples-main/deploy-arm-templates-az-cli.sh" id="upload_model":::

2. After uploading the file, use the template to create a model registration. In the following example, the modelUri parameter contains the path to the model:

   :::code language="azurecli" source="~/azureml-examples-main/deploy-arm-templates-az-cli.sh" id="create_model":::

3. Part of the environment is a conda file that specifies the model dependencies needed to host the model. The following example demonstrates how to read the contents of the conda file into environment variables:

   :::code language="azurecli" source="~/azureml-examples-main/deploy-arm-templates-az-cli.sh" id="read_condafile":::

4. The following example demonstrates how to use the template to register the environment. The contents of the conda file from the previous step are passed to the template using the condaFile parameter:

   :::code language="azurecli" source="~/azureml-examples-main/deploy-arm-templates-az-cli.sh" id="create_environment":::

Your deployment configuration uses the registered model that you wish to deploy and your registered environment..

Use the registered assets (model and environment) in your deployment definition. The following snippet shows the endpoints/online/managed/sample/blue-deployment-with-registered-assets.yml file, with all the required inputs to configure a deployment:

blue-deployment-with-registered-assets.yml

:::code language="yaml" source="~/azureml-examples-main/cli/endpoints/online/managed/sample/blue-deployment-with-registered-assets.yml":::

To configure a deployment, use the registered model and environment:

model = "azureml:my-model:1"
env = "azureml:my-env:1"
 
blue_deployment_with_registered_assets = ManagedOnlineDeployment(
    name="blue",
    endpoint_name=endpoint_name,
    model=model,
    environment=env,
    code_configuration=CodeConfiguration(
        code="../model-1/onlinescoring", scoring_script="score.py"
    ),
    instance_type="Standard_DS3_v2",
    instance_count=1,
)

When you deploy from the studio, you'll create an endpoint and a deployment to add to it. At that time, you'll be prompted to provide names for the endpoint and deployment.

To define the endpoint and deployment, this article uses the Azure Resource Manager templates online-endpoint.json and online-endpoint-deployment.json. To use the templates for defining an online endpoint and deployment, see the Deploy to Azure section.

You can specify the CPU or GPU instance types and images in your deployment definition for both local deployment and deployment to Azure.

Your deployment definition in the blue-deployment-with-registered-assets.yml file used a general-purpose type Standard_DS3_v2 instance and a non-GPU Docker image mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest. For GPU compute, choose a GPU compute type SKU and a GPU Docker image.

For supported general-purpose and GPU instance types, see Managed online endpoints supported VM SKUs. For a list of Azure Machine Learning CPU and GPU base images, see Azure Machine Learning base images.

You can specify the CPU or GPU instance types and images in your deployment configuration for both local deployment and deployment to Azure.

Earlier, you configured a deployment that used a general-purpose type Standard_DS3_v2 instance and a non-GPU Docker image mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest. For GPU compute, choose a GPU compute type SKU and a GPU Docker image.

For supported general-purpose and GPU instance types, see Managed online endpoints supported VM SKUs. For a list of Azure Machine Learning CPU and GPU base images, see Azure Machine Learning base images.

When using the studio to deploy to Azure, you'll be prompted to specify the compute properties (instance type and instance count) and environment to use for your deployment.

For supported general-purpose and GPU instance types, see Managed online endpoints supported VM SKUs. For more information on environments, see Manage software environments in Azure Machine Learning studio.

The preceding registration of the environment specifies a non-GPU docker image mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04 by passing the value to the environment-version.json template using the dockerImage parameter. For a GPU compute, provide a value for a GPU docker image to the template (using the dockerImage parameter) and provide a GPU compute type SKU to the online-endpoint-deployment.json template (using the skuName parameter).

For supported general-purpose and GPU instance types, see Managed online endpoints supported VM SKUs. For a list of Azure Machine Learning CPU and GPU base images, see Azure Machine Learning base images.

Next, deploy your online endpoint to Azure.

1. Create the endpoint in the Azure cloud.

   ::: code language="azurecli" source="~/azureml-examples-main/cli/deploy-managed-online-endpoint.sh" ID="create_endpoint" :::

2. Create the deployment named blue under the endpoint.

   az ml online-deployment create -name blue --endpoint $ENDPOINT_NAME -f endpoints/online/managed/sample/blue-deployment-with-registered-assets.yml --all-traffic
   

   The deployment creation can take up to 15 minutes, depending on whether the underlying environment or image is being built for the first time. Subsequent deployments that use the same environment are processed faster.

   [!TIP]

   • If you prefer not to block your CLI console, you can add the flag --no-wait to the command. However, this option will stop the interactive display of the deployment status.

   [!IMPORTANT] The --all-traffic flag in the code az ml online-deployment create that's used to create the deployment allocates 100% of the endpoint traffic to the newly created blue deployment. Though this is helpful for development and testing purposes, for production, you might want to route traffic to the new deployment through an explicit command. For example, az ml online-endpoint update -n $ENDPOINT_NAME --traffic "blue=100".

1. Create the endpoint:

   Using the endpoint you defined earlier and the MLClient you created earlier, you can now create the endpoint in the workspace. This command starts the endpoint creation and returns a confirmation response while the endpoint creation continues.

   ml_client.online_endpoints.begin_create_or_update(endpoint)

2. Create the deployment:

   Using the blue_deployment_with_registered_assets that you defined earlier and the MLClient you created earlier, you can now create the deployment in the workspace. This command starts the deployment creation and returns a confirmation response while the deployment creation continues.

   ml_client.online_deployments.begin_create_or_update(blue_deployment_with_registered_assets)

   [!TIP]

   • If you prefer not to block your Python console, you can add the flag no_wait=True to the parameters. However, this option will stop the interactive display of the deployment status.

   # blue deployment takes 100 traffic
   endpoint.traffic = {"blue": 100}
   ml_client.online_endpoints.begin_create_or_update(endpoint)

Use the studio to create a managed online endpoint directly in your browser. When you create a managed online endpoint in the studio, you must define an initial deployment. You can't create an empty managed online endpoint.

One way to create a managed online endpoint in the studio is from the Models page. This method also provides an easy way to add a model to an existing managed online deployment. To deploy the model named model-1 that you registered previously in the Register your model and environment section:

1. Go to the Azure Machine Learning studio.

2. In the left navigation bar, select the Models page.

3. Select the model named model-1 by checking the circle next to its name.

4. Select Deploy > Real-time endpoint.

   :::image type="content" source="media/how-to-deploy-online-endpoints/deploy-from-models-page.png" lightbox="media/how-to-deploy-online-endpoints/deploy-from-models-page.png" alt-text="A screenshot of creating a managed online endpoint from the Models UI.":::

   This action opens up a window where you can specify details about your endpoint.

   :::image type="content" source="media/how-to-deploy-online-endpoints/online-endpoint-wizard.png" lightbox="media/how-to-deploy-online-endpoints/online-endpoint-wizard.png" alt-text="A screenshot of a managed online endpoint create wizard.":::

5. Enter an Endpoint name that's unique in the Azure region. For more information on the naming rules, see endpoint limits.

6. Keep the default selection: Managed for the compute type.

7. Keep the default selection: key-based authentication for the authentication type. For more information on authenticating, see Authenticate clients for online endpoints.

8. Select Next, until you get to the "Deployment" page. Here, toggle Application Insights diagnostics to Enabled to allow you to view graphs of your endpoint's activities in the studio later and analyze metrics and logs using Application Insights.

9. Select Next to go to the "Code + environment" page. Here, select the following options:

   • Select a scoring script for inferencing: Browse and select the \azureml-examples\cli\endpoints\online\model-1\onlinescoring\score.py file from the repo you cloned or downloaded earlier.
   • Select environment section: Select Custom environments and then select the my-env:1 environment that you created earlier.

   :::image type="content" source="media/how-to-deploy-online-endpoints/deploy-with-custom-environment.png" lightbox="media/how-to-deploy-online-endpoints/deploy-with-custom-environment.png" alt-text="A screenshot showing selection of a custom environment for deployment.":::

10. Select Next, accepting defaults, until you're prompted to create the deployment.

11. Review your deployment settings and select the Create button.

Alternatively, you can create a managed online endpoint from the Endpoints page in the studio.

1. Go to the Azure Machine Learning studio.

2. In the left navigation bar, select the Endpoints page.

3. Select + Create.

   :::image type="content" source="media/how-to-deploy-online-endpoints/endpoint-create-managed-online-endpoint.png" lightbox="media/how-to-deploy-online-endpoints/endpoint-create-managed-online-endpoint.png" alt-text="A screenshot for creating managed online endpoint from the Endpoints tab.":::

This action opens up a window for you to select your model and specify details about your endpoint and deployment. Enter settings for your endpoint and deployment as described previously, then Create the deployment.

1. Use the template to create an online endpoint:

   :::code language="azurecli" source="~/azureml-examples-main/deploy-arm-templates-az-cli.sh" id="create_endpoint":::

2. Deploy the model to the endpoint after the endpoint has been created:

   :::code language="azurecli" source="~/azureml-examples-main/deploy-arm-templates-az-cli.sh" id="create_deployment":::

To debug errors in your deployment, see Troubleshooting online endpoint deployments.

1. Use the show command to display information in the provisioning_state for the endpoint and deployment:

   ::: code language="azurecli" source="~/azureml-examples-main/cli/deploy-managed-online-endpoint.sh" ID="get_status" :::

2. List all the endpoints in the workspace in a table format by using the list command:

   az ml online-endpoint list --output table
   

1. Check the endpoint's status to see whether the model was deployed without error:

   ml_client.online_endpoints.get(name=endpoint_name)

2. List all the endpoints in the workspace in a table format by using the list method:

   for endpoint in ml_client.online_endpoints.list():
       print(endpoint.name)

   The method returns a list (iterator) of ManagedOnlineEndpoint entities.

3. You can get more information by specifying more parameters. For example, output the list of endpoints like a table:

   print("Kind\tLocation\tName")
   print("-------\t----------\t------------------------")
   for endpoint in ml_client.online_endpoints.list():
       print(f"{endpoint.kind}\t{endpoint.location}\t{endpoint.name}")

You can view all your managed online endpoints in the Endpoints page. Go to the endpoint's Details page to find critical information including the endpoint URI, status, testing tools, activity monitors, deployment logs, and sample consumption code:

1. In the left navigation bar, select Endpoints. Here, you can see a list of all the endpoints in the workspace.

2. (Optional) Create a Filter on Compute type to show only Managed compute types.

3. Select an endpoint name to view the endpoint's Details page.

   :::image type="content" source="media/how-to-deploy-online-endpoints/managed-endpoint-details-page.png" lightbox="media/how-to-deploy-online-endpoints/managed-endpoint-details-page.png" alt-text="Screenshot of managed endpoint details view.":::

1. Use the show command to display information in the provisioning_state for the endpoint and deployment:

   ::: code language="azurecli" source="~/azureml-examples-main/cli/deploy-managed-online-endpoint.sh" ID="get_status" :::

2. List all the endpoints in the workspace in a table format by using the list command:

   az ml online-endpoint list --output table
   

Check the logs to see whether the model was deployed without error.

1. To see log output from a container, use the following CLI command:

   :::code language="azurecli" source="~/azureml-examples-main/cli/deploy-managed-online-endpoint.sh" ID="get_logs" :::

   By default, logs are pulled from the inference server container. To see logs from the storage initializer container, add the --container storage-initializer flag. For more information on deployment logs, see Get container logs.

1. You can view log output by using the get_logs method:

   ml_client.online_deployments.get_logs(
       name="blue", endpoint_name=endpoint_name, lines=50
   )

2. By default, logs are pulled from the inference server container. To see logs from the storage initializer container, add the container_type="storage-initializer" option. For more information on deployment logs, see Get container logs.

   ml_client.online_deployments.get_logs(
       name="blue", endpoint_name=endpoint_name, lines=50, container_type="storage-initializer"
   )

To view log output, select the Logs tab from the endpoint's page. If you have multiple deployments in your endpoint, use the dropdown to select the deployment whose log you want to see.

:::image type="content" source="media/how-to-deploy-online-endpoints/deployment-logs.png" lightbox="media/how-to-deploy-online-endpoints/deployment-logs.png" alt-text="A screenshot of observing deployment logs in the studio.":::

By default, logs are pulled from the inference server. To see logs from the storage initializer container, use the Azure CLI or Python SDK (see each tab for details). Logs from the storage initializer container provide information on whether code and model data were successfully downloaded to the container. For more information on deployment logs, see Get container logs.

1. To see log output from a container, use the following CLI command:

   :::code language="azurecli" source="~/azureml-examples-main/cli/deploy-managed-online-endpoint.sh" ID="get_logs" :::

   By default, logs are pulled from the inference server container. To see logs from the storage initializer container, add the --container storage-initializer flag. For more information on deployment logs, see Get container logs.

1. Use either the invoke command or a REST client of your choice to invoke the endpoint and score some data:

   ::: code language="azurecli" source="~/azureml-examples-main/cli/deploy-managed-online-endpoint.sh" ID="test_endpoint" :::

2. Get the key used to authenticate to the endpoint:

   [!TIP] You can control which Microsoft Entra security principals can get the authentication key by assigning them to a custom role that allows Microsoft.MachineLearningServices/workspaces/onlineEndpoints/token/action and Microsoft.MachineLearningServices/workspaces/onlineEndpoints/listkeys/action. For more information on managing authorization to workspaces, see Manage access to an Azure Machine Learning workspace.

   :::code language="azurecli" source="~/azureml-examples-main/cli/deploy-managed-online-endpoint.sh" ID="test_endpoint_using_curl_get_key":::

3. Use curl to score data.

   ::: code language="azurecli" source="~/azureml-examples-main/cli/deploy-managed-online-endpoint.sh" ID="test_endpoint_using_curl" :::

   Notice you use show and get-credentials commands to get the authentication credentials. Also notice that you're using the --query flag to filter only the attributes that are needed. To learn more about the --query flag, see Query Azure CLI command output.

4. To see the invocation logs, run get-logs again.

Using the MLClient created earlier, get a handle to the endpoint. The endpoint can then be invoked using the invoke command with the following parameters:

• endpoint_name - Name of the endpoint
• request_file - File with request data
• deployment_name - Name of the specific deployment to test in an endpoint

1. Send a sample request using a json file.

   # test the blue deployment with some sample data
   ml_client.online_endpoints.invoke(
       endpoint_name=endpoint_name,
       deployment_name="blue",
       request_file="../model-1/sample-request.json",
   )

Use the Test tab in the endpoint's details page to test your managed online deployment. Enter sample input and view the results.

1. Select the Test tab in the endpoint's detail page.

2. Use the dropdown to select the deployment you want to test.

3. Enter the sample input.

4. Select Test.

   :::image type="content" source="media/how-to-deploy-online-endpoints/test-deployment.png" lightbox="media/how-to-deploy-online-endpoints/test-deployment.png" alt-text="A screenshot of testing a deployment by providing sample data, directly in your browser.":::

1. Use either the invoke command or a REST client of your choice to invoke the endpoint and score some data:

   az ml online-endpoint invoke --name $ENDPOINT_NAME --request-file cli/endpoints/online/model-1/sample-request.json
   

If you want to update the code, model, or environment, update the YAML file, and then run the az ml online-endpoint update command.

To understand how update works:

1. Open the file online/model-1/onlinescoring/score.py.

2. Change the last line of the init() function: After logging.info("Init complete"), add logging.info("Updated successfully").

3. Save the file.

4. Run this command:

   az ml online-deployment update -n blue --endpoint $ENDPOINT_NAME -f endpoints/online/managed/sample/blue-deployment-with-registered-assets.yml
   

   [!Note] Updating by using YAML is declarative. That is, changes in the YAML are reflected in the underlying Azure Resource Manager resources (endpoints and deployments). A declarative approach facilitates GitOps: All changes to endpoints and deployments (even instance_count) go through the YAML.

   [!TIP]

   • You can use generic update parameters, such as the --set parameter, with the CLI update command to override attributes in your YAML or to set specific attributes without passing them in the YAML file. Using --set for single attributes is especially valuable in development and test scenarios. For example, to scale up the instance_count value for the first deployment, you could use the --set instance_count=2 flag. However, because the YAML isn't updated, this technique doesn't facilitate GitOps.
   • Specifying the YAML file is NOT mandatory. For example, if you wanted to test different concurrency setting for a given deployment, you can try something like az ml online-deployment update -n blue -e my-endpoint --set request_settings.max_concurrent_requests_per_instance=4 environment_variables.WORKER_COUNT=4. This will keep all existing configuration but update only the specified parameters.

5. Because you modified the init() function, which runs when the endpoint is created or updated, the message Updated successfully will be in the logs. Retrieve the logs by running:

   :::code language="azurecli" source="~/azureml-examples-main/cli/deploy-managed-online-endpoint.sh" ID="get_logs" :::

The update command also works with local deployments. Use the same az ml online-deployment update command with the --local flag.

If you want to update the code, model, or environment, update the configuration, and then run the MLClient's online_deployments.begin_create_or_update method to create or update a deployment.

To understand how begin_create_or_update works:

1. Open the file online/model-1/onlinescoring/score.py.

2. Change the last line of the init() function: After logging.info("Init complete"), add logging.info("Updated successfully").

3. Save the file.

4. Run the method:

   ml_client.online_deployments.begin_create_or_update(blue_deployment_with_registered_assets)

5. Because you modified the init() function, which runs when the endpoint is created or updated, the message Updated successfully will be in the logs. Retrieve the logs by running:

   ml_client.online_deployments.get_logs(
       name="blue", endpoint_name=endpoint_name, lines=50
   )

The begin_create_or_update method also works with local deployments. Use the same method with the local=True flag.

Currently, the studio allows you to make updates only to the instance count of a deployment. Use the following instructions to scale an individual deployment up or down by adjusting the number of instances:

1. Open the endpoint's Details page and find the card for the deployment you want to update.
2. Select the edit icon (pencil icon) next to the deployment's name.
3. Update the instance count associated with the deployment. You can choose between Default or Target Utilization for "Deployment scale type".
   • If you select Default, you cal also specify a numerical value for the Instance count.
   • If you select Target Utilization, you can specify values to use for parameters when autoscaling the deployment.
4. Select Update to finish updating the instance counts for your deployment.

There currently isn't an option to update the deployment using an ARM template.

Autoscale automatically runs the right amount of resources to handle the load on your application. Managed online endpoints support autoscaling through integration with the Azure monitor autoscale feature. To configure autoscaling, see How to autoscale online endpoints.

To view metrics and set alerts based on your SLA, complete the steps that are described in Monitor online endpoints.

The get-logs command for CLI or the get_logs method for SDK provides only the last few hundred lines of logs from an automatically selected instance. However, Log Analytics provides a way to durably store and analyze logs. For more information on using logging, see Monitor online endpoints.

Delete the endpoint and all its underlying deployments:

::: code language="azurecli" source="~/azureml-examples-main/cli/deploy-managed-online-endpoint.sh" ID="delete_endpoint" :::

Delete the endpoint and all its underlying deployments:

ml_client.online_endpoints.begin_delete(name=endpoint_name)

If you aren't going use the endpoint and deployment, you should delete them. By deleting the endpoint, you also delete all its underlying deployments.

1. Go to the Azure Machine Learning studio.
2. In the left navigation bar, select the Endpoints page.
3. Select an endpoint by checking the circle next to the model name.
4. Select Delete.

Alternatively, you can delete a managed online endpoint directly by selecting the Delete icon in the endpoint details page.

Delete the endpoint and all its underlying deployments:

::: code language="azurecli" source="~/azureml-examples-main/cli/deploy-managed-online-endpoint.sh" ID="delete_endpoint" :::

• Perform safe rollout for online endpoints
• Deploy models with REST
• How to autoscale managed online endpoints
• How to monitor managed online endpoints