Welcome back to our detailed guide meant to help you troubleshoot the plethora of agents available to you as part of your broader security and monitoring strategy!
In our last article, we covered the design matrix to help you select which agents and product plans you need to enable to achieve your goals. In this blog, we will cover the onboarding to Arc, networking, and more advanced troubleshooting steps for errant agents. Make sure you stay tuned for the remainder of the series for some handy scripts to help you deploy Defender for Servers per server and at scale!
What's in this blog:
- Onboarding to Arc
- Arc Networking Deep Dive
- Private Link Set up
- Preparation for Defender for Servers
Prepare for Arc Agent Onboarding
Perhaps, after reading Pt. 1 of the series, you’ve decided that you would like the server management capabilities of Arc and would like to enable it on your servers before you onboard them to Defender for Servers as well. That way you will be able to monitor and push agents/extensions to your servers across Azure, on-premises, and other clouds. Let’s get started with onboarding to Arc!
First, ensure the Azure subscription in which you will be onboarding the Arc servers has the following Azure Resource Providers enabled:
- Microsoft.HybridCompute
- Microsoft.GuestConfiguration
- Microsoft.HybridConnectivity
After enabling the required resource providers, review the list of supported OS to also ensure that your device can be Arc-enabled.
NOTE: Arc does not run-on 32-Bit systems.
Once you’ve confirmed your operating system is supported by Arc, review the networking requirements and make sure all the endpoints are reachable. You may need to whitelist those URLs in your firewall. The Arc service uses Traffic Manager, a DNS-based load balancer, to route you to the appropriate regional URLs for Arc (and the related services needed for the Arc agent to function), but that does not limit you from onboarding your on-premises servers to the Azure region of your choice. Thus, you may see a response from one region when you try to ping the Arc URLs, but you can certainly choose a different region in which to create your Arc resources, as long as you’ve whitelisted the URLs for that particular region in your firewall.
You have three network connection options for Arc-enabled servers, and each level introduces a different level of complexity:
- You can connect to the Azure Arc service over Public Internet using HTTPS (TCP/443) and SSL/TSL. Usually, this is suitable for PoC, but for any enterprise, production Arc deployment, Private Link is strongly encouraged.
- You can connect via proxy. Be sure to allow the following Azure Service tags on the Virtual Network in Azure:
- AzureActiveDirectory
- AzureTrafficManager
- AzureResourceManager
- AzureArcInfrastructure
- Storage
- WindowsAdminCenter (if using Windows Admin Center to manage Arc-enabled servers)
- You can connect via Private Link. If you are planning on using Private Link setup with Azure Arc, you can allow it to deploy the default Private DNS labels and then set up DNS forwarding so that your on-premise resolution translates it.
- Azure Private Link DNS - Microsoft Community Hub
- Design your Azure Private Link setup - Azure Monitor | Microsoft Learn
- Configure your private link - Azure Monitor | Microsoft Learn
Deployment options for Azure Arc
Installing the Azure Arc Connected Machine agent on a non-Azure server is at no cost to you. Once you start using any Guest Configuration policies, there is a cost of $6/server. Guest Policy Configuration and service provides the ability to govern the Arc-enabled server as though it were in Azure. This allows you to use Guest OS policy and perform Azure Policy compliance evaluation and remediation. Once you’ve enabled Arc on your non-Azure servers, it allows Defender for Cloud to discover them and gives Defender for Cloud the capability to onboard Defender for Servers.
For more information, see: Understand machine configuration assignment resources.
Per-server Deployment Method: This requires that you provide the Azure Connected Machine Onboarding role (Role Based Access Control - RBAC) to the entity who will be onboarding the server to Arc. This process requires interactive authentication to Azure to onboard servers and can serve as a test instance or future state based on your requirement to have servers onboarded weekly; it is not suitable for deployment at scale.
Service Principal Deployment Method: This is an Azure Enterprise Application/Application Registration with the Azure Connected Machine Onboarding role assigned to it. This is used in an unattended process to deploy the Arc agent at scale. The Application Registration itself doesn’t expire, but rather the certificate it uses to authenticate will, so you should set that expiration date accordingly. Learn more about adding servers with a service principal.
Deployment of Azure Arc
Click Azure Arc => Machines => +Add/Create
You will then see the following deployment options:
Things to be on the lookout for during and after Arc deployment:
There was an anomaly with the gcservice log file, and the gcservice (Guest Configuration service) would sometimes surpass the CPU utilization percentage at which it is typically capped. The Guest Configuration service evaluates Azure Policy on your Arc servers. This issue should be resolved and no longer present, but it is recommended you stay up-to-date with the latest Arc agent updates and monitor agent CPU usage in case an abnormal spike is observed. You can use VM Insights to monitor your Arc servers’ performance metrics and set up alerts.
For the Application Registration you used for Machine onboarding, the expiration of the client secret can be set to expire to preset defaults of 1 day, 1 week, 1 month, but you do have the ability to modify this and set a custom expiration date. Once the secret expires, the Service Principal will not have the ability to onboard another machine, until you provision a new one.
To check if the Secret for your Arc onboarding Service Principal has expired, navigate to your Entra tenant => Select App Registration.
When it expires, you can simply select the Application Registration => Certificates & Secrets => select Client Secret => + New Client Secret, which will allow you to create a new secret for the application and put it back into operation.
The most secure approach to managing a client secret for an application with this level of authority is to manage the client secret with Azure Key Vault.
Private Link Scope configuration for Azure Arc and Azure Monitor
If you are new to the concept of Private Link, review this guide. For a short primer on Private Link in the context of Log Analytics workspace (this would be the Azure Monitor use case), see: Use Azure Private Link to connect networks to Azure Monitor - Azure Monitor | Microsoft Learn.
Now that you have chosen Private Link as your network connection option for Azure Arc/Azure Monitor, you will need a method of resolving the DNS entry from on-premises to Azure. To achieve this resolution over your private network, Azure needs a Private DNS Resolver.
Azure DNS Private Resolver is a new service that enables you to query Azure DNS private zones from an on-premises environment and vice versa without deploying VM based DNS servers. You can leverage a previously provisioned Virtual Network (VNET) for this configuration or build one. On the VNET add a subnet for inbound and for outbound traffic that will be resolved by the DNS Resolver.
Once that step is complete now navigate to Azure Arc and provision your Arc enablement script using the Private Endpoint option.
Associate your Private Endpoint with your modified or newly created VNET. Choose the Inbound Subnet you previously created. The guided deployment method will offer you an opportunity to use integrated private DNS zone.
This will provide you with three private DNS configurations for the Azure Arc service.
You complete the process of downloading the script to deploy in your desired process. The Deployment script now has an entry specifying the use of private link scope.
The deployment also provided Azure Arc Private Link Scope, Private Endpoint and Network Interface for the service connection.
For an extensive breakdown of Azure Private DNS and how we route it, review Azure Private Link DNS - Microsoft Community Hub.
On the DNS Private Resolver, the on-premises Conditional Forwarder entries you previously created will map to the Inbound endpoints of IP address, which is automatically provisioned. All the URLs will use this same IP.
After configuring the Private Link Scope for Arc in Azure, you should modify your DNS forwarder on-premises as follows: add these conditional forwarder settings for the inbound IP address for the Private DNS Resolver based on the Azure Services DNS zone Configuration. The intent is to avoid HTTPS 443 over the internet (red lines) and move all traffic from on-premises into internal routing channels (green lines). Please see the required conditional forwarder settings below.
Note: For your Arc-enabled servers to forward data to Log Analytics over Private Link, you will need to modify the FQDNs for Azure Monitor as well.
Azure Arc
|
Azure equivalent resolution |
On-premises Conditional Forwarder |
|
privatelink.his.arc.azure.com |
his.arc.azure.com |
|
privatelink.guestconfiguration.azure.com |
guestconfiguration.azure.com |
|
privatelink.dp.kubernetesconfiguration.azure.com |
dp.kubernetesconfiguration.azure.com |
Azure Monitor
|
Azure equivalent resolution |
On-premises Conditional Forwarder |
|
privatelink.monitor.azure.com |
monitor.azure.com |
|
privatelink.oms.opinsights.azure.com |
oms.opinsights.azure.com |
|
privatelink.ods.opinsights.azure.com |
ods.opinsights.azure.com |
|
privatelink.agentsvc.azure-automation.net |
agentsvc.azure-automation.net |
|
privatelink.blob.core.windows.net |
blob.core.windows.net |
Thank you for reading! Stay tuned for Part 3, focusing entirely on deployment of Defender for Servers.