Azure Open AI is widely used in industry but there are number of security aspects that must be taken into account when using the technology. Luckily for us, Audrey Long, a Software Engineer at Microsoft, security expert and renowned conference speaker, gives us insights into securing LLMs and provides various tips, tricks and tools to help developers use these models safely in their applications.

Media file: https://azpodcast.blob.core.windows.net/episodes/Episode502.mp3

YouTube: https://youtu.be/64Achcz97PI

Resources:

AI Tooling:

1. Azure AI Tooling Announcing new tools in Azure AI to help you build more secure and trustworthy generative AI applications | Microsoft Azure Blog
   • Prompt Shields to detect and block prompt injection attacks, including a new model for identifying indirect prompt attacks before they impact your model, coming soon and now available in preview in Azure AI Content Safety. 
   • Groundedness detection to detect “hallucinations” in model outputs, coming soon. 
   • Safety system messagesto steer your model’s behavior toward safe, responsible outputs, coming soon.
   • Safety evaluations to assess an application’s vulnerability to jailbreak attacks and to generating content risks, now available in preview.  
   • Risk and safety monitoring to understand what model inputs, outputs, and end users are triggering content filters to inform mitigations, coming soon, and now available in preview in Azure OpenAI Service.
2. AI Defender for Cloud
   • AI Security Posture Management AI security posture management (Preview) - Microsoft Defender for Cloud | Microsoft Learn
   • AI Workloads Enable threat protection for AI workloads (preview) - Microsoft Defender for Cloud | Microsoft Learn      
3.  AI Red Teaming Tool
   • Announcing Microsoft’s open automation framework to red team generative AI Systems | Microsoft Security Blog

AI Development Considerations:

1. AI Assessment from Microsoft
   • Conduct an AI assessment using Microsoft’s Responsible AI Impact Assessment Template
   • Responsible AI Impact Assessment Guide for detailed instructions
2. Microsoft Responsible AI Processes
   • Follow Microsoft’s Responsible AI principles: fairness, reliability, safety, privacy, security, inclusiveness, transparency, and accountability
   • Utilize tools like the Responsible AI Dashboard for continuous monitoring and improvement
3. Define Use Case and Model Architecture
   • Determine the specific use case for your LLM
   • Design the model architecture, focusing on the Transformer architecture
   •  
4. Content Filtering System
   • How to use content filters (preview) with Azure OpenAI Service - Azure OpenAI | Microsoft Learn
   • Azure OpenAI Service includes a content filtering system that works alongside core models, including DALL-E image generation models. This system uses an ensemble of classification models to detect and prevent harmful content in both input prompts and output completions
   • The filtering system covers four main categories: hate, sexual, violence, and self-harm
   • Each category is assessed at four severity levels: safe, low, medium, and high
   • Additional classifiers are available for detecting jailbreak risks and known content for text and code. JailBreaking Content Filters
5. Red Teaming the LLM
   • Plan and conduct red teaming exercises to identify potential vulnerabilities
   • Use diverse red teamers to simulate adversarial attacks and test the model’s robustness
   • Microsoft AI Red Team building future of safer AI | Microsoft Security Blog
6. Create a Threat Model with OWASP Top 10
• owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-slides-v1_1.pdf
• Develop a threat model and implement mitigations based on identified risks

Other updates:

• Los Angeles Azure Extended Zones
• Carbon Optimization
• App Config Ref GA
• OS SKU In-Place Migration for AKS
• Operator CRD Support with Azure Monitor Managed Service
• Azure API Center Visual Studio Code Extension Pre-release
• Azure API Management WordPress Plugin
• Announcing a New OpenAI Feature for Developers on Azure

In this episode, we explore the fascinating world of ARM processors on Azure. We're thrilled to have Arpita Chatterjee and Noah Wood, both Program Managers at Azure, join us. They share valuable insights into the new Microsoft ARM processor, currently in Preview, and discuss when and how customers can best leverage these innovative SKUs.

Media file: https://azpodcast.blob.core.windows.net/episodes/Episode501.mp3

YouTube: https://youtu.be/z5GN4-WjiJo

Resources: 

https://aka.ms/Cobalt100-VM-Preview-Blog

https://aka.ms/Cobalt100-VM-Preview-Signup

Other updates:

https://azure.microsoft.com/en-us/updates/v2/convert-to-azure-premium-ssd-v2

https://azure.microsoft.com/en-us/updates/v2/Azure-Monitor-Basic-Logs-Updates

https://azure.microsoft.com/en-us/updates/v2/App-Service-Environment-Retirement-Update

https://azure.microsoft.com/en-us/updates/v2/Azure-Functions-supports-NET8-using-in-process-model

https://azure.microsoft.com/en-us/updates/v2/Redis-Extension-Azure-Functions

https://azure.microsoft.com/en-us/updates/v2/Pgvector-support-in-Azure-Database

https://azure.microsoft.com/en-us/updates/v2/Update-Records-in-Kutso-Databases

The team catches up with Daniel Taylor and Bradley Ball of the Azure Fast Track team and learns about how Database Watcher can help customers better manage and debug their Azure SQL, Elastic Pools, SQL Managed Instances, and Hyperscale and the replicas. 

Daniel and Bradley also share the really cool work they are doing helping Azure customers learn about Azure with their Tales from the Field YouTube channel, plus we hear about some upcoming events where you can see Daniel and Bradley in person!!!

Media file: https://azpodcast.blob.core.windows.net/episodes/Episode500.mp3

YouTube: https://youtu.be/Z_jBWLUBY9g

Resources: 

https://learn.microsoft.com/en-us/azure/azure-sql/database-watcher-overview?view=azuresql

https://youtu.be/qr1Qxwao68M?si=mO24KZF3PAvTnnmn

Other updates:

https://azure.microsoft.com/en-us/updates/v2/Guest-OS-Family-Retirement

https://azure.microsoft.com/en-us/updates/v2/cmk-for-backup-vaults-ga/

https://azure.microsoft.com/en-us/updates/v2/Azure-Elastic-SAN-Feature-Updates

Azure classic resource providers will be retired on 31 August 2024 | Azure updates | Microsoft Azure

Prashant Nayak, a Cloud Solutions Architect at Microsoft, gives us a quick background on AVS and enlightens us with the latest updates in migration and licensing to support customers wanted to move to Azure.

Media file: https://azpodcast.blob.core.windows.net/episodes/Episode499.mp3

YouTube: https://youtu.be/ut-iUJZXtI4

Resources:

• Microsoft and Broadcom to support license portability for VMware Cloud Foundation on Azure VMware Solution | Microsoft Azure Blog
• Introducing the VMware Rapid Migration Plan | Microsoft Azure Blog
• Tutorial - Network planning checklist - Azure VMware Solution | Microsoft Learn
• Introduction to the Azure VMware Solution adoption scenario - Cloud Adoption Framework | Microsoft Learn
• Enterprise-scale network topology and connectivity for Azure VMware Solution - Cloud Adoption Framework | Microsoft Learn
• Run VMware resources on Azure VMware Solution - Training | Microsoft Learn
• Azure VMware Solution Documentation Hub - Azure VMware Solution | Microsoft Learn
• Azure VMware Solution Videos - YouTube (Videos)

Other updates:

https://techcommunity.microsoft.com/t5/azure-observability-blog/advanced-network-observability-for-your-azure-kubernetes-service/ba-p/4176736

https://azure.microsoft.com/en-us/updates/v2/Databricks-Jobs-Serverless-Compute

https://azure.microsoft.com/en-us/updates/v2/interactive-serverless-compute-AU

https://azure.microsoft.com/en-us/updates/v2/DLT-Serverless-Compute 

https://azure.microsoft.com/en-us/updates/v2/Cross-Subscription-Load-Balancer

We get an update on the fascinating topic of Quantum Computing on Azure from Cloud Solutions Architect and Quantum enthusiast Delbert Murphy. He gives us the latest status on Quantum, Quantum Elements and the Q# programming language and provides guidance on the type of use-cases that can take advantage of these services.

Media file: https://azpodcast.blob.core.windows.net/episodes/Episode498.mp3

YouTube: https://youtu.be/2bZD2Ns91O8

Resources:

https://www.microsoft.com/en-us/research/blog/microsoft-has-demonstrated-the-underlying-physics-required-to-create-a-new-kind-of-qubit/ 

https://cloudblogs.microsoft.com/quantum/2024/04/03/how-microsoft-and-quantinuum-achieved-reliable-quantum-computing/ 

https://cloudblogs.microsoft.com/quantum/2024/05/30/in-collaboration-with-microsoft-photonic-demonstrates-quantum-entanglement-at-telecom-wavelengths/

https://www.qir-alliance.org/

https://github.com/microsoft/qsharp/

https://learn.microsoft.com/en-us/azure/quantum/hybrid-computing-overview

https://quantum.microsoft.com/en-us/experience/quantum-katas

https://devblogs.microsoft.com/qsharp/design-fault-tolerant-quantum-computing-applications-with-the-open-source-resource-estimator/

https://azure.microsoft.com/en-us/solutions/quantum-innovator-series/

https://quantum.microsoft.com/en-us/quantum-elements/product-overview

https://cloudblogs.microsoft.com/quantum/2024/06/18/introducing-two-powerful-new-capabilities-in-azure-quantum-elements-generative-chemistry-and-accelerated-dft/

https://cloudblogs.microsoft.com/quantum/2024/01/09/unlocking-a-new-era-for-scientific-discovery-with-ai-how-microsofts-ai-screened-over-32-million-candidates-to-find-a-better-battery/

https://cloudblogs.microsoft.com/quantum/2024/02/29/microsoft-and-1910-genetics-partner-to-turbocharge-rd-productivity-for-the-pharmaceutical-industry/

https://blogs.microsoft.com/blog/2024/06/18/empowering-every-scientist-with-ai-augmented-scientific-discovery/

https://quantum.microsoft.com/en-us/quantum-elements/azure-quantum-elements 

https://quantum.microsoft.com/en-us/experience/quantum-cryptography

Other Updates:

Generally Available: az command invoke in AKS

https://learn.microsoft.com/en-us/azure/aks/access-private-cluster?tabs=azure-cli

Public Preview: Cluster operation status for AKS

https://learn.microsoft.com/en-us/azure/aks/manage-abort-operations?tabs=azure-cli

Public Preview: Geo-Replication for Azure Service Bus Premium

https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-geo-replication

Public Preview: Force detach zone redundant data disks during zone outage

 https://techcommunity.microsoft.com/t5/azure-updates/week-of-june-20-2024-azure-updates/ba-p/4173016

How hollow core fiber is accelerating AI

https://azure.microsoft.com/en-us/blog/how-hollow-core-fiber-is-accelerating-ai/

Rishabh Tewari, a Software Engineer on the Azure team, gives us insights into Project Sirius. This is an upgrade to the Azure software that lights up Accelerated Connections for any resource that uses a network interface. Using a specialized SDN, customers and ISVs can achieve very high number of connections per second throughput across all of their Azure and non-Azure resources.

Media file: https://azpodcast.blob.core.windows.net/episodes/Episode497.mp3

YouTube: https://youtu.be/ZVgBBrRlZGQ

Resources:

https://www.usenix.org/system/files/nsdi23-bansal.pdf

Other updates:

https://azure.microsoft.com/en-us/updates/general-availability-audit-logging-in-azure-api-management-developer-portal/ 

https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-analytics-simple-mode

Cale, Evan and Sujit have an insightful discussion with Niko Pamboukas and Siri Velauthapillai about the new Azure Boost feature that is being deployed into the Azure substrate for VMs. They explain the specialized FPGAs and ARM SoCs that provide extreme IOPs for storage and networking and almost no downtime for system upgrades.

Media file: https://azpodcast.blob.core.windows.net/episodes/Episode496.mp3

YouTube: https://youtu.be/4FPQCAmYrzw

Resources:

Learn more about Azure Boost at https://aka.ms/AzureBoost

Learn about accelerated networking with Boost at https://aka.ms/MANA

Learn about accelerated storage with Boost at https://aka.ms/NVMe

Other updates:

Public preview: Azure Load Balancer now supports Admin State | Azure updates | Microsoft Azure

https://azure.microsoft.com/en-us/updates/public-preview-azure-bastion-premium/

https://azure.microsoft.com/en-us/updates/kubernetesmetadataandlogsfilteringpublicpreview/

https://azure.microsoft.com/en-us/updates/public-preview-azure-load-balancer-health-event-logs/

https://azure.microsoft.com/en-us/updates/generally-available-azure-chaos-studio-supports-a-new-pause-process-fault-for-windows-virtual-machines/

Hearing how our ISVs are using Azure in innovative ways is always eye-opening. So, we are fortunate to have Saquib Rashid, a Technical Strategist in the Azure partner organization, give us insights into how Backbase used APIM and the Azure Service Operator to integrate with their customers and backend systems.

Media file: https://azpodcast.blob.core.windows.net/episodes/Episode495.mp3

 YouTube: https://youtu.be/qFwOoSNpDME

Resources:

• Backbase Grand Central
• Microsoft Customer Story-Backbase seeks to usher in a new era of engagement banking with its Grand Central platform and Azure API Management
• APIM links: Azure API Management - v2 tiers | Microsoft Learn
• ASO links:
• Azure/azure-service-operator: Azure Service Operator allows you to create Azure resources using kubectl (github.com)
• Azure Service Operator v2

Other updates:

Public Preview - Azure Compute Fleet | Azure updates | Microsoft Azure

https://azure.microsoft.com/en-us/products/copilot/#Usecases

https://techcommunity.microsoft.com/t5/azure-database-for-postgresql/gen-ai-simplified-the-azure-ai-extension-now-generally-available/ba-p/4146139

https://techcommunity.microsoft.com/t5/azure-sql-blog/microsoft-copilot-in-azure-extends-capabilities-to-azure-sql/ba-p/4075408

We are fortunate to have Vlad Joanovic to give us the valuable insights into ARM. He shares the importance of ARM in Azure, provides details on performance and reliability investments and gives us tips on how to make effective use of it to deploy and manage your workloads in Azure.

Media file: https://azpodcast.blob.core.windows.net/episodes/Episode494.mp3

YouTube: https://youtu.be/kaeXdi2khTc

Resources:

https://techcommunity.microsoft.com/t5/azure-tools-blog/azure-cli-and-powershell-tools-build-2024-announcement/ba-p/4148584

https://learn.microsoft.com/en-us/azure/governance/resource-graph/changes/view-resource-changes

https://techcommunity.microsoft.com/t5/azure-governance-and-management/what-s-new-across-azure-governance-services-microsoft-build-2024/ba-p/4150428

Other updates:

Public preview: Azure Log Analytics enhances resilience with workspace replication across regions | Azure updates | Microsoft Azure

Update on Inter-Availability Zone Data Transfer Pricing | Azure updates | Microsoft Azure

General availability: ExpressRoute Seamless Gateway Migration | Azure updates | Microsoft Azure

App Service Environment version 1 and version 2 will be retired on 31 August 2024 | Azure updates | Microsoft Azure

https://azure.microsoft.com/en-us/updates/public-preview-azure-application-gateway-v2-basic-sku/

https://azure.microsoft.com/en-us/updates/cloud-services-classic-retirement-announcement-may2024/

https://azure.microsoft.com/en-us/updates/gpt4-turbo-vision/

https://azure.microsoft.com/en-us/updates/public-preview-sensitive-data-protection-for-azure-front-door-web-application-firewall/

The team catches up with Priyanka Khare to learn about recent advancements in Azure Advisor and how customers can use Advisor to help ensure their workloads are more stable.

Media file: https://azpodcast.blob.core.windows.net/episodes/Episode493.mp3

YouTube: https://youtu.be/Y9LVYhSiGgE

Resources:

Azure Advisor: Advisor - Microsoft Azure

Advisor Workbooks: Advisor Workbooks

Advisor WAF Assessments: Advisor WAF Assessments

Advisor Recommendations: Advisor Recommendations

Advisor Manage Reviews: Advisor Manage Reviews

What's new on Azure Advisor: What's new in Azure Advisor - Azure Advisor | Microsoft Learn

Engage with Microsoft for a Resiliency Review: Azure Advisor resiliency reviews - Azure Advisor | Microsoft Learn