*Thank you to John Gardner and Shikha Jain from the Azure Monitor Workbooks team for helping with this content*
With Workbooks 202, a sample workbook is provided in the same manner as Workbooks 101. This new workbook will contain examples and samples of each item covered in this document, as well as instructions on how to use the components. Workbooks 202 will cover:
Please note, this blog is very long. If looking to break up the content, please use the workbook as the same content will exist in the workbook for each section. The workbook template can be found here and will be available soon in Microsoft Sentinel.
----------------------------------------------------------------
Oh, you are still here. Let's begin.
Additional Sources
While Azure Log Analytics is the main source of data in a workbook, there are several other options available.
Azure Resource Graph
Azure Resource Graph (ARG) is one valuable source of data for a workbook. ARG presents current information about resources and configurations within the Azure environment that are not recorded as logs within Azure Log Analytics. ARG uses KQL and can be used to grab information such as:
When it comes to utilizing ARG, it is most commonly used for fetching existing resources within the environment. This is used in combination with parameters in order to create items like drop down selectors of resources.
To use ARG in a workbook:
An example query is:
resources
| where type has 'microsoft.insights/workbooks'
| extend DisplayName = properties.displayName
For more information on ARG, please refer to: Azure Workbooks data sources - Azure Monitor | Microsoft Learn.
Azure Resource Manager
Azure Resource Manager (ARM) operates like an API but does not require an authorization token. The source still uses a GET call and the URI is just the regular URI but without ‘https://management.azure.com/’. ARM is great to use when looking to gather information that is not available in a log but can be brought in via API. Some examples may be:
The main differences between ARM and a custom endpoint are:
The response when using ARM will be in JSON format, so the results will need to be parsed via JSONPath to appear in a log format. To format the response:
For more information on parsing, please refer to https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-jsonpath. For more information on ARM as a source, please refer to https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-data-sources#azure-resourc....
Azure Data Explorer
Azure Data Explorer (ADX) is another option for data. This source type will take in the database and table names in order to route to the data. From there, KQL can be used to query the data. An example database is https://dataexplorer.azure.com/clusters/help/databases/Samples.
To use ADX as a source:
An example query is:
StormEvents
| project column_ifexists("Capital", State)
| summarize by State
| order by State asc
For more information on ADX as a source, please refer to https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-data-sources#azure-data-ex....
Merge
Merge queries allow for two or more queries to be merged together in a similar fashion as a join. The queries will be joined together via a shared column key. There are several types of merges:
To use a merge query:
For more information on merges, please refer to Azure Workbooks data sources - Azure Monitor | Microsoft Learn.
External
External locations can be pulled as a data source. The most common source will be from a website that is hosting data. The most common example is GitHub. Two options are available for pulling the data are via custom endpoint as a source or the externaldata operator.
As an example, we will reference a publicly hosted GitHub JSON file at https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/Sample%20Data/CEF/SymantecDLP.json and https://raw.githubusercontent.com/nsacyber/Event-Forwarding-Guidance/master/Events/RecommendedEvents....
For more information on the externaldata operator, please refer to https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/externaldata-operator?pivots=azure....
Custom endpoint as a source is highly valuable as it means the data being brought in does not need to be ingested into Azure Log Analytics and the data can be dynamic over time.
To configure an external source:
An example of an externaldata query is:
externaldata (SourceSystem:string, TimeGenerated: datetime, ReceiptTime: datetime, DeviceVendor:string, DeviceProduct: string, DeviceEventClassID: string, LogSeverity: string, OriginalSeverity: string, DeviceAction:string, SimplifiedDeviceAction: string, Computer: string, CommunicationDirection: string, DeviceFacility:string, DestinationPort:string, Activity:string, AdditionalExtensions:string) [
@'https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/Sample%20Data/CEF/SymantecDLP.json'
] with (format="multijson")
Additionally, external sources can be brought in via API. This will utilize the full URI and require an auth token, unlike with ARM. The API call can be performed like any other API call where the version and token can be placed in the header of the call. If calling a URL that is just a file, no auth token will be needed. To perform this:
An example of parsing a JSON response is:
print events=todynamic({CECall:parsejson})
| extend events = events['Recommended Events to Collect']// First we pivot the data
| mv-apply events on (
extend Category = tostring(bag_keys(events)[0])
| extend Category = tostring(bag_keys(events)[0]), description = events[Category].description, events = events[Category].events
)
| mv-apply events on (
extend eventName = tostring(bag_keys(events)[0])
| extend eventId = tostring(bag_keys(events[eventName])[0])
| extend properties = events[eventName][eventId]
| project-away events
)
| order by eventId asc
For more information about custom endpoints, please refer to Azure Workbooks data sources - Azure Monitor | Microsoft Learn.
Visualizations
Graphs
Graphs are an advanced visualization option within workbooks that depict relations between items and directions of actions if built correctly. This visualization is challenging to create as it requires the user to build out the data, nodes, and relations within the KQL itself before configuring how to the graph will look.
Structure
The structure of the KQL can be summarized as:
Building the query utilizes several let statements in order to store results or components. First, it will need the data portion. This can be as simple as:
let data = SigninLogs | Account = split(UserDisplayName, '@')[0] | summarize by Account, UserDisplayName, IPAddress | join kind = leftouter SecurityEvent on Account;
Once the data is configured, it's time to set the links. This will involve defining what the link relations are, notated with a '->' to signal the direction of the relation. Links are essential for dictating the links between nodes. An example would look like:
let links = data
| summarize by Source = Account, Target = Computer, Kind = 'Account -> Machine';
Once the links are made, it's time for nodes. This will involve defining what the nodes will be. Think of nodes as the plots on a chart in that they will be interconnected via a common data piece. This may look like:
let nodes = data
| summarize by Id = Account, Name = Account, Kind = 'Account'
| union (data
| summarize by Id = Computer, Name = Computer, Kind = 'Machine');
Building
Once the query is done and the results are in, it’s time to configure the graph. To do so:
For more information on graphs, please refer to Azure Workbooks graph visualizations - Azure Monitor | Microsoft Learn.
Linking
Linking other blades and views within Azure is a highly valuable feature within workbooks. This allows users to open other blades and items within the Azure portal without having to leave the workbook. This allows users to do things such as:
The resource can be set dynamically or statically. If desired, resource paths can be inserted manually to make the value static. If looking to switch between different resources, values can be set dynamically via Azure Resource Graph.
Workbooks: Workbooks can be linked within another workbook. This allows users to tie two or more workbooks together if they are related. There are two ways to do so:
Azure Blades: Blades within the Azure Portal can be linked. This item is a bit more advanced as it involves using developer tools in the web browser. This method allows for blades within Azure to be linked, allowing users to pivot to another part of the portal without having to leave the workbook. This also allows for key blades from several Azure services to be linked in one location for ease of use.
External Locations: This method allows for external links to be tied to a button. This allows for key web locations or items to be statically linked within the workbook. While a URL is static, values of the link can be set dynamically via parameters in the workbook.
Ex. https://www.virustotal.com/gui/file/ENTERDYNAMICPARAMETERHERE
For more information on linking, please see https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-link-actions.
Azure Resources
If looking to link an Azure resource, the best way to do so is by leveraging parameters with ARG. The purpose of using these two items is to locate the resource via ARG, then save the resource path via a parameter. This allows for the button linking to the resource to be dynamic. To make this:
Workbooks
Linking workbooks is powerful in that it combines one or more workbooks with the current one that is being used. This can create a chain of functionality and use cases in one convenient location.
Methods
There are two main methods:
Both options are viable options for linking a workbook. Linking workbooks dynamically via the Azure resource method is a little easier while also allowing users to avoid having to pass parameters in the configuration. The workbook (template) action is great to use when statically linking workbooks while providing better performance.
Azure Resource Based
To link via an Azure resource:
Workbook Templates
The Workbook (Template) action allows for workbooks to be linked in a workbook by directing the current workbook to where the template can be found for the other workbook of interest. This action contains several values:
To link via the Workbook (Template) action:
For more information on the Workbook (Template) action, please refer to https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-link-actions#workbook-temp....
Template Based
Externally hosted templates will be a JSON file hosted on another platform, such as GitHub or even from Microsoft Sentinel. Workbooks are able to call upon external template files if directed to the proper location and file name. For this to work, two items are needed:
Examples would be:
An example of mapping the file names to a workbook name would appear as so:
[
{"value": "WorkspaceUsage.json", "label": "Workspace Usage Report", "tags": ["usage"]},
{"value": "AMAmigrationTracker.json", "label": "AMA Agent - Migration Tracker", "tags" : ["migration", "Agents", "deployment"]},
{"value": "MicrosoftSentinelDeploymentandMigrationTracker.json", "label": "Sentinel Deployments and Migration Tracker", "tags": ["migration","deployment"]},
{"value": "ArchivingBasicLogsRetention.json", "label": "Archiving and Basic Logs Retention", "tags": ["retention","archiving","basic logs"]}
]
To link:
For more information on linking workbooks, please refer to Azure Workbooks link actions - Azure Monitor | Microsoft Learn.
Mapping Blade Options
Linking blades allows other blades within Azure to be opened from the workbook without having to leave. There are two main ways to link blades:
Both methods leverage the custom view option, which is made up of 3 components:
URL
To use this method:
Metadata
This method uses metadata from the Azure Portal via developer tools within the web browser. This method is more tedious but proves to be more accurate when linking blades. Additionally, this method allows for sub-blades to be linked. To use this method:
For more information on linking, please refer to Azure Workbooks link actions - Azure Monitor | Microsoft Learn.
Advanced Concepts
Creating Tables from Parameters
Tables can be created from API responses and used to query and modify. Normally, the response will be in a nested JSON format. This can be parsed via KQL and made into a table format that allows for querying. An example of this would be running an ARM call to list all of the deployed Microsoft Sentinel analytic rules or all deployed data collection rules from Azure Monitor.
For parsing, the query will leverage different operators, such as mv-expand, bag_unpack, or pack_all.
To do so:
Creating ARM Templates
ARM templates can be both created and deployed within workbooks. This will involve a combination of parameters, links, and making an editor in the workbook. There are two main ways to go about this:
Parameters
This portion will leverage storing values in parameters in order to make the template generation dynamic. Storing values in parameters allows for the values stored to be changed, which will update the template dynamically.
Links
This portion will leverage creating buttons that will contain the API URI call. If dynamically updating the template, this will also store the main portions of the template.
Editor
This portion will create a parameter that allows for user input and will behave like a barebones JSON editor. The value of this parameter is passed to the API button for deployment.
To make it dynamically:
To make it manually:
Running Automation from a Workbook
There are two main ways to run automation from a workbook:
Linking
This method will leverage the steps highlighted in the linking section. There are a few items that will be needed:
To do this:
API
This method will leverage the steps highlighted in the linking section. There are a few items that will be needed:
To do this:
{
"LogicAppsResourceId":"{Playbook}",
"TenantId":"{TenantId}"
}
10. Once each of the items are populated, make sure that an incident is clicked on.
11. When ready, click the run playbook button.
-------------------------------------------------------------------------------------------
Aaaaaaand exhale. This was a lot of content to cover but hopefully this sparks some inspiration for new, advanced workbooks for your environment or clients. It is recommended to go through the content and review how everything was built so that the magic behind the curtain is fully shown. Again, the workbook template can be found here and will be provided in Microsoft Sentinel soon. Best of luck.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.