logstash-output-azure_loganalytics is a logstash plugin to output to Azure Log Analytics. Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite destinations. Log Analytics is a service in Operations Management Suite (OMS) that helps you collect and analyze data generated by resources in your cloud and on-premises environments. It gives you real-time insights using integrated search and custom dashboards to readily analyze millions of records across all of your workloads and servers regardless of their physical location. The plugin stores in-coming events to Azure Log Analytics by leveraging Log Analytics HTTP Data Collector API
[NOTICE] logstash-output-azure_loganalytics >= 0.6.0
- Multithreading support
- New parame
max_batch_items
is added- No longer
flush_items
andflush_interval_time
params are supported in the plugin configuration
You can install this plugin using the Logstash "plugin" or "logstash-plugin" (for newer versions of Logstash) command:
bin/plugin install logstash-output-azure_loganalytics
# or
bin/logstash-plugin install logstash-output-azure_loganalytics (Newer versions of Logstash)
Please see Logstash reference for more information.
output {
azure_loganalytics {
customer_id => "<OMS WORKSPACE ID>"
shared_key => "<CLIENT AUTH KEY>"
log_type => "<LOG TYPE NAME>"
key_names => ['key1','key2','key3'..] ## list of Key names
key_types => {'key1'=> 'string' 'key2'=>'double' 'key3'=>'boolean' .. }
max_batch_items => <MAX BATCH ITEMS (num)>
}
}
- customer_id (required) - Your Operations Management Suite workspace ID
- shared_key (required) - The primary or the secondary Connected Sources client authentication key.
- log_type (required) - The name of the event type that is being submitted to Log Analytics. It must only contain alpha numeric and _, and not exceed 100 chars. sprintf syntax like
%{my_log_type}
is supported. - time_generated_field (optional) - Default:''(empty string) The name of the time generated field. Be carefule that the value of field should strictly follow the ISO 8601 format (YYYY-MM-DDThh:mm:ssZ). See also this for more details
- key_names (optional) - Default:[] (empty array). The list of key names in in-coming record that you want to submit to Log Analytics.
- key_types (optional) - Default:{} (empty hash). The list of data types for each column as which you want to store in Log Analytics (
string
,boolean
, ordouble
)- The key names in
key_types
param must be included inkey_names
param. The column data whose key isn't included inkey_names
is treated asstring
data type. - Multiple key value entries are separated by
spaces
rather than commas (See also this) - If you want to store a column as datetime or guid data format, set
string
for the column ( the value of the column should beYYYY-MM-DDThh:mm:ssZ format
if it'sdatetime
, andGUID format
if it'sguid
). - In case that
key_types
param are not specified, all columns that you want to submit ( you choose withkey_names
param ) are stored asstring
data type in Log Analytics.
- The key names in
- max_batch_items (optional) - Default 50. Maximum number of log events to put in one request to Log Analytics.
[NOTE] There is a special param for changing the Log Analytics API endpoint (mainly for supporting Azure sovereign cloud)
- endpoint (optional) - Default: ods.opinsights.azure.com
Here is an example configuration where Logstash's event source and destination are configured as Apache2 access log and Azure Log Analytics respectively.
input {
file {
path => "/var/log/apache2/access.log"
start_position => "beginning"
}
}
filter {
if [path] =~ "access" {
mutate { replace => { "type" => "apache_access" } }
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
azure_loganalytics {
customer_id => "818f7bbc-8034-4cc3-b97d-f068dd4cd659"
shared_key => "ppC5500KzCcDsOKwM1yWUvZydCuC3m+ds/2xci0byeQr1G3E0Jkygn1N0Rxx/yVBUrDE2ok3vf4ksXxcBmQQHw==(dummy)"
log_type => "ApacheAccessLog"
key_names => ['logid','date','processing_time','remote','user','method','status','agent']
flush_items => 10
flush_interval_time => 5
}
# for debug
stdout { codec => rubydebug }
}
You can find example configuration files in logstash-output-azure_loganalytics/examples.
Now you run logstash with the the example configuration like this:
# Test your logstash configuration before actually running the logstash
bin/logstash -f logstash-apache2-to-loganalytics.conf --configtest
# run
bin/logstash -f logstash-apache2-to-loganalytics.conf
Here is an expected output for sample input (Apache2 access log):
Apache2 access log
106.143.121.169 - - [29/Dec/2016:01:38:16 +0000] "GET /test.html HTTP/1.1" 304 179 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
Output (rubydebug)
{
"message" => "106.143.121.169 - - [29/Dec/2016:01:38:16 +0000] \"GET /test.html HTTP/1.1\" 304 179 \"-\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36\"",
"@version" => "1",
"@timestamp" => "2016-12-29T01:38:16.000Z",
"path" => "/var/log/apache2/access.log",
"host" => "yoichitest01",
"type" => "apache_access",
"clientip" => "106.143.121.169",
"ident" => "-",
"auth" => "-",
"timestamp" => "29/Dec/2016:01:38:16 +0000",
"verb" => "GET",
"request" => "/test.html",
"httpversion" => "1.1",
"response" => "304",
"bytes" => "179",
"referrer" => "\"-\"",
"agent" => "\"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36\""
}
If you need to debug and watch what this plugin is sending to Log Analytics, you can change the logstash log level for this plugin to DEBUG
to get additional logs in the logstash logs.
One way of changing the log level is to use the logstash API:
> curl -XPUT 'localhost:9600/_node/logging?pretty' -H "Content-Type: application/json" -d '{ "logger.logstash.outputs.azureloganalytcs" : "DEBUG" }'
{
"host" : "yoichitest01",
"version" : "6.5.4",
"http_address" : "127.0.0.1:9600",
"id" : "d8038a9e-02c6-411a-9f6b-597f910edc54",
"name" : "yoichitest01",
"acknowledged" : true
}
You should then be able to see logs like this in your logstash logs:
[2019-03-29T01:18:52,652][DEBUG][logstash.outputs.azureloganalytics] Posting log batch (log count: 50) as log type HealthCheckLogs to DataCollector API. First log: {"message":{"Application":"HealthCheck.API","Environments":{},"Name":"SystemMetrics","LogLevel":"Information","Properties":{"CPU":3,"Memory":83}},"beat":{"version":"6.5.4","hostname":"yoichitest01","name":"yoichitest01"},"timestamp":"2019-03-29T01:18:51.901Z"}
[2019-03-29T01:18:52,819][DEBUG][logstash.outputs.azureloganalytics] Successfully posted logs as log type HealthCheckLogs with result code 200 to DataCollector API
Once you're done, you can use the logstash API to undo your log level changes:
> curl -XPUT 'localhost:9600/_node/logging/reset?pretty'
Bug reports and pull requests are welcome on GitHub at https://github.com/yokawasa/logstash-output-azure_loganalytics.