Detect scene changes in remote areas with AWS IoT Events and Amazon SageMaker

Organizations with large numbers of assets need to monitor their physical and operational health, in order to detect issues and act upon them. This post covers the use case of a fictitious industrial organization AcmeDrone that uses drone devices to inspect assets periodically such as infrastructure components like valves, oil/gas pipelines or power transmission lines, located in hard-to-access areas. These inspections consist of having drone devices capture scene images of the assets and validating them against a machine learning model that detects changes to the asset, such as physical damage or physical obstacles that may affect the proper asset’s operation.

In this post, we provide an operational overview of the aforementioned asset inspection solution, and then describe how to set up the applicable AWS IoT services:

• AWS IoT Events to manage the asset’s geolocation route
• Amazon SageMaker to detect scene changes
• Amazon S3 and AWS Lambda function to upload asset images and inspect them
• Amazon Simple Notification Service (SNS) to receive notifications of scene changes
• AWS IoT Core for devices’ communication with AWS

Solution overview

Consider a scenario where a drone device must follow a specific route to inspect various assets within a large industrial facility. During the inspection, the drone device would stop at each location on the route, in order to capture scene images of the assets. Those images will be sent to AWS and used to detect scene changes. If significant changes are detected, the solution notifies the asset operators.

The drone device uses AWS IoT Core to authenticate with AWS and uses the AWS IoT Device SDK for Python.

This asset inspection solution involves the following sequence of data and message exchanges:

1. The drone device publishes a notification to AWS IoT Core over a MQTT topic, in order to start the inspection route.
2. The AWS IoT Core rules engine retrieves the notification from the MQTT topic.
3. The AWS IoT Core rules engine sends the notification to AWS IoT Events.
4. AWS IoT Events has a detector model that monitors incoming IoT events (e.g. drone device notifications) to start the inspection route (e.g. move the device to the recording scene) by sending a request back to the AWS IoT Core MQTT topic.
5. The device reads the request from the MQTT topic.
6. The device physically moves to a recording area and captures the scene image of the asset.
7. The device uploads the scene image to an Amazon S3 bucket.
8. Upon the scene image upload in the bucket, an AWS Lambda function gets executed.
9. The AWS Lambda function requests Amazon SageMaker to detect scene changes in the image uploaded by validating it against a model defined and deployed in Amazon SageMaker.
10. If scene changes were detected, the AWS Lambda function sends a notification message to an Amazon Simple Notification Service (SNS) topic.
11. The end-users subscribed to the SNS topic receive a notification message to inform them of the asset scene changes.

Pre-requisites

To follow along and set up the asset inspection solution, be sure to have the following:

• An AWS account.
• A device or laptop/computer with an access to your AWS account, Python installed, and the AWS IoT Device SDK for Python.

To set up AWS IoT Events to manage drone device route

In AWS IoT Events, we create the following components to start the drone device’s route:

• One input representing the notification, sent by the drone device to the AWS Cloud.
• A detector model with two states (and two transitions). Upon receipt of a notification, the device state is changed to “Area1”. This state is used to trigger the drone to move to the recording area.

The following sections explain how the above components are implemented and how to configure AWS IoT Core to forward the device notification to AWS IoT Events.

To create an input in AWS IoT Events

You can create an input in AWS IoT Events by following the guide to create an input. In our example, we create an input with the following details:

• An Input name set to “deviceNotifInput”.
• An example JSON payload/event with the content below:

{
  "deviceID": "1",
  "notifType": "StartRoute",
  "dateTm": "05/26/2021  08:03:20.200"
}

To create and publish a detector model in AWS IoT Events

In our example, we create a detector model with the following details:

• Two states (Stopped and Area1).
• Two transitions (StartTransition and StopTransition) in order to transition the device from one state to another.
• Upon the reception of a notification from the device, the start transition (StartTransition) triggers the sending of a notification to the outbound AWS IoT Core’s MQTT topic (See Steps 1 to 4 in previous section for further details) and changes the device state from Stopped to Area1.

To create a detector model, you can download this sample detector model file and import it into AWS IoT Events by following these steps:

1. Create an IAM role for the Detector Model. For more information, see the documentation for setting up permissions for AWS IoT Events.
2. Download the sample detector model file from the repository at this location.
3. Update the file with the Detector Model IAM role’s ARN (replace value “#detectorModelRoleArn#”with a value in the following format: arn:aws:iam::account:role/service-role/detectorRoleName).
4. Go to AWS Management Console and select AWS IoT Events.
5. Select Create detector model and select Import detector model.
6. Select Import, select the file from your local system and select Open. This will create your Detector Model.
7. Once the Detector Model is created, you can publish it by following the guide Create a Detector Model. In our example, the detector generation model is set to “Create a detector for each key value” with a key set to the deviceID. The Detector evaluation model is set to “Batch evaluation” as shown in the below screenshot.

To configure AWS IoT Core rules

An AWS IoT Core rule needs to be configured to forward device data from AWS IoT Core (MQTT topic) to AWS IoT Events.

1. Go to the AWS Management Console and select AWS IoT Core.
2. Select Act, then Rules, and then select the Create button.
3. Set the Name for the rule and set the rule query statement to SELECT * FROM ‘deviceInboundTopic’.
4. Select “Add action”.
5. Select Send a message to an IoT Events Input and then select Configure action as shown in the below screenshot.
6. Select the Input previously created.
7. Select Create Role and enter a role name.
8. Select Add action.
9. Select Create rule.

To set up Amazon SageMaker to detect scene changes

Amazon SageMaker builds a model that evaluates if an asset (i.e. scene image of the asset) changed.

To create an Amazon SageMaker model, you can download this notebook and follow the steps:

1. Create an Amazon SageMaker Notebook Instance by following the steps here: Create an Amazon SageMaker Notebook Instance.
2. Once done, select your notebook instance by choosing Open JupyterLab for the JupyterLab interface.
3. Select the Upload Files icon as shown in the following screenshot:
4. Select this this notebook (previously downloaded) in your file system.
5. Once the file is uploaded, select the Create a new folder icon (as shown in the following screenshot), and set the folder name to “images.”
6. Go into the images folder and upload images from this location.
7. On the Run tab, select Run All Cells. Once done, your model endpoint should be visible in the Amazon SageMaker console.
8. Go to the Amazon SageMaker console.
9. Select Inference and and select Endpoints to confirm the availability of your model endpoint.

To upload asset images in Amazon S3 and inspect them with AWS Lambda

When a device captures a scene image, it is uploaded to an Amazon S3 bucket. Then, an AWS Lambda function gets triggered in order to evaluate the image against the Amazon SageMaker model. If changes are detected, the Lambda function publishes a notification to an Amazon Simple Notification Service (SNS) topic.

In order to set up this solution, follow these steps:

Amazon Simple Notification Service setup:

1. Create a SNS topic. For more information, see the Amazon Simple Notification Service Developer Guide and, more specifically, the documentation of the CreateTopic operation in the Amazon Simple Notification Service API Reference.
2. Subscribe to a SNS topic by following the guide To Subscribe an Endpoint to an Amazon SNS Topic Using the AWS Management Console.

AWS Lambda setup:

1. Create an IAM policy with the below JSON policy document. For more information on the procedure, please refer to the steps in the IAM user guide.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sagemaker:InvokeEndpoint"
],
"Resource": [
"arn:aws:sagemaker:region:account_number:endpoint/#endpoint_name#"
]
},
{
"Effect": "Allow",
"Action": [
"sns:Publish*"
],
"Resource": [
"sns_topic_arn"
]
}
]
}

1. • Note: In the above policy, replace the content in bold with a valid region, account number, Amazon SageMaker endpoint name (Example: linear-learner-2021-01-01-22-39-49-363) and the SNS topic’s ARN (format: arn:aws:sns:region:account:snsTopicName)
2. Create an IAM role for your AWS Lambda function as described in this user guide.
   • When prompted to select permissions, choose the policy previously created.
   • Choose the Amazon managed policy AmazonS3ReadOnlyAccess.
3. Create an AWS Lambda function as described in creating a Lambda function.
   • For the Runtime, choose a Python version (Example: Python 3.8).
   • For the Role, choose the IAM previously created.
4. Once the AWS Lambda function is created, do the following:
   • Under the Configuration tab, go to General Configuration, select the Edit button to set the Memory to 256 (MB) and the Timeout to 1 minute. Then select the Save button.
   • Under the Code tab, go to the Layers section and select the Add a layer button.
   • Select AWS Layers in the Layer source field, choose the AWS Lambda SciPy Layer for Python option.
   • Select a version in the Version field and select the Add button.
5. Create a directory on your computer, download the AWS Lambda source code available at this location and copy it in the directory created.
   • In this directory, open the lambda_function.py file and add a valid Amazon SageMaker Endpoint name (#SAGEMAKER_ENDPOINT_NAME#) and a valid SNS topic’s ARN (#SNS_TOPIC_ARN#). Once done, save the file.
   • Run the command: pip3 install –upgrade -r requirements.txt -t . —no-dependencies
   • Select all the files and folders in this directory and create a zip file.
   • Note: The Pip version version must match the AWS Lambda’s Python version (Example: Pip 3.8 for Python 3.8).
6. In the AWS Lambda console, go to the Code Source section, select the Upload From button, select the Upload button, select your zip file on your computer, and then select the Save button.

Amazon S3 setup:

1. Create an Amazon S3 bucket as described in this user guide.
2. Create an Amazon S3 event notification in order to trigger an AWS Lambda function when a new image is uploaded in the Amazon S3 bucket. See the documentation to enable event notifications:
   • In the Event types section, select the “All object create events” event type option (“s3:ObjectCreated:*”).
   • In the Destination section, select Lambda function as a Destination type and specify the ARN of the AWS Lambda function previously created.

To set up AWS IoT Core for device connection

Create device credentials

You can create device credentials by following the guides below. Once completed, you will have a certificate, private key, public key and root CA certificate.

1. Create an AWS IoT Core Policy
2. Create Device Credentials and attach IoT Core Policy to a Device Certificate.
3. Attach the following policy to the device certificate.

Note: In the following policy, replace the content in bold with a valid region and AWS account ID.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iot:Connect",
"Resource": "arn:aws:iot:region:account:client/clientid"
},
{
"Effect": "Allow",
"Action": ["iot:Publish", "iot:Receive"],
"Resource": "arn:aws:iot:region:account:topic/deviceInboundTopic"
},
{
"Effect": "Allow",
"Action": ["iot:Publish", "iot:Receive"],
"Resource": "arn:aws:iot:region:account:topic/deviceOutboundTopic"
},
{
"Effect": "Allow",
"Action": ["iot:Subscribe"],
"Resource": "arn:aws:iot:region:account:topicfilter/*"
}
]
}

Connect to AWS IoT Core

Once the device credentials are created, you can now connect your IoT device to AWS IoT Core and publish the device notification by following the steps outlined.

Note: you must install the Python SDK (visit this page) and the AWS IoT Device SDK (visit this page) to run the following code snippets.

import boto3
import json
import logging
import time
from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClient
logging.basicConfig(filename='pythonIotDevice.log', filemode='w', format='%(name)s - %(levelname)s - %(message)s',level=logging.DEBUG)
logger = logging.getLogger('pythonIotDevice')
logger.info("pythonIotDevice")

#Connection to the AWS IoT Core with Root CA certificate and unique device credentials (keys and certificate) previously retrieved
# For certificate based connection
myMQTTClient = AWSIoTMQTTClient("clientid")

# For TLS mutual authentication
myMQTTClient.configureEndpoint("your.iot.endpoint", 8883) #Provide your AWS IoT Core endpoint (Example: "abcdef12345-ats.iot.us-east-1.amazonaws.com")
myMQTTClient.configureCredentials("/root/ca/path", "/private/key/path", "/certificate/path") #Set paths for Root CA, for private key and private certificate

myMQTTClient.configureOfflinePublishQueueing(-1)
myMQTTClient.configureDrainingFrequency(2)
myMQTTClient.configureConnectDisconnectTimeout(10)
myMQTTClient.configureMQTTOperationTimeout(5)
logger.info("Connecting...")
myMQTTClient.connect()

#Publish device notification to AWS IoT Core
deviceId = 1
timestamp = time.time();
logger.info("Publishing...")
myMQTTClient.publish("deviceInboundTopic", "{\"deviceID\":\"" + str(deviceId) + "\",\"notifType\":\"StartRoute\",\"dateTm\":\""+ str(timestamp) +"\"}", 0)

def handleResponse(client, userdata, message):
jsonMessage = json.loads(message.payload)
logger.info('jsonMessage=%s', jsonMessage)
logger.info("Subscribing...")
myMQTTClient.subscribe("deviceOutboundTopic", 1, handleResponse);

#Wait until reception of subscription confirmation (wait 60 seconds)
time.sleep(60)
logger.info("Disconnecting...")
myMQTTClient.disconnect()

Connect to Amazon S3

In order for your device to upload images directly into Amazon S3, AWS IoT Core has a credentials provider that allows you to use the built-in X.509 certificate as the unique device identity to authenticate AWS requests. This eliminates the need to store an access key ID and a secret access key on your device.

To set it up, please refer to the documentation for further information on the procedure. Attach the following IAM policy to the IAM role assumed by the credentials provider.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::#bucket_name#/*"
}
]
}

Note: Replace the content in bold with the bucket name previously created.

The following code snippet only shows how to upload an image to Amazon S3. To run the code snippet, download one of the images from this location. Then, specify its name and path as indicated below:

import boto3
import json
import logging
import time
import requests

logging.basicConfig(filename='pythonS3IotDevice.log', filemode='w', format='%(name)s - %(levelname)s - %(message)s',level=logging.DEBUG)
logger = logging.getLogger('pythonS3IotDevice')
logger.info("pythonS3IotDevice")

iot_credentials_endpoint='https://#iot_credentials_provider_endpoint#/role-aliases/#role_alias#/credentials' #Provide your AWS IoT Credentials endpoint (Example: "acbcdef12345.credentials.iot.us-east-1.amazonaws.com") and the IAM role alias previously created.

response = requests.get(iot_credentials_endpoint, cert=("/certificate/path", "/private/key/path")) #Set paths for private certificate and private key

if response:
tmp_credentials = response.json()
access_key_id = tmp_credentials['credentials']['accessKeyId']
secrete_access_key = tmp_credentials['credentials']['secretAccessKey']
session_token = tmp_credentials['credentials']['sessionToken']

s3client = boto3.client(
's3',
aws_access_key_id=access_key_id,
aws_secret_access_key=secrete_access_key,
aws_session_token=session_token,
)

s3client.upload_file('/jpegimage/path', "#bucket_name#", '#image_name#.jpg') #Set path for jpeg image to upload. Then specify the bucket name (name of the bucket previously created) and the image name.

Note: For further AWS IoT SDK (for Python) samples, please visit this page.

Clean up

After you are done, if you want to ensure that no additional cost is incurred, you can remove the resources provisioned in your account. This includes the deletion of the following resources:

1. Delete the AWS Lambda function with the following AWS CLI command: aws lambda delete-function —function-name #lambda_function_name# —region #region#
2. Delete the Amazon S3 bucket as explained in this guide.
3. Clean up the Amazon SageMaker resources (model, endpoint and notebook) as explained in this guide.
4. Delete the detector model created in AWS IoT Events with the following AWS CLI command: aws iotevents delete-detector-model —detector-model-name #detector_model_name# —region #region#

Note that for the above AWS CLI commands:

• The #lambda_function_name# is the name of the AWS Lambda’s function previously created.
• The #region# is the AWS region where your resources were created (Example: us-east-1).
• The #detector_model_name# is the name of the detector model previously created in AWS IoT Events.

Conclusion

In this post, we walked through a use case where drone devices are used to capture scene images of a physical asset (e.g. gas pipe), which are then validated against a machine learning model to detect changes to the asset that may affect the proper operation of the asset.

To summarize the solution steps:

1. First, we created a detector model in AWS IoT Events, that manages the drone device route.
2. Second, we provisioned a sample machine learning model that validates the images received from the device. We also created an Amazon S3 bucket to upload the images captured by the device.
3. Then we provisioned an AWS Lambda function to retrieve the images uploaded in the Amazon S3 bucket and send them to the Amazon SageMaker model for validation. We also created an Amazon Simple Notification System (SNS) topic to notify the end-users, if scene changes were detected by the Amazon SageMaker model.
4. Finally, we explained how the device can connect with AWS services (AWS IoT Core and Amazon S3) by providing AWS configuration procedures as well as Python code samples for the device.

For further information on the services used, you can consult the AWS IoT Core, AWS IoT Events and Amazon SageMaker web pages.