So, if you’ve been following me for a while you know that I’ve been playing with AI for Earth and using Logic Apps to pull images from OneDrive for analysis. I had the fortune of working with Elmbrook Launch for two seasons and over that time we’ve incrementally improved the platform. The iterations went like this:
- Iteration 1: Capture images from OneDrive and send to AI for Earth platform
- Iteration 2: Send tweets from identified images to generic address
- Iteration 3: Capture tweets, pull the image, populate OneDrive, and reply to the tweet
- Iteration 4: Simplify with Logic Apps and eliminate OneDrive portion
- Iteration 5: (upcoming) – to replace Logic App with Azure Function with faster polling interval
- Iteration 6: (upcoming) – add Cosmos DB and Synapse Link to the end and enable Power BI
- Iteration 7: (upcoming) – enable Cosmos / Synapse with AI operations
The goal of iteration 4 was to prove that a Logic App could pull a tweet, capture its image, send it to AI for Earth, then tweet back out the image and the identified wildlife. The iteration was successful at accomplishing this goal, but for two limitations. The first limitation is that the Logic App twitter agent will only poll hourly, even though it can be configured on a faster interval. The second is that the Logic App twitter agent cannot @ or quote tweet the original tweet, leading us to have to just post on the wildlifeai timeline. This is fine, but not ideal, which leads to what we’ll be looking to accomplish in iteration 5.
The structure of the logic app is as follows:
The first section looks for new tweets to the @wildlifeai twitter handle and then passes those tweets to future stages. You’ll note that the interval says “1 minute”, but actually it polls only 1 time hourly. This is a bug in the interface, as the documented API shows otherwise.
The next stage is a for-each on the media urls coming out of the tweet. If for instance the tweet had multiple media urls, we would take each separately and process them. There is a limitation on the number of posts per hour (12) that could cause people to not get responses. This is another reason for iteration 5.
The next stage of the process copies the media to an Azure storage account, which is later used to pull the image from and we’ll eventually use for further image categorization and AI processes.
The next stage is to get blob content so it can be sent to AI for Earth, which requires the image to be stored as a blob so it can be passed to the following HTTP stage.
In this stage you need to send the blob content to Microsoft AI for Earth. If you want to use it, you need to request a key from Microsoft for your meaningful project. In this case I’ve hidden the key we’re using.
You then need to parse the JSON. I’m using the JSON parser Logic Apps step. The code in the JSON is as follows, but I generated it by grabbing the output from AI for Earth and then using the “use sample payload to generate schema”
{
"properties": {
"bboxes": {
"items": {
"properties": {
"confidence": {
"type": "number"
},
"x_max": {
"type": "number"
},
"x_min": {
"type": "number"
},
"y_max": {
"type": "number"
},
"y_min": {
"type": "number"
}
},
"required": [
"confidence",
"x_max",
"x_min",
"y_max",
"y_min"
],
"type": "object"
},
"type": "array"
},
"predictions": {
"items": {
"properties": {
"class": {
"type": "string"
},
"class_common": {
"type": "string"
},
"confidence": {
"type": "number"
},
"family": {
"type": "string"
},
"family_common": {
"type": "string"
},
"genus": {
"type": "string"
},
"genus_common": {
"type": "string"
},
"kingdom": {
"type": "string"
},
"kingdom_common": {
"type": "string"
},
"order": {
"type": "string"
},
"order_common": {
"type": "string"
},
"phylum": {
"type": "string"
},
"phylum_common": {
"type": "string"
},
"species": {
"type": "string"
},
"species_common": {
"type": "string"
},
"subphylum": {
"type": "string"
},
"subphylum_common": {
"type": "string"
},
"subspecies": {
"type": "string"
},
"subspecies_common": {
"type": "string"
}
},
"type": "object"
},
"type": "array"
}
},
"type": "object"
}
Finally, from the prediction I’m then posting the tweet with only the species_common output, the confidence, and the image as attached media.
The result is this! The twitter bot saw my initial tweet, captured it, and then posted a new tweet indicating the species and the confidence. You can try yourself at wildlifeai (@wildlifeai) / Twitter
Nathan Lasnoski 