Category Archives: Azure

Power of PowerBI with Azure Stream Analytics and IoT Hub

Recently i gave a talk at a Microsoft Integration Commuity Event on Azure IoT Hub. In the demo i used Azure Stream Analytics for ingestion and PowerBi as its output.

To demo the IoT Hub bidirectional feature it had an abnormal detection scenario using Stream Analytics together with EventHub and a Azure Function sending a message back to the device.

The scenario is illustrated below:

Azure IoT Hub demo scenario

I used two real devices and one multi-device simulator written in C#.

  • Raspberry Pi 3 + GrovePi+ with sensors for temperature/humidy, rotary and a LCD-display. OS was Windows 10 IoT Core and a custom UWP app that sent the telemetry data to IoT Hub every second.
  • Adafruit Feather M0 (sensor: BME280 for temperature/humidity). The SDK used was the one from here https://github.com/Azure/azure-iot-arduino. It has only HTTP support at the moment but MQTT is said to be coming very soon.
  • C# Simulator was a commandline application with a random telemetry simulator to get somewhat realistic movements on its faked sensors.

The Azure Stream analytics jobs had 1 input, 2 outputs and a two part query. One for the PowerBi output where “group by TumblingWindow” was used and one for the abnormal detection (vibrationLevel > 400).

Stream Analytics Query:

The TumblingWindow (together with Sliding and Hopping -window) is the star of Stream Analytics. Thousands (or millions!) of devices sending telemetry data every second can be grouped using a time window. For demo purposes i used a short window.

The EventHub output for the abnormal scenario was read by an Azure Function that sent a Cloud-2-Device message back to the device that caused a background color change and a message on the LCD.

Azure Function code :

 

After some simplified PowerBI reporting the demo looked like this:

2016-11-17-19_03_48-power-bi

The map highlights the location of the devices and the circle size are related to the vibration level.

The real power here is the scaling. Knowing that one could take this from 6 devices to millions (and back down again!) without any investment or upfront costs is pretty amazing and unthinkable just a couple years ago. That is what cloud computing means to me.

Code on GitHub

Azure Logic Apps Text-to-speech with TalkToMe API

Many years ago i made a text-to-speech WCF binding that i used in demos and proof-of-concepts with BizTalk Server. Its time to bring that up to 2015!

This time i will use the Azure App Service stack with a custom Azure API App that can be used from a Azure Logic App. The API App will let you connect a browser session to it that will act as the “loudspeaker”.

TalkToMe GitHub repo – instructions and code

The API app has two parts, both hosted together. First the API that hosts a SignalR hub and the second one is the UI that connects one (or more) browser as the API App “Loudspeaker”. As an Azure API App is just a ‘normal’ Web App it can host both parts and no other deployment is needed. Sweet!

  • WebAPI
    • SignalR Hub
  • Client UI (Hosted inside the API App!)

SignalR is the brilliant communication framework that will allow us to trigger functionality in the client (browser) yet abstracting away the actual connection details (web-sockets, long-polling etc.).

To test the API App i used a Logic App that gets weather data from api.openweathermap.com that TalkToMe API then will read out. Note that by design the API will return 202/Accepted even if no browser (ie. SignalR client) is connected and no queuing or similar is performed.

How to use in Azure Logic Apps

  1. Deploy Azure API App
  2. Create and deploy a Logic App where TalkToMe is used as an ‘action’ Logic App
  3. Connect one or more browsers to the API by browsing the root URL of the deployed API. If you are unsure of the URL you can find the link in the ‘essentials’ section of the API App in the portal.image image
  4. Run the Logic App

When the logic app runs and the TalkToMe action fires the portal should show something like image

And you should hear the artificial voice read the weather :) image

Test API using HTTP

If you just want to try the API App, outside of an Logic App, you can use a simple REST call.

REST param Value
URL http://{your-apiapp-url}/api/TalkToMe
Method HTTP POST
Content-Type application/json
Body { “TextToRead” : “Nice weather today” }

 

 

 

Azure Logic App with File Connector for on-premise filesystem integration

Azure Logic Apps is Microsofts new shiny integration PaaS offering building on API Apps as the integration building-blocks. Sure – the tooling is far from ready and the management capabilities is lacking in functionality. But i like the way this is moving. For cloud/SaaS scenarios there is already an impressive number of connectors available and a simple API App model for custom extensions.

I decided to document the steps for simplified integration scenario of Azure Logic App with FILE Connector for on-premise filesystem integration. Its really very simple but there are some small quirks and even a small bug!

The scenario is that the Logic App should pick up files on the local file-system without requiring a custom developed scheduled process/script/application to upload the files to Azure. The FILE Connector uses a “Hybrid Connection service” to make an outgoing connection to the ServiceBus relay. This means (IT-dept Security responsible – please cover your ears) you normally don´t need any firewall openings.

To get this scenario running start by creating a “Service Bus Namespace” using the old (current) portal at https://manage.windowsazure.com and copy the connection string using the “Connection Information” function. You do not need to create a relay as this will be created automatically for you by the FILE Connector API app.

soapfault_com_api-app-fileconnector10

Click on the Connection Information as seen in the screenshot above.

Next step is to create the actual File Connector API using the the new/preview portal. Search for ‘File Connector’ if you are having trouble finding it.

Fill in the Service Bus Connection string copied earlier and the local root folder to be used. You will specify a relative folder to this when configuring the File Connector API APP in the Logic App later on.soapfault_com_api-app-fileconnector11

 

When the API app is created after a couple of minutes its time to setup the Hybrid Connection. Or at least that is what i thought…

The summary page of the API app is supposed to show some Hybrid Connection information but this never appears. This seems to be some kind of a bug (it surely can’t be me can it…) in the portal (see below) not saving the parameters specified in Package Settings correctly.

soapfault_com_api-app-fileconnector12

 

If this happens to you the easiest way of solving it is to click the Host link in the essentials section (marked in a square above) followed by Application Settings. Scroll down to the App settings section and have a look at the File_RootFolderPath and ServiceBusConnectionString. They are empty!

soapfault_com_api-app-fileconnector13

 

Edit them manually.soapfault_com_api-app-fileconnector14

Click Save and then reopen the File Connector API app blade. After some time you should now see the “Hybrid Connection” icon saying its setup is incomplete. Sweet!

soapfault_com_api-app-fileconnector15

 

Click to open the Hybrid Connection configuration.

soapfault_com_api-app-fileconnector16

 

You can either download the Hybrid Connection msi manually from here or use the Click-Once-Installation using the provided link ‘Download and Configure’.

soapfault_com_api-app-fileconnector6

soapfault_com_api-app-fileconnector7

 

If everything works out you should now see this on the summary page.

soapfault_com_api-app-fileconnector8

Great! Now lets try the File Connector API app as an Logic App trigger.

Lets assume this on-premise legacy system produces XML files and that we want to access the data. In this heavily simplified test-scenario i use the Biztalk JSON Encoder and Slack Connector as Logic App actions but anything goes.

soapfault_com_api-app-fileconnector24

Note that the folder specified is relative to the folder specified as a root folder.

Ok, lets drop an XML file in my fictitious on-premise system and see what happens.

soapfault_com_api-app-fileconnector21

 

The file disappears from the filesystem and if we look in the Azure portal both actions run successfully.

soapfault_com_api-app-fileconnector25

 

And the message appeared in Slack. Nice!

soapfault_com_api-app-fileconnector23

WCF service hosted in Azure Websites

While WCF might not be the most viable .NET technology stack on the open web as opposed to WEB API 2.0 it is still very relevant in enterprise and B2B scenarios.

It is sometimes considered hard to configure and host WCF. Lets see how hard it really is now with .NET 4.5.

The other day i quickly needed a simple test SOAP endpoint exposed on the internet. I thought i would host it in Azure Websites.

First let us create the web site in Azure. As an alternative to normal ftp-deploy lets use Azure Websites great Kudu features and GIT support.

 

Lets crate a simple untyped WCF service echoing any SOAP-request back as a response. This is probably not a real world scenario although the untyped nature of System.ServiceModel.Channels.Message is really powerful. But any WCF service you find appropriate would work.

The sample below is a minimal single-file wcf service without any code-behind. Save it to a file called EchoService.svc.

 

Save the file and push it to the remote repository automatically created in your Azure Website.

 

Done!

GIT pushed the EchoService.svc file to the remote repository and Kudu automatically deployed it into the website wwwroot folder. If you want to learn more on the amazing kudu stuff in Azure Websites i highly recommend having a look at the short videos made by Scott Hanselman and David Ebbo.

You can reach the service at http://yourwebsitename.azurewebsites.net/EchoService.svc and maybe use something like SoapUI to try it out. The WCF default configuration will expose an endpoint using the BasicHttpBinding meaning any SOAP 1.1 envelope will work. Metadata publication is disabled by default but as this is an untyped service there is really no need for it. If needed it can easily be enabled in code or configuration.

soapui_echoservice

As show Microsoft PaaS service Azure Websites is i really simple way to host a WCF endpoint in the cloud. With the help of .NET 4.5 this is easier than ever.

Great times for cloud developers

The last couple of years has meant a incredible things for us developers as they now can stay focused on the important stuff and forget about stuff that used to take hours, days and sometimes even weeks.

Lets take an example. Using the Azure command-line tools (which are built in node/javascript and are platform agnostic running on Windows, Linux, OSX) and GIT i want to:

  • Create new cloudbased WebSite
  • Create a distributed source control repository locally and at my remote website
  • Setup automatic continuous-integration from the remote GIT repository
  • Create a html file
  • Add file to local repository
  • Push to remote repository and have the site automatically deployed from there.

Ok, lets begin…

Thats it! Done! Thanks to Azure, Kudu and GIT it took 4 (!) commands. Well, 5 if you count creating the website homepage. The site could be ASP.NET, Node.JS, PHP or static HTML.

If this was an ASP.NET application i could attach to and debug the application running in Azure directly from my local Visual Studio.

Now what if we want to use the real power of cloud computing, scaling! Can we do that? Well then we need to issue another command…

Now we have a website running load-balanced on 3 server instances. I can scale up and down and pay for the minutes i use. No IIS, no applicationppol, no NLB-setup and no script editing. The developer can focus on the site content and functionality.

When i need more control i just go to the default created log/debug/diagnostic site created by Kudu at https://<yoursite>.scm.azurewebsites.net. Here you can copy files, look at deployment details and have a go at the shiny new debug console.

2014-02-02 09_45_16-Diagnostic Console

 

The abstraction and simplification provided by Azure Websites is really really effective and powerful yet lets you gain more detailed control when you need it.

Warmup of blog in new Azure home

I have now moved this blog to a new home in Windows Azure.

Pretty cool experience to see how seamless PHP/MYSQL based WordPress runs on Azure WebSites which is one of the PaaS features of Windows Azure.

One thing i have noticed working with Azure WebSites prior to this is the long initial load time after site has been idle for a while and the process is unloaded. This is easily solved by some keep-alive or warm-up task that periodically makes a HTTP GET to the site.

After reading a post by Sandrino Di Mattia i did this in Azure by using the Azure Mobile Services Job Scheduler which is currently in preview. This is probably not the intended usage of Mobile Services Job Scheduler but simple enough to be replaced with another solution if needed in the future.

The node.js script i use is shown here

Azure Mobile Services scheduled script

Azure Mobile Services scheduled script

The script is scheduled to run every 15 minutes which seems to do the trick both for ASP.NET and PHP based WebSites.

keep-alive-script-schedule

 

If the blog is not responsive – let me know :)

Connecting BizTalk with a mobile app using Azure Appfabric ServiceBus Queues, REST and WCF

The Azure Appfabric Service Bus 1.5 released in September had a feature I really have been looking forward to, durable queues and topics (pub/sub). A cloud based MSMQ ‘like’ queue can help both enterprise-enterprise scenarios as well as in connecting mobile applications with on-premise or cloud-hosted services in a reliable way.

Appfabric Service Bus queues and topics support three programming models:

 

  • .NET API
  • REST API
  • WCF Bindings

While the.NET API offers great control and simplicity, the REST API is supported on almost all platforms. Finally the WCF binding allows .NET programmers not having to learn yet another programming model.

In BizTalk the easiest way to utilize queues and topics are by the WCF binding NetMessagingBinding. Although we surely could use the REST API, just getting the security token out-of-band and keeping it updated, in the WCF-pipeline or elsewhere, would make it a bit cumbersome. NetMessagingBinding will automatically do the token acquisition before accessing the queue/topic.

If both sender and receiver use WCF things are pretty straight forward but if different programming models are used there are some things that need to be addressed.

Scenario

This scenario consists of an Android device which will use REST to send messages to a Service Bus queue then picked up by BizTalk using WCF.

I want the the client to be able to send any data (XML, text or binary) to the same Receive Location in BizTalk.

The Service Bus authorization is claims-based and the Azure Appfabric ACS (Access Control Service) is automatically configured when the Service Bus namespace is created. For simplicity I use ACS Service identities for both BizTalk and Android. In real world, the Android application would probably use something else such as an external IdP together with ACS. More on that in a later post!

Setup

Appfabric 1.5 SDK does not add entries to the machine.config as the previous versions did and we need those in BizTalk. I copy/pasted the necessary lines from http://msdn.microsoft.com/en-us/library/windowsazure/hh148155.aspx. Make sure to add them both to 32 & 64 bit machine.config!

Create a new one-way Receive Location hosted ‘in-process’ using the WCF-Custom adapter.

To be able to receive anything the client sends in the body of the HTTP POST we have to use CustomBinding instead of the pre-configured NetMessagingBinding. This allows us to replace the default TextMessageEncoding binding element with a WebMessageEncodingElement instead. Remember to move it above the NetMessagingTransport element after it is added.

WebMessageEncoding can output XML, JSON or binary data and it defaults to binary. It is possible to control this behavior using a custom WebContentTypeMapper but in this scenario it works fine with the default.

On the general tab enter the address of the already existing queue using the following format :  “sb://{servicebusnamespace}.servicebus.windows.net/{Queue}”.

Authentication and authorization

To receive messages from the queue the authorization will require a claim of type ‘net.windows.servicebus.action’ with value ‘Listen’.

The pre-configured ACS Service Identity ‘owner’ has this claim but it is considered more secure to create a separate identity with a shared symmetric key or an X509 certificate as identification. If you use something different than ‘owner’ do not forget to update the Rule Group to create the claim.

To specify how to authenticate against Service Bus add a TransportClient Endpoint Behavior and specify the identity and shared key to use.

Addressing

By default the NetMessageTransport assumes WCF and that WS-Addressing “To” is present in the message. So if you make a HTTP POST with a non-SOAP body to the queue the WCF trace will show that the adapter throws this exception:

The message with To ” cannot be processed at the receiver, due to an AddressFilter mismatch at the EndpointDispatcher. Check that the sender and receiver’s EndpointAddresses agree.

To fix this we need to insert the more forgiving Address Filter such as “System.ServiceModel.Dispatcher.MatchAllMessageFilter”. This must be done in a custom behavior extension.

Add this BehaviorExtensionElement to machine.config (32 & 64-bit) in the behaviorExtensions section.

Go to the Receive Location and add this in the EndpointBehaviors.

BizTalk message source

If you would make a REST HTTP POST to the queue now BizTalk would publish the message from WebMessageEncoding and it would look something like this:

<Binary>Qk3WAAAAAAAAADYAAAAoAAAA==</Binary>

This is Base64 encoded data and to fix that we just need to change what BizTalk publishes to the message box.

Go to the Messages tab and enter /*[local-name()=’Binary’] as the Body path expression and Base64 as Node encoding.

Now the WCF adapter will decode and publish everything inside <Binary> to the messagebox where a subscribing orchestration or send port can use the message.

Client

To test it I would use a REST capable test-client such Fiddler, SoapUI or the test client in WCF WEB API

Make a HTTP POST to “https://{servicebusnamespace}.servicebus.windows.net/{QueueName}/Messaging” with the token received from ACS in the authorization http header.

To test the mobile scenario i made a heavily simplified Android application that can send an image from the phones image gallery to the Azure Appfabric Service Bus Queue.

Code handling image-resize and some basic Android stuff are not shown above.

In BizTalk we now get the binary representation of the image published to the messagebox and a simple FILE Sendport makes the image viewable. In a real world scenario, metadata about the image could go as custom http headers or similar.

The takeaway here is that we can use the same queue and BizTalk Receive Location no matter what client or Service Bus API (WCF, REST or .NET) is used on the other side!