Forem: Surya Venkat

Azure Functions in the Portal – ALM

Surya Venkat — Tue, 24 Sep 2019 12:53:51 +0000

Author Credits: Michael Stephenson, Microsoft Azure MVP.

Originally Published at Serverless360 Blogs.

This article is part of #ServerlessSeptember. You'll find other helpful articles, detailed tutorials, and videos in this all-things-serverless content collection. New articles are published every day — that's right, every day — from community members and cloud advocates in the month of September.

Find out more about how Microsoft Azure enables your Serverless functions at https://docs.microsoft.com/azure/azure-functions/.

One of the advantages of Azure is that for some use cases you can develop solutions in the Azure Portal. This has the benefits that you can just focus on writing some code and not have to worry about versions of Visual Studio and extensions and all of the other overheads which turn a few simple lines of code which can be written by anyone into something which requires an additional level of developer skills. Let’s face it the ALM processes have been around for years but there are still large portions of the developer community who don’t follow them.

The ability to just get the job done in the portal is compelling and I expect that we will see more of that in the future but it does give you a challenge when it comes to ALM activities like keeping a safe version of the code and being able to move between environments reliably

This article explores the options for being able to develop an Azure Function in the portal but use some of the basic ALM type activities which would only be a minor overhead but gives some good practices so that developing in the portal would be ok in the real world.

My Process

The process I am going to follow is as follows:

I will have a development resource group which will contain my Azure Function and the code for it
The resource group will also contain other assets for the function like AppInsights and Storage
I will create a 2^nd resource group called Test. In my Build process, I will refresh the Test resource group with the latest version so I can do some testing
Once I am happy with the test resource group I will then execute a release pipeline which will copy the latest for the function to other environments such as UAT which I assume are used by other testers etc

To summarise the pipeline usage see below:

Dev -> Test = Build Pipeline
-> UAT and beyond = Release Pipeline.

Assumptions

I am going to make a few assumptions:

The function apps and Azure resources will be created by hand in advance
Any config settings will be added to the function apps by hand.

In this simple example, we are assuming that everything is quite simple, and we can just update the code between environments. In a future example, we will look at some more complex scenarios.

Walk-through

To begin the walk-through, let’s have a look at the code for our function below:

You can see this is a very simple function which is just reading some config settings and returning them.

Build Process

From here we need to go to our Build process in Azure DevOps. The build process looks like the following:

I have defined a build process which I could use for any function app in the portal. I would simply need to change the variables and subscriptions references and it could be reused easily via the Clone function.

The build executes the following steps:

Show all build variables = I use this for troubleshooting as it shows the values for all build variables
Export code = This uses the App Service Kudu API features to download the source code for the function app as a zip file
Publish Artifact = This attaches the zip file to the build so I can use it in Release pipelines later
Azure Function App Deploy = This will deploy the zip file to the Test function app so that I can do some manual testing if I want.

A closer look at Export

I think in the build process the Export Function Code step warrants a closer look. This step uses Powershell to execute a web request to download the code as a zip file. I have used the publisher profile in the function app to get the publisher credentials which I can save as build variables and then use as a basic authentication header for the web request to do the download. See the below piece of code

$user = '$(my.functionapp.deployment.username)'
$pass = '$(my.functionapp.deployment.password)'
$pair = "$($user):$($pass)"

Write-Host $pair
$encodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($pair))

$basicAuthValue = "Basic $encodedCreds"

$Headers = @{
    Authorization = $basicAuthValue

}

Write-Host $(my.functionapp.name)

Invoke-WebRequest -Uri 

"https://$(my.functionapp.name).scm.azurewebsites.net/api/zip/site/wwwroot/" -OutFile 
"$Env:BUILD_STAGINGDIRECTORY\Function.zip" -Headers $Headers

A closer look at Azure Function App Deploy

The Azure Function app deploy is simply using the out of the box task. I am pointing to the zip file I have just downloaded in the earlier step and it will automatically deploy it for me. I have set the deployment type on this task to Zip deployment.

Release Process

We now have a repeatable build process which will take the latest version of the code from my development function app and push it to the test instance and package the zip file so I can at some future point release this version of the code.

To do the release to other environments I have an Azure DevOps Release pipeline. You can see this below:

The Release pipeline contains a reference to the build output I should use and then contains a set of tasks for each environment we want to deploy to. In this case its just UAT. The UAT release process looks like the following:

You can see that in this case the Release process is very simple and really it’s a cut down version of the Build process. In this case, I am downloading the artifact we saved in the build. We then use the Azure Function deploy to copy the function to the UAT function app. We are just using the same OOTB configuration as in the above Build process but this time we are pointing to the UAT Function app.

I now just need to run the Release process to deploy the function to other environments.

Limitations

I am not using any visual studio so I am unlikely to be automatically testing my functions much. I could potentially look at doing something in this area but it's out of the scope of this article
I am not keeping the code in source control in this article. I am happy that the zip file attached to the build is sufficient. I could possibly look to saving the zip to source control or unpacking it and saving files to source control if I wanted
I am not using any continuous integration here, you could maybe monitor Azure events with Logic Apps and then develop your own trigger.

Summary

Hopefully, you can see that it is very simple to implement the most basic of ALM processes for your development in the portal effort which will add some maturity to it.

Azure Service Bus Overview

Surya Venkat — Fri, 30 Aug 2019 06:37:00 +0000

What is Azure Service Bus?

Azure Service Bus is a messaging service on cloud used to connect any applications, devices, and also services running in the cloud to any other applications or services. As a result, it acts as a messaging backbone for applications available in the cloud or across any devices.

Service Bus Concepts

At the time of this webinar, there were three basic concepts in Azure Service Bus:

Relay -> To establish a secure connection between applications which is bridged through the cloud
Messaging -> Used to push/pull messages from the queue
Event Hub -> High performing stream-based event system

Relays and also Event Hubs were moved out as separate services and Service Bus will only refer to Messaging that consists of Queues / Topics

Multi-Tenant Service Bus environment

In this environment, we can create a namespace which would be the container for our Service Bus artifacts in the cloud. The namespace can be available in different regions.

Each namespace comprises messaging entities like Queues, Relays or Topics. Most importantly, a namespace will be dedicated only to one type of Service Bus.

Message processing statistics (as of 2015)

The statistics generated by the Microsoft team working on Service Bus are as follows,

Event Hub = 30 Billion Events per day
Messaging = 7 Billion Messages per day

It can handle a colossal number of messages in a very executable manner.

Service Bus Relay

In Service Bus Relay, we will have an endpoint that will be hosted in the cloud, but the difference would be the fact that our on-premise service will make an outbound connection to register with the Service Bus relay endpoint.

At a conceptual level, we send a message to an endpoint in the cloud from where it will be forwarded to an on-premise service. As a result, it gives us the advantage to connect from one organization to another organization without having to set VPNs.

Some of the key features are,

Service Bus Relay can be used to solve problems in scenarios like,

Information passed between two data centers.
Building an integration architecture through SAP to forward messages into microservice architecture using Service Bus Relay.

Example Scenarios:

Many of the Microsoft API apps connect to on-premise systems use relay under the hood
Danish pension and also insurance supplier exposes IBM WebSphere integration platform to partners through relay
Northumbria University use the relay to bridge cloud-hosted application to integrate with on-premise WCF service

Demo-Relay

This demo explains the process of hybrid connectivity with Service Bus Relay. We would initially work on a WCF service (on-premise service). The demo constructed on a .NET framework with console application is to call the back-end service.

In Service Bus Explorer, we will have a registered endpoint under the “relay” section. Now, a virtual directory (router) will be linked to the configuration file “web” under user defined location to establish a connection. The folder would also comprise of two more files, which are,

Bin directory
WcfRoutingService(SVC file)

In the SVC file, there would be service pointing to WCF routing service endpoint. In the configuration file, there would a WCF service which uses the HTTP relay binding along with other credentials to access the Service Bus Relay.

Finally, Service Bus Relay accepts a message and forwards it to the back-end service. When executed on the console application, every message will be processed within a time frame of 100 milliseconds.

Messaging

In Azure Service Bus, We can transfer the data between services or applications in message format like XML, JSON, or text.

In this messaging protocol, Microsoft would handle most of the scaling. We just have to create a namespace using platforms like Azure Portal, Service Bus Explorer or Serverless360.

Let us consider some example scenarios,

Construction companies use sensors to monitor buildings in real time to ensure adherence to safety more standards.
Using Service Bus, “Qoros” (Chinese car manufacturer) connects a car to the internet to stream messages.
Metro Bank Use Service Bus Messaging to manage near real-time messaging between SharePoint and also Dynamics CRM.

Queue

A messaging protocol based on the first-in-first-out pattern. Here a sender would send a message to a queue from where the receiver would collect it in initial or later stage. Besides that, queues enable us to store the message until the receiver is available to receive and also process them. The user will receive the message upon request.

Topics

Topics are similar to queues. Most importantly, topics make multiple subscriptions available. A sender will send a message to a topic then a routing rule can determine which subscription may receive the slotted copy of the message.

The receiver can receive the message from the subscription. Multiple receivers could be allocated to a subscription.

Other messaging patterns include,

RPC
Scatter Gather
Pub/Sub

Demo-Messaging

In this demo, we will send a message to a queue and also collect it back. Initially, a queue (testqueue) will be specified and also the following properties will be determined in Service Bus Explorer as mentioned below,

Path
Queue properties
Lock duration

To send a message to a queue and also to collect it back, we define the Service Bus connection string along with reference to Service Bus NuGet package.

We will now create a Service Bus message and sent it. On receiving side, we have a competing consumer pattern which enables multiple concurrent consumers to process messages received on the messaging channel. Dynamic message delivery method is employed in a single-thread pattern i.e., first-in-first-out. But in multi-threading pattern, messages will be processed at a higher throughput for multiple receivers.

On receiving side, using “OnMessagingOption” we can specify the number of concurrent receivers required,

We will then access the body of the message through the event (OnMessage) and also write it to the console window.

The message can either be removed from the queue or abandoned. Service Bus handles serialization with the help of SDK’s.

Let us consider, when we send a message it would be processed in a loop format. So, we will take the .NET type specified as “TestMessage” and also moves it to “BrokeredMessage” to handle the serialization of the .NET type when passed to the queue.

The content of the message will be extracted. Then the format of the message which delivered, received should be verified.

Now if we run our demo on two separate console applications, one to send and also another to receive. The respective messages would be sent and also received from the queue.

Event Hubs:

Event hubs will manage massive scale event stream processing and are similar to Service Bus messaging. It is similar in the case that it supports sending the message over AMQP (Advanced Message Queuing Protocol).

The key difference between Event Hubs and also Service Bus messaging is that there is no concept of deleting a peek lock message like in Service Bus. Instead, Microsoft has used a reader across a stream which provides a high level of scaling.

Let us consider an IOT scenario where Event Hub comes in use,

The above picture comprises many subsystems, each with an individual offering. Event Hub can stream a high volume of data from different applications and also store it.

Most importantly, users can process the streamed data and apply it for their individual purpose.

Let us consider some example scenarios:

Microsoft uses Event Hub under the hood as a part of application insights and also API management.
A company called “Autolib” uses Event Hub as a part of Microsoft IOT stack to exchange information between devices and also vehicles as a part of a car-sharing scheme in Paris.

In Event Hub, multiple senders store their telemetry data in the message stream. A receiver cannot remove the message from the stream. Rather, they only get deleted when their retention period gets over. A consumer group can access message from any stream.

Application Telemetry

The above picture represents a typical message flow of a system. The idea is to feed telemetry data to Power BI through Event Hub. Message from the user would pass through various systems and also would reach Event Hub.

Here, Stream Analytics and Machine learning process collect the data from Event Hub and send it to Power BI for any business analytics service.

Demo-Event Hubs

This demo illustrates the purpose of Event Hub and is similar to Azure Service Bus messaging.

On Sender Side,

In the console application, we have a connection string followed by an Event Hub name (key name). In the code portion, we have

EventHubClient -> To create an instance of a client through the connection string
Client.Send -> To send an object

On Receiver Side,

In the console application, we have an “EventHubName” and also storage account followed by a ServiceBus.EventProcessorHost which is used to create a listener for different partitions in the Event Hub instance. In the Azure storage account, this package creates a blob for each partition to manage the index of its location in the stream.

When a message is received, it will be fired to the event listed in the below piece of code,

Our position in the stream can also be acknowledged so if the whole process blew up, we can simply move to the last index we read and also continue our progress. Now, we can see how the operation gets executed. As a result, Event Hub can receive and store any kind of data and events.

Fact

Service Bus team provided a cost estimation analysis for processing a billion events per day which lead to,

12 Throughput units
12,000 events per second
720,000 events per minute
43,200,000 events per hour
Total Cost -> $37.67

Service Bus Costs

Relay:

10,000 listening hours=6.60 pounds
100 million messages=6.20 pounds

Messaging:

5 million operations per month=6.20 pounds
13-100 million operations = 50pounds per million
100-2500 million = 31pounds per million

Event Hubs:

100 million events = 1.72 pounds
The rate of Processing = Throughput units = 14 pounds per month per unit

Service Bus + BizTalk

Although Service Bus and BizTalk provide connectivity in many aspects for a seamless experience, they could be more aligned in the following aspects;

SB- Adapter (latest version not available)

Service Bus Relay
Service Bus Messaging
Event Hubs-it does not provide out-of-box support

Addition of features list includes,

AMQP Adapter- Supports non-Microsoft Service Bus type products
Event Hub Adapter

User scenarios of BizTalk+Relay

Scenario 1:

Let us consider a BizTalk server hosted in Azure as a Virtual Machine needs to connect to an on-premise web service. The simplest method to do that would be through SB adapter in BizTalk.

Here, BizTalk Server will send a message to Relay and the rest of the process will be carried out in WCF service.

Scenario 2:

If we have BizTalk server on-premise followed by a SaaS application in the cloud, a communication can be established between them through Service Bus Relay.

This can be achieved through Service Bus Relay or by assigning a REST API between SaaS component and the server. So, the Relay offers an easy way to expose a BizTalk endpoint from an organization.

Scenario 3:

Let us consider the scenario where two BizTalk Servers in separate companies needs to communicate. The best way to do that will be to send the message through Relay by exposing their endpoints. This flow does not require any separate infrastructure support.

User scenarios of BizTalk+Messaging

Scenario 1:

This scenario would focus more on transferring a massive number of messages from a web application in the cloud to on-premise Server. Here, BizTalk Server pulls the messages from the queue at a manageable level.

Scenario 2:

To publish an event in pub/sub format to various receivers, we will push the message from on-premise server to a queue where the application can access it. You cannot directly link the applications to the BizTalk Server located on-premise.

The key difference between Service Bus and BizTalk is that,

BizTalk does not support poll pattern
Service Bus does not support push pattern

Therefore, combining both the services would lead to dynamic productivity. The receiver can access the message available on Service Bus Topic at any time.

Scenario 3:

Let us consider a scenario where Service Bus Topic will share the resource between two BizTalk application. But if we want to delete the shared resource in the application and patch a new resource, the dependency management in BizTalk would make it harder.

When the BizTalk application has to be updated, we cannot undeploy it because the subscriptions would be eliminated. Therefore, this causes overcomplexity when trying to solve it.

Demo

This demo explains on combining both BizTalk and Service Bus, through which messages can be sent and also received between BizTalk applications. Most importantly, you can execute the process in JSON format.

Let us assume there are three BizTalk applications (app1, app2, app3), each comprising of “Hello World” scheme followed by a unique namespace. We will have a sender in each application to receive a message from the file system, which will be routed to Service Bus through Send Ports. Furthermore, you can convert the messages to JSON format before sending it using JSON encoder.

Most importantly, when we receive the message in Service Bus, it would hit the topic comprising of two subscriptions. Each subscription will forward the content to the desired queue. The Topic can specify who can send and also receive the message in it, but if there are multiple receivers, there is a probability where one receiver may fetch the message of another receiver. Hence, this problem can be solved by using “Forward to” option in Topic and can also specify which receiver should receive the appropriate message.

Most importantly, the agenda is to send a message from any application to topics before it gets forward to the selected queues.

After the initialization process gets over, the virtual machine will receive a message. Using the JSON decode, we will change the format of the message in the receive pipeline of the virtual machine. Finally, the respective Biztalk applications receive the message.

User scenarios of BizTalk+Event Hub

Scenario 1:

Here Event Hubs forwards the messages to BizTalk server after processing from event processor host. If the scenario does not involve a huge number of messages to forward to BizTalk server, then the event processor host will act as an isolated BizTalk host without any adapter for a smoother transition.

Scenario 2:

In this scenario, you share the telemetry data and logging information with Service Bus Event Hub when a message flows through the pipeline of BizTalk server. Most importantly, powerful analysis machines like Power BI and also Cortana Analytics make use of the telemetry data from Service Bus.

Wrap up

This blog gives an overview of various Service Bus components available with demo scenarios for a better understanding of the audience. There are a lot of features available in Azure Service Bus and also more to come soon.

How To Build A Reactive Solution With Azure Event Grid

Surya Venkat — Wed, 14 Aug 2019 06:10:59 +0000

Author Credits: Steef-Jan Wiggers, Azure MVP

Originally published at Serverless360 Blogs

Azure Event Grid was the missing piece for messaging in the cloud. Instead of extensive polling or ‘hammer-polling,’ a very resource intensive process, you now get ‘tap on the shoulder’ – here’s an event ‘an image is available for processing.’ Event Grid is a service in Azure that enables central management of events. It provides intelligent routing with filters and standardizes on an event schema.

Event Grid

With Event, Grid offers a mechanism for reactive event handling. An application can subscribe to events and handle them accordingly – for instance, handle events like storage blob events, provisioning notifications in an Azure subscription, IoT device signals, or even custom event.

Event Grid handles events, not messages. A message has an intent and requires an action – for instance, an order that needs to be fulfilled. An event is a fact, something that has happened and you can react to it or not. Hence there is a clear distinction between messaging and events. Thus, there are services for both available in the Azure Platform - Clemens Vasters explains this in the blog post from last year: Events, Data Points, and Messages - Choosing the right Azure messaging service for your data.

Pub-Sub Model

The concept of Azure Event Grid is simple – it is a pub-sub model. An event source pushes events to Azure Event Grid, and event handlers subscribe to events. An event source or publisher - an Azure service like Storage, IoT Hub, or a third party source emits an event, for instance, blobCreated or blobDeleted. You can send the event to a topic. Each topic can have one or multiple subscribers (event handlers). You can configure an Azure Service, if supported, as an event publisher or you create a custom Azure Event Grid Topic. Subsequently, the event handlers in Azure like Functions, WebHook, and Event Hubs can react to the events and process them.

The pub-sub model of Azure Event Grid is not similar to service bus Topics and subscriptions. Moreover, Topics and subscription are push-pull, while Azure Event Grid is push-push. Furthermore, there are other differences too, as shown in the picture below.

Consider a function listening to a blob storage container and each time a new blob is created it will send a message to a message queue. Subsequently, you can listen to the queue with another function, process the blob with the function and call two other functions (or place a queue between that function and the other two functions). The diagram below shows how pull-push would work using a Service Bus Queue.

With Event Grid, you can have a different approach as shown in the diagram below.

You can view Event Grid as the backbone of event-driven computing in Microsoft Azure - and it brings a few benefits to the enterprise. With the push-push pub-sub model you can be more efficient with resources. Event Grid delivers besides efficiency:

Native integrations (point and click) in a serverless way;
Fan out to multiple places;
Reliable delivery of events within 24 hours;
Intelligent routing of events with types and filters.

These benefits will become apparent in this blog post.

Reversal of dependencies

With Event Grid in Azure, you can use as a developer can or adopt reverse dependencies in an existing cloud solution – where services are supporting business processes by depending on each other. In this type of solution, each service has some logic to communicate with others across the architecture. Moreover, the solution or architecture becomes complex or spaghetti of services.

However, with leveraging Event Grid, you can loosen these dependencies by making the services more event-driven. The services communicate by pushing events to a central place, i.e., Event Grid.

By pushing events to a central place you fundamentally unify the communication between services. Tom Kerhove discusses this in his blog Exploring Event Grid.

Toll booth Scenario

A typical use-case or sample of Event Grid is processing an image leveraging the Vision API. Azure Blob Storage emits an event blobCreated, which is subsequently handled by a function or Logic App subscribing to that event. In the following scenario, the event grid plays a crucial role in handling blobCreated events where several functions subscribe to that event.

In the above scenario, a camera registers every car going through a toll booth. The picture of the vehicle includes a license plate. These images are uploaded in batches to an Azure Storage account – configured as event publisher. Every image is stored as a blob and thus results in an event being emitted to the Event Grid topic. Subsequently, several functions subscribe to the events:

A function to push the blob (image) to a Computer Vision API – extracting license plate number through Optical Character Recognition (OCR) feature. The function will push the result as an event to another Event Grid Topic.
A Function to save the license plate (the result of the Vision API) to Cosmos DB.
A Function to save a reference of the image in the storage account to Cosmos DB.

The above scenario shows how a developer can build several functions to handle the events. Moreover, the developer is a part of making reactive components, i.e., functions (tasks) in this case. The process of managing events (car passing a toll booth) is through a loosely coupled, scalable, and serverless type of solution. In the following paragraphs, we will further discuss some of the critical concepts of Event Grid with the given scenario.

Azure Event Grid Schema

The events in Event Grid adhere to a predefined schema – a JSON schema with several properties. Furthermore, events are sent as a JSON array to Event Grid with a maximum size of 1 MB – each event in the array is limited to 64 KB. In case you send an array of an event exceeding the limits – you will receive the HTTP response 413 Payload Too Large. Note that even a single event is sent as an array.

The following schema shows the properties of an event:

[
 {
  "topic": string,
  "subject": string,
  "id": string,
  "eventType": string,
  "eventTime": string,
  "data":{
  object-unique-to-each-publisher
  },
  "dataVersion": string,
  "metadataVersion": string
 }
]

For example, the schema published for an Azure Blob storage event in our scenario:

[{
"topic": "/subscriptions/0bf166ac-9aa8-4597-bb2a-a845afe01415/resourceGroups/RG_LicensePlates_Demo/providers/Microsoft.Storage/storageAccounts/licenseplatesstorage",
"subject": "/blobServices/default/containers/images/blobs/LicensePlate1.png",
"eventType": "Microsoft.Storage.BlobCreated",
"eventTime": "2018-06-05T08:35:26.2778593Z",
"id": "2c6f7c22-201e-00b7-0aa8-fca7ce06e319",
"data": {
"api": "PutBlob",
"clientRequestId": "32a51839-6bf5-44d6-acfe-7d284229988c",
"requestId": "2c6f7c22-201e-00b7-0aa8-fca7ce000000",
"eTag": "0x8D5CABF497519C6",
"contentType": "application/octet-stream",
"contentLength": 41860,
"blobType": "BlockBlob",
"url": "https://licenseplatesstorage.blob.core.windows.net/images/LicensePlate1.png",
"sequencer": "000000000000000000000000000008BC0000000003fd850f",
"storageDiagnostics": {
"batchId": "fbb4e831-82b2-4e64-b0e5-0735449f9cb7"
}
},
"dataVersion": "",
"metadataVersion": "1"
}]

The image processing function in our scenario can handle the event and use the URL in the data payload to download the image from the storage account container. The Rest Call pushes the downloaded image to the Computer Vision endpoint. Finally, this function will read the JSON result and push to another Event Grid topic as an event.

Security

The security of Event Grid consists of several parts. The first part is the Role-based Access Control (RBAC) on the Event Grid resource - where the person creating a new subscription needs to have the Microsoft.EventGrid/EventSubscriptions/Write permissions. Second, there is a validation on WebHooks – the first and foremost mechanism for listening to events – every newly registered WebHook needs to be validated by Event Grid first. This is what is implicitly happening when creating subscriptions to events in an event resource like Storage Account V2 – you can hook events to a Function or Logic App using the events feature in the storage account. In our scenario, the arrangement of subscriptions to events is implicit.

In case you want to subscribe to events using custom code, i.e., the on-premise client you will need to respond to the validation token the Event Grid will send out. The validation request Event Grid will send to the client will look like below:

[{
"id": "2d1781af-3a4c-4d7c-bd0c-e34b19da4e66",
"topic": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"subject": "",
"data": {
"validationCode": "512d38b6-c7b8-40c8-89fe-f46f9e9622b6"
},
"eventType": "Microsoft.EventGrid.SubscriptionValidationEvent",
"eventTime": "2018-01-25T22:12:19.4556811Z",
"metadataVersion": "1",
"dataVersion": "1"
}]

The client will respond with the validation code in the response with a 200-OK. If you want to send custom events, use SAS tokens or key authentication to publish an event to a topic. For more details on security and authentication of Event Grid, see https://docs.microsoft.com/en-us/azure/event-grid/security-authentication

Guaranteed delivery of events

A key aspect of the scenario is the guaranteed delivery of events. The default behavior of Event Grid is that it will try at least once to deliver the event and retry using exponential backoff - Event Grid will keep on sending events to the Event Handler until it acknowledges the request with either an HTTP 200 OK or HTTP 202 Accepted. The back-off mechanism will use progressively longer waits between retries for following error responses. Furthermore, it will last until 24 hours have passed - after 24 hours the Event Grid will delete the event. See also https://docs.microsoft.com/en-us/azure/event-grid/delivery-and-retry.

Fan-in fan-out Scenario

You can view Event Grid as serverless that is you do not have to manage any infrastructure, you pay per operation, and Microsoft handles the scaling for you. Event Grid leverages Service Fabric under the hood and thus scales when the workload increases.

Event Hubs and Event Grid together work seamlessly in ingestion (fan-in) and push (fan-out) scenarios. Both scale easily when workloads increases and do not require you manage infrastructure. In the following situation, the capture feature will send the wind farm data (per turbine) to the event hub.

The Event Hubs Capture pulls the data from the stream of windfarm data and generates storage blobs with the data in Avro format. Subsequently, when Capture creates the blob (1), the blobCreated event is generated and emitted to Event Grid (2). Subscribers to the event will receive it and handle it accordingly, in this case, an Azure Function (3) will get the event. Furthermore, the function will retrieve the file (URL to the Avro file) from the blob storage (4) and store it into a SQL Data Warehouse (5). This sample is available on the Microsoft docs: https://docs.microsoft.com/en-gb/azure/event-grid/event-grid-event-hubs-integration

Wrap up

This blog discusses Event Grid scenarios where Event Grid plays a central role, and how developers can leverage this service. Another scenario showed how Event Grid could collaborate with Event Hubs in high volume ingestion and migration of event data.

Event Grid is globally available in several Azure regions of the U.S., Europe, and Asia, with more to come soon. For pricing of Event Grid, details see the pricing page. Furthermore, the full documentation on the Azure Event Grid service is available on the Microsoft Documentation site.

Exposing Azure Services using Azure API Management

Surya Venkat — Mon, 05 Aug 2019 12:37:31 +0000

Originally Published at Serverless360 Blogs

Azure API Management

With Azure API Management we can expose our services in a managed way, allowing to take control through policies, add security, gain insights, provide decoupling between frontend and backend, and much more. API Management enables us to create a service repository, where we can expose all our services to clients, which can quickly start using these – thanks to the documentation and testing capabilities presented by the developer portal. By implementing various of the available policies, we can enrich our services with capabilities like caching, throttling, advanced security, and even protocol and data translations. In short, API Management gives the option to create a complete ecosystem around our services, including everything from development and publishing to customization and monitoring.

This gives us various possibilities around exposing our services, not only in the cloud but on-premises as well. Moreover, we can also present several of our Azure services this way. Some of these can be exposed in API Management out of the box, like API Apps or Logic Apps, but as the other Azure services also provide REST endpoints, we can use these as well. This way, we can easily publish a large variety of services to our clients through a single repository, with a homogenous experience to consume all these different services.

Exposing services

All of this starts by creating an API management instance, which is used to publish our services. If you haven’t done so yet, follow these instructions to create your own. During this blog post, we will assume you have API Management set up, including the basics like users, groups, and products. When we add a new API to API management, we will find that there are several different options to do this. Some of these will be discussed in the following paragraphs, while others provide the capability to add our own services running outside of Azure. For example, an API with an OpenAPI specification, which is the successor of Swagger, can be added through here. You just point to the location of you’re the OpenAPI definition, and API Management will pull in all information around the operations, messages, etc. Another option is to load an API from a WSDL, allowing to even expose SOAP services. Consequently, the SOAP service can then be exposed directly, however, we even have the option to expose this as a REST API.

API Apps

Now let’s have a look at the various options for presenting our Azure services through API Management. Besides the options to load definitions like OpenAPI, WADL or WSDL, we also have the opportunity to load several of the Azure services directly. The first example of this is API Apps, which allow to build and host RESTful services in Azure. By choosing the API Apps option on the Add a new API screen, we will be presented with an overview of all API Apps available in our subscription.

We can just select one of the API Apps, update the names and suffix if required, and the API definition will be created in our API Management. Altogether, it took us only a few seconds to add the API App through API Management, where we can now configure it as desired, like adding additional policies or security.

Logic Apps

Similarly, we can also use API Management to expose our Logic Apps to our consumers. This does require the Logic App to be triggered using an HTTP based trigger like the Request trigger.

When we choose to add a Logic App from API Management, we will find an overview of all the Logic Apps which are usable in this manner. Once again, this allows us to quickly and easily publish another Azure service through API Management.

Functions

Next, we have Azure Functions, which once again can be imported from the Add a new API screen. Important to note here is that to expose a Function through API Management, the Function needs to use an HTTP trigger, and the authorization needs to be on an anonymous or function level. Additionally, it’s even possible to select multiple Functions inside a single Function App, which will then all be exposed through the same API in API Management as different operations.

Service Bus

We now have seen how to expose various services directly through the API Management user interface, but how about services which do not offer this option? Luckily, we can leverage the REST endpoints of these services. For example, Service Bus offers various endpoints, which we can use to send and retrieve messages to queues and topics. Service Bus does, however, require us to use a WRAP token to authenticate against the service. In API Management we can leverage a policy to expose the Service Bus entities, allowing our clients to, for example, send in messages over HTTP, without them needing to know all details about how to create a WRAP token. To do this, we will create a blank API in API Management using our service namespace URI as the Service URI (f.e. https://yournamespace.servicebus.windows.net/), and then add a new POST operation to the API. Next, we open the policy editor for the inbound processing of the operation, where we will set up the policies needed.

The policies we need to set up will set the necessary headers while using variables to pass in the needed data. As you will find, we will execute some C# code from the policy to set the Authorization header, which is needed since we need to calculate some of the values expected in this header. Remember, the SAS key should be created in Service Bus with the correct permissions, like send or listen.

<policies>

    <inbound>

        <base />

        <set-variable name="resourceUri" value="https://yournamespace.servicebus.windows.net/yourqueuename" />

        <set-variable name="sasKeyName" value="yoursaskeyname" />

        <set-variable name="sasKey" value="yoursaskeyvalue" />

        <set-header name="Authorization" exists-action="override">

            <value>@{

// Load variables

string resourceUri = (string) context.Variables.GetValueOrDefault("resourceUri");

string sasKeyName = (string) context.Variables.GetValueOrDefault("sasKeyName");

string sasKey = (string) context.Variables.GetValueOrDefault("sasKey");

// Set the token lifespan

System.TimeSpan sinceEpoch = System.DateTime.UtcNow.Subtract(new System.DateTime(1970, 1, 1));

var expiry = System.Convert.ToString((int)sinceEpoch.TotalSeconds + 60); //1 minute

string stringToSign = System.Uri.EscapeDataString(resourceUri) + "\n" + expiry;

System.Security.Cryptography.HMACSHA256 hmac = new System.Security.Cryptography.HMACSHA256(System.Text.Encoding.UTF8.GetBytes(sasKey));

var signature = System.Convert.ToBase64String(hmac.ComputeHash(System.Text.Encoding.UTF8.GetBytes(stringToSign)));

// Format the sas token

var sasToken = String.Format("SharedAccessSignature sr={0}&sig={1}&se={2}&skn={3}",

System.Uri.EscapeDataString(resourceUri), System.Uri.EscapeDataString(signature), expiry, sasKeyName);return sasToken;

}</value>

        </set-header>

        <set-backend-service base-url="https:// yournamespace.servicebus.windows.net/" />

    </inbound>

    <backend>

        <base />

    </backend>

    <outbound>

        <base />

    </outbound>

    <on-error>

        <base />

    </on-error>

</policies>

Now when we save the policy, we can use API Management to pass in any message, which then places these in the specified queue.

Blob Storage

Similarly, we can expose Azure Blob Storage using API Management as well. In this case, we can use API Management to expose a storage container, allowing us to have clients send messages which are then created as a blob inside the container. We will also use some C# code here to calculate the Authorization header.

<policies>

    <inbound>

        <set-variable name="UTCNow" value="@(DateTime.UtcNow.ToString("R"))" />

        <set-variable name="ContainerName" value="@(context.Request.Headers.GetValueOrDefault("Container"))" />

        <set-variable name="BlobName" value="@(context.Request.Headers.GetValueOrDefault("Blob"))" />

        <base />

        <set-header name="Authorization" exists-action="override">

            <value>@{

                    string storageAccount = " yournamespace ";

                    string storageKey = "yourstoragesaskeyvalue";

                    string containerName = context.Variables.GetValueOrDefault<string>("ContainerName");

                    string blobName = context.Variables.GetValueOrDefault<string>("BlobName");

                    string contentLength = context.Request.Headers.GetValueOrDefault("Content-Length");

                    string ContentType = context.Request.Headers.GetValueOrDefault("Content-Type");

                    string dateToSign = context.Variables.GetValueOrDefault<string>("UTCNow");

                    string headerResource = $"x-ms-blob-type:BlockBlob\nx-ms-date:{dateToSign}\nx-ms-version:2016-05-31";

                    string urlResource = $"/{storageAccount}/{containerName}/{blobName}";

                    var stringToSign = string.Format("PUT\n\n\n{0}\n\n{1}\n\n\n\n\n\n\n{2}\n{3}",contentLength,ContentType,headerResource,urlResource);

                    HMACSHA256 hmac = new HMACSHA256(Convert.FromBase64String(storageKey));

                    string signature = Convert.ToBase64String(hmac.ComputeHash(Encoding.UTF8.GetBytes(stringToSign)));

                    string authorizationHeader = String.Format("{0} {1}:{2}", "SharedKey", storageAccount, signature);

                    return authorizationHeader;

                }</value>

        </set-header>

        <set-header name="x-ms-date" exists-action="override">

            <value>@(context.Variables.GetValueOrDefault<string>("UTCNow"))</value>

        </set-header>

        <set-header name="x-ms-version" exists-action="override">

            <value>2016-05-31</value>

        </set-header>

        <set-header name="x-ms-blob-type" exists-action="override">

            <value>BlockBlob</value>

        </set-header>

        <set-backend-service base-url="@{

            string containerName = context.Variables.GetValueOrDefault<string>("ContainerName");

            string blobName = context.Variables.GetValueOrDefault<string>("BlobName");

            return String.Format("https://yournamespace.blob.core.windows.net/{0}/{1}", containerName, blobName);

        }" />

    </inbound>

    <backend>

        <base />

    </backend>

    <outbound>

        <base />

    </outbound>

    <on-error>

        <base />

    </on-error>

</policies>

Conclusion

As we have seen, API Management allows us to easily expose our Azure services, both out of the box or by making use of policies. We can now assign the various APIs we created in API Management to our products, which expose them to the clients. As the consumers won’t need to know anything about the underlying services, they can consume the exposed APIs in a consistent manner over HTTP, using a single key, and from a single repository. In the meantime, we gain the advantage of advanced insights into the usage of the services and can apply all the capabilities provided by API Management, like additional security and advanced policies.