Forem: Ishwar398

Using Azure Doc intelligence for OCR

Ishwar398 — Mon, 06 Nov 2023 22:04:35 +0000

Whenever we need to read text from a PDF File, Image, Doc file etc. we use Optical Character Recognition (OCR). With OCR, we can read a document, handwritten or typed, across all the supported formats.
Azure Document Intelligence is an AI service which embeds the intelligence of AI in performing OCR. There are many use cases of Azure Document Intelligence other than OCR, but for this post we will stick to OCR.

Creating the Azure Document Intelligence service

Search for Azure Document intelligence on the Azure portal and the click on Create.
Fill in the details like the Subscription, Resource Group, Region, Name and the Pricing Tier.

Performing OCR

We can perform OCR on a document using 2 ways.

Using the File URL (40MB size limitation for F0 tier)
Using the actual file (4MB size limitation for F0 tier)

Using the File URL

When we need to perform OCR on the file that's present on some storage or is hosted, we can use the URL of the file to perform OCR on it. The only requirement here is that the URL should be publicly accessible. If the URL is not publicly available, Document Intelligence will not be able to read it.
When using this way, the POST request body should contain the URL of the file.

{ 
   'urlSource': 'URL_OF_THE_FILE'
}

Along with this, the header value for Content-Type should be as follows:

Content-Type: "application/json"

Using the actual file

Now, if we need to send the file directly to the Document intelligence service, we can use this way.
Here, the POST request body will contain the file, and the header value for Content-Type will be as follows:

Content-Type: "application/octet-stream"

Document Intelligence in action for doing OCR

Once everything is setup, we can use the Document Intelligence service using REST API. There are other options available as well. But for this post, we will be focusing on REST API.
Getting the OCR results from Document Intelligence is a two step process.
The first step is to upload the file using any of the desired way i.e. either by sending a file directly or by providing the File URL.
This will provide us with the Result ID.

The second step is to use the Result ID to get the results.

First. let's try using the File URL.
I'll be using Postman to call the API endpoints.

Step 1:
We need to send the document for analysis. The API endpoints should be as follows:

{endpoint}/formrecognizer/documentModels/{modelID}:analyze?api-version=2023-07-31

Endpoint: The endpoint provided on the Azure portal for the Document Intelligence service
modelID: prebuilt-document (using this model for OCR)

Step 2: Setting up the headers
Apart from the normal headers, we need to add two headers.

Ocp-Apim-Subscription-Key: Get this key from the Azure portal
Content-Type: "application/json"

Body:
Currently, I'm considering a dummy hosted PDF file

{ 
   "urlSource": "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"
}

Send this as a POST request. If everything is correct, you'll get a 202 Accepted response. Check the response header. You'll get a apim-request-id in the headers.
Copy this request id.

Step 3: Getting the results
To Get the results for OCR, we need to make a GET request.

https://{endpoint}/formrecognizer/documentModels/{modelId}/analyzeResults/{resultId}?api-version=2023-07-31

Endpoint: your azure document intelligence service endpoint
modelID: prebuilt-document
resultId: apim-request-id from the first step

Make this GET request. If everything is correct, you'll get a 200OK response along with the content of the PDF file.

Using the actual file

Now, if we need to send the file to the service instead of the File URL, only 2 things will change in the Step 1.

Content-Type: "application/octet-stream"

In the body, instead of the File URL, we need to send the actual file.

Send the request, if everything is correct you'll get the apim-request-id, which can then be used in similar way.

Adding Cosmos DB to Azure Search index

Ishwar398 — Mon, 06 Nov 2023 19:30:39 +0000

In continuation to Setting Cognitive Search with Blob Storage

In the above post, we saw how can we integrate Azure blob storage with Azure cognitive search. In this post, we will learn how can we create an Indexer with Cosmos DB, and push the indexer to the same index where Blob Storage indexer is pushing results to.
Due to this, search will be done across multiple sources i.e. Blob and Cosmos.

We already have Azure cognitive search and Cosmos DB set up.

Cosmos DB data:

We have a cosmos DB data present in a specific format. We can decide on which fields we want to be searchable. Based on that selection, we need to decide on the fields which we will be using in the index.
Below is a format which we will be using for setting up the index.

{
        "id": UNIQUE_ID,
        "filename": FILENAME,
        "format": FORMAT,
        "tags": [
            {
                "type": EXAMPLE_SEARCH_FIELD,
                "city": EXAMPLE_SEARCH_FIELD
            }
        ]
    }

In the above format, we will be keeping the filename, format and the tags as a searchable entity.

Setting up the data source in Azure Cognitive Search

Just as we added the Blob Storage as the data source, we will be adding a Cosmos DB data source as well.

Click on the Data Sources option from the left pane and click on the Add Data Source button.

Select Cosmos DB from the Data Source and give it a name. Then click on the Choose an existing connection link. This will bring the results of the Cosmos DB accounts in that resource group.

Then, the dropdown for the Database and Collection will be filled. Select appropriate items from the dropdown.

Now, we need to write a query which will run against the DB and bring the relevant results. This query should always bring the results in incremental manner, which means it should only fetch the results which are new and not present in the Search index. To do so, it takes help of the _ts field automatically provided by the Cosmos DB. And it maintains the values of the last _ts value in @HighWaterMark field.
In our case, we need the above mentioned fields as Searchable.
So, our query will look like this.

SELECT c.id, c.filename, c.format, t.type, t.city FROM c join t in c.tags WHERE c._ts >= @HighWaterMark ORDER BY c._ts

Save the data source, it will be now visible in the Data Sources list.

Setting up the Indexer

Click on the Indexers option from the left pane, and then click on Add Indexer.
Give the name to the Indexer, select the target index, and then select the newly created data source

Fill the schedule as per requirement. As per the schedule, the Indexer will run and fetch the data from the Cosmos DB.

Save the indexer, and then run it. It will fetch the data from the DB.

Setting up the index
Now, once we have the data source and indexer working fine, lets set up the index.
Click on the index which we selected as the target index in the cosmos indexer. And then click on Fields. You can also see the number of documents currently present in the index.

Add the fields which we selected from the cosmos db query.
You can select which fields you want as Searchable, Retrievable, Sortable, Facetable etc. which adding the fields.

You can try the Search option in the Index. The entries from the Cosmos will be shown in the Search results.

Azure Cognitive Search - Searching documents with Blob Storage

Ishwar398 — Mon, 02 Oct 2023 07:31:08 +0000

We will setup a complete flow on how we can use both Cosmos DB and Blob storage together in Azure Cognitive Search.
Azure Cognitive Search is a powerful tool which can help in searching documents across both Cosmos and Blob based on various parameters.
Cosmos DB provides the parameters of the file which can be helpful in searching like extension, filename, author.
Blob Storage, where the actual files are present, can be used in Search when we have to search the document based on the content in the document.
The terms which we will be using in this post will be referenced from this post which talks about the theoretical details of Azure Search.

Setting up the data sources (assuming Blob and Cosmos are already created)

From the left pane in the Azure Cognitive Search, select the data sources option. Click on the Add Data Source button.

Cognitive Search gives us option to select from a variety of sources.

For this post, we will be selecting the Azure blob storage. On selecting Azure blob storage, it will ask for selecting the 'Connection string'. There an option below to select from existing connection.
On clicking this, you'll get the list of Blob storages in the subscription. Select the blob storage and then select the container.

Once you select the instance, the connection string will be automatically populated. Provide a name to it.
If you need to read the files from a specific folder in the container, you can add the path in the 'Blob Folder' box.

On the Search's data source section, you'll be able to see the data source.
Now click on Indexes option, and click on Add Index.
Then we need to add the fields on which we will be making a search.
By default, when we select Blob storage as Data Source, the indexer is able to perform an OCR on the text documents (PDF, Word, RTF etc.) and add the content of the document in the search index. To make the document searchable by the content within them, you'll need to add the content field in the index and mark it as Searchable.
This will be a string search and there needs to be an Analyzer selected for it.
Analyzer defines your searching strategy, how your search query will be parsed, how the AND, OR, NOT will be handled. For this example, we will be using the Standard Lucene parser.

Once the index is created, it will be shown in the Index list.

Now we will start creating the Indexer. Indexer is the link between the Data Source and the Index.

Click on Indexers and then click on Add Indexer.
Fill the basic details.

Give a proper name to the indexer
Select the target index where the indexer will push the data
Choose the data source from where the indexer will pull the data

Skills is something we will see in a different post, so for now skip it.
Select the schedule when the indexer should run and fetch the data from the data source.

In the Advanced settings, you can set which data types you want to include or exclude. Also, add enrichments etc.

Indexer in action

The data source configured has 2 documents in it.

When we run the indexer, it fetches the documents in this container.

Same document count we'll be able to see on the index as well since the indexer has pushed the data to the target index.

Search in action

The document uploaded to the search is as follows:

By default, the Azure cognitive search provides OCR on the text documents (PDF, DOCX, RTF etc.), so it is able to fetch the content from the PDF file we uploaded and index the content as well.
So we if try to search for anything from the content, it brings back the file as result.

Publishing your Xamarin app to Apple Store

Ishwar398 — Sat, 16 Sep 2023 12:49:29 +0000

1 Introduction

Cross platform mobile application development is a growing phenomenon due to its single code approach. Development of cross platform application with these new technologies is quick, but what matters more is how the apps that are developed on a single code base can be published to the respective application stores, what all kinds of testing is required for it and many more. This paper helps to cover these topics which will be useful to test single code base Applications before they are published to the App Stores.

2 Submitting an iOS application

2.1 Preparing an application for release

2.1.1 Set up AppID and Entitlements

An AppId is needed for submitting an application to Apple App Store. AppId is unique for every iOS App. Every AppId is associated with a set of entitlements. To generate an AppId, follow the below process.

Click on the ‘Identifiers’ option from the side menu and Click the Add button.

Select ‘AppId’ from the list and click on ‘Continue’. Give a ‘BundleID’ and Description in the required boxes.

Select the ‘Capabilities’ from the list below. Capabilities are defined as the permissions to the application to the services provided by Apple like CloudKit, In-App purchases etc.

2.1.2 Set up App Store Provisioning Profile

Provisioning profile is used to control how the application is deployed. A developer can create profiles like Alpha, Beta and Production to have an efficient release of the application to a selected set of people. To create a Provisioning Profile, click on the Profiles option in the side menu.

Click on the ‘+’ icon to create a new profile. On the next screen, select ‘iOS App Development’ radio button. In the Distribution section, there are various options.

*1. Ad Hoc: *
This option can be considered as Beta release where the build is only available for a limited number of registered devices.

2. Developer ID:
This option can be considered as Alpha release where the build is only available for devices with same AppleID signed devices.

3. App Store:
This option can be considered as Production release, as the build is available on the App Store and can be accessed by the public.

2.1.3 Prepare the set up for building the application

After the development stage of the application is complete, the next step is to publish the app to App Store. Building an iOS app requires a Mac Book or a Mac Build Host setup. This is required because building Native iOS requires Apple’s Build tools which only is compatible with Mac.

2.1.4 iOS Build Configuration

Considering Visual Studio as the IDE, right click the Project Name and click properties. Navigate to the iOS build tab.

Set the Configuration to ‘Release’ and Platform to ‘iPhone’.

If the app is targeted to a specific SDK version, select it from the dropdown, else keep it ‘Default’.

Linker behavior is the property which helps in reducing the size of the application by deleting the unused code. Mostly, this should be set to ‘Link Framework SDKs only’ to avoid any issues with third party libraries.

The targeted architecture is to be selected in the ‘Supported Architecture’ dropdown. All apps targeted to iOS11 and above need to support 64-bit architectures.

LLVM (Low Level Virtual Machine) compiler can be used to build smaller and faster code. This increases the compile time of the application.
Optimize PNG images can be checked to further reduce the size of the application

2.1.5 iOS Bundle Signing

Click on the ‘iOS Bundle Signing’ option from the side menu.

Set the configuration to ‘Release’ and Platform to ‘iPhone’.

Select the ‘Manual Provisioning’ radio button in the Bundle Signing Scheme. Selecting this option enables to manually map the Generated AppId from the Developer Account portal to the application.
In the ‘Manual Provisioning’ section, set the ‘Signing Identity’ as ‘Distribution (Automatic)’.

Select the ‘Provisioning Profile’ wherein the application is to be released.

Save the configuration and close it.

2.2 iTunes Connect

iTunes connect is a suite of web-based tools provided by Apple for managing iOS Applications on the App Store.

2.2.1 Configuring an App in iTunes Connect

Create a new App on iTunes connect using the ‘+’ button.

Upon clicking the ‘+’ button, a prompt asks for information as below:
a. Platform: iOS / tvOS
b. Name
c. Primary Language of the Application
d. Bundle ID
e. SKU (Unique Identification for the app)
f. User Access (Choosing the audience)

2.2.2 Provisioning an App for App Store Distribution

Irrespective of the release plan, a Distribution profile is needed for releasing an app to App Store. This profile allows the application to be digitally signed for release so that it can be installed on an iOS device. A Distribution Profile contains the following things.

App Id
Distribution Certificate

2.2.2.1 Creating a Distribution Certificate

Click on the ‘Certificates, Identifiers and Profiles’ Section on the side menu. Then select ‘Certificates’ from the side menu and click ‘+’ button.

Select the ‘iOS App Development’ in the Software Section.

Also select the services that are being used by the application.

2.2.2.2 Creating a Certificate Signing Request (CSR)

Certificate Signing Request is created using the Keychain Access tool on Mac. In the dropdown menu, click Keychain Access > Certificate Assistant > Request Certificate from a Certificate Authority.

Enter the Email ID and name associated with the Apple Account and check ‘Save to Desktop’ and click continue.

Once the CSR certificate is created, upload the certificate to Apple Developer Account page and click on Generate.

Download the certificate and install it. Once the certificate is installed, the app can be published to the App Store.

2.2.3 Publish and Submit the App to App Store

When all the release and build configuration is complete, set the Solution Configuration to Release and Solution Platform as iPhone.

From the Build menu, select Archive. Archive Manager will pop up. Once the archive is created click on ‘Distribute’.

After clicking Distribute, Publishing Wizard will open. Select the desired Distribution Channel and fill the identity details and Provisioning Profile name.

Click on ‘Upload to Store’. A prompt will ask for Apple ID and Password. Fill in the ID and Password and click ‘OK’. The app will start to upload to App Store.

Once the upload is complete, login to iTunes and select the app. It will show the status as ‘Waiting for review’ with version details.

2.2.4 Managing App Screenshots

For publishing an app, App Store requires a set of screenshots and videos (optional). The screenshot consist of actual views of the application running and showing the unique features. Apple requires screenshots for every screen size and resolution that the application supports.

Publishing your Xamarin app to Google Play Store

Ishwar398 — Sat, 16 Sep 2023 12:34:36 +0000

1 INTRODUCTION

2 DEVELOPMENT STAGE

2.1 Identifying Targets

Before beginning development, the target devices and platforms are to be identified. Because this will smoothen the process of technology selection. The questions asked during this stage revolve around few major areas consisting of Device Types, Operating Systems and Versions.
.

2.2 Finalizing technology

Based on the selection of targets in the prior stage, the technology needs to be selected. Main factor while the selection is how feasible is it to be used on the identified target platforms and what the native abilities offered.

2.3 Implementation

Once all the technology is finalized, the architectures are kept on the tables. Upon drafting the architecture of the application, coding for the application is started. Over the time and the choice of Software Engineering methodology, like Agile or waterfall, the application starts taking shape. Once few deployable modules are complete, testing of the modules start.

3 TESTING STAGE

Analysis and Testing is an important activity in any software development process and specially in case of Mobile Applications, it is important for providing dependable solutions for end users. Currently there are various types and stages of testing involved in testing mobile applications. Along with the testing team, developers are also responsible for performing some kind of testing.

3.1 Manual Testing

Testing teams manually has to execute test cases without using any kind of Automation Tool. This is the most primitive type of testing. Any new application has to be tested manually at initial stages before automating the test cases.

3.2 Unit Testing

This testing is to be performed by Developers. Unit is the smallest part of the software. Whenever a developer develops a module, he has to test that module. Main aim of this testing is to ensure that the basic functionality of the developed module are working.

3.2.1 Platform Dependency testing

When the technology which is to be used is a single code-oriented design, the features developed are to be scrupulously tested on the platforms which the app is supposed to run on. This ensures that the native features used in the code are working as expected on all the intended platforms.

3.3 Integration Testing

Once all the units are developed which can together, logically serve as an independent functionality, these units are merged.
The testing that takes plan on these merged units is Integration testing. This testing ensures that the units which were developed individually, when integrated with each other, work properly.

3.4 Beta Testing/Third Party Testing

Beta Testing is one of the Acceptance Testing types, which adds value to the product as the end-user validates the product for functionality, usability, reliability, and compatibility. Beta Test provides a complete overview of the true experience gained by the end users while experiencing the product. This testing is done by external members which are not the part of development or testing team.

3.4 GUI Testing

GUI testing is defined as the process of testing the system's Graphical User Interface of the Application Under Test. GUI testing involves checking the screens with the controls like menus, buttons, icons, and all types of bars - toolbar, menu bar, dialog boxes, and windows, etc. This stage ensures that the UI look and feel is the same across all the supported devices and platforms.

4 PREPARING AN APPLICATION FOR RELEASE

4.1 Preparing an Android app.

4.1.1 Preparing an application for release

4.1.1.1 Specify the app icon

It is strongly recommended that each Xamarin.Android application specify an application icon. Some application marketplaces will not allow an Android application to be published without one. The Icon property of the Application attribute is used to specify the application icon for a Xamarin.Android project. In these examples, @drawable/icon refers to an icon file that is located at Resources/drawable/icon.png

4.1.1.2 Version the Application

Versioning is important for Android application maintenance and distribution. Without some sort of versioning in place, it is difficult to determine if or how an application should be updated. To assist with versioning, Android recognizes two different types of information:

Version Number – An integer value (used internally by Android and the application) that represents the version of the application. Most applications start out with this value set to 1, and then it is incremented with each build. This value has no relationship or affinity with the version name attribute (see below). Applications and publishing services should not display this value to users. This value is stored in the AndroidManifest.xml file as android:versionCode.

Version Name – A string that is used only for communicating information to the user about the version of the application The version name is intended to be displayed to users or in Google Play. This string is not used internally by Android. The version name can be any string value that would help a user identify the build that is installed on their device. This value is stored in the AndroidManifest.xml file as android:versionName. You cannot upload an APK to the Play Store with a versionName you have already used for a previous version.

4.1.1.3 Understanding Android API levels

Target Framework – Specifies which framework to use in building your application. This API level is used at compile time by Xamarin.Android.

Minimum Android Version – Specifies the oldest Android version that you want your app to support. This API level is used at run time by Android.

Target Android Version – Specifies the version of Android that your app is intended to run on. This API level is used at run time by Android.

Before you can configure an API level for your project, you must install the SDK platform components for that API level.

On the Android Manifest page, set the Minimum Android version to Use Compile using SDK version and set the Target Android version to the same value as the Target Framework version (in the following screenshot, the Target Android Framework is set to Android 7.1 (Nougat)):

If you want to maintain backward compatibility with an earlier version of Android, set Minimum Android version to target to the oldest version of Android that you want your app to support

4.1.2 Compile

Select Build > Rebuild Solution to verify that it builds successfully in Release mode. Note that this step does not yet produce an APK.

4.1.2 Archive for Publishing

4.1.2.1 Archive

To begin the publishing process, right-click the Solution in the Solution Explorer and select Archive All..., which builds the solution and archives all Xamarin projects that can generate an archive:

This will load the archive manager.

4.1.2.2 Distribution

When an archived version of the application is ready to publish, select the archive in the Archive Manager and click the Distribute... button:

The Distribution Channel dialog shows information about the app, an indication of distribution workflow progress, and a choice of distribution channels. On the first run, two choices are presented:

It is possible to choose one of the following distribution channels:

Ad hoc (We’ll be considering this option)
Google Play

4.1.2.2.1 Publishing using Ad hoc option

After Ad-Hoc is selected, To publish the .APK, it must first be signed with a signing key (also referred to as a certificate).

An existing certificate can be used by clicking the Import button and then proceed to Sign the APK. Otherwise, click the click the + button to create a new certificate:

The Create Android KeyStore dialog is displayed; use this dialog to create a new signing certificate that can be used for signing Android applications. Enter the required information (outlined in red) as shown in this dialog:

After the information is provided, click on create button.
The resulting keystore resides in the following location:

C:\Users\USERNAME\AppData\Local\Xamarin\Mono for Android\Keystore\ALIAS\ALIAS.keystore

Once the keystore is generated, make a backup of the same.

Sign the APK
When Create is clicked, a new key store (containing a new certificate) will be saved and listed under Signing Identity as shown in the next screenshot. To publish ad-hoc, select the signing identity to use for signing and click Save As to publish the app for independent distribution.

Next, the Archive Manager displays the publishing progress. When the publishing process completes, the Save As dialog opens to ask for a location where the generated .APK file is to be stored.

If the key password is unknown, the Signing Password dialog will appear to prompt for the password for the selected certificate

After the signing process completes, click Open Distribution

This causes Windows Explorer to open the folder containing the generated APK file. At this point, Visual Studio has compiled the Xamarin.Android application into an APK that is ready for distribution. The following screenshot displays an example of the ready-to-publish app, App.apk.

4.1 Publishing the app to Google Play Store

Before publishing the application to Google Play store, there are various security checks have to be validated and various operations are to be performed within security domain of the application.

4.1.1 Checks before publishing the app to Play Store

4.1.1.1 Obfuscate Source Code

When we build the app, the source code gets compiled into the library and files as per the selected build configuration. In case of Android, the code is bundled into an .apk file or app bundle files. But there are software packages which can convert the library files back to source code files which can result in security issues. Obfuscation is a process which will not allow to convert the library files back to source code files. To obfuscate code in Android Studio, navigate to the build.gradle file in the Android Studio Project and set the minifyEnabled property to true from false.

4.1.1.2 Disable Debugging

The debugging flag in the AndroidManifest.xml file should be set to false. The name of the flag is android:debuggable.

4.1.1.3 Other checks

The other checks which are the be performed are

Ensuring that the application transmits data over HTTPS.
Disable logging in the application
Ensuring that the application has emulator checks.
Ensuring the passwords are hashed and not encrypted.
Ensuring Local Data storage is encrypted
Avoid use of Custom Encryption Algorithm

4.1.2 Submitting the application to Play Store

4.1.2.1 Using the Google Play Console

Once logged in, select the ‘All Applications’ tab from the side menu.

Click the create application button in the top right corner.

A prompt opens up asking for the default language of the application and the name of the application.

Once the app is created, select the ‘App Releases’ tab from the side menu. Google Play console offers three tracks for App Releases.
A. Production Track
This is the Main track which will be available for public

B. Open Track
This is the Beta track which will only be available for beta testers. Beta testers are people outside the development team.

C. Closed Track
This is the Alpha track which is only available for people inside the development team.

4.1.2.2 Upload the APK or app bundle files for your app

This stage give developer the ability to limit the reach of the application to specific people only. Here the application can be assigned to a Alpha Track, Beta Track or the Production Track. Choosing the correct track is important because Alpha and Beta track help in getting feedback from real end users. After selection of the track, the .apk and bundle files are to be selected and uploaded to the Play Store.
Select a track where the application is to be deployed. It will ask you for the APK file and a release name. It will also ask for release notes which consists of new features in the release.

4.1.2.3 Preparing the information about the application

After uploading the APK file, click on the ‘Store Listing’ tab from the side menu. Basic information about the application is required before submitting the application to Play Store. The information required is listed below:

Short description about the application
Brief description about the application
Screenshots of various screens
Hi-resolution Icon
Video link
Application Type
Category

4.1.2.4 Set the rating of the Application

Once the application files are submitted successfully, in the next stage the submission team has to answer some questions. Based on the responses to these questions, the age rating is calculated for the application. This helps in identifying if the application is appropriate for which age group.

4.1.2.5 Set pricing and distribution plan

The developer can select if the application needs to be a paid one or a free one. The countries in which the application should be available can also be selected on this page.

4.1.2.6 Publish the App

Once all the details are filled, the developer can publish the app. Once published, it will be reviewed by the people at Google for validity of the information. If any of the security check is violated the application will be rejected. If no security violations are found, the application will be published successfully.

Getting started with Application Insights in .NET Web API

Ishwar398 — Sun, 20 Aug 2023 05:18:26 +0000

Monitoring an application is one of the most important part in Software Lifecycle. With help of Monitoring tools, we can see whats going inside the application, predict any infra scaling, see how users are using the application etc.
Azure Application Insights offers powerful application performance and monitoring tools. It tracks user interactions, traces requests, measures performance metrics, and monitors application health. With features like custom event tracking and real-time analytics, it helps developers identify issues, optimize performance, and enhance user experiences in cloud-based applications.

Pre-Requisites:

Azure Account
.NET Web API project
Visual Studio

Configuring Application Insights in Web API project

Open the solution in Visual Studio
Right click on the solution.
Click on 'Configure Application Insights'

This will open a pop up asking for connecting to the dependency. If you need to use application insights on local machine, click on Application Insights SDK (Local). If you want to connect to an instance on Azure, click on Azure Application Insights. For this post, we will be connecting to an instance on Azure.

On clicking next, we need to select the Application Insights instance. If not present, there is an option to create a new and then select it.

Next, it will show us the Application Insights connection string which is to be saved in the User secret file.
On the next step, it will add the necessary config files and Nuget packages to the application.

Deployment and Application Insights portal

Once the changes are deployed, you can visit the Azure portal to check the newly created Application Insights instance. It will start showing stats for your application in some time.

Checking Failures in API

Here's an example on how we can check the exceptions which were throw by the API and their details.
In the overview page of Application Insights, we will see a Failures chart. This option is also present in the left pane.

Once you click on any of the above highlighted option, you'll get the list of the failures along with the exceptions by HTTP Status code, exception types etc. along with the operation name.

If we click on the type of exception, we will get complete list of the exception of the selected type. We can then click on any item from the list to get the complete details about the exception.

Getting started with Azure Cognitive Search

Ishwar398 — Mon, 14 Aug 2023 06:10:41 +0000

Azure Cognitive Search is a cloud-based search service by Microsoft. It empowers seamless integration of powerful search capabilities into applications. It offers AI-driven indexing, faceted navigation, and natural language processing. It scales dynamically, supports diverse data sources, and enhances user experiences with accurate, fast, and relevant search results.

Creating a Cognitive Search Service

On the Azure Portal, search for Cognitive Search. Then click on 'Create Search Service'

Select the resource group and give a unique name as this will be the part of the Search URL
Click on Next, which will take us to the Scaling section. For this post, we will not look into the scaling part.
Click on Next, and then 'Create + Review'
Once the resource is created, head over to the newly created service.

Important parts of Search Service

The above Search Management options are most important options in setting up the Search Service. We will look at them one by one.

Data Sources

In this section, we can setup the input data sources for the Search Service.
There are multiple options we can use as Data Source.

Blob Storage
Data Lake
Table
Cosmos DB
SQL Database

On selecting any option, it will ask us for setting up the connection to the respective resource. Specially for Cosmos DB and SQL Database, we need to provide a database query which the Search Service will execute on the database and fetch the data into Search Service.

Indexers

Once the data sources are created, we need to create Indexers. The main responsibility of Indexers is to get the data from data sources and feed it to the Search Index. It acts like a middle ware. Here, we can add Skillsets to the data like performing OCR, Entity Extraction etc.
We can set a schedule for Indexers which will ensure that the Indexers fetch the latest data from the data source at particular time intervals.

Once the basic details are filled, we can scroll and set the schedule.

Enabling the Base64 encoding ensure that if there are any special characters in the Key like '.' etc., this characters will get encoded so that this keys will not cause any issue in the URL.
Enable Incremental Enrichment - This option will ensure that if any skillsets or AI Processing is applicable to the resource, the enrichments will be cached so that enrichments are only applicable to new documents. This enrichments are stored in Azure storage.
Default Batch size - The number of documents which are to be processed by the indexer at a time, by default this value is dependent on the data source.
Indexer Cache Location - The storage where the enrichments are to be stored
Max Failed Items - Threshold to set which can define a max number of documents that can fail during indexing and still the indexing can be termed as successful.
Max Failed Items per Batch - Number of items that can be fail.
Managed identity authentication - If the Search Service has to be authenticated using Managed Identity, then this option is helpful.

Index

Indexes are the most important part, where the search queries are executed.
In we think about the flow:

Data sources establish connection to read the data
Indexers fetch the data, clean it, send it to the index
Index gets the data from Indexers, and we can decide which fields should be Searchable, Filterable, Sortable, Facetable.

While creating the index, we can create the fields which are be included in the index, and set the properties of the fields.

Flow

Aliases

As the index name is included in the URL when a Search query is executed, we can give a different alias to the Search index so that the alias can be included in the query.

Azure Key Vault with .NET - Reading & Writing secrets from a C# application

Ishwar398 — Sun, 13 Aug 2023 14:07:14 +0000

Once you have created a Key Vault resource, and you've set the Access Policies, the next step is to establish a connection between the application and Key Vault to perform operations like reading, writing and deleting values from Key Vault.

Install the required Nuget Packages

dotnet add package Microsoft.Extensions.Azure
dotnet add package Azure.Security.KeyVault.Secrets

Setup the appsettings file

In the Overview page of the Key Vault resource in Azure portal, copy the VaultURI.
Add a section in the appsettings file or the config file.

"KeyVault": {
    "VaultUri": "VAULT-URI"
  }

Adding the KeyVault service

In the ConfigureServices method, we need to configure our KeyVault connection
In WebApp this will be present in the Program.cs, in Console Application it will be present in the StartUp.cs

builder.Services.AddAzureClients(azureClientFactoryBuilder =>
{

    azureClientFactoryBuilder.AddSecretClient(

    Configuration.GetSection("KeyVault"));

});

Create an Interface for Dependency Injection

Create an interface which can help us in injecting the dependency.

builder.Services.AddSingleton<IKeyVaultManager, KeyVaultManager>();

Add three classes to Write, Read and Delete a secret from KeyVault

public interface IKeyVaultManager
    {
        public Task<bool> WriteSecret(string key,string secret);
        public Task<string> ReadSecret(string key);
        public Task<bool> DeleteSecret(string key);
    }

Setup the class for the interface

Using the interface above, create a class and inherit it from the above interface and implement the three methods in the class.

using Azure.Security.KeyVault.Secrets;

namespace KeyVaultConnectivity.KeyVault
{
    public class KeyVaultManager: IKeyVaultManager
    {
        public SecretClient SecretClient { get; set; }

        public KeyVaultManager(SecretClient secretClient)
        {
            SecretClient = secretClient;
        }

        public async Task<bool> WriteSecret(string key, string secret)
        {
            try
            {
                await SecretClient.SetSecretAsync(key,secret);
                return true;
            }
            catch(Exception ex)
            {
                //Log the exception
                Console.WriteLine(ex.Message);
                return false;
            }
        }

        public async Task<string?> ReadSecret(string key)
        {
            try
            {
                var secret = await SecretClient.GetSecretAsync(key);
                return secret != null ? secret.Value.ToString() : string.Empty;
            }
            catch (Exception ex)
            {
                //Log the exception
                Console.WriteLine(ex.Message);
                return string.Empty;
            }
        }

        public async Task<bool> DeleteSecret(string key)
        {
            try
            {
                await SecretClient.StartDeleteSecretAsync(key);
                return true;
            }
            catch (Exception ex)
            {
                //Log the exception
                Console.WriteLine(ex.Message);
                return false;
            }
        }
    }
}

Using this service

Inject the service in the class which has to read secrets from Key Vault and the respective method from the class.

Azure Key Vault with .NET - Vault Access Policies

Ishwar398 — Sun, 13 Aug 2023 11:29:59 +0000

Link to the post explaining the creation of Azure Key Vault resource.

Now, we will see the Vault Access Policy setup in detail.

Key Vault has to be accessed from any where. Be it App Service, Functions, Azure VM etc.
We need to ensure that all these services can access the Key Vault to read the secrets without any authentication error.

If the Key Vault Access Policies has been selected to ensure that access policies are added for all the resources which have to connect to the Key Vault.

This is a two step process:

Assign a system id to the resource which will access the Key Vault resource

We need to make sure that a System ID is assigned to the resource for which we will create an access policy
Open the Overview page of the Azure Resource

Click on Identity from the left pane
In the System Assigned section, turn on the Enabled button.
Click on Save.
Once the save is successful, an Object (Principle) ID will be assigned to the resource.

Creating the access policy

Open the Azure Key Vault resource overview and click on the Access Policies option in the left pane.

Click on 'Create(+)' button
The next step is to grant permissions
The permissions are divided in 3 parts: Keys, Secrets, Certificates
You can select from pre-defined templates or assign the permissions you need.
Example, if you want an application to just read and get list of the secrets, then just provide Get and List permission under the Secret option.

Paste the Object(Principle) ID from the first step in search, it will bring the result.

Select the application and click on Next. (Multiple applications can be selected here)
Since we have already selected the applications, skip the 3rd step
Create the access policy

Once this policy is created, the selected application will now get all the selected accesses to the Key Vault Resource.

Azure Key Vault with .NET - Creating a Key Vault Resource

Ishwar398 — Sun, 13 Aug 2023 10:27:30 +0000

Traditional Methods:
Storing passwords in config files is insecure and risky. Config files are often stored in plain text, making passwords easily accessible to anyone with access to the file. This approach leaves sensitive credentials vulnerable to unauthorized access, data breaches, and insider threats. Unlike encrypted storage solutions, config files lack proper encryption and access controls, making it difficult to safeguard the passwords effectively. Furthermore, updating or changing passwords requires manual intervention in the config files, posing challenges in terms of security and management. Adopting secure alternatives like credential vaults or key management systems is essential to prevent potential security breaches and protect sensitive data.

How Azure Key Vault can help here:
Azure Key Vault helps in storing credentials and providing it to the application during runtime. With this approach, we can eliminate the need for storing passwords in plain text config files.
Apart from credentials, Azure key vault can also store certificates and provide them as and when needed.

How to setup Azure Key Vault
There are three ways in which an Azure Key Vault resource can be created:

A. Using the Azure Portal

Open the Azure Portal (portal.azure.com)
Click on 'Create a new resource' and search for 'Key Vault'

Step 1: General Settings

Fill in all the required information for the resource group, name of the resource, region and pricing tier.
Soft delete would be enabled by default. This ensures that if a secret is deleted, it would stay in a soft deleted state from where, if required, it can be restored.
Time to retain defines how many days should the soft deleted key be retained before it is deleted permanently.
Purge Protection setting defines if the soft deleted keys can be deleted manually before the retention period is complete.

Step 2: Authorizations

Two options are provided here:
a. Azure Role Based Access control
With this option, the access to all the key vaults can be saved and used centrally.
Objects in the Azure Active Directory can be assigned to these roles
Most of the roles are already created, but if required we can create a customized roles
Any changes to the roles can affect all the resource using that role
b. Vault Access Policy
If we need an access policy to control the specific instance of the Key Vault
The access policy created only applies to the specific instance and not present centrally

Step 3: Networking

If the Key Vault is to be assigned to any VNet or Public Access has to be disabled etc. All these settings can be applied here

Step 4: Tags

Any tags to be assigned to the resource can be assigned here

Step 5: Review and Create

Review all the details
If everything is correct, create the resource

B. Using Azure CLI

We can create a Key Vault resource using Azure CLI commands
If resource group is not present, create a resource group az group create --name "DemoResourceGroup" --location "northeurope"
Then run command to create Key Vault az keyvault create --name "Unique-KeyVault-Name" --resource-group "DemoResourceGroup" --location "northeurope"

This will create a Key Vault resource. In a later blog, we will see how can we store, read, delete secrets from this Key Vault resource from a dotnet application in C#.