Forem: Wai Liu

Deleting all blobs in an Azure Storage / Data Lake container quickly

Wai Liu — Wed, 20 Sep 2023 05:36:25 +0000

The Azure CLI contains a delete-batch command to delete multiple blobs at once.

az storage blob delete-batch --account-key <storage_account_key> --account-name <storage_account_name> --source <container_name>

This article explains this in detail.

However, this command is sloooooow. If you have 100k+ records, this could take hours.

Another way to do it is programmatically. Here is an Azure function that can be deployed to delete all your blobs much faster.

public static class DeleteAllFilesInContainer
    {
        [FunctionName("DeleteAllFilesInContainer")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequest req,
            ILogger log)
        {

            string containerName = "mycontainer";
            string directoryPath = $"mydirectory";
            var client = GetDataLakeServiceClient("mydatalakeaccount", "<insert key>");
            var fileSystemClient = client.GetFileSystemClient(containerName);

            IAsyncEnumerator<PathItem> enumerator =
                fileSystemClient.GetPathsAsync("message").GetAsyncEnumerator();

            await enumerator.MoveNextAsync();

            PathItem item = enumerator.Current;

            var lstOfTasks = new List<Task>();
            int i = 0;

            while (item != null)
            {

                if (!await enumerator.MoveNextAsync())
                {
                    break;
                }

                item = enumerator.Current;
                lstOfTasks.Add(DeleteFile(item.Name, fileSystemClient));
                i++;
            }

            await Task.WhenAll(lstOfTasks.ToArray());

            string responseMessage = $"You deleted {i} files";
            return new OkObjectResult(responseMessage);
        }

        public static DataLakeServiceClient GetDataLakeServiceClient(string accountName, string accountKey)
        {
            StorageSharedKeyCredential sharedKeyCredential =
                new StorageSharedKeyCredential(accountName, accountKey);

            string dfsUri = $"https://{accountName}.dfs.core.windows.net";

            DataLakeServiceClient dataLakeServiceClient = new DataLakeServiceClient(
                new Uri(dfsUri),
                sharedKeyCredential);

            return dataLakeServiceClient;
        }


        public async static Task DeleteFile(
            string fileName, DataLakeFileSystemClient fileSystemClient)
        {
            await fileSystemClient.GetFileClient(fileName).DeleteAsync();
        }


    }

The important bits are below.

var lstOfTasks = new List<Task>();

Create a list of Tasks.

lstOfTasks.Add(DeleteFile(item.Name, fileSystemClient));

For each blob, create an asynchronous task to delete it

await Task.WhenAll(lstOfTasks.ToArray());

Wait until all the tasks to delete each of their files are finished.

This method will make an make a request for each file to be deleted and fire these requests all at the same time.

Method can be adjusted to delete only some blobs based on some sort of filter as needed.

In a container with 130k blobs it took less than 4 minutes to delete everything.

Deploying Angular apps to Azure Static Web apps

Wai Liu — Fri, 26 May 2023 04:53:13 +0000

Introduction

Azure Static Web Apps is a perfect hosting option for SPA frameworks like Angular or React.

The best thing is it's going to be free (or at least dirt cheap if you have a highly trafficked site) but not only that it contains a whole host of goodies out of the box. This includes the following

Content Delivery Network

Your files are served from edge servers so wherever your users are, there should be minimal latency.

CI/CD Pipelines

As development teams try to increase their velocity, one critical prerequisite is the ability to deploy quickly, automatically and consistently and to do that, you need to build CI/CD pipelines. This can sometimes be an afterthought, but with azure Static Web apps, this is the default. You have to specify your Github / DevOps repo as part of spinning one up and it automatically creates a Github Action / Pipeline for you.

Multi deployment environments

The "static" in Static Web Apps implies files (eg. HTML, JS or CSS or other asset files) that do not change - at least not without a deployment. This means it's really easy to create environments for different versions of them. This is built out of the box - you can easily spin up a new environment to test the changes of a PR or have dedicated test or UAT environments. It's all very flexible and very easy to implement.

Demonstration

Rather than explaining how it works in detail, it's much easier for me to just show you in the attached video. In this video, I'll be showcasing the features of Azure Static Web App to deploy a sample Angular solution. Feel free to like or subscribe!

Finally, the codebase can be viewed here

Retrieving deleted passwords in Chrome

Wai Liu — Sun, 24 Jul 2022 23:19:00 +0000

Scenario

I accidentally cleared my browsing data including Passwords in Chrome whilst synced to my chrome account

By clicking on Clear Data here

This is how you know if you're syncing to your google account

When this happens, it'll delete the passwords from the google cloud servers as well as all your devices.

How to fix

First, this is a bad situation. The only way you can retrieve the passwords is if you have a second device that is also synced to google. This could be another laptop, a mobile etc. but whatever it is - immediately, turn off the wifi on it.

This is because as soon as Chrome has a chance to go on the Internet, it'll sync and delete all the passwords on there too. For me, I had a second laptop and I drove a couple of blocks from my house to make sure it didn't accidentally pick up the wifi whilst I was unlocking it.

Exporting your password

After you have deprived the device of network access, let's immediately export the passwords so whatever happens, we know we still have them.

You can do that by going to the password screen

And clicking on Export password under the ellipsis in Saved Passwords

Note this will create a cleartext CSV file with all your passwords. This is just for backup purposes - I would delete this once everything is running again.

Turn off the sync

After I deprived the device of network access, I turned off the sync on it.

Go to Chrome Settings, You and Google and click Turn Off.

Very important here is to make sure to leave the Clear bookmarks, history, passwords checkbox unchecked.

Turn the sync back on

Once you've logged out, turn your wifi back on.

You should notice that your browser profile no longer is logged in but it still has your passwords stored against it.

If you log back in and turn on sync, it should reinstate your password back to your Google account.

If you check your other devices you should be able to see that the passwords have been reinstated on the cloud account.

Setting up a Power BI deployment pipeline with environment-specific data sources

Wai Liu — Wed, 25 May 2022 04:50:06 +0000

Introduction

Power BI Deployment Pipelines allow you to implement Application Lifecycle Management (ALM) for your Power BI Service workspaces. Today I'll show you how to use one-click deployment to elevate changes from dev through to production.

Prerequisites

You must have Power BI Premium license to use Deployment Pipelines. You should be able to get a trial for 60 days.
You'll also need Power BI Desktop installed.

Creating Premium Workspaces

The first thing to do is to create premium workspaces - you'll need one for each environment.

At app.powerbi.com, go to Workspaces / Create Workspaces

Make sure you pick Premium per user as license mode.

In our case, we'll create two workspaces - Dev and Test.

Creating pipeline

Once the workspaces are created, go and create a pipeline.

Assign your dev workspaces to the Development stage

Your Pipeline View

This is your pipeline view - you have three stages - Development, Test and Production. Each stage is attributed to one workspace. In this example, we'll demonstrate going from Dev to Test but going to Production should be exactly the same process.

The last thing to do in this view is to assign the Test workspace to the Test stage

Creating a report in PowerBI Desktop

We want to connect a report in PowerBI Desktop to demonstrate how the pipelines work. A common difference between each of your power bi environments is going to be the data source. Your dev environment should connect to the dev database, your test environment to test database and so on.

How to use Power BI Desktop is outside the scope of this article but the quickest way is to open up the application and select Get Data

Any data source is fine but for my case, I'd like to create visualisations for a Power App so I'm going to select Dataverse as the data source. It'll ask you to sign in using your Azure AD account and select the Power App environment.

In this report, I've similarly grabbed all the data in the "accounts" table.

Adding a parameter

I want to add a parameter for the data source because that is going to be customised for each stage. If you go to Transform data, on the ribbon, there should be a New Parameter button

Create the parameter by giving it a name.

Select "List of Values" under Suggested Values and put in the URLs of all the environments that you would like to deploy to and select the dev URL as the environment you'd like to deploy to.

Using the parameter in the Data Model

Instead of the report pointing to the Dev as the data source, we want it to point to the current parameter value. Therefore, for each table under the Source applied step, change the URL to be #"DataSource".

Creating and publishing a report

We'll now create a report to demonstrate the data being in different environments. For this example, I have created a simple report that simply counts the number of accounts in different statuses. These should different which will allow me to verify it's getting data from different environments.

Once you have created your report, click the Publish button on the ribbon and select the Dev workspace.

Deployed to Dev

You've successfully published your report onto the Dev Workspace inside your Power BI Service. Go to the deployment pipeline and check that inside the dev environment, you should see 1 data set and 1 report deployed. If you go to the report, you should be able to see the report in Power BI Service.

Pointing the parameter variable to Test

We've successfully deployed to Dev, we now want to deploy to Test - we want to use the same artefacts but we want to change the data source so it's pointing to the Test Database.

The first thing we should do is Deploy to Test.

Once the report is deployed to Test, we should set up a parameter rule. Click on the lightning settings icon for Test and select the Dataset you've just deployed.

Set up a parameter rule and point it to Test URL

Deploy again. Once completed, if you go to the Test workspace open the report, it should show data in Test.

Extra tips

Verifying the parameters

If you want to double check whether the parameters have been set correctly for each workspace, go to the Dataset settings and you can check under Parameters what the value is.

Setting the connection

Also, if you're having connection issues, you can also go to the Dataset settings, under Data source credentials, either log in or check "Report viewers can only access this data source with their own Power BI identifies". I generally tick this to ensure only the data the user has access to is in the report.

Moving your Azure subscription to another tenant

Wai Liu — Fri, 29 Apr 2022 01:51:31 +0000

Scenario

I have an Azure subscription attached to my university email which I mainly use just as a sandpit to try new things out on. Microsoft offers a $100 credit if you have an email from an eligible educational institution. These are handy because you have a year instead of 30 days to use it.

Unfortunately, these student subscriptions are attached to your university Azure AD tenant which sensibly - many unis are not going to give you any access to do much on.

That means I couldn't try things which required say creating app registrations or turning on P1 or P2 licenses.

This article will explain how to move this subscription over to your own tenant where you'll have full control of Azure AD. You'll still be able to have your $100 credit too and you'll still be able to access the subscription using your university credentials.

Prerequisites

You'll need a subscription with Owner access and you'll need a tenant you want to move this subscription to. There's various ways to do this - the easiest is to just create a new Azure free account.

Adding your uni account as a guest on destination tenant

Your university account needs to be a user in both tenants, so go to Azure AD and invite them in as a guest user. Your university mailbox will get an email where you'll need to accept the invite.

You should also grant your university account global admin access too.

Change directory of subscription

Once access is granted, your university mailbox is now a user in both tenants. It's time to go to the subscription and click on Change Directory and select the new tenant.

Once you click 'Change', be aware that it may take some time to cascade.

Verify it all works

After a little while, you're going to find the subscription will have disappeared. If you go and Switch Directory to your new tenant and you should see it.

On the Subscription overview page, you should see the Directory is now your new tenant.

Go ahead and try to create a new resource. You should be able to indicating you still have your $100 credit.

Accessing environment variables in your custom workflow activity or plugin

Wai Liu — Fri, 25 Mar 2022 06:01:02 +0000

Introduction

Environment variables are a great way to store global variables that differ in each of your environment. Some common use cases are storing the AppID so it can be referenced in your flows or maybe a Sharepoint site url if you are storing data there. They’re stored on the Dataverse under the EnvironmentVariableDefinition and EnvironmentVariableValue table.

Whilst they’re built into Power Automate and you can access them relatively easy, if you need to reference them in a traditional plugin or custom workflow activity, you’ll need use FetchXML to retrieve them.

Code Snippet

protected override void Execute(CodeActivityContext context)
{

    try
    {
        IOrganizationServiceFactory serviceFactory = context.GetExtension<IOrganizationServiceFactory>();
        IWorkflowContext workflowContext = context.GetExtension<IWorkflowContext>();
        // Use the context service to create an instance of IOrganizationService.
        IOrganizationService service = serviceFactory.CreateOrganizationService(workflowContext.InitiatingUserId);

        string appID = GetEnvironmentVariable(service, "appID");

    }
    catch (Exception ex)
    {
        throw new InvalidPluginExecutionException(ex.ToString());
    }
}

private string GetEnvironmentVariable(IOrganizationService service, string envVariableName)
{

    string fetchXml = string.Empty;
    EntityCollection result = null;
    string envVariableValue = string.Empty;

    fetchXml = String.Format(@"<fetch version=""1.0"" output-format=""xml-platform"" mapping=""logical"" distinct=""false"">
            <entity name=""environmentvariablevalue"">
            <attribute name=""environmentvariablevalueid"" />
            <attribute name=""value"" />
            <link-entity name=""environmentvariabledefinition"" from=""environmentvariabledefinitionid"" to=""environmentvariabledefinitionid"" link-type=""inner"">
                <attribute name=""displayname"" />
                <filter type=""and"">
                <condition attribute=""displayname"" operator=""eq"" value=""{0}"" />
                </filter>
            </link-entity>
            </entity>
        </fetch>", envVariableName);

    result = service.RetrieveMultiple(new FetchExpression(fetchXml));

    if (result != null && result.Entities.Count > 0)
    {
        envVariableValue = result.Entities[0].GetAttributeValue<string>("value");
    }

    return envVariableValue;

}

Explanation

So under Execute method, we have the standard boilerplate code to get the classes to be able to access the DataVerse.

Inside the GetEnvironmentVariable method is really where all the action is. It takes a parameter called envVariableName and then we make a call using FetchXML to retrieve the Environment Value with that name.

Trigger an instant flow from a model-driven app

Wai Liu — Sat, 12 Mar 2022 05:37:56 +0000

In this article, I demonstrated how to trigger an instant flow from a canvas app, now I will demonstrate the same but in a model driven app.

The Flow button

In a model driven app, there is a Flows button on the details page that allows you to run adhoc commands on the current record.

It's a little bit confusing because they use the word "Flow" but there's actually two ways to create these "flows". The first is the old fashioned way using on-demand workflows which you can do by just ticking the "As an on-demand process" in any clasic workflow.

The other method is by creating an instant flow in Power Automate and this article is will explain how.

Scenario

As a simple example to demonstrate the concept, what I'd like to do is create a button to allow me to email the details of a Lead to a hard-coded email address.

Create the Instant Flow in a solution

Don't click the "Create a flow" button cause that will create a personal flow. We want the flow in a solution so that we can exercise good ALM practices.

Select the "When a row is selected" trigger. This is in the old Dataverse (legacy) connector. At the time of this writing, the new connector doesn't have this trigger yet.

Your Environment should be "Default" and under Table, select the record type. For this example, I've selected "Leads".

Adding inputs

You can feed information into the instant flow by clicking "Add an input" - here I've added a field to enter extra info about the lead.

Adding the Email action

Select the Office 365 Outlook connector and Send an email (v2) action.

Fill in the email as you wish.

Once this is done, you should save the flow and publish the solution.

Executing the instant flow

To execute the flow, simply navigate to the Lead that you want. It should now appear under "Flow".

As we asked for extra input, it'll prompt for this.

Conclusion

Below is the email I get once the flow is executed.

Instant flows allows developers to easily use the features of Power Automate to add extra functionality to their model driven apps.

Copying your http request from browser into Postman

Wai Liu — Mon, 14 Feb 2022 11:28:45 +0000

Postman is a great tool for examining API calls.

Here is a quick way to get that call you want from your browser into Postman.

Copy it from Chrome

Open up the Developer Tools (Ctrl-I), go to the Network tab. Find the call you want. Right click and select Copy as cURL

Import it into Postman

Inside Postman, open up File / Import (or Ctrl+O).

Paste it into the raw text tab and press Continue

Once it's in Postman, you can now examine, manipulate, re-send the request as you wish.

Set up automation to regularly clean up your Azure environment

Wai Liu — Sat, 29 Jan 2022 06:28:57 +0000

Introduction

I'm sure I'm not the only one this happens to but how often have you set up something Azure to maybe learn about a new feature or do a demo of some sort.

So you do your work, it works but forget to delete your resource groups - then next time you check, you've just realised you've just burnt through a whole bunch of Azure compute.

This article helps you avoid this by setting up Azure Automation to delete these resources up at a regular interval.

Word of warning

Before we begin, this isn't something you should do on a production subscription or ereally any subscription where holds any data you don't want to lose.

This is intended to be set up in a PAYG subscription that you've set up to experiment on and you'd like to keep pristine.

What we want to do

The requirement is that once a day, a job will run and delete all the resource groups in the subscription beginning with "todelete".

For example, this resource group and everything inside it will be deleted once the job runs.

This may be not exactly what you want but it should be easy to customise it once you understand what's being done.

Prerequisite

You'll need an Azure subscription with contributor or higher access. We'll be using Azure Automation to accomplish this

Create an Automation account

Go into the portal and create an Automation account - make sure the resource group you assign to it DOESN'T start with todelete or it'll get deleted when the job runs.

Once it is created, Go to Identity, and assign Contributor access across the whole tenant. This creates a managed identity that gives it access to the whole tenant.

Create a runbook

Now we have set up the Automation account and given it the access we want, it's time to tell the automation account what to do.

Runbooks allows you to specify a script to run. You can run in a number of languages but for our purposes we should select PowerShell.

Once the runbook has finished, paste this gist and press Publish.,

Examining the script

The script is really simple - the first part just logs in using the managed identity of the Automation account.

try
{
    "Logging in to Azure..."
    Connect-AzAccount -Identity
}
catch {
    Write-Error -Message $_.Exception
    throw $_.Exception
}

The second part gets all resource groups starting with "todelete", pipes it to the next command which is to delete them one by one - the -force flag means there is no warning.

get-azresourcegroup -name todelete* | remove-azresourcegroup -force

If you're a little concerned about this step, you can always run get-azresourcegroup -name todelete* on its own and see what resource groups are listed - those are the ones that will be deleted.

Test to see if this works.

We've set up the automation account with the right access, we've created a runbook that has a script to delete the right resources, at this point, let's test it to see if it's all functioning.

To prove that it works, create two resource groups called "todeleteRG" and "dontdeleteRG".

Now start the runbook - once it finishes, the todeleteRG will be gone and the dontdeleteRG will remain.

Set up the schedule

Once we prove the logic works, lets set it up so that it runs overnight automatically and we do that by linking a schedule to the runbook.

Summary

And that's it - every night at 3am, your Azure subscription reverts back to an original state. Now of course, there's lots of things you can do to customise this. Maybe you want to use tags to determine what needs to be deleted instead of the resource group names? Maybe you want to do weekly deletes instead of daily deletes (maybe you can even have two runbooks - one for weekly deletes and one for dailys) - all of this is possible with just a bit of tinkering.

For me, every time I create anything in this subscription, all I have to do is remember to create the resource group with a "todelete" prefix and there's no need to worry it about being there in the morning.

Field level security in the Dataverse

Wai Liu — Mon, 03 Jan 2022 02:01:09 +0000

Introduction

Whilst the Dataverse supports permissions at the table or row level, there may be times you need more granular security permissions on the individual field level.

Dataverse supports this with Column Security Profiles. They are easy to set up and this article will provide a quick example to get you started.

1. In any table, create new field called PrivateValue and make sure Column Security is enabled.

2. Add this field to a form

3. Create a new Column Security Profile

4. Configure it so you have access to the field

5. Publish and test by navigating to the form

For someone with access, it should appear like this

For someone without access, it should appear like this

6. Resources and final comments

Note that this isn't available for some of the system fields (for example, you can't enable field security for a Status Reason or Status field in a table).

Note that in your column security profile, you can provide different levels of permissions on the field (read only, create etc).

For more information see this link.

Trigger an instant flow from a canvas app - simple example

Wai Liu — Thu, 23 Dec 2021 05:27:29 +0000

1. Create a canvas app with three components

Text input will what we send to the Instant Flow
Create Task button will be the trigger
LabelResult will be what is returned

2. Create Instant Flow

In a solution, create a new Instant Flow

Select Power Apps as the Trigger and call it "Instant Flow"

Edit the Instant Flow

There should be 3 steps

PowerApps - That's the first step - as it's an instant step, you don't have to do anything

Add a new row - this is specific to this example and all I'm really doing is going to create a task with the subject of all the input I'm putting into the flow

(Optional) to create inputs go to the Dynamic Content tab and click Ask in PowerApps

Respond to a Powerapp or Flow - this step returns an output back to the canvas app. For our experience we'll just put a success message.

4. Hook up the Canvas app to use the Instant Flow

Select your button, click on Action at the top, then Power Automate and select the Flow you want

Then Edit the formula so it is Set(Var1, InstantFlow.Run(User().Email, TextInput1.Text, "extra"))

Breaking down the formula

InstantFlow.Run will trigger the flow - it asks for 3 variables so I've put them in there.
Set is setting a variable with the variable being the first parameter and the second being the value of the variable. So here you're saying set Var1 to be the output of the InstantFlow.

Set the LabelResult

Lastly you want the LabelResult to display the returning message from the flow so click on it and put in Var1.result as the text message

Auto-incrementing your Power Platform solution version in your DevOps Pipeline

Wai Liu — Thu, 09 Dec 2021 12:11:10 +0000

Just a quick enhancement to this post where we exported a solution from Power Platforms in order to commit to a source repo.

If you manually exported the solution in Power Platform, it would normally offer to automatically increment the version by one. This post will demonstrate how to do it with DevOps Pipeline as well.

Setting the build number format

This sets the format of the Pipeline's Build.BuildNumber property. What 1.0.1.$(rev:rr) means is that we want the version to be 1.0.1.x with x being appended by one each time.

Add Set Solution Version Step

Add the Set Solution Version task before the export solution task. This step will actually go into your Dev environment's solution and update the version number to be the same as the BuildNumber.

Running the pipeline

When you run the pipeline, notice that the version number has changed from just a number to now in the 1.0.1.x format.

When you do a PR, you'll be able see that the version in the solution has also changed.

Forem: Wai Liu

Deleting all blobs in an Azure Storage / Data Lake container quickly

Deploying Angular apps to Azure Static Web apps

Introduction

Content Delivery Network

CI/CD Pipelines

Multi deployment environments

Demonstration

Retrieving deleted passwords in Chrome

Scenario

How to fix

Exporting your password

Turn off the sync

Turn the sync back on

Setting up a Power BI deployment pipeline with environment-specific data sources

Introduction

Prerequisites

Creating Premium Workspaces

Creating pipeline

Your Pipeline View

Creating a report in PowerBI Desktop

Adding a parameter

Using the parameter in the Data Model

Creating and publishing a report

Deployed to Dev

Pointing the parameter variable to Test

Extra tips

Verifying the parameters

Setting the connection

Moving your Azure subscription to another tenant

Scenario

Prerequisites

Adding your uni account as a guest on destination tenant

Change directory of subscription

Verify it all works

Further reading

Accessing environment variables in your custom workflow activity or plugin

Introduction

Code Snippet

Explanation

Trigger an instant flow from a model-driven app

The Flow button

Scenario

Create the Instant Flow in a solution

Adding inputs

Adding the Email action

Executing the instant flow

Conclusion

Copying your http request from browser into Postman

Copy it from Chrome

Import it into Postman

Set up automation to regularly clean up your Azure environment

Introduction

Word of warning

What we want to do

Prerequisite

Create an Automation account

Create a runbook

Examining the script

Test to see if this works.

Set up the schedule

Summary

Field level security in the Dataverse

Introduction

1. In any table, create new field called PrivateValue and make sure Column Security is enabled.

2. Add this field to a form

3. Create a new Column Security Profile

4. Configure it so you have access to the field

5. Publish and test by navigating to the form

6. Resources and final comments

Trigger an instant flow from a canvas app - simple example

1. Create a canvas app with three components

2. Create Instant Flow

Edit the Instant Flow

4. Hook up the Canvas app to use the Instant Flow

Breaking down the formula

Set the LabelResult

Auto-incrementing your Power Platform solution version in your DevOps Pipeline

Setting the build number format

Add Set Solution Version Step