Forem: Nam Phuong Tran

Getting Started with AWS Infrastructure as Code: A Terraform Guide

Nam Phuong Tran — Tue, 26 Sep 2023 14:27:10 +0000

If you're looking to start working with AWS and dive into the world of Terraform, this article is your comprehensive guide. We'll walk you through setting up your Terraform project to work seamlessly with AWS and share essential best practices for a smooth transition.

Section 1: Getting Started with AWS
1.1 Create Your AWS Account
We begin by creating your AWS account, which is your gateway to AWS services. Follow our step-by-step guide to set up your AWS account quickly and efficiently.

1.2 Configure Users
Learn how to configure user accounts in AWS, ensuring proper access control for your team members

1.3 Set Up the AWS CLI
Discover how to install and configure the AWS CLI (Command Line Interface) for managing AWS resources from your local environment.

We have to Configure AWS CLI credentials. In this article I will use Long-term credentials option. But it would be better if you use AWS IAM Identity Center (SSO). This is more secure. However you have to use your root user credentials or an IAM user with sufficient permissions to enable AWS SSO to setup this.

Let’s start with Long-term credentials options first.
Go to the AWS Portal, choice user account then Security Credentials

Scrolling to the Access Key

Click to the Create access key button

Choice the first option (Command line Interface – CLI)

Tick to the confirmation checkbox and Click to the Next button.

Because the Secret key only available for the first time so please note that we should copy the value of secret to some where or download csv file before closing the tab.

Section 2: AWS CLI and Terraform
2.1 Installing AWS CLI
Get hands-on with the AWS CLI installation process. Check your installation with aws --version.

2.2 Configure AWS CLI Credentials
Explore different options for configuring AWS CLI credentials, including long-term credentials and more secure AWS IAM Identity Center (SSO) configurations.

Filling the value corresponding to aws configure command
Now, we are going to create a resource group, that helps us to manage resources better via Tags management.

It will show you the result

Section 3: Setting Up Remote State Storage
3.1 Using AWS S3 for Remote State
Secure Remote Backends: Store the Terraform state in secure remote backends such as AWS S3.

In AWS, the Services should be enabled tagging the tag. That can help us to manage services better. It is also the tagged the service into the resource group

After creating the S3 bucket, we can see it on the AWS portal.

Go to the Resource group and check. Awesome, they look as expected.

Section 4: DynamoDB for State Locking
4.1 Understanding State Locking
As you know that, using DynamoDB for state locking enhances the reliability and scalability of Terraform workflows, especially in environments with multiple users or automation processes working concurrently. It helps ensure that infrastructure changes are applied safely and consistently while preventing conflicts and data corruption.
Terraform uses state files to keep track of resource information and dependencies. When multiple users or automation processes work with Terraform concurrently, there's a risk of conflicts and data corruption. DynamoDB provides built-in concurrency control mechanisms, ensuring that only one process can acquire a lock for a particular state file at a given time. This prevents conflicts and ensures consistent state management.
Terraform provides built-in support for using DynamoDB as a backend for state locking. Configuring Terraform to use DynamoDB is straightforward, and it seamlessly integrates with the Terraform CLI.
So, we are going to create Dynamo table

4.2 Creating a DynamoDB Table
Step-by-step instructions on creating a DynamoDB table to serve as the state locking mechanism

It should be also tagged then it will show in the resource group

Awesome, now let’s start write Terraform code

Section 5: Writing Terraform Modules
5.1 Terraform Configuration
Setting up your Terraform project with the required providers, including AWS and optional Kubernetes configurations.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.16.2"#"~> 4.16"
    }

    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.16.1"
    }
  }

  required_version = ">= 1.2.0"
}
provider "aws" {
  region = "us-east-1"
}
terraform {
  backend "s3" {
    bucket         = "s3-terraform-state-use1"
    key            = "terraform.tfstate"   # State file name
    region         = "us-east-1"           # Use your desired AWS region
    encrypt        = true                  # Optionally, enable server-side encryption
    dynamodb_table = "ddb-statelock-table" # Optional, use a DynamoDB table for state locking
  }
}

5.2 Creating a Resource Group
Learn how to define a Terraform resource group to organize AWS resources based on tags.

resource "aws_resourcegroups_group" "rg" {
  name = join("-", [var.resource_type, var.application, var.application_environment, var.region_short])

  resource_query {
    query = <<JSON
    {
      "ResourceTypeFilters": [
        "AWS::AllSupported"
      ],
      "TagFilters": [
        {
          "Key": "Environment",
          "Values": ["${var.workload_environments}"]
        },
        {
          "Key": "ApplicationEnvironment",
          "Values": ["${var.application_environment}"]          
        },
        {
          "Key": "OpsTeam",
          "Values": ["${var.ops_team}"]
        },
        {
          "Key": "Owner",
          "Values": ["${var.owner}"]
        },{
          "Key": "Criticality",
          "Values": ["${var.business_criticality}"]          
        },{
          "Key": "OpsCommitment",
          "Values": ["${var.ops_commitment}"]          
        },{
          "Key": "ApplicationName",
          "Values": ["${upper(var.application)}"]
        }
      ]
    }
  JSON
  }
}

5.3 Configuring S3 Buckets
Create S3 buckets for your infrastructure components, making them public and configuring website settings.

resource "aws_s3_bucket" "s3" {
  bucket = join("-", [var.resource_type, var.application, var.application_environment, var.region_short]) #"s3-static-website-bucket"                                                                               # Make the bucket public

  tags = {
    Environment            = var.workload_environments
    ApplicationEnvironment = var.application_environment
    OpsTeam                = var.ops_team
    Owner                  = var.owner
    Criticality            = var.business_criticality
    OpsCommitment          = var.ops_commitment
    ApplicationName        = upper(var.application)
  }
  depends_on = [ var.resource_group]
}

resource "aws_s3_bucket_website_configuration" "s3_website" {
  bucket = aws_s3_bucket.s3.id

  index_document {
    suffix = "index.html"
  }

  error_document {
    key = "error.html"
  }

  routing_rule {
    condition {
      key_prefix_equals = "docs/"
    }
    redirect {
      replace_key_prefix_with = "documents/"
    }
  }

  depends_on = [ var.resource_group]
}

Section 6: Module Integration in Main Configuration
6.1 Integrating the Resource Group Module
See how to call and configure the resource group module in your

main configuration.
module "rg" {
  for_each                = var.application_environments
  source                  = "./modules/resource-group"
  resource_type           = "rg"
  application             = var.organization
  workload_environments   = var.workload_environments
  application_environment = each.value #var.application_environment# 
  region                  = var.region
  region_short            = var.region_short
  ops_team                = var.ops_team
  owner                   = var.owner
  business_criticality    = var.business_criticality
  ops_commitment          = var.ops_commitment
}

6.2 Integrating the S3 Bucket Module
Integrate the S3 bucket module into your main configuration and link it to your resource group.

module "s3" {
  for_each                = var.application_environments
  source                  = "./modules/bucket"
  resource_type           = "s3"
  application             = var.organization
  workload_environments   = var.workload_environments
  application_environment = each.value #var.application_environment#
  region                  = var.region
  region_short            = var.region_short
  ops_team                = var.ops_team
  owner                   = var.owner
  business_criticality    = var.business_criticality
  ops_commitment          = var.ops_commitment
  resource_group          = module.rg[each.value]
  resource_group_id       = module.rg[each.value].resource_group_id
  resource_group_arn      = module.rg[each.value].resource_group_arn
}

Source code: https://github.com/namphuongtran/aws-infrastructure
By the end of this article, you'll have a firm grasp of setting up AWS and Terraform for your infrastructure needs. The included best practices will ensure you're working efficiently and effectively in your new AWS environment.

Provisioning Azure Databricks with Terraform

Nam Phuong Tran — Tue, 12 Sep 2023 14:32:32 +0000

Azure Databricks is a powerful analytics platform built on Apache Spark, tailor-made for Azure. It fosters collaboration between data engineers, data scientists, and machine learning experts, facilitating their work on large-scale data and advanced analytics projects. On the other hand, Terraform, an open-source Infrastructure as Code (IaC) tool developed by HashiCorp, empowers users to define and provision infrastructure resources using a declarative configuration language.
In this guide, we'll delve into the seamless integration of these two technologies using the Databricks Terraform provider. This combination offers several compelling advantages and is the recommended approach for efficiently managing Databricks workspaces and their associated resources in Azure.

Setting Up Your Terraform Environment
Before we dive into the specifics, there are some prerequisites for successfully using Terraform and the Databricks Terraform provider:

Azure Account: Ensure you have an Azure account.
Azure Admin User: You need to be an account-level admin user in your Azure account.
Development Machine Setup: On your local development machine, you should have the Terraform CLI and Azure CLI installed and configured. Make sure you are signed in via the az login command with a user that has Contributor or Owner rights to your subscription.

Project Structure
Organize your project into a folder for your Terraform scripts, let's call it "Terraform-Databricks." We will create several configuration files to handle authentication and resource provisioning.
Version and Provider Configuration
In your Terraform project, create a versions.tf file to specify the Terraform version and required providers:



terraform {
  required_version = ">= 1.2, < 1.5"
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
    }
    databricks = {
      source  = "databricks/databricks"
      version = "1.24.1"
    }
  }
}

Now, let's define the providers in a providers.tf file:



provider "azurerm" {
  features {}
}

# Use Azure CLI authentication.
provider "databricks" {
  # We'll revisit this section later
}

We'll return to the Databricks provider configuration shortly.
Backend Configuration
To store the Terraform state, create a backend.tf file. In this example, we're using an Azure Storage account to store the state:



terraform {
  backend "azurerm" {
    resource_group_name  = "rg-terraform-non-prod-weu"
    storage_account_name = "stteranonprodweu"
    container_name       = "terraform-databricks"
    key                  = "terraform.tfstate"

    # rather than defining this inline, the Access Key can also be sourced
    # from an Environment Variable - more information is available below.
    access_key = "your_key_storage"
  }
}

With these initial configurations in place, we can proceed to set up Terraform for Azure.

Getting started writing the Terraform script to build Azure Databricks infrastructure.
Step 1: Retrieve the Current Client Configuration and User in Azure



data "azurerm_client_config" "current" {
}

data "databricks_current_user" "me" {  
   depends_on = [azurerm_databricks_workspace.dbw]
}

Step 2: Create a resource group



resource "azurerm_resource_group" "rg" {
  name     = "rg-analytics-test-weu"
  location = "westeurope"
}

Step 3: Create a Virtual Network (Vnet) - Optional, but Important
Whether to create a Vnet depends on your specific use case. If you're exclusively using Azure Databricks and don't require outbound access or a high level of security, you can skip this step. However, if you need to interact with services outside of Azure, it's advisable to create a Vnet.
Consider the scenario where you want Azure Databricks to access MongoDB Atlas, which resides outside of Azure. MongoDB Atlas secures its infrastructure by allowing specific IPs in a whitelist. However, exposing Azure Databricks to the internet isn't an ideal solution. Instead, you can create a Vnet and set up peering or a private endpoint.
It's essential to note that you can't add a Vnet to an existing workspace. Once a workspace is created, its configurations are registered in the Control Plane and can't be modified.

Here, we'll create a Vnet:



resource "azurerm_virtual_network" "vnet" {
  name                = "vnet-analytics-test-weu"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  address_space       = [var.cidr]
}

Within this Vnet, we'll create two subnets: a public subnet and a private subnet. We'll also implement a network security group (NSG) to manage security for the Vnet.



resource "azurerm_network_security_group" "nsg" {
  name                = "nsg-analytics-test-weu"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
}

resource "azurerm_subnet" "public" {
  name                 = "subnet-public"
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = [cidrsubnet(var.cidr, 3, 0)]

  delegation {
    name = "databricks"
    service_delegation {
      name = "Microsoft.Databricks/workspaces"
      actions = [
        "Microsoft.Network/virtualNetworks/subnets/join/action",
        "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action",
      "Microsoft.Network/virtualNetworks/subnets/unprepareNetworkPolicies/action"]
    }
  }
}

resource "azurerm_subnet_network_security_group_association" "public" {
  subnet_id                 = azurerm_subnet.public.id
  network_security_group_id = azurerm_network_security_group.nsg.id
}

resource "azurerm_subnet" "private" {
  name                 = "subnet-private"
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = [cidrsubnet(var.cidr, 3, 1)]

  delegation {
    name = "databricks"
    service_delegation {
      name = "Microsoft.Databricks/workspaces"
      actions = [
        "Microsoft.Network/virtualNetworks/subnets/join/action",
        "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action",
      "Microsoft.Network/virtualNetworks/subnets/unprepareNetworkPolicies/action"]
    }
  }
}

resource "azurerm_subnet_network_security_group_association" "private" {
  subnet_id                 = azurerm_subnet.private.id
  network_security_group_id = azurerm_network_security_group.nsg.id
}

Azure Databricks Workspace Setup
Now, let's focus on creating an Azure Databricks workspace. This workspace will be our central hub for running analytics tasks:



resource "azurerm_databricks_workspace" "dbw" {
  name                          = "dbw-analytics-test-weu"
  resource_group_name           = azurerm_resource_group.rg.name
  location                      = azurerm_resource_group.rg.location
  sku                           = "premium"
  managed_resource_group_name   = "rg-databricks-managed-weu"
  public_network_access_enabled = var.public_network_access_enabled
  custom_parameters {
    no_public_ip                                         = var.no_public_ip
    virtual_network_id                                   = azurerm_virtual_network.vnet.id
    private_subnet_name                                  = azurerm_subnet.private.name
    public_subnet_name                                   = azurerm_subnet.public.name
    public_subnet_network_security_group_association_id  = azurerm_subnet_network_security_group_association.public.id
    private_subnet_network_security_group_association_id = azurerm_subnet_network_security_group_association.private.id
  }
  depends_on = [
    azurerm_subnet_network_security_group_association.public,
    azurerm_subnet_network_security_group_association.private
  ]
}

You can see that in the depends_on, I listed 2 subnet. We need this, otherwise destroy doesn't cleanup things correctly.
This workspace creation includes some crucial parameters like SKU, location, and network configurations. We're ensuring that the workspace is integrated into your Azure environment.
Next, we are going to create Databricks cluster, however, before creating cluster. We have to define note type, spark version and instance pool.



data "databricks_node_type" "dbr_node_type" {
  local_disk = true
  depends_on = [azurerm_databricks_workspace.dbw]
}

data "databricks_spark_version" "dbr_spark" {
  long_term_support = true
  depends_on        = [azurerm_databricks_workspace.dbw]
}

resource "databricks_instance_pool" "dbr_instance_pool" {
  instance_pool_name = "pool-analytics-test-weu"
  min_idle_instances = 0
  max_capacity       = 10
  node_type_id       = data.databricks_node_type.dbr_node_type.id

  idle_instance_autotermination_minutes = 10

  azure_attributes {
    availability       = "ON_DEMAND_AZURE"
    spot_bid_max_price = -1
  }

  disk_spec {
    disk_type {
      azure_disk_volume_type = "PREMIUM_LRS"
    }
    disk_size  = 80
    disk_count = 1
  }
}

Azure Databricks Cluster Creation
Next, we'll create an Azure Databricks cluster, which will serve as the computational engine for our analytics tasks:



resource "databricks_cluster" "cluster" {

  cluster_name  = "dbc-analytics-test-weu"
  spark_version = data.databricks_spark_version.dbr_spark.id
  node_type_id = data.databricks_node_type.dbr_node_type.id
  autotermination_minutes = 20
  autoscale {
    min_workers = 1
    max_workers = 50
  }
  spark_conf = {
    "spark.databricks.io.cache.enable" : true
  }
  depends_on       = [azurerm_databricks_workspace.dbw]
  # instance_pool_id = databricks_instance_pool.dbr_instance_pool.id
}

We have also declared that the cluster will be dependent on Databricks workspace. We can define the note type id or instance pool id.
To reduce cluster start time, you can attach a cluster to a predefined pool of idle instances. When attached to a pool, a cluster allocates its driver and worker nodes from the pool. If the pool does not have sufficient idle resources to accommodate the cluster’s request, it expands by allocating new instances from the instance provider. When an attached cluster changes its state to TERMINATED, the instances it used are returned to the pool and reused by a different cluster.
In this article, I use note type id instead of instance pool id.
In addition to creating more stuff such as: jobs or notebook we can define as below



resource "databricks_notebook" "nb" {
  path          = "${data.databricks_current_user.me.home}/${var.notebook_subdirectory}/${var.notebook_filename}"
  language      = var.notebook_language
  source        = "./notebooks/${var.notebook_filename}"
}

resource "databricks_job" "job" {
  name = var.job_name
  existing_cluster_id = databricks_cluster.cluster.cluster_id
  notebook_task {
    notebook_path = databricks_notebook.nb.path
  }
  email_notifications {
    on_success = [ data.databricks_current_user.me.user_name ]
    on_failure = [ data.databricks_current_user.me.user_name ]
  }
}

This is almost completely.
This configuration includes cluster details such as its name, Spark version, and autoscaling properties.

Databricks Provider Update
We need to revisit the Databricks provider configuration and update it with the necessary authentication information:



provider "databricks" {
  azure_workspace_resource_id = azurerm_databricks_workspace.dbw.id
  host                        = azurerm_databricks_workspace.dbw.workspace_url
}

This help authorization Azure Databricks service.
Initialize the working directory containing the *.tf file by running the terraform init command



terraform init

Terraform downloads the specified providers and installs them in a hidden subdirectory of your current working directory, named .terraform. The terraform init command prints out which version of the providers were installed. Terraform also creates a lock file named .terraform.lock.hcl which specifies the exact provider versions used, so that you can control when you want to update the providers used for your project.
Check whether your project was configured correctly by running the terraform plan command. If there are any errors, fix them, and run the command again.



terraform plan

If everything looks good, apply the changes to your Azure environment. Apply the changes required to reach the desired state of the configuration by running the terraform apply command.



terraform apply -auto-approve

This guide covers the fundamental setup and provisioning steps for Azure Databricks using Terraform. In upcoming articles, we'll explore more advanced configurations and automation options to help you harness the full potential of this powerful analytics platform.
Stay tuned for more in-depth insights into managing Databricks workspaces and resources with Terraform!

How to Create a Beautiful PowerShell Prompt with Oh My Posh and Windows Terminal

Nam Phuong Tran — Mon, 04 Sep 2023 13:25:55 +0000

As a developer working with tools like PowerShell, Kubernetes, Terraform, and others, having a well-customized and visually appealing terminal can significantly enhance your workflow. In this guide, we'll walk through the steps to create a stunning PowerShell prompt using Oh My Posh and Windows Terminal.

Prerequisites
Before we start, make sure you have the following prerequisites in place:

PowerShell Core: If you haven't already installed PowerShell Core, you can find installation instructions here.

Windows Terminal: You can install Windows Terminal by following the guide here.

After installing you can see the default UI as below:

Oh My Posh: Install Oh My Posh by running the following command in PowerShell or Terminal:

winget install JanDeDobbeleer.OhMyPosh -s winget

Create a PowerShell Folder: Create a folder for PowerShell scripts on your C drive, for example, C:\Users<your-username>\Documents\PowerShell. Inside this folder, create a file named profile.ps1.
Terminal Icons: Install the Terminal Icons module by running the following command in Windows Terminal:

Install-Module -Name Terminal-Icons -Repository PSGallery

Fonts: Download and install the Caskaydia Cove Nerd Font Complete from here. Install this font in C:\Windows\Fonts.
Configuration Steps
Open the profile.ps1 file you created in the PowerShell folder (C:\Users<your-username>\Documents\PowerShell\profile.ps1).

Add the following lines to set up aliases for your tools:

$pwd = pwd
write-host "Using profile in '$pwd\Documents\PowerShell\profile.ps1':"
Import-Module oh-my-posh
Import-Module -Name Terminal-Icons

Set-PoshPrompt -Theme 'C:\Users\<your-username>\Documents\PowerShell\ohmyposhv3-v2.json'

After that the Terminal UI will look like this

Download the ohmyposhv3-v2.json theme file from Scott Hanselman's GitHub. Save it in your PowerShell folder (C:\Users<your-username>\Documents\PowerShell).
Open Windows Terminal and go to Settings.

Set up PowerShell as the default shell for Windows Terminal.
In the Windows Terminal settings, navigate to Profiles > > Appearance and change the font to CaskaydiaCove NFM.

Save your settings.
Now, you have a beautifully customized PowerShell prompt with Oh My Posh and Windows Terminal. Enjoy your new, stylish, and efficient development environment!

Creating a Private Endpoint for Azure Storage Account using Terraform

Nam Phuong Tran — Fri, 01 Sep 2023 13:33:17 +0000

Enhancing the security of our infrastructure is paramount. Today, I'll guide you through the process of setting up a private endpoint for Azure Storage Account using Terraform, step by step, leveraging distinct services.
One of the best architectures is highly recommended from MS official as below

Deeping dive to the detail we can see how it goes behind the scenes

We can see that, all almost connections to the Azure services such as: Storage account, Container registry, Keyvault, etc. It should be used private endpoint for more secure.
So, Today we will focus on the Azure Storage Account first. After we creating private endpoint for Storage account it looks like

Let’s start
Step 1: Create a Resource Group



resource "azurerm_resource_group" "rg" {
  name = "rg-sd2488-non-prod-weu-infra"
  location = "westeurope"
  tags = {
    Owner="sd2488"
  }
}

Step 2: Create a Storage Account



resource "azurerm_storage_account" "st" {
  name                     = "stsd2488nonprodweu"
  resource_group_name      = azurerm_resource_group.rg.name
  location                 = azurerm_resource_group.rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
  min_tls_version = "TLS1_2"
  network_rules {
    default_action             = "Deny"
    ip_rules                   = []
  }

  tags = {
    Owner="sd2488"
  }
}

Step 3: Create a Virtual Network with Subnets



resource "azurerm_virtual_network" "vnet" {
  name                = "vnet-sd2488-non-prod-weu"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  address_space       = ["10.1.0.0/16"]  

  tags = {
    Owner="sd2488"
  }
}


resource "azurerm_subnet" "snet_endpoint" {
  name = "PrivateSubnet"
  virtual_network_name = azurerm_virtual_network.vnet.name
  resource_group_name = azurerm_resource_group.rg.name
  address_prefixes      = ["10.1.0.0/24"]  

}

resource "azurerm_subnet" "snet_bas" {
  name = "AzureBastionSubnet"
  virtual_network_name = azurerm_virtual_network.vnet.name
  resource_group_name = azurerm_resource_group.rg.name
  address_prefixes      = ["10.1.1.0/24"]
}

The Virtual Network comprises two subnets:
The PrivateSubnet: This is intended for creating the private endpoint. It also includes a Virtual Machine for testing purposes.
The AzureBastionSubnet: This subnet is designated for the Azure Bastion service, which enables secure connections to Virtual Machines.
Step 4: Set Up Public IP Address and Azure Bastion



resource "azurerm_public_ip" "pip" {
  name                = "pip-sd2488-non-prod-weu"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  allocation_method   = "Static"
  sku                 = "Standard"
}

resource "azurerm_bastion_host" "bas" {
  name                = "bas-sd2488-non-prod-weu"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name

  ip_configuration {
    name                 = "bas-configuration"
    subnet_id            = azurerm_subnet.snet_bas.id
    public_ip_address_id = azurerm_public_ip.pip.id
  }
}

Create a Public IP address and configure Azure Bastion.
Step 5: Establish a Private Endpoint
Before creating the private endpoint, generate a Private DNS Zone. Link the DNS Zone with the Virtual Network (VNet) and define the A record within the DNS Zone.



resource "azurerm_private_dns_zone" "pdns_st" {
  name                = "privatelink.blob.core.windows.net"
  resource_group_name = azurerm_resource_group.rg.name
}
Then,


resource "azurerm_private_endpoint" "pep_st" {
  name                = "pep-sd2488-st-non-prod-weu"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  subnet_id           = azurerm_subnet.snet_endpoint.id

  private_service_connection {
    name                           = "sc-sta"
    private_connection_resource_id = azurerm_storage_account.st.id
    subresource_names              = ["blob"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "dns-group-sta"
    private_dns_zone_ids = [azurerm_private_dns_zone.pdns_st.id]
  }
}

This step, we need to link the DNS Zone with Vnet and define A record in DNS Zone



resource "azurerm_private_dns_zone_virtual_network_link" "dns_vnet_lnk_sta" {
  name                  = "lnk-dns-vnet-sta"
  resource_group_name   = azurerm_resource_group.rg.name
  private_dns_zone_name = azurerm_private_dns_zone.pdns_st.name
  virtual_network_id    = azurerm_virtual_network.vnet.id
}

resource "azurerm_private_dns_a_record" "dns_a_sta" {
  name                = "sta_a_record"
  zone_name           = azurerm_private_dns_zone.pdns_st.name 
  resource_group_name = azurerm_resource_group.rg.name
  ttl                 = 300
  records             = [azurerm_private_endpoint.pep_st.private_service_connection.0.private_ip_address]
}

It seems almost thing done. Now, we are going to create an Virtual Machine and verify
Step 6: Create a Virtual Machine
Before creating Virtual machine, we need to create Network Interface and Network Security Group and link 2 services with each together



resource "azurerm_network_security_group" "nsg" {
  name                = "nsg-sd2488-non-prod-weu"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name

  tags = {
    Owner="sd2488"
  }
}

resource "azurerm_network_interface" "nic" {
  name                = "nic-sd2488-non-prod-weu"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name

  ip_configuration {
    name                          = "nic-configuration"
    subnet_id                     = azurerm_subnet.snet_endpoint.id
    private_ip_address_allocation = "Dynamic"
  }
}

resource "azurerm_network_interface_security_group_association" "nsgnic" {
  network_interface_id      = azurerm_network_interface.nic.id
  network_security_group_id = azurerm_network_security_group.nsg.id
}

resource "azurerm_windows_virtual_machine" "vm" {
  name                = "vm-sd2488-non"
  location              = azurerm_resource_group.rg.location
  resource_group_name   = azurerm_resource_group.rg.name
  size                = "Standard_F2"
  admin_username      = "adminuser"
  admin_password      = "P@$$w0rd1234!"  
  network_interface_ids = [
    azurerm_network_interface.nic.id,
  ]

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }

  source_image_reference {
    publisher = "MicrosoftWindowsServer"
    offer     = "WindowsServer"
    sku       = "2019-Datacenter"
    version   = "latest"
  }
}

Creating a Virtual Machine involves setting up a Network Interface and a Network Security Group. These components need to be connected.
Execute the Terraform script with the "terraform apply" command.
After running the command successfully, the resources will be created on Azure portal. It looks like

Step 7: Now, we will verify the result.
Step 7.1: Accessing through Storage Explorer

As we can see the result. If trying to access the Storage Account through Storage Explorer. It should be evident that direct access is prohibited.
Step 7.2: Now, we will use virtual machine that is the same virtual network as Remote Access using Azure Bastion

Using Azure Bastion to remotely access the Azure Virtual Machine is the next step. This entails connecting to the VM through the Azure Portal and selecting the Bastion option.

Install Azure Storage Explorer to access the VM, retrieve the Access Key from the Storage Account, and establish a connection through the Explorer. Create a blob container as an additional validation.

Step 7.3: Using nslookup for FQDN Verification

To confirm the setup, we can use the nslookup command for the FQDN (fully qualified domain name) generated after creating the DNS A record. This step helps ensure that the DNS record is accurately resolving to the expected IP address.
By following these meticulous steps, we create a private endpoint for an Azure Storage Account, significantly bolstering the security of our infrastructure.