Forem: CloudDude

Terraform for dummies Part1 : Launch an instance with a static website on OCI

CloudDude — Sat, 10 Jun 2023 00:44:18 +0000

Terraform for dummies Part1 : Launch an instance with a static website on OCI

Intro

Terraform brings a new paradigm where Infrastructure becomes a Code, and with Cloud becoming what it is today, everyone is invited at the (devops) table. Therefore, after provisioning with oci-cli in my previousBlogPost, I will explore the same task using terraform.To add more fun, we won’t just deploy an instance but also configure a website linked to its public IP.

Note This lab will also help you practice if you are preparing forOCI Operations Associate exam(1Z0–1067) .

Topology

The following illustration shows the layers involved between your workstation an Oracle cloud infrastructure while running the terraform commands along with the instance attributes we will be provisioning .

Besides describing my GitHub repo before starting this tutorial, I’ll just briefly discuss some principles.

Infrastructure As Code Manages and provisions cloud resources using a declarative code (i.e Terraform) and definition files avoiding interactive configuration. Terraform is an immutable Orchestrator that creates and deletes all resources in the proper sequence.Each Cloud vendor has what we call a provider that terraform uses in order to convert declarative texts into API calls reaching the Cloud infrastructure layer.
Terraform Files
Can be a single file or split into multiple tf or tf.json files, any other file extension is ignored
Files are merged in alphabetical order but resource definition order doesn’t matter (subfolders are not read)
Common configurations have 3 type of tf files and a statefile
Terraform resource declaration syntax looks like this:

Component "Provider_Resource_type" "MyResource_Name" { Attribute1 = value ..
                                                       Attribute2 = value ..}

Where the hell do I find a good deployment sample?

The most important thing when learning a new program is accomplishing your first HelloWorld. Unfortunately, google can’t always make the cut as samples I used had errors. Luckily, OCI Resource Manager had some samples I managed to export and tweak which was a good starting point for this lab.

Terraform lab content: I have deliberately split this lab in 2 :

VCN Deployment: To grasp the basics of a single resource deployment
FULL Instance Deployment: which is a more complex deployment (instance provisioning with a hosted web sever).

I.Terraform setup

Although I’m on windows I tried the lab using WSL (Ubuntu) terminal (but same applies to Mac).

Windows: Download and run the installer from their website (32-bit ,64-bit)
Linux: Download, unzip and move the binary to the local bin directory

$ wget https://releases.hashicorp.com/terraform/1.0.3/terraform_1.0.3_linux_amd64.zip 
$ unzip terraform_1.0.3_linux_amd64.zip 
$ mv terraform /usr/local/bin/

Once installed run the version command to validate your installation

$ terraform --version
  Terraform v1.0.3

OCI API Key based authentication

Tenancy_ocid, Compartment_ocid, user_ocid and the region
The private API key path and its fingerprint to authenticate with your tenancy account
The SSH key pair (Private/Public) required when launching the new compute instance

Assumptions

Terraform shares most of the authentication parameters with oci-cli (located in ~/.oci/config ). Please refer to my Other post for details on how to setup oci-cli if it isn’t done yet.
However, terraform also allows using environment variables to define these parameters. This is why I will be using a shell script that sets them before the deployment (I still needed oci-cli for API keys).

II. Clone the repository

Pick an area that is close to your oci-terraform directory on your file system and issue the following command

$ git clone https://github.com/brokedba/terraform-examples.git

Note: As explained earlier you will find 2 directories inside the repository which will make things easier:

terraform-provider-oci/create-vcn/ To grasp how we deploy a VCN.
terraform-provider-oci/launch-instance/ For the full instance deploy, once comfortable with terraform.

III. Provider setup

INSTALL AND SETUP THE OCI PROVIDER

Cd Into the subdirectory terraform-provider-oci/create-vcn where our configuration resides (i.e vcn )

$ cd ~/terraform-examples/terraform-provider-oci/create-vcn

OCI provider plugin is distributed by HashiCorp hence it will be automatically installed by terraform init.

$ terraform init
  Initializing the backend...

  Initializing provider plugins...
  - Checking for available provider plugins...
  - Downloading plugin for provider "oci" (hashicorp/oci) 3.83.1...
  * provider.oci: version = "~> 3.83"

$ terraform --version
  Terraform v0.12.24
  + provider.oci v3.83.1 ---> the privider is now installed

Let’s see what’s in the directory. Here, only files matter along with (click to see content)

$ tree
  .
  |-- env-vars ---> TF_environment_variables needed to authenticate to OCI 
  |-- outputs.tf ---> displays the resources detail at the end of the deploy
  |-- schema.yaml ---> Contains the stack (variables) description    
  |-- variables.tf ---> Resource variables needed for the deploy   
  `-- vcn.tf ---> Our vcn terraform declaration code (configuration)

Adjust the required authentication parameters in file according to your tenancy and key pairs .

$ vi env-vars 

  export TF_VAR_tenancy_ocid="ocid1.tenancy.oc1..aaaaaaaa" # change me 
  export TF_VAR_user_ocid="ocid1.user.oc1..aaaaaaaa" # change me 
  export TF_VAR_compartment_ocid="ocid1.tenancy.oc1..aaaaaaaa" # change me 
  export TF_VAR_fingerprint=$(cat PATH_To_Fing/oci_api_key_fingerprint)# change me 
  export TF_VAR_private_key_path=PATH_To_APIKEY/oci_api_key.pem # change me 
  export TF_VAR_ssh_public_key=$(cat PATH_To_PublicSSH/id_rsa.pub) # change me 
  export TF_VAR_ssh_private_key=$(cat PATH_To_PrivateSSH/id_rsa) # change me 
  export TF_VAR_region="ca-toronto-1" # change me 
  $ . env-vars

IV. Partial Deployment

DEPLOY A SIMPLE VCN

Now that env-vars values are set and sourced, we can run terraform plan command to create an execution plan (quick dry run to check the desired state/actions )

$ terraform plan
   Refreshing Terraform state in-memory prior to plan... 
  ------------------------------------------------------------------------
  An execution plan has been generated and is shown below.
    Terraform will perform the following actions:

    # oci_core_default_route_table.rt will be created
    + resource "oci_core_default_route_table" "rt" 
    {..}
    # oci_core_internet_gateway.gtw will be created
    + resource "oci_core_internet_gateway" "gtw" 
    {..}

    # oci_core_security_list.terra_sl will be created
    + resource "oci_core_security_list" "terra_sl" {
        + egress_security_rules {..}
        + ingress_security_rules {..
            + tcp_options {+ max = 22 + min = 22}}
        + ingress_security_rules {..
             + tcp_options { + max = 80 + min = 80}}
     }

    # oci_core_subnet.terrasub[0] will be created
    + resource "oci_core_subnet" "terrasub" {
        + availability_domain = "BahF:CA-TORONTO-1-AD-1"
        + cidr_block = "192.168.78.0/24"
        ...}

    # oci_core_vcn.vcnterra will be created
    + resource "oci_core_vcn" "vcnterra" {
        + cidr_block = "192.168.64.0/20"
        ...}

  Plan: 5 to add, 0 to change, 0 to destroy.

The output being too verbose I deliberately kept only relevant attributes

for each VCN component
Next, we can finally run terraform deploy to apply the changes required to create our VCN ( listed in the plan )

 $ terraform apply -auto-approve
oci_core_vcn.vcnterra: Creating...
...
Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

default_dhcp_options_id = ocid1.dhcpoptions.oc1.ca-toronto-1.aaaaaaaaasxxxx
default_route_table_id = ocid1.routetable.oc1.ca-toronto-1.aaaaaaaaaxxx
default_security_list_id = ocid1.securitylist.oc1.ca-toronto-1.aaaaaaaaxx
internet_gateway_id = ocid1.internetgateway.oc1.ca-toronto-1.aaaaaaaaxxxx
subnet_ids = ["ocid1.subnet.oc1.ca-toronto-1.aaaaaaaaxxx,]
vcn_id = ocid1.vcn.oc1.ca-toronto-1.amaaaaaaaxxx

Observations :

The deploy started by loading the resources variables in variables.tf which allowed the execution of vcn.tf
Finally terraform fetched the variables (ocids) of the resources listed in outputs.tf (lookup)

Note : In order to continue the lab we will need to destroy the vcn as the full instance launch will recreate it.

$ terraform destroy -auto-approve

Destroy complete! Resources: 5 destroyed.

V. Full deployment (Instance)

OVERVIEW

Awesome, After our small test let’s launch a full instance from scratch .
First we need to switch to the second directory terraform-provider-oci/launch-instance/ Here's its content:

$ tree ./terraform-provider-oci/launch-instance
.
|-- cloud-init ---> SubFolder
| `--> vm.cloud-config ---> script to install a web server & add a Webpage at startup
|-- compute.tf ---> Instance related terraform configuration
|-- env-vars ---> authentication envirment variables
|-- outputs.tf ---> displays the resources detail at the end of the deploy
|-- schema.yaml ---> Containes the stack (variables)
|-- variables.tf ---> Resource variables needed for the deploy   
|-- vcn.tf ---> same vcn terraform declaration

Note: As you can see we have 2 additional files and one Subfolder.

compute.tf is where the compute instance and all its attributes are declared. All the other tf files come from my vcn example with some additions for variables.tf and output.tf

Cloud-init : is a cloud instance initialization method that executes tasks upon instance startup by providing the user_data entry in the metadata block of the Terraform oci_core_instance resource definition (See below).

$ vi compute.tf
resource "oci_core_instance" "terra_inst" {
...
metadata = {
  ssh_authorized_keys = file("../../.ssh/id_rsa.pub") ---> Upload sshkey
  user_data = base64encode(file("./cloud-init/vm.cloud-config")) ---> Run tasks 
      }      
...

In my lab, I used cloud-init to install nginx and write an html page that will be the server’s HomePage at startup.

2. LAUNCH THE INSTANCE

Once in the launch-instance directory make sure you copied the adjusted file and sourced it (see ). You can then run the plan command (output is truncated for more visibility)

$ terraform plan
   Refreshing Terraform state in-memory prior to plan... 
  ------------------------------------------------------------------------
  An execution plan has been generated and is shown below.
    Terraform will perform the following actions:

  ... # VCN declaration 
  # oci_core_instance.terra_inst will be created
  + resource "oci_core_instance" " terra_inst" {
      + ...
      + defined_tags = (known after apply)
      + display_name = "TerraCompute"
      + metadata = {
         + "ssh_authorized_keys" =...
         + "user_data" = " ...
      + shape = "VM.Standard.E2.1.Micro"
      + ...
      + create_vnic_details {
      + hostname_label = "terrahost"
      + private_ip = "192.168.78.51"
      ..}
      + source_details {
          + boot_volume_size_in_gbs = "50"
          + source_type = "image"
           ..}
   # oci_core_volume.terra_vol will be created
   + resource "oci_core_volume" "terra_vol" {..}
   # oci_core_volume_attachment.terra_attach will be created
   + resource "oci_core_volume_attachment" "terra_attach" {..}
   ...
  Plan: 8 to add, 0 to change, 0 to destroy.

Now let the cloud party begin and provision our instance (output has been truncated for more visibility)

$ terraform apply -auto-approve
...
oci_core_instance.terra_inst: Creation complete after 1m46s
oci_core_volume.terra_vol: Creation complete after 14s  
oci_core_volume_attachment.terra_attach: Creation complete after 33s  
...
Apply complete! Resources: 8 added, 0 changed, 0 destroyed.

Outputs:
...
private_ip = ["192.168.78.51",]
public_ip = ["132.145.108.51",]

3. CONNECTION TO YOUR INSTANCE WEB PAGE

— — — — — — — — — — — — Your Web Page is 🔥 :)

Once the instance is provisioned, just copy the public IP address(132.145.108.51) in your browser and Voila!
You have made yourself a fresh and shiny website ready to roll.
Here I just embedded a video link into the index page but you can adapt the cloud-config to your own liking
You can also tear down this configuration by simply running terraform destroy from the same directory

CONCLUSION

We have demonstrated in this tutorial how to quickly deploy an instance using terraform in OCI and leverage Cloud-init to bootstrap your instance into a webserver .
Remember that all used attributes in this exercise can be modified in the variables.tf file.
In my next blog post we will explore how to provision instances using oci ansible modules.

Terraform for dummies part 3: Launch a vm with a static website on Azure

CloudDude — Fri, 09 Jun 2023 23:57:38 +0000

Intro

In this tutorial we will try provisioning a vm on azure using their terraform provider and reproduce the same deployment I have completed on AWS and Oracle Cloud. As usual we won’t just deploy an instance but also configure an nginx website linked to its public IP. I’ll end this post with some notes on Azure/AWS differences.

I used azurerm_linux_virtual_machine resource instead of the classic vm resource because azure decided it was better to split both windows/Linux for vm deployments(provider 2.0) which is unlike anything seen elsewhere.

Note: I have done the same task with my AZ-CLI based bash scripts (avaialble in Github) which was very helpful to understand the logic behind components needed during the vm creation (More details: deploy-webserver-vm-using-azure-cli).

Content :

I. Terraform setup

IV. Partial deployment

V. Full deployment

Tips & Conclusion

Overview and Concepts

Topology

The following illustration shows the layers involved between your workstation an Oracle cloud infrastructure while running the terraform actions along with the instance attributes we will be provisioning.

Besides describing my GitHub repo before starting this tutorial, I’ll just briefly discuss some principles.

Infrastructure As Code Manages and provisions cloud resources using a declarative code (i.e Terraform) and definition files avoiding interactive configuration. Terraform is an immutable Orchestrator that creates and deletes all resources in the proper sequence. Each Cloud vendor has what we call a provider that terraform uses in order to convert declarative texts into API calls reaching the Cloud infrastructure layer.

Terraform Files

Can be a single file or split into multiple tf or tf.json files, any other file extension is ignored.
Files are merged in alphabetical order but resource definition order doesn’t matter (subfolders are not read).
Common configurations have 3 type of tf files and a statefile.

Terraform resource declaration syntax looks like this:

Component "Provider_Resource_type" "MyResource_Name" { Attribute1 = value .. 
                                                       Attribute2 = value ..}

Where do I find a good Azure deployment sample?

The easiest way is to create/locate an instance from the console and then use the import function from terraform to generate each of the related components in HCL format (vnet, instance,subnet,etc..) based on their id.

Example for a Vnet >>

1- Create a shell resource declaration for the vnet in a file called vnet.tf

2- Get the id of the vnet resource from your Azure portal

3- Run the Terraform import then run Terraform show to extract the vnet full declaration from azure to the same file ( vnet.tf)

4- Now you can remove the id attribute with all non required attributes to create a vnet resource (Do that for each resource)

1- # vi vnet.tf 
 provider "azurerm" { 
 features {}
 } 
 resource "azurerm_virtual_network" "terra_vnet" {
 }
2- # terraform import azurerm_virtual_network.terra_vnet /subscriptions/00*/resourceGroups/providers/Microsoft.Network/virtualNetworks/terra_vnet
3- # terraform show -no-color > vnet.tf

If you want to import all the existing resources in your account in bulk mode (not one by one) there are tools likepy-az2tf or terraformer , which can import both code and state from your azure account automatically.

Terraform lab content: I have deliberately split this lab in 2:

Vnet Deployment: To grasp the basics of a single resource deployment.
Instance Deployment: Includes the instance provisioning configured as web sever(includes above vnet) .

I.Terraform setup

Although I’m on windows I tried the lab using WSL (Ubuntu) terminal (but same applies to Mac).

Windows: Download and run the installer from their website (32-bit ,64-bit)
Linux: Download, unzip and move the binary to the local bin directory

$ wget https://releases.hashicorp.com/terraform/1.0.3/terraform_1.0.3_linux_amd64.zip 
$ unzip terraform_1.0.3_linux_amd64.zip 
$ mv terraform /usr/local/bin/

Once installed run the version command to validate your installation

$ terraform --version
  Terraform v1.0.3

Azure authentication

To authenticate with your azure account, Terraform will only need you to login using az cli . This can be done by running az cli commands described here >> az-cli installation

Assumptions

I will assume that below authentication option is present in your workstation:
AZCLI default profile configured with your azure credentials. Refer to my Blog post for more details

$ az account show 
EnvironmentName HomeTenantId IsDefault Name State 
----------------- ---------------- ----------- ------------- -------
AzureCloud 00000000-00000000... True BrokeDba Lab Enabled

I’ll also assume the presence of an ssh key pair to attach to your vm instance. If not here is a command to generate a PEM based key pair.

$ ssh-keygen -P "" -t rsa -b 2048 -m pem -f ~/id_rsa_az
 Generating public/private rsa key pair.
 Your identification has been saved in /home/brokedba/id_rsa_az.
 Your public key has been saved in /home/brokedba/id_rsa_az.pub.

II. Clone the repository

Pick an area that is close to your azure-terraform directory on your file system and issue the following command

$ git clone https://github.com/brokedba/terraform-examples.git

Note: As explained earlier you will find 2 directories inside the repository which will make things easier:

terraform-provider-azure/create-vnet/ To grasp how we deploy a single Vnet.
terraform-provider-azure/launch-instance/ For the final instance deploy.

III. Provider setup

INSTALL AND SETUP THE AZURE PROVIDER

Cd Into “ terraform-provider-azure/create-vnet” where our configuration resides (i.e vnet)

$ cd /mnt/c/Users/brokedba/azure/terraform-examples/terraform-provider-azure/create-vnet

Azure provider plugin will be automatically installed by running "terraform init".

$ terraform init
  Initializing the backend...

  Initializing provider plugins...
  - Finding latest version of hashicorp/azurerm...
  - Installing hashicorp/azurerm v2.80.0...
  * Installed hashicorp/azurerm v2.80.0 (signed by HashiCorp)
    provider "azurerm" {
Terraform has been successfully initialized!

$ terraform --version
  Terraform v1.0.3 on linux_amd64
  + provider registry.terraform.io/hashicorp/azurerm v2.80.0

Let’s see what’s in the directory. Here, only files matter (click to see content)

$ tree
  .
  |-- outputs.tf ---> displays resources detail after the deploy
  |-- variables.tf ---> Resource variables needed for the deploy   
  |-- vnet.tf ---> Our vnet terraform declaration

IV. Partial Deployment

DEPLOY A SIMPLE VNET

Once the authentication is setup and provider installed , we can run terraform plan command to create an execution plan (quick dry run to check the desired state/actions). Specify the prefix for your resource names

$ terraform plan
var.prefix The prefix used for all resources in this example
Enter a value: Demo
Terraform used selected providers to generate the following execution plan. 
Resource actions are indicated with the following symbols: + create
  ------------------------------------------------------------------------
  Terraform will perform the following actions:

    # azurerm_network_security_group.terra_nsg will be created
    + resource "azurerm_network_security_group" "terra_nsg" 
    {..
     + security_rule = [
     + destination_port_ranges = [+ "22", + "3389",+ "443",+ "80",]
     + direction = "Inbound"
     + name = "Inbound HTTP access"

    # azurerm_resource_group.rg will be created
    + resource "azurerm_resource_group" "rg" 
    {..} 
    # azurerm_subnet.terra_sub will be created
    + resource "azurerm_subnet" "terra_sub" 
    {..
     + address_prefixes = ["192.168.0.0/16”]
     …}
    # azurerm_subnet_network_security_group_association.nsg_sub will be created
    + + resource "azurerm_subnet_network_security_group_association" "nsg_sub"{

    # azurerm_virtual_network.terra_vnet will be created
    + resource "azurerm_virtual_network" "terra_vnet" {
    ...
    + address_space = [ "192.168.0.0/16”
   ...}

  Plan: 5 to add, 0 to change, 0 to destroy.

The output being too verbose I deliberately kept only relevant attributes for the Vnet resource plan
Next, we can run ”terraform deploy” to provision the resources to create our Vnet (listed in the plan)

$ terraform apply -auto-approve
azurerm_resource_group.rg: Creating...
azurerm_virtual_network.terra_vnet: Creating...
azurerm_network_security_group.terra_nsg: Creating...
azurerm_subnet.terra_sub: Creating...
...
Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

Subnet_CIDR = 192.168.10.0/24
Subnet_Name = internal
vnet_CIDR = 192.168.0.0/16
vnet_Name = Terravnet
vnet_dedicated_security_group_Name = "Demo-nsg"
vnet_CIDR = 192.168.0.0/16
vnet_dedicated_security_ingress_rules = toset([
{…
"access" = "Allow" 
"description" = "RDP-HTTP-HTTPS ingress trafic"
"destination_port_ranges" = toset([
"22",
"3389",
"443",
"80",])
…

Observations:

The deploy started by loading the resources variables in variables.tf which allowed the execution of vnet.tf
Finally terraform fetched the attributes of the created resources listed in outputs.tf

Note : We’ll now destroy the Vnet as the next instance deploy contains the same Vnet specs.

$ terraform destroy -auto-approve

Destroy complete! Resources: 6 destroyed.

V. Full deployment (Instance)

After our small intro to Vnet creation, let’s launch a vm and configure nginx in it in one command.
First we need to switch to the second directory terraform-provider-azure/launch-instance/ Here's the content:

$ tree ./terraform-provider-azure/launch-instance
.
|-- cloud-init --> SubFolder
| `--> centos_userdata.txt --> script to config a webserver+ modify the index
| `--> sles_userdata.txt --> for SUSE
| `--> ubto_userdata.txt --> for Ubunto 
| `--> el_userdata.txt --> for Enteprise linux distros 
| `--> Win_userdata.ps1 --> for windows 

|-- compute.tf ---> Instance related terraform configuration
|-- outputs.tf ---> displays the resources detail at the end of the deploy
|-- variables.tf ---> Resource variables needed for the deploy   
|-- vnet.tf ---> same vnet we deployed earlier

Note: As you can see we have 2 additional files and one Subfolder. is where the compute instance and all its attributes are declared. All the other “. tf” files come from my vnet example with some additions for variables.tf and output.tf

Cloud-init : is a cloud instance initialization method that executes tasks upon instance Startup by providing the user_data entry of the azure vm resource definition (See below). I have created enough for 6 OS’ (Centos,Ubuntu,Windows,RHEL,OL,SUSE)

...variable "user_data" { default = "./cloud-init/centos_userdata.txt"} 

 $ vi compute.tf
resource "azurerm_linux_virtual_machine" "terravm" {
...
custom_data = base64encode ("${file(var.user_data)}")
...

In my lab, I used cloud-init to install nginx and write an html page that will replace the HomePage at Startup.
Make sure you your public ssh key is in your home directory or just modify the path below (see variables.tf)

$ vi compute.tf
resource "azurerm_linux_virtual_machine" "terravm" {
.. 

admin_ssh_key {

   username = var.os_publisher[var.OS].admin

   public_key = file("~/id_rsa_az.pub") } ## Change m

LAUNCH THE INSTANCE

Once in “ launch-instance” directory, you can run the plan command to validate the 9 resources required to launch our vm instance. The output has been truncated to reduce verbosity

$ terraform plan
 Terraform used selected providers to generate following execution plan. 
 Resource actions are indicated with the following symbols:  
  ------------------------------------------------------------------------
    Terraform will perform the following actions:

  ... # Vnet declaration (see previous vnet deploy) 
  ...  
 # azurerm_linux_virtual_machine.terravm will be created
  + resource "azurerm_linux_virtual_machine" "terravm" {
    + ...
    + admin_username = "centos"  
    + location = "eastus"   
    + size = "Standard_B1s" 
    + name = "TerraDemo-vm"
    + private_ip = "192.168.10.51"
    + admin_ssh_key { = {
      + public_key = <<-EOT ssh-rsa AAAABxxx…*
         EOT 
         username =”centos” }
     + custom_data = (sensitive value)
     + os_disk {…}
     + source_image_reference {
         + offer = "CentOS"
         + publisher = "OpenLogic"
         + version = "latest"
         }   
     }
  # azurerm_network_interface.Terranic will be created
  # azurerm_network_interface_security_group_association.terra_assoc_pubip_nsg will be created
  # azurerm_network_security_group.terra_nsg will be created
  # azurerm_public_ip.terrapubip will be created
    ...}
   ...
  }
  Plan: 9 to add, 0 to change, 0 to destroy.

Now let’s launch our CENTOS7 vm using terraform apply (I left a map of different OS ids in the variables.tf you can choose from)

$ terraform apply -auto-approve
...
azurerm_resource_group.rg: Creating...
azurerm_virtual_network.terra_vnet: Creating...
azurerm_network_security_group.terra_nsg: Creating...
azurerm_subnet.terra_sub: Creating...
azurerm_subnet_network_security_group_association.nsg_sub: Creating...
azurerm_network_interface.Terranic: Creating...
azurerm_linux_virtual_machine.terravm: Creating......
Apply complete! Resources: 9 added, 0 changed, 0 destroyed.

Outputs:
... 
Subnet_Name = "internal"
vnet_CIDR = 192.168.0.0/16
Subnet_CIDR = 192.168.10.0/24
vnet_dedicated_security_ingress_rules = toset([
 {…
  "access" = "Allow" 
  "description" = "RDP-HTTP-HTTPS ingress trafic"
  "destination_port_ranges" = toset([
  "22",
  "3389",
  "443",
  "80",])
]
SSH_Connection = ssh connection to instance TerraCompute ==> sudo ssh -i ~/id_rsa_az centos@’’ <---- IP not displayed due to bug with az provider
private_ip = "192.168.10.4"
public_ip = ""

Once the instance is provisioned, juts copy the public IP address(i.e 52.191.26.102 ) in chrome and Voila!
Although a bug of azurerm_linux_virtual_machine doesn’t show the public IP , just go to the console and copy past the IP into your browser
You can also tear down this configuration by simply running terraform destroy from the same directory

Tips

You can fetch any of the specified attributes in outputs.tf using terraform output command i.e:

$ terraform output SSH_Connection
ssh connection to instance TerraCompute ==> sudo ssh -i ~/id_rsa_az centos@ ’’

Terraform Console: Although terraform is a declarative language, there are still myriads of functions you can use to process strings/number/lists/mappings etc. There is an excellent all in one script with examples of most terraform functions >> here
I added cloud-init files for different distros you can play with by adapting var.user_data & var.OS

Differences between Azure/AWS & things I wish azure kept simple

Network In Azure every subnet is a public subnet because as soon as you associate a public IP to a Vm’s VNIC, you’ll magically have internet access. Internet gateway is not needed here because system routes are taking care of that.
CIDR range in azure is slightly larger than aws ( from /8 to /29 ).
ID Azure doesn’t provide regular alpha numeric IDs for its resources but a sort of path based identification (see below)

$ SUBNET ID  
/subscriptions/xx/resourceGroups/my_group/providers/Microsoft.Network/virtualNetworks/MY-VNET/subnets/My_SUBNET

Naming Case insensitive but unique, a resource group can’t have 2 resources of the same type with the same name
Bummer: I wish azure kept one terraform resource to create vms instead of having one for Linux and one for Windows since v2.0. This implies that we will have extra blocks/modules if we want to do both in one stack instead of leveraging loops. The classic one is not handy for Linux either.

CONCLUSION

We have demonstrated in this tutorial how to quickly deploy a web server instance using terraform in Azure and leverage Cloud-init to configure the vm during the bootstrap .
Remember that all used attributes in this exercise can be modified in the file.
Route table and internet gateway setting in our code were replaced by the Public IP and VNICs
I found azure vm spinning time so slow comparing to AWS or OCI, not sure if it’s it’s regional thing
Improvement: I will look to improve the compute.tf code to allow both windows & Linux type of resource to provision. Validate that userdata works for windows too, unlike on az cli. Another improvement can be reached in terms of display of the security rules using formatlist . stay tuned

Thank you for reading!

Originally published at http://www.brokedba.com.

Terraform for dummies part 2: Launch an instance with a static website on AWS

CloudDude — Fri, 09 Jun 2023 23:35:02 +0000

Intro

This has become a habit so far to explore different ways of automated provisioning for each cloud provider. This time, I will try Terraform on AWS and reproduce the same deployment I have completed on Oracle Cloud, and as usual we won’t just deploy an instance but also configure a website linked to its public IP. I’ll end this post with some notes on OCI/AWS differences.

Note: I have done a similar task with my bash scripts and which was very helpful to understand the logic behind each IAC interaction during the resource creation (More details > here).

Topology

Besides describing my GitHub repo before starting this tutorial, I’ll just briefly discuss some principles.

Infrastructure As Code Manages and provisions cloud resources using a declarative code (i.e Terraform) and definition files avoiding interactive configuration. Terraform is an immutable Orchestrator that creates and deletes all resources in the proper sequence. Each Cloud vendor has what we call a provider that terraform uses in order to convert declarative texts into API calls reaching the Cloud infrastructure layer.
Terraform Files
Can be a single file or split into multiple tf or tf.json files, any other file extension is ignored.
Files are merged in alphabetical order but resource definition order doesn’t matter (subfolders are not read).
Common configurations have 3 type of tf files and a statefile.
Terraform resource declaration syntax looks like this:

Component "Provider_Resource_type" "MyResource_Name" { Attribute1 = value .. 
                                                       Attribute2 = value ..}

Where do I find a good AWS deployment sample? The easiest way is to create/locate an instance from the console and then use the import function from terraform to generate each of the related components in HCL format (vpc,instance,subnet,etc..) based on their id.

Example for a VPC >>

1- Create a shell resource declaration for the vpc in a file called vpc.tf

2- Get the id of the vpc resource from your AWS Console

3- Run the Terraform import then run Terraform show to extract the vpc’s full declaration from aws in the same file (vpc.tf)

4- Now you can remove the id attribute with all non required attributes to create a vpc resource (Do that for each resource)

1- # vi vpc.tf 
provider "aws" { region = "us-east-1" }
 resource "aws_vpc" "terra_vpc" {
 }
2- # terraform import aws_vpc.terra_vpc vpc-0091141e28608813c
3- # terraform show -no-color > vpc.tf

Note: If you want to import all the existing resources in your account in bulk mode (not one by one) there is a tool called Terraforming , which can import both code and state from your AWS account automatically.

Terraform lab content: I have deliberately split this lab in 2:

VPC Deployment: To grasp the basics of a single resource deployment.
Instance Deployment: includes the instance provisioning (with above vpc) with a hosted web sever.

I.Terraform setup

Since I’m on windows I tried the lab using both Gitbash and WSL (Ubuntu) terminal clients (same applies for Mac).

Windows: Download and run the installer from their website (32-bit ,64-bit)

Linux: Download, unzip and move the binary to the local bin directory

$ wget https://releases.hashicorp.com/terraform/0.12.28/terraform_0.12.28_linux_amd64.zip 
$ unzip terraform_0.12.18_linux_amd64.zip 
$ mv terraform /usr/local/bin/

Once installed run the version command to validate your installation

$ terraform --version   
Terraform v0.12.24

AWS authentication

To authenticate with your aws account, Terraform will need to provide both access_key_id & secret_access_key . This can be done either by sharing the authentication parameters with aws-cli or by Including the access_key and key_id within the Terraform config (i.e. variables.tf)

Assumptions

I will assume that either of the two below authentication options are present/configured in your workstation:

AWSCLI default profile configured with your aws credentials (Access keys). Refer to my Blog post for more details

$ aws configure list
Name Value Type Location
---- --------- ---- -------- 
profile <not set> None None
access_key ********* J2WA shared-credentials-file
region us-east-1 
config-file ~/.aws/config

Or AWS credentials imbedded in provider specific config section on one of your terraform files (See vpc.tf)

provider "aws" {
# access_key = "${var.aws_access_key}" - uncomment & replace accordingly 
# secret_key = "${var.aws_secret_key}" - uncomment & replace accordingly
region = var.aws_region --- uncomment and replace accordingly}

I’ll also assume the presence of an ssh key pair to attach to your ec2 instance. If not here is a command to generate a PEM based key pair.

$ ssh-keygen -P "" -t rsa -b 2048 -m pem -f ~/id_rsa_aws
     Generating public/private rsa key pair.
 Your identification has been saved in /home/brokedba/id_rsa_aws.
 Your public key has been saved in /home/brokedba/id_rsa_aws.pub.

II. Clone the repository

Pick an area that is close to your aws-terraform directory on your file system and issue the following command

$ git clone https://github.com/brokedba/terraform-examples.git

Note: As explained earlier you will find 2 directories inside the repository which will make things easier:

terraform-provider-aws/create-vpc/ To grasp how we deploy a VPC.
terraform-provider-aws/launch-instance/ For the final instance deploy.

III. Provider setup

INSTALL AND SETUP THE AWS PROVIDER
Cd Into the subdirectory terraform-provider-aws/create-vpc where our configuration resides (i.e vpc)

 $ cd ~/brokedba/aws/terraform-examples/terraform-provider-aws/create-vpc

AWS provider plugin is distributed by HashiCorp hence it will be automatically installed by terraform init.

$ terraform init Initializing the backend... 
Initializing provider plugins...
 - Checking for available provider plugins... 
- Downloading plugin for provider "aws" (hashicorp/aws) 3.11.0... 
* provider.aws: version = "~> 3.11"

$ terraform --version 
Terraform v0.12.24 
+ provider.aws v3.11.0 ---> the provider is now installed

Let’s see what’s in the directory. Here, only files matter

$ tree . 
|-- outputs.tf ---> displays resources detail after the deploy 
|-- variables.tf ---> Resource variables needed for the deploy 
|-- vpc.tf ---> Our vpc terraform declartion

IV. Partial Deployment

DEPLOY A SIMPLE VPC

Once the authentication configured (access_key_id/secrete set) , we can run terraform plan command to create an execution plan (quick dry run to check the desired state/actions).

$ terraform plan
   Refreshing Terraform state in-memory prior to plan... 
  ------------------------------------------------------------------------
  An execution plan has been generated and is shown below.
    Terraform will perform the following actions:

    # aws_internet_gateway.terra_igw will be created
    + resource "aws_internet_gateway" "terra_igw" 
    {..}
    # aws_route_table.terra_rt will be created
    + resource "aws_route_table" "terra_rt" {
    {..} 
    # aws_route_table_association.terra_rt_sub will be created
    + "aws_route_table_association" "terra_rt_sub" 
    {..}
    # aws_security_group.terra_sg will be created
    + resource "aws_security_group" "terra_sg" {
    + arn = (known after apply)
    + description = "SSH ,HTTP, and HTTPS"
    + egress =[{...}]
    + id = (known after apply)
    + ingress = [
        + {... rules for HTTP/SSH/HTTPS ingress access}
    # aws_subnet.terra_sub will be created
    + resource "aws_subnet" "terra_sub" {
    ...
    + cidr_block = "192.168.10.0/24”
   ...}
    # aws_vpc.terra_vpc will be created
    + resource "aws_vpc" "terra_vpc" {
    ...
    + cidr_block = "192.168.0.0/16”
    ...
    + tags = {
        + "Name" = "Terravpc" } }

  Plan: 6 to add, 0 to change, 0 to destroy.

The output being too verbose I deliberately kept only relevant attributes for the VPC resource plan

$ terraform apply -auto-approve
aws_vpc.terra_vpc: Creating...
...
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

Outputs:

Subnet_CIDR = 192.168.10.0/24
Subnet_Name = terrasub
internet_gateway_Name = terra-igw
map_public_ip_on_launch = true
route_table_Name = terra-rt
vpc_Name = Terravpc
vpc_id = vpc-09d491059eb740562
vpc_CIDR = 192.168.0.0/16
vpc_dedicated_security_group_Name = terra-sg
vpc_dedicated_security_ingress_rules = 
["Inbound HTTP access : 80 , CIDR: 0.0.0.0/0",
  "Inbound HTTPS access : 443 , CIDR: 0.0.0.0/0",
  "Inbound RDP access : 3389 , CIDR: 0.0.0.0/0",
  "Inbound SSH access: 22 , CIDR: 0.0.0.0/0",]

Observations:

The deploy started by loading the resources variables in variables.tf which allowed the execution of vpc.tf
Finally terraform fetched the attributes of the created resources listed in outputs.tf

Note : We’ll now destroy the vpc as the next instance deploy contains the same vpc specs.

$ terraform destroy -auto-approve 
...
Destroy complete! Resources: 6 destroyed.

V. Full deployment (Instance)

OVERVIEW

Sweet, After our small test let’s launch a full instance from scratch.
First we need to switch to the second directory terraform-provider-aws/launch-instance/

Here's its content:

$ tree ./terraform-provider-aws/launch-instance
.
|-- cloud-init ---> SubFolder
| `--> vm.cloud-config ---> script to config a webserver & add a HomePage
|-- compute.tf ---> Instance related terraform configuration
|-- outputs.tf ---> displays the resources detail at the end of the deploy
|-- variables.tf ---> Resource variables needed for the deploy   
|-- vpc.tf ---> same vpc terraform declaration deployed earlier

Note: As you can see we have 2 additional files and one Subfolder. is where the compute instance and all its attributes are declared. All the other . tf files come from my vpc example with some additions for variables.tf and output.tf

Cloud-init : is a cloud instance initialization method that executes tasks upon instance startup by providing the user_data entry of the aws_instance resource definition (See below).

...variable "user_data" { default = "./cloud-init/vm.cloud-config"} 

 $ vi compute.tf
resource "aws_instance" "terra_inst" {
...
user_data = filebase64(var.user_data)
...

In my lab, I used cloud-init to install nginx and write an html page that will be the server’s HomePage at Startup.
Make sure you your public ssh key is in your home directory or just modify the path below (see variables.tf)

resource "aws_key_pair" "terra_key" {
key_name = var.key_name
public_key = file("~/id_rsa_aws.pub") } ## Change me

LAUNCH THE INSTANCE

Once in “ launch-instance” directory, you can run the plan command to validate the 10 resources required to launch the ec2 instance (end-state). The output has been truncated to reduce verbosity

$ terraform plan
   Refreshing Terraform state in-memory prior to plan... 
  ------------------------------------------------------------------------
  An execution plan has been generated and is shown below.
    Terraform will perform the following actions:

  ... # VPC declaration (see previous vpc deploy 
  ...
  # aws_key_pair.terra_key will be created
   + resource "aws_key_pair" "terra_key" {
      + key_name = "demo_aws_KeyPair"
      + key_pair_id = (known after apply)
      + public_key = "ssh-rsa AAAAB3Nz…"
      ...}
  # aws_ebs_volume.terra_vol[0] will be created
  + resource "aws_ebs_volume" "terra_vol" {...}

 # aws_instance.terra_inst will be created
  + resource "aws_instance" "terra_inst" {
    + ...
    + ami = "ami-01861c2f0a2adfdb7"  
    + availability_zone = "us-east-1a"   
    + instance_type = "t2.micro" 
    + key_name = "demo_aws_KeyPair"
    + private_ip = "192.168.10.51"
    + tags = {
     ----+ "Name" = "TerraCompute"
        }
     + user_data = "c8c701575f9c76db131ccf77cf352da"
     + ebs_block_device {
     + network_interface {
     + root_block_device {
     + ...
     + ...}

  # aws_volume_attachment.terra_vol_attach[0] will be created
  + resource "aws_volume_attachment" "terra_vol_attach" {
      + device_name = "/dev/xvdb"
      ...}
   ...
  }
  Plan: 10 to add, 0 to change, 0 to destroy.

Let’s launch our CENTOS7 instance using terraform apply (I left a map of different OS ami’s in the variables.tf you can choose)

$ terraform apply -auto-approve ...aws_vpc.terra_vpc: Creating...

$ terraform apply -auto-approve
...
aws_vpc.terra_vpc: Creating...
aws_key_pair.terra_key: Creation complete after 0s [id=demo_aws_KeyPair]
aws_security_group.terra_sg: Creation complete after 3s [id=sg-0b04564e8b]
aws_instance.terra_inst: Still creating... [40s elapsed]
aws_instance.terra_inst: Creation complete after 44s [id=i-046f1b1406bff]
aws_ebs_volume.terra_vol[0]: Creation complete after 11s [id=vol-037c1a9f9]aws_volume_attachment.terra_vol_attach[0]: Creation complete after 21s [.. 
...
Apply complete! Resources: 10 added, 0 changed, 0 destroyed.

Outputs:
...
vpc_Name = Terravpc
vpc_CIDR = 192.168.0.0/16
Subnet_CIDR = 192.168.10.0/24
SecurityGroup_ingress_rules = [
  "Inbound HTTP access : 80 , CIDR: 0.0.0.0/0",
  "Inbound HTTPS access : 443 , CIDR: 0.0.0.0/0",
  "Inbound RDP access : 3389 , CIDR: 0.0.0.0/0",
  "Inbound SSH access: 22 , CIDR: 0.0.0.0/0",
]
SSH_Connection = ssh connection to instance TerraCompute ==> sudo ssh -i ~/id_rsa_aws centos@35.173.222.166
private_ip = "192.168.10.51"
public_ip = "54.91.27.50"

Once the instance is provisioned, just copy the public IP address(54.91.27.50 ) in your browser and Voila!
Here I just embedded a video link into the index page but you can adapt the cloud-init file to your own liking (with a new content in the write_file section.
You can also tear down this configuration by simply running terraform destroy from the same directory

Tips

You can fetch any of the specified attributes in outputs.tf using terraf orm out put command i.e:

$ terraform output SSH_Connection
ssh connection to instance TerraCompute ==> sudo ssh -i ~/id_rsa_aws 
centos@54.91.27.50

Terraform Console:

Although terraform is a declarative language, there are still myriads of functions you can use to process strings/number /lists/mappings etc. In my case I had to do some tests to get the right syntax for outputting the ingress SG rules:

$ terraform console
> formatlist("% s: %s" ,aws_security_group.terra_sg.ingress[*].description,aws_security_group.terra_sg.ingress[*].to_port)
[ "Inbound HTTP access : 80", "Inbound HTTPS access : 443",
"Inbound SSH access: 22",]

There is an excellent all in one script with examples of most terraform functions >> here

Differences between OCI/AWS & things I wish AWS provider had

Unlike OCI, most of aws resources don’t have a display-name attribute, hence a tag is necessary each time you want to create a resource in AWS
If you are used to DNS-labels for Subnet/VCNs in OCI, this is not something you’ll find in AWS
There is no direct way of setting the instance hostname like in OCI’s hostname_label attribute, you will have to do it via user-data
Bummer: I wish AWS had a data source for aws_key_pair as we can’t check if a key pair exists while launching an instance. This is a drawback for cases where a dev team needs to share a key. The only option is to give it a different keyname for each deployment even if the key is the same, which will generate a lot of unnecessary duplicates .

CONCLUSION

We have demonstrated in this tutorial how to quickly deploy an instance using terraform in AWS and leverage Cloud-init to bootstrap an instance into a webserver.
Remember that all used attributes in this exercise can be modified in the variables.tf file.
Improvement: In my next blog post, I will look to improve the vpc.tf code to leverage for_each function on map collections to make security group rules creation dynamic and conditional. stay tuned

Thank you for reading!

Originally published at http://www.brokedba.com.

Oracle 19c Data Guard builds on Rhel8 /Oracle linux8/Centos8 (Vagrant)

CloudDude — Fri, 09 Jun 2023 22:58:41 +0000

With Tools like Vagrant and virtualbox nowadays , developers can seamlessly re/create different environments for their apps by just checking vagrantfile definition, run a vagrant up, and start designing.

This is more a description of new vagrant builds I’ve been working on lately. They are based on the work that Tim hall started a while now on his Github. So hat tip to Tim, he’s done the hard work, I merely adapted the build to the new Enterprise linux 8 distros (rhel/centos/oracle linux) and made sure nothing bugged during each provisioning.

Oracle 19c database and Data Guard builds on RHEL8/OL8/CENTOS8

After forking his repo on github, I then started to adapt above Oracle 19c buildsto run on Red Hat 8 and Oracle linux 8. Centos 8 being left alone, I decided to include it to the rest of the pack :).

As I didn’t write a blog post for this . I thought it’d be convenient for the reader to have an excerpt of the builds’ descriptions (Readme files) before kicking it off and trying them.

Required Software

Clone Repository

Pick an area on your file system to act as the base for this git repository and issue the following command.

git clone [https://github.com/brokedba/vagrant.git](https://github.com/KoussHD/vagrant.git)

You can also use Git GUI to clone it as shown below

I. Oracle 19c database on RHEL8:

Note: let’s pick rhel_19_gen directory which is a build that uses a generic rhel8 vagrant box. You could, however, choose any of the builds listed above as it works the same.

¤¤ rhel_19_gen :README.md ¤¤

Oracle 19c on Red Hat Linux 8

A simple Vagrant build for Oracle Database 19c on Red hat linux 8.

Note: The vagrant box here is generic and will be provisioned with the prerequisite rpm packages for Oracle 19c through a RHEL8 Beta repository (see install_os_packages.sh for details).

Place the software in the “software” directory before calling the vagrant up command.

Directory contents when software is included.

$ tree . 
+--- README.md 
+--- scripts 
| +--- dbora.service 
| +--- install_os_packages.sh 
| +--- oracle_create_database.sh 
| +--- oracle_service_setup.sh 
| +--- oracle_software_installation.sh 
| +--- oracle_user_environment_setup.sh 
| +--- ords_software_installation.sh 
| +--- prepare_u01_u02_disks.sh 
| +--- root_setup.sh 
| +--- server.xml 
| +--- setup.sh 
+--- software 
| +--- apache-tomcat-9.0.22.tar.gz *optional 
| +--- apex_19.1_en.zip *optional 
| +--- LINUX.X64_193000_db_home.zip 
| +--- openjdk-12.0.2_linux-x64_bin.tar.gz *optional 
| +--- ords-19.2.0.199.1647.zip *optional 
| +--- put_software_here.txt 
| +--- sqlcl-19.2.1.206.1649.zip *optional 
+--- Vagrantfile $

Locate the rhel_19_gen directory and the run the below vagrant command.

cd D:\VM\vagrant\database\rhel_19_gen vagrant up ..

II. Oracle 19c Data Guard on CENTOS8:

Note: Here also I just chose one of the builds but they all work the same .

¤¤ cent8_19:README.md ¤¤

Vagrant 19c Data Guard Build on Centos 8

Note: The vagrant base box of centos 8 is generic and will be provisioned with the prerequisite rpm packages for Oracle 19c from a centos repository during vagrant start.

The Vagrant scripts here will allow you to build a 19c Data Guard system on Centos 8. by just starting the VMs in the correct order.

This configuration is slightly modified comparing to Tim hall’s original build but the modification merely resides in the below environment variables values (that are easy to adapt to your liking) and two start/stop scripts.

export DOMAIN_NAME=evilcorp.com export NODE1_HOSTNAME=montreal export NODE2_HOSTNAME=toronto export NODE1_PUBLIC_IP=192.168.78.54 export NODE2_PUBLIC_IP=192.168.78.55 export ORACLE_SID=montreal export NODE1_DB_UNIQUE_NAME=montreal export NODE2_DB_UNIQUE_NAME=toronto export ROOT_PASSWORD=racattack export ORACLE_PASSWORD=oracle export SYS_PASSWORD="racattack" export PDB_PASSWORD="PdbPassword1!"

You will also need to download the 19c database software.

Copy the Oracle software under the “dataguard/cent8_19/software” directory. From the “dataguard” subdirectory, the structure should look like this.

tree . 
+--- config 
| +--- install.env 
| +--- vagrant.yml 
+--- node1 
| +--- scripts 
| | +--- oracle_create_database.sh 
| | +--- oracle_user_environment_setup.sh 
| | +--- root_setup.sh 
| | +--- setup.sh 
| +--- Vagrantfile +--- node2 
| +--- scripts 
| | +--- oracle_create_database.sh 
| | +--- oracle_user_environment_setup.sh 
| | +--- root_setup.sh 
| | +--- setup.sh 
| +--- Vagrantfile 
+--- README.md 
+--- shared_scripts 
| +--- configure_chrony.sh 
| +--- configure_hostname.sh 
| +--- configure_hosts_base.sh 
| +--- configure_shared_disks.sh 
| +--- install_os_packages.sh 
| +--- oracle_db_software_installation.sh 
| +--- prepare_u01_disk.sh +--- software 
| +--- LINUX.X64_193000_db_home.zip 
| +--- put_software_here.tx

Build the Data Guard System

The following commands will leave you with a functioning Data Guard installation.

Start the first node and wait for it to complete. This will create the primary database.

cd node1 vagrant up

Start the second node and wait for it to complete. This will create the standby database and configure the broker.

cd ../node2 vagrant up

Turn Off System

Perform the following to turn off the system cleanly. (Stop_all script will set state to apply-off or transport-off according to the role of the node’s DB before db shutdown)

On Node 1

oracle@# /home/oracle/scripts/stop_all.sh cd ../node2 vagrant halt

On Node 2

oracle@# /home/oracle/scripts/stop_all.sh cd ../node1 vagrant halt

Restart the System after first install and shutdown

Perform the following to turn on the system cleanly (start_all.sh script will run a startup if local DB role is Primary or startup mount if the local DB role is Standby).

cd node1 vagrant up oracle@# /home/oracle/scripts/start_all.sh cd node2 vagrant up oracle@# /home/oracle/scripts/start_all.sh

Remove the whole System

The following commands will destroy all VMs and the associated files, so you can run the process again.

cd ../node2 vagrant destroy -f cd ../node1 vagrant destroy -f

Originally published at https://www.linkedin.com.

Terraform for Dummies Part 4: Launch a VM with a Static Website on a Google Cloud Platform —…

CloudDude — Fri, 29 Oct 2021 17:49:50 +0000

Terraform for Dummies Part 4: Launch a VM with a Static Website on a Google Cloud Platform — Eclipsys

Terraform for Dummies Part 4: Launch a VM with a Static Website on a Google Cloud Platform

Intro

After AWS, Oracle Cloud, and Azure, GCP (Google Cloud Platform) is the 4th cloud platform in our Terraform tutorial series. We will describe what it takes to authenticate and provision a compute engine using their Terraform provider. The instance will also have an Nginx website linked to its public IP. If you want to know about the differences GCP brings in terms of networking, you can check out my blog.

Note: GCP Terraform provider authentication was hell to get a hold on and counter-intuitive compared to other Cloud platforms. I wasted a lot of time just trying to figure if I could avoid hardcoding project id.

Here’s a direct link to my GitHub repo linked to this lab: terraform-examples/terraform-provider-gcp

Content:

I. Terraform Setup

IV. Partial Deployment

V. Full Deployment

Tips and Conclusion

Overview and Concepts

Topology

The following illustration shows the layers involved between your workstation and GCP cloud while running the Terraform actions along with the instance attributes we will be provisioning.

Besides describing my GitHub repo before starting this tutorial, I’ll just briefly discuss some principles.

Terraform Files

Can be a single file or split into multiple tf or tf.json files, any other file extension is ignored.

Files are merged in alphabetical order but resource definition order doesn’t matter (subfolders are not read).

Common configurations have 3 type of tf files and a statefile.

main.tf: Terraform declaration code (configuration) . The file name can be anything you choose
variables.tf: Resource variables needed for the deploy
outputs.tf: displays the resources detail at the end of the deploy
terraform.tfstate: keeps track of the state of the stack(resources) after each Terraform apply run

Terraform resource declaration syntax looks like this:

Component "Provider_Resource_type" "MyResource_Name" { Attribute1 = value .. 
                                                       Attribute2 = value ..}

Where do I find a good GCP deployment sample?

The easiest way is to create/locate an instance from the console and then use the import function from Terraform to generate each of the related components in HCL format (VPC, instance,subnet,etc..) based on their id.

Example for a VPC >>

Create a shell resource declaration for the VPC in a file called ** vpc.tf**
Get the id of the VPC resource from your GCP portal
Run the Terraform import then run Terraform show to extract the VPC full declaration from GCP to the same file ( vpc.tf)
Now you can remove the id attribute with all non required attributes to create a vpc resource (Do that for each resource)

1- # vi vpc.tf 
  provider "google" { 
              features {}
    }
  resource "google_compute_network" "terra_vpc" {
   }
2- # terraform import google_compute_network.terra_vpc {{project}}/{{name}}
3- # terraform show -no-color > vpc.tf

Note

If you want to import all the existing resources in your account in bulk mode terraformer can help import both code and state from your GCP account automatically.

Terraform Lab Content: I purposely split this lab in 2 for more clarity

VPC Deployment: To grasp the basics of single resource deployment.
Instance Deployment: Includes the instance provisioning configured as a web server(includes above VPC).

I. Terraform Setup

Although I’m on windows I tried the lab using WSL (Ubuntu) terminal (but same applies to Mac).

Windows: Download and run the installer from their website (32-bit ,64-bit)
Linux: Download, unzip and move the binary to the local bin directory

$ wget https://releases.hashicorp.com/terraform/1.0.3/terraform_1.0.3_linux_amd64.zip 
$ unzip terraform_1.0.3_linux_amd64.zip 
$ mv terraform /usr/local/bin/

Once installed run the version command to validate your installation

$ terraform --version
  Terraform v1.0.3

GCP authentication (least user friendly)

To authenticate to GCP with Terraform you will need GCloud , service account credentials key file, and the projectId

Prerequisites

Using dedicated service accounts to authenticate with GCP is recommended practice (not user accounts or API keys)

GCLOUD authentication configured with your GCP credentials. Refer to my Blog post for more details

$ gcloud auth login --activate
…
$ gcloud config list --format='table(account,project)'
 ACCOUNT PROJECT
-------------- ------------- 
bdba@gmail.com brokedba2000

Service account: Either you create a service account with “ owner role “ in the console or run the below cli commands

1 -- Create service account
$ gcloud iam service-accounts create terraform-sa --display-name="Terra_Service"
$ gcloud iam service-accounts list --filter="email~terraform" --format='value(email)'

2 -- Bind it to a project and add owner role 
$ gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:email" --role="roles/owner"

3 -– Generate the Key file for the service account
$ gcloud iam service-accounts keys create ~/gcp-key.json --iam-account=email

I’ll also assume the presence of an ssh key pair to attach to your VM instance. If not here is a command to generate a PEM based key pair.

$ ssh-keygen -P "" -t rsa -b 2048 -m pem -f ~/id_rsa_az
     Generating public/private rsa key pair.
 Your identification has been saved in /home/brokedba/id_rsa_az.
 Your public key has been saved in /home/brokedba/id_rsa_az.pub.

II. Clone the Repository

Pick an area that is close to your gcp-terraform directory on your file system and issue the following command.

$ git clone https://github.com/brokedba/terraform-examples.git

Note: As explained earlier you will find 2 directories inside the repository which will make things easier:

terraform-provider-gcp/create-vpc/ To grasp how we deploy a VPC.
terraform-provider-gcp/launch-instance/ For the final instance deploy.

III. Provider Setup

1. INSTALL AND SETUP THE GCP PROVIDER

Cd Into “ terraform-provider-gcp/create-vpc” where our configuration resides (i.e vpc)

$ cd /brokedba/gcp/terraform-examples/terraform-provider-gcp/create-vpc

GCP provider plugin will be automatically installed by running ”terraform init”.

$ terraform init
  Initializing the backend...

  Initializing provider plugins...
  - Finding latest version of hashicorp/google...
  - Installing hashicorp/google v3.88.0...
  * Installed hashicorp/google v3.88.0 (signed by HashiCorp)
Terraform has been successfully initialized!

$ terraform --version
  Terraform v1.0.3 on linux_amd64
  + provider registry.terraform.io/hashicorp/google v3.88.0

Let’s see what’s in the ”create-vpc” directory. Here, only *.tf files matter

$ tree
  .
  |-- outputs.tf ---> displays resources detail after the deploy
  |-- variables.tf ---> Resource variables needed for the deploy   
  |-- vpc.tf ---> Our vpc terraform declaration

IV. Partial Deployment

DEPLOY A SIMPLE VPC

Once the authentication is setup and provider installed , we can run terraform plan command to create an execution plan (quick dry run to check the desired end-state).

$ terraform plan
var.prefix The prefix used for all resources in this example
Enter a value: Demo
Terraform used selected providers to generate the following execution plan. 
Resource actions are indicated with the following symbols: + create
  ------------------------------------------------------------------------
  Terraform will perform the following actions:

    # google_compute_network.terra_vpc will be created
    + resource "google_compute_firewall" "web-server" 
    {..
     + name = "allow-http-rule"
     + allow {
     + ports = [+ "80", + "22",+ "443",+ "3389",]
     + protocol = "tcp"
    ...

    # google_compute_firewall.web-server will be created
    + resource "google_compute_firewall" "web-server" {
    {..} 
    # google_compute_subnetwork.terra_sub will be created
    + resource "google_compute_subnetwork" "terra_sub"  
    {..
     ip_cidr_range = ["192.168.10.0/24”]
     ...}

  Plan: 3 to add, 0 to change, 0 to destroy.

The output being too verbose I deliberately kept only relevant attributes for the VPC resource plan
Next, we can run ”terraform deploy” to provision the resources to create our VPC (listed in the plan)

$ terraform apply -auto-approve
google_compute_network.terra_vpc: Creating...
google_compute_firewall.web-server: Creating...
google_compute_subnetwork.terra_sub: Creating...
...
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

project = "brokedba2000"

Observations:

The deployment started by loading the resources variables in variables.tf which allowed the execution of vpc.tf
Finally Terraform fetched the attributes of the created resources listed in outputs.tf

Note : We’ll now destroy the VPC as the next instance deploy contains the same VPC specs.

$ terraform destroy -auto-approve

Destroy complete! Resources: 3 destroyed.

V. Full deployment (Instance)

1. OVERVIEW

After our small intro to VPC creation, let’s launch a VM and configure Nginx in it in one command.
First, we need to switch to the second directory terraform-provider-gcp/launch-instance/ Here's the content:

$ tree ./terraform-provider-gcp/launch-instance
.
|-- cloud-init --> SubFolder
| `--> centos_userdata.txt --> script to config a webserver the Web homepage
| `--> sles_userdata.txt --> for SUSE
| `--> ubto_userdata.txt --> for Ubunto 
| `--> el_userdata.txt --> for Enteprise linux distros 

|-- compute.tf ---> Compute engine Instance terraform configuration
|-- outputs.tf ---> displays the resources detail at the end of the deploy
|-- variables.tf ---> Resource variables needed for the deploy   
|-- vpc.tf ---> same vpc we deployed earlier

Note: As you can see we have 2 additional files and one Subfolder. is where the compute instance and all its attributes are declared. All the other “. tf” files come from my VPC example with some additions for variables.tf and output.tf

Cloud-init : is a cloud instance initialization method that executes scripts upon instance Startup. see below metadata entry of the vm instance definition (startup-script). There are 5 OS’ scripts (Centos,Ubuntu,Windows,RHEL,SUSE) windows was not tested.

...variable "user_data" { default = "./cloud-init/centos_userdata.txt"} 

 $ vi compute.tf 
resource "google_compute_instance" "terravm" { 
metadata = {
startup-script = ("${file(var.user_data)}")
...

In my lab, I used cloud-init to install Nginx and write an HTML page that will replace the HomePage at Startup.
Make sure your public ssh key is in your home directory or just modify the path below (see variables.tf)

$ vi compute.tf
resource "google_compute_instance" "terravm" {
metadata = {

admin_ssh_key {

   ssh-keys = var.admin":${file("~/id_rsa_gcp.pub")}" ## Change me

2. LAUNCH THE INSTANCE

Once in “ launch-instance” directory, you can run the plan command to validate the 9 resources required to launch our vm instance. The output has been truncated to reduce verbosity

$ terraform plan
  ------------------------------------------------------------------------
    Terraform will perform the following actions:

  ... # VPC declaration (see previous VPC deploy) 
  ...  
 # google_compute_instance.terra_instance will be created
 + resource "google_compute_instance" "terra_instance" {
    + ...
    + hostname = "terrahost"
    + machine_type = "e2-micro"
    + name = "Terravm"
    + tags = [+ "web-server",]
    + boot_disk {
    + initialize_params { + image = "centos-cloud/centos-7"
    + network_interface {
      ... 
    + network_ip = "192.168.10.51"
     } 
    + metadata = {
  + "ssh-keys" = <<-EOT ssh-rsa AAAABxxx…*
         EOT 
   + "startup-script" = <<-EOT”        
      EOT}

  # google_compute_address.internal_reserved_subnet_ip will be created
  + resource "google_compute_address" "internal_reserved_subnet_ip" {
      ...}
   ...
  }
  Plan: 5 to add, 0 to change, 0 to destroy. terraform apply -auto-approve ... google_compute_network.terra_vpc: Creating... google_compute_firewall.web-server: Creating... google_compute_subnetwork.terra_sub: Creating... google_compute_address.internal_reserved_subnet_ip: Creating... google_compute_instance.terra_instance: Creating... Apply complete! Resources: 5 added, 0 changed, 0 destroyed. Outputs: vpc_name = "terra-vpc" Subnet_Name = "terra-sub" Subnet_CIDR = "192.168.10.0/24" fire_wall_rules = toset([{... "ports" = tolist([ "description" = "RDP-HTTP-HTTPS ingress trafic" "destination_port_ranges" = toset([ "80", "3389", "443", "3389",]) ] hostname = "terrahost.brokedba.com" project = "brokedba2000" private_ip = "192.168.10.51" public_ip = "35.227.81.2" SSH_Connection = "ssh connection to instance TerraCompute ==> sudo ssh -i ~/id_rsa_gcp centos@35.227.81.2"

Now let’s launch our CENTOS7 vm using terraform apply (I left a map of different OS ids in the variables.tf you can choose from)

$ terraform apply -auto-approve
...
google_compute_network.terra_vpc: Creating...
google_compute_firewall.web-server: Creating...
google_compute_subnetwork.terra_sub: Creating...
google_compute_address.internal_reserved_subnet_ip: Creating...
google_compute_instance.terra_instance: Creating...
Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:
vpc_name = "terra-vpc"
Subnet_Name = "terra-sub"
Subnet_CIDR = "192.168.10.0/24"
fire_wall_rules = toset([
 {…
  "ports" = tolist([
  "description" = "RDP-HTTP-HTTPS ingress trafic"
  "destination_port_ranges" = toset([
  "80",
  "3389",
  "443",
  "3389",])
]
hostname = "terrahost.brokedba.com"
project = "brokedba2000"
private_ip = "192.168.10.51"
public_ip = "35.227.81.2"
 SSH_Connection = "ssh connection to instance TerraCompute ==> sudo ssh -i ~/id_rsa_gcp centos@35.227.81.2"

Once the instance is provisioned, juts copy the public IP address(i.e 52.191.26.102 ) in chrome and Voila!
You can also tear down this configuration by simply running terraform destroy from the same directory

Tips

You can fetch any of the specified attributes in outputs.tf using terraform output command i.e:

$ terraform output SSH_Connection
ssh connection to instance TerraCompute ==> sudo ssh -i ~/id_rsa_gcp centos@ ’public_IP’

Terraform Console: Although Terraform is a declarative language, there are still myriads of functions you can use to process strings/number/lists/mappings etc. There is an excellent all in one script with examples of most terraform functions >>here
I added cloud-init files for different distros you can play with by adapting var.user_data & var.OS

CONCLUSION

We have demonstrated in this tutorial how to quickly deploy a web server instance using terraform in GCP and leverage Cloud-init (Startupscript) to configure the VM during the bootstrap .
We had to hardcode the projectId although it’s embedded in the config credentials (key file) which make it tedious and rigid
Remember that all user attributes in this exercise can be modified in the variables.tf file.
Route table and internet gateway didn’t need to be created
Improvement: Validate that the startup script works for windows too. Another improvement can be reached in terms of the display of the security rules usingformatlist . stay tuned

Originally published at https://eclipsys.ca

Fixes for Virtualbox errors due to latest Windows 10 updates

CloudDude — Sun, 02 Aug 2020 23:14:58 +0000

Fixes for Vagrant/Virtualbox errors due to latest Windows 10 updates

"Failed to create the host-only adapter" and "VT-x not available"

It is always hard to keep up with Operating systems upgrades and patches for software companies, not only vagrant or Virtualbox. But if there was an OS that could make it even worse for us — end-users — that would definitely be Windows 10. Microsoft never ceases to amaze us, so much their updates side effects can be dumbfounding.

Ok, let’s cut the Vendetta and show the workarounds for these unfortunate UX Sucker punches instead.

1- Failed to create the host-Only adapter

A Host-Only adapter can usually be created from virtualbox GUI (see below) or from VboxManage.exe call as admin (manually or from an external app).

Note: I could still do it from vagrant before.

Create a Host Only adaptor from the vbox GUI

Symptoms

In my case I just recall applying windows 10 (v1909) updates few days ago. But when I started running my vagrant build yesterday, I then bumped into this unsavory (yet so insightful 😛) error from the get go.

Stderr: 0%...
Progress state: E_FAIL
VBoxManage.exe: **error:**  **Failed to create the host-only adapter**
VBoxManage.exe: error: Operation canceled by the user
VBoxManage.exe: error: Details: **code E\_FAIL (0x80004005)**, component VirtualBoxWrap, interface IVirtualBox
VBoxManageHostonly.cpp

ENVIRONMENT

OS: Windows 10 Home

Update: July 14, 2020 — KB4565483 (1903 OS Builds 18362.959 and 1909 18363.959)

Vagrant version : Vagrant 2.2.7

Vagrant provider : VirtualBox

VirtualBox version : 6.1

Workaround

You will have to temporarily change the user account Control setting. That way the vagrant background task won’t hang then crash as windows won’t show the confirmation pop up.

**Actions

Type UAC in the search field on your taskbar and click Change User Account Control settings.
- Turn UAC off by dragging the slider down to Never notify and click ** OK.

Disable notification when apps make changes to the system

Note: ** Please remember to Re-Enable the notification once your vagrant build is provisioned.**

2- VT-x is not available

After solving the earlier issue, I tried to run the vagrant up again but here comes another error message in my output.

Symptoms

error seems to indicate that the virtualization capabilities were not detected

... The command and stderr is shown below.
Command: [" **startvm**", "6cef1e57-d4d3-4633-a122-1be55e990eec", "--type", " **headless**"]

Stderr: VBoxManage.exe: **error: VMMR0\_DO\_NEM\_INIT\_VM failed** : **VERR\_NEM\_MISSING\_KERNEL\_API\_2**.
VBoxManage.exe: error: **VT-x is not available (VERR\_VMX\_NO\_VMX)**
VBoxManage.exe: error: Details: **code E\_FAIL (0x80004005)**, component ConsoleWrap, interface IConsole

Workaround

ThereforeVT-x is a set of intel CPU instructions that include hardware virtualization features which accelerate virtual machines. This must be enabled at BIOS level. Coincidentally, I also had to update my laptop BIOS the same week, but it’s not what caused the error.

Windows is known to automatically disable VT-x if Hyper-V is active. In my case I am sure the new updates have enabled it back. You will, therefore, have to unset that again.

Actions

Disable Hyper-V using the below Command and reboot your system

PS C:\Users\brokedba> **bcdedit** /set hypervisorlaunchtype **off**
 The operation completed successfully.

Conclusion

We just confirmed through these small yet annoying issues that windows updates aren’t that harmless. Therefore, It’s wise to test your favorite applications functionalities after each major patch. Hopping this will help those who’ll face the same errors with vagrant on windows 10.