Streamline Your AWS Asset Inventory with Automated Discovery and Reporting

Praveen HA — Mon, 03 Jun 2024 07:23:41 +0000

Managing resources across multiple AWS accounts and regions can be a daunting task, especially as your cloud infrastructure grows. To simplify this process, I have developed an AWS Inventory Discovery tool that scans all your AWS accounts across all regions and compiles a comprehensive, searchable HTML report. In this blog, I'll walk you through the features, benefits, and technical details of this solution.

Introduction
As organizations scale their use of AWS, keeping track of resources scattered across various accounts and regions becomes increasingly challenging. Manual inventory management is not only time-consuming but also prone to errors. This is where the AWS Inventory Discovery tool comes in.

Features
Comprehensive Scanning: The tool scans all your AWS accounts across all regions, ensuring no resource is left unaccounted for.
Detailed Reporting: Generates a detailed HTML report that lists all discovered resources, including their ARNs and regions.
Searchable Interface: The HTML report includes a search functionality, making it easy to find specific resources quickly.
Visual Representation: Includes pie charts to provide a visual summary of your resources, helping you quickly understand the distribution of resources across various dimensions.
User-Friendly: The generated report is easy to navigate, allowing users to drill down into resource details with just a few clicks.
Benefits
Improved Visibility: Gain a clear overview of all your AWS resources across accounts and regions.
Time Savings: Automate the discovery process, freeing up time for your team to focus on more critical tasks.
Error Reduction: Minimize the risk of overlooking resources or making mistakes in manual inventories.
Enhanced Security: Quickly identify and manage resources to ensure compliance with security policies and best practices.
How It Works
Prerequisites
Before you begin, ensure you have the following set up:

AWS CLI: Install the AWS Command Line Interface (CLI).

pip install awscli
AWS Credentials: Export your AWS profile or access keys and secrets, and specify the region.
export AWS_PROFILE=your-aws-profile

export AWS_ACCESS_KEY_ID=your-access-key-id
export AWS_SECRET_ACCESS_KEY=your-secret-access-key
export AWS_DEFAULT_REGION=your-default-region`

#!/bin/bash

AccountID="<YOUR_AWS_ACCOUNT>"
regions=$(aws ec2 describe-regions --query "Regions[].RegionName" --output text)
#regions=(ap-south-1 eu-north-1  eu-west-3  eu-west-1  ap-northeast-3  ap-northeast-2  ap-northeast-1  ca-central-1  sa-east-1   ap-southeast-1  ap-southeast-2  eu-central-1  us-east-1  us-east-2  us-west-1  us-west-2 )

# Initialize arrays to store data
declare -a summary_data=()
declare -a detailed_data=()

# Fetch resources from all regions
for region in $regions; do
    # Fetch resource ARNs and process them
    data=$(aws --region $region resourcegroupstaggingapi get-resources | jq -r '.ResourceTagMappingList[].ResourceARN' | \
    awk -v region=$region -F '[:/]' '
        {
            resourceType = "unknown"
            if ($3 == "s3") {
                resourceType = "bucket"
            } else if ($3 == "sns") {
                resourceType = "topic"
            } else {
                resourceType = $6  
            }

            # Construct a unique key for each service-resource pair
            pair = $3 ":" resourceType
            count[pair]++
        }
        END {
            for (pair in count) {
                split(pair, s, ":")  # Split the pair back into service and resource
                service = s[1]
                resource = s[2]
                print "{\"service\":\"" service "\", \"resource\":\"" resource "\", \"count\":" count[pair] "}"
            }
        }
    ')

    # Append to summary_data
    summary_data+=($data)

    # Fetch detailed resource information
    detailed_info=$(aws resourcegroupstaggingapi get-resources --region $region | jq -r --arg region $region '.ResourceTagMappingList[] | {Resource_Arn: .ResourceARN, Region: ($region // "Global"), Detailed_Service: (.ResourceARN | split(":")[2])}')

    # Append to detailed_data
    detailed_data+=($detailed_info)
done

# Process summary data
summary_json=$(printf "%s\n" "${summary_data[@]}" | jq -s 'group_by(.service, .resource) | map({service: .[0].service, resource: .[0].resource, count: map(.count) | add})')
total_resources=$(echo "$summary_json" | jq 'map(.count) | add')

# Process detailed data
detailed_json=$(printf "%s\n" "${detailed_data[@]}" | jq -s .)

# Generate HTML
cat <<EOF > index.html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>AWS Resources</title>
    <style>
        body {
            font-family: Arial, sans-serif;
            background-color: #f4f4f9;
            margin: 0;
            padding: 20px;
        }
        h1 {
            color: #333;
        }
        #searchBox, #searchBox2 {
            width: 100%;
            padding: 10px;
            margin-bottom: 20px;
            box-sizing: border-box;
            font-size: 16px;
        }
        table {
            width: 100%;
            border-collapse: collapse;
        }
        th, td {
            border: 1px solid #ddd;
            padding: 8px;
            text-align: left;
        }
        th {
            background-color: #4CAF50;
            color: white;
        }
        tr:nth-child(even) {
            background-color: #f2f2f2;
        }
        tr:hover {
            background-color: #ddd;
        }
        .total-count {
            font-size: 18px;
            margin: 20px 0;
        }
        #chartContainer {
            width: 50%;
            margin: 0 auto;
        }
    </style>
</head>
<body>
    <h1>AWS Resources for Account ID: $AccountID</h1>
    <div class="total-count">Total Resources: $total_resources</div>
    <input type="text" id="searchBox" onkeyup="filterTable()" placeholder="Search for services..">
    <table id="resourcesTable">
        <thead>
            <tr>
                <th>Service</th>
                <th>Resource</th>
                <th>Count</th>
            </tr>
        </thead>
        <tbody>
        </tbody>
    </table>

    <h1>Detailed AWS Resources</h1>
    <input type="text" id="searchBox2" onkeyup="filterTable2()" placeholder="Search for resources..">
    <table id="detailedResourcesTable">
        <thead>
            <tr>
                <th>Resource ARN</th>
                <th>Region</th>
                <th>Detailed Service</th>
            </tr>
        </thead>
        <tbody>
        </tbody>
    </table>

    <div id="chartContainer">
        <canvas id="resourceChart"></canvas>
    </div>

    <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>
    <script>
        const data = $summary_json;
        const resourceData = $detailed_json;

        function loadTableData(data) {
            const tableBody = document.getElementById('resourcesTable').getElementsByTagName('tbody')[0];
            tableBody.innerHTML = "";

            data.forEach(item => {
                let row = tableBody.insertRow();
                let cellService = row.insertCell(0);
                let cellResource = row.insertCell(1);
                let cellCount = row.insertCell(2);

                cellService.innerHTML = item.service;
                cellResource.innerHTML = item.resource;
                cellCount.innerHTML = item.count;
            });

            updateChart(data);
        }

        function loadDetailedTableData(resourceData) {
            const tableBody = document.getElementById('detailedResourcesTable').getElementsByTagName('tbody')[0];
            tableBody.innerHTML = "";

            resourceData.forEach(item => {
                let row = tableBody.insertRow();
                let cellArn = row.insertCell(0);
                let cellRegion = row.insertCell(1);
                let cellService = row.insertCell(2);

                cellArn.innerHTML = item.Resource_Arn;
                cellRegion.innerHTML = item.Region;
                cellService.innerHTML = item.Detailed_Service;
            });
        }

        function filterTable() {
            const searchValue = document.getElementById('searchBox').value.toLowerCase();
            const filteredData = data.filter(item => item.service && item.service.toLowerCase().includes(searchValue));
            loadTableData(filteredData);
        }

        function filterTable2() {
            const searchValue = document.getElementById('searchBox2').value.toLowerCase();
            const filteredResourceData = resourceData.filter(item => 
                item.Resource_Arn.toLowerCase().includes(searchValue) ||
                item.Region.toLowerCase().includes(searchValue) ||
                item.Detailed_Service.toLowerCase().includes(searchValue)
            );
            loadDetailedTableData(filteredResourceData);
        }

        function updateChart(data) {
            const services = data.reduce((acc, item) => {
                acc[item.service] = (acc[item.service] || 0) + item.count;
                return acc;
            }, {});

            const labels = Object.keys(services);
            const counts = Object.values(services);

            const ctx = document.getElementById('resourceChart').getContext('2d');
            new Chart(ctx, {
                type: 'pie',
                data: {
                    labels: labels,
                    datasets: [{
                        data: counts,
                        backgroundColor: [
                            '#FF6384', '#36A2EB', '#FFCE56', '#4CAF50', '#FF9800', '#9C27B0', '#00BCD4'
                        ]
                    }]
                },
                options: {
                    responsive: true,
                    plugins: {
                        legend: {
                            position: 'top',
                        },
                        title: {
                            display: true,
                            text: 'Resource Distribution by Service'
                        }
                    }
                }
            });
        }

        document.addEventListener('DOMContentLoaded', () => {
            loadTableData(data);
            loadDetailedTableData(resourceData);
        });
    </script>
</body>
</html>
EOF

AMI Age Calculator of Running AWS EC2 Instances and Generate CSV Report

Praveen HA — Sun, 01 Oct 2023 10:32:51 +0000

Instances running on outdated AMIs pose a security risk. Keeping our infrastructure secure requires us to regularly update these instances with the latest AMIs. Manually identifying and updating instances can be a time-consuming process, especially in large AWS accounts.

How does the automation work?

Our automation script leverages AWS CLI and the power of scripting to identify instances using older AMIs automatically.
Account and Region Selection: We can specify the AWS account ID and region you want to scan when executing the Jenkins Job.

AMI Age Assessment: The script checks every instance in the selected account and region and determines the age of the associated AMI based on its creation date.

Automated Reporting: It generates a CSV report with details of instances using AMIs that are 4 months or older. This report includes the Account ID, Region, Instance ID, AMI ID, and AMI Age (Months).

Actionable Insights: With this report, we can easily identify which instances require updates. This information can be used to patch instances with the latest AMIs, ensuring they remain secure and optimized.

How can we use it?

Running this automation is simple. we can execute the script filling the AWS account ID and region as parameters. script will do the rest providing us with a detailed report.

What are the benefits?

Enhanced Security: We can proactively identify and patch instances with older AMIs, reducing security vulnerabilities.

Efficiency: Manual instance assessment is time-consuming, but this automation speeds up the process, allowing us to focus on other critical tasks.

Consistency: By automating this process, we ensure that all instances are assessed uniformly and regularly.

`#!/bin/bash

Initialize variables with default values

ACCOUNTID=""
REGION=""

OUTPUT_CSV="$ACCOUNTID-$REGION-ami_age_report.csv" # Define the CSV file name
rm -rf $OUTPUT_CSV

Parse command line options

while getopts "a🅱️" option; do
case $option in
a) ACCOUNTID=${OPTARG} ;;
b) REGION=${OPTARG} ;;
*) echo "usage: $0 [-a ACCOUNTID] [-b REGION]" >&2
exit 1 ;;
esac
done

List instances and AMI IDs in the specified region

instances_json=$(aws ec2 describe-instances --region "$REGION" --query 'Reservations[].Instances[].[InstanceId,ImageId]' --output json) # For local

Get the current timestamp

current_time=$(date -u +%s)

Initialize the CSV file with headers

echo "AccountID,Region,InstanceID,AMIID,AMIAge (months)" > "$OUTPUT_CSV"

Iterate through instances and append to the CSV file

for row in $(echo "$instances_json" | jq -r '.[][] | @base64'); do
_jq() {
echo "$row" | base64 --decode | jq -r "$1"
}

instance_id=$(_jq '.[0]')
ami_id=$(_jq '.[1]')

# Get the creation date of the AMI in the specified region
ami_create_time=$(aws ec2 describe-images --region "$REGION" --image-ids "$ami_id" --query 'Images[0].CreationDate' --output text) # For local

# Calculate the age in months based on the AMI creation time
ami_create_timestamp=$(date -u -d "$ami_create_time" +%s)
months_diff=$(( (current_time - ami_create_timestamp) / 60 / 60 / 24 / 30 ))

# Check if the AMI is older than 4 months
if [ "$months_diff" -ge 4 ]; then
# Append to the CSV file
echo "$ACCOUNTID,$REGION,$instance_id,$ami_id,$months_diff" >> "$OUTPUT_CSV"
fi
done

echo "CSV report saved to $OUTPUT_CSV"
cat $OUTPUT_CSV`

Forem: Praveen HA