Forem: Tom Granot

Finding and Remediating Idle AWS Databases Made Easy

Tom Granot — Tue, 08 Apr 2025 17:17:10 +0000

It’s easy to overlook idle databases in AWS. Spun up in a hurry for testing, migration, or “just in case” scenarios, they often outlive their purpose and quietly accumulate charges in the background.

This isn’t just about wasted money—though that’s a major factor. It’s also about infrastructure sprawl, increased attack surface, and the risk of unintentionally deleting something still critical because no one remembers what it was for.

Over time, this clutter adds friction to your operations. Auditing becomes harder, deployment pipelines get noisier, and cost forecasting gets skewed by ghost workloads.

What is an Idle AWS Database?

An idle AWS database is any database instance—RDS, Aurora, Redshift, or others—that shows little to no meaningful engagement over time. That means no active connections, low or flatline read/write IOPS, and negligible CPU usage. It's not about a single moment of quiet—it's about a sustained pattern of inactivity confirmed by metrics and usage history.

These instances often originate from valid use cases. A developer might create a new RDS instance for performance testing and forget to decommission it. A data team might launch a Redshift cluster for a one-time warehouse import. Or a migration project might leave behind a “just in case” snapshot that never gets used. What begins as a useful resource turns into a silent cost center.

Idle doesn’t always mean harmless. These forgotten databases consume compute and storage; they also carry operational and security risks. When left unmonitored, they miss patches, drift from configuration baselines, and become more vulnerable to misconfigurations or open endpoints. Multiply that across regions, environments, or accounts, and the scale of the problem grows fast.

The key characteristics of an idle instance usually follow a clear trend:

Zero or near-zero active connections: Measured through CloudWatch’s DatabaseConnections metric across a 7–31 day span.
Minimal IOPS activity: ReadIOPS and WriteIOPS metrics show little to no disk activity, often under 20 operations per day.
Consistently low CPU usage: Less than 5% average CPU utilization over time is a strong indicator of inactivity—especially for provisioned instances, where you're paying for compute regardless.

What makes these databases tricky is that they don’t always show up when scanning for errors or failures. They’re not broken. They're just not doing anything. Unless you're specifically looking for them, they simply blend into the noise. That’s why a structured approach to AWS idle database management, like the one used at Tom Granot, helps teams identify and clean up these costly, hidden inefficiencies without risking disruption.

Why Do Idle Databases Matter?

The biggest offender here is cost. Idle RDS instances, for example, continue charging for compute, provisioned IOPS, and storage—even if the database hasn't handled a single query in weeks. A multi-AZ SQL Server Enterprise instance left running for 90 days can quietly rack up over $10,000 in charges. These aren’t rare edge cases—they’re the result of default behaviors like forgetting to disable instances after a migration, or leaving replica clusters online “just in case.”

Beyond cost, idle databases introduce unnecessary decision friction across engineering teams. During remediation or infrastructure reviews, teams get stuck triaging long lists of resources no one remembers provisioning. Old project databases hang around, cluttering dashboards and inflating usage graphs. CI/CD pipelines that spin up temporary databases often forget to tear them down, adding even more noise. This mess slows down operations, introduces confusion in ownership, and makes enforcing resource tagging standards harder.

Attack surface is another issue. Idle doesn’t mean isolated—many of these databases still have active endpoints, permissive security groups, or unused IAM roles attached. Without owners watching them, they miss routine hardening: no patching, no version upgrades, and often no alarms. That’s how they become convenient entry points for attackers or compliance violations waiting to happen. In one real-world audit, a previously decommissioned staging database was discovered still accepting external connections—with default credentials.

Common Types of Idle AWS Databases

Idle databases tend to follow predictable origin stories. Most of them weren’t mistakes at the time—they were part of legitimate workflows that just never had a proper exit. Understanding how they get there is key to designing reliable cleanup processes.

Short-Lived Environments That Overstay

Temporary environments are a known source of idle sprawl, but the root issue often lies in automation gaps. CI/CD pipelines may provision new RDS instances for integration tests or staging deployments, yet fail to include teardown logic on failure paths or timeouts. These instances don’t just persist—they accumulate silently across accounts and regions, especially when the automation lacks tagging or TTL enforcement.

Another overlooked source comes from internal training or onboarding exercises. Teams spin up realistic environments to teach new hires or simulate customer setups, but unless someone owns the cleanup step, those environments become permanent residents. These databases often look like legitimate workloads—especially if they mimic production naming conventions—and are rarely flagged until a cost spike triggers a deeper review.

Post-Migration Leftovers and Dormant Archives

Infrastructure migrations tend to leave echoes: clusters, read replicas, or legacy endpoints that were supposed to be temporary but got lost in the shuffle. Often, the switch to a new instance happens cleanly, but the old one remains available out of habit or as a fallback that no one wants to touch. These drift further into obscurity when they’re excluded from IaC or drift detection tooling, making them invisible to standard audits.

Then there are the archival workloads—datasets that serve compliance or audit use cases but are no longer part of day-to-day operations. These aren't abandoned, but they’re grossly overprovisioned. For example, an RDS instance storing quarterly financial logs might idle at under 1% CPU for months, yet still runs on a db.m5.2xlarge tier. Without scheduled shutdowns or a shift to more storage-efficient models, they quietly burn through budget while doing almost nothing.

Where Do Idle Databases Usually Live?

Idle databases tend to hide in the edges of your infrastructure—the places no one checks unless something breaks. Non-production environments lead the pack: staging, QA, sandbox accounts. These environments exist to mirror production closely enough for testing, but they often evolve organically without consistent controls. Feature branches, parallel testing workflows, or hotfix validation setups leave behind full-stack environments that no longer map to active code. Over time, these “temporary” databases blend into the noise because no one knows if they’re still important—or who to ask.

Automated systems, ironically, are another source of debris. Infrastructure-as-code tools, scheduled builds, or ephemeral environments generated on the fly often lack teardown hooks that execute reliably in every failure path. Over time, you’ll find RDS or Redshift instances created for one-off data loads, dry runs, or schema validation tests that never completed—but the instance stuck around. These workloads are rarely tagged with consistent metadata, and even when they are, they’re often excluded from cleanup policies due to lack of ownership mapping.

Multi-region activity introduces another layer of complexity. Organizations experimenting with latency optimization, geo-redundancy, or cross-border compliance often deploy workloads in secondary or tertiary regions. These databases serve a purpose for a few days—but months later, they’re still online in regions like me-south-1 or ap-northeast-3, quietly accruing storage and networking costs. Since most billing dashboards default to primary regions, and many teams work within isolated scopes, these idle workloads often escape detection until someone performs a CUR deep dive or a cross-region resource inventory.

Then there’s the forgotten legacy tier—databases linked to deprecated services or abandoned proof-of-concepts. These are rarely part of modern CI/CD or infrastructure automation. Nobody’s watching the metrics, no alerts are configured, and no one has logged in for months. They’re still online because they’ve become “too old to touch”—deleting them feels risky without documentation, and the original owner probably left the organization or moved on to a different team. So they sit, draining budget and widening your security perimeter.

How to Find and Safely Remediate Idle Databases on AWS

Identifying idle databases starts with surfacing underutilized infrastructure across your AWS footprint. Use the Cost and Usage Report (CUR) to isolate long-running database instances with consistent spend. From there, pull runtime metrics like CPUUtilization, FreeableMemory, and NetworkThroughput via CloudWatch to detect usage anomalies or consistent inactivity patterns over time. These aren't just raw stats—they form the behavioral fingerprint of whether an instance is doing real work or just existing.

To accelerate this process, use CLI queries to capture a snapshot of current state:

aws rds describe-db-instances \
  --query "DBInstances[*].{DB:DBInstanceIdentifier,Class:DBInstanceClass,State:DBInstanceStatus,Created:InstanceCreateTime}" \
  --output table

Then, compare that against CloudWatch data collected over a 14–31 day window:

aws cloudwatch get-metric-statistics \
  --metric-name CPUUtilization \
  --start-time 2024-05-01T00:00:00Z \
  --end-time 2024-05-31T23:59:59Z \
  --period 86400 \
  --namespace AWS/RDS \
  --statistics Average \
  --dimensions Name=DBInstanceIdentifier,Value=my-db

Instances with low CPU and negligible network activity—especially those with stable or declining memory usage—are strong idle candidates.

Validate Business Context Before Acting

Before selecting a remediation path, establish the current relevance of the instance. Use existing tags (Environment, Project, Owner, etc.) to identify responsible teams. If tags are missing or stale, use IAM access patterns, CloudTrail events, or recent login history to trace back recent activity. You can also leverage AWS Resource Groups to list assets tied to specific teams or projects.

When ownership is unclear, append a review tag to the instance:

aws rds add-tags-to-resource \
  --resource-name arn:aws:rds:us-east-1:123456789012:db:my-db \
  --tags Key=ReviewStatus,Value=NeedsVerification

This creates an audit trail and signals to others that the instance is under evaluation. Once ownership is confirmed or dismissed, update the tag accordingly.

Choose the Right Remediation Strategy

Each idle instance should follow a remediation path based on cost, risk, and future accessibility.

Delete with Backup: When the instance is confirmed unused and not tied to compliance retention, create a snapshot and remove it entirely:

  aws rds create-db-snapshot \
    --db-instance-identifier my-db \
    --db-snapshot-identifier my-db-final-snapshot

Then:

  aws rds delete-db-instance \
    --db-instance-identifier my-db \
    --final-db-snapshot-identifier my-db-final-snapshot

Retain with Adjustments: For rarely accessed but necessary databases, reduce instance size or switch to burstable classes like db.t4g.micro to minimize costs:

  aws rds modify-db-instance \
    --db-instance-identifier my-db \
    --db-instance-class db.t4g.micro \
    --apply-immediately

Suspend or Pause: For Aurora Serverless clusters, configure auto-pause to suspend compute when idle:

  aws rds modify-db-cluster \
    --db-cluster-identifier audit-logs-cluster \
    --scaling-configuration MinCapacity=0,AutoPause=true,SecondsUntilAutoPause=3600

These options give you flexibility—whether you need to decommission, minimize, or preserve the database with reduced cost exposure.

Lock Down What You Keep

Idle databases left online must be isolated from unnecessary access. Start by limiting network exposure:

aws rds modify-db-instance \
  --db-instance-identifier my-db \
  --vpc-security-group-ids sg-00000000 \
  --apply-immediately

Attach a security group that denies all inbound connections except from trusted internal ranges or jump boxes. Then remove any IAM roles, Lambda triggers, or SNS topics previously tied to the database. If the database must remain online for archival or reference, add a Quarantine tag and restrict its visibility in dashboards and monitoring tools.

Automate Detection and Review

For ongoing idle detection, build a Lambda function that runs daily and queries metrics across all RDS instances. Use thresholds like < 1% CPU and < 5 connections over 14 days to flag candidates. When flagged, tag them with IdleCandidate=true and send alerts via SNS or Slack for manual review.

You can also integrate with EventBridge to trigger evaluations on instance creation, ensuring new databases follow lifecycle policies from day one. Pair this with a runbook that outlines remediation protocols, backup policies, and escalation paths. That way, cleanup isn’t a one-time effort—it becomes part of the infrastructure lifecycle.

1. Inventory Your Databases

Start with coverage. Not partial, not “just the prod account”—full-region, full-account visibility. The point isn’t just to surface what’s online, it’s to expose what’s quietly burning budget with no clear owner. Begin with your AWS Organizations account structure. Use list-accounts with sts assume-role to pull RDS inventory from every child account. This way, you’re not blind to idle resources running in sandbox or legacy projects someone forgot to shut down three quarters ago.

Here’s how to list RDS instances across multiple accounts and regions using a loop:

for account in $(aws organizations list-accounts --query 'Accounts[*].Id' --output text); do
  creds=$(aws sts assume-role --role-arn arn:aws:iam::$account:role/OrgAuditAccess --role-session-name audit-session)
  export AWS_ACCESS_KEY_ID=$(echo $creds | jq -r .Credentials.AccessKeyId)
  export AWS_SECRET_ACCESS_KEY=$(echo $creds | jq -r .Credentials.SecretAccessKey)
  export AWS_SESSION_TOKEN=$(echo $creds | jq -r .Credentials.SessionToken)

  for region in $(aws ec2 describe-regions --query 'Regions[*].RegionName' --output text); do
    echo "Account: $account | Region: $region"
    aws rds describe-db-instances --region $region \
      --query "DBInstances[*].{DB:DBInstanceIdentifier,Type:Engine,Class:DBInstanceClass,Status:DBInstanceStatus}" \
      --output table
  done
done

This gives you a multi-account, multi-region view of your database landscape. Once you’ve got the raw list, push it into a central repository—DynamoDB, Athena, or even a tagged S3 CSV—so you can filter by engine type, region, or creation date later. Tracking it in a living system makes it easier to attach review states over time.

Cross-Reference Metrics Over Time

Once inventory is established, focus on behavior. You’re not just looking for what exists—you’re looking for databases that consistently do nothing. Use AWS Cost Explorer to flag RDS instances with flat spend trends over 30+ days. Then layer in CloudWatch metrics like DatabaseConnections, ReadIOPS, and WriteIOPS to validate inactivity.

Instead of reviewing one instance at a time, automate the metric pull using get-metric-data. This lets you batch-query multiple instances in a single request:

aws cloudwatch get-metric-data \
  --metric-data-queries file://metric-queries.json \
  --start-time $(date -u -d '30 days ago' +%Y-%m-%dT00:00:00Z) \
  --end-time $(date -u +%Y-%m-%dT00:00:00Z) \
  --region us-east-1

Where metric-queries.json contains:

[
  {
    "Id": "readiops1",
    "MetricStat": {
      "Metric": {
        "Namespace": "AWS/RDS",
        "MetricName": "ReadIOPS",
        "Dimensions": [
          {
            "Name": "DBInstanceIdentifier",
            "Value": "my-db-instance"
          }
        ]
      },
      "Period": 86400,
      "Stat": "Sum"
    },
    "ReturnData": true
  }
]

Batching metrics like this helps you identify low-activity patterns at scale, without drowning in one-off CLI calls per instance. You’ll start to see which ones never spike, never connect, and never move data—your top idle suspects.

Capture Fallbacks Before You Touch Anything

Before tagging or terminating anything, document the state. Store metadata like engine version, allocated storage, parameter groups, and security group associations. This makes rollback easier if someone flags the instance post-cleanup. For retention-sensitive environments, record your findings in a separate audit log and attach a flag like IdleCandidate=True or RemediationStatus=PendingReview.

If deletion is on the table, use a snapshot strategy that includes naming conventions and timestamping:

aws rds create-db-snapshot \
  --db-instance-identifier my-db-instance \
  --db-snapshot-identifier idle-snap-my-db-instance-$(date +%Y%m%d)

Don’t just rely on snapshots for backup—use them to prove due diligence. When you’re working across teams or regulated environments, a standardized naming scheme for snapshots and tags helps signal intent and avoids accidental overlaps with active workloads.

2. Evaluate Each Database’s Role

After surfacing idle candidates, the next step is confirming whether they still matter. Metrics show you what’s happening under the hood—but they don’t tell you whether the database still holds business value. That context lives in ownership, integrations, and historical decisions made outside the console.

When metadata is missing or unreliable, trace the origin using CloudTrail’s creation logs. This gives you a timestamp and the IAM identity that created the instance—often enough to connect the dots. Run this to surface the creator:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateDBInstance \
  --query 'Events[*].{Time:EventTime,User:Username,Resource:Resources[0].ResourceName}' \
  --max-results 20

If the user is active, a short message asking whether the database supports anything critical can resolve ambiguity fast. If not, look for context in provisioning pipelines, resource policies, or existing role bindings. You’ll often find clues in IAM permissions, ECS task definitions, or Lambda environment variables that reference the database identifier.

Determine Usage Through Connected Systems

Rather than guessing at purpose, check whether the database appears in active configurations. You can scan Secrets Manager and Parameter Store for stored credentials that are still in use by apps or services:

aws secretsmanager list-secrets \
  --query "SecretList[?contains(Name, 'my-db')].Name"

aws ssm describe-parameters \
  --query "Parameters[?contains(Name, 'my-db')].Name"

Also look at task queues, data pipelines, or scheduled jobs tied to the database. For example, a Redshift cluster may not see daily queries but could be processing monthly reports via a Step Functions state machine or AWS Glue job. In those cases, you’re dealing with low-frequency usage—not true idleness.

Apply Clear, Future-Friendly Tagging

Once you’ve confirmed the role—or lack thereof—tag the instance to document its status and avoid reevaluating it later. Use clear labels like Role=HistoricalReporting, RemediationStatus=IdleCandidate, or ReviewDate=2024-06-01 to give future teams context at a glance:

aws rds add-tags-to-resource \
  --resource-name arn:aws:rds:us-west-2:123456789012:db:legacy-db \
  --tags Key=Role,Value=LegacyData \
          Key=RemediationStatus,Value=NeedsApproval \
          Key=ReviewDate,Value=2024-06-01

If the database is still under review or lacks a clear owner, tag it accordingly and schedule a follow-up. This creates a paper trail—and avoids false positives the next time someone runs a cleanup script.

The goal here isn’t just to clean up one instance. It’s to embed context into your infrastructure so that future reviews move quickly, decisions are traceable, and idle database management becomes a lightweight, continuous process.

3. Confirm Data Retention Requirements

Before any remediation action—termination, downsizing, or archiving—you need to understand the data’s contractual or regulatory weight. Usage metrics don’t reveal whether a database contains GDPR-affected records, financial logs tied to tax filings, or archive content required by HIPAA. Removing an idle database without validating retention obligations exposes you to compliance violations, not just operational risk.

Start by reviewing your internal data classification policy, if one exists. Many organizations mandate different retention periods for PII, logs, analytical models, or system telemetry. If the database falls under a formal data governance category—like “Regulatory Financials” or “Customer Interaction Logs”—you’ll want to escalate the review to a compliance lead before taking any action. When no classification is available, fall back to metadata like schema names (audit_logs, transactions, etc.) or table row counts to infer its historical value.

Export for Long-Term Retention Without Keeping Infrastructure

If you confirm the data must remain accessible beyond the life of the instance, exporting it is a more durable and cost-efficient approach than keeping the instance running or relying solely on RDS snapshots. For example, you can export a snapshot to S3 in Parquet format, which supports downstream querying via Athena without spinning up compute:

aws rds export-db-snapshot \
  --db-snapshot-identifier audit-db-snap-20240601 \
  --source-engine mysql \
  --s3-bucket-name org-compliance-archive \
  --iam-role-arn arn:aws:iam::123456789012:role/RDSExportToS3 \
  --kms-key-id arn:aws:kms:us-east-1:123456789012:key/abc12345-6789-0123-4567-abcdef123456

This export strategy allows you to retain data in a queryable format, minimize storage costs, and avoid compute resources entirely. It’s especially useful for compliance teams that require access to historical data but don’t need a live database to retrieve it.

Put Lifecycle Controls Around Retained Data

Once exported, the data must follow structured lifecycle controls to avoid becoming the next generation of idle sprawl. Use S3 object tags and bucket lifecycle configurations to enforce expiration:

{
  "Rules": [
    {
      "ID": "ExpireComplianceDataAfter7Years",
      "Filter": {
        "Tag": {
          "Key": "RetentionPolicy",
          "Value": "Finance"
        }
      },
      "Status": "Enabled",
      "Expiration": {
        "Days": 2555
      }
    }
  ]
}

Couple this with versioning and access logging to create an auditable trail. If your organization uses centralized security tooling, integrate these exported records into your compliance vault or archival domain, and restrict access via resource-level IAM policies that reference specific tags like DataSensitivity=High or LegalHold=True.

Instead of just preserving data reactively, this approach embeds retention logic into the export process—ensuring archived databases follow the same rigor as live systems but without the operational overhead.

4. Apply the Best Remediation Strategy

Once you've finished auditing, tagging, and confirming the purpose of each idle database, the next step is choosing the right remediation path. Not every instance needs to be deleted—your decision should reflect the level of risk, the need for historical access, and the frequency of future use.

Full Termination with Lifecycle Controls

For databases that serve zero ongoing purpose and carry no retention requirements, complete removal is appropriate—but it still requires a structured approach. Before deletion, register the instance in your internal cleanup log and check for any downstream dependencies (e.g., secrets, parameter references, or IAM bindings that might still reference the database). Rather than relying on a single snapshot, consider exporting the database contents to S3 using the ExportDBSnapshot feature, especially if the data may need to be queried later via Athena or Redshift Spectrum.

aws rds export-db-snapshot \
  --db-snapshot-identifier legacy-db-snap-20240601 \
  --source-engine postgres \
  --s3-bucket-name longterm-archive-bucket \
  --iam-role-arn arn:aws:iam::123456789012:role/RDSExportRole \
  --kms-key-id arn:aws:kms:us-east-1:123456789012:key/abc123

After confirming the export, follow up with deletion and attach a record of the action to your central resource inventory or security audit trail.

Downsizing to Align with Intermittent Use

Some databases are technically idle but still serve low-frequency workloads—quarterly audits, monthly reports, or ad-hoc queries by internal teams. In those cases, full termination may be excessive. Instead, reduce the footprint to match actual usage. Choose instance classes with burstable capacity (t3, t4g) and evaluate whether storage throughput or IOPS provisioning can be scaled down. This tactic preserves availability while significantly reducing runtime costs.

For example, to update a database to a smaller, ARM-based instance class:

aws rds modify-db-instance \
  --db-instance-identifier archive-db \
  --db-instance-class db.t4g.micro \
  --storage-type gp3 \
  --apply-immediately

This method is particularly effective when paired with scheduled backups and a defined retention period, allowing the database to remain functional but cost-aligned.

Suspend or Quarantine for Low-Risk Retention

When deletion isn't an option and downsizing doesn’t go far enough—such as in compliance-heavy environments or when ownership is unclear—quarantine is a viable middle ground. This involves isolating the instance from the network, locking down IAM policies, and removing any triggers or downstream integrations. Instead of relying on auto-scaling or pause features alone, you enforce a state of intentional disconnection, reducing exposure without losing the data.

For Aurora Serverless v1 or v2 clusters, where pausing is supported, configure inactivity thresholds explicitly. This allows the cluster to suspend compute resources automatically while data remains intact.

aws rds modify-db-cluster \
  --db-cluster-identifier dormant-reporting-cluster \
  --scaling-configuration MinCapacity=1,MaxCapacity=2,AutoPause=true,SecondsUntilAutoPause=7200 \
  --apply-immediately

For engines that don’t support pause, simulate the effect by disabling inbound access and scheduling instance stops. Use Systems Manager Automation documents or EventBridge rules to shut down the database outside known access windows, reducing runtime costs without removing the instance entirely. This gives you a predictable, low-risk fallback plan while buying time to confirm long-term decisions.

5. Update Network and Access Configurations

Idle databases that remain online often carry legacy access paths—open ports, broad CIDR blocks, or deprecated roles—that were never revisited once the database stopped serving live traffic. These misconfigurations don’t raise alarms on their own but present quiet, persistent risks. If a database is inactive, its exposure should reflect that status with strict, intentional controls.

Start by identifying which security groups are still attached to databases flagged as idle. Use describe-db-instances to retrieve the associated security group IDs, then inspect the ingress rules:

aws ec2 describe-security-groups \
  --group-ids sg-0123456789abcdef0 \
  --query "SecurityGroups[*].IpPermissions"

Instead of just removing security groups, consolidate idle instances into a dedicated isolation group with no inbound rules defined—effectively dead-ending all network access:

aws ec2 create-security-group \
  --group-name rds-isolation-zone \
  --description "Blocked access for inactive RDS" \
  --vpc-id vpc-0abc123def456ghij

Apply it to the instance:

aws rds modify-db-instance \
  --db-instance-identifier dormant-db \
  --vpc-security-group-ids sg-0abc123def456ghij \
  --apply-immediately

This replaces any prior rules—public or private—with a locked-down profile. From a network perspective, the database becomes inert without requiring full deletion or snapshot restoration.

IAM roles and policies tied to idle databases often go unchecked. These roles may have been granted permissions like rds:StartDBInstance or rds:RestoreDBInstanceFromSnapshot months—or years—ago, and they persist long after their operational use ends. Use CloudTrail to pinpoint which IAM entities have interacted with the instance in the last 90 days:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=dormant-db \
  --max-results 50

If no recent activity is found, simulate the role’s permissions to see whether it still has access to sensitive RDS actions:

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/ObsoleteRDSRole \
  --action-names rds:StartDBInstance rds:RestoreDBInstanceFromSnapshot

For roles still permitted to act on idle databases, either remove the policy or modify the trust relationship to prevent future assumptions. For example, to explicitly deny assumptions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyObsoleteAccess",
      "Effect": "Deny",
      "Action": "sts:AssumeRole",
      "Principal": "*",
      "Condition": {
        "ArnEquals": {
          "aws:PrincipalArn": "arn:aws:iam::123456789012:role/ObsoleteRDSRole"
        }
      }
    }
  ]
}

Outside of network and IAM, configuration bloat can also linger in parameter and option groups. These often accumulate enabled features—like audit logs, minor version preferences, or legacy extensions—that no longer serve their original purpose. Instead of defaulting to a full reset, duplicate the current group, strip out unused settings, and reassign it to the instance:

aws rds modify-db-instance \
  --db-instance-identifier dormant-db \
  --db-parameter-group-name minimal-config \
  --apply-immediately

This approach reduces unnecessary complexity and aligns the database’s configuration with its current (or lack of) workload. It also makes future remediation simpler—leaner configurations mean fewer unknown variables when it’s time to archive, downsize, or delete.

6. Automate Ongoing Idle Detection

Manual reviews don’t scale—not across multiple teams, regions, or accounts. Automation ensures idle database detection becomes a continuous process, not a quarterly fire drill. By embedding scheduled metric evaluation into lightweight Lambda functions, you can catch drift before it becomes budget bloat, and remove guesswork from the equation entirely.

Start with a Lambda function that consumes instance IDs from a centralized inventory—stored in DynamoDB or fetched dynamically—and runs targeted metric queries via CloudWatch APIs. Instead of relying on fixed thresholds, implement configurable logic using environment variables or a lookup table to account for different environments and usage profiles (e.g., production vs. test). This flexibility allows you to tune for noise reduction without missing meaningful signals.

Here’s a basic implementation that uses a DynamoDB table to load instance configurations and supports adjustable thresholds per instance:

import boto3
import datetime
import os

cloudwatch = boto3.client('cloudwatch')
dynamodb = boto3.resource('dynamodb')
rds = boto3.client('rds')

def lambda_handler(event, context):
    table = dynamodb.Table(os.environ['INSTANCE_CONFIG_TABLE'])
    end = datetime.datetime.utcnow()
    start = end - datetime.timedelta(days=7)

    items = table.scan()['Items']
    for item in items:
        db_id = item['DBInstanceIdentifier']
        threshold_cpu = float(item.get('CPUThreshold', 2))
        threshold_conn = float(item.get('ConnThreshold', 1))

        conn_stats = cloudwatch.get_metric_statistics(
            Namespace='AWS/RDS',
            MetricName='DatabaseConnections',
            Dimensions=[{'Name': 'DBInstanceIdentifier', 'Value': db_id}],
            StartTime=start,
            EndTime=end,
            Period=86400,
            Statistics=['Average']
        )

        cpu_stats = cloudwatch.get_metric_statistics(
            Namespace='AWS/RDS',
            MetricName='CPUUtilization',
            Dimensions=[{'Name': 'DBInstanceIdentifier', 'Value': db_id}],
            StartTime=start,
            EndTime=end,
            Period=86400,
            Statistics=['Average']
        )

        conn_avg = sum(dp['Average'] for dp in conn_stats['Datapoints']) / len(conn_stats['Datapoints']) if conn_stats['Datapoints'] else 0
        cpu_avg = sum(dp['Average'] for dp in cpu_stats['Datapoints']) / len(cpu_stats['Datapoints']) if cpu_stats['Datapoints'] else 0

        if conn_avg < threshold_conn and cpu_avg < threshold_cpu:
            rds.add_tags_to_resource(
                ResourceName=item['DBInstanceArn'],
                Tags=[{'Key': 'IdleCandidate', 'Value': 'true'}]
            )
            print(f"Marked {db_id} as idle")

This version doesn’t assume one-size-fits-all logic. It adjusts based on the intended behavior of each instance and stores evidence in tagging, enabling downstream systems to take action—or escalate—without human intervention.

To extend this pipeline, integrate the function with an SNS topic or webhook that feeds into your internal ops tooling. Instead of plain alerts, push structured payloads that include instance metadata, thresholds breached, and time-series summaries. This gives reviewers the context they need to make decisions quickly without digging through metrics dashboards.

For long-term traceability, log all detection results and tagging actions to a dedicated S3 bucket or a time-partitioned Athena table. Include not just the instance ID and metric values but also the evaluation config that triggered the action. When teams review past decisions or investigate unexpected deletions, this audit log becomes the source of truth.

Capture your automation stack and detection logic in a shared runbook accessible to all teams involved in infrastructure lifecycle management. Include a decision tree for each remediation path, escalation contacts, and examples of valid idle patterns versus anomalies. When detection becomes embedded into your review cadence, idle cleanup turns from a one-off effort into a predictable, low-friction routine.

Reasons to Keep an Eye on AWS Idle Databases

Idle databases are easy to ignore—until they interfere with your ability to move fast. When your AWS inventory is bloated with unused resources, even simple tasks like running compliance scans or provisioning new environments take longer than they should. Visibility suffers, and so does the confidence that what you're deploying into is actually being used, secured, and maintained.

Unmonitored databases tend to operate outside of your standard guardrails. They might still be accessible from legacy IP ranges, tied to IAM roles no one remembers creating, or logging to buckets that are no longer monitored. One idle PostgreSQL instance left with unrestricted ingress for weeks can quietly sidestep your perimeter defenses—not because of negligence, but because no one knew it was still there.

Infrastructure That Doesn’t Fight Back

Clean infrastructure doesn’t just save money—it reduces resistance. When teams spin up new workloads or migrate existing ones, they aren’t forced to decipher what’s safe to reuse or what might break something. There’s no guessing if a database labeled "test-db-v2" is critical or disposable. By keeping idle databases out of the way, you prevent them from becoming blockers in future planning or sources of accidental regression.

As environments scale up, the risk of misclassifying resources grows. A database that supported a proof-of-concept last quarter might still sit on a large instance class, untouched but protected by deletion safeguards. Without a consistent way to surface and review these kinds of leftovers, capacity planning and cost forecasting get skewed—and so do your infrastructure decisions.

aws rds describe-db-instances \
  --query "DBInstances[?contains(DBInstanceIdentifier, 'poc') && @.DBInstanceStatus=='available']" \
  --output table

Use this to isolate candidate instances by naming convention and status—then evaluate whether they’ve had any active connections or IOPS in the last 30 days. It’s not just about finding what’s idle. It’s about knowing what’s safe to ignore—and what isn’t—before it affects your next deployment.

Tips on AWS Database Remediation

1. Tag Rigorously

Tagging is only useful if it’s opinionated and enforced. It's not enough to label something “dev” or “test”—your tags should declare intent. Tags like Owner or Purpose are just the start. Introduce tags that reflect lifecycle (TTL), cost center (BillingCode), and compliance scope (DataSensitivity). These tags help automation scripts decide what to ignore, what to archive, and what to shut down without involving a human.

For example, when provisioning a database, inject a tag set that answers three questions: who owns it, when does it expire, and what kind of data lives inside?

aws rds create-db-instance \
  --db-instance-identifier campaign-metrics \
  --db-instance-class db.t3.small \
  --engine postgres \
  --tags Key=Owner,Value=team.analytics \
          Key=TTL,Value=2024-09-30 \
          Key=DataSensitivity,Value=PII \
          Key=BillingCode,Value=MKT-Q3-2024

To enforce tagging at org level, use AWS Organizations tag policies to define allowed tag keys and value patterns. Combine that with Config rules that flag instances not matching policy. For high-sensitivity environments, pair tag-based access control with IAM condition keys to prevent untagged resources from being created altogether.

2. Embrace Snapshots

Snapshots are more than just a safety net—they’re a critical part of your idle database strategy. Instead of keeping dormant databases online “just in case,” convert them into snapshots that can be retained in low-cost storage tiers or exported to S3 in a queryable format. This gives teams access to historical data without incurring compute or high IOPS charges.

Use snapshot exports when the data needs to be queried periodically but doesn’t justify a running instance. For example, exporting a snapshot to S3 in Parquet format allows it to be queried directly with Athena:

aws rds export-db-snapshot \
  --db-snapshot-identifier campaign-metrics-final-snap \
  --source-engine postgres \
  --s3-bucket-name analytics-historical-data \
  --iam-role-arn arn:aws:iam::123456789012:role/RDSExportRole \
  --kms-key-id arn:aws:kms:us-east-1:123456789012:key/abc123xyz

For retention tracking, tag your snapshots with RetentionWindow and ArchiveStatus. Then use a scheduled Lambda function to delete anything beyond your defined policy. This keeps storage lean and aligns with internal data governance requirements—without needing a human to remember what’s safe to delete.

Final Thoughts

Treat idle database management like documentation: not glamorous, but essential, and painful when neglected. The teams that stay ahead don’t rely on quarterly cleanups—they build systems that flag drift automatically and make deletion feel safe, not risky. It’s less about tooling and more about culture. Infrastructure that reflects the present, not the past, is easier to operate and trust.

Cleaning up unused databases reduces more than cost—it removes cognitive load. Every forgotten RDS instance introduces friction: in audits, in migration plans, in monitoring noise. When you've got fewer unknowns, you don’t waste cycles second-guessing what’s still in use. Instead, your team can focus on the infrastructure that actually powers the business.

The strongest environments aren’t the ones with the most controls—they’re the ones with the fewest surprises. Retiring what doesn’t serve a clear purpose clears the way for faster decisions, simpler security, and cleaner growth.

How to Find and Safely Remediate Idle Databases on AWS: Frequently Asked Questions

Q: What if I suspect a DB is idle but might be used quarterly or yearly?

Start by expanding the observation window. Instead of reviewing week-over-week usage, query CloudWatch for trends over the past 90 to 180 days. Look for recognizable usage spikes tied to business cycles—end-of-quarter reporting, annual audits, or seasonal traffic simulations. If the pattern holds, label the instance accordingly and avoid full remediation.

For these low-frequency workloads, consider shifting the database to a suspended or paused state if the engine supports it. Aurora Serverless, for example, can be configured to auto-pause after inactivity, which cuts compute costs while preserving availability. If pause isn’t supported, use Systems Manager or EventBridge to implement recurring stop/start routines aligned to expected activity windows.

Q: How can automation help me avoid future idle database sprawl?

The most effective automation happens before the database exists. Integrate tagging requirements into your provisioning pipelines so every new instance carries metadata like Owner, Project, and ExpectedTTL. Then use AWS Config to continuously monitor for missing or misaligned tags and trigger remediation workflows.

For ongoing detection, implement a scheduled Lambda that pulls metrics like DatabaseConnections and ReadIOPS across your RDS estate. Cross-reference those with a DynamoDB table that tracks expected activity profiles. When a database falls outside its expected behavior, tag it for review or trigger a notification for the owning team. This creates a lightweight but persistent idle detection loop that evolves with your infrastructure.

Q: Are there any data compliance or regulatory concerns with deleting a DB?

Yes—and they’re often not immediately visible in the AWS resource itself. Compliance risks usually live in the data schema, not the instance metadata. Before any deletion, assess whether the database contains records tied to financial audits, customer interactions, medical logs, or legal holds. These datasets often fall under industry or jurisdictional mandates for long-term retention.

If data must be preserved, export it to an immutable medium. Use RDS snapshot exports to store the data in columnar formats like Parquet, which supports long-term archival and serverless querying. Apply S3 object tags like DataClassification=Regulated and configure bucket policies that enforce encryption and access restrictions. This gives you retention without incurring compute costs or leaving services running.

Q: Is there a single best solution for all idle databases?

No, because each idle database represents a different context. Some are no-brainers to delete—test environments with zero connections for 30+ days. Others need to stick around for compliance, but not in their current form. The remediation strategy should reflect how often the data is accessed, what kind of data it holds, and the cost of keeping it online.

Think in terms of tiers. For databases that are completely dormant, snapshot and delete. For those with intermittent access or unknown status, isolate and shrink. For compliance-hardened workloads, export and archive. The key is to match remediation depth with operational risk—so that the cleanup effort doesn’t create more problems than it solves.

Managing idle databases isn’t just about cost—it's about clarity, control, and confidence in your infrastructure. When you know what's running and why, you can scale smarter and avoid painful surprises down the line.

Backend options for front-end developers - a deep overview

Tom Granot — Sat, 15 Aug 2020 12:05:16 +0000

I got into a Twitter chat culminating in this tweet with Lee Robinson:

Lee Robinson

@leeerob

@TomGranot @purpleorangeye4 I'm searching for a 500-1000 word document explaining tradeoffs at a high level for why you might choose:

- Vanilla database
- Database client / ORM
- Auto-generated API over a database
- Fullstack framework

It probably depends on what you want to build. That's the hard part.

12:46 PM - 10 Aug 2020

I love this suggestion for an article. I really do - his tweets suggest that he's truly in a bind about all the possibilities, which means (since he's a prominent developer) that a lot of other, more silent developers are too. He wrote his own version here, but I figured I'd roll my own as well.

Some context: up until recently I was a Site Reliability Engineer - an ops guy, tasked with making sure our entire stack works as it should, with all its different parts behaving nicely. This gives me some understanding of how different pieces fit together, and I think I could shed some light on the darker sides of the stack.

Lee's article is very practical and down to the point. This article is a bit more "philosophical" in nature, and is aimed at people who want to get a "feel" for what all the different options out there are like. This usually implies more experienced developers, so if you're either just starting out or want the very practical, to the point answers to your questions - go with Lee. Otherwise - strap in.

What's in a backend?

I get the feeling that when Lee talks about a backend he's talking about a "data machine" - one that knows how to do your regular CRUD activities, and lets you focus on your front-end logic instead of focusing on operational concerns.

The backend, from my perspective, is the cornerstone of two - very different - concerns:

Running "correct" software - your backend is responding correctly to your requests
Running "performant" software - your backend is capable of handling the traffic you throw at it without wasting too much resources, in a quick and cost-effective fashion

Generally speaking, this is also the order of importance - your software first and foremost has to do what it should, and then do it as fast and with as little operational concerns as possible.

Following Lee's tweet, I am going to enumerate 4 different options, show some examples, and then discuss the tradeoffs.

I'm making 4 (valid, in my book) assumptions here:

We're talking about websites, and not various system services or more low-level applications / Machine Learning / Data Science stuff. Those "other" types of software are usually using a different type of front-end than the ones front-end devs are used to. Qt comes to mind for desktop apps, for example.
We're intentionally disregarding the fact that multiple developers - and DevOps people and DBAs and sysadmins - need to work, maintain and run this software in production. We are talking about a single developer, working on a single application, on their own. The human facet of things plays so, so much into technology selection, and it's way too large a concept to dive into here.
The "usual" flow of work for front-end devs is "call API, parse data, send to front". That means a lot of different backend APIs, all tailored towards a specific, "small" goal like setting a property of for an object or getting information about a cross-section of objects.
Most front-end devs use JavaScript and its myriad of frameworks to write their application logic.

Option 1 - Vanilla Database (Database Client)

This means that your backend is simply a database that you interface with directly. There are basically four variants of databases you can go with here:

Key-value Stores - Redis, DynamoDB, etc.
Relational Databases - MySQL, PostgreSQL, etc.
NoSQL Databases - MongoDB, CouchDB, etc.
Graph Databases - Don't, unless you specifically have a need for them (and then you'd probably know everything in this article already).

The choice of database changes the way you interact with it. Relational databases use SQL, NoSQL databases have a variety of data models and thus have a variety of ways of interacting with them, and key-value stores usually allow you to get and set key-value pairs.

The list above is actually ordered by the level of complexity each database system presents to you, in my opinion. Using a key-value store is more like dealing with localStorage, so should be somewhat familiar to front-end devs. SQL / NoSQL are.... more tricky.

There's a misconception in the tweet, by the way - a database client and an ORM are two different things. A client is usually just library that allows you to run commands on the database (read: write SQL queries), whereas an ORM is usually another layer of abstraction above the database itself (read: write JavaScript code). I'll deal with ORMs in Option 2.

Considerations

How complicated to deploy?

Relatively easy. Setting up the database is really easy, especially with database addons / plugins by the leading push-to-deploy tools like Netlify. The hard thing is choosing which database to use, maintaining the database, watching that it behaves, optimising it, creating a schema for it, etc. It's the "cleanest" way to go about storing data - no layers of abstraction between you and the database - but it's for people who want to deal with databases (like me!).

There is so much documentation about databases out there, it's insane. It's really easy to get confused. Choosing a database holds with it a very large set of considerations - most of which are completely irrelevant to the front-end developer.

I can abstract some of the mystery away by noting that the choice of which database to use mainly depends on where your code runs. Figure out where you want to deploy to, then google for "How to set up a database on X", where "X" is your platform of choice (Heroku, Netlify, etc). Most of the platforms have a huge amount of documentation already, since they want you to come aboard.

There's also the installation of the client library for that database, but that's usually an npm install away.

How much code do I have to write?

A large amount (SQL / NoSQL) or a medium amount (key-value stores). Note that there's no API here. That means that where you would to do a fetch before, you'd now need to write an SQL query to get the data you want, dispatch it to the database using a client (most databases have JS clients implemented as open-source libraries), then parse the response into form you want the data in. Same goes for updating data, just inversely (you have some data, then need to parse it into an SQL query to dispatch to the database). With data-heavy applications, that can mean hundreds (and often thousands) of different queries with varying length.

Working with key-value stores is a bit easier, since you're writing JSON-like (and sometimes actual JSON) to the database. It still requires defining a general schema for your data, however, or you will quickly have a real mess on your hands.

How complex will my code be?

Quite complex (SQL / NoSQL), or not very (key-value stores). I actually wanted to write that using SQL simplifies your code greatly - no extra APIs to learn - but that's assuming that SQL flows through your fingers. Most (good) backend devs I know speak fluent SQL, but from what I gather it's not something front-end tutorials and videos focus on. I'm doing my best to step out of my shoes and into the shoes of a front-end dev, so SQL fluency is not necessarily a common skill.

That means that any code that has complex SQL queries can be considered complex. Same goes for whatever data structure NoSQL databases use, with the added concern that they are often less represented in online tutorials as their SQL counterparts. There's material out there, sure, just not as in the line of sight as SQL stuff.

I have to note, however, that key-value stores are relatively straightforward if you're coming from JS, and aren't necessarily foreign-looking to most JavaScript devs, who are used to working with JSON and JavaScript objects.

Conclusion

I would opt for a database only if you really want to understand the lowermost abstraction in your stack that deals with persisting data. If that's not interesting for you, choose one of the other options.

Option 2 - an ORM (Object Relational Mapper)

An ORM is another level of abstraction between you and the database. It allows you to call "familiar" constructs (read: objects) to perform common activities, instead of relying on raw queries.

An example: you want to create a new item, that has a few values for the properties that define it. With an ORM, you would do so by calling the relevant ORM API for items:

Item.create({property1: 'value1' , property2: 'value2', property3: 'value3'})

With a raw SQL query, you would do it like so:

INSERT INTO items (property1, property2, property3) VALUES (value1, value2, value3)

This saves you a lot of SQL work, but is actually not the same as using a "normal" API endpoint. It's just a more comfortable wrapper around SQL queries, that is not custom-tailored to a specific need.

In other words, you still work with tables - they're just exposed to you as JavaScript objects. There are much more sophisticated ORMs that read your database schema and do all sorts of magic with it, but at their core - ORMs are just wrappers around tables. They prevent you from dropping down to raw SQL.

In option 3 I talk about another approach for the same idea, that tackles the same idea from a different approach.

Considerations

How complicated to deploy?

Relatively easy. ORMs still require you to deploy a database, and then install an ORM library for your framework of choice or vanilla JS (Sequelize is an example of a JavaScript ORM). That's not that different from deploying a raw database.

How much code do I have to write?

A large amount (models + accessing the ORM). Since your ORM doesn't actually know how you want your data to be structured, you need to define Models for your code. Sequlize's docs make a great intro for understanding what this means in practice, but for the sake of discussion you can think about as creating "virtual" tables.

This means that you're still doing basically the same thing you were doing with raw SQL queries - but instead of defining the tables in the database and then querying them from your code, you're defining your models in your code and the ORM creates the tables for you. This can take quite a lot of code if you have a lot of tables.

The rest is interacting with those tables via the ORM - which is usually around same amount of code as using raw SQL queries.

How complex will my code be?

Not very. Your code will be entirely JavaScript - no SQL. This provides for a much more native experience. The only "new" thing will be the ORM library's code, which is usually straightforward (Tablename.CRUDAction({propertiesObject}).

Conclusion

This choice is still somewhat verbose, and is basically one step up from interacting with the database directly. Option 3 details a path that offers a somewhat different way of thinking and resembles your current way of working, with REST-style APIs, more closely.

Option 3 - Auto-generated API over a database

This option is somewhat tricky to explain, because there are a few technologies that are all considered some variant of "API auto-generation", but are in fact very different things. These include software that turns a database into an API (like Hasura), and databases that come with an auto-generated API out of the box (like CouchDB).

These are more like "traditional" backend APIs, in the sense that they abstract away the need to deal with the database at all - and instead just give you an API you can fetch to and from. This means that you get all the information in the format you're used to - JSON - and there're no parts in the middle.

Note that this does not mean you are exempt from modelling the data in your database. The auto-generated API still relies on you telling it how is the information you want to use is modelled like. The nice part, though, is that once you model your data you don't really need to touch it anymore. Everything else is done via familiar APIs.

One comment - there's a technology called GraphQL that allows you to query APIs just like you would query a database, i.e. using a query language. This means you can use a single GraphQL call to the queryroot (a GraphQL system's main API endpoint) instead of mixing-and-matching different, multiple API queries.

Hasura creates a GraphQL API over a database, while CouchDB only allows you to access the database via an API. It's a tricky differentiation to make, but I would say those are two completely different worlds, and one should not confuse the two. What I'm referring to in this article is Hasura-like services, not CounchDB-like ones.

Considerations

How complicated to deploy?

Really easy. Especially with Hasura and HasuraCloud, getting up and running is very fast. The service is there, you model your data and you're good to go.

How much code do I have to write?

Probably less than you would have before. An auto-generated API is basically not a change at all from the way you used to work. You call an API exactly like you used before. The only difference is that the source of the API is not some backend code crafted by a developer, but an automated API over your database.

Especially with GraphQL, you're looking at shaving off a lot of different API calls, which will result in you writing less code. You will have to, however, define your models in your database / HasuraCloud console, which - as you can probably see by now - is part of the cost of playing.

One comment though: since you're working with a model of the database, expect that building your logic might sometimes be more verbose than what you would have with dedicated backend API endpoints. This really depends on what you are trying to create, and deserves an entirely different discussion. Creating data models is truly an art form, and part of the reason why hardcore programmers are so, so much more efficient than their peers - they're using the correct model for their problem.

How complex will my code be?

Generally simple. An auto-generated API is, in many ways, a front-ender's dream come true - an almost full abstraction of the backend. There's no SQL to write and the flow of work is similar to what you're used to - there's an API, right there in front of you, for the taking.

If you modelled your data correctly before, then the same logic you used previously will probably work here as well. If you're migrating, though, it's probably a good idea to re-think the model and see whether you can simplify it to reduce the number of API calls you're making.

If your old APIs were very complicated and specific, you might find that this new model allows for much more expressiveness with significantly less code. I dislike generalisations and catchphrases, but these services are a gold mine for most applications.

GraphQL itself is, however, somewhat foreign even for SQL veterans. It has a small learning curve, but there is legitimately amazing material out there - like this - that will take you all the way with your existing set of tools and frameworks.

Conclusion

If you're attempting to abstract away the backend, go with GraphQL over a database, like Hasura.

Option 4 - Full Stack Framework

A full-stack JavaScript framework - like Redwood - combines all you need to get a fully-functional web-app without the hassles of the separation of concerns - namely a backend and a frontend. It's a different type of philosophy, aiming to create a "unified" experience for you as a developer.

In practice, a full-stack framework is usually a combination of one of the ideas I mentioned before with the other, "normal" front-end parts of the application. Redwood is based around Prisma, which is a database toolkit (but you can think of it, for the sake of simplicity, as a type of very advanced and easy to use ORM), and uses GraphQL and React under the hood. The beauty of wrapping all of the relevant tools needed for an application in one bundle comes from the ability to stay in the same "state of mind" all of the way - everything is JavaScript, everything is available from the same "vendor" (i.e. your framework) and generally speaking you can "do it all" on your own.

If I had to guess, I'd say that this is where the web is going to - a consolidated JS experience for developers, operations people and anyone in between.

Considerations

How complicated to deploy?

Relatively easy. Everything is available outside of the box, which means that deploying the framework is as easy as finding a place to host it. Pretty much the same as the other options, albeit with all the documentation and concepts under the same roof - the framework's docs.

How much code do I have to write?

Probably less that you would have before. Since you are modelling your own data under the hood, you still need to define how it's going to be built. So writing full-stack code is comprised of defining how your data looks like and then using those definitions to write actual application logic. Pretty similar to the amount of code you would have written in Option 3.

How complex will my code be?

Generally simple. Again, it's all JavaScript - but you have to get familiar with the framework's syntax, which might scare away some people afraid of being "boxed" you into the framework. Fear not - Redwood, for example, utilises well-known open-source projects in the mix, so knowledge you gain by using the platform can generally be later transformed into other, adjacent worlds.

Conclusion

Full-stack frameworks are not yet popular enough to be considered the "de facto standard" of the future, but it sure does fell like they're getting there. I would suggest going first with something a bit more established like an auto-generated API (Hasura) and then make your way to a full-stack framework if it becomes too much to handle.

Wrapping it all up

We've went on quite a journey here.

I'd like to sign off with a personal message. I'm a systems guy - I like dealing with the nitty gritty, with trying different deployment options, with looking at why is my memory running out, on destructuring complicated infrastructure and building it all up again. That means I'm a generalist, rather than a specialist.

That doesn't mean, though, that you have to be one too. There's a whole world of content on both ends of the spectrum. Learn about what interests you most, go deep instead of wide if you want to, and mostly just enjoy the run. There are enough people working on the foundations for your dream project right now - you don't have to build (or even understand) everything yourself.

It does, however, mean you need to share your knowledge, so that other - oppositely-inclined - people would be able to benefit the same that you have. Spend time writing detailed GitHub issues, and blog posts, and tutorial videos.

We're all in this together.

Questions? Comments? Hit me up in a private message or leave a comment here.

Learning with books vs. tutorials

Tom Granot — Sun, 02 Aug 2020 09:39:51 +0000

I'm currently re-learning Rust - I picked it up a while back to be able to better debug issues at work, but didn't have the chance to write a lot of production-ready code with it.

In order to fully understand all the intricacies of the language, I picked up an open-source side project to participate in. But before that, I'm going to go deep on the language's syntax and methodology by reading The Book, i.e. The Rust Programming Languge.

What is your preference? Do you run through tutorials and then reach out for project, or do are you a bookworm like me?

Why shall you shell as a webdev? And other terminal topics

Tom Granot — Wed, 29 Jul 2020 07:13:15 +0000

I have a great affinity for textual interfaces.

Knowing my way around a terminal has, by far and wide, been the skill with the highest ROI (Return On Investment) in my toolbelt.

It enables you to, in no particular order:

Create highly-customized, used-on-the-daily programmer workspaces
Automate routine tasks with a single command
Easily and immediately observe the status of your machine, external services and basically everything else
Run matrix-style commands:

Yes, this is a Windows CMD GIF, and not a Linux terminal one. I regret nothing.

Discuss!

I can not recommend learning the terminal enough. It's a real treat, and I think its versatility practically ensures there's a use case for it somewhere in your workflow - even for non-programmers!

What do you think? Shall everyone shell, or is GUI king of the realm?

And specifically - how important are terminal skills for the average web developer?

Side Note

See the cover image? Try running curl parrot.live from your terminal, and you can get the party parrot too!

See here for more information.

Docker shell shortcuts - because writing full commands is hard!

Tom Granot — Tue, 28 Jul 2020 22:30:31 +0000

Containers for fun and profit!

So you're now officially excited about Docker and containers.

Good! They're an awesome tool to create portable configurations for your software and make sure it works seamlessly across multiple devices.

If you've been around them long enough, though, you're probably using Docker engine's CLI (command line interface) quite often.

Commands like:

docker exec -ti 1x81hs72k2s9 /bin/bash

docker start container-name

Are probably all over your shell history.

I had the same problem, and so I made a small hack and uploaded it to my slowly-expanding Useful Snippets repository (which you can totally star if you feel like it :) ):

TomGranot / useful-snippets

Useful things I made/found over time

useful-snippets

In chronological order of addition, a bunch of useful stuff kept here for persistence:

Base64-Encoded Flag Images (as JSON!) - link
Some shell snippets for a better Docker experience - link

View on GitHub

Money time

Specifically, if you go here you'll see the following snippets:

#!/bin/bash

# Get container id of existing container
dgrep(){
    docker ps -a | grep "$1" | cut -c1-12
}

# bash into an existing container
dbash() {
    docker exec -ti $(dgrep "$1") /bin/bash
}

# sh into an existing container
dsh() {
    docker exec -ti $(dgrep "$1") /bin/sh
}

# Execute something in an existing container
dex() {
    docker exec -ti $(dgrep "$1") $2
}

# Start an existing container
dstart(){
    docker start $(dgrep "$1")
}

# Stop an existing container
dstop(){
    docker stop $(dgrep "$1")
}

# Remove an existing container
drm(){
    docker rm $(dgrep "$1")
}

# Stop and remove an existing container
dsrm(){
    dstop $(dgrep "$1") && drm $(dgrep "$1")
}

# Remove an existing image
dirm(){
    docker image remove $(dgrep "$1")
}

# Get logs of an existing container
dlog(){
    docker logs $(dgrep "$1")
}

These go inside your shell's configuration file, probably located at ~/.bashrc if you're on one of the Linux distros and older macs, or ~/.zshrc if you're on newer macs.

After you put those commands inside your shell configuration file, whenever your shell starts up they will become available for use inside your session - just like regular shell commands.

Shell functions as command aliases

Note that what I did up there was define shell functions that perform specific commands, and take arguments just like regular shell commands.

This goes in contrast to defining shell aliases, like Nick Taylor does here (for the most part), and allows us to pass arguments to our "aliases".

This was a bit of a mouthful. Let's consider, as an example, the dgrep function I defined earlier:

# Get container id of existing container
dgrep() {
    docker ps -a | grep "$1" | cut -c1-12
}

dgrep() is a shell function that accepts an unlimited amount of arguments (as shell functions do). In order to access the arguments provided to the function inside the function's body, we use the $X notation, where X is an integer number.

$0 always refers to the name of the executing command, so if you're running:

my-command first-tom second-tom

And my-command is defined as

my-command() {
    echo $0;
    echo $1;
    echo $2;
}

Then the output will be:

my-command
1
2

Coming back to our dgrep command:

# Get container id of existing container
dgrep() {
    docker ps -a | grep "$1" | cut -c1-12
}

Note that what it's doing is getting the contents of docker ps (that is a list of all running containers), then piping the input into grep, which returns any match to the searched text with the provided argument.

In our case, it's "grepping" for $1, which is the first argument provided to dgrep. It finally pipes that output into cut -c1-12 that gets the first 12 characters of the piped text (which is, coincidentally, exactly the length of Docker container IDs).

For example, if you're running:

dgrep tom-container

Then if tom-container is a running container, docker ps will return a string containing all the information the Docker engine has about that container.

dgrep will then pipe it into the cut command, and print out only the container ID of that container.

But why do I need that?

Excellent question. Remember our example from the beginning of the article, where we used the full container ID to refer to the container?

docker exec -ti 1x81hs72k2s9 /bin/bash

Nobody expects you to remember the full names or the IDs of your containers. We can now re-write the command as:

docker exec -ti $(dgrep substring-in-container-name) /bin/bash

Where $() is a subshell - a special command that executes whatever is provided to it, and sends out the output as text before running the rest of the command.

In our case, $(dgrep substring-in-container-name) will find us a container ID based on some substring in the name (like my in my-very-long-and-hard-to-remember-container-name), and pass it to the docker exec -ti CONTAINER-ID /bin/bash command as CONTAINER-ID.

Wait, but that's still kinda long...

You're right! "Bashing" into a container (i.e. running bash inside that container interactively, which is basically "opening" a terminal to that container) is something we do quite often to debug long-running containers, and to do other fun filesystem shenanigans (ask me about that if you'd like to know more!).

In comes dbash():

# bash into an existing container
dbash() {
    docker exec -ti $(dgrep "$1") /bin/bash
}

As you can see, dbash is calling the previously mentioned dgrep in a subshell, and passes that subshell the argument provided to dbash - namely a substring of the relevant container.

This eventually lets us do something like:

dbash tom

And get a terminal to the first container on the list of running containers that has tom in its human-readable name.

Pretty neat, right?

Conclusion

Containers are fun. And so is working with the shell!

I literally just made a really nice hack using Docker for an open-source project I'm working on with my good friend, Federico. Feel free to take a look if you're into container wizardry!

Side note

An observant reader would note that I can save one pipe by using Docker command formatting. That reader would be correct, but I think my syntax is simpler and easier to explain to beginners, so I stuck with it for this tutorial (and, honestly, for my own ~/.zshrc as well).

Quick tip: Increase keyboard repeat rate for faster typing

Tom Granot — Mon, 27 Jul 2020 21:33:01 +0000

Why?

This a short post to remind myself that some fruits are hanging so, so low, that you're a dummy for not thinking about them yourself.

If you're not a master of keyboard shortcuts to run around textual interfaces (like your code editor or your terminal), you've probably done the following many a time:

Write many words of text.
Figure out you made a mistake a few words back.
Press and hold the left key button until you reach the word you need to fix.
Fix word.
Press and hold the right key button until you reach the place you were before.
Repeat ad infinitum.

There are two potential paths for improvement here, both revolving around steps 3 & 5.

Become better at keyboard shortcuts

This will replace steps 3 & 5 with a combination of keys that performs the required action.

This is a worthwhile pursuit, as you will be spending insane amounts of time riding a keyboard. Check out this dev.to article for some great ones.

It does, however, take some time to get used to keyboard shortcuts, and if you're anything like me - i.e. always changing environments between remote servers, VMs, containers, different local code editors and basically any modern website with WYSISWYG capabilities - then you will have to memorize a large amount of shortcuts. This can get tiring very fast.

Solution: if you can't be faster, make your computer run faster.

Increase keyboard repeat rate

This will increase the speed of steps 3 & 5 with a neat keyboard trick.

The keyboard repeat delay of your machine is the amount of time that the computer waits between each repeat action you make.

In our case, it's the amount of time that passes between each "go left" (step 3) or "go right" (step 5) operations.

If we decrease that delay, we increase the rate at which those actions can happen in a sequence. This rate is called the keyboard repeat rate.

A cursor on a machine with a fast keyboard repeat rate looks much, much faster than it does on your current machine. It will basically ping-pong across lines of text, dashing to wherever you need it to be.

Here's how to employ this trick on different platforms - add a comment if you need some help or find this super-duper-cool like I did!

Mac OS

This can be done either graphcially, using System Preferences --> Keyboard:

Or textually, using the command line - just add the following commands to your shell user's config file, usually ~/.zshrc on newer macs:

# Set a blazingly fast keyboard repeat rate
defaults write NSGlobalDomain KeyRepeat -int 1
defaults write NSGlobalDomain InitialKeyRepeat -int 10

Most Linux distros

Same, except swap ~/.zshrc for ~/.bashrc. Graphical way changes between distros, so I can't really attest to that.

Also, you're a Linux user. Use your shell! :)

Windows

Untested - see here.

Props

Original idea from here and the ever-awesome Mathias Bynens.