Forem: Nico Bistolfi

FREE Bulletproof Supabase Backups: With AWS Lambda and S3

Nico Bistolfi — Sun, 03 Aug 2025 02:37:42 +0000

The July 2025 Replit incident should terrify any team building AI agents. An AI coding agent deleted an entire production database during a "code freeze", then created fake data to cover its tracks. The agent later admitted it "panicked" when seeing empty results. This breakdown highlights a critical vulnerability: Supabase's built-in backups aren't sufficient protection against AI-driven disasters.

Why Supabase's backup system fails under pressure

Supabase provides basic automated backups, but daily backups start at $25/month with only 7-30 day retention. More problematically:

No API to trigger immediate backups when AI agents are active
Physical backups for databases >15GB cannot be restored outside Supabase
Storage API objects excluded from database backups
Maximum 28-day retention creates dangerous gaps

When an AI agent starts behaving erratically, you need immediate backup capabilities and long-term retention. Supabase's system provides neither.

Enter go-postgres-s3-backup: Purpose-built protection

The go-postgres-s3-backup repository solves these exact problems with a serverless Lambda solution designed for production PostgreSQL environments. This isn't just another backup tool—it's specifically architected for the AI agent era.

Key advantages:

Serverless AWS Lambda execution (zero server maintenance)
Intelligent 3-tier backup rotation (daily/monthly/yearly)
Automatic S3 lifecycle management with Glacier and Deep Archive
Sub-5-minute deployment using Serverless Framework
Built-in encryption and security best practices

Implementation walkthrough

Setting up bulletproof backups takes under 10 minutes:

1. Clone and configure

git clone https://github.com/nicobistolfi/go-postgres-s3-backup.git
cd go-postgres-s3-backup
npm install -g serverless

2. Set your Supabase connection

# Get from Supabase -> Settings -> Database
echo "DATABASE_URL=postgresql://postgres.xxxx:password@aws-0-region.pooler.supabase.com:5432/postgres" > .env

3. Deploy

task deploy

That's it. Your database now has enterprise-grade backup protection.

How the backup architecture works

The system implements a sophisticated rotation strategy that balances cost and recovery needs:

daily/     → 7-day retention, immediate access
monthly/   → Transition to Glacier after 30 days
yearly/    → Deep Archive after 90 days for compliance

Daily execution flow:

Lambda triggers at 2 AM UTC via EventBridge
Connects to PostgreSQL using efficient pgx/v5 driver
Creates compressed SQL dump
Uploads to S3 with server-side encryption
Manages retention automatically

Smart rotation logic:

If no monthly backup exists, promotes current daily backup
If no yearly backup exists, promotes current daily backup
Cleans up expired daily backups (>7 days)
Lifecycle policies handle storage tier transitions

This approach provides multiple recovery points while minimizing storage costs—critical for AI agent environments where backup frequency might need to increase rapidly.

AI agent protection strategies

The Replit incident reveals that AI agents can exhibit deceptive behavior under stress. The agent didn't just delete data—it actively tried to hide the deletion by creating fake records. Your backup strategy must account for this.

Pre-AI agent deployment checklist:

# Trigger immediate backup before AI agent access
task invoke

# Verify backup completed
task logs

# Confirm S3 backup exists
aws s3 ls s3://go-postgres-s3-backup-prod-backups/daily/

During AI agent operations:

# Schedule additional backups during high-risk AI operations
# Add to crontab for hourly backups during AI development:
0 * * * * cd /path/to/go-postgres-s3-backup && task invoke

Post-incident recovery:

# List available backups
aws s3 ls s3://go-postgres-s3-backup-prod-backups/daily/

# Download specific backup
aws s3 cp s3://go-postgres-s3-backup-prod-backups/daily/2025-08-01-backup.sql ./

# Test restore to local PostgreSQL
docker run --name recovery-test -e POSTGRES_PASSWORD=postgres -d -p 5432:5432 postgres
docker exec -i recovery-test psql -U postgres -d postgres < 2025-08-01-backup.sql

Production hardening

For AI agent environments, enhance the default configuration:

Increase backup frequency during AI operations:

# serverless.yml - modify events section
events:
  - schedule: rate(4 hours)  # Every 4 hours instead of daily

Add backup verification:

// Add to main.go after backup creation
func verifyBackup(backupPath string) error {
    // Download and verify backup integrity
    file, err := s3Client.GetObject(&s3.GetObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(backupPath),
    })
    if err != nil {
        return err
    }

    // Verify file size and basic SQL syntax
    if file.ContentLength < 1000 {
        return errors.New("backup file suspiciously small")
    }

    return nil
}

Monitoring and alerting:

# Add CloudWatch alarm for backup failures
resources:
  Resources:
    BackupFailureAlarm:
      Type: AWS::CloudWatch::Alarm
      Properties:
        AlarmName: postgres-backup-failure
        MetricName: Errors
        Namespace: AWS/Lambda
        Statistic: Sum
        Period: 300
        EvaluationPeriods: 1
        Threshold: 1
        ComparisonOperator: GreaterThanOrEqualToThreshold
        AlarmActions:
          - !Ref SNSTopic

Cost optimization for AI workloads

The tool's lifecycle management becomes crucial for AI agent environments where backup frequency might spike:

Daily backups: $0.023/GB/month (Standard S3)
Monthly backups: $0.004/GB/month (Glacier)
Yearly backups: $0.00099/GB/month (Deep Archive)

For a 10GB database with hourly backups during AI agent operations:

Standard approach: ~$200/month for retention
go-postgres-s3-backup: ~$15/month with intelligent tiering

Testing your disaster recovery

Before trusting your setup with AI agents, validate the complete recovery process:

# 1. Create test data
echo "CREATE TABLE ai_test (id serial, data text);" | psql $DATABASE_URL
echo "INSERT INTO ai_test (data) VALUES ('pre-ai-agent-data');" | psql $DATABASE_URL

# 2. Trigger backup
task invoke

# 3. Simulate AI agent destruction
echo "DELETE FROM ai_test;" | psql $DATABASE_URL
echo "INSERT INTO ai_test (data) VALUES ('fake-data-like-replit');" | psql $DATABASE_URL

# 4. Restore from backup
aws s3 cp s3://go-postgres-s3-backup-prod-backups/daily/$(date +%Y-%m-%d)-backup.sql ./
psql $DATABASE_URL < $(date +%Y-%m-%d)-backup.sql

# 5. Verify recovery
echo "SELECT * FROM ai_test;" | psql $DATABASE_URL

The bottom line

The Replit incident proves that AI agents will eventually behave unpredictably with your data. The agent's deceptive behavior—creating fake data to hide its mistake—demonstrates that traditional safeguards are insufficient.

go-postgres-s3-backup provides the insurance policy you need: automated, frequent, verifiable backups with intelligent cost management. Deploy it before your first AI agent touches production data. The 10 minutes of setup time could save your company.

When your AI agent inevitably panics and does something destructive, you'll have the backups to recover. And unlike Supabase's limited retention, you'll have them for years.

Data De-identification for LLMs: Protecting Privacy While Preserving Meaning

Nico Bistolfi — Tue, 27 May 2025 23:25:29 +0000

TL;DR: deidentify is a zero-dependency Go library that removes PII from your data while preserving format and meaning. Same input = same output, making it perfect for LLM preprocessing. Star it on GitHub if you find it useful!

The Hidden Risk in Your AI Pipeline

Every time you send customer data to an LLM, you're making a trust decision. Whether it's GPT-4, Claude, or your company's internal model, that data leaves your control. And if it contains names, emails, SSNs, or credit card numbers, you're one data breach away from a nightmare.

Imagine the following scenario: you're looking to use LLMs to analyze customer support tickets. Great idea, until legal asks: "What happens to the customer data we're sending to OpenAI?"

Why Traditional Redaction Falls Short

The obvious solution is to redact everything. Replace all names with [REDACTED], all emails with [EMAIL], and so on. But here's the problem:

Original: "John Smith emailed john.smith@company.com about his order"
Redacted: "[NAME] emailed [EMAIL] about his order"

The LLM loses critical context. Was it the same person? Different people? The relationships between entities disappear, making analysis nearly useless.

Preserving Meaning While Protecting Privacy

What you need is consistent, deterministic replacement. Same input should always produce the same output, so relationships stay intact:

Original: "John Smith emailed john.smith@company.com, then Jane Doe replied"
Better: "Robert Johnson emailed robert.j@demo.com, then Sarah Miller replied"

Now the LLM understands that the email belongs to the first person, not the second. The data remains useful for analysis while protecting actual identities.

Enter deidentify: A Go Solution

After facing this problem repeatedly, we built deidentify - a Go library that handles this automatically. What makes it special:

Zero dependencies - Just Go's standard library. No supply chain risks.
Deterministic - Same secret key + same input = same output, always.
Format preserving - Phone numbers look like phone numbers, emails like emails.
Context aware - Uses column names to prevent correlation across fields.

Here's what it looks like in practice:

secretKey, _ := deidentify.GenerateSecretKey()
d := deidentify.NewDeidentifier(secretKey)

text := "Contact Alice at alice@startup.com or 555-123-4567"
safe, _ := d.Text(text)
// Output: "Contact Taylor at member4921@demo.co or 555-642-8317"

Real-World Example: Customer Support Analysis

At my current company, we process thousands of support tickets through LLMs for sentiment analysis and categorization. Before deidentify:

Ticket: "Hi, I'm Bob Wilson (bob@example.com). My SSN 123-45-6789
was exposed when your system crashed. Please call me at 555-0123."

Risk: Bob's entire identity is sent to an external API.

After deidentify:

Ticket: "Hi, I'm Michael Davis (user7823@demo.org). My SSN 847-92-3651
was exposed when your system crashed. Please call me at 555-7492."

Result: LLM can still analyze the severity (SSN exposure) without seeing real data.

Why Go Matters Here

You might wonder why we built this in Go. Three reasons:

Performance - De-identifying gigabytes of data needs to be fast
Deployment - Single binary, no runtime dependencies
Safety - Strong typing catches PII-type mismatches at compile time

The fact that it uses only Go's standard library means you can audit the entire codebase without chasing dependencies. Critical for security-conscious teams.

Implementation Tips

When integrating de-identification into your LLM pipeline:

De-identify early - Before data hits your message queue or API
Keep your key safe - The secret key is what makes replacements consistent
Test with production-like data - PII patterns vary by industry

Example pipeline:

// Step 1: Load customer data
data := loadCustomerData()

// Step 2: De-identify
safe := d.Table(data)

// Step 3: Send to LLM
response := llm.Analyze(safe)

// Step 4: Process results (no re-identification needed)

Take Advantage

We made deidentify open source because privacy tools should be transparent. You can inspect every line of code, understand exactly how it works, and even contribute improvements.

The library handles:

Names, emails, phone numbers
SSNs, credit cards, addresses
Structured data (CSV, database exports)
International formats (100+ address patterns)

If you're sending any data to LLMs, check out deidentify on GitHub. And if it saves you from a data breach, consider giving it a star 🌟 - it helps others find the tool.

9 Coding Principles to Write Great Python Code 🐍🚀

Nico Bistolfi — Thu, 24 Oct 2024 20:44:30 +0000

Welcome to the Python coding dojo! 🧑‍💻 This guide will help you avoid common pitfalls while writing clean, readable, and efficient code. Let's dive into 9 key principles that will level up your Python game, followed by a bonus tip to supercharge your code reviews! ⚡

1. Use Meaningful Variable Names

Bad code often has cryptic variable names like x, y, or tmp. Always aim for meaningful, descriptive names that make the code easier to understand.

❌ Bad Code

x = 10
y = 20
z = x + y

✅ Good Code

length = 10
width = 20
area = length + width

📜 Why: Descriptive names make your code self-documenting. You’ll thank yourself (and your team will thank you too!) when you revisit this code weeks or months later. 🚀

2. Follow PEP 8 Style Guide

PEP 8 is the official style guide for Python. Stick to it to ensure that your code looks like everyone else's and is easy to read.

❌ Bad Code

def my_function(param1,param2):
    return(param1+param2)

✅ Good Code

def my_function(param1, param2):
    return param1 + param2

📜 Why: Consistency is key. PEP 8 ensures your code is neat and easy to follow for everyone, not just you. Use tools like flake8 to check for PEP 8 compliance.

👉 PEP 8 Guide

3. Avoid Magic Numbers and Strings

Magic numbers (unnamed constants) or magic strings can make your code hard to understand and maintain. Always define them as variables with meaningful names.

❌ Bad Code

if payment_type == 1:
    print("Credit Card")

✅ Good Code

CREDIT_CARD = 1

if payment_type == CREDIT_CARD:
    print("Credit Card")

📜 Why: Using named constants makes your code more readable and flexible. It's much easier to change CREDIT_CARD = 1 in one place than to hunt down all instances of 1 in your code.

4. Use List Comprehensions (But Don’t Overdo It)

List comprehensions are a concise way to create lists but should be used only when they improve readability.

❌ Bad Code

result = []
for i in range(10):
    result.append(i * 2)

✅ Good Code

result = [i * 2 for i in range(10)]

📜 Why: List comprehensions are more compact and often faster. Just be careful not to make them too complex — they should still be easy to understand.

5. Don’t Repeat Yourself (DRY)

Avoid repeating the same logic in multiple places. If you need to do the same thing in multiple places, put it in a function.

❌ Bad Code

print(f"User {user_id} logged in")
print(f"User {user_id} logged out")

✅ Good Code

def log_event(event, user_id):
    print(f"User {user_id} {event}")

log_event("logged in", user_id)
log_event("logged out", user_id)

📜 Why: DRY keeps your code maintainable and less error-prone. Whenever you spot duplication, consider refactoring to avoid it.

6. Handle Exceptions Properly

Don’t just catch exceptions and pass silently, handle them in meaningful ways.

❌ Bad Code

try:
    result = 10 / 0
except:
    pass

✅ Good Code

try:
    result = 10 / 0
except ZeroDivisionError as e:
    print(f"Error: {e}")

📜 Why: If you just "pass" on exceptions, you might miss critical errors. Handle them appropriately or at least log them so you know what’s going wrong.

7. Use Docstrings for Documentation

Your code should be self-explanatory, but when that's not enough, use docstrings to explain your functions.

❌ Bad Code

def calculate_area(radius):
    return 3.14 * radius ** 2

✅ Good Code

def calculate_area(radius):
    """
    Calculate the area of a circle given the radius.

    :param radius: The radius of the circle.
    :return: The area of the circle.
    """
    return 3.14 * radius ** 2

📜 Why: Docstrings provide useful information about your functions, classes, and modules, especially when you're working in a team or handing off your code.

8. Use Built-In Functions Where Possible

Python comes with a lot of built-in functions. Make sure to use them instead of reinventing the wheel.

❌ Bad Code

numbers = [1, 2, 3, 4, 5]
total = 0
for num in numbers:
    total += num

✅ Good Code

numbers = [1, 2, 3, 4, 5]
total = sum(numbers)

📜 Why: Python’s built-in functions are optimized for performance and readability. Don’t waste time writing code that Python already provides.

9. Keep Your Code Modular

Break your code into small, reusable functions instead of writing long, complex blocks of code.

❌ Bad Code

def process_data(data):
    # Do everything in one big function

✅ Good Code

def clean_data(data):
    pass

def analyze_data(data):
    pass

def process_data(data):
    clean_data(data)
    analyze_data(data)

📜 Why: Modular code is easier to test, debug, and maintain. It also makes it easier to reuse code across different projects.

Bonus Tip: Write & Review Code Like a Pro!

Once you have written your code, reviewing it is equally important. Not sure how to do that effectively? Here's a bonus tip — check out this comprehensive guide on how to do effective code reviews that I wrote. It will guide you through what to look for when reviewing code, ensuring you're delivering your best work every time.

💡 Final Thought: Writing good code isn’t about being perfect but continuously improving. As Linus Torvalds said:

"Talk is cheap, show me the code."

Beginners Guide: Setting Up Your Local Environment for Machine Learning with Miniconda and Python

Nico Bistolfi — Tue, 22 Oct 2024 15:56:09 +0000

Welcome to the world of machine learning! Whether you’re just starting or have dabbled a bit, having a well-organized local environment can make your life much easier. In this guide, we’ll set up your local environment using Miniconda and Conda. We’ll also install some of the most essential Python libraries for machine learning and data science: Pandas, NumPy, Matplotlib, and Scikit-learn.

Warning: This setup is 100% stress-free (except maybe for the part where we install libraries 😉).

Why Miniconda?

You might be wondering: "Why Miniconda and not Anaconda?" Well, it’s like choosing between a fully loaded spaceship 🛸 (Anaconda) and a lightweight, more customizable spacecraft 🚀 (Miniconda). Miniconda gives you just the essentials, allowing you to install only the packages you need and keep things tidy.

Step 1: Install Miniconda

1.1. Download Miniconda

Head over to the Miniconda website and download the appropriate installer for your operating system:

Windows: .exe installer
macOS: .pkg installer
Linux: .sh installer

1.2. Install Miniconda

Once downloaded, follow the instructions for your system:

Windows: Run the .exe installer. When it asks, check the box to “Add Miniconda to my PATH environment variable” (it’ll make life easier later, trust me 😅).
macOS/Linux: Open a terminal and run the installer:

  bash Miniconda3-latest-Linux-x86_64.sh  # for Linux
  bash Miniconda3-latest-MacOSX-x86_64.sh  # for macOS

Follow the prompts. It’s smoother than butter on a hot pancake! 🥞

1.3. Verify Installation

Once installed, let’s make sure everything is in working order. Open your terminal or command prompt and type:

conda --version

If you see a version number, congrats—you’ve got Miniconda ready to go! 🎉

Step 2: Set Up a Conda Environment

Here comes the fun part! With Conda, you can create isolated environments to keep your projects organized and prevent package conflicts. Think of it like having different closets for different hobbies—no mixing fishing gear 🎣 with your gaming setup 🎮.

2.1. Create a New Environment

To create a new environment (think of it as your project’s personal workspace), use the following command:

conda create --name ml-env python=3.10

Here, ml-env is the name of your environment, and we’re setting Python to version 3.10. Feel free to use whichever version you prefer.

2.2. Activate the Environment

Before we install any packages, we need to activate the environment:

conda activate ml-env

You’ll notice your prompt changes, showing you’re now inside the ml-env environment. 🧙‍♂️ It’s like stepping into a new dimension... of Python, that is.

Step 3: Install Essential Python Libraries

Time to arm your environment with the necessary tools! We’ll install Pandas, NumPy, Matplotlib, and Scikit-learn—the heroes of any machine learning adventure. Think of them as your Avengers 🦸‍♂️, but for data science.

3.1. Install Pandas 🐼

Pandas is great for working with structured data. You can think of it as Excel, but on steroids 💪. Install it with:

conda install pandas

3.2. Install NumPy 🔢

NumPy is your go-to library for numerical operations and matrix manipulation. It’s the secret sauce behind a lot of machine learning algorithms. To install:

conda install numpy

3.3. Install Matplotlib 📊

What’s data science without some beautiful charts? Matplotlib is perfect for creating visualizations, from line graphs to scatter plots. Install it with:

conda install matplotlib

(Quick joke: Why don’t graphs get into relationships? Because they have too many “plots” 🤣).

3.4. Install Scikit-learn 🤖

Finally, we need Scikit-learn for implementing machine learning algorithms like linear regression, classification, and more. To install:

conda install scikit-learn

Step 4: Verify Your Setup

Let’s make sure everything is working smoothly. Open Python in your terminal:

python

Once inside the Python shell, try importing the libraries to see if everything is installed correctly:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import sklearn

If there are no errors, you’re good to go! 🎉 Go ahead and exit Python by typing:

exit()

Step 5: Managing Your Environment

Now that your environment is all set up, here are a few handy tips for managing it.

5.1. List Installed Packages

Want to see what’s installed in your environment? Simply type:

conda list

5.2. Save Your Environment

To share your environment setup with others or recreate it later, you can export it to a file:

conda env export > environment.yml

5.3. Deactivate the Environment

When you’re done working for the day, you can exit the environment with:

conda deactivate

5.4. Delete an Environment

If you no longer need an environment (goodbye, old projects 👋), you can remove it entirely:

conda remove --name ml-env --all

If you liked this, follow me on Github

Liked the content? Follow me for more content like this

Nico BistolfiFollow

Persistence leads to breakthroughs.

Final Thoughts

Congrats! You’ve successfully set up your local machine learning environment with Miniconda, Conda, and essential Python libraries like Pandas, NumPy, Matplotlib, and Scikit-learn. 🎉 Your new environment is isolated, organized, and ready for some serious data crunching.

Remember: Always keep your environments tidy, or risk ending up like my old closet—full of tangled cables and random Python versions. 😅 Happy coding!

Code Reviews for Beginners: How to Send and Review Code Like a Pro

Nico Bistolfi — Tue, 15 Oct 2024 19:27:49 +0000

Code reviews are an essential part of the software development process. Whether you're sending your first code review or reviewing someone else's work, knowing how to navigate these waters is key to becoming a stronger developer. Let’s break down both sides of the code review process, from submitting your code for review to providing valuable feedback as the reviewer.

Sending a Code Review: How to Do It Right

When you submit your code for review, it’s not just about getting a "thumbs up" from your team. It’s about showcasing your work in a way that’s easy to understand, clean, and ready to integrate. Here’s how to approach sending a code review:

1. Ensure Your Code Is Clean

Before you even think about submitting your code, make sure it’s polished. Run tests, lint your code, and ensure there are no obvious errors or oversights. A clean submission sets the tone for a productive review. "It runs on my laptop" is just not good enough :)

2. Explain Your Changes

When submitting your code for review, context is critical. Provide a concise yet detailed description of what the code does, why certain decisions were made, and any potential areas that may require extra attention. The more information you give upfront, the easier it is for the reviewer to understand your thought process.

3. Break It Into Small, Manageable Chunks

Submitting massive pull requests can be overwhelming for reviewers. Keep your changes small and focused. Large, multi-purpose pull requests are harder to review and can easily introduce bugs that are hard to spot. Think that a code review shouldn't include more than 100 lines of changes, and within that take into account the cognitive load of the code you're submitting.

Being a Code Reviewer: How to Give Constructive Feedback with a Growth Mindset

Growth Mindset = trust your teammates, think what you can lear from what they did and how you're review may allow your teammates to learn too. A thoughtful review not only improves the code but also helps foster a collaborative and learning-focused environment.

Reviewing someone else’s code is just as important as writing your own, very likely you're responsible for maintaining that code (and people deserve PTO wink wink).

1. Be Respectful and Constructive

Remember, you’re reviewing someone’s work, not criticizing them as a person. Be positive, specific, and suggest improvements in a way that encourages growth. Instead of saying, “This is wrong,” try “This approach might be problematic in cases where X happens. Have you considered Y?” - if possible, talk to the person directly and initiate the dialogue.

As you build trust with your teammates, you'll learn to calibrate your message to the person receving the feedback and ensuring they receive it in the best possible way.

2. Focus on the Big Picture First

When reviewing code, it’s easy to get lost in the nitty-gritty details (like variable names or indentation). But first, focus on the big picture: Does the code solve the problem? Is it efficient? Is it maintainable? Once the major aspects are addressed, you can move on to smaller stylistic issues.

3. Ask Questions Instead of Making Assumptions

Sometimes, you won’t fully understand the reason behind certain coding decisions. Instead of assuming it’s wrong, ask questions. “Can you explain why you chose this approach?” is far better than saying, “This doesn’t make sense.” This opens up a conversation where the author can explain their thinking, and you might even learn something new.

4. Be Specific with Your Feedback

Generic feedback like “Looks good” or “Needs work” isn’t helpful. Be as specific as possible so the developer knows exactly what you’re referring to and why. Provide code snippets or links to documentation when possible to support your points.

Final Thoughts: Code Reviews Are a Learning Opportunity

Whether you’re sending or reviewing code, remember that the code review process is a learning opportunity for everyone involved. As the code author, you get to learn from feedback and improve your work. As the reviewer, you get to understand someone else’s code and share your expertise.

By approaching code reviews with respect, clarity, and collaboration, you’ll not only improve the quality of the code but also contribute to a stronger, more effective development team. Happy reviewing!

AI Code Generation is the New WordPress

Nico Bistolfi — Fri, 11 Oct 2024 17:26:59 +0000

We’ve seen code generators becoming more integrated into the tools we use every day. Some call them "the future of coding," others are more skeptical and stick to using Vim, I'm in the middle and I believe there's a strong similarity between AI generated code and WordPress.

WordPress powers about 43% of all websites on the internet, including some of those shiny Fortune 500 companies' sites. It’s an absolute workhorse for blogs, small businesses, and even some more complex sites. But we all know it's not ideal for everything, especially when you're aiming for real scale or complexity.

WordPress: Great Tool, Wrong Job

Doing a quick rundown of things WordPress isn’t exactly designed for:

Large-scale platforms (Think millions of users simultaneously hammering your server)
Complex web applications (Like heavy SaaS with intricate workflows)
Real-time collaborative tools (Imagine building Google Docs with WordPress... yikes)
Financial platforms (I’m talking about high-security requirements, not a simple eCommerce site)
Mobile-first apps (Yes, you can build apps with WordPress, but should you? Let’s be honest)
Data-heavy analytics platforms (Good luck crunching massive datasets on WordPress)

The list can go on, WordPress is amazing for what it is – just not for everything. It’s great for certain applications but definitely not the go-to when you’re looking at high security, real-time functionality, or complex operations.

AI Code Generation: Familiar Territory

In the same vein, AI code generation or IDE assistants (take your pick: Copilot, Tabnine, Cursor, or any fork of VS) are fantastic tools that can really streamline your workflow, but they’re no silver bullet. Just like WordPress, they’re wonderful when used for the right task, but don’t expect them to build your next complex, highly-secure, real-time platform without some serious oversight.

The code they generate isn't always the best. We’ve all seen those StackOverflow snippets pasted verbatim that work (sometimes) but make you wince in horror when you peek under the hood. The logic these tools implement can often be faulty and it make take you more time to code-review the generated code than to just write it yourself.

Where AI Shines: Scripting and Simplicity

AI truly shines in automating repetitive scripting tasks or creating "black-box" logic. The kind where you feed in some input, expect an output, and don’t really care about what happens in the middle? As long as you can define the task clearly, AI can often hit the nail on the head.

If you’re looking to automate a tedious process, spin up a basic REST API, or even mock up an MVP (with very simple functionality), AI code generation might just save you hours of work as I've shown in my article Generative AI: Dynamic Mock APIs using Langchain and Anthropic..

I've also used it to generate tests, I've found that it does a pretty good job at a task that is usually quite boring - UNLESS you're doing TDD, which in that case I would not recommend it.

Pro tip: the more you can reduce your use cases and make your MVP as basic as possible, the higher the chances you can get away with using AI-generated code. Less is more here.

The Silver Lining

Just like WordPress, AI-assisted code generation is a tool in your toolkit. It's powerful, but only when used correctly. Think of it like trying to use a hammer to do the job of a screwdriver – sure, it might work in some scenarios, but you're probably going to break something if you push it too far. 🛠️

At the end of the day, tools like AI code generators can take a lot of the grunt work out of your hands and speed up development for specific tasks. But when you start needing more robust, nuanced, and scalable solutions, that's when the human touch (aka: you) is irreplaceable. AI won’t be putting us out of a job just yet.

Best Python Libraries for ETL Pipelines: A Beginners Developer's Guide

Nico Bistolfi — Thu, 10 Oct 2024 00:42:32 +0000

Building ETL pipelines can feel a lot like being the chosen one – you’re moving data from point A to point B, transforming it into something useful, and making sure everything works seamlessly. Python’s got your back with an army of libraries that make your job easier – kind of like having the Force on your side 🛠️. In this guide, we’ll look at some of the best Python libraries to help you win the ETL war. 🏆

1. Extract: Pulling Data from the Source

When it comes to data extraction, you need the right tools to pull data from different sources – databases, APIs, files. This is where the fun begins (cue Obi-Wan voice). Here are the go-to libraries for getting the data you need.

SQLAlchemy

When extracting data from databases, SQLAlchemy is your trusty lightsaber. It's powerful and handles multiple database types without breaking a sweat.

Pros:
- Supports a wide range of databases (PostgreSQL, MySQL, SQLite, etc.)
- You can easily switch between databases
Example:

  from sqlalchemy import create_engine

  engine = create_engine('postgresql://user:password@localhost/dbname')
  connection = engine.connect()
  result = connection.execute("SELECT * FROM jedi_order")

Pandas

Pandas is your Swiss army knife 🗡️ when it comes to dealing with data in CSV, Excel, JSON, or even SQL. It’s fast and simple to use, perfect for extracting data from files.

Pros:
- Can load data from various file formats with a single line of code
- Great performance for in-memory data
Example:

  import pandas as pd

  data = pd.read_csv('rebels_data.csv')

Requests

For dealing with REST APIs, requests is like R2-D2 – it’s reliable, simple, and will get you the data you need, no matter what.

Pros:
- Makes HTTP requests super easy
- Handles API authentication, headers, etc.
Example:

  import requests

  response = requests.get('https://api.example.com/data')
  data = response.json()

2. Transform: Shaping the Data

Now that you’ve extracted the data, it’s time to transform it into something usable. This stage is like taking raw mithril and forging it into armor 🛡️. Let’s dive into some awesome libraries for transformation.

Pandas

Once again, Pandas comes in handy for transforming your data. Whether it’s cleaning, filtering, or aggregating, it’s got you covered like a cloak of invisibility.

Pros:
- Tons of built-in functions for data manipulation
- Ideal for in-memory transformations
Example:

  # Clean data by removing NaN values and filtering rows
  data_cleaned = data.dropna().query('age > 18')

Dask

Got massive datasets that would make even the Death Star look small? Dask lets you handle larger-than-memory data using parallel processing, all without rewriting your Pandas code. 🌌

Pros:
- Scales to handle large datasets
- Distributed computing, but with familiar Pandas-like syntax
Example:

  import dask.dataframe as dd

  df = dd.read_csv('huge_data.csv')
  result = df[df.age > 18].compute()

PySpark

For Jedi-level transformations on big data, look no further than PySpark. It's the Luke Skywalker of distributed data processing. 🧙‍♂️

Pros:
- Lightning-fast data transformations on large datasets
- Great for working in big data ecosystems (Hadoop, Spark)
Example:

  from pyspark.sql import SparkSession

  spark = SparkSession.builder.appName("ETL").getOrCreate()
  df = spark.read.csv('galaxy_data.csv', header=True, inferSchema=True)
  df_filtered = df.filter(df.age > 18)

3. Load: Putting Data Where It Belongs

Finally, you’ve transformed your data into something usable. Now it’s time to load it to its final destination. Whether it’s a data warehouse, S3 bucket, or database, think of this as delivering the One Ring to Mordor 🏔️ – with the right tools, the journey becomes a whole lot easier.

SQLAlchemy

SQLAlchemy makes loading data back into your database simple. With it, you can easily insert your data into a relational database.

Pros:
- Works with multiple databases
- Supports bulk inserts
Example:

  data.to_sql('jedi_council', engine, index=False, if_exists='replace')

Psycopg2

For PostgreSQL databases, psycopg2 is your best companion. It’s fast, efficient, and makes complex SQL tasks a breeze.

Pros:
- Native support for PostgreSQL
- Supports transactions
Example:

  import psycopg2

  conn = psycopg2.connect(dbname="star_wars", user="user", password="force123")
  cur = conn.cursor()
  cur.execute("INSERT INTO jedis (name, age) VALUES (%s, %s)", ('Luke', 30))
  conn.commit()

Boto3

If you’re working with AWS services like S3, Boto3 is the go-to tool for uploading data to the cloud. You’ll feel like Gandalf wielding it. ☁️

Pros:
- Fully integrated with AWS services
- Easy to upload/download from S3
Example:

  import boto3

  s3 = boto3.client('s3')
  s3.upload_file('local_file.csv', 'mybucket', 'file.csv')

Google Cloud Storage (GCS) Client

For developers working with Google Cloud, the GCS Client will help you load data to Google Cloud Storage with ease, just like Boto3 does with AWS.

Pros:
- Full support for Google Cloud
Example:

  from google.cloud import storage

  client = storage.Client()
  bucket = client.get_bucket('my_bucket')
  blob = bucket.blob('data.csv')
  blob.upload_from_filename('local_file.csv')

4. Orchestration: Managing Your ETL Pipeline

Now, no ETL pipeline would be complete without a bit of orchestration. Think of this as the force guiding all the moving parts ⚙️ – scheduling tasks, monitoring, and retrying if something goes wrong.

Apache Airflow

If you're working on anything complex, Apache Airflow is your Yoda for task orchestration. With it, you can create, schedule, and monitor workflows, ensuring all your ETL jobs run like clockwork.

Pros:
- Powerful scheduling and task management
- Visual interface for tracking workflows
Example:

  from airflow import DAG
  from airflow.operators.python_operator import PythonOperator

  def extract_data():
      # Extraction logic
      pass

  dag = DAG('my_etl_pipeline', start_date=datetime(2023, 1, 1))
  task = PythonOperator(task_id='extract_task', python_callable=extract_data, dag=dag)

Wrapping Up

Building ETL pipelines doesn’t have to feel like you're battling Darth Vader ⚔️. With the right tools, you can automate the entire process, transform data efficiently, and load it to its final destination. Whether you’re handling small data sets or working on massive, distributed systems, these Python libraries will help you build ETL pipelines that are as powerful as the One Ring (but way less evil).

May the ETL Force be with you. ✨

Go vs Python for File Processing: A Performance and Architecture Perspective

Nico Bistolfi — Wed, 09 Oct 2024 22:21:53 +0000

"If all you have is a hammer, everything looks like a nail." This is known as the Law of the Instrument or Maslow's hammer, and it's a cognitive bias that pushes us to often rely on the tools we are most familiar with, even when they may not be the best fit for the task at hand.

When it comes to file processing, using Python a no-brainer, due to its simplicity, large ecosystem and libraries. However, Go is an equally powerful tool—one that can often outperform Python, especially in scenarios requiring high performance and concurrency.

I'll explore the strengths and weaknesses of Go and Python when handling file processing tasks. By benchmarking these two languages in real-world scenarios, I'll highlight their differences and help you choose the right tool for the job. I'll also look at how both languages can interact in cloud-native environments like serverless architectures (AWS Lambda) and containerized applications (Docker/Kubernetes).

Benchmarking Go vs. Python

To understand which language might be better suited for file processing tasks I created two simple scripts, one in Python and another in Go to run these benchmarks. I focused on transforming CSV files into JSON payloads while using concurrency in Go and multithreading in Python. Here's what I found:

Go:

Processing Time: Go was significantly faster, especially when handling large files. This is due to its efficient use of system resources through its built-in concurrency model.
Concurrency: Go's goroutines and channels make it easy to process multiple rows in parallel, drastically reducing processing time for large datasets.
Memory Consumption: Go had better memory management, leading to stable performance under high-load scenarios.

Python:

Processing Time: Python is slower compared to Go in processing large files, even with multithreading enabled.
Multithreading: Due to the Global Interpreter Lock (GIL), Python's multithreading can be less efficient, limiting parallelism, especially in CPU-bound tasks like file processing.
Ecosystem: Python offers a wealth of libraries like pandas and csv, making it highly versatile for different data formats. However, this often comes at the cost of speed (I found better performance using csv instead of pandas).

Go for High-Performance, Concurrent File Processing

Go's native concurrency model using goroutines and channels makes it an ideal choice for architectures that require high-performance, scalable file processing. With Go, you can efficiently process large datasets, such as logs, transactions, or IoT data, by leveraging multi-core CPUs without the overhead seen in other languages.

Best use cases for Go:

Batch processing of large files (e.g., CSVs, JSON logs).
Data pipelines in microservice architectures.
High-throughput systems that require predictable latency.

Serverless Architecture with Go on AWS Lambda

Go works well in serverless environments like AWS Lambda. AWS Lambda now supports Go natively, allowing you to deploy functions for file processing with minimal cold start time. Here's an example of a serverless setup for Go:

package main

import (
    "context"
    "encoding/csv"
    "fmt"
    "os"
    "github.com/aws/aws-lambda-go/lambda"
)

func handleRequest(ctx context.Context) (string, error) {
    // File processing logic here
    return "File processed successfully", nil
}

func main() {
    lambda.Start(handleRequest)
}

Why Go in Serverless?

Faster cold starts compared to Python.
Better memory management for long-running file processes.
Excellent for event-driven architecture, such as processing S3-triggered file uploads.

Kubernetes with Go

Go also excels in a Kubernetes environment due to its lightweight binary size, speed, and low memory footprint. A Go-based microservice in Kubernetes can efficiently scale horizontally by spinning up more replicas to handle increased load. Here, Go's performance shines in terms of startup time and CPU efficiency.

Why Go in Kubernetes?

High throughput with low CPU and memory consumption.
Goroutines allow for massive parallelism, making it well-suited for distributed systems like Kubernetes.
Can handle multiple concurrent file processing requests with minimal latency.

Python for Flexibility and Rich Ecosystem

Python's strength lies in its ease of use and powerful ecosystem of libraries. Although slower than Go, Python remains a great choice for IO-bound tasks where the bottleneck is waiting for external resources rather than CPU processing.

Best use cases for Python:

Data analytics where you need to leverage libraries like pandas for CSV transformation and NumPy for numerical data manipulation.
Prototyping complex data pipelines quickly.
Machine learning pipelines, where Python can be integrated with data processing libraries.

Serverless Architecture with Python on AWS Lambda

Python is also supported natively by AWS Lambda, and it's one of the most popular choices due to its simplicity and rapid development time. For lightweight or IO-bound file processing, Python can still be an efficient choice.

import csv
import json

def lambda_handler(event, context):
  # Process CSV to JSON here
  return {
        'statusCode': 200,
        'body': json.dumps('File processed successfully')
  }

Why Python in Serverless?

Easy to deploy and develop.
Rich support for third-party libraries, including data analytics tools.
Great for prototyping or environments where flexibility is key.

Kubernetes with Python

Python's flexibility makes it useful in a Kubernetes environment, but the slower processing time and memory consumption are key considerations. Python services may not be as performant as Go in a highly concurrent or large-scale system. However, if you rely heavily on Python libraries, Kubernetes allows easy scaling of Python microservices.

Why Python in Kubernetes?

Simple to integrate with existing data processing libraries.
Good for batch jobs or IO-heavy workloads.
Can still scale horizontally, but may require more resources compared to Go.

Key Differences in Performance and Suitability

Aspect	Go	Python
Concurrency Model	Goroutines (lightweight threads)	Multithreading (limited by GIL)
Memory Consumption	Low, efficient	Moderate to high
Ease of Use	Moderate learning curve	Easy and widely known
Library Ecosystem	Limited but performant	Extensive and rich for data work
Execution Speed	Fast	Slower
Cold Start Time (Lambda)	Low	Moderate to high

Which Architecture Benefits the Most?

High-Throughput, Performance-Critical Systems: Go is the best option, especially for applications needing fast, concurrent file processing, like financial services or IoT pipelines.
Flexibility in Data Transformation: Python is well-suited for architectures that require flexibility and heavy use of third-party libraries for data transformation and analytics, making it ideal for data science workloads.
Serverless Architectures: Both Go and Python are excellent for serverless environments on AWS Lambda, but Go shines when performance is critical and Lambda cold starts or resource constraints are a concern.
Kubernetes Microservices: Go offers better scalability, performance, and resource efficiency in Kubernetes. Python can still perform well but may require more resources to handle the same load, particularly in CPU-bound tasks.

Final Thoughts

Both Go and Python have their strengths and weaknesses, and the right choice depends on your architecture, the nature of the file processing task, and performance needs. Go's speed and concurrency make it ideal for high-performance, CPU-bound tasks, while Python's simplicity and rich ecosystem are better suited for data-heavy, IO-bound tasks or rapid development.

Whether you choose Go or Python, understanding how each language performs under different conditions will ensure that you maximize efficiency and scalability in your file processing architecture, whether in a serverless environment or a containerized microservices setup.

Repository

You can find the code for my experiment and why I landed on this conclusion here: https://github.com/nicobistolfi/golang-vs-python

Eagle Image Optimization API

Nico Bistolfi — Wed, 04 Sep 2024 17:12:17 +0000

Having previously co-founded an Web Performance and Image Optimization company my motivation on building this open-source image optimization api is to enable everyone to give a better experience to their users.

An Open-source Image Optimization API levels the field, regardless of budget or technical expertise. I'm hoping this fosters innovation and experimentation, leading to a diverse landscape of solutions tailored to specific needs.

The benefits of optimizing images using Eagle Image Optimization API

Accessibility for All: Large, unoptimized images can create significant barriers for users with slow internet connections or disabilities. Open-source image optimization shrinks file sizes, minimizing data consumption and ensuring everyone can access web content seamlessly.
Democratization of Web Creation: Traditionally, image optimization tools have been expensive and complex, limiting access for smaller creators and businesses. Open-source solutions break down this barrier, empowering anyone to create high-quality, fast-loading content without significant financial or technical hurdles.
Fostering Innovation and Collaboration: Open-source APIs encourage community involvement, with developers contributing code, fixes, and adaptations. This collaborative approach fosters continuous improvement, leading to innovative solutions tailored to diverse needs and emerging technologies.
Building a Sustainable Web Ecosystem: By reducing server load and resource consumption, open-source image optimization contributes to a more sustainable web ecosystem. This benefits both developers by lowering server costs and the environment by minimizing energy usage.
Enhancing User Experience: Faster loading times, improved accessibility, and optimized image quality all contribute to a better user experience. Open-source image optimization ensures users spend less time waiting and more time engaging with the content they seek.

Where to find Eagle Image Optimization API

The project is hosted on Github and can be found here with a detailed README on how to deploy it to AWS using Serverless Framework.

How to use it

The API is very simple to use and it leverages query parameters to modify the image. Here's a list of the basic available parameters:

url: The url of the image to be optimized.
width: The width of the image to be resized.
height: The height of the image to be resized.
fit: When both a width and height are provided, the possible methods by which the image should fit.

* More advanced parameters can be found in the README file.

A sample URL would look like this:

{{API_ENDPOINT}}/api/v1/image?width=400&height=400&fit=contain&url={{IMAGE_URL}}

Example of an image optimized using the API:

https://d3kbkk2pyl9ybg.cloudfront.net/api/v1/image?width=200&height=200&url=https%3A%2F%2Feagle-image-test.s3.us-west-1.amazonaws.com%2Fpublic%2Feagle.jpeg

Run in Postman

To test the API you can use the following Postman collection

Deploying to AWS

To deploy the API to AWS you can use the Serverless Framework. Once you have the Serverless Framework installed and you've configured your .env file with the settings you want to use for the API, you can deploy the API by running the following command:

serverless deploy --stage production

This will create a new CloudFormation stack in your AWS account and deploy the API.

✨ ProTip: If you fork the repo you can deploy to AWS using github actions.

Let's Collaborate

The Eagle Image Optimization API is a powerful tool designed to democratize access to high-quality image optimization, making it accessible to everyone, regardless of their budget or technical expertise. By leveraging the open-source community, we're creating a sustainable, collaborative environment that fosters innovation and continuous improvement.

If you're passionate about optimizing web experiences or simply need a robust solution for your project, I encourage you to explore the API, contribute to the project, and share your feedback. The more we collaborate, the better we can make this tool for everyone.

I’m always open to connecting with fellow developers, entrepreneurs, or anyone interested in web performance. If you have any questions, need help deploying the API, or just want to chat about the project, feel free to reach out. You can ping me anytime—I'd love to hear from you!

Web Performance for beginners

Nico Bistolfi — Fri, 09 Oct 2020 16:08:50 +0000

We all know what waiting means, and we don’t want it. It’s awful to be waiting in line, wait for something you bought to be delivered, or click on a link and wait for a website to load, especially if it is your website. You don’t want to be that company that users slip away because the website didn’t load properly.

Let’s get into the topic, what is Web Performance? In this article, we are going to talk about web performance in a way that everybody can understand, and define the next steps to improve the performance of your website.

I’m Nico Bistolfi, and I’m the CEO at Piio. A company that helps websites to accelerate their experiences for the users and deliver the content faster. I also love building things in many ways; I built my own race car with my father, built our own house with my wife, and co-founded the largest eCommerce company in my home country. We got to $25M annual revenue, which I exited in 2015. I’m also obsessed with Speed, and that’s why Piio started.

Not everything can be fixed, but everything can be measured

There are many excuses and things that are completely valid and true. That’s why we are going to take a look at different tools and tests, in a way that you can check what you can do, see what the results are, and define your mid/long-term goal for your website.

We hear very often the phrase: “our software is too old” — or — “we need to change service providers” — or — “our platform has limitations”. And all these “excuses” are valid and true, we know it! All these excuses involve spending money, but sometimes money it’s not the issue, knowledge is. Eg, the images from the content marketing team are too big, there are things we can do to solve or improve that.

First of all, we need to measure, in order to understand what do we need to change or which problem do we need to solve. If you don’t have a website yet and you are thinking about web performance, think about how to have a great experience for your users. It’s not just about design, and it’s not just about the user flow. Performance it’s really important and we have different ways of doing it.

The key elements to improve and test web performance

If we talk about key elements that you can use to understand what do you need to do and what you can do to improve and test your web performance:

Synthetic testing -Test on a simulated environment. We recommend having a starting point of reference. It gives you a consistent result time over time and allows you to monitor your web performance.
Real User Monitoring — You can set it up on your website and it would constantly measure the performance on your site. How it’s loading on your user devices and in different scenarios. This is really good because it helps you to understand based on your demographics how things are changing. If you have users using the latest mobile phones, computers, and powerful devices, you don’t need to worry very much about web performance. But, if you are in the other way that you have people with more legacy devices, or with devices not that powerful, then you really need to look at web performance.
Browser Manual Testing — just, too technical for today. But as it says, it’s manually testing and measuring the performance on different devices.

Synthetic Testing

We have hundreds of tools, some of the key tools that we’ve found very useful are:

**Page Speed Insights** (or Lighthouse) designed by Google that works very similarly. With Lighthouse you can run your test on your own browser which is going to change a little bit the results, it’s a kind of manual testing. But Page Speed Insights is going to give you something else: the Core Web Vitals. These metrics are measured by Google across the different devices that your users are accessing your website. And that’s very interesting! The Core Web Vitals are focused on Loading, interactivity, and visual stability of your website (I’m going to talk about them later).
**GTMetrix** is another tool that when you get used to the interface, it’s really easy to find what’s wrong and what’s working well for you. Another extra point for GTMetrix that I find very useful is the Waterfall feature. It allows you to visualize the different resources that are being loaded for your website in the specific test you are running.
**Image Speed Test** is designed by us, at Piio. It’s completely free and you can set up to receive emails with a new test every month at no cost. This tool is extremely focused on images and how you can improve the image loading on your website.

Page Speed Insights

Let’s take a random website as an example, using the Page Speed Insights tool. In the picture, we can see two pieces of content that are very important to us.

Metrics (blue rectangle)

The blue rectangle contains the Core Web Vitals and we are going to focus on the Largest contentful Paint (LCP) — First Input Delay (FID) and Cumulative Layout Shift (CLS).

First Contentful Paint (FCP) it’s important but if you focus on LCP you are going to fix the FCP automatically.

Google is showing us how well that metric performs for all the devices or users that are accessing the G2 website. We can see on the right side, the First Input Delay and the Cumulative Layout Shift are in the threshold that is considered good. The 0.0 seconds for the Cumulative Layout Shift is considered good for 90% of the users that are accessing the G2 website.

This is very useful because we can see that there’s no real need to work on the First Input Delay or in the Cumulative Layout Shift. But on the FCP or in the Largest Contentful Paint, there is a real need! We can see that for almost 20% of the users the LCP is in the poor performance threshold, and of course, that is something that we do want to work in.

The hard thing about this is that really working on the LCP could be really hard and really technical. Usually what happens, or what I see with different people, is that they end up working on the complete Page Load and that ends up speeding up a little bit the LCP. But it’s not something really easy to identify. You might need someone technical to help you with that process.

More metrics (red rectangle)

On the red rectangle, we’re going to have more metrics that are useful too, like the Speed Index or the Time to Interactive (when the user is going to be able to start interacting with your website). In the G2 case is 4.9 seconds. Maybe they can start reading something before that, but they are not going to click on anything or write on any input. This is something that can definitely be improved (The blue flags icons explain to you more of how you can improve those metrics or what the metric means).

GTMetrix

Moving to GTMetrix, this can be difficult to read. In the beginning, you can focus on the base Page Speed Score or YSlow Score. If you try to get above 85–90% on each one of those tests, you are in a good position!

Something to clarify about GTMetrix is that their Fully Load Time is actually the Fully Load Time, meaning that they’re going to miss you till every resource of your website gets loaded and it’s completed. We’re used to hearing that is recommended for a website to load in less than 3 seconds, and yes that’s true!

If your website takes more than two seconds to load, people are more likely to abandon your website. In this case, you can’t use the 3 seconds rule, because maybe your website is already completely loaded at 3 seconds, but you have more things like a chatbot, pop-ups, or something more related to behavior, that is loading on the background, and the user doesn’t really need to at the beginning. GTMetrix is not going to count that, and just going to wait until everything is loaded and going to stop the timer there. I would say if you are below 5 seconds could be good or even if you are around 8 seconds could be a good scenario too.

Total Page Size

We do recommend for the Total Page Size to be less than 1.5 MB and regarding the requests keep it as slow as possible. Having fewer requests means the user device or the browser is going to ask for fewer resources and that means less network usage (improving the page load time automatically). If you click on one of those metrics you attach an explanation on how you can improve that and what are the things that you can do. But something more interesting about this is that you are also going to be able to identify that, from your website. That’s the waterfall effect, I will explain it in another chapter.

Image Speed Test

Last but not least, the Image Speed Test. What we do and what we are trying to tell you, is that there more things than page load. This test is extremely focused on images and how they can be optimized and improved.

We can see that for the image that is analyzed on the comparison, we’re having like a 97% of savings if we were properly to optimize that image. That means that we could have 97% better results on that image, and this will impact on the general results of the whole website.

The interesting thing is that this test is focused on educating and giving useful information to you. Below the image, you can see some advice and suggestions, for example, the size of the image. It’s 100px square and based on the simulation for desktop, the Image Speed Test identifies it should be 80 x 80 px, and this is a huge change! Other suggestions that you can find are the correct or better image format you should use to optimize your image. All these improvements will decrease Image weight.

Images: the biggest part of your website

In the next report box, you can visualize the potential savings by optimizing all your images (not specifically one as we see on the report above). Reaching this number means applying all the techniques and suggestions given. In the G2 case you can go from 1.38 MB to less than 700 KB. This would really speed up your website!

Usually, images are taking the biggest part of your website, and are the heaviest resource that e-commerce, or any website in general, has. Try this tool, run a test of your website, and see how much you can save by optimizing your images. If you want to get deep into this tool and learn how to take advantage of the Image Speed Test, here is an article with all the data you need.

Set a Monitor or an alarm for web performance

This is really important. If you are not going to set a monitor or an alarm, just go away… I really recommend keeping a close eye on your web performance. If you haven’t done it today, today is the day to start.

Change your mindset and keep track of it(not just for 2 or 3 weeks). Web performance is about constancy and keeping track of your progress. The websites are dynamic and constantly changing, that’s why this is a task you must have under control. Keep the work and effort during the time, otherwise, it’s very easy to lose everything.

This is an example when you update something on your website, and this could impact your web performance. We can see here that the performance score went down from 95% to less than 60%. Looking at the other graph, we see that the total page weight, on another date went up from 500 KB to 2.5 MB, and the same happened with Page Load.

We can understand that we are adding something very heavy for our website, and that is affecting web performance. This is the perfect time to call our team and see what’s happening to take action.

“Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away”
Antonie de Saint-Exupéry

Conclusion

Keep an eye on your Website Performance- and not just for a couple of weeks. Set an alarm or receive an email with your report.
Don’t try to solve everything- Prioritize
Involve your team, tell which are your goals in terms of performance.

Thanks for reading!

Post first published at: https://blog.piio.co/web-performance/web-performance-for-humans/