Forem: ryo ariyama

I Reduced a Week-Long Dev Task to 1 Hour with Claude Code

ryo ariyama — Sun, 08 Mar 2026 08:53:23 +0000

Introduction

Recently, I tried out Claude Code on a development project. I'm writing this down as a memo of what I learned and my thoughts on it.

What is Claude Code?

Claude Code is an AI agent developed by Anthropic, specialized for code generation. The official documentation describes it as follows:

Claude Code is an agentic coding tool that reads codebases, edits files, runs commands, and integrates with development tools. It's available in terminals, IDEs, desktop apps, and browsers. Claude Code is an AI-powered coding assistant that helps you build features, fix bugs, and automate development tasks. It can understand your entire codebase and work across multiple files and tools to complete tasks.

In short, it's a tool that autonomously handles the full development workflow: reading code, implementing changes, running tests, and creating pull requests on GitHub.

Background

In my day-to-day work, I maintain and develop various systems — one of which is an ETL tool. This ETL tool connects to various data sources to extract, transform, and load data. Development on it can broadly be divided into two workstreams:

Developing the tool's interfaces and shared components
Developing plugin-style modules that connect to individual data sources

The second workstream — plugin module development — involves a wide range of data sources such as MySQL, PostgreSQL, and Oracle, which demands significant time for development and research. As the customer base grows and development requests increase, the burden on engineers rises accordingly. Ideally, we'd be able to bring on additional engineers, but chronic understaffing has made that impossible.

On the other hand, plugin development follows fairly predictable patterns. So I thought: if I could standardize the development process as much as possible and automate it with Claude Code, I could reduce the workload — and that's what led me to adopt it.

Installation

Before getting started, you need to install the CLI:

npm install -g @anthropic-ai/claude-code

Once installed, run claude and complete API authentication. Just follow the terminal prompts and you should be fine.

npm: https://www.npmjs.com/package/@anthropic-ai/claude-code

Practices

CLAUDE.md

Starting with the official documentation, you can get up and running by following How Claude Code Works:

When you give Claude a task, it works through three phases: gathering context, taking actions, and verifying results. These phases blend together. Claude uses tools to search files to understand your code, edit them to make changes, and run tests to verify its work.

The agent autonomously handles research, development, and testing. To enable it to do this efficiently, engineers need to provide the necessary resources. This information goes into a markdown file called CLAUDE.md.

CLAUDE.md can be placed in several locations:

Location	Description
`~/.claude/CLAUDE.md`	Applies to all Claude sessions
`./CLAUDE.md` (project root)	Check into git to share with your team
`./CLAUDE.local.md`	Add to `.gitignore` for local-only settings
Parent directories	Useful for monorepo setups
Child directories	Picked up on demand when Claude works in that directory

For guidance on how to write a good CLAUDE.md, these articles are worth reading:

✅ What to include

Bash commands Claude can't infer on its own
Code style rules that differ from defaults
Testing instructions and recommended test runners
Repository etiquette (branch naming, PR conventions)
Project-specific architectural decisions
Development environment quirks (required environment variables)
Common pitfalls or non-obvious behaviors

❌ What to exclude

Anything Claude can understand by reading the code
Standard language conventions Claude already knows
Detailed API documentation (link to docs instead)
Frequently changing information
Long explanations or tutorials
Per-file codebase explanations
Self-evident practices like "write clean code"

💡 Best practice: Keep CLAUDE.md under 500 lines. Since its contents are loaded into Claude Code's memory, shorter is better.

To get started, run /init to generate a sample template. Here's an example:

# Project Context

When working with this codebase, prioritize readability over cleverness.
Ask clarifying questions before making architectural changes.

## About This Project

FastAPI REST API for user authentication and profiles.
Uses SQLAlchemy for database operations and Pydantic for validation.

## Key Directories

- `app/models/` - database models
- `app/api/` - route handlers
- `app/core/` - configuration and utilities

## Standards

- Type hints required on all functions
- pytest for testing (fixtures in `tests/conftest.py`)
- PEP 8 with 100 character lines

## Common Commands

uvicorn app.main:app --reload  # dev server
pytest tests/ -v               # run tests

## Notes

All routes use `/api/v1` prefix. JWT tokens expire after 24 hours.

Skills

In addition to CLAUDE.md, preparing Skills files is another recommended practice.

Skills function like procedure manuals that guide the agent through specific tasks. By preparing a SKILL.md file, the agent will follow its steps during development.

Anthropic has published example skills in a public repository — a great reference for getting started:

https://github.com/anthropics/skills

For the ETL tool I described earlier, I prepared one skill per workstream. I can then give simple instructions like "create a new module" or "add a parameter to the interface", and the agent takes it from there.

The directory structure looks like this:

CLAUDE.md                          # Concise project overview & conventions
docs/skills/
  new-plugin.md                    # Steps for creating a new plugin
  add-interface-parameter.md       # Steps for adding an interface parameter

Results

By automating development with the agent, tasks that would take a senior engineer about a week can now be completed in roughly an hour.

Additionally, using near-identical template-like prompts, virtually any engineer can now produce the same quality output — achieving faster delivery without sacrificing quality.

Closing Thoughts

That wraps up my introduction to Claude Code. What I've covered here is just the basics, and I'm sure there are many more ways to improve on this. I encourage you to try it out and share any useful patterns you discover!

As a side note — I often hear that AI will take engineers' jobs and previously had the same idea as well, but working with Claude Code has made me think the opposite. Using Claude Code effectively requires knowing good development processes, designing modules that are easy for AI to learn from, knowing how to write proper tests, and how to write clear markdown. Right now, software engineers are best positioned to do all of this. Rather than disappearing, I think the demand for engineers who can efficiently leverage AI will only grow.

What do you think? I'd love to hear your perspective.

Multi-Tenant Design for Bedrock Knowledge Base: Solving the Account Limit with Metadata Filtering

ryo ariyama — Thu, 01 Jan 2026 18:26:59 +0000

Introduction

Recently, while working with Bedrock KnowledgeBase in my daily work, I encountered some challenges related to its specifications that I'd like to share.

Background

Currently, I'm developing a multi-tenant application using Bedrock KnowledgeBase (referred to as KB below). To briefly explain KB, it's an orchestrator for implementing LLM RAG that handles vectorization of files into vector stores and can generate context-aware conversations when combined with Bedrock Agent.

We're using OpenSearch as our vector store, and our design creates separate KBs and indices for each tenant. This approach ensures data isolation between tenants, which seemed like a natural design choice at the time.

The Problem

At some point, when checking Bedrock's quotas, I found (Knowledge Bases) Knowledge bases per account quota. This is a limit on the maximum number of KBs you can create within an account, with a hard limit of 100. With our initial design, this meant our application could only support up to 100 tenants. Therefore, we needed to reconsider our design.

Solution

In reconsidering the design, we modified it so that KBs and indices are shared across multiple tenants. Since KBs have several parameters related to vectorization, such as ChunkStrategy, we created several combinations of ChunkStrategy and MaxToken parameters and let users select from these options for sharing.

An important consideration with this approach is ensuring that tenant data isn't referenced during conversations with other tenants. KB provides functionality to attach custom metadata during vectorization, so we adopted a method of attaching tenant_id-like metadata and filtering documents by that ID during conversations.
https://docs.aws.amazon.com/bedrock/latest/userguide/kb-metadata.html

Here's the conceptual approach:
Architecture:

Shared Knowledge Base across multiple tenants
Custom metadata (tenant_id) attached to each document
Metadata filtering during retrieval to ensure data isolation

Below is sample code for attaching metadata to documents:

# ingest_documents
response = client.ingest_knowledge_base_documents(
    knowledgeBaseId='string',
    dataSourceId='string',
    clientToken='string',
    documents=[
        {
            'metadata': {
                'type': "IN_LINE_ATTRIBUTE",
                'inlineAttributes': [
                    {
                        'key': 'tenant_id',
                        'value': {
                            'type': "STRING",
                            'stringValue': "$tenant_id",
                        }
                    },
                ]
            },
            'content': {
                ...
            }
        },
    ]
)

To filter documents by metadata when conversing with the agent, you can implement it with code like this:

## invoke_agent
response = boto3.client.invoke_agent(
        'knowledgeBaseConfigurations': [
                {
                    "knowledgeBaseId": "$vector_store_id",
                    "description": "Knowledge base for document retrieval",
                    "retrievalConfiguration": {
                        "vectorSearchConfiguration": {
                            "filter": {
                                 "equals": {
                                    "key": "tenant_id", "value": "$tenant_id"
                                    }
                                 }
                            }
                    },
                }            
        ]
)

Future Plans

With the above implementation, we can now build the application while successfully avoiding the constraints.

In multi-tenant applications, it's crucial to monitor that one tenant cannot access another tenant's data. I'm thinking of creating a monitoring mechanism to ensure this isn't possible. For example, I'm considering creating multiple test tenants, inserting different documents into the same vector store for each, asking questions about other tenants' documents, and verifying that no answers are returned. This script could be executed regularly in staging environments. While monitoring system resources like CPU is important, I believe it's equally crucial to monitor data to ensure that data that shouldn't exist according to system specifications doesn't exist.

Conclusion

These are the issues related to KB specifications and our countermeasures. Through this experience, I realized the importance of checking cloud service specifications before deciding on system design. I hope this article will be helpful to you.

Migrating Masking Database from Amazon Aurora to BigQuery: Performance Improvement and Security Implementation

ryo ariyama — Thu, 01 Jan 2026 12:51:18 +0000

Introduction

In a previous project, we used Amazon Aurora as our business database and synchronized this data to BigQuery for use as an analytics platform.
We had an opportunity to improve the performance of this synchronization process, and I'd like to share what we did.

Target Audience

This article is intended for:

Those who want to build an analytics platform with BigQuery
Those who want to learn about BigQuery security measures such as access restrictions
Those who want to know practical examples of data masking implementation

Background

In that project, we used Amazon Aurora as our application database. Since we stored sensitive information such as customer data, access to the database was restricted to specific operational terminals and limited operational members only.

On the other hand, the development team had the following needs, so we built a database with masked sensitive information daily and provided it to the development team:

Connect with backend applications for development
Query the database for troubleshooting and debugging purposes
Use as a data source for the data analytics platform

In the existing method, we created the masking database by cloning the production database and directly updating records with UPDATE statements.

We had been operating this way for some time, but the masking process performance had been degrading.
One reason was table lock contention occurring during data masking.
When directly updating records with UPDATE statements, tables need to be locked to synchronize with other tables.
During the masking process, we were executing UPDATE statements on all records of almost all tables in the database, which made table locks more likely to occur, resulting in massive write wait times.

The masking database was also used as a data source for the data analytics platform in addition to development and testing.
Using this analytics platform, we provided KPI dashboards to management by a specified time, but the masking process became a bottleneck, and there was concern that dashboard creation would regularly fail to meet the deadline in the near future.
Therefore, the challenge was to improve the masking process mechanism and reduce KPI dashboard creation time.

Improved Architecture

When considering improvement plans, we explored improvements from the following perspectives:

Offload to some storage other than Aurora
Perform masking processing by methods other than updating records

For the first perspective, since the analytics platform was using BigQuery as the data warehouse, we thought we could achieve this by transferring Aurora data to BigQuery.
For the second perspective, instead of updating database values, we thought we could achieve this by executing the masking logic that was previously done with UPDATE statements through SELECT statements when retrieving data.

Specifically, we adopted the following configuration:

We created a new project for the masking database and built a database equivalent to the masking database in BigQuery of this project with the following flow:

Export Aurora records to S3
Transfer data from S3 to BigQuery tables using BigQuery Data Transfer Service
Query Views with masking logic written based on the transferred data and link to analytics platform tables

The implemented View looks like this:

-- Conventional masking with UPDATE statement
UPDATE User
SET Name = "***"  -- Replace Name column in User table with *** uniformly as it contains sensitive information

-- Improved masking with SELECT statement
SELECT
  "***" AS Name
FROM
  User

By making these improvements, we achieved the following performance improvements and reduced processing time from about 3 hours to about 1 hour:

Improved overall query performance by offloading to BigQuery
Eliminated the need to synchronize with other tables by not writing values directly

While we could improve performance, this configuration required storing sensitive information in BigQuery.
In the next chapter, I'll introduce the security measures we implemented when storing sensitive information.

Implemented Security Measures

Applying Access Restrictions to BigQuery

To treat BigQuery equivalently to Aurora, we needed to apply the following access restrictions to datasets containing personal information:

Only infrastructure members and system accounts necessary for the service can access
Accessible only from specific operational terminals
BigQuery data export is restricted to the analytics platform project

The first requirement could be addressed with IAM restrictions, but we needed to consider another method for the second requirement.

VPC Service Controls is a service for managing access to GCP resources. By using this, we can place GCP resources in a private perimeter and restrict service access by specifying allowed IP addresses and IAM.
We used this service to allow operations on production resources by permitting operational member IDs and the global IP used by operational terminals as inbound rules.

Also, we use Terraform Cloud for deploying BigQuery resources, and we needed to make changes to the Terraform Cloud execution environment to accommodate these access restrictions.
When using Terraform Cloud, you can select the command execution environment, which is normally within the Terraform Cloud environment. In this case, the global IP of the execution environment also changes dynamically.
To fix the global IP permitted by inbound rules, we decided to use Terraform Cloud Agent. Using Agent allows you to specify the execution location of terraform commands to your own environment.
We built the Agent by constructing a GCE instance and starting a container within the instance.

While the Agent explanation and detailed configuration methods are described in the official documentation, by executing the following shell script as a GCE startup script based on the obtained Agent name and token, we could start the Agent when the instance runs:

#!/bin/bash

sudo apt-get update && sudo apt-get install -y \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update && sudo apt-get install -y \
  containerd.io \
  docker-ce \
  docker-ce-cli \
  docker-compose-plugin

TFC_AGENT_TOKEN=${tfc_agent_token} \
TFC_AGENT_NAME=${tfc_agent_name} \
docker run -d -e TFC_AGENT_TOKEN -e TFC_AGENT_NAME hashicorp/tfc-agent:latest

For the third requirement, since we were performing ETL across GCP using Cloud Workflows, we solved this by permitting the Cloud Workflows service account and analytics platform project as outbound rules.

Using Service Account Keys Without Issuing Credentials

This time, to access GCP resources from Terraform Cloud and AWS, we needed to use service accounts to operate resources from outside GCP.
When using service accounts from outside GCP such as other clouds, there's a method of issuing credentials, but we decided not to issue them for the following reasons:

If the key is leaked, anyone can execute the service account's permissions
Credentials have no expiration date, so rotation needs to be managed manually

Instead, we used Workload Identity to operate GCP resources from AWS and Terraform Cloud without issuing service account keys.
For example, when using Workload Identity with Terraform Cloud, the flow is as follows:

Following the official documentation, authentication is performed with the following steps:

Send an authentication token to GCP when executing Terraform commands
Verify the token's validity, and if no problems, issue a temporary token and send to Terraform Cloud
Set the temporary token in environment variables and execute commands
Discard the temporary token after command completion

The configuration method follows the GCP Configuration guide. The procedure roughly consists of:

Create Workload Identity Pool and Provider
- The attribute mapping values can be confusing, but the sample code is helpful
Create a service account and grant necessary permissions
Set the following values in Terraform Cloud environment variables:
- TFC_GCP_PROJECT_NUMBER: GCP project number
- TFC_GCP_PROVIDER_AUTH: true
- TFC_GCP_RUN_SERVICE_ACCOUNT_EMAIL: Email of the service account to use
- TFC_GCP_WORKLOAD_POOL_ID: Workload Pool ID
- TFC_GCP_WORKLOAD_PROVIDER_ID: Workload Provider ID

By using temporary authentication information instead of service account keys in this way, we avoided the risk of key leakage.

Conclusion

By migrating the masking database from Amazon Aurora to BigQuery, we were able to reduce processing time from 3 hours to 1 hour.
Additionally, by utilizing VPC Service Controls and Workload Identity, we were able to store sensitive information in BigQuery while ensuring security.

I hope this will be helpful for those who have similar challenges.

How I Stay Healthy While Working 250 Hours a Month as a Software Engineer

ryo ariyama — Tue, 30 Dec 2025 11:56:54 +0000

Introduction

I work as a software engineer with a full-time position plus freelance contracts, totaling around 250 hours per month. Compared to the typical 160-hour workload, that's a significant amount of time in front of a screen.

Over time, I've developed habits that help me maintain both productivity and health. Here's what has worked for me, ranked by impact.

What Has Worked

1. Let AI Agents Do the Heavy Lifting

This is the most impactful change I've made. The key insight: minimize cognitive load to maintain quality over long hours.

My typical day starts around 10:30 AM and ends around midnight, sometimes later when deadlines hit. By 7 PM, my brain is noticeably fatigued—similar to an athlete losing stamina in the second half of a game.

I noticed that bugs in my code correlated strongly with late-night commits. The root cause? Mental fatigue leading to shortcuts and oversights.

Writing complex logic and tests from scratch requires peak mental energy. So now, I delegate the implementation to coding agents (like GitHub Copilot Agent or Claude Code) and focus on reviewing their output. Reviewing is cognitively lighter than creating from scratch, which lets me maintain reasonable quality even when tired.

Since adopting this approach, bug reports have noticeably decreased—though I admit I haven't measured this quantitatively yet.

2. Take Supplements for Eye Health

Earlier this year, eye strain became a serious problem. Staring at monitors for 12+ hours daily was catching up with me.

After trying several solutions, lutein supplements made the biggest difference for my eye fatigue. For general health, I take Ebios (a brewer's yeast supplement popular in Japan) which helps with digestion and provides B vitamins. Reducing digestive stress has noticeably improved my daily energy levels.

3. Use a Standing Desk

I switched to a standing desk a while ago and rarely sit while working now.

Prolonged sitting puts significant strain on your lower back. I didn't notice it much when I was younger, but recently I've started feeling the effects. Senior engineers have repeatedly warned me: "Your back will give out suddenly one day."

Now I stand by default and only sit during breaks. Prevention is easier than recovery.

4. Build an Exercise Routine

This one's obvious but essential:

3 days/week: 1-hour strength training at the gym
Regular walks: About 2 hours of outdoor walking
Monthly hiking: 5-6 hour climbs on nearby mountains

Physical activity is the most effective way to offset the stress of knowledge work. I used to try unwinding with video games, but that just added more screen time and eye strain.

Hiking has a bonus benefit: you're constantly looking at distant scenery, which helps your eyes recover from close-up screen focus.

Final Thoughts

None of this is revolutionary. But consistently applying these habits has allowed me to sustain a demanding workload without burning out.

Technical skills matter, but so does knowing how to maintain the body and mind that use those skills. I hope this helps someone else navigating a similar schedule.

What are your strategies for staying healthy with long coding hours? I'd love to hear what works for you in the comments.

Building an Interpreter in Rust: A Journey Through Lexical Analysis, Parsing, and Evaluation

ryo ariyama — Sun, 21 Dec 2025 16:51:20 +0000

Introduction

I previously read Writing An Interpreter In Go and implemented the interpreter in Rust for learning purposes. I'll put what I learned in this article.

The Output

The completed interpreter can be found here. Since it's Rust source code mimicking Go, I named the repository imitation_interpreter.

GitHub Repository

Since it's for learning purposes, there are some minor issues like lack of newline support, but it can perform minimum viable programming including variable assignment, if statements, and function usage.

For the runtime environment, if you have cargo*2 installed you can run it locally. If you don't but want to try it, you can run the Dockerfile with the command described in the README.

About the Book

This book explains the implementation of an interpreter that evaluates a fictional C-like programming language called Monkey, centered around Go source code. By reading through this book, you can create an interpreter that evaluates programs with the following features:

C-style syntax
Variable binding
Integers and booleans
Arithmetic expressions
Built-in functions
First-class and higher-order functions
Closures
String data type
Array data type
Hash data type

The implementation divides the interpreter into components, implementing them step by step.

In the following sections, I'll explain the implementation of these components using actual source code snippets and diagrams.

Lexer (Lexical Analyzer)

The first thing we develop is the lexer. Lexical analysis means converting source code into a format that's easier to interpret - splitting the program into words and assigning a type to each word.

For example, if we have a program like let x = 5;, this program is split into words with types: let, x, =, 5, ;, and passed to the parser. These words are called tokens.

In the implementation, we define the types of words that Monkey outputs as tokens, and the lexer analyzes them through pattern matching.

pub enum TokenKind {
    ILLEGAL,     // ILLEGAL
    EOF,         // EOF

    // identifier and literal
    IDENT,       // IDENT
    INT,         // 123...

    // operator
    ASSIGN,      // =
    PLUS,        // +

    LPAREN,      // (
    RPAREN,      // )

    // keyword
    FUNCTION,    // FUNCTION
    LET,         // LET
}

pub struct Token {
    pub token_type: TokenKind,
    pub literal: String
}

pub struct Lexer {
    input:        &'a str,
    position:     usize, // current input position
    read_position: usize, // next input position
    ch:           u8, // a letter which is currently read
}

impl Lexer {
    pub fn new(input: &'a str) -> Self {
        let mut l = Lexer{
            input,
            position: 0,
            read_position: 0,
            ch: 0
        };
        l.read_char();
        return l;
    }

    pub fn new_token(token_type: TokenKind, ch: u8) -> Token {
        Token {
            token_type,
            literal: String::from_utf8(vec![ch]).unwrap(),
        }
    }

    pub fn next_token(&mut self) -> Token { 
        // Skip spaces as they have no meaning
        self.skip_whitespace();
        let token;
        match self.ch { // Pattern match source code to define tokens
            b'*' => {
                token = Self::new_token(TokenKind::ASTERISK, self.ch);
            }
            b'/' => {
                token = Self::new_token(TokenKind::SLASH, self.ch);
            }
            b'<' => {
                token = Self::new_token(TokenKind::LT, self.ch);
            }
            b'>' => {
                token = Self::new_token(TokenKind::GT, self.ch);
            }
            // ... more patterns
        }
        self.read_char();
        return token;
    }
}

The new() method receives the source code, and the next_token() method pattern matches words and returns token structures. Since Monkey doesn't assign meaning to spaces, we skip them in this method.

The result for let five = 5; looks like this:

Token { token_type: LET, literal: "let" }
Token { token_type: IDENT, literal: "five" }
Token { token_type: ASSIGN, literal: "=" }
Token { token_type: INT, literal: "5" }
Token { token_type: SEMICOLON, literal: ";" }

Parser (Syntax Analyzer)

Next, we develop the parser. By implementing this parser, we can correctly calculate expressions like 1 + 2 * 3 with the proper precedence.

In this parser implementation, we build a tree structure to simply represent the program's precedence. Taking 1 + 2 * 3 as an example, we get the following tree:

For more complex calculations, we nest the tree deeper. For example, 4 + 5 * (6 - 7) + 8 * 9 becomes:

This tree is called an Abstract Syntax Tree (AST), and we calculate it from the deepest left to right. It's called "abstract" because the tree is represented with only the minimum necessary elements. For example, parentheses are necessary for calculation but are redundant for building the tree, so they don't appear in the AST.

I used arithmetic operations as an example to explain AST, but general programs are also interpreted by creating this AST.

In the implementation, we parse the program with Program as the root, decomposing it into Statements and Expressions. A statement is a language element with no return value that's complete by itself, while an expression has a return value and becomes part of a statement.

For example, when parsing let x = 1 * (2 + 3);, we get an AST like the following. In this case, the statement is let x = 1 * (2 + 3);, and the expression is 1 * (2 + 3). 1 * (2 + 3) is part of the let statement, and when executed, returns 5. Understanding this might be a bit difficult, but writing expressions as ASTs on paper makes it easier to understand.

In the implementation, we create Statement and Expression enums, exhaustively define patterns, and the parser analyzes them through pattern matching.

pub struct Program {
    pub statements: Vec
}

pub enum Statement {
    LetStatement{identifier: Expression,
                 value: Expression},
    Return(Expression),
    ExpressionStatement(Expression),
    Block(Vec),
}

pub enum Expression {
    Identifier(String),
    String(String),
    Integer(i32),
    // ... more variants
}

pub struct Parser {
    lexer: lexer::Lexer,
    current_token: Token,
    next_token: Token,
}

impl Parser {
    pub fn new(l: lexer::Lexer) -> Self {
        let mut p = Parser{
            lexer: l,
            current_token: Token{token_type: TokenKind::DEFAULT, literal: "default".to_string()},
            next_token: Token{token_type: TokenKind::DEFAULT, literal: "default".to_string()},
        };
        p.next_token();
        p.next_token();
        p
    }

    pub fn parse_program(&mut self) -> Result {
        let mut statements: Vec = vec![];

        // read token until it reaches the end of the sentence
        while !self.is_current_token(TokenKind::EOF){
            let statement = self.parse_statement()?;
            statements.push(statement);
            self.next_token();
        };
        Ok(Program {statements: statements})
    }

    fn parse_statement(&mut self) -> Result {
        match self.current_token.token_type {
            TokenKind::LET => {
                Ok((self.parse_let_statement()?))
            },
            TokenKind::RETURN => {
                Ok(self.parse_return_statement()?)
            },
            _ => {
                Ok(self.parse_expression_statement()?)
            }
        }
    }

    fn parse_expression(&mut self, precedence: Precedence) -> Result {
        let mut exp = match self.current_token.token_type {
            TokenKind::IDENT => {Expression::Identifier(self.parse_identifier()?)},
            TokenKind::STRING => {Expression::String(self.parse_string()?)},
            TokenKind::INT => Expression::Integer(self.parse_integer()?),
            TokenKind::TRUE => Expression::Bool(true),
            _ => return Err(Errors::TokenInvalid(self.current_token.clone()))
        };
        // ... more logic
    }
}

The new() method receives the lexically analyzed result, and the parse_program() method recursively constructs the AST.

The result for let x = 1 * (2 + 3) looks like this:

LetStatement { 
    identifier: Identifier("x"),
    value: InfixExpression { 
        left_expression: Integer(1),
        operator: "*",
        right_expression: InfixExpression {
            left_expression: Integer(2),
            operator: "+",
            right_expression: Integer(3)
        }
    }
}

Evaluation

Finally, we develop the evaluation. In this book, we sequentially execute the AST constructed by the parser in order using the host language (Rust). This method is called a Tree-walking interpreter. As a side note, this method was actually used in Ruby 1.8 and earlier. (From 1.9 onwards, to improve performance, they changed to a method that compiles the AST to bytecode and evaluates it on a virtual machine.)

By the way, from around this point I started to feel comfortable with Rust syntax and implementation became easier.

In the implementation, we define a type called Object to represent it in the host language, converting AST values to Objects.

pub enum Object {
    Identifier(String),
    String(String),
    Integer(i32),
    Boolean(bool),
    Return(Box),
    Let(Box),
    Array(Vec),
    // ... more variants
}

pub fn evaluate(&mut self, program: &ast::Program) -> Result {
    let mut result = Object::Default;
    // evaluate sentence per semicolon
    for statement in program.statements.iter() {
        result = self.evaluate_statement(statement)?;
        // if statement contains 'return', process should be broken and return value
        if let Object::Return(value) = result {
            return Ok(*value)
        }
        // if the result of evaluation, process should be broken
        if result == Object::Error(Errors::InvalidInfix){
            return Ok(result)
        }
    }
    Ok(result)
}

fn evaluate_statement(&mut self, statement: &ast::Statement) -> Result {
    match statement {
        ast::Statement::ExpressionStatement(expression) => self.evaluate_expression(expression),
        ast::Statement::Block(stmt) => self.evaluate_block_statements(stmt),
        ast::Statement::Return(expression) => {
            let return_expression = self.evaluate_expression(expression)?;
            Ok(Object::Return(Box::new(return_expression)))
        },
        ast::Statement::LetStatement{identifier, value} => {
            if let Expression::Identifier(identifier) = identifier {
                // if expression is identifier, evaluate value and append identifier as variable
                let evaluated_value = self.evaluate_expression(&value)?;
                let value = self.set(identifier.to_owned(), evaluated_value);
                return Ok(value)
            }
            Ok(Object::Null)
        },
        _ => Err(Errors::NodeError),
    }
}

fn evaluate_expression(&mut self, expression: &ast::Expression) -> Result {
    match expression {
        ast::Expression::String(value) => Ok(Object::String(value.to_owned())),
        ast::Expression::Integer(value) => Ok(Object::Integer(*value)),
        ast::Expression::Bool(bool) => Ok(Object::Boolean(*bool)),
        ast::Expression::Array(value) => {
            let array = self.evaluate_arguments(value.to_vec())?;
            Ok(Object::Array(array))
        }
        // ... more patterns
    }
}

The evaluate_expression() function continuously converts received expressions to Objects.

When evaluating let x = 5; x;, the result Integer(5) is returned. We want the terminal to display 5, so I decided to use Rust's Formatter feature, which nicely formats and displays values.

pub enum Object {
    Integer(i32),
    // ... more variants
}

impl fmt::Display for Object {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        match self {
            // Only the internal value of Integer is displayed
            Object::Integer(value) => write!(f, "{}", value),
            // ... more patterns
        }
    }
}

What I Did to Become Proficient in Rust

Rust is generally considered a language with higher learning costs compared to other languages. The main reason is that it's a feature-rich language, but separately, there's also less knowledge available on the web compared to other languages.

While it's impossible to explain all of Rust's syntax in this blog, I hope to help improve the knowledge gap a bit, so I'll briefly explain the books and documents I referenced in chronological order.

Before Implementation

Before starting development, I worked through the official documentation. Since it's free and very carefully written, I think it was excellent as an initial document to read.

Knowledge up to about chapter 11 seemed essential, so I read through it all, memorizing syntax while copying the number guessing game program from the early chapters. For later chapters, I only worked on parts that seemed relevant. Therefore, there are areas I barely touched, which I think is a challenge.

During Implementation

During implementation, I mainly referenced "Programming Rust" alongside the official documentation. This book is written for beginners so that even those without base knowledge of Rust like C can understand it, making it very accessible.

Rust has a more difficult impression regarding string handling and type conversion compared to other languages, but perhaps the author sympathized with this feeling, as they devoted many pages to carefully explaining it. It's also attractive that it contains sample code that seems useful in everyday development, such as database access and web API usage, so I can recommend it to those who want to use Rust in production.

I also searched for GitHub repositories doing similar things in Rust and referred to commit logs. Some might avoid this as if looking at problem set answers, but you can learn about notation and libraries not covered in books, and by following commit logs you can see how others avoided the same pitfalls you encountered, which was very helpful.

Things You Should Do When Writing Rust

Build a development environment using RLS (Rust Language Server)

Rust is a restrictive language due to its type system and unique ownership system. Therefore, you'll encounter many compile errors, and in this development too, I had the feeling that working code emerged from fixing errors (partly because I was inexperienced).

I usually use VSCode as my editor, and by installing the rls*3 extension, I could check syntax errors and types in the editor, which saved development time. Especially since Rust has type constraints but it's difficult to check types in the program, being able to check types in the editor was very helpful.

Also, when running tests, you can execute them at the function level, which was very convenient when debugging only newly developed functions in pattern matching.

Installation reference:
Rust Analyzer - Visual Studio Marketplace

Use the Result Type for Return Values

As an implementation strategy for the parser and evaluator, we pattern match tokens and execute functions to process them. Therefore, we implement functions step by step for each token and extend the match statement, but for unimplemented tokens, there's a problem that we can't return a return value with the appropriate type because it's not implemented.

In this case, we use the Result type, which can define return values for successful termination Ok(T) and abnormal termination Err(E). In the code example above, we define the Expression type defined as an enum for successful termination and the Errors type for abnormal termination as return values. Wanting to return errors within functions is common, such as in API implementations, and using the Result type allows high flexibility in implementation.

I think it becomes easier to handle errors if you define an Errors type in detail in a dedicated errors.rs file using an enum.

Conclusion

I was able to implement an interpreter in about 2,000 lines including test code, and it was excellent study as an introduction to language processing and Rust. (The preface mentioned aiming for something between a technical book and a blog, which seemed accurate.)

Particularly when implementing the parser logic and successfully parsing complex expressions, I felt I understood why calculation formulas work correctly. As its reputation suggests, Rust is a language with high learning costs, and I had quite a difficult time in the beginning, but learning Rust-specific syntax like the ownership system means learning programming at a lower layer, so as someone who usually develops with scripting languages, I now think it was a good language worth learning.

I just learned while writing this blog that this book has a sequel called "Writing A Compiler In Go," so to prevent the Rust knowledge I've acquired from fading away, I'm thinking of trying to make a compiler next time.

Footnotes

*1: A tool that can build, run, test, and manage packages of Rust code - essential for Rust development. Cargo Documentation

*3: Abbreviation for Rust Language Server, a backend server that provides various completions for Rust programs in IDEs and editors. RLS GitHub

BuildingRetrieval-AugmentedGenerationRAGSystemonAmazonBedrock

ryo ariyama — Mon, 15 Sep 2025 10:02:39 +0000

How to Build a RAG System with Amazon Bedrock

Introduction

I am currently implementing an AI Chatbot with RAG functionality using Amazon Bedrock for a project.
Now that things have settled down, I'd like to document what I've researched as a reference.
Since LLM and RAG functionality cover a wide range, this article provides a conceptual overview.

What is RAG Functionality?

Basic Knowledge

Before diving into the explanation, let me explain RAG.
RAG stands for "Retrieval-Augmented Generation."
RAG is a technique for improving the capabilities of large language models (LLMs), and it works with the following mechanism:

Retrieval: Search for information related to the user's question from external databases or documents
Augmented: Add the relevant information obtained from the search to the LLM input
Generation: The LLM generates responses using both the original question and the searched information

Main Benefits of RAG

When using traditional LLMs, responses are generated within the scope of knowledge from data used during training. In this case, if the information you want to know through dialogue is recent content or if the knowledge and terminology are not common, you may not be able to get correct answers.
On the other hand, by using RAG, you can generate responses that better reflect local context by adding external documents as reference information.

Here are some practical examples:

Chatbots that search internal company documents to answer questions
Research support systems that reference the latest academic papers
Customer support that searches product manuals for information

RAG has become an important technology for building AI systems that can better meet user-specific requirements by combining the general knowledge of LLMs with specific contexts and the latest information.

How to Build with AWS?

About Bedrock

AWS has a service called Bedrock for generative AI, which we will use.
Bedrock is a service that enables access to various LLM models like Claude and makes them accessible via API.
Bedrock has various services, but to realize dialogue with LLMs using RAG functionality, you need to use the following services:

Agent: Provides API access to LLM models for dialogue and response generation.
KnowledgeBase: A service for RAG that connects to data sources and vector stores. Performs data vectorization.

Using Agent, you can have general LLM conversations via API, but by adding KnowledgeBase, you can have conversations using RAG functionality.

How KnowledgeBase Works

Bedrock KnowledgeBase is an orchestration tool to realize RAG functionality.
It provides functions such as importing training data and data retrieval to improve LLM dialogue performance.
KnowledgeBase connects to various data sources, vectorizes documents, and registers them in vector databases.
Data sources include S3, SharePoint, Confluence, Salesforce, etc., and vector databases can use OpenSearch, Pinecone, Amazon Aurora, etc.

Vectorization Process

In RAG systems, documents stored as context are converted to numerical vectors and stored.
Vectorization is an important process that converts text to numerical vectors in a format that computers can understand.
The following steps are involved in vectorization:

Chunk splitting
Embedding model selection
Vector conversion

1. Chunk Splitting

First, long documents are divided into appropriate sizes according to the following policy:

Size: Usually around 500-1000 characters
Overlap: Provide 50-100 character overlap between chunks
Boundaries: Split at paragraph or sentence breaks, not in the middle of sentences

Example:
Original document: "Amazon Bedrock is a generative AI service. It provides access to various models..."
↓
Chunk1: "Amazon Bedrock is a generative AI service. It provides access to various models and can be used via API."
Chunk2: "It can be used via API. Main features include Agent functionality and KnowledgeBase functionality."

2. Embedding Model Selection

Next, select an embedding model. Use this model to vectorize the chunked text.
Bedrock uses embedding models such as Amazon Titan Embeddings:

Amazon Titan Embeddings: General-purpose text embedding
Cohere Embed: Multi-language support

3. Vector Conversion

Convert each chunk to vectors and store them in the vector store. When stored, vectors are typically saved as arrays of floating-point numbers.
During dialogue creation, similarity calculations are performed from these vectors to search for information needed for dialogue.
During search, the user's question is also vectorized using the same method, and cosine similarity or Euclidean distance is calculated to search for documents with high similarity.

Sample Implementation

Now let's write a sample implementation for Bedrock KnowledgeBase.
The implementation language is Python, the vector store is OpenSearch managed cluster, and the data source is S3.
The implementation flow is as follows:

Create index in OpenSearch
Create VectorStore
Create data source
Ingest vector data into VectorStore
Execute agent combined with KnowledgeBase

Create Index in OpenSearch

First, as preparation, let's create an index in OpenSearch.
There are various configuration items, but there is a sample in the official documentation, so it's good to customize based on this.

{
    "settings": {
        "index": {
            "knn": true
        }
    },
    "mappings": {
        "properties": {
            "<vector-name>": {
                "type": "knn_vector",
                "dimension": <embedding-dimension>,
                "data_type": "binary",          # Only needed for binary embeddings
                "space_type": "l2" | "hamming", # Use l2 for float embeddings and hamming for binary embeddings
                "method": {
                    "name": "hnsw",
                    "engine": "faiss",
                    "parameters": {
                        "ef_construction": 128,
                        "m": 24
                    }
                }
            },

            "AMAZON_BEDROCK_METADATA": {
                "type": "text",
                "index": "false"
            },
            "AMAZON_BEDROCK_TEXT_CHUNK": {
                "type": "text",
                "index": "true"            
            }
        }
    }
}

For explanations of values like ef_construction, the OpenSearch official documentation has explanations and recommended parameter combinations, which I think would be helpful as reference.
https://opensearch.org/blog/a-practical-guide-to-selecting-hnsw-hyperparameters/#:~:text=efficiency%20%5B3%2C%204%5D.-,Recommended,-HNSW%20configurations

Create VectorStore

Next, use create_knowledge_base to create a VectorStore.

import boto3

def main():
    client = boto3.client(
        "bedrock-agent", region_name='<your-aws-region>'
    )
    bedrock_kb_role_arn = "<KB-IAMrole-arn>"
    embedding_model_arn = "<KB-Embedding-model-arn>"
    opensearch_domain_endpoint = "https://<os-domain-endpoint>"
    opensearch_domain_arn = "<os-domain-arn>"
    vector_index_name = "<vector-store-index-name>"
    vector_field_name = "<vector_field_name>"
    response = client.create_knowledge_base(
                    name="kb-sample",
                    roleArn=bedrock_kb_role_arn,
                    knowledgeBaseConfiguration={
                        "type": "VECTOR",
                        "vectorKnowledgeBaseConfiguration": {
                            "embeddingModelArn": embedding_model_arn,
                            # Set this if implementing multimodal processing for images like PDFs
                            "supplementalDataStorageConfiguration": {
                                "storageLocations": [
                                    {
                                        "s3Location": {
                                            "uri": "<s3-multimodal-bucket-arn>"
                                        },
                                        "type": "S3",
                                    }
                                ]
                            }
                        },
                    },
                    storageConfiguration={
                        "type": "OPENSEARCH_MANAGED_CLUSTER",
                        "domainEndpoint": opensearch_domain_endpoint,
                        "domainArn": opensearch_domain_arn,
                        "vectorIndexName": vector_index_name,
                        "fieldMapping": {
                            "vectorField": vector_field_name,
                            "textField": "AMAZON_BEDROCK_TEXT_CHUNK",
                            "metadataField": "AMAZON_BEDROCK_METADATA",
                        },
                    }
                )
    ...

If you want to vectorize images embedded in PDFs, you need to set an S3 bucket for multimodal processing in supplementalDataStorageConfiguration.

Create Data Source

Next, create a data source for the vector store. Here's an example when the text chunk strategy is set to fixed_size.

import boto3

def main():
    client = boto3.client(
        "bedrock-agent", region_name='<your-aws-region>'
    )
    knowledge_base_id = <knowledge-base-id>
    datasource_s3_arn = <S3-arn>
    aws_account_id = <aws-account-id>
    inclusion_prefixes = <inclusion-prefixes-array>
    foundation_model_arn = <foundation-model-arn>
    chunking_max_tokens = <chunking-max-tokens>
    overlap_percentage = <overlap-percentage>
    response = client.create_data_source(
        knowledgeBaseId=knowledge_base_id,
        name=<knowledge-base-name>,
        dataSourceConfiguration={
            "type": "S3",
            "s3Configuration": {
                "bucketArn": datasource_s3_arn,
                "bucketOwnerAccountId": aws_account_id,
                "inclusionPrefixes": inclusion_prefixes,
            },
        },
        vectorIngestionConfiguration={
            "chunkingConfiguration": {
                "chunkingStrategy": "FIXED_SIZE",
                "fixedSizeChunkingConfiguration": {
                    "maxTokens": chunking_max_tokens,
                    "overlapPercentage": overlap_percentage
                }
            },
            "parsingConfiguration": {
                "bedrockFoundationModelConfiguration": {
                    "modelArn": foundation_model_arn,
                    "parsingModality": "MULTIMODAL",
                },
                "parsingStrategy": "BEDROCK_FOUNDATION_MODEL",
            },
        }
    )
    ...

Ingest Vector Data into VectorStore

Next, ingest vector data. This time, we'll use ingest_knowledge_base_documents to specify S3 files and insert vector data.

import boto3

def main():
    client = boto3.client(
        "bedrock-agent", region_name='<your-aws-region>'
    )
    knowledge_base_id = <knowledge-base-id>
    datasource_id = <datasource-id>
    s3_uri = <s3-uri-where-the-source-file-is-located>
    metadata_s3_uri = f"{s3_uri}.metadata.json"
    aws_account_id = <aws-account-id>

    response = client.ingest_knowledge_base_documents(
        knowledgeBaseId=knowledge_base_id,
        dataSourceId=datasource_id,
        documents=[
            {
                # Set this if you want to add custom metadata
                'metadata': {
                    'type': 'S3_LOCATION',
                    's3Location': {
                        'uri': metadata_s3_uri,
                        'bucketOwnerAccountId': aws_account_id
                    }
                },
                'content': {
                    'dataSourceType': 'S3',
                    's3': {
                        's3Location': {
                            'uri': s3_uri
                        }
                    }
                }
            },
        ]
    )

When importing files, you can get page numbers and source file information by default, but you can also set custom metadata. You can add metadata to the vector store by preparing a file named filename.metadata.json in the same directory as the target file and specifying that path.
https://docs.aws.amazon.com/bedrock/latest/userguide/kb-metadata.html

Execute Agent Combined with KnowledgeBase

Finally, let's call the agent for dialogue. You can easily invoke dialogue by calling invoke_inline_agent. To simplify the explanation, I'll only include the parts related to KnowledgeBase.

import boto3

def main():
    client = boto3.client(
        "bedrock-agent-runtime", region_name='<your-aws-region>'
    )

    vector_store_id = <vector-store-id>
    override_search_type = <override-search-type>
    response = client.invoke_inline_agent(
        knowledgeBases=[
            {
                "knowledgeBaseId": vector_store_id,
                "description": "Knowledge base for document retrieval",
                "retrievalConfiguration": {
                    "vectorSearchConfiguration": {
                        "overrideSearchType": override_search_type
                    }
                },
            }            
        ]
    )

After the API search, you can get metadata of referenced documents from the following fields in the response:

{
    'completion': EventStream({
        'trace': {
            'trace': {
                'orchestrationTrace': {
                    'observation': {
                        'knowledgeBaseLookupOutput': {
                            'metadata': {
                                'clientRequestId': 'string',
                                'endTime': datetime(2015, 1, 1),
                                'operationTotalTimeMs': 123,
                                'startTime': datetime(2015, 1, 1),
                                'totalTimeMs': 123,
                                'usage': {
                                    'inputTokens': 123,
                                    'outputTokens': 123
                                }
                            },
                            'retrievedReferences': [
                                {
                                    'content': {
                                        'byteContent': 'string',
                                        'row': [
                                            {
                                                'columnName': 'string',
                                                'columnValue': 'string',
                                                'type': 'BLOB'|'BOOLEAN'|'DOUBLE'|'NULL'|'LONG'|'STRING'
                                            },
                                        ],
                                        'text': 'string',
                                        'type': 'TEXT'|'IMAGE'|'ROW'
                                    },
                                    'location': {
                                        'confluenceLocation': {
                                            'url': 'string'
                                        },
                                        'customDocumentLocation': {
                                            'id': 'string'
                                        },
                                        'kendraDocumentLocation': {
                                            'uri': 'string'
                                        },
                                        's3Location': {
                                            'uri': 'string'
                                        },
                                        'salesforceLocation': {
                                            'url': 'string'
                                        },
                                        'sharePointLocation': {
                                            'url': 'string'
                                        },
                                        'sqlLocation': {
                                            'query': 'string'
                                        },
                                        'type': 'S3'|'WEB'|'CONFLUENCE'|'SALESFORCE'|'SHAREPOINT'|'CUSTOM'|'KENDRA'|'SQL',
                                        'webLocation': {
                                            'url': 'string'
                                        }
                                    },
                                    'metadata': {
                                        'string': {...}|[...]|123|123.4|'string'|True|None
                                    }
                                },
                            ]
                        },
                    },
                },
            }
        },
    }),
}

Among these, retrievedReferences[*].metadata contains metadata about referenced files such as:

x-amz-bedrock-kb-source-uri: S3 URL
x-amz-bedrock-kb-chunk-id: Chunk Id used to generate response
x-amz-bedrock-kb-data-source-id: Datasource id
x-amz-bedrock-kb-document-page-number: page number of the source document In addition, if you specify a metadata.json file, metadata will be added here.

Conclusion

This has been an explanation of RAG functionality using KnowledgeBase.
KnowledgeBase and RAG functionality have much to discuss, and it's impossible to cover everything in this blog, so I've only introduced the basics.
If I have time, I'd like to touch on parameter tuning for OpenSearch index as well.
I hope this article will be helpful for everyone.

Reflections on My First 10 Blog Posts: Lessons, Challenges, and Next Steps

ryo ariyama — Sun, 06 Jul 2025 16:33:12 +0000

Introduction

I’ve now published 10 blog posts, so I’d like to take a moment to reflect on the experience so far.

What Went Well

As I wrote in this post, I originally started blogging because I didn’t have many opportunities to share my thoughts or output my ideas.

My purpose hasn’t really changed since then, but by writing these posts, I’ve been able to review and reinforce what I learn day to day — and that’s been really helpful.
Personally, I tend to forget the details of what I’ve implemented after about a month, but I feel like I remember the contents of my blog posts quite well.
Also, writing a blog often requires doing additional research, so I feel like I’ve deepened my knowledge in the process.

On top of that, this experience has helped me understand what kind of engineer I am.
I usually work across different technical areas and don’t have a clear title like “Backend Engineer.” or “Infrastructure Engineer.” So I hadn’t really known how to define myself. But looking back at the posts I’ve written, many of them are about infrastructure-related topics — so I guess I’m more of an infrastructure engineer who can also do backend development.
I think this could be useful when I look for new opportunities in the future — though, unfortunately, I haven’t actually had any interviews during this time.

Challenges

While there have been a lot of positives, there have definitely been challenges too.
First of all, to be honest, publishing a blog post every week is pretty tough.
I’ve often found myself thinking that I could have spent my precious days off doing something else — for example, working on a side project or planning a personal trip instead.

Also, although I didn’t mention this in my introduction post, I have a personal rule: I must write each post within two hours.
The reasons are simple — I don’t want to spend too much of my weekends on blogging, and I feel that spending more time would just result in longer, possibly harder-to-read posts.
Keeping this rule has actually been pretty hard for me. But on the bright side, it forced me to find ways to write more efficiently, which I think was a good thing.

For example, I used to create diagrams myself in draw.io, but I switched to using AI to generate ASCII art instead.
I also experimented with generating drafts based on my headings and outlines using AI, then adding my own explanations.
Of course, most of the sample code is also generated by AI and I even ask AI to review my drafts.
I still come up with the article titles myself — though sometimes I get suggestions from AI too.
Obviously, it wouldn’t make sense to have AI write a reflection post like this one, so I wrote this one entirely by myself.

Sometimes I wonder if relying on AI like this is a bit of a shortcut, but if I can achieve my goal in a short amount of time, I think it’s good enough.

What’s Next

I still believe that outputting my thoughts is important, so I plan to keep blogging until I run out of ideas.
Maintaining motivation to post regularly is really tough, so I hope to keep exploring better ways to stay motivated — maybe I’ll write about that too.

Ideally, I hope this blog will help me find new work opportunities or expand my network.

Lastly, I’d like to say thank you to the people who shared my posts and reacted positively on SNS like X and LinkedIn.
Thanks to them, some of my articles reached a wider audience than I ever expected.
I really appreciate it.

That’s all for now.

Introduction to LocalStack

ryo ariyama — Sun, 29 Jun 2025 13:23:03 +0000

Introduction

Recently, I've been increasingly using LocalStack to set up local development environments at work, so I'd like to explain the reasons behind this.

What is LocalStack?

LocalStack is a tool that can emulate AWS services on your local machine.
https://www.localstack.cloud/

An emulator is software that mimics the behavior of other software or systems and can be used as an alternative. By using LocalStack, you can reproduce AWS services like S3 and Lambda in your local environment.

The main use cases are individual learning and testing in local environments.

The free version supports around 30 services, which is sufficient for individual learning purposes. However, for commercial services or web services of a certain scale, I believe you would need to use the Base plan.

Motivation for Adoption

In our usual backend development, we follow the flow of developing in a local environment → deploying to the cloud for functional verification.

When developing in a local environment, we set up AWS profiles and SSH tunneling to resolve authentication and network issues.

This is sufficient for regular development, but we decided to use LocalStack due to the following requirements:

We want members from other departments to run the applications we operate, but for various reasons, we don't want to grant them AWS account permissions.
We want to use AI agents like Devin for development, but it's difficult to resolve network and authentication issues.
Even if we could resolve these issues, we want to avoid excessive usage of pay-per-use services due to AI agent malfunctions.

Environment Setup

Environment setup is done using Docker and Docker Compose. Here is the image of the difference between AWS Cloud and Localstack.

// AWS Cloud
┌─────────────────┐           ┌─────────────────────────────────┐
│  Local env      │           │         AWS Cloud               │
│                 │           │                                 │
│  ┌───────────┐  │  HTTPS    │  ┌─────────┐  ┌─────────┐       │
│  │Backend    │  │ ────────► │  │   S3    │  │ Lambda  │       │
│  │Code       │  │  Request  │  │         │  │         │       │
│  └───────────┘  │           │  └─────────┘  └─────────┘       │
│                 │           │                                 │
│  ┌───────────┐  │           │  ┌─────────┐  ┌─────────┐       │
│  │Terraform  │  │ ────────► │  │DynamoDB │  │   SQS   │       │
│  │           │  │           │  │         │  │         │       │
│  └───────────┘  │           │  └─────────┘  └─────────┘       │
└─────────────────┘           └─────────────────────────────────┘

// Localstack
┌───────────────────────────────────────────────────────────────┐
│                       Local env                               │
│                                                               │
│  ┌───────────┐           ┌─────────────────────────────────┐  │
│  │Backend    │  HTTP     │        Docker Container         │  │
│  │Code       │ ────────► │                                 │  │
│  └───────────┘  :4566    │  ┌──────────┐  ┌──────────┐     │  │
│                          │  │   S3     │  │ Lambda   │     │  │
│  ┌───────────┐           │  │(emulated)│  │(emulated)│     │  │
│  │Terraform  │ ────────► │  └──────────┘  └──────────┘     │  │
│  │           │           │                                 │  │
│  └───────────┘           │  ┌──────────┐  ┌──────────┐     │  │
│                          │  │DynamoDB  │  │   SQS    │     │  │
│                          │  │(emulated)│  │(emulated)│     │  │
│                          │  └──────────┘  └──────────┘     │  │
│                          │                                 │  │
│                          │        LocalStack Process       │  │
│                          └─────────────────────────────────┘  │
└───────────────────────────────────────────────────────────────┘

The official documentation provides installation instructions, and you can follow these to set it up.

Prepare a YAML file like the following:

version: "3.8"

services:
  localstack:
    container_name: localstack
    image: localstack/localstack-pro:latest
    ports:
      - "4566:4566"
    environment:
      - LOCALSTACK_AUTH_TOKEN="${LOCALSTACK_AUTH_TOKEN}" # required for base, pro...
      - SERVICES=<AWS services> # comma delimited AWS services list like s3, lambda, batch...
      - DEFAULT_REGION=<aws-region>
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "./:/tmp/app"
      - "/var/run/docker.sock:/var/run/docker.sock"

Start the Docker process with docker-compose up -d. This process creates emulated AWS endpoints, and you can create AWS services by sending requests to these endpoints.

There's an AWS CLI called awslocal that can be used with LocalStack, which allows you to easily create resources. To create an S3 bucket, execute the following command:

awslocal s3 mb s3://${S3_BUCKET_NAME}

You can also use Terraform with LocalStack.

In the provider configuration, change the endpoint to LocalStack's endpoint as follows. The default endpoint is http://localhost:4566, so use that:

provider "aws" {
  region = "<region>"
  endpoints { // specify localstack endpoint to the services you use.
    dynamodb = "http://localhost:4566"
    s3       = "http://localhost:4566"
    lambda   = "http://localhost:4566"
    ...
  }

  s3_use_path_style = true

  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  access_key = "test"
  secret_key = "test"
}

After configuring the above settings, execute terraform apply to create emulated resources using Terraform with LocalStack's endpoint.

That covers how to create AWS resources.

To use AWS services created with LocalStack from backend code, change the AWS service endpoint to LocalStack's endpoint, similar to Terraform. Here's sample code for initializing an S3 client in Go:

var configOptions []func(*config.LoadOptions) error

cfg, err := config.LoadDefaultConfig(context.TODO(), configOptions...)
if err != nil {
    return nil, err
}
localstackEndpoint := "http://localhost:4566"
cfg.BaseEndpoint = aws.String(localstackEndpoint)

client := s3.NewFromConfig(cfg)
...

Conclusion

That's all there is to it. It can be used with relatively simple changes. The use of AI agents, as mentioned in the motivation section, will likely increase in the future, so using LocalStack as a measure to prevent excessive service usage seems like a good strategy.

I hope this article will be helpful to everyone.

Introduction to Data Analytics Platform with Databricks

ryo ariyama — Sun, 22 Jun 2025 17:31:21 +0000

Introduction

Recently, I've had several opportunities to build data analytics platforms using Databricks.
As a reference, I'd like to summarize what I've learned.

Prerequisites for this Article

This article explains the basic concepts of Databricks.
A construction article using sample scripts will be introduced in a separate article.
It is written with the assumption of building in combination with AWS.

What is Databricks

Databricks is a data analytics platform based on Apache Spark.
Databricks is a platform that can handle everything related to data analytics, including data collection, processing, analysis, and visualization.
Construction can be performed using services from multiple cloud vendors such as AWS, Azure, and GCP.

Architecture Overview

Databricks has an account as the top-level resource, with workspaces underneath it.
The account manages billing, users, and workspaces.
A workspace is where actual data analysis takes place, managing notebooks, jobs, SQL warehouses, and more.
Using AWS Organizations as an example, the Databricks account corresponds to the management account, and workspaces correspond to individual AWS accounts.

┌──────────────────────────────────────────────────────────────────┐
│                    Databricks Account                            │
│                                                                  │
│  ┌─ Management functions ──────────────────────────────────┐     │
│  │ • Billing management                                    │     │
│  │ • User management                                       │     │
│  │ • Workspace management                                  │     │
│  │ • Security configuration                                │     │
│  └─────────────────────────────────────────────────────────┘     │
│                                                                  │
│  ┌─ Workspace A ──────────────┐  ┌─ Workspace B ──────────────┐  │
│  │                            │  │                            │  │
│  │ ┌─ Control Plane ────────┐ │  │ ┌─ Control Plane ────────┐ │  │
│  │ │ • Web UI               │ │  │ │ • Web UI               │ │  │
│  │ │ • Job Scheduler        │ │  │ │ • Job Scheduler        │ │  │
│  │ │ • Metadata Store       │ │  │ │ • Metadata Store       │ │  │
│  │ │ • Security Manager     │ │  │ │ • Security Manager     │ │  │
│  │ └────────────────────────┘ │  │ └────────────────────────┘ │  │
│  │            │               │  │            │               │  │
│  │            ▼               │  │            ▼               │  │
│  │ ┌─ Compute Plane ────────┐ │  │ ┌─ Compute Plane ────────┐ │  │
│  │ │ • Clusters             │ │  │ │ • Clusters             │ │  │
│  │ │ • SQL Warehouses       │ │  │ │ • SQL Warehouses       │ │  │
│  │ │ • Notebooks            │ │  │ │ • Notebooks            │ │  │
│  │ │ • Jobs                 │ │  │ │ • Jobs                 │ │  │
│  │ └────────────────────────┘ │  │ └────────────────────────┘ │  │
│  └────────────────────────────┘  └────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────┘

Within a workspace, there are two main components: the control plane for management and the compute plane for data processing.
The control plane is a service provided by Databricks - a web application managed by Databricks.
The compute plane consists of resources for actual data processing. There are two configuration methods:

Classic configuration: Create clusters as EC2 instances within the user's AWS account
Serverless configuration: Execute resources within Databricks-managed AWS accounts (utilizing underlying AWS resources invisibly to users)

The following page provides a clear explanation:
Databricks architecture overview

Databricks Features

What are the benefits of adopting Databricks as a data analytics platform? The main Databricks-specific features for analytics platforms include:

Delta lake
Unity Catalog

Delta Lake is an open-source storage format.
It consists of Apache Parquet format files, delta logs (JSON format), and metadata, enabling ACID transactions that were not possible with traditional data lakes and data warehouses.
Specifically, while traditional data lakes could potentially retrieve inconsistent data when reading during data updates, Delta Lake enables reading data in a consistently coherent state through its transaction functionality.
Databricks delta lake
Databricks provides default support for Delta Lake.

Unity Catalog is a data governance feature provided by Databricks.
It enables access management and quality management for storage across workspaces.
Databricks unity catalog

Specific use cases include:

Why Choose Databricks

While simple analytics systems that store data in S3 or DWH and analyze with Python or SQL could be implemented using only AWS services, Databricks offers the following advantages:

Strong data consistency guarantee: Transaction functionality through Delta Lake
Integrated data quality management: Quality monitoring and governance through Unity Catalog
Multi-cloud support: Enables integrated data collection and analysis even when there are multiple cloud environments within an organization These features enable the operation of high-quality data platforms, which I believe are the main reasons for adopting Databricks.

Conclusion

That's all for now. This article briefly explained the basic knowledge about Databricks.
I plan to publish an actual construction article using Terraform in the future.

How We Manage ECS with Terraform and GitHub Repos

ryo ariyama — Sun, 08 Jun 2025 12:43:17 +0000

Introduction

This may seem like a thoroughly discussed topic already, but since the same conversation recently came up internally at work, I decided to write about it.

In our day-to-day operations, we use Amazon ECS as the platform for running our applications. AWS resources are provisioned with Terraform, and we manage our code on GitHub with separate repositories for infrastructure and backend.

A frequent point of discussion is how much of the ECS-related resources should be managed by the infrastructure side, which I’ll break down in this article.

Intended audience: Engineers using ECS and Terraform, especially those working in teams with separate infrastructure and application repositories

Estimated reading time: 7 minutes

How We Manage It

To cut to the chase: ECS task definitions and ECS service definitions are created in the infrastructure repository but updated from the backend repository. Everything else is managed in the infrastructure repository.

Before explaining why, let’s list the common triggers for updating these definitions:

Changing the tag of a container image in ECR
Updating environment variables or CPU/MEM parameters
Deploying a new revision of the task definition via the service definition

Especially with the first point, changing the container image tag typically results from updates to backend source code rather than infrastructure changes. Because of this dependency on application implementation, we believe it’s better not to manage such values from the infrastructure repository. Instead, they should be handled in the backend repository.

If you’re unsure what this looks like, consider a case where another team manages the infrastructure repository and also owns the ECS task definitions. When you want to deploy:

You build the container and push the image to ECR
You inform the infra team of the new image tag so they can create a new task definition
They update the ECS service to use the new task definition

Doing this every time you deploy creates a lot of communication overhead. Ideally, both teams should be able to deploy independently. That’s why we create the ECS service and task definitions in the infrastructure repo but handle updates from the backend.

Sample Implementation

Here’s a reference implementation, starting with Terraform code for ECS:

# ECS task definition 
resource "aws_ecs_task_definition" "main" {
  family             = var.container_name
  task_role_arn      = var.task_role_arn
  execution_role_arn = var.exec_role_arn

  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = var.task_cpu
  memory                   = var.task_mem

  lifecycle {
    ignore_changes = [container_definitions]
  }

  container_definitions = jsonencode([
    {
      essential   = true
      name        = var.container_name
      image       = "${var.ecr_repo_url}:latest"
      cpu         = 0
      environment = []
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          awslogs-create-group  = "true"
          awslogs-group         = "/ecs/example"
          awslogs-region        = var.region
          awslogs-stream-prefix = "ecs"
        }
      }
      portMappings = [
        {
          appProtocol   = var.container_protocol
          containerPort = var.container_port
          hostPort      = var.container_port
          name          = "${var.container_name}_port"
          protocol      = "tcp"
        }
      ]
    }
  ])
}

# ECS service definition
resource "aws_ecs_service" "main" {
  name                               = var.service_name
  cluster                            = var.cluster_id
  deployment_maximum_percent         = var.deployment_maximum_percent
  deployment_minimum_healthy_percent = var.deployment_minimum_healthy_percent
  desired_count                      = var.desired_count
  enable_execute_command             = var.enable_execute_command
  health_check_grace_period_seconds  = var.health_check_grace_period_seconds
  launch_type                        = "FARGATE"
  platform_version                   = "1.4.0"
  task_definition                    = aws_ecs_task_definition.main.arn

  deployment_controller {
    type = "ECS"
  }

  load_balancer {
    container_name   = var.container_name
    container_port   = var.container_port
    target_group_arn = var.tg_arn
  }

  network_configuration {
    assign_public_ip = false
    security_groups  = [var.sg_id]
    subnets          = var.subnets
  }

  lifecycle {
    ignore_changes = [
      task_definition,
      desired_count
    ]
  }
}

The lifecycle.ignore_changes block ensures that attributes dependent on the backend won’t show as diffs in Terraform plans. In this example, we ignore container versions and environment variables to allow safe initial creation.

For backend deployments, we use ecspresso, a CLI tool that lets you deploy using JSON definitions for task and service configurations.

Here’s what the definitions look like:

// ecs_task_def.json
{
  "containerDefinitions": [
    {
      "cpu": {{ env `ECS_CPU` `256` }},
      "essential": true,
      "image": "{{ env `ECR_REPO_IMAGE_URL` }}:{{ env `IMAGE_TAG` }}",
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-create-group": "true",
          "awslogs-group": "/ecs/example",
          "awslogs-region": "ap-northeast-1",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "memory": {{ env `ECS_MEMORY` `512` }},
      "name": "example",
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80,
          "protocol": "tcp"
        }
      ],
      "secrets": [
        {
          "name": "ENV",
          "valueFrom": "{{ env `SECRETS_MANAGER_ARN` }}:ENV::"
        }
      ]
    }
  ],
  "cpu": {{ env `ECS_CPU` `256` }},
  "executionRoleArn": "{{ env `EXEC_ROLE_ARN` }}",
  "family": "example",
  "memory": {{ env `ECS_MEMORY` `512` }},
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "taskRoleArn": "{{ env `TASK_ROLE_ARN` }}"
}

// ecs_service_def.json
{
  "deploymentConfiguration": {
    "deploymentCircuitBreaker": {
      "enable": false,
      "rollback": false
    },
    "maximumPercent": 200,
    "minimumHealthyPercent": 100
  },
  "desiredCount": {{ env `DESIRED_COUNT` }},
  "enableECSManagedTags": false,
  "enableExecuteCommand": {{ env `ENABLE_ECS_EXEC` `true` }},
  "healthCheckGracePeriodSeconds": 5,
  "launchType": "FARGATE",
  "loadBalancers": [
    {
      "containerName": "example",
      "containerPort": 80,
      "targetGroupArn": "{{ env `TARGET_GROUP_ARN` }}"
    }
  ],
  "networkConfiguration": {
    "awsvpcConfiguration": {
      "subnets": [
        "{{ env `SUBNET_1` }}",
        "{{ env `SUBNET_2` }}",
        "{{ env `SUBNET_3` }}"
      ],
      "securityGroups": [
        "{{ env `SECURITY_GROUP_ID` }}"
      ],
      "assignPublicIp": "DISABLED"
    }
  },
  "placementConstraints": [],
  "placementStrategy": [],
  "platformVersion": "1.4.0",
  "schedulingStrategy": "REPLICA",
  "serviceRegistries": []
}

You can deploy with a YAML config like this:

# ecspresso.yaml
cluster: example
service: example
service_definition: ecs_service_def.json
task_definition: ecs_task_def.json
timeout: 15m0s

$ ECS_CPU=256 \
  ECR_REPO_IMAGE_URL=<ecr-repository-url> \
  IMAGE_TAG=<ecr-container-image-tag> \
  ECS_MEMORY=512 \
  SECRETS_MANAGER_ARN=<arn-of-secrets-manager> \
  EXEC_ROLE_ARN=<arn-of-ecs-task-exec-role> \
  TASK_ROLE_ARN=<arn-of-ecs-task-role> \
  DESIRED_COUNT=<amount-of-ecs-tasks> \
  ENABLE_ECS_EXEC=<enable-ecs-task-exec> \
  TARGET_GROUP_ARN=<amount-of-target-group> \
  SUBNET_1=<subnet-id1> \
  SUBNET_2=<subnet-id2> \
  SUBNET_3=<subnet-id3> \
  SECURITY_GROUP_ID=<security-group-id> \
  ecspresso deploy --config ecspresso.yaml

You simply change the IMAGE_TAG based on your code updates and run the command.
For more advanced usage, check out how to pull values from tfstate or GitHub Actions integration, though these are beyond the scope of this article.

Conclusion

That’s it. I’ve summarized our approach to managing ECS resources. I hope this is helpful for your development workflows.

Deployment practices like these can vary between teams and organizations, so if you have other approaches or improvements, I’d love to hear about them.

Building a Scalable Event-Driven System on AWS with DynamoDB Streams, SQS, and ECS

ryo ariyama — Sun, 25 May 2025 15:10:49 +0000

Introduction

I previously built an event-driven system on AWS and would like to introduce it here. The event-driven approach I'm referring to is an architecture that processes events triggered by DynamoDB updates and similar events.
As a prerequisite, we're building a multi-tenant system where servers are shared but each tenant has its own DynamoDB table.
This blog can be read in about 5-10 minutes and should be helpful for those who want to learn system architecture on AWS or understand SQS auto-scaling functionality.

System Architecture Overview

Originally, we were operating the following system:

Lambda functions were triggered based on DynamoDB streams events to perform processing. While this met the requirement of processing events as they occurred, it had the following problems:

When event sizes are too large, Lambda may timeout before processing completes within the execution time limit. Additionally, limited CPU and memory resources could lead to OOM (Out of Memory) errors.
Events could be lost in the following scenarios:
- When there are too many events to process within DynamoDB Streams' record retention period (24 hours)
- When ECS cannot operate normally due to AWS outages or other issues

To address these issues, we made the following improvements:

Replaced Lambda with ECS
Added SQS between DynamoDB and ECS

These changes provide the following benefits:

Using a platform without time constraints allows processing to continue until completion, even with large inputs
Larger machine resources prevent CPU and memory-related issues that Lambda couldn't handle
When event processing fails, events are stored in the queue for retry once the system returns to normal

The updated architecture looks like this:

The pipeline between DynamoDB and SQS uses a service called EventBridge Pipes. This eliminates the need to create Lambda functions for event sending and allows filtering and error handling (previously done in code) to be handled without code, reducing maintenance costs.
Here's a sample Terraform code using the aws_pipes_pipe resource:

###########################
# Event
###########################
resource "aws_pipes_pipe" "event_bulk" {
  depends_on = [aws_iam_role_policy.event_pipe_role]
  name       = "event-sqs"
  role_arn   = aws_iam_role.event_pipe_role.arn
  source     = aws_dynamodb_table.events.stream_arn
  target     = var.sqs_event_queue_arn

  source_parameters {
    dynamodb_stream_parameters {
      batch_size                    = var.batch_size
      starting_position             = var.starting_position
      maximum_record_age_in_seconds = 86400
    }
    filter_criteria {
      filter {
        pattern = jsonencode({
          eventName : ["INSERT"]
          dynamodb = {
            "NewImage" : {
              "status" : { "S" : ["pending"] }
            }
          }
        })
      }
    }
  }
}

###########################
# SQS
###########################
resource "aws_sqs_queue" "event" {
  name                       = "${var.batch_container_name}-sqs"
  delay_seconds              = var.sqs_delay_seconds
  max_message_size           = var.sqs_max_message_size
  message_retention_seconds  = var.sqs_message_retention_seconds
  receive_wait_time_seconds  = var.sqs_receive_wait_time_seconds
  visibility_timeout_seconds = var.sqs_visibility_timeout_seconds
  sqs_managed_sse_enabled    = true
}

###########################
# DynamoDB
###########################
resource "aws_dynamodb_table" "events" {
  name             = "events"
  billing_mode     = "PAY_PER_REQUEST"
  hash_key         = "_global_id"
  range_key        = "_event_type"
  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"
}

###########################
# IAM
###########################
resource "aws_iam_role" "event_pipe_role" {
  name = "event-pipe-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "pipes.amazonaws.com"
        }
        Action = "sts:AssumeRole"
        Condition = {
          StringEquals = {
            "aws:SourceAccount" = var.aws_account_id
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy" "event_pipe_role" {
  name = "event-pipe-role-policy"
  role = aws_iam_role.event_pipe_role.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "dynamodb:DescribeStream",
          "dynamodb:GetRecords",
          "dynamodb:GetShardIterator",
          "dynamodb:ListStreams"
        ]
        Resource = aws_dynamodb_table.events.stream_arn
      },
      {
        Effect = "Allow"
        Action = [
          "sqs:SendMessage"
        ]
        Resource = aws_sqs_queue.event.arn
      }
    ]
  })
}

The application code within the ECS task is simplified but looks like this.

import json
import logging
import os
import time

import boto3


logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Get the SQS queue URL from environment variables
QUEUE_URL = os.environ.get("SQS_QUEUE_URL")

# Initialize the SQS client
sqs = boto3.client("sqs", region_name=os.environ.get("AWS_REGION", "ap-northeast-1"))

def process_event(event: dict):
    """
    Business logic to process a single event.
    Replace this with your own implementation.
    """
    logger.info(f"Processing event: {event}")
    # Simulate some processing time
    time.sleep(1)
    # You can raise an exception to simulate a failure
    # raise Exception("Simulated failure")

def main():
    """
    Poll messages from the SQS queue and process them one by one.
    """
    while True:
        try:
            response = sqs.receive_message(
                QueueUrl=QUEUE_URL,
                MaxNumberOfMessages=10,
                WaitTimeSeconds=20,        # Enable long polling
                VisibilityTimeout=60       # Time for processing before making the message visible again
            )

            messages = response.get("Messages", [])
            if not messages:
                logger.info("No messages received. Waiting...")
                continue

            for message in messages:
                try:
                    # Parse the message body
                    body = json.loads(message["Body"])
                    logger.info(f"Received message: {body}")

                    # If EventBridge Pipes is used, actual DynamoDB record is nested inside
                    # Uncomment and adjust based on actual structure
                    # dynamodb_record = json.loads(body["detail"])["dynamodb"]

                    process_event(body)

                    # Delete the message only if processing succeeds
                    sqs.delete_message(
                        QueueUrl=QUEUE_URL,
                        ReceiptHandle=message["ReceiptHandle"]
                    )
                    logger.info("Message processed and deleted successfully.")

                except Exception as e:
                    logger.error(f"Error processing message: {e}", exc_info=True)
                    # Do not delete the message to allow for retry

        except Exception as e:
            logger.error(f"Error polling messages: {e}", exc_info=True)
            # Wait a bit before retrying on polling failure
            time.sleep(10)

There are several considerations for this system, but determining how many ECS tasks to launch is one of the main topics. The next section explains this in detail.

ECS Task Auto Scaling

For cost and performance optimization, we want to dynamically adjust the number of ECS tasks based on the number of messages to process. Since ECS can auto-scale based on specific events, we configured it to auto-scale based on SQS message count metrics.

Step scaling

We use Step scaling policies for auto-scaling. Step scaling scales in or out based on specified thresholds. You can configure scaling settings like:

1~100 messages -> Set total task to 1
101~200件 messages -> Set total tasks to 2
201~300件 messages -> Set total tasks to 3

Here's the terraform code for scale-out:

resource "aws_appautoscaling_target" "ecs_target" {
  max_capacity       = 3 // Maximum number of tasks that can be run in the ECS service.
  min_capacity       = 0
  resource_id        = "service/${aws_ecs_cluster.batch.name}/${aws_ecs_service.batch.name}"
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace  = "ecs"
}

resource "aws_appautoscaling_policy" "scale_out" {
  name               = "ecs-batch-scale-out"
  policy_type        = "StepScaling"
  resource_id        = aws_appautoscaling_target.ecs_target.resource_id
  scalable_dimension = aws_appautoscaling_target.ecs_target.scalable_dimension
  service_namespace  = aws_appautoscaling_target.ecs_target.service_namespace

  step_scaling_policy_configuration {
    adjustment_type         = "ExactCapacity"
    cooldown                = 60
    metric_aggregation_type = "Average"

    # scale out to 1 task: 0 < metrics value <= 100
    step_adjustment {
      metric_interval_lower_bound = 0  # Applied when greater than this value (boundary not included)
      metric_interval_upper_bound = 100 # Applied when less than or equal to this value (boundary included)
      scaling_adjustment          = 1
    }

    # scale out to 2 tasks: 100 < metrics value <= 200
    step_adjustment {
      metric_interval_lower_bound = 101
      metric_interval_upper_bound = 200
      scaling_adjustment          = 2
    }

    # scale out to 3 tasks: 200 < metrics value
    step_adjustment {
      metric_interval_lower_bound = 201
      scaling_adjustment          = 3
    }
  }
}

resource "aws_cloudwatch_metric_alarm" "message_alarm_high" {
  alarm_name = "ecs_batch_FiveSecondsApproximateNumberOfMessages_high"

  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "ApproximateNumberOfMessagesVisible"
  namespace           = "AWS/SQS"
  period              = 10
  statistic           = "Maximum"

  threshold = 1

  dimensions = {
    QueueName = var.sqs_queue_name
  }

  alarm_actions = [aws_appautoscaling_policy.scale_out.arn]
}

This code works as follows

Monitors the ApproximateNumberOfMessagesVisible metric, which represents the number of pending messages, and sends alerts to the autoscaling policy when thresholds are exceeded
The aws_appautoscaling_policy's step_adjustment conditions determine how many ECS tasks to set as the total count. Pay attention to the calculation formulas for lower_bound and upper_bound (inclusive vs exclusive)

Scale-in can be configured similarly.

resource "aws_appautoscaling_policy" "scale_in" {
  name               = "ecs-batch-${aws_ecs_service.batch.name}-scale-in"
  policy_type        = "StepScaling"
  resource_id        = aws_appautoscaling_target.ecs_target.resource_id
  scalable_dimension = aws_appautoscaling_target.ecs_target.scalable_dimension
  service_namespace  = aws_appautoscaling_target.ecs_target.service_namespace

  step_scaling_policy_configuration {
    adjustment_type         = "ExactCapacity"
    cooldown               = 1200
    metric_aggregation_type = "Average"

    # 200 < message <= 300: -1 task
    step_adjustment {
      metric_interval_lower_bound = 200
      metric_interval_upper_bound = 250
      scaling_adjustment         = 2
    }

    # 100 < message <= 200: -2 task
    step_adjustment {
      metric_interval_lower_bound = 100
      metric_interval_upper_bound = 200
      scaling_adjustment         = 1
    }
  }
}

resource "aws_appautoscaling_policy" "scale_in_to_zero" {
  name               = "ecs-batch-${aws_ecs_service.batch.name}-scale-in-to-zero"
  policy_type        = "StepScaling"
  resource_id        = aws_appautoscaling_target.ecs_target.resource_id
  scalable_dimension = aws_appautoscaling_target.ecs_target.scalable_dimension
  service_namespace  = aws_appautoscaling_target.ecs_target.service_namespace

  step_scaling_policy_configuration {
    adjustment_type         = "ExactCapacity"
    cooldown               = 1200
    metric_aggregation_type = "Average"

    step_adjustment {
      metric_interval_lower_bound = null
      metric_interval_upper_bound = null
      scaling_adjustment         = 0
    }
  }
}

resource "aws_cloudwatch_metric_alarm" "message_alarm_low" {
  alarm_name          = "ecs_batch_messages_low"
  comparison_operator = "LessThanOrEqualToThreshold"
  evaluation_periods  = 3
  metric_name         = "ApproximateNumberOfMessagesVisible"
  namespace           = "AWS/SQS"
  period              = 60
  statistic           = "Average"
  treat_missing_data  = "notBreaching"
  threshold           = 100

  dimensions = {
    QueueName = var.sqs_queue_name
  }

  alarm_actions = [aws_appautoscaling_policy.scale_in.arn]
}

resource "aws_cloudwatch_metric_alarm" "messages_completely_empty" {
  alarm_name          = "ecs_batch_completely_empty"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = 3
  threshold           = 1
  treat_missing_data  = "notBreaching"

  metric_query {
    id          = "e1"
    expression  = "IF(m1 <= 0 AND m2 <= 0, 1, 0)"
    label       = "Empty Messages Condition"
    return_data = "true"
  }

  metric_query {
    id = "m1"
    metric {
      namespace   = "AWS/SQS"
      metric_name = "ApproximateNumberOfMessagesVisible"
      dimensions = {
        QueueName = var.sqs_queue_name
      }
      stat   = "Average"
      period = 60
    }
  }

  metric_query {
    id = "m2"
    metric {
      namespace   = "AWS/SQS"
      metric_name = "ApproximateNumberOfMessagesNotVisible"
      dimensions = {
        QueueName = var.sqs_queue_name
      }
      stat   = "Average"
      period = 60
    }
  }
  alarm_actions = [aws_appautoscaling_policy.scale_in_to_zero.arn]
}

This basically reverses the scale-out settings to gradually scale in based on message count. When messages reach 0, we want to scale tasks to 0, but we need to combine this with the ApproximateNumberOfMessagesNotVisible metric (representing currently processing messages) to ensure instances don't scale to 0 when messages are still being processed.

System monitoring

Next, let's consider operational monitoring. For this system (and generally), monitoring items should be considered from the perspective of whether the system is operating according to requirements. For this system, the following items come to mind:

Tasks not starting
- Pending message count ≥ 1 and no ECS tasks running
Error rate monitoring
- ApproximateNumberOfMessagesNotVisible backlog in SQS
- Dead Letter Queue (DLQ) transfer rate
Resource utilization
- ECS task CPU/Memory usage
- Task OOM detection

Due to space constraints, I can't cover everything in this blog, but as a sample, here's how we implemented the first item using CloudWatch composite metrics:

resource "aws_cloudwatch_metric_alarm" "sqs_has_messages" {
  alarm_name          = "event-ecs-sqs-has-messages"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = "1"
  metric_name         = "ApproximateNumberOfMessagesVisible"
  namespace           = "AWS/SQS"
  period              = "60"
  statistic           = "Average"
  threshold           = "1"
  alarm_description   = "This alarm monitors if SQS queue has messages waiting to be processed"
  dimensions = {
    QueueName = var.aws_sqs_queue_name
  }
}

resource "aws_cloudwatch_metric_alarm" "ecs_no_tasks" {
  alarm_name          = "event-ecs-no-tasks"
  comparison_operator = "LessThanOrEqualToThreshold"
  evaluation_periods  = "1"
  metric_name         = "RunningTaskCount"
  namespace           = "AWS/ECS"
  period              = "60"
  statistic           = "Average"
  threshold           = "0"
  alarm_description   = "This alarm monitors if ECS has no running tasks"
  dimensions = {
    ClusterName = aws_ecs_cluster.batch.name
    ServiceName = aws_ecs_service.batch.name
  }
}

resource "aws_cloudwatch_composite_alarm" "sqs_messages_but_no_ecs" {
  alarm_name        = "sqs-messages-but-no-ecs"
  alarm_description = "Alarm when SQS has messages but no ECS tasks are running"

  alarm_rule = "ALARM(${aws_cloudwatch_metric_alarm.sqs_has_messages.alarm_name}) AND ALARM(${aws_cloudwatch_metric_alarm.ecs_no_tasks.alarm_name})"

  alarm_actions = [var.monitoring_sqs_arn]
  ok_actions    = [var.monitoring_sqs_arn]
}

Alternative Approaches

While we've introduced the complete system, there are still cases where problems can occur. This happens when one tenant has significantly larger data compared to other tenants, causing that tenant's data processing to take longer and affect other tenants. This is known as the "noisy neighbors" problem, which is well-known in multi-tenant systems.
To solve this problem, you need to provide separate queues and servers for each tenant.

However, compared to tenant-shared systems, this approach requires more servers and increases costs in various ways. Deciding which system approach to take is difficult, but it's probably best to start with a simple implementation and make comprehensive decisions while monitoring performance. If you have good decision-making methods for this or any other alternative ways to resolve this multi-tenant problem, I'd appreciate comments sharing them.

Conclusion

That's it. We were able to create a simple yet scalable event-driven system using AWS. I hope this article helps with your development efforts.

Introduction to Burp Suite: Basic Usage of Web Application Vulnerability Assessment Tool

ryo ariyama — Sun, 18 May 2025 16:20:35 +0000

Introduction

[Reading time: approximately 10 minutes]
This article explains the basic usage of "Burp Suite," a web application vulnerability assessment tool. It covers everything from installation to basic operations, simple attack methods, and the use of extensions, all explained step-by-step with actual screenshots. Recommended content for those interested in web security, especially beginners.

What is Burp Suite?

Burp Suite is a tool for security testing of web applications.
It works by modifying request parameters to input values that might exploit application vulnerabilities, and then checking the responses to determine if vulnerabilities exist.
In this article, I'll be using the free version, Community Edition, Version 2024.9.5.

Basic Operations

First, download the Community Edition from the link below.
https://portswigger.net/burp/communitydownload
When launched, the following screen appears. There's nothing in particular to set up, so press Next -> Start Burp to proceed.
Open the Proxy tab and click Intercept on. This allows you to intercept requests, or in other words, temporarily stop requests so you can inject parameters.
Click Open browser to launch a web browser. Let's try searching on Google.
The request is intercepted, but you can proceed by pressing Forward.
Another request is intercepted, but if you want to use custom input values, you can change the Pretty values.

Simple attack

Now, let's try changing parameter values to perform an attack. Please note that you should never attack real web applications! Always use your own development environment or test pages where no one will be affected.
Burp Suite provides the following test page for practice, so I'll demonstrate using this page.
https://portswigger.net/web-security/logic-flaws/examples/lab-logic-flaws-excessive-trust-in-client-side-controls

Press the LOGIN button in the top right to go to the login page. Enter some arbitrary values and press Login.
Click on the POST request to see the request body, where you can enter arbitrary values for EmailAddress and Password to send the request.
While this method works, you can also modify requests using the Intruder feature. Right-click on the POST request and select Send to intruder.
You can see the API request body, surround the parts you want to change with §, and enter the values you want in the Payload configuration. Then press Start attack to attack with the entered values.
Once the attack is performed, the payloads used and the responses are displayed.

So far, I've introduced the basic usage of Burp Suite. Next, I'll introduce attack methods that combine other tools.

Import attack parameter values

Even if you understand how to attack, it might be hard to imagine what parameters to use. Additionally, preparing parameters one by one is frankly time-consuming. If you search on GitHub, you'll find several repositories that compile typical input patterns for each vulnerability, which might be helpful to reference.
You can import these files to perform attacks, and I'll explain how below.
I used the following repository as an example:
https://github.com/fuzzdb-project/fuzzdb/tree/master

First, clone the repository to your local environment.
Go to the Intruder tab and select Runtime file as the Payload type. Then select a file from ./attack under the fuzzdb-project for Select file and click Start attack.
This allows you to attack using the input values in the file.

OpenAPI extension

Next, I'll introduce a method to create parameters based on OpenAPI specifications. Burp Suite has various extensions, and this time we'll use one called OpenAPI Parser.

First, install OpenAPI Parser from the BAppstore.
Select the OpenAPI specification from the Browser button in the top right and click Load. This will display requests for each API path. You can select a path to transition to the Intruder screen and launch attacks.

Impressions after use

That's how to use Burp Suite. I think the free version of the tool should be sufficient for simple diagnostics. However, one issue I felt while using it is that it would be nice to have a feature that automatically determines whether an API is vulnerable for each attack. Especially when preparing a large number of attack patterns, it's challenging to go through all the responses, so an automatic detection feature would be very helpful.
It seems possible with Pro version extensions, but apparently not with the free version. Similar tools to Burp Suite include OWASP ZAP, but I haven't tried it, so if anyone knows if this is possible with other tools, I'd appreciate your input.