Forem: Eugene Zimin

API Authentication: Part II. API Keys

Eugene Zimin — Mon, 02 Dec 2024 08:38:33 +0000

Few Words of History

Following our exploration of Basic Authentication, the next in line are API Keys - a widely adopted authentication mechanism that addresses many of the limitations of Basic Authentication while maintaining simplicity in implementation. API Keys represent an evolution in API authentication, offering a balance between security and usability that has made them a popular choice for modern web services.

The concept of API Keys emerged as web services began to scale beyond simple and single client-server interactions. Unlike Basic Authentication's username-password paradigm, API Keys introduced a single, long-lived token approach that better suited programmatic access to APIs. This shift reflected the growing need for machine-to-machine communication in distributed systems, where traditional username-password combinations proved cumbersome.

The rise of public APIs in the mid-2000s catalyzed the widespread adoption of API Keys. Services like Google Maps, Twitter, and Amazon Web Services popularized this authentication method, demonstrating its effectiveness in managing access to public APIs at scale. This adoption marked a significant step forward in API security, introducing concepts like rate limiting, usage tracking, and granular access control that weren't easily achievable with Basic Authentication.

Understanding API Keys

At their core, API Keys are long, unique strings that serve as both identifier and authenticator for API clients. Unlike Basic Authentication's two-part credential system, an API Key combines identification and authentication into a single token. This architectural choice fundamentally changes how systems handle authentication and brings several important implications for system design.

Consider a practical example of how this unified approach manifests in real-world applications:

# Traditional Basic Auth approach
def authenticate_with_basic(username, password):
    stored_password_hash = database.get_password_hash(username)
    if not stored_password_hash:
        return False  # Username doesn't exist

    return verify_password(password, stored_password_hash)

# API Key approach
def authenticate_with_api_key(api_key):
    key_data = database.get_key_data(api_key)
    if not key_data:
        return False  # Invalid key

    return key_data  # Contains identity and permissions

In this example, the basic authentication process requires two distinct steps: first finding the user, then verifying their password. The API Key approach consolidates these steps - the key itself contains or points to all necessary information. When using Basic Authentication, the system must maintain separate storage for usernames and password hashes, while API Keys allow for a more simplified data structure.

Let's examine a more complex example that demonstrates the practical implications of this unified approach:

class APIKeyAuthenticator:
    def __init__(self, database_connection):
        self.db = database_connection

    def issue_key_for_service(self, service_name, permission_level):
        # Generate a cryptographically secure random key
        raw_key = secrets.token_urlsafe(32)

        # Create a structured key with embedded metadata
        key_prefix = self._generate_prefix(service_name)
        timestamp = int(time.time())
        api_key = f"{key_prefix}.{timestamp}.{raw_key}"

        # Store key metadata
        key_metadata = {
            'service': service_name,
            'permissions': permission_level,
            'created': timestamp,
            'last_used': None
        }
        self.db.store_key_metadata(api_key, key_metadata)

        return api_key, key_metadata

    def validate_request(self, api_key, requested_action):
        key_data = self.db.get_key_metadata(api_key)
        if not key_data:
            return False

        # Update usage timestamp
        key_data['last_used'] = int(time.time())
        self.db.update_key_metadata(api_key, key_data)

        return self._check_permissions(key_data['permissions'], requested_action)

This implementation demonstrates several key concepts. The issue_key_for_service method creates a structured API key that embeds useful metadata directly in the key format. The key consists of three parts:

a service-specific prefix,
a timestamp, and
a random component. This structure allows for quick identification of the key's origin and age without accessing the database.

The validate_request method shows how authentication and authorization become intertwined with API Keys. A single database lookup retrieves all necessary information about the key's associated service and permissions. This contrasts with Basic Authentication, where separate lookups might be needed for authentication and permission checking.

The practical implications of this unified approach become clear when examining system behavior. For instance, when a service needs to rotate its API key:

def rotate_service_key(authenticator, service_name, old_key):
    # Verify the old key is valid
    if not authenticator.validate_request(old_key, 'rotate_key'):
        return None, False

    # Get existing permissions from old key
    old_key_data = authenticator.db.get_key_metadata(old_key)
    permission_level = old_key_data['permissions']

    # Issue new key with same permissions
    new_key, _ = authenticator.issue_key_for_service(
        service_name, 
        permission_level
    )

    # Invalidate old key after grace period
    authenticator.db.schedule_key_deletion(old_key, grace_period_hours=24)

    return new_key, True

This rotation process demonstrates the elegance of the unified token approach. The entire service identity and permission set transfers seamlessly to the new key, while the old key can be gracefully deprecated. In a Basic Authentication system, changing credentials might require updating multiple database records and managing password change workflows.

Key Management and Storage

The management and storage of API keys present unique challenges that require careful consideration. In a production environment, the storage mechanism must balance security, performance, and scalability. Let's explore a comprehensive implementation that addresses these concerns with the class called APIKeyManader

from datetime import datetime, timedelta
import hashlib
from typing import Optional, Dict
import secrets
import base64

class APIKeyManager:
    def __init__(self, database_connection):
        self.db = database_connection
        self.hash_algorithm = 'sha256'
        self.key_prefix_length = 8

Hashing

Our class will contain few instance variables defining DB connection, hashing algorithm and the prefix length. Then we need to consider a method to create a hash key:

    def hash_key(self, api_key: str) -> str:
        return hashlib.new(self.hash_algorithm, api_key.encode()).hexdigest()

Usage of the method is pretty simple:

>>> manager.hash_key("pk_test_abc123")
"8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92"

Creating a hash instead of storing an API Key directly in the database is another security implication. If an attacker gains read access to this database through SQL injection, backup file exposure, or any other security breach, they immediately obtain valid API keys for all clients. These keys remain fully functional until manually revoked.

Here's what happens when an attacker breaches this database:

They see only hashes, which are one-way mathematical transformations
Even with modern computing power, it's practically impossible to reverse these hashes to obtain the original API keys
Each key uses a unique salt, preventing attackers from using rainbow tables (pre-computed hash databases)

# Example of what an attacker might see in the database
compromised_data = {
    'record_1': {
        'key_hash': '8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92',
        'salt': 'a1b2c3d4e5f6g7h8',
        'client_id': 'client_123'
    },
    'record_2': {
        'key_hash': '472b07b9fcf2c2451e8781e944bf5f77cd8457488d8f45f2318eb1b5b4175276',
        'salt': '9i8h7g6f5e4d3c2',
        'client_id': 'client_456'
    }
}

it won't be helpful as it is practically impossible to restore the API Key based on the hash information only. The security of this approach relies on some mathematical principles.

Hash functions are one-way operations
The same input always produces the same hash
Small changes in input produce completely different hashes
It's computationally infeasible to find an input that produces a specific hash

For instance:

# Demonstration of hash properties
api_key = "pk_live_abc123"
hash1 = hashlib.sha256(api_key.encode()).hexdigest()
print(f"Hash of '{api_key}': {hash1}")

# Changing just one character produces a completely different hash
api_key_modified = "pk_live_abc124"
hash2 = hashlib.sha256(api_key_modified.encode()).hexdigest()
print(f"Hash of '{api_key_modified}': {hash2}")

This will output two completely different hashes, even though the input strings differ by only one character:

Hash of 'pk_live_abc123': 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92
Hash of 'pk_live_abc124': 472b07b9fcf2c2451e8781e944bf5f77cd8457488d8f45f2318eb1b5b4175276

Adding Metadata

However creating an abstract key is almost meaningless as we need to associate this key with specific user information. For this purpose we need to be able to link the API Key to the use data.

    def create_key_with_metadata(self, 
                               client_id: str, 
                               permissions: list[str],
                               expiry_days: int = 90) -> Dict:
        # Generate random bytes and encode as base64
        random_bytes = secrets.token_bytes(32)
        random_part = base64.urlsafe_b64encode(random_bytes).decode()

        # Create structured API key with prefix
        prefix = "pk_live" if self.is_production() else "pk_test"
        api_key = f"{prefix}_{random_part}"

        # Calculate expiration date
        created_at = datetime.utcnow()
        expires_at = created_at + timedelta(days=expiry_days)

        # Prepare metadata for storage
        metadata = {
            'client_id': client_id,
            'permissions': permissions,
            'created_at': created_at.isoformat() + 'Z',
            'expires_at': expires_at.isoformat() + 'Z',
            'key_hash': self.hash_key(api_key),
            'status': 'active'
        }

        return {
            'raw_key': api_key,
            'metadata': metadata
        }

The create_key_with_metadata method represents a fundamental operation in API key management - the creation of a new API key for a client.

The method accepts three parameters: a client identifier (who the key is for), a list of permissions (what they can do), and an optional expiry period in days (when the key will stop working). By default, keys expire after 90 days to enforce regular rotation.

The output consists of two critical components packaged together: the API key itself and its associated metadata. The API key is what we'll provide to the client - it's their credential for accessing our API. The metadata, on the other hand, contains everything our system needs to know about this key: who owns it, what they're allowed to do with it, when it was created, when it expires, and a secure hash of the key for verification purposes.

Think of this like issuing a security badge at a facility. The physical badge (analogous to our API key) goes to the person, while all the information about that badge - who it belongs to, what doors it can open, when it expires - gets stored in the security system's database (our metadata).

The key itself follows a specific format: a prefix indicating whether it's for production or testing, followed by a cryptographically secure random string. This structured format helps with key management and debugging, while maintaining the security properties we need.

The method ensures several security properties: unpredictability through cryptographic randomness, safe storage through hashing, and automatic expiration through timestamp tracking. These elements combine to create a robust, manageable authentication token system.

Validating API Key

    def validate_key(self, api_key: str) -> Optional[Dict]:
        key_hash = self.hash_key(api_key)
        metadata = self.db.get_key_metadata(key_hash)

        if not metadata:
            return None

        # Check if key has expired
        expires_at = datetime.fromisoformat(
            metadata['expires_at'].replace('Z', '+00:00')
        )
        if datetime.utcnow() > expires_at:
            return None

        # Update last used timestamp
        metadata['last_used'] = datetime.utcnow().isoformat() + 'Z'
        self.db.update_key_metadata(key_hash, metadata)

        return metadata

The validate_key method serves as the gatekeeper for our API authentication system. When a client makes a request to our API with their key, this method determines whether that key is valid and active.

The process begins with the client's provided API key. The method first calculates its hash - the same process we used when storing the key. This hash acts as our lookup identifier in the database. Remember, we never store the actual API keys, only their hashes.

If we can't find any metadata associated with this hash in our database, it means the key either never existed or has been revoked. In this case, we immediately return None, indicating an invalid key.

For keys that do exist, we perform a temporal validation. The stored metadata includes an expiration timestamp - we parse this timestamp (converting it from ISO format with UTC indicator to a Python datetime object) and compare it with the current time. If the key has expired, we again return None. This expiration check is vital for enforcing security through key rotation.

When a key passes both these checks (exists and hasn't expired), we perform one final housekeeping task: updating the last_used timestamp. This tracking helps with key management, letting us identify unused keys that might need to be revoked.

Finally, for valid keys, we return the complete metadata associated with that key. This metadata contains essential information like the client's identity and permissions, which our API can use to make authorization decisions.

Think of this process like checking an ID card at a secure facility:

We first verify the ID exists in our system
We check if it hasn't expired
We log when it was last used
If everything checks out, we retrieve all the information about what the ID holder is allowed to do

This method is called frequently - potentially for every API request - so it needs to be efficient while maintaining strict security standards.

Implementation Patterns

When implementing API key authentication, there are few different transmission methods of API Keys between client and server. We'll examine the main approaches and their implications for real-world applications.

Header-based Authentication

The HTTP header approach represents the industry standard for API key transmission. By placing the API key in a custom header, we benefit from several security advantages:

def make_api_request(url: str, api_key: str, method: str = 'GET', 
                    data: dict = None) -> requests.Response:
    headers = {
        'X-API-Key': api_key,
        'Content-Type': 'application/json'
    }

    response = requests.request(
        method=method,
        url=url,
        headers=headers,
        json=data,
        verify=True  # Always verify SSL certificates
    )

    return response

and here is the usage example:

api_key = "pk_8a4c6f3e_1a2b3c4d5e6f7g8h9i0j"
response = make_api_request(
    url="https://api.example.com/data",
    api_key=api_key
)

The header-based approach offers several advantages. HTTP headers are not typically logged by default in most web servers and proxy systems and they're not directly visible by user. This means our API keys won't appear in log files, reducing the risk of accidental exposure. Additionally, headers aren't cached by browsers or included in browser history, providing another layer of security.

When implementing header-based authentication, we commonly prefix our custom header with X- to indicate it's a non-standard header. The X-API-Key header has become a de facto standard in the industry, though some systems use variations like Authorization with a custom prefix.

Query Parameter Authentication

Query parameter authentication places the API key directly in the URL as a parameter. Here's how this manifests in practice.

https://api.example.com/v1/data?api_key=pk_test_abc123def456

This approach faces several security challenges:

Server Logs Exposure: Most web servers include the complete URL in their logs:

`192.168.1.1 - - [01/Dec/2024:10:00:00 +0000] "GET /v1/data?api_key=pk_test_abc123def456 HTTP/1.1" 200 2326`

Browser History: The key becomes part of browsing history:

# Browser history entry
{
    'url': 'https://api.example.com/v1/data?api_key=pk_test_abc123def456',
    'timestamp': '2024-12-01T10:00:00',
    'title': 'API Request'
}

Referrer Headers: When clicking links on the API response page, the API key might be sent as a referrer:

Referer: https://api.example.com/v1/data?api_key=pk_test_abc123def456

While not recommended for production systems, query parameter authentication remains in use for some specific scenarios.

Static HTML/JavaScript environments where header modification isn't possible:

<img src="https://api.example.com/v1/image?api_key=pk_test_abc123" />

Direct browser access for development and testing:

curl "https://api.example.com/v1/test?api_key=pk_test_abc123"`

Legacy system compatibility where header modification isn't supported.

Security Considerations

When implementing API key authentication, security considerations extend far beyond the mere generation and validation of keys.

The fundamental challenge lies in the nature of API keys themselves - they are long-lived credentials that grant access to potentially sensitive resources. Unlike session-based authentication systems where credentials expire relatively quickly, API keys often persist for extended periods, making them attractive targets for malicious actors. This persistence, while convenient for legitimate users, requires us to implement robust security measures throughout the key lifecycle.

Consider an API key like a physical key to a building. Just as a physical key requires secure storage, controlled distribution, and periodic replacement, API keys demand similar careful management. The loss or compromise of an API key can have far-reaching consequences, potentially exposing not just individual resources but entire systems to unauthorized access.

Key Rotation and Revocation

Key rotation and revocation form the cornerstone of long-term API security strategy. While API keys might seem permanent, treating them as such introduces significant security risks. Let's examine how we implement these crucial security practices:

from datetime import datetime, timedelta

class APIKeyRotationManager:
    def __init__(self):
        self.rotation_period = timedelta(days=90)
        self.grace_period = timedelta(days=7)

The rotation period represents our security lifecycle. Setting it to 90 days balances security needs with operational convenience. The grace period allows systems to transition smoothly between old and new keys without service interruption.

Think of this like replacing locks in a building - we need to ensure new keys are distributed before old ones stop working. Here's how we manage this process:

def should_rotate(self, key_metadata: dict) -> bool:
    key_age = datetime.utcnow() - datetime.fromisoformat(
        key_metadata['created_at'].replace('Z', '+00:00')
    )

    # Standard age-based rotation
    if key_age >= self.rotation_period:
        return True

    # Check for suspicious activity patterns
    if self._detect_unusual_usage(key_metadata):
        return True

    return False

def rotate_key(self, old_key_metadata: dict) -> tuple[str, dict]:
    # Creates a new key while maintaining the old one during grace period.
    # Generate new key with same permissions
    new_key = self._generate_key()
    new_metadata = {
        'client_id': old_key_metadata['client_id'],
        'permissions': old_key_metadata['permissions'],
        'created_at': datetime.utcnow().isoformat() + 'Z',
        'rotated_from': old_key_metadata['key_hash'],
        'status': 'active'
    }

    # Update old key metadata
    old_key_metadata['status'] = 'rotating'
    old_key_metadata['rotates_at'] = (
        datetime.utcnow() + self.grace_period
    ).isoformat() + 'Z'

    return new_key, new_metadata

Our rotation strategy includes several critical patterns:

Regular rotation based on key age
Forced rotation when suspicious activity is detected
Overlapping validity periods to prevent service disruption
Maintenance of key lineage for audit purposes

Sometimes, we need more immediate action. Key revocation provides this capability:

def revoke_key(self, key_metadata: dict, reason: str) -> None:
    # this is a soft revoke
    revocation_data = {
        'status': 'revoked',
        'revoked_at': datetime.utcnow().isoformat() + 'Z',
        'revocation_reason': reason,
        'last_known_use': key_metadata.get('last_used')
    }

    # Update key metadata with revocation information
    key_metadata.update(revocation_data)

    # Trigger any necessary security alerts
    if reason in ['suspicious_activity', 'security_breach']:
        self._trigger_security_alert(key_metadata)

We also implement a validation system that respects both rotation and revocation states:

def validate_key_status(self, key_metadata: dict) -> bool:
    """
    Validates a key's status considering rotation and revocation.
    """
    if key_metadata['status'] == 'revoked':
        return False

    if key_metadata['status'] == 'rotating':
        rotation_deadline = datetime.fromisoformat(
            key_metadata['rotates_at'].replace('Z', '+00:00')
        )
        if datetime.utcnow() > rotation_deadline:
            return False

    return True

This approach to key lifecycle management ensures our system maintains security while remaining operationally viable. Regular rotation prevents the risks associated with permanent credentials, while our revocation system provides immediate response capability for security incidents.

Rate Limiting and Usage Tracking

Rate limiting and usage tracking serve as essential defensive mechanisms in API authentication systems. Just as a security guard monitors the frequency of entries into a building, these systems protect our API from abuse while providing valuable insights into usage patterns.

Let's examine a comprehensive implementation that handles both concerns:

from datetime import datetime, timedelta
from collections import defaultdict
import threading

class RateLimiter:
    def __init__(self):
        self.lock = threading.Lock()
        self.limits = {
            'requests_per_second': 10,
            'requests_per_minute': 300,
            'requests_per_hour': 5000,
            'requests_per_day': 50000
        }
        self.usage = defaultdict(lambda: {
            'second_count': 0,
            'minute_count': 0,
            'hour_count': 0,
            'day_count': 0,
            'windows': {
                'second': datetime.utcnow(),
                'minute': datetime.utcnow(),
                'hour': datetime.utcnow(),
                'day': datetime.utcnow()
            }
        })

This implementation introduces a multi-window rate limiting approach. Think of it like a building's security system that tracks entries across different time scales - from immediate access control to daily visitor limits. Here's how we enforce these limits:

def check_rate_limit(self, api_key: str) -> tuple[bool, dict]:
    with self.lock:
        now = datetime.utcnow()
        usage = self.usage[api_key]
        limits_status = {}

        # Reset counters if time windows have elapsed
        if now - usage['windows']['second'] >= timedelta(seconds=1):
            usage['second_count'] = 0
            usage['windows']['second'] = now

        if now - usage['windows']['minute'] >= timedelta(minutes=1):
            usage['minute_count'] = 0
            usage['windows']['minute'] = now

        # Update counters
        usage['second_count'] += 1
        usage['minute_count'] += 1

        # Check against limits
        limits_status = {
            'second': usage['second_count'] <= self.limits['requests_per_second'],
            'minute': usage['minute_count'] <= self.limits['requests_per_minute']
        }

        # Record usage pattern for analysis
        self._record_usage_pattern(api_key, now)

        return all(limits_status.values()), limits_status

Alongside rate limiting, we implement comprehensive usage tracking:

class UsageTracker:
    def __init__(self):
        self.usage_patterns = defaultdict(list)

    def record_request(self, api_key: str, request_data: dict):
        usage_record = {
            'timestamp': datetime.utcnow().isoformat() + 'Z',
            'endpoint': request_data.get('endpoint'),
            'method': request_data.get('method'),
            'response_status': request_data.get('status'),
            'response_time': request_data.get('response_time'),
            'client_ip': request_data.get('client_ip')
        }

        self.usage_patterns[api_key].append(usage_record)

    def analyze_usage_pattern(self, api_key: str) -> dict:
        patterns = self.usage_patterns[api_key]
        if not patterns:
            return {'risk_level': 'unknown'}

        recent_patterns = [p for p in patterns 
                          if self._is_recent(p['timestamp'])]

        # these methods are separate part of implementation:
        # self._calculate_risk_level
        # self._detect_anomalies
        # self._summarize_usage
        return {
            'risk_level': self._calculate_risk_level(recent_patterns),
            'unusual_activity': self._detect_anomalies(recent_patterns),
            'usage_summary': self._summarize_usage(recent_patterns)
        }

These systems work together to provide several critical security functions:

Prevention of abuse through multi-window rate limiting
Detection of unusual usage patterns that might indicate compromised keys
Collection of usage metrics for billing and capacity planning
Early warning system for potential security incidents

Finally we may create adaptive rate limiting based on observed patterns:

def adjust_limits(self, api_key: str, usage_analysis: dict):
    # get data fromt the previous method
    usage_analysis = self.analyze_usage_pattern(api_key)
    risk_level = usage_analysis['risk_level']
    normal_usage = usage_analysis['usage_summary']['average_daily_requests']

    if risk_level == 'high':
        # Implement stricter limits for high-risk usage patterns
        self.limits['requests_per_minute'] = min(
            self.limits['requests_per_minute'],
            int(normal_usage / (24 * 60) * 1.5)  # 50% above normal
        )
    elif risk_level == 'low':
        # Gradually relax limits for trusted keys
        self.limits['requests_per_minute'] = int(
            self.limits['requests_per_minute'] * 1.1  # 10% increase
        )

This approach to rate limiting and usage tracking provides both immediate protection against abuse and long-term insights into API usage patterns.

When to Use API Keys

The decision to implement API key authentication should be based on a careful analysis of system requirements, security needs, and usage patterns. Let's examine the primary scenarios where API keys excel as an authentication mechanism.

Public APIs with Moderate Security Requirements

In the context of public APIs, such as weather data services or public reference databases, API keys serve as an ideal authentication method. For instance, consider a public mapping service:

def get_location_data(api_key: str, coordinates: tuple) -> dict:
    headers = {'X-API-Key': api_key}
    return requests.get(
        f"https://maps.example.com/v1/location",
        headers=headers,
        params={'lat': coordinates[0], 'lng': coordinates[1]}
    ).json()

This approach works well because it balances accessibility with basic security and usage tracking. The data, while valuable, doesn't contain sensitive information that would require more robust authentication methods.

Developer-Focused Services

When building services primarily used by developers, API keys provide a straightforward integration path. Consider a code analysis service:

class CodeAnalysisAPI:
    def __init__(self, api_key: str):
        self.api_key = api_key
        self.base_url = "https://analysis.example.com/v1"

    def analyze_repository(self, repo_url: str) -> dict:
        response = requests.post(
            f"{self.base_url}/analyze",
            headers={'Authorization': f"Bearer {self.api_key}"},
            json={'repository': repo_url}
        )
        return response.json()

Developers appreciate this approach because it requires minimal setup and integrates easily with common development tools and workflows.

Services Requiring Usage Tracking

When business requirements demand precise usage monitoring, API keys excel. Consider a content delivery network:

class ContentDelivery:
    def __init__(self):
        self.usage_tracker = UsageTrackingSystem()

    def serve_content(self, api_key: str, content_id: str) -> Response:
        # Validate key and record usage
        if not self.usage_tracker.check_quota(api_key):
            return Response("Quota exceeded", status=429)

        # Track bandwidth consumption
        content = self.get_content(content_id)
        self.usage_tracker.record_bandwidth(
            api_key,
            len(content),
            content_type=content.type
        )

        return Response(content)

This system allows for precise tracking of resource consumption, enabling accurate billing and capacity planning.

Internal Microservices

Within secure networks, API keys provide an efficient service-to-service authentication mechanism. Consider a microservice architecture:

class InternalServiceClient:
    def __init__(self, service_name: str, api_key: str):
        self.service_name = service_name
        self.api_key = api_key

    def request_internal_data(self, endpoint: str) -> dict:
        headers = {
            'X-Service-Name': self.service_name,
            'X-API-Key': self.api_key
        }

        return requests.get(
            f"http://internal-service/{endpoint}",
            headers=headers
        ).json()

In this context, API keys provide a lightweight yet effective way to maintain service boundaries and track inter-service communications.

However, there are scenarios where API keys might not be the best choice:

High-security environments requiring user-specific actions
Systems handling sensitive personal data
Financial services requiring transaction-level authentication
Applications needing fine-grained user permissions

In these cases, more robust authentication methods like OAuth 2.0 or JWT-based systems might be more appropriate. The key is to match the authentication method to the specific security and operational requirements of your system.

This comprehensive understanding of when to use API keys helps ensure we implement the right authentication mechanism for our specific use case, balancing security needs with operational efficiency.

Conclusion

API Keys represent a significant evolution in API authentication, offering a balance of security and simplicity that makes them ideal for many modern use cases. While they may not provide the robust security features of OAuth 2.0 or JWT tokens, their straightforward implementation and management make them an excellent choice for many applications.

The key to successful API Key implementation lies in proper key management, secure transmission, and comprehensive monitoring. By following the best practices outlined in this guide and implementing appropriate security measures, developers can leverage API Keys to build secure and scalable API authentication systems.

As we continue our journey through API authentication methods, API Keys serve as a bridge between the simplicity of Basic Authentication and the complexity of token-based systems like OAuth 2.0, which we'll explore in our next article in this series.

API Authentication: Part I. Basic Authentication

Eugene Zimin — Mon, 02 Dec 2024 04:10:15 +0000

Introduction

API authentication serves as the fundamental gatekeeper in the inter service communication, establishing a mechanism for identifying and verifying the clients attempting to interact with our systems. At its core, authentication acts as a digital handshake between client and server, ensuring that only legitimate requests gain access to protected resources.

In distributed systems architecture, this process involves multiple layers of verification. When a client - whether a mobile application, web service, or another system - attempts to access an API endpoint, it must first prove its identity. This proof typically comes in the form of credentials, tokens, or cryptographic signatures. Much like a passport serves as a trusted form of identification in international travel, these digital credentials provide a way to verify the authenticity of incoming requests.

The story of API authentication begins in the early days of web services, when the primary concern was simply identifying the caller, who often was a user. The initial mechanisms were straightforward - a simple username and password combination transmitted with each request. As web applications grew more complex and interconnected, these basic authentication methods proved insufficient. Client applications, third-party services, and automated systems began consuming APIs alongside human users, each requiring different security considerations.

The rise of service-oriented architectures in the late 1990s and early 2000s introduced new challenges. APIs needed to handle machine-to-machine communication, delegate access rights, and manage varying levels of permissions - all while maintaining security and performance.

The evolution reflected a fundamental shift in how systems interact with each other. No longer was it sufficient to simply verify a username and password - systems needed to establish trust, manage sessions, handle token expiration, and implement multiple layers of security. This transformation mirrors the broader evolution of the internet from a simple information-sharing platform to a complex ecosystem of interconnected services.

Authentication vs. Authorization

Before we begin, we need to differentiate and understand the difference between authentication and authorization. In scientific terms, authentication and authorization represent distinct but complementary security processes that work together to create a comprehensive security model.

Authentication answers the question: "Who is making this request?" Think of it like airport security screening. When passing through security, travelers must present valid identification (passport or ID) to prove they are who they claim to be. Similarly, API requests must present valid credentials to prove their identity. Just as a passport contains specific security features that can be validated, API credentials contain unique identifiers and cryptographic elements that verify their authenticity.

Authorization, on the other hand, addresses the question: "What is this authenticated entity allowed to do?" Continuing with our airport analogy, once a traveler's identity is confirmed through their passport (authentication), their boarding pass determines which flight they can board and which areas of the airport they can access (authorization). A first-class ticket might grant access to exclusive lounges, while a standard ticket restricts access to general waiting areas.

In the context of API security, this distinction becomes critical to keep in mind. Consider a banking system: when logging into a banking application, providing correct username and password authenticates the user (proves identity), but the bank's authorization system determines which accounts they can view or what types of transactions they can perform. A customer service representative might be authenticated into the system but authorized only to view customer information, while a financial advisor might be authorized to perform trades on behalf of clients.

Basic Authentication: The Core

Basic Authentication represents one of the foundational authentication methods in web services. Despite its simplicity, understanding its mechanics provides crucial insights into authentication principles. Born in the early days of web protocols, Basic Authentication introduces core concepts that persist in modern authentication systems.

Figure 1. Basic Authentication Flow

At its heart, Basic Authentication operates on a straightforward premise - each request carries credentials in its header. These credentials consist of a username and password pair, combined and encoded using a base64 algorithm with the prefix word Basic before encoded string. Below is the example how to encode the username and password into the base64 string using command line interface.

$ echo -n 'username:password' | base64
dXNlcm5hbWU6cGFzc3dvcmQ=

To create a Basic Authentication token we need to get that string and simply add the word Basic before:

# This is the client code, which may be another service or browser
import base64

def create_basic_auth_header(username, password):
    # Combine username and password with a colon
    credentials = f"{username}:{password}"

    # Convert the string to bytes, encode in base64, and decode back to string
    encoded_credentials = base64.b64encode(credentials.encode('utf-8')).decode('utf-8')

    # Create the authorization header
    return f"Basic {encoded_credentials}"

# Example implementation
auth_header = create_basic_auth_header("researcher", "secure_pwd123")
print(f"Generated header: {auth_header}")

# Output example:
# Generated header: Basic dXNlcm5hbWU6cGFzc3dvcmQ=

This generated Basic Authentication token in form of the string Basic dXNlcm5hbWU6cGFzc3dvcmQ= should be injected by client application for every request made towards the secured API endpoint. When examining network traffic, such a header should appear in every request, made by client application (web browser or another service):

GET /api/resource HTTP/1.1
Host: api.example.com
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=

The server's role in this exchange involves reversing the encoding process and validating the credentials. This creates a simple yet complete authentication cycle:

# This is server code, which validates credentials on the server
def validate_basic_auth(auth_header):
    try:
        # check the authentication approach
        auth_tokens = auth_header.split(' ')

        # return False if the approach is not Basic Auth
        if ('Basic' != auth_tokens[0])
            return False

        # Decode from base64
        decoded_bytes = base64.b64decode(auth_tokens[1])

        # Convert to string and split credentials
        decoded_str = decoded_bytes.decode('utf-8')
        username, password = decoded_str.split(':')

        # In practice, compare against securely stored credentials in Database
        return verify_credentials_in_DB(username, password)

    except Exception:
        return False

Each request follows this validation cycle, establishing a stateless authentication pattern.

Base64 Encoding

When examining Basic Authentication's encoding mechanism, understanding why base64 is chosen over other encoding methods reveals fascinating insights into network protocol design. Let's take a look at it through both theoretical explanation and practical demonstrations.

import base64

# Let's examine what happens with special characters in credentials
def demonstrate_base64_necessity(username, password):
    # Original credentials with special characters
    raw_credentials = f"{username}:{password}"
    print(f"Original string: {raw_credentials}")

    # Show raw bytes representation
    raw_bytes = raw_credentials.encode('utf-8')
    print(f"Raw bytes: {raw_bytes}")

    # Show base64 encoded version
    encoded = base64.b64encode(raw_bytes).decode('utf-8')
    print(f"Base64 encoded: {encoded}")

    return encoded

# Example with special characters
result = demonstrate_base64_necessity("research@lab.com", "p@ssw:rd!123")

Considering we provide the following data for input research@lab.com as the username and p@ssw:rd!123 as the password, we should expect the following output:

# Original string: research@lab.com:p@ssw:rd!123
# Raw bytes: b'research@lab.com:p@ssw:rd!123'
# Base64 encoded: cmVzZWFyY2hAbGFiLmNvbTpwQHNzdzpyZCExMjM=

Base64 encoding serves several critical purposes in Basic Authentication.

First, HTTP headers must comply with ASCII character restrictions. Consider credentials containing non-ASCII characters or special symbols that might interfere with HTTP header parsing. Base64 encoding solves this by converting any binary data into a subset of ASCII characters (A-Z, a-z, 0-9, +, /, and =).

Let's demonstrate this with a more complex example:

def analyze_character_safety(credentials):
    # Original string might contain problematic characters
    print(f"Original: {credentials}")

    # Show problematic characters in HTTP headers
    problematic = [c for c in credentials if not (32 <= ord(c) <= 126)]
    if problematic:
        print(f"Problematic characters: {problematic}")

    # Encode to base64
    encoded = base64.b64encode(credentials.encode('utf-8')).decode('utf-8')
    print(f"Safe base64: {encoded}")

    # Verify all characters are HTTP-safe
    safe_chars = all(32 <= ord(c) <= 126 for c in encoded)
    print(f"All characters HTTP-safe: {safe_chars}")

# Example with various challenging characters
analyze_character_safety("user:パスワード")  # Japanese characters
analyze_character_safety("user:password\n\r")  # Control characters

Again, using user:パスワード" in first case and user:password\n\r in the second case we would expect to get the following output:

# Original: user:パスワード
# Problematic characters: ['パ', 'ス', 'ワ', 'ー', 'ド']
# Safe base64: dXNlcjrjg5Hjgrnjg/rjg7zjg4k=
# All characters HTTP-safe: True

Another aspect is the colon character : used as a delimiter between username and password. Without encoding, this would create ambiguity in parsing the credentials. Base64 encoding ensures the colon in the original credentials doesn't interfere with the username-password delimiter:

def demonstrate_colon_handling():
    # A username containing a colon would be problematic
    problematic_username = "research:team"
    password = "secure123"

    raw = f"{problematic_username}:{password}"
    print(f"Ambiguous raw format: {raw}")

demonstrate_colon_handling()

Without encoding, this would be ambiguous and we won't know which colon is the delimiter?:

# Ambiguous raw format: research:team:secure123

To solve that we need to add the following code at the end of the demonstrate_colon_handling method:

    # Base64 encoding resolves this ambiguity
    encoded = base64.b64encode(raw.encode('utf-8')).decode('utf-8')
    print(f"Unambiguous encoded format: {encoded}")

    # Demonstrate successful decoding
    decoded = base64.b64decode(encoded).decode('utf-8')
    print(f"Decoded back: {decoded}")

As the result now we should expect the following:

# Ambiguous raw format: research:team:secure123
# Unambiguous encoded format: cmVzZWFyY2g6dGVhbTpzZWN1cmUxMjM=
# Decoded back: research:team:secure123

Base64 encoding also provides a minor level of obfuscation. While it's important to note that this is not encryption and offers no real security, it prevents casual observers from immediately reading credentials in network traces.

It is clear that base64 encoding in Basic Authentication isn't about security but about ensuring reliable transport of credential information across HTTP protocols. It's a elegant solution to the challenge of sending potentially complex credential data within the constraints of HTTP headers.

Security Considerations

While Basic Authentication provides a straightforward implementation, its security profile requires careful consideration, as Transport Layer Security (TLS) becomes critical when using Basic Authentication. Consider the following example:

import requests
from urllib.parse import urlparse

def create_basic_auth_credentials():
    url = "http://api.example.com/data"
    auth_header = construct_basic_auth_header("researcher", "password123")

    # Examining the request details
    parsed_url = urlparse(url)
    if parsed_url.scheme != "https":
        print("Credentials will be transmitted in encoded but unencrypted form")
        print(f"Raw auth header visible in transit: {auth_header}")

If we run the following code with the HTTP schema (not HTTPS) we will see the following:

# Unencrypted HTTP request (vulnerable)
GET /api/data HTTP/1.1
Host: api.example.com
Authorization: Basic cmVzZWFyY2hlcjpwYXNzd29yZDEyMw==

and if we run it over HTTPS connection - we will get encrypted data:

# HTTPS encrypted request (secure)
[encrypted data stream]

HTTP Secured connection is important here because it provides encryption on transport layer and prevents MITM attacks.

A man-in-the-middle attack occurs when a malicious actor positions themselves between the client and server during communication. Think of it like intercepting a postal mail - the attacker can read, modify, or even replace the contents before forwarding them to the intended recipient. In Basic Authentication, since credentials are merely encoded in base64 (not encrypted), an intercepted request reveals the username and password as clearly as reading an open letter.

What makes MITM particularly dangerous for Basic Authentication? Every request to the server includes these encoded credentials. This repeated transmission creates multiple opportunities for credential theft. Once intercepted, these credentials remain valid until explicitly changed, as Basic Authentication lacks built-in expiration or revocation mechanisms. The attacker can reuse the stolen credentials indefinitely, potentially accessing sensitive data or performing unauthorized operations.

The situation changes dramatically when HTTPS enters the picture. HTTPS establishes a secure encrypted tunnel between client and server through Transport Layer Security (TLS). Continuing our postal analogy, HTTPS is like sending a letter in a secure diplomatic pouch that can only be opened by the intended recipient. Even if intercepted, the encrypted contents remain unreadable to the attacker.

When to use Basic Authentication

Despite its limitations, Basic Authentication remains relevant in specific scenarios where its simplicity and ease of implementation outweigh its constraints. Understanding these use cases helps make informed decisions about authentication strategies.

Development and Testing Environments

During the development phase, teams often need quick, straightforward authentication mechanisms to test API functionality. Basic Authentication serves this purpose effectively, allowing developers to focus on core functionality without the complexity of more sophisticated authentication systems.

In development environments, this approach facilitates rapid iteration and testing. However, it's important to ensure these simplified credentials never make their way into production systems. Development configurations should remain strictly separated from production environments and never stored in Git. When building and testing applications, secrets such as credentials, API keys, and tokens should be managed through environment variables or dedicated secrets management systems.

from os import environ
from dotenv import load_dotenv

class Config:
    def __init__(self):
        # Load environment variables from .env for development
        load_dotenv()

        self.basic_auth = {
            'username': environ.get('AUTH_USERNAME'),
            'password': environ.get('AUTH_PASSWORD')
        }

Development teams should maintain:

A .env.example file in version control showing the required variables without actual values
A .gitignore file that explicitly excludes .env files and other secret-containing configurations
Separate configuration management for each environment (development, staging, production)
Documentation explaining how to securely obtain and configure credentials

For local development, create a .env file that's never committed:

# .env
AUTH_USERNAME=dev_user
AUTH_PASSWORD=dev_password

Internal Network Communications

Within controlled, internal networks, Basic Authentication can provide adequate security when combined with proper transport layer protection. This is particularly relevant for microservices architectures where services communicate within a trusted network boundary. In such environments, services often need to authenticate each other rapidly and efficiently which makes Basic Authentication is the preferable approach. The controlled nature of internal networks, combined with additional security measures like network segmentation, firewalls, and intrusion detection systems, creates multiple layers of protection. Organizations commonly implement Basic Authentication for service-to-service communication in scenarios where the network perimeter is well-defined and access is strictly controlled through VPNs or private networks.

def internal_service_call():
    internal_url = "https://internal-api.local/data"
    response = requests.get(
        internal_url,
        auth=('service_account', 'internal_token'),
        verify=True  # Still enforce SSL certificate verification
    )

Command-Line Tools and Scripts

Basic Authentication proves valuable for command-line interfaces and automation scripts where simplicity and direct implementation are priorities. These tools often operate in controlled environments where the additional complexity of token-based authentication might be unnecessary. When building automation tools for internal use, developers need quick, reliable authentication methods that can be easily implemented across different programming languages and platforms. The simplicity of implementation allows teams to focus on core functionality while maintaining adequate security through proper credential management and secure communication channels.

Monitoring and Health Check Endpoints

For simple monitoring endpoints that provide basic system health information, Basic Authentication offers a straightforward way to prevent unauthorized access while maintaining easy integration with monitoring tools. Monitoring systems require consistent, reliable access to health check endpoints while preventing unauthorized access to potentially sensitive system information. The stateless nature of Basic Authentication makes it particularly suitable for these endpoints, as monitoring tools can easily include credentials in their requests without managing complex token lifecycles or session states. System administrators can quickly configure monitoring tools with Basic Authentication credentials, enabling automated health checks and alerts while maintaining a basic security barrier against unauthorized access attempts. The simplicity of Basic Authentication also facilitates easy integration with various monitoring platforms and tools, making it a practical choice for health check endpoints in both development and production environments.

Conclusion

Basic Authentication, despite its age and simplicity, continues to serve as a foundational element in API security architecture. The security considerations highlight the critical importance of implementing Basic Authentication exclusively over HTTPS connections. Without proper transport layer security, the base64 encoding offers no real protection against man-in-the-middle attacks, potentially exposing sensitive credentials to malicious actors. This vulnerability underscores why Basic Authentication should be deployed thoughtfully and in appropriate contexts.

Basic Authentication remains valuable when matched with the right requirements and security context. In development environments, it facilitates rapid testing and iteration. Within internal networks, it enables efficient service-to-service communication. For command-line tools and monitoring systems, it provides a straightforward authentication mechanism that balances security with simplicity.

However, it's important to recognize that Basic Authentication is just one tool in the modern API security toolkit. While it excels in specific scenarios, more complex applications often require more sophisticated authentication mechanisms like OAuth 2.0 or JWT tokens. The key to successful implementation lies in understanding these limitations and choosing Basic Authentication only when its simplicity aligns with the security requirements and operational context of your system.

As we continue to build and secure modern APIs, Basic Authentication serves as a reminder that sometimes the simplest solution, when properly implemented and appropriately deployed, can be the right choice. Its longevity in the field of web security testifies to the enduring value of straightforward, well-understood security mechanisms in specific contexts.

Debugging SSH connections: A Comprehensive Guide

Eugene Zimin — Tue, 26 Nov 2024 21:35:26 +0000

SSH (Secure Shell) is the backbone of remote system administration and secure remote access, serving millions of developers and system administrators daily. However, when SSH connections fail, the cryptographic nature of the protocol can make debugging challenging. The complex interplay between authentication mechanisms, encryption algorithms, and network layers often obscures the root cause of connection issues. This complexity is further compounded by the protocol's security-first design, where error messages are intentionally vague to prevent potential attackers from gathering system information. Whether we're dealing with key authentication failures, network connectivity issues, or configuration mismatches, understanding the underlying SSH architecture becomes critical for effective troubleshooting.

Understanding SSH Connection Process

Before diving into debugging, it's important to understand how SSH establishes connections. The process follows these steps:

TCP connection establishment (Port 22 by default)
Protocol version exchange
Key exchange and server authentication
Client authentication
Session establishment

Detailed process is highlighted in another article: Configuring SSH to Access Remote Server dedicated to overview this process in a deeper level.

Each step can fail for different reasons, producing various error messages. Understanding these steps helps pinpoint where issues occur.

Common SSH Errors and Solutions

When working with SSH connections, we frequently encounter various error messages that might seem cryptic at first glance. While SSH's error reporting is intentionally terse for security reasons, each error message provides valuable clues about the underlying issue. Understanding these common errors and their potential solutions not only helps us resolve issues faster but also deepens our knowledge of SSH's security model and connection process. Many of these errors stem from incorrect configurations, permission issues, or network problems, and they often appear during system upgrades, after server migrations, or when setting up new environments. Let's explore the most frequent SSH connection issues and their systematic solutions.

Connection Refused: Understanding Server Accessibility

One of the most common SSH errors we encounter during daily operations looks deceptively simple:

ssh: connect to host example.com port 22: Connection refused

This error message indicates that our SSH server is completely unreachable, though the underlying cause requires careful investigation. Most frequently, we discover that the SSH daemon (sshd) has stopped running on the target server. This typically happens after system updates, configuration changes, or occasional service crashes. A quick check of the service status often reveals the problem:

$ sudo systemctl status sshd
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled)
     Active: inactive (dead) since Mon 2024-01-15 09:23:45 UTC

Pay attention that the server is shown as inactive (dead) and usually it should be started with the following command:

$ sudo systemctl start sshd

Network connectivity issues present another common culprit. Corporate or cloud provider firewalls might be blocking the default SSH port 22, requiring us to verify firewall rules. In cloud environments like AWS or GCP, this often means checking both the instance's security group and network ACLs:

$ sudo iptables -L | grep ssh
# No output indicates potential firewall blocking

Sometimes the issue stems from DNS resolution problems or incorrect IP addressing. We can validate basic network connectivity using standard networking tools:

$ ping example.com
ping: example.com: Name or service not known

$ telnet example.com 22
telnet: Unable to connect to remote host: Connection refused

If all these checks pass but the connection still fails, the server itself might be experiencing issues, possibly due to resource exhaustion or hardware problems. In such cases, server logs become our primary diagnostic tool:

$ sudo tail -f /var/log/syslog
Jan 26 10:15:32 server kernel: Out of memory: Kill process 1234 (sshd) score 28 or sacrifice child

Permission Denied

One of the most frustrating SSH errors occurs during the authentication phase, presenting itself as a seemingly simple message:

Permission denied (publickey,password)

This deceptively brief message indicates an authentication failure, though its resolution often requires careful investigation. The error message's format actually provides our first clue - the terms "publickey" and "password" in parentheses tell us which authentication methods the server attempted before denying access.

When troubleshooting this error, we often find that the username doesn't match the remote system's records. For example, while connecting to an Ubuntu server, we might mistakenly use:

$ ssh admin@ubuntu-server
Permission denied (publickey,password)

When the correct username should be 'ubuntu':

$ ssh ubuntu@ubuntu-server
Welcome to Ubuntu 22.04.2 LTS...

Private key mismatches represent another common scenario. The SSH server maintains a strict relationship between private and public key pairs, and even a slight mismatch will trigger this error. We can investigate key-related issues by enabling verbose output:

$ ssh -v ubuntu@ubuntu-server
...
debug1: Trying private key: /home/user/.ssh/id_rsa
debug1: Trying private key: /home/user/.ssh/id_ed25519
debug1: No more authentication methods to try.

This verbose output shows SSH attempting to use each private key file it finds. The sequence shows that SSH couldn't successfully authenticate with any available key file. When you see 'No more authentication methods to try', it means SSH has exhausted all configured authentication methods (in this case, trying both RSA and ED25519 keys) without success.

A particularly tricky variant occurs when all credentials appear correct, but file permissions are preventing SSH from accepting the keys. SSH enforces strict permission requirements for security reasons. We commonly see this when copying key files between systems:

$ ls -la ~/.ssh/id_ed25519
-rw-rw-r-- 1 user user 464 Nov 26 10:15 /home/user/.ssh/id_ed25519

$ ssh ubuntu@ubuntu-server
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0664 for '/home/user/.ssh/id_ed25519' are too open.

The solution requires setting appropriate permissions:

$ chmod 600 ~/.ssh/id_ed25519
$ ssh ubuntu@ubuntu-server
Welcome to Ubuntu 22.04.2 LTS...

For those cases where the server has our public key but still denies access, examining the server's auth log often reveals the underlying issue:

$ sudo tail -f /var/log/auth.log
Nov 26 10:15:32 ubuntu-server sshd[12345]: Authentication refused: bad ownership or modes for directory /home/ubuntu/.ssh

Host Key Verification Failed

During routine SSH operations, we occasionally encounter an alarming message that stops us in our tracks:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:abcdef1234567890abcdef1234567890.
Please contact your system administrator.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/user/.ssh/known_hosts:3

This error, while alarming, serves as a critical security feature in SSH's trust model. When we first connect to a server, SSH stores its unique fingerprint in our known_hosts file. Any subsequent changes to this fingerprint trigger this warning, protecting us from potential security breaches.

In cloud environments, this error frequently occurs after server rebuilds. For instance, when working with AWS EC2 instances:

$ ssh ubuntu@ec2-12-34-56-78.compute-1.amazonaws.com
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

The instance might have been terminated and recreated, generating a new host key. We can verify this through AWS console or CLI:

$ aws ec2 describe-instance-history --instance-id i-1234567890abcdef0
{
    "InstanceHistoryEvents": [
        {
            "EventType": "instanceStop",
            "EventTime": "2024-01-25T10:00:00Z"
        }
    ]
}

For known legitimate changes, we can remove the old key:

$ ssh-keygen -R ec2-12-34-56-78.compute-1.amazonaws.com
# Host ec2-12-34-56-78.compute-1.amazonaws.com found: line 3
/home/user/.ssh/known_hosts updated.

However, when this error occurs unexpectedly, particularly with production servers, it warrants immediate investigation. We can examine the server's SSH fingerprint directly:

$ ssh-keyscan -t ed25519 hostname
# hostname:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
hostname ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI...

For added security, we can compare this with the fingerprint provided by our infrastructure provider or system administrator:

$ ssh-keygen -lf <(ssh-keyscan -t ed25519 hostname 2>/dev/null)
256 SHA256:abcdef1234567890abcdef1234567890 hostname (ED25519)

In cases involving DNS changes, we might see this error when a domain name starts pointing to a different server. A quick DNS lookup can confirm such changes:

$ dig +short hostname
93.184.216.34  # Note if this IP has changed

Slow Connections

While SSH generally provides very responsive connections, we occasionally encounter frustratingly slow sessions that impact us. These performance issues often manifest in various ways: delayed command responses, laggy terminal output, or prolonged initial connection times.

One of the most common culprits involves DNS resolution delays. When establishing an SSH connection, the server attempts to resolve the client's hostname by default. In environments with misconfigured DNS servers or slow network responses, this resolution process can add significant delays:

$ time ssh server.example.com date 
Warning: Reverse DNS lookup failed 
Tue Nov 26 10:15:32 UTC 2024 
real 0m3.245s 
user 0m0.035s 
sys 0m0.012s

The output shows three timing measurements: 'real' indicates the actual elapsed wall-clock time (3.245 seconds), while 'user' and 'sys' show CPU time spent in user and kernel mode respectively. The large difference between real time (3.245s) and CPU time (0.047s total) indicates the connection is spending most of its time waiting for DNS resolution, not processing.

We can significantly improve connection times by disabling DNS lookups in the server configuration:

$ sudo nano /etc/ssh/sshd_config
# Add or modify the following line
UseDNS no

# Restart the SSH service to apply changes
$ sudo systemctl restart sshd

# Test the connection speed again
$ time ssh server.example.com date
Tue Nov 26 10:15:32 UTC 2024
real    0m0.532s
user    0m0.034s
sys     0m0.011s

For connections over high-latency networks or when transferring large amounts of data, enabling SSH compression can yield substantial performance improvements. SSH compression becomes particularly effective when working with text-heavy sessions or transferring compressible data:

$ cat ~/.ssh/config
# Global SSH client configuration
Host *
    # Enable compression for all connections
    Compression yes
    # Use compression level 6 for optimal balance
    CompressionLevel 6

Perhaps one of the most powerful optimizations involves SSH connection multiplexing. Instead of establishing new TCP connections for each SSH session, multiplexing reuses an existing connection, dramatically reducing connection overhead. This becomes especially valuable when working with remote Git repositories or running multiple SSH sessions to the same server:

$ cat ~/.ssh/config
Host *
    # Enable automatic multiplexing
    ControlMaster auto
    # Define the control socket location
    ControlPath ~/.ssh/control/%C
    # Keep the master connection alive for an hour
    ControlPersist 1h
    # Optional: Configure keepalive to prevent timeouts
    ServerAliveInterval 60
    ServerAliveCountMax 3

We can verify multiplexing is working by examining the control socket:

$ ls -la ~/.ssh/control/
total 0
drwx------ 2 user user 100 Nov 26 10:15 .
drwx------ 8 user user 160 Nov 26 10:15 ..
srw------- 1 user user   0 Nov 26 10:15 example.com-22-user

The srw at the start of the last line indicates this is a socket file (s) with read-write permissions (rw). The size showing as 0 is normal for socket files. The filename example.com-22-user follows the format hostname-port-username, indicating an active multiplexed connection for this specific combination.

The presence of the socket file indicates an active multiplexed connection. Subsequent SSH commands to the same host will reuse this connection, resulting in nearly instantaneous session establishment:

$ time ssh server.example.com date
Tue Nov 26 10:15:32 UTC 2024
real    0m0.087s
user    0m0.012s
sys     0m0.008s

For production environments where consistent performance is critical, we might also consider adjusting TCPkeepalive settings to prevent connection drops over problematic networks:

Host production-*
    # More aggressive keepalive for production servers
    TCPKeepAlive yes
    ServerAliveInterval 30
    ServerAliveCountMax 6

Resolving SSH Key-Related Issues

SSH key problems often emerge as some of the most perplexing authentication challenges. Let's explore two critical categories of key-related issues that frequently impact SSH connections.

One of the most common SSH key errors presents itself with an alarming warning message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/home/user/.ssh/id_rsa' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.

This error reflects SSH's strict security requirements for private key files. Common scenarios leading to this issue include copying keys from another system, extracting them from backups, or creating them with incorrect default permissions. Let's examine a typical scenario:

$ ls -la ~/.ssh/id_rsa
-rw-rw-r-- 1 user user 1876 Nov 26 10:15 /home/user/.ssh/id_rsa

The permissions shown above (664) allow group members to read the private key, creating a security vulnerability. We can resolve this by applying proper permissions:

# Secure the private key file
$ chmod 600 ~/.ssh/id_rsa
$ ls -la ~/.ssh/id_rsa
-rw------- 1 user user 1876 Nov 26 10:15 /home/user/.ssh/id_rsa

# Secure the SSH directory itself
$ chmod 700 ~/.ssh
$ ls -la ~ | grep .ssh
drwx------ 2 user user 4096 Nov 26 10:15 .ssh

Another subtle but frustrating issue occurs when SSH refuses to read seemingly valid key files:

Load key "/home/user/.ssh/id_rsa": invalid format

This error often surfaces after migrating keys between different SSH implementations or when working with keys generated by third-party tools. Let's investigate a problematic key:

$ ssh-keygen -l -f ~/.ssh/id_rsa
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: INVALID KEY FILE FORMAT DETECTED!           @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

The -l flag attempts to show the key's fingerprint and bit length. This error indicates the key file exists but isn't in a format SSH can parse. This often happens when the key file has been corrupted during transfer or when it's been modified by a text editor that changed line endings or character encoding.

The error might occur because the key is in a modern OpenSSH format while connecting to an older server, or vice versa. We can examine the key's content (being careful not to expose private key material):

$ head -n 1 ~/.ssh/id_rsa
-----BEGIN OPENSSH PRIVATE KEY-----

If we see a different header format or unexpected content, we might need to convert the key to a compatible format. The PEM format offers the widest compatibility:

# Backup the original key first
$ cp ~/.ssh/id_rsa ~/.ssh/id_rsa.backup

# Convert the key to PEM format
$ ssh-keygen -p -f ~/.ssh/id_rsa -m PEM
Key has comment 'user@hostname'
Enter new passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved with the new passphrase.

# Verify the key format
$ head -n 1 ~/.ssh/id_rsa
-----BEGIN RSA PRIVATE KEY-----

For keys that appear completely unreadable, we might need to check their encoding:

$ file ~/.ssh/id_rsa
/home/user/.ssh/id_rsa: PEM RSA private key

Sometimes, keys might become corrupted during transfer, especially when copying between Windows and Unix systems. In such cases, checking for hidden characters or incorrect line endings can help:

$ dos2unix ~/.ssh/id_rsa
dos2unix: converting file /home/user/.ssh/id_rsa to Unix format...

Advanced Debugging Techniques

When standard troubleshooting steps fall short, we often need to delve deeper into SSH's internal workings to identify and resolve complex connection issues. SSH provides sophisticated debugging capabilities that, while potentially overwhelming at first glance, offer invaluable insights into the connection process. These advanced techniques become particularly necessary when dealing with enterprise environments, complex network configurations, or when standard error messages prove insufficient for diagnosis.

During production incidents or when supporting mission-critical systems, these debugging approaches help us understand the intricate dance between client and server configurations, network interactions, and authentication mechanisms. Let's explore the advanced tools and techniques that experienced system administrators rely on for resolving challenging SSH connection problems.

Verbose Logging

The very first step which should be done - enable excessive logging. SSH's logging capabilities represent one of our most powerful diagnostic tools, offering three distinct levels of detail (-v, -vv, -vvv). Each level peels back another layer of the connection process:

# Basic debugging output
$ ssh -v user@hostname
OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to hostname port 22 [192.168.1.100]

# More detailed protocol debugging
$ ssh -vv user@hostname
debug2: resolving "hostname" port 22
debug2: ssh_connect_direct: needpriv 0
debug2: fd 3 setting O_NONBLOCK

# Maximum verbosity for complex issues
$ ssh -vvv user@hostname
debug3: send packet: type 5
debug3: receive packet: type 6
debug3: rekey after 134217728 bytes, 3600 seconds

Server-Side Logging

Understanding what's happening on SSH server gives better vision of root cause of the problems, as well as security and troubleshooting. By enabling detailed logging, we can monitor authentication attempts, track user sessions, and investigate potential security incidents with precision.

To unlock the full potential of SSH logging, at first we need to modify SSH daemon configuration. Open /etc/ssh/sshd_config and set the logging level to its most verbose setting:

LogLevel DEBUG3

SSH activity can be monitored in real-time through system logs. The log location varies by Linux distribution:

# For Debian-based systems (Ubuntu, Debian)
sudo tail -f /var/log/auth.log

# For Red Hat-based systems (RHEL, CentOS, Fedora)
sudo tail -f /var/log/secure

The logs contain detailed information about SSH connections. Here's an example of a successful login with its associated IP address:

Apr 15 14:23:21 server sshd[12345]: Accepted publickey for alice from 192.168.1.100 port 52413

Failed authentication attempts are also recorded, providing valuable security insights:

Apr 15 14:25:33 server sshd[12346]: Failed password for invalid user admin from 203.0.113.1 port 59632 ssh2

Log rotation helps manage the increased volume of data from DEBUG3 level logging. This can be configured in /etc/logrotate.d/sshd to maintain disk space while preserving historical data.

Note that verbose logging creates additional system overhead. In high-traffic production environments, consider reducing the log level after completing specific monitoring or investigation tasks.

Testing Connectivity

Before diving into complex SSH issues, establishing basic connectivity helps narrow down potential problems. Let's start by examining network paths and connections.

The netcat utility provides a straightforward way to verify if the SSH port accepts connections:

nc -zv hostname 22

The -z flag tells netcat to scan for listening daemons without sending data, while -v enables verbose output. A successful response looks like 'Connection to hostname port 22 succeeded!', while a failure might show 'Connection refused' or 'Connection timed out'. This test confirms basic TCP connectivity without attempting SSH authentication.

When connection issues arise, tracing the network path often reveals routing problems or blocked ports:

traceroute hostname

DNS resolution problems can manifest as connection failures, so checking name resolution adds another layer of verification:

dig hostname

Moving beyond basic connectivity, validating SSH configuration prevents common setup issues. The SSH client includes built-in configuration testing:

ssh -G hostname

This command displays the exact configuration that will be used when connecting to the specified host, including inherited defaults and matching Host blocks.

For server-side verification, the SSH daemon offers similar diagnostic capabilities:

sudo sshd -T

This command performs a comprehensive check of the server configuration, displaying the active settings after processing all included files and applying default values. The output helps identify misconfigurations before they impact users.

Best Practices for SSH Troubleshooting

When troubleshooting SSH issues, following a methodical approach leads to faster resolution. Starting with basic connectivity checks establishes a foundation for further investigation. Moving through permission verification and authentication methods helps isolate problems systematically. System logs and verbose SSH output often reveal the root cause of connection issues.

Maintaining clear documentation strengthens our troubleshooting capabilities. Recording configuration changes, preserving working configurations, and keeping configuration backups creates a reliable reference point when issues arise. This documentation becomes particularly valuable when dealing with complex multi-server environments.

During troubleshooting, maintaining security remains paramount. Avoiding temporary security bypasses prevents accidental exposure. Host key changes warrant careful verification, and proper file permissions must be maintained throughout the debugging process.

Conclusion

SSH debugging requires a methodical approach and understanding of both the protocol and common failure points. By following this guide, you can efficiently diagnose and resolve SSH connection issues while maintaining security. Remember that SSH's complexity is a feature, not a bug – it's designed to be secure first and convenient second.

Future maintenance of SSH connections can be simplified by implementing proper monitoring, maintaining documentation, and following security best practices. When issues do arise, a systematic debugging approach will help resolve them quickly and effectively.

Understanding SSH Key Pairs: A Developer's Guide

Eugene Zimin — Tue, 26 Nov 2024 07:22:45 +0000

In today's interconnected development world, secure authentication is not just a luxury—it's a necessity. Whether you're a seasoned DevOps engineer or a junior developer just starting your journey, understanding SSH key pairs is crucial for your daily workflow. They're the unsung heroes that keep our git pushes secure, our server access protected, and our deployments safe from prying eyes.

But let's be honest: SSH keys can be confusing. With terms like "public key infrastructure," "cryptographic algorithms," and "key fingerprints" floating around, it's easy to feel overwhelmed. This guide aims to demystify SSH key pairs, breaking down complex concepts into digestible pieces that will help you make informed decisions about your security setup.

What Are SSH Key Pairs?

SSH key pairs are cryptographic credentials consisting of two parts:

A private key that stays on your local machine (keep this secret!)
A public key that you can share freely

Think of them as a sophisticated lock and key system. The public key is like a special lock you can give to anyone, while the private key is the unique key that opens only your locks. When you connect to a remote system, it checks if your private key matches the public key it has stored—if they match, you're granted access. This eliminates the need for password-based authentication, which can be vulnerable to brute force attacks and keyloggers.

What makes SSH key pairs particularly powerful is their ability to provide secure authentication without transmitting sensitive information over the network. Your private key never leaves your machine, making it virtually impossible for attackers to intercept your credentials during the authentication process.

Types of SSH Key Pairs and Their Mathematical Foundations

The cryptographic algorithms behind SSH keys represent some of the most elegant applications of number theory and algebraic geometry in modern computing. Let's delve deep into their mathematical foundations and understand how they provide the security we rely on daily.

RSA: The Prime Numbers Guardian

RSA's brilliance lies in the elegant use of prime numbers and modular arithmetic. Let's walk through a simplified but illustrative example of how RSA actually works in practice.

Suppose we want to create a small (insecure, but educational) RSA key:

First, choose two prime numbers:
p = 61 and q = 53
Calculate n (the modulus):
n = p × q = 61 × 53 = 3,233
Calculate the totient φ(n):
φ(n) = (p-1) × (q-1) = 60 × 52 = 3,120
Choose a public exponent e (commonly 65537):
Let's use e = 17 for this example (must be coprime with φ(n))
Calculate the private exponent d:
d = e⁻¹ mod φ(n) = 2,753

This gives us:

Public key: (n=3,233, e=17)
Private key: (n=3,233, d=2,753)

Here's how the encryption works with these numbers:

Message (m) = 123
Encryption: c = m^e mod n
c = 123^17 mod 3,233 = 855

Decryption: m = c^d mod n
m = 855^2,753 mod 3,233 = 123

In real RSA implementations, we use numbers that are typically 2048 or 4096 bits long. To put this in perspective, a 4096-bit number is roughly 1,234 decimal digits long. The security comes from the fact that factoring such large numbers is computationally infeasible with current technology.

Here's how you'd generate such a key in practice:

# Create a 4096-bit RSA key with custom settings
ssh-keygen -t rsa -b 4096 \
  -C "RSA-4096_$(date +%Y%m%d)" \
  -f ~/.ssh/id_rsa_4096 \
  -N "your_secure_passphrase"

Ed25519: Elliptic Curve Elegance

Ed25519 operates on a specialized Edwards curve, defined by the equation:
-x² + y² = 1 - (121665/121666)x²y²

This curve was carefully chosen for several mathematical properties that make it both secure and efficient. Let's break down how a point multiplication works on this curve:

The base point B has coordinates:

x = 15112221349535400772501151409588531511454012693041857206046113283949847762202
y = 46316835694926478169428394003475163141307993866256225615783033603165251855960

A private key is a 256-bit scalar k
The public key is the point k·B (scalar multiplication)

Here's a simplified example of point addition on the curve:

Given two points P1(x₁,y₁) and P2(x₂,y₂):

x₃ = (x₁y₂ + y₁x₂)/(1 + dx₁x₂y₁y₂)
y₃ = (y₁y₂ - ax₁x₂)/(1 - dx₁x₂y₁y₂)

where d = -121665/121666

The actual Ed25519 implementation uses field arithmetic modulo the prime 2²⁵⁵ - 19, chosen for efficient computation on modern 64-bit processors. When you generate an Ed25519 key:

# Generate Ed25519 key with maximum entropy
ssh-keygen -t ed25519 \
  -C "Ed25519_$(hostname)_$(date +%Y%m%d)" \
  -f ~/.ssh/id_ed25519 \
  -N "$(head -c 32 /dev/urandom | base64)"

The real power of Ed25519 comes from its resistance to various implementation attacks:

Signing equation:
R = rB
S = (r + H(R,A,M)a) mod l

where:
r = H(h_b,...,h_2b-1,M)
H = SHA-512
a = private key
A = public key
M = message
l = 2²⁵² + 27742317777372353535851937790883648493

ECDSA: The NIST Curves

ECDSA uses the NIST P-curves, which are defined over prime fields. The P-256 curve is defined by:
y² = x³ - 3x + b

where b = 0x5AC635D8AA3A93E7B3EBBD55769886BC651D06B0CC53B0F63BCE3C3E27D2604B

Let's examine a point multiplication on this curve:

Start with a base point G:

Gx = 0x6B17D1F2E12C4247F8BCE6E563A440F277037D812DEB33A0F4A13945D898C296
Gy = 0x4FE342E2FE1A7F9B8EE7EB4A7C0F9E162BCE33576B315ECECBB6406837BF51F5

For a private key k, the public key Q = kG is computed through repeated point doubling and addition:

Point Addition (P + Q):
s = (y₂ - y₁)/(x₂ - x₁) mod p
x₃ = s² - x₁ - x₂ mod p
y₃ = s(x₁ - x₃) - y₁ mod p

Point Doubling (P + P):
s = (3x₁² + a)/(2y₁) mod p
x₃ = s² - 2x₁ mod p
y₃ = s(x₁ - x₃) - y₁ mod p

Generate an ECDSA key using the P-521 curve:

# Generate ECDSA key with P-521 curve
ssh-keygen -t ecdsa -b 521 \
  -C "ECDSA_P521_$(date +%Y%m%d)" \
  -f ~/.ssh/id_ecdsa_521 \
  -N "$(openssl rand -base64 32)"

Extended Security Considerations

The security of these algorithms depends on different hard mathematical problems:

RSA: Integer Factorization Problem (IFP)

Given n = pq, find p and q
Time complexity: O(exp((log n)^(1/3) * (log log n)^(2/3)))

Ed25519: Elliptic Curve Discrete Logarithm Problem (ECDLP)

Given P and Q = kP, find k
Time complexity: O(√n) using Pollard's rho algorithm

ECDSA: Same as Ed25519, but with different curve parameters

Security level comparison:
RSA 3072-bit ≈ ECDSA/Ed25519 256-bit
RSA 15360-bit ≈ ECDSA/Ed25519 512-bit

Quantum Computing Impact

The advent of quantum computers poses different threats to these algorithms. Using Shor's algorithm:

RSA factoring time complexity:
Classical: O(exp((log N)^(1/3) * (log log N)^(2/3)))
Quantum: O((log N)^3)

ECDLP solving time complexity:
Classical: O(√n)
Quantum: O((log n)^3)

This is why post-quantum cryptography is becoming increasingly important, though it's not yet implemented in standard SSH keys.

Making an Informed Choice

The choice between these algorithms goes beyond mathematics—it's about balancing security, compatibility, and performance. Ed25519 represents the future: mathematically elegant, computationally efficient, and designed with modern threats in mind. Its implementation provides consistent security properties across different platforms, making it the ideal choice for new deployments.

For systems requiring broad compatibility, RSA with 4096 bits remains a solid choice. Its mathematical foundation has withstood decades of cryptanalysis, and while it may be computationally more intensive than modern elliptic curve approaches, its security margins are well understood.

When implementing any of these algorithms, the key is to ensure proper entropy during key generation. A strong random number generator is crucial for security, as even the most mathematically secure algorithm can be compromised by poor randomness. Modern systems use hardware random number generators and entropy pools to ensure strong key generation, but it's worth being aware of this critical foundation.

SSH Config File - Forgotten Gem

Eugene Zimin — Tue, 26 Nov 2024 07:04:16 +0000

For developers and system administrators managing multiple remote servers, the conventional approach of typing lengthy SSH commands such as those incorporating identity files, usernames, and complex domain names presents a significant operational burden. Consider the following typical command structure:

ssh -i ~/.ssh/special_key.pem username@ec2-123-45-67-89.compute-1.amazonaws.com

Such verbose commands, while explicit in their intent, can be transformed into more elegant alternatives through proper configuration. The SSH config file enables concise commands like ssh staging or ssh prod, reducing cognitive load and potential typing errors. This often overlooked tool enhances SSH workflow efficiency substantially, while maintaining the robust security measures inherent in SSH protocols.

Understanding the SSH Config File

The SSH config file, generally located at ~/.ssh/config, serves as a sophisticated configuration repository for SSH connections, embodying the Unix philosophy of maintaining simple, text-based configuration files. It functions as a sophisticated bookmark system with extensive customization capabilities, allowing for the definition of aliases and default settings. This approach to configuration management reflects the fundamental principles of systems administration: clarity, maintainability, and scalability.

Basic Configuration Implementation

Consider this foundational example of an SSH config file (~/.ssh/config), which demonstrates the essential elements of host configuration. Each section defines a specific connection profile, encapsulating all necessary parameters for establishing secure connections:

Host github
    HostName github.com
    User git
    IdentityFile ~/.ssh/github_key

Host staging
    HostName ec2-123-45-67-89.compute-1.amazonaws.com
    User ubuntu
    IdentityFile ~/.ssh/staging.pem
    Port 22

This configuration reduces verbose SSH commands to simplified versions, demonstrating the principle of abstraction in system administration. The complex underlying connection details remain hidden yet accessible when needed:

ssh github
# or
ssh staging

Advanced Features and Implementations

The SSH configuration system provides several sophisticated mechanisms for managing complex connection scenarios. These advanced features demonstrate the extensive capabilities of OpenSSH's configuration framework and its ability to handle diverse operational requirements.

Wildcard Pattern Implementation

Pattern matching facilitates efficient host management through the implementation of glob-style patterns, enabling sophisticated matching rules that apply configurations across multiple hosts. This pattern-based approach significantly reduces configuration redundancy and maintains consistency across similar environments. The pattern matching system employs asterisks (*) and question marks (?) as wildcards, following similar principles to shell globbing patterns.

Basic Pattern Examples

# Development environment configurations
Host dev-*
    User development-user
    IdentityFile ~/.ssh/dev_key.pem
    ForwardAgent yes
    StrictHostKeyChecking ask
    LogLevel INFO
    Port 22000

# Staging environment configurations
Host staging-*
    User staging-user
    IdentityFile ~/.ssh/staging_key.pem
    ForwardAgent yes
    StrictHostKeyChecking ask
    LogLevel INFO
    Port 22001

# Production environment configurations
Host prod-*
    User production-user
    IdentityFile ~/.ssh/prod_key.pem
    ForwardAgent no
    StrictHostKeyChecking yes
    LogLevel ERROR
    Port 22

Advanced Pattern Matching

The pattern matching system supports complex configurations through hierarchical rule application:

# Default settings for all hosts
Host *
    Compression yes
    TCPKeepAlive yes
    ServerAliveInterval 60
    ForwardAgent no

# Regional data center configurations
Host *.eu-west-*
    ProxyCommand ssh eu-jumphost -W %h:%p
    User european-user
    IdentityFile ~/.ssh/eu_key.pem

Host *.us-east-*
    ProxyCommand ssh us-jumphost -W %h:%p
    User american-user
    IdentityFile ~/.ssh/us_key.pem

# Service-specific patterns
Host db-*
    User database-admin
    Port 5022
    StrictHostKeyChecking yes

Host app-*
    User application-admin
    Port 5023
    StrictHostKeyChecking yes

Pattern Precedence Examples

The pattern matching system follows a hierarchical precedence model, where more specific patterns override general ones. This enables sophisticated configuration layering:

# Global defaults
Host *
    ForwardAgent no
    Compression yes
    ServerAliveInterval 60

# Environment-specific overrides
Host *-prod
    ForwardAgent no
    Compression no
    ServerAliveInterval 120
    StrictHostKeyChecking yes

# Region-specific overrides
Host *-prod-eu
    ProxyCommand ssh eu-prod-bastion -W %h:%p
    IdentityFile ~/.ssh/eu_prod_key.pem

# Service-specific configurations within production
Host db-*-prod
    User database-prod-admin
    Port 5022
    IdentityFile ~/.ssh/db_prod_key.pem

Real-world Implementation Examples

Consider a multi-environment, multi-region infrastructure setup:

# Development environments
Host dev-app-* dev-db-*
    User devops
    IdentityFile ~/.ssh/dev_key.pem
    ForwardAgent yes
    StrictHostKeyChecking no

    # Development-specific settings
    LogLevel DEBUG
    Compression yes
    TCPKeepAlive yes
    ServerAliveInterval 30

# Regional production configurations
Host prod-app-eu-* prod-db-eu-*
    User prod-admin
    IdentityFile ~/.ssh/prod_eu_key.pem
    ProxyCommand ssh eu-prod-bastion -W %h:%p

    # Production-specific settings
    LogLevel ERROR
    Compression no
    TCPKeepAlive no
    ServerAliveInterval 60
    StrictHostKeyChecking yes

# Database-specific configurations
Host *-db-*
    # Database server specific settings
    Port 5022
    IPQoS throughput
    Ciphers aes256-gcm@openssh.com,aes128-gcm@openssh.com

# Application-specific configurations
Host *-app-*
    # Application server specific settings
    Port 5023
    IPQoS lowdelay
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com

# Monitoring system access
Host monitor-*
    User monitoring
    IdentityFile ~/.ssh/monitoring_key.pem
    PermitLocalCommand yes
    LocalCommand logger "Monitoring system access: %h"

This comprehensive pattern matching implementation enables:

Environment segregation (development, staging, production)
Regional configuration management
Service-specific settings
Security policy enforcement
Performance optimization per service type
Audit logging for specific access patterns

The pattern matching system proves particularly valuable in large-scale infrastructures where maintaining individual host entries would become unmanageable. Through careful pattern design, administrators can implement consistent policies while maintaining the flexibility to override specific settings where necessary.

ProxyJump Configuration for Bastion Hosts

Bastion host traversal becomes straightforward through proper configuration, implementing the security principle of defense in depth. This approach enables secure access to internal resources while maintaining strict access controls. The ProxyJump feature, introduced in OpenSSH 7.3, replaces the older ProxyCommand methodology:

# Bastion host configuration
Host bastion
    HostName bastion.company.com
    User jumpuser
    IdentityFile ~/.ssh/bastion_key
    StrictHostKeyChecking yes
    LogLevel VERBOSE

# Internal server accessed via bastion
Host internal-server
    HostName 10.0.0.5
    User appuser
    ProxyJump bastion
    IdentityFile ~/.ssh/internal_key
    ForwardAgent no

For more complex scenarios, multiple jump hosts can be specified in sequence:

# Multi-hop bastion configuration
Host internal-private
    HostName 192.168.1.100
    ProxyJump bastion1,bastion2
    User internal-user
    IdentityFile ~/.ssh/internal_key

Connection Persistence Configuration

Optimizing connection management and minimizing authentication requests through persistent connections represents a significant improvement in both security and efficiency. The ControlMaster feature enables multiple SSH sessions to share a single network connection, reducing overhead and improving response times:

Host *
    # Connection sharing configuration
    ControlMaster auto
    ControlPath ~/.ssh/control/%C
    ControlPersist 1h

    # Connection keepalive settings
    ServerAliveInterval 60
    ServerAliveCountMax 3

    # TCP keepalive and compression
    TCPKeepAlive yes
    Compression yes

This configuration implements several important features:

ControlMaster auto: Automatically creates a master connection for subsequent sharing.
ControlPath: Defines the socket file location using %C for a unique hash of connection parameters.
ControlPersist: Maintains the master connection in the background for the specified duration.

The implementation can be further enhanced with environment-specific adjustments:

# Development environments - longer persistence
Host dev-*
    ControlPersist 4h
    ServerAliveInterval 30

# Production environments - stricter settings
Host prod-*
    ControlPersist 30m
    ServerAliveInterval 90
    ServerAliveCountMax 2

Advanced Authentication Configurations

The SSH config file supports sophisticated authentication mechanisms, including multi-factor authentication and certificate-based access:

Host secure-server
    HostName secure.company.com
    User secure-user
    CertificateFile ~/.ssh/user_cert.pub
    IdentityFile ~/.ssh/secure_key
    PKCS11Provider /usr/local/lib/opensc-pkcs11.so
    RequestTTY force
    PreferredAuthentications publickey,keyboard-interactive

Port Forwarding and Tunnel Configuration

Advanced port forwarding configurations enable secure access to remote services while maintaining security boundaries:

Host tunnel-host
    HostName gateway.company.com
    User tunnel-user
    # Forward local port 8080 to remote host's port 80
    LocalForward 8080 internal.company.com:80
    # Forward remote port 5432 to local PostgreSQL instance
    RemoteForward 5432 localhost:5432
    # Dynamic SOCKS proxy on local port 1080
    DynamicForward 1080
    # Ensure tunnel stays active
    ExitOnForwardFailure yes
    ServerAliveInterval 30

Advanced Configuration Patterns

Service-Specific Key Management
The implementation of distinct keys for different services enhances security through isolation and granular access control:

   Host github.com
       IdentityFile ~/.ssh/github_key

   Host gitlab.com
       IdentityFile ~/.ssh/gitlab_key

Environment-Specific Configurations
Different environments often require distinct logging levels and access patterns, reflecting the operational requirements of various deployment stages:

   Host prod-*
       User prod-user
       LogLevel QUIET

   Host dev-*
       User dev-user
       LogLevel VERBOSE

Port Configuration by Environment
Security requirements often necessitate different port configurations across environments, implementing the principle of least privilege:

   Host staging-web
       HostName staging.company.com
       Port 2222

   Host prod-web
       HostName prod.company.com
       Port 22

Security Implementation Guidelines

The implementation of robust security measures in SSH configuration requires a methodical approach to multiple aspects of system security. Each element of the configuration must be carefully considered to maintain the integrity and confidentiality of SSH connections while ensuring operational efficiency.

File System Security

The cornerstone of SSH security begins with proper file system permissions. These permissions form the first line of defense against unauthorized access and potential security breaches:

# Set restrictive permissions on SSH directory
chmod 700 ~/.ssh

# Set proper permissions on configuration file
chmod 600 ~/.ssh/config

# Secure private keys
chmod 600 ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_ed25519

# Ensure public keys are readable
chmod 644 ~/.ssh/*.pub

# Protect known_hosts file
chmod 600 ~/.ssh/known_hosts

Key Management Protocols

The implementation of a robust key management strategy requires careful consideration of several factors:

# Example of key-specific configurations
Host production-*
    # Specify permitted key types
    PubkeyAcceptedKeyTypes ssh-ed25519,rsa-sha2-512

    # Define preferred authentication methods
    PreferredAuthentications publickey

    # Disable password authentication
    PasswordAuthentication no

    # Restrict key forwarding
    ForwardAgent no

    # Enable strict host key checking
    StrictHostKeyChecking yes

    # Use specific identity file
    IdentityFile ~/.ssh/prod_%h_ed25519

Network Security Configurations

Implementation of network-level security measures helps protect against various attack vectors:

# Enhanced security configuration
Host *
    # Prefer modern, secure ciphers
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com

    # Specify secure key exchange algorithms
    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group16-sha512

    # Define secure MAC algorithms
    MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com

    # Disable X11 forwarding
    ForwardX11 no

    # Set connection timeout
    ConnectTimeout 60

    # Enable verbose logging for security events
    LogLevel VERBOSE

Environment-Specific Security Policies

Different environments require varying levels of security controls:

# Development environment
Host dev-*
    StrictHostKeyChecking ask
    UserKnownHostsFile ~/.ssh/known_hosts.dev
    LogLevel DEBUG3

# Staging environment
Host staging-*
    StrictHostKeyChecking yes
    UserKnownHostsFile ~/.ssh/known_hosts.staging
    LogLevel VERBOSE

# Production environment
Host prod-*
    StrictHostKeyChecking yes
    UserKnownHostsFile ~/.ssh/known_hosts.prod
    LogLevel ERROR
    IdentitiesOnly yes
    MaxAuthTries 3
    NoHostAuthenticationForLocalhost no

Secret Management Integration

Modern security practices often involve integration with external secret management systems:

# Example of retrieving credentials securely
Host secure-service
    # Use environment variables for sensitive data
    Match exec "test -n '$SSH_SECRET_KEY_PATH'"
        IdentityFile $SSH_SECRET_KEY_PATH

    # Integration with external secret management
    Match exec "vault-ssh-helper --verify"
        IdentityFile ~/.ssh/vault-signed-key

Audit and Logging Configurations

Implementing comprehensive logging helps in security monitoring and incident response:

Host *
    # Enable detailed logging
    LogLevel VERBOSE

    # Log connection attempts
    Match exec "logger 'SSH connection attempt to %h'"
        LogLevel DEBUG

    # Additional security logging
    PermitLocalCommand yes
    LocalCommand logger "SSH connection established to %h by %r"

Conclusion

The SSH config file represents a powerful tool for optimizing SSH workflows and implementing robust security practices. Through proper configuration, complex SSH commands can be condensed into efficient, memorable aliases while maintaining comprehensive security standards. This approach exemplifies the balance between usability and security in systems administration.

The management of multiple servers becomes considerably more efficient through effective SSH config file utilization. Beginning with basic host configurations and progressively implementing more advanced features allows for natural skill progression and workflow enhancement, following the principle of incremental improvement in systems administration.

It is worth noting that the examples presented constitute only a subset of the available functionality. The official OpenSSH documentation provides extensive additional options for advanced configuration and customization, offering numerous possibilities for further optimization and security enhancement.

Access to Google Cloud Virtual Machine through SSH

Eugene Zimin — Tue, 26 Nov 2024 05:30:06 +0000

Earlier in another article, we provided a detailed description of how to create and set up a Virtual Machine (VM) instance in the Google Cloud Platform (GCP) using Google Compute Engine (GCE). It is accessible here - Google Cloud: Provisioning a Virtual Machine and Accessing it via SSH.

Provisioning a GCP VM allows us to create our own powerful computing environment to serve as a sandbox or workspace for our further testing and experiments. Having successfully deployed a VM in GCP, the next crucial step is to establish a secure remote connection to this virtual machine. It is required by many reasons, starting from efficient administration and management up to the monitoring and observing it. Google Cloud Platform provides a straightforward method to configure SSH access to Compute Engine instances, facilitating an encrypted communication channel. In this article, we will outline the steps required to set up SSH access to a Google Compute Engine (GCE) virtual machine.

Few Words About SSH

Secure Shell (SSH) is a cryptographic network protocol and the de facto standard for secure remote login and command execution on Unix-like operating systems. SSH provides a secure encrypted channel over an unsecured network, preventing potential eavesdropping, packet sniffing, or man-in-the-middle attacks that could compromise login credentials or sensitive data. By leveraging strong encryption algorithms and authenticated key exchange, SSH ensures data integrity and confidentiality, making it an essential tool for securely administering remote servers, transferring files, and managing networked systems over insecure networks such as the internet.

SSH operates through the use of encrypted key pairs - a public key that gets placed on the remote server, and a private key that is kept secure by the client. The keys are very large numbers derived via a one-way mathematical function, making them virtually impossible to derive if intercepted during transmission.

When initiating an SSH connection, the client and server negotiate a secure symmetrical encryption cipher and session keys to use. This key exchange utilizes the private/public keys for authentication and protection against man-in-the-middle attacks. Once the secure tunnel is established, all subsequent commands, file transfers, and communications are encrypted between the client and server.
Figure 1. Communication schema for SSH exchange

SSH supports several strong modern encryption algorithms such as AES, Blowfish, 3DES, and ciphers like ChaCha20. The specific algorithms used can be configured on both ends based on policy and requirements. SSH also provides data integrity verification through cryptographic message authentication codes like HMAC-SHA1 to detect tampering.

By default, SSH operates on TCP port 22, but can be configured to use any port. It supports various user authentication mechanisms like passwords, public keys, security keys, and more. SSH key-based authentication is considered more secure than basic password authentication.

Configuring SSH on Local Machine

Before initiating an SSH connection to our Google Cloud VM instance, we'll need to have SSH configured and set up on our local machine. The steps vary slightly depending on the operating system, as Linux and macOS usually have preinstalled SSH client which requires minimal configuration, whereas Window requires a bit more steps to make initial setup ready.

Linux and macOS

Most Linux distributions and macOS come pre-installed with OpenSSH, a free open source SSH client and server utility. To check if it's installed, we can open a terminal and run:

ssh -V

# Response should be something like the line below
# OpenSSH_9.6p1, LibreSSL 3.3.6

This should display the installed OpenSSH version. If not installed, we can get it through the operating system's package manager, for Linux it may be usually apt or yum for macOS brew is very popular.

Windows

Windows 11 has pre-installed SSH client. To check it we need to open cmd application (a.k.a Command Prompt) and run the same command:

ssh -V

If we will see a similar output we saw for Linux and macOS (some names may vary), it means we have installed SSH client and we are good to go.

A little bit more complicated situation happens if Windows doesn't have preinstalled client. In that case we may use one of the following 3rd party applications to run it:

PuTTY for Windows
Git Bash for Windows

Generating Key Pairs

To establish secure SSH connections, we need to generate cryptographic key pairs consisting of a public and private key. The ssh-keygen utility allows us to create these key pairs locally on our machine. Before we start creating a key pair it may worth to overview file schema for the it.
Figure 2. Schema of SSH key pairs

We can run ssh-keygen in a terminal/command prompt and follow the prompts. Some common options include:

-t to specify the key type (e.g. rsa, ecdsa, ed25519)
-b to set the key length in bits (e.g. 2048, 4096)
-C to add a comment to identify the key

For example:

ssh-keygen -t rsa -b 4096 -C "user@example.com"

This generates an RSA key pair with 4096 bits length associated with the user@example.com email.

There are various key types we may use. Here are the most popular ones we may want to generate:

RSA - One of the oldest and most widely used key types. Recommended minimum key length is 2048 bits, with 4096 bits being more secure.
ECDSA (Elliptic Curve Digital Signature Algorithm) - This key type is based on elliptic curve cryptography which provides equal security strength with smaller key sizes compared to RSA.
Ed25519 - A modern EdDSA (Edwards-curve Digital Signature Algorithm) key type that provides better performance and security than RSA/ECDSA. Ed25519 uses elliptic curve cryptography with a 256-bit key length.

While RSA is the most widely compatible, Ed25519 keys are considered more secure and efficient for modern systems. We can check which key types our SSH server and client support using ssh -Q key.

During key generation, we have the option to protect the private key with a passphrase. While providing a passphrase enhances security, we need to enter it every time the key is used.

After generating, both keys - private and public - will be saved in the local SSH configuration folder. Usually (for Linux and macOS systems) this folder is under the following path - ~/.ssh/. Both keys by default use their name pattern as id-<algorithm-name>[.pub] , where extension .pub is used only for public key from the pair. As an example, user may get id-rsa and id-rsa.pub if there was RSA algorithm used, or id-ed25519 and id-ed25519.pub there was Ed25519 algorithm in use during key pair generation.

Next step would be to copy the content of the public key (which has .pub extension) to the remote server, in accordance with the Figure 2. We may copy the content of the public key and paste it on the new line of the ~/.ssh/authorized_keys file for the user which we will be connecting to. We can do that by many ways, but usually we should have access to the remote server via screen sharing or through the web interface, like with Google Compute Engine in GCP, where we can paste our public key.

NOTE FOR GCP
GCP requires to add at the end of the key userName and expireOn in form of JSON. We should do that manually to get the ready for pasting. Eventually we should have something like that:
ecdsa-sha2-nistp521 AAAAE2...== user@laptop.local {"userName":"<user@yourgcp.email>","expireOn":"2024-05-06T08:43:20+0000"}

At the same time private key (without .pub extension) remains securely stored on our local machine. We must keep the private key safe and never share it.

With our key pair ready, we can now proceed to configure server to use key pair only and disallow further access using username and password.

Configuring Server Side

While using SSH keys provides a much more secure authentication mechanism compared to passwords, many cloud providers, including Google Cloud Platform, enable password-based logins to SSH servers by default. However, relying solely on password authentication for SSH connections introduces potential security risks and is considered a poor practice, so we should consider to disable it and keep it disabled.

Main reasons why we should do that lay in the security considerations. Having password authentication enabled we allow anyone to try to connect to remote server and try out to brute force the password. Sometimes password are not that strong and such an attack might be successful.

That being said the advantages of disabling Password authentication for the remote server are as follows:

Enhanced Security: SSH keys utilize strong encryption algorithms and prevent brute-force and dictionary attacks that are common threats to password-based authentication. Keys are virtually impossible to guess or decrypt.
Eliminate Weak Passwords: Enforcing key-based authentication eliminates the risk of users setting weak or easily guessable passwords, which is a common vulnerability.
Audit Trail: SSH key pairs provide better audit trails and monitoring capabilities, as each key can be associated with a specific user or system.
Compliance: Many security standards and best practices, such as PCI-DSS, HIPAA, and NIST, recommend or mandate the use of key-based authentication over passwords for remote access to servers.

At the same time nothing comes without price. If any of the parts for the key pair would be lost or corrupted, it means we may also lost the access to the server, thus we should be very careful and keep generated key pair as good as possible.

To disable password authentication for the SSH server we need to modify the SSH daemon (sshd) configuration file. To do that we need to connect to the VM instance over SSH. We will use our new generated private key to connect to the server (don't forget to replace username and address of the server).

ssh -i ~/.ssh/id_ed25519 remote-user@server-ip-address

On the remote server open the SSH daemon configuration file as a superuser:

sudo nano /etc/ssh/sshd_config # may be changed to vim

In the file editor find the line #PasswordAuthentication yes and change it to PasswordAuthentication no and save the file and exit the editor.

Finally we need to restart the SSH daemon to make our changes take effect:

sudo systemctl restart sshd # (or command for your OS)

After making this change, the SSH server will only allow key-based authentication, and password logins will be disabled.

It's important to note that we should have at least one authorized SSH key added to the VM instance before disabling password authentication. Otherwise, we may get locked out of the system.

By enforcing key-based authentication and disabling password logins, we significantly enhance the security of our SSH server and remote access to our Google Cloud virtual machines, aligning with industry best practices for secure system administration.

Continuance

The next article in the cycle is dedicated to the debugging and troubleshooting different connection errors happening during establishing SSH connections - Debugging SSH connections: A Comprehensive Guide

JWT at a Glance

Eugene Zimin — Mon, 19 Aug 2024 04:32:11 +0000

Many of us heard about JWT and while JWT has become a buzzword in tech circles, it's frequently misunderstood or confused with OAuth 2.0 and OIDC, particularly among those who use it without fully grasping its intricacies. Mixing up JWT with OAuth 2.0 and OIDC is like tossing different fruits into a blender and calling it all "smoothie." It's especially messy when folks treat these tech tools like magic wands, waving them around without peeking under the hood.

What is JWT? By itself is a small piece of data that contains information about someone or something. It's like a small container which is able to keep and transfer information between two different destinations. Or it's like a digital badge that proves who the user is and what this user is authorized to do.

Let's imagine a library where anyone could walk in and borrow books without showing any identification. It would be chaos! The library needs a way to know who's borrowing what, and to ensure that only members can borrow books. In the digital world, JWT serves a similar purpose.

Session Management Based Authorization

Traditional web applications often use sessions to keep track of who's logged in. This is like giving someone a special badge when they enter the library. Library staff can see the badge and then if they need to know details - they will check them out inside the computer. Let's take a look at the example below.
Figure 1. Regular request-response flow

This diagram illustrates the basic flow of a web application, where the browser sends a request to the server, the server processes the request (potentially interacting with a database), and then sends a response back to the browser. Nothing complex, it's just a business as usual:

User sends a request to the Service, seeking for some information.
Service goes to the Database to retrieve that data from the long term storage.
Finally Service returns that data to the user in his browser to view.

Now, let's imagine if the Service holds sensitive personal information. Without security system to verify who's knocking on the door, there's a real danger of handing out private details to the wrong hands. It's like throwing a party and accidentally inviting the whole neighborhood when you meant to host a small gathering. In the digital world, this kind of mix-up isn't just embarrassing - it's a serious security risk that could expose confidential data to anyone who comes asking. That's a scenario is a "no-go" situation!

To prevent this situation to ever happen, let's think about implementing another service, which would be in charge to authenticate users and grant them special permissions which data they allow to access and which is not. Look at the diagram below:

Figure 2. Regular request-response flow with Login Service

What is changed on the diagram? Let's take a look again. We've introduced a new service which will be in charge for user's authentication and authorization. In other words we make this service exists to ensure proper user verification before granting access to the system's resources.

Prior to allowing any interaction with the data, this service requires users to authenticate themselves through a login process. Upon successful identification, the system initiates a session and issue a session identifier, or SessionID. In this context, a session is essentially a database record in Auth Users database indicating user is online and interacting with the system.

Upon successful authentication, all the rest interaction with the system no longer require the user to re-authenticate. Instead, the user's client (browser) simply keeps and provides back with every request the session identifier, which is securely stored in the Auth Users database. This identifier serves as a temporary credential, allowing the system to recognize and authorize the user for the duration of their session.

It may be shown like on the diagram below.

Figure 3. Session Management

Let's walk through this web application flow step by step, in a way that's easy to understand:

The process begins with the browser initiating an HTTP request to the server. This should happen only after user has been authenticated and have a Session ID in hands as a response from the authentication flow.
1. HTTP request itself is comprehensive and contains at least:
  - The destination URL
  - A token in the headers, any relevant cookies and optionally - a request body
2. The token in the headers carries a Session ID - a unique identifier acting as a digital passport for the user's session.
Upon receiving the request, the Service doesn't immediately process it. Instead, it has to forwards the session information to the Auth component for verification of the request sender, i.e user.
The Auth service cross-references the Session ID with the database, essentially checking the list of sessions and its mapping to the provided Session ID. If there is a match, the Auth component retrieves and returns the associated user data, which may include any necessary data about the user, like user ID, first and last name of the user, his email and user's role and permissions.
Armed with this user information, Service can now safely decide whether user is able to interact with the system or not. Eventually, Service compiles the requested information and sends it back to the browser as an HTTP response if the user is authorized or rejects it if the user is not authorized.

That's it! As this process occurs with each interaction between user and the system, we may consider the system is safe unless the session token is not stolen or brute forced.

Is that good or bad approach? There is no straightforward answer. One of its greatest benefits is that the backend fully controls the access to all protected resources. By any chance if any request would be considered suspicious, user might be easily kicked off of the system by erasing a record in the Auth Users database with the Session ID associated to him.

On the other hand - backend needs to keep the state about user's status - whether he authenticated or not. This stateful nature of session-based authentication brings challenges.

As every request to the system necessitates a database lookup to verify the session we can easily imagine what happens if our UI sends N (where N > 0) concurrent requests to retrieve information from the backend (hello React and other SPA):

Figure 4. Potential bottleneck with the session management

Take a look at the diagram above. On every user action, for example an initial page load, the UI may send 4 requests to fill out different page sections. That may happen as those data might be provided by different API Endpoints, like information about the user should be obtained from /api/v1/users and information about their appointments from another one, like /api/v1/appointments. More importantly, those requests are usually sent concurrently, as we want rapid page rendering in the user's browser.

Now let's estimate the cost of such an approach. Every user action with that page will push us to send 4 concurrent requests to the Service. Okay, that's doable. However each of those requests will also need be authorized, which means Service should ask Auth is that possible to proceed with them as it doesn't have any other information than Session ID which is nothing more than a link in Auth Users database. In that case Service needs to issue another 4 requests to the Auth and ask it about user information and details.

Easy calculation gives us that 1 user action creates 4 requests to one service in our backend and same amount for another service on the backend. In total, these eight actions will all result in database interactions. Let's imagine there are another services which also require confirmed authorization and we will understand the full picture.

Is it too much? The answer depends on the use case - sometimes it's necessary. However, is it possible to decrease the load on the database and other services?

Stateless Authorization Management

There is another way to handle system security and authorize users without using session management. This alternative approach tackles some of the scalability headaches we saw earlier and it is often implemented using JSON Web Tokens (JWT).

As we mentioned earlier JWT is a small container which holds some information for us. Compare it with the Session ID:

Figure 5. Compare Sessions with JWT approaches

In a JWT-based system, the server no longer needs to store session information. Instead, when user authenticates, the Auth generates a JWT - a compact, self-contained token that encapsulates all necessary user information and permissions. This token is then sent back to the client and stored, typically in local storage or a cookie.

Interesting thing happens on the next stage - when user needs to reach protected resources. Using previous approach we have to trigger authorization logic on every upcoming request on a different service which is Auth in the example above. What happened when we employ JWT?

Take a look at the diagram below.

Figure 6. JWT based Authorization flow

When a user makes a request to access a protected resource, their client (typically the browser) automatically includes the JWT in the request headers. This is usually done by setting the Authorization header to "Bearer token", where token is the actual JWT.

Upon receiving the request, the Service no longer needs to consult with the Auth service for each incoming request. Instead, it can verify the JWT's integrity directly and decode the JWT on its own. This is possible because the JWT is cryptographically signed, and the Service has access to the JWT secret key or public key (in case of asymmetric signing) used to sign the token by Auth at the JWT creation time.

Once the signature is verified, the Service can decode the JWT payload. This payload contains all the necessary information about the user - their ID, roles, permissions, and any other relevant data that was included when the token was created. All of this happens without any database lookups or calls to the Auth service.

Finally with the user information available from the decoded JWT, the Service can make immediate authorization decisions. It can check if the user has the necessary permissions to access the requested resource, all without any additional network calls or database queries.

JWT Structure

To make this flow happen JWT should be able to carry on some information. That includes not only some useful information we may want to store in the token, but also data, allowing to make the flow secured.

RFC 7519 defines the JWT as follows.

JSON Web Token (JWT) is a compact, URL-safe means of representing
claims to be transferred between two parties. The claims in a JWT
are encoded as a JSON object that is used as the payload of a JSON
Web Signature (JWS) structure or as the plaintext of a JSON Web
Encryption (JWE) structure, enabling the claims to be digitally
signed or integrity protected with a Message Authentication Code
(MAC) and/or encrypted.

Using that we can describe the structure of a JWT as of (usually) consisting of three parts: a header, a payload, and a signature, separated by . (dot) symbol. The payload contains claims about the user, such as user ID, role, and expiration time. The signature, created using a secret key known only to the server, ensures the token's integrity and authenticity.

Figure 7. JWT Structure

All these 3 parts are separated by dot symbol and following in the order of:

header . payload . signature

These three components are encoded using the BASE64URL algorithm. This encoding method is closely related to standard BASE64, but with a few key differences. In BASE64URL, the output replaces the plus sign + with a minus sign -, and the forward slash / becomes an underscore _. Additionally, BASE64URL omits the typical padding found in standard BASE64, which usually consists of equal signs = at the end of the encoded string. These modifications make the encoded output more suitable for use in URLs and other contexts where certain characters might cause issues.

Payload

If the payload (middle) part is more or less clear, as it is the container for user defined data, it still worth to take a look at some predefined fields over there.

RFC 7519 standard calls data inside the payload a Climes Set and defines three classes of JWT Claim Names

The JWT Claims Set represents a JSON object whose members are the
claims conveyed by the JWT.

There are three classes of JWT Claim Names: Registered Claim Names,
Public Claim Names, and Private Claim Names.

Registered Climes are those which defined by the standard and used by applications in case they considered as mandatory by developers. Here they are:

jti (JWT ID) Claim a case-sensitive unique identifier for the JWT, designed to prevent token replay and ensure uniqueness across multiple issuers
iss (Issuer) Claim identifies the JWT's issuer using a case-sensitive StringOrURI value, with processing typically application-specific
iat (Issued At) Claim specifies the JWT's creation time, allowing for age determination of the token
sub (Subject) Claim identifies the JWT's principal using a case-sensitive StringOrURI value, which must be locally or globally unique, and typically serves as the subject of the token's claims
aud (Audience) Claim specifies the intended recipients of the JWT as a case-sensitive string or array of strings, each containing a StringOrURI value, and requires the processing principal to be identified within this claim for token acceptance
exp (Expiration Time) Claim specifies the latest valid processing time for the JWT, with a small leeway often permitted for clock skew
nbf (Not Before) Claim sets the earliest time the JWT can be processed, with a small leeway often allowed for clock skew

Different applications may use different claims from the list above, but many of them rely at least on 2 - which are iat and exp to determine the lifecycle of the token itself and when it should be regenerated.

Public Claims contains user defined information and may include there everything in JSON format. Usually developers create it with user data, which is necessary to be shown on the UI, user permissions and some additional information.

Header

The JWT header plays a critical role in the token's functionality. This component, appearing at the beginning of the token, contains metadata about the token itself. Two key elements stored in the header are the token type and the hashing algorithm used for the signature.

The token type, denoted by the typ claim, usually specifies JWT to indicate that the token is indeed a JSON Web Token. This information helps systems quickly identify and process the token appropriately. The hashing algorithm, represented by the alg claim, specifies the cryptographic algorithm employed to create the token's signature. Common algorithms are HMAC SHA256 (HS256) or RSA SHA256 (RS256), but there are many others which are possible.

Signature

This section of the JWT is supposed to provide a mechanism of verification of the token integrity and authenticity. It is important and serves only for a one reason - prevent modification of any information at any stage.

This validation of the JWT happen on the backend Service, right after the Service received user's request for data. Server actually can recreate the signature using the same secret key (which is secretly stored on the backend) and algorithm (which is defined in the JWT header section). If the newly generated signature matches the one in the token, it confirms that the contents haven't been altered since the token's creation.

Refreshing JSON Web Tokens

There is a last but not least thing to consider - JWT lifetime. After issuing a JWT to a user, we actually allowing him to access the protected data in accordance with his level of permissions. However do we want to limit it in time? In other words - do we want that user confirms his authenticity after some time or not?

Even though this question is not that obvious, the answer is straightforward and positive. User must to re-confirms his authenticity and he should do that within a reasonable amount of time. It matters because we can't exclude the situation or JWT leakage, as what goes to browser is available to the rest of the world. JWT provides safety mechanism from data modifications (by signing them and later signature verification), but they do not save from reusing them.

This brings us to the concept of JWT refreshing. While limiting the lifetime of a JWT is necessary for security reasons, it can also lead to a poor user experience if the token expires too quickly, forcing frequent re-authentication. And this is where refresh tokens come into play.

A refresh token is another special token that can be used to obtain a new JWT once the original JWT has expired. Unlike JWTs, refresh tokens are typically stored server-side only and are associated with a specific user, using some of its unique data, like a hash of the JWT itself. They have a longer lifespan than JWTs but are only used to verify one thing - the old JWT is valid even if it is expired and we can safely issue a new one. In other words it let us know that original JWT was not modified and still can be trusted.

This approach balances security and user experience. It allows for short-lived JWT reducing the window of potential token misuse, while also allowing users to remain logged in for extended periods without explicit re-authentication.

Scalability Advantages of JWT

The stateless nature of JWT provides a significant boost especially for distributed systems in terms of scalability, when compared to traditional session-based authentication systems. This makes JWT an attractive approach for modern, distributed applications, such as those deployed in cloud environments or utilizing load-balanced architectures.

In traditional session-based systems, user authentication data is stored on the server side, typically in a database or in-memory store. Each time a user makes a request, the server must retrieve this session information to verify the user's identity and permissions. As the number of concurrent users grows, this approach can lead to increased load on the session store, potentially becoming a bottleneck in the system.

JWT, on the other hand, encapsulate all necessary authentication and authorization information within the token itself. When a user authenticates, the server generates a JWT containing the user's identity and permissions, then sends this token back to the client. For subsequent requests, the client includes this token, allowing the server to verify the user's identity and permissions without querying a central session store.

JWT's stateless nature aligns well with microservices architectures. In a microservices environment, different services can easily validate the JWT and extract necessary information without maintaining complex, centralized session management systems. This decoupling of authentication state from individual services enhances the overall scalability and flexibility of the system.

Next Steps

In the next article, we'll take a look further into the authentication and authorization cased by integrating JWT and two other important protocols: OAuth 2.0 and OpenID Connect (OIDC). Stay tuned to learn how combining JWT with OAuth 2.0 and OIDC can enhance your application's security posture while maintaining the scalability benefits we've discussed!

Using Domain-Driven Design to to Create Microservice App

Eugene Zimin — Mon, 24 Jun 2024 07:18:01 +0000

Creating scalable and maintainable applications remains a constant challenge in the software development industry. As digital ecosystems grow more complex and user demands evolve rapidly, developers face increasing pressure to build systems that can adapt and expand seamlessly.

Scalability presents a multifaceted challenge. Applications must handle growing user bases, increasing data volumes, and expanding feature sets without compromising performance. This requires careful architectural decisions, efficient resource utilization, and the ability to distribute workloads effectively across multiple servers or cloud instances.

Maintainability, on the other hand, focuses on the long-term health of the codebase. As applications grow in complexity, keeping the code clean, understandable, and easily modifiable becomes crucial. This involves creating modular designs, adhering to coding standards, and implementing robust testing strategies. A maintainable codebase allows for faster bug fixes, easier feature additions, and smoother onboarding of new team members.

The intersection of scalability and maintainability often leads to trade-offs. Highly optimized systems for scalability may sacrifice code readability, while overly modular designs for maintainability might introduce performance overhead. Striking the right balance requires deep domain knowledge, technical expertise, and a forward-thinking approach to software design.

The rapid pace of technological change adds another layer of complexity. Developers must create systems that not only meet current needs but also remain flexible enough to incorporate future technologies and methodologies. This forward-looking approach is essential in preventing technical debt and ensuring the longevity of the application.

Rapid pace of technological change adds another layer of complexity. Developers must create systems that not only meet current needs but also remain flexible enough to incorporate future technologies and methodologies. This forward-looking approach is essential in preventing technical debt and ensuring the longevity of the application.

Domain-Driven Design, first introduced by Eric Evans, emphasizes aligning software design with business needs. It provides a set of patterns and practices that help architects and developers create flexible, modular systems that can evolve with changing requirements. By centering the design process around the core domain and domain logic, DDD facilitates the creation of software that truly reflects the underlying business model.

Domain-Driven Design provides a framework for creating systems that are both scalable in terms of functionality and maintainable in terms of code organization. It offers strategies for managing complexity, facilitating communication between technical and non-technical stakeholders, and creating flexible systems that can evolve with changing business needs.

Domain-Driven Design Quick Overview

Domain-Driven Design (DDD) is a software development approach that places the project's core focus on the domain and domain logic. Introduced by Eric Evans in his seminal book, DDD provides a set of principles and patterns for creating complex software systems that closely mirror the business domain they serve.

At the heart of DDD lies the concept of the ubiquitous language. This shared vocabulary, meticulously crafted by developers in collaboration with domain experts, forms the foundation of the model. It ensures that all stakeholders, from business analysts to programmers, communicate effectively using terms and concepts directly related to the domain. This alignment significantly reduces misunderstandings and helps create software that truly reflects business needs.

Bounded contexts represent another cornerstone of DDD. These are explicit boundaries within which a particular domain model applies. In large systems, different parts may have different domain models, each with its own ubiquitous language. Recognizing and defining these boundaries helps manage complexity and allows teams to work independently on different parts of the system.

Aggregates are a key tactical pattern in DDD. An aggregate is a cluster of domain objects treated as a single unit for data changes. The aggregate root, a single entity within the aggregate, ensures the consistency of changes within the aggregate and controls access to its members. This pattern is particularly useful in defining clear boundaries for transactions and maintaining data integrity.

Domain events are another crucial concept in DDD. These represent significant occurrences within the domain. By modeling important changes as events, DDD facilitates loose coupling between different parts of the system, enabling more flexible and scalable architectures.

In the context of microservices architecture, DDD proves exceptionally valuable. The bounded contexts in DDD often align well with individual microservices, providing a natural way to decompose a complex system into manageable, independently deployable services. This alignment helps in creating a modular system where each service has a clear responsibility and a well-defined interface.

The strategic patterns of DDD, such as context mapping, help in understanding and defining the relationships between different bounded contexts or microservices. This is crucial for managing the complexity that arises from distributed systems and ensuring that the overall system remains coherent.

By applying DDD principles, developers can create software that's not only aligned with business needs but also inherently more maintainable and scalable. The focus on the core domain ensures that development efforts are concentrated where they provide the most value. The clear boundaries and well-defined interfaces facilitate easier changes and extensions to the system over time.

In the following sections, we will explore how these DDD concepts can be applied to create a Twitter-like application using a two-microservice architecture. This practical example will demonstrate how DDD can guide the design of complex systems, resulting in a flexible and maintainable solution.

Defining the Bounded Contexts

In applying Domain-Driven Design (DDD) to our Twitter-like application, the first step is to identify and define the bounded contexts. Bounded contexts are crucial in DDD as they delineate the boundaries within which a particular domain model applies. For our simplified Twitter-like platform, we'll focus on two primary bounded contexts, each corresponding to a microservice: UserManagementService and MessagingService.

UserManagementService Bounded Context

The UserManagementService encompasses all aspects related to user accounts and profiles. This context is responsible for managing user identities, authentication, and user-specific information. Within this bounded context, the core concepts include:

User - the central entity representing an individual account on the platform, which contains user-specific details such as display name, bio, and profile picture, authentication information like username and password, etc.
Role - defines user role in the whole system, whether he's a messages producer or messages consumer only

The ubiquitous language within this context includes terms like "register", "login", "logout", "authentication" and "authorization". The UserManagementService operates independently, focusing solely on user-related operations without direct concern for messaging or content creation.

MessagingService Bounded Context

The MessagingService handles all aspects of content creation, distribution, and consumption. This context is more complex as it manages both the creation of messages (tweets) and the subscription system. Key concepts in this bounded context include:

Message: The core entity representing a piece of content (similar to a tweet).
Subscription: Represents the follow relationship between users.
Feed: A collection of messages from subscribed users.
Producer: A user who creates content (messages).
Consumer: A user who consumes content from their subscriptions.

The ubiquitous language here includes terms like "post message", "subscribe", "unsubscribe", "consume message". This context deals with the dynamic aspects of the platform, managing the flow of content between users.

Context Interactions

While these bounded contexts are separate, they do interact. The MessagingService needs to reference users, but it does so without delving into the internal complexities of user management. Instead, it might use a simplified representation of a user, containing only the necessary information for messaging purposes.

Figure 1. Context Interactions between bounded contexts

For instance, when a user posts a message, the MessagingService doesn't need to know about the user's password or email address. It only needs a user identifier and perhaps a display name. This separation allows each context to evolve independently while maintaining necessary connections.

This separation is further reinforced by implementing the database-per-service pattern. In this approach, each microservice maintains its own dedicated database. The UserManagementService has a database that stores comprehensive user information, including sensitive data like passwords and email addresses. On the other hand, the MessagingService database focuses on message content, user subscriptions, and a minimal set of user data required for its operations.

To maintain necessary connections between services, the MessagingService keeps a minimal replica of user data required for its operations. This typically includes the user ID only. If MessagingService requires some data related to the user, it may request that information from UserManagementService. This may be useful for checking user roles for example or user name to display.

Technically speaking, this approach introduces some data redundancy, however organized correctly, such a redundancy may be reviewed as nothing much but as foreign keys used between different tables in RDBMS.

It allows each service to operate independently most of the time, enhancing overall system resilience. It also aligns well with the DDD principle of respecting bounded context boundaries, as each service has full control over its own data model and storage.

Context Mapping

To manage the relationship between these bounded contexts, we employ context mapping. In this case, we might use a Customer-Supplier relationship, where the UserManagementService (supplier) provides necessary user data to the MessagingService (customer). This could be implemented through API calls or message queues, ensuring that each service has the information it needs without tightly coupling the two contexts.

Implementation of this context mapping involves several key aspects. First, the UserManagementService exposes a well-defined API that the MessagingService can use to retrieve essential user information. This API is designed to provide only the necessary data, such as user IDs and user roles, without exposing sensitive information. For example, when a new message is posted, the MessagingService might make an API call to the UserManagementService to verify the user's existence and his role.

Figure 2. Context Mapping between different bounded contexts

As part of the Customer-Supplier relationship, we define clear Service Level Agreements (SLAs) between the services. These SLAs specify the expected request and response structure, timeline for API calls, the frequency of event publications, and the handling of service outages.

Leveraging these patterns we will create a foundation for a modular, maintainable system. Each context has a clear responsibility and a well-defined interface, allowing for independent development and scaling. This separation also provides flexibility for future expansions, such as adding new features or integrating with external systems.

In the following sections, we'll delve deeper into each of these bounded contexts, exploring their domain models, aggregates, and services in detail.

User Management Service

The UserManagementService forms a crucial part of our Messaging application, handling all aspects related to user accounts and roles. This service embodies its own bounded context, focusing solely on user-related operations. Let's delve into the key components of this service, structured according to Domain-Driven Design principles.

Domain Model

At the core of the UserManagementService lies the User aggregate. In DDD, an aggregate is a cluster of domain objects treated as a single unit. The User aggregate encapsulates all user-related data and behavior.

The User aggregate root contains essential attributes such as userId, username, email, and password. It also includes methods for user authentication, user roles and user authorization. By centralizing these responsibilities within the User aggregate, we ensure that all user-related operations maintain data consistency and adhere to business rules.

Associated with the User aggregate are several value objects. These are immutable objects that describe characteristics of the User but have no conceptual identity. Examples include Email and Password. These value objects encapsulate validation logic, ensuring that email addresses are properly formatted in accordance with RFC 2822 and passwords meet security requirements by its length and other criteria.

The Profile value object represents user-specific details like displayName, bio, and profilePictureUrl. By modelling Profile as a separate value object, we allow for easy expansion of profile-related features without cluttering the User aggregate.

Repository

The UserRepository interface defines methods for persisting and retrieving User aggregates. This abstraction allows us to separate the domain model from the underlying data storage mechanism. The implementation of this repository interfaces with our chosen database system, in this case, MySQL.

The repository includes methods such as findById, findByUsername, and regular CRUD operations (CREATE-READ-UPDATE-DELETE). It also provides more specific query methods like findByEmail, which might be used during the user registration process to ensure email uniqueness.

Data model, which is also scoped to User aggregation only, might be represented as on the figure below.

Figure 3. User Management Data Model

Domain Services (Domain Flows)

While most business logic resides within the User aggregate, some operations involve multiple aggregates or require external resources. For these, we use domain services which may exist as separate services or might be defined as modules inside a single service. As of now we will prefer to define those operations as separate reusable modules inside one service.

The UserRegistrationModule handles the process of user registration, which might involve checking for existing users, validating input, and triggering welcome emails. This module coordinates between the User aggregate, UserRepository, and potentially external services like email providers.

The AuthModule manages user login processes, including password verification and generation of authentication tokens, as well as handles requests to identify user roles. This module works closely with the User aggregate but keeps the authentication and authorization logic separate, allowing for easier updates to authentication and authorization mechanisms in the future.

ApplicationGatewayModule is responsible for communication with other external components and services. In some other sources it is also called a Controller, however we try to include here only logic, responsible for terminating incoming traffic, data serialization, input validation and forming responses to return them to the requesters. This module acts like a communication facade between different external services and internal modules, providing SLA for them, and at the same time keeping flexibility of internal modules.

The last module we may want to use is DataAccessLayerModule. We already defined the data model, now we need to determine the way to communicate with it. Using direct calls to the database from any point of the code is possible but may not be considered as a good idea because of many reasons, main ones are correlated to the high coupling and low cohesion between data model and application logic. Using this module we may decompose and abstract access our logic from accessing data. Instead of creating a specific SQL query to select specific user and access his permissions, we may simply create a method, called getUserWithRoles, and hide the implementation behind it. In that case we reach high consistency by calling the only one method, which we will be a single point to access data.

Figure 4. Block Diagram of Domain Modules

Infrastructure Concerns

While not strictly part of the domain model, the UserManagementService also addresses several infrastructure concerns. These include implementing security measures like password hashing and token-based authentication, setting up database connections and ORM mappings, and configuring API endpoints.

The service also implements event publishing for significant domain events. When a user updates their profile, for instance, the service publishes a UserProfileUpdated event. Other services, like our further considering NotificationService, can subscribe to these events to maintain consistency across the system and deliver different state changes.

By structuring the UserManagementService according to DDD principles, we create a robust, maintainable service that clearly encapsulates all user-related functionality. This design allows for easy extension of user management features and provides a solid foundation for our Messaging application.

MessagingService

The MessagingService forms the core of our Messaging application's content creation and distribution system. This service encompasses its own bounded context, managing messages (tweets) and subscriptions. Let's explore the key components of this service, structured according to Domain-Driven Design principles.

Domain Model

The central aggregate in the MessagingService is the Message. This aggregate encapsulates the content created by users, along with metadata such as creation time and author information. The Message aggregate root contains attributes like messageID, content, authorID (which is a userID), and createdAt. It also includes methods for creating, editing (if allowed), and deleting messages.

Another crucial aggregate is the Subscription, representing the follow relationship between users. The Subscription aggregate contains a producerID and a subscriberID, which are no more than userID correlated to the appropriate data from the UserManagementService, along with subscription status and creation date. This aggregate is responsible for managing the relationships between content producers and consumers.

Value objects in this context might include MessageContent, which encapsulates the actual text or media content of a message and its author ID, to establish and determine the subscription status.

Repository

The MessagingService might include several repositories in case we would like to split it into standalone services to manage messages and subscriptions. In this article we consider that functionality as closely related and manage it under the single umbrella of the MesagingService and within the single database for this service.

The MessageRepository handles persistence and retrieval of Message aggregates. It includes methods like regular CRUD, findByMessageId, findByAuthorId, and findBySubscriberId. We may also implement another method allowing users to filter messages by the time when they were created, which may be another convenient use case, and call this method as findByCreatedAt.

As it is seen, this database includes all those entities related to the messages and subscriptions. It's schema might be represented like on the figure below.

Figure 5. Messaging Service Data Model

Domain Services (Domain Flows)

In this service we may define the same amount of modules as in the UserManagementService - 4, where 2 of them would be identical - DataAccessLayerModule and ApplicationGatewayModule, as these modules' responsibilities are generic and service agnostic. These 2 modules are intended provide border modules for data communication between MessagingService other ones. Eventually their goal is to convert and adjust incoming or outgoing data with its internal representation happening inside this Domain.

MessagingModule in this case this is the core module for the whole service as it manages messages creation, retrieval and editing (if supported). This modules accept data from different sources, including user input, like message's text, user data, obtained from UserManagementService by using User ID, provided by user and using internal logic performs the requested operation - create, retrieve or edit (if supported) message.

The SubscriptionModule handles the complex logic of managing user subscriptions, including creating new subscriptions, handling unsubscribes, and managing subscription statuses like muting or blocking.

Figure 6. Block Diagram of Domain Modules

Infrastructure Concerns

The MessagingService implements several infrastructure-level features to support its operations, as concerns for this service should be different than for the UserManagementService. These include setting up message queues for handling high volumes of new messages, implementing caching mechanisms for frequently accessed subscriptions, and managing database connections for efficient data retrieval.

The service also may publish domain events such as MessageCreated and SubscriptionChanged to be consumed by other parts of the system or external services for features like notifications or analytics.

By structuring the MessagingService according to DDD principles, we create a robust and scalable system for handling the core functionality of our Messaging application. This design allows for efficient management of messages, subscriptions, while providing clear boundaries for future extensions and optimizations.

Scalability and Performance Considerations

As our application grows in popularity, scalability and performance become critical concerns. This section explores strategies to ensure our microservices architecture can handle increasing loads while maintaining responsiveness and reliability. Generally speaking those techniques below are not critical for application functionality, however they become those when the system starts experiencing the high load.

Horizontal Scaling

Both the UserManagementService and MessagingService are designed to be stateless, allowing for easy horizontal scaling. As user traffic increases, we can deploy multiple instances of each service behind a load balancer. This approach distributes the load across multiple servers, improving overall system capacity and resilience.

Database Scaling

As the volume of data grows, database performance can become a bottleneck. To address this, we may implement several strategies for that.

For the UserManagementService, we establish a system of read replicas, as this service load assumes prevail retrieval operations over writing ones. This setup allows us to distribute read operations across multiple database instances, significantly reducing the load on the primary database. By directing read-heavy operations to these replicas, we ensure that the primary database can focus on write operations, thereby improving overall system performance and responsiveness.

The MessagingService, which deals with a substantially larger volume of data, requires a more robust scaling solution. Here, we may implement database sharding, a technique that involves partitioning data across multiple database instances based on user ID or message ID ranges. This approach not only distributes the data itself but also spreads the query load across multiple servers. As a result, we can handle a much larger number of concurrent operations and store a greater volume of data than would be possible with a single database instance.

In addition to these structural changes, we should place a strong emphasis on query optimization and indexing. Our database administrators and developers work in tandem to continuously monitor query performance, identifying frequently used query patterns and ensuring that appropriate indexes are in place to support them. This ongoing process of refinement helps maintain database efficiency even as the volume and complexity of data grow.

Caching Strategies

Caching plays a pivotal role in performance optimization strategy, serving as a crucial layer between our services and the database. In the UserManagementService, we implement a sophisticated caching mechanism for user profile data. We may utilize any in-memory storage, like Redis, for high-performance in-memory data store, where we may cache frequently accessed user information. This approach significantly reduces the number of database queries required for common operations, such as retrieving user details for display or authentication purposes.

The MessagingService employs a more complex, multi-tiered caching strategy to handle the high-volume, real-time nature of social media feeds. We may use a combination of in-memory caching with Caffeine for the most recent and frequently accessed feed entries, providing near-instantaneous access to this hot data. For older or less frequently accessed feed data, we leverage Redis as a second-level cache. This tiered approach allows us to balance the need for ultra-fast access to recent content with the ability to efficiently store and retrieve a larger volume of historical data.

As our system scales horizontally, maintaining cache consistency across multiple service instances becomes critical. To address this, we configure Redis in cluster mode, ensuring that our caching layer is both distributed and consistent. This setup allows us to scale our caching infrastructure in tandem with our application services, maintaining performance as user numbers grow.

Eventual Consistency and CAP Theorem Considerations

In designing our distributed system, we've had to grapple with the fundamental trade-offs described by the CAP theorem, which states that in the event of a network partition, a distributed system must choose between consistency and availability. Given the nature of our application, where the immediacy of information flow is a key feature, we've opted to prioritize availability and partition tolerance over strict consistency in many scenarios.

Figure 7. CAP Theorem Problem

This decision manifests in our embrace of eventual consistency. For instance, when a user updates their profile information in the UserManagementService, we acknowledge and apply this change immediately in that service. However, we accept that there may be a short delay before this update is reflected in the MessagingService or in cached data across the system. During this brief period, different parts of the system may have slightly different views of the user's data.

While this approach introduces the possibility of temporary inconsistencies, it allows our system to remain highly available and responsive, even in the face of network issues or high load. We mitigate the impact of these inconsistencies through careful system design, such as including timestamps with data updates to help resolve conflicts, and by setting appropriate user expectations about the speed of data propagation.

This eventual consistency model extends to other areas of our system as well, such as follower counts or message distribution. By relaxing the requirement for immediate, system-wide consistency, we can process high volumes of operations more efficiently, leading to a more scalable and responsive system overall. However, we're also careful to identify those areas of our application where strong consistency is necessary, such as in financial transactions or security-critical operations, and design those components accordingly.

Conclusion

The journey of creating a Twitter-like application using Domain-Driven Design (DDD) and microservices architecture offers valuable insights into building complex, scalable systems. Through our exploration of the UserManagementService and MessagingService, we've demonstrated how DDD principles can guide the development of a robust, maintainable, and flexible application architecture.

By focusing on clearly defined bounded contexts, we've created services that are independently deployable and scalable, yet cohesive in their overall functionality. The use of aggregates, repositories, and domain services has allowed us to encapsulate complex business logic within each service, promoting code that is both more understandable and more closely aligned with the underlying business domain.

Our approach to data management, including the implementation of the database-per-service pattern and sophisticated caching strategies, showcases how modern distributed systems can handle large volumes of data while maintaining performance and responsiveness. The adoption of event-driven architecture for inter-service communication highlights the power of loose coupling in creating systems that are both resilient and adaptable to change.

Throughout this process, we've had to make important architectural decisions, balancing concerns such as data consistency, scalability, and system complexity. Our choice to embrace eventual consistency in certain areas of the application demonstrates the nuanced thinking required when designing distributed systems, taking into account the trade-offs described by the CAP theorem.

While our implementation focuses on core functionalities, omitting features like notifications and commenting, the architecture we've designed provides a solid foundation for future expansion. The principles and patterns we've applied can be extended to accommodate new features and services as the application grows.

It's important to note that the journey doesn't end with implementation. Continuous monitoring, performance optimization, and iterative improvement are crucial for maintaining and evolving such a system. As user behavior changes and new technologies emerge, our Twitter-like application must be ready to adapt and scale accordingly.

The combination of Domain-Driven Design and microservices architecture offers a powerful approach to building complex, scalable applications. By focusing on clear domain boundaries, embracing eventual consistency where appropriate, and leveraging modern technologies for data management and inter-service communication, we can create systems that are not only capable of handling current demands but are also well-positioned to evolve with future needs.

Database per Service as a Design Pattern

Eugene Zimin — Thu, 13 Jun 2024 18:46:55 +0000

In the world of modern software development, the microservices architecture has emerged as a popular approach to building scalable and maintainable applications. This architecture breaks down a monolithic application into smaller, independent services that can be developed, deployed, and scaled independently.

However, if there are multiple approaches to perform the decomposition of the service functionality, it is still questionable how to split the data and whether it is necessary at all. This brings us to the heart of the Database per Service pattern and its role in the modern architecture.

A Bit of History

The journey of software architecture and data management is one of evolving complexity and increasing distribution. To understand where we are today with patterns like Database per Service, it's illuminating to trace the path that led us here. This journey begins with traditional monolithic architectures, where applications were built as single, tightly-integrated units sharing a common database.

Traditional Monolithic Architecture

Before diving deeper into this design pattern, it's important to understand the traditional approach and why the problem arose.

In the epoch of monolithic applications, there were no such problems with how to access data, because there was no definition of concurrent data access from multiple applications. The monolithic architecture, with its single, shared database, provided a straightforward approach to data management. All components of the application had direct access to the entire database, and data consistency was primarily managed through database transactions and locking mechanisms.

Figure 1. Application - Database communication model

This simplicity, however, came at a cost. As applications grew in scale and complexity, the limitations of the monolithic approach became increasingly apparent. The tight coupling between application components and the database made it difficult to evolve the data model without impacting the entire system. Schema changes often required coordinated updates across multiple parts of the application, leading to complex deployment processes and potential downtime.

Moreover, the lack of data encapsulation meant that any part of the application could potentially access and modify any data, increasing the risk of unintended side effects and data corruption. This became particularly problematic as teams grew and different groups of developers worked on different features within the same monolith.

The advent of service-oriented architectures and, later, microservices, brought these data access and management challenges into sharp focus. Suddenly, architects and developers were faced with questions that didn't exist in the monolithic world: How do we ensure that each service has access to the data it needs without creating tight coupling? How do we maintain data consistency when transactions span multiple services? How do we scale data access patterns independently for services with different performance requirements?

N-Layer Architecture

The first attempt to answer this question was the N-layer architecture style. This approach sought to introduce a degree of separation between different concerns within an application or by organizing code into distinct layers, typically presentation, business logic, and data access, or even further - extracting that functionality into separate application. Each layer is supposed to have a specific responsibility and would communicate with adjacent layers through well-defined interfaces.

Figure 2. Example of layered system with external DAL

In the context of data management, the N-layer architecture often involved creating a dedicated data access layer (DAL). This layer encapsulated all interactions with the database, providing a set of methods or services that the business logic layer could use to retrieve or manipulate data. The idea was to abstract away the complexities of data storage and retrieval, making it easier to change the underlying database technology or structure without impacting the rest of the application.

While the N-layer architecture brought some benefits, such as improved code organization and the potential for reusability of data access components, it did not fully address the challenges of distributed systems. The data access layer, though separate, was still typically tightly coupled to a single, shared database. This meant that while individual services or modules might have their own business logic, they were still dependent on a central data store, which could become a bottleneck and a single point of failure.

Moreover, the N-layer approach did not inherently solve issues of data ownership and autonomy. Different services or modules might need to evolve their data models at different rates or have distinct performance and scaling requirements. A shared data access layer and database made it difficult to cater to these diverse needs without complex coordination and potential disruptions.

All these problems lead system and software architects to the next step - creating a design pattern called "Database per service".

Database per Service

As systems continued to grow in complexity and the need for genuine decoupling became more pressing, architects began to look beyond layered architectures to more distributed patterns. This led to the consideration of approaches like the Database per Service pattern, which takes the concept of separation a step further by giving each service not just its own data access code, but its own dedicated database.

The transition from N-layer architectures to patterns like Database per Service reflects a fundamental shift in thinking about data management in distributed systems. Rather than seeing data access as a shared concern to be abstracted away, modern approaches often treat data as an integral part of each service's domain, to be owned and managed as autonomously as possible.

Figure 3. Database per Service Design Pattern in Action

This evolution, however, is not without its challenges. While the N-layer architecture grappled with issues of code organization and reusability, Database per Service patterns must contend with questions of data consistency, duplication, and inter-service communication. The decoupling of data stores provides greater flexibility and resilience, but it also requires careful consideration of how to maintain a coherent view of data across the system.

As we continue to explore the Database per Service pattern, we'll see how it builds upon the lessons learned from earlier architectural styles like N-layer, while introducing new paradigms that are better suited to the realities of modern, distributed applications. The goal remains the same: to create systems that are scalable, maintainable, and responsive to change. But the means of achieving this goal have evolved, reflecting the ever-increasing complexity and scale of the software we build.

Benefits of the Database per Service Pattern

We need to understand that Database per Service pattern is not a silver bullet. However, even introducing its own set of challenges, it offers several compelling benefits that have made it an increasingly popular choice in microservices architectures. These advantages stem from the fundamental principle of decoupling, not just at the application level, but at the data level as well.

Enhanced Scalability

One of the primary drivers behind the adoption of microservices, and by extension the Database per Service pattern, is scalability. Consider 2 different examples, where the first one is a traditional monolithic application with a shared database.

Figure 4. Loading Map Inside Monolith Application

With this approach scaling often means scaling the entire system, even if only one component is under heavy load. This can lead to inefficient resource utilization and increased costs. Let's count it up.

Imagine that our application runs on a VM with 4 CPU and 16Gb of RAM. That's a simple web application, providing functionality for storing and reading messages by different users based on their subscriptions, like in Twitter. Application, being built as a monolith one, is able to consume no more than 25% resources of the VM to leave some room for scaling and operational overhead.

Now, for the sake of simplicity let's imagine that each of the component consumes about 300Mb of RAM and about 100Mb of RAM is integration overhead for the monolith. In this case we notice that our application is lack of resources and we would like to create another instance. Would it be a good idea? No. Do we have a choice? No.

Figure 5. Scaling Monolith Application

Creating another instance of the whole application, we have to scale all components, sitting inside it, even those ones, which are totally not under high load, because we can't differentiate them. Eventually it will lead us to the state where we will have 2 applications with redundant functionality, consuming memory and CPU, which cost us money.

Figure 6. Scaling Microservice Application

With a database per service, each service and its corresponding database can be scaled independently based on its specific demands. A service handling creating messages and reading them, for instance, might need to scale to handle a high volume of requests, while an user registration might not require scaling at all. By allowing each service to scale its own database, we can fine-tune their infrastructure, allocating resources where they're most needed and optimizing performance without unnecessary overhead.

Cleaner Data Ownership

In large systems, it can sometimes be unclear which part of the application "owns" particular data, leading to confusion, inconsistencies, or unintended side effects when data is modified. The Database per Service pattern establishes clear boundaries of data ownership.

Imagine the scenario with the messaging application.

Figure 7. Data Ownership in Messaging Application

Adopting a Data-Driven Design approach, we can identify two primary services with distinct data responsibilities: the User Management Service and the Messaging Service. When examining the data model for the Messaging Service, it's evident that each message must include information about its author—a user registered within the User Management Service.

A simplistic solution might involve replicating user and role data within the Messaging Database, which is directly linked to the Messaging Service. The rationale seems straightforward: every time a message is created and persisted, we need to authenticate the user and verify their permissions for message production and storage.

However, duplicating user and role information in the Messaging Database would result in the User and Roles tables existing concurrently in two separate databases. If we consider the User Management Database as the authoritative source for user-related data, we introduce a data synchronization challenge—ensuring that information remains consistent across both locations. This scenario effectively violates the Single Responsibility Principle from the S.O.L.I.D. pattern. The Messaging Database would now require updates not only when message data changes but also when user information is modified, conflating its responsibilities.

Such an arrangement complicates data ownership and management. Ideally, each service should have clear authority over its domain-specific data, with well-defined boundaries that promote maintainability and reduce interdependencies. Duplicating user data within the Messaging Service blurs these lines, creating potential for data inconsistencies and increasing the complexity of system-wide updates.

A more robust approach would respect the natural boundaries between services, allowing the User Management Service to retain sole ownership of user-related data. Rather than replicating this information, the Messaging Service would reference it as needed, storing only links or pointers, while relying on the User Management Service for authoritative data. This strategy upholds the principles of service autonomy and data integrity, cornerstones of effective microservices architecture.

Figure 8. Correct Data Ownership

Now each service becomes the sole writer to its database and the authoritative source for its domain of data. This clarity reduces complexity in data management, helps maintain data integrity, and makes it easier to reason about the system's behavior. While services may still need to share data, this is typically done through well-defined APIs rather than direct database access or redundant data, reinforcing the principles of encapsulation and loose coupling.

Improved Fault Isolation

In a system with a shared database, issues like outages, performance degradation, or data corruption can have wide-ranging impacts, potentially bringing down the entire application. The Database per Service pattern provides a higher degree of fault isolation.

If one service's database experiences problems, the impact is largely contained to that service. Other parts of the system can continue to function, improving overall resilience. This isolation also simplifies troubleshooting and recovery, as teams can focus on addressing issues within a specific service and its database without needing to coordinate across the entire system.

Let's take a look at the example. We will use the same Messaging application and now consider the worst-case situation: the application relies on a shared database, and this database suddenly becomes unavailable.
Figure 9. Database/DAL failure

The consequences of this failure are immediate and far-reaching. With the database inaccessible, the entire service effectively grinds to a halt in terms of its core functionality. No component within the system can retrieve or manipulate data, rendering the application useless from an end-user perspective.

This outage isn't merely an inconvenience; it represents a single point of failure that can paralyze operations across the board. User authentication may fail without access to account data. The messaging feature, the heart of the application, cannot display existing conversations or allow the sending of new messages. Even secondary functions like user settings or search capabilities are compromised.

The ripple effects extend beyond just data retrieval. Write operations—such as storing new messages, updating user profiles, or logging system activities—also come to a standstill. This not only disrupts current user interactions but can lead to data loss if the system isn't designed with robust queueing and retry mechanisms.

Furthermore, the shared nature of the database means that even services or components with minimal data needs are impacted. A status monitoring tool that might normally operate with cached data could fail if it periodically needs to check in with the database. Administrative interfaces, often crucial for diagnosing and addressing issues, may themselves be rendered inoperable.

This scenario underscores one of the fundamental vulnerabilities of a monolithic architecture with a shared database: lack of isolation. When all services depend on the same data store, they share the same fate when that store fails. There's no graceful degradation, no ability for some parts of the system to continue functioning while others are impaired.

Figure 10. Database failure in distributed system

Now let's reimagine the same scenario, but within the context of a system employing the Database per Service pattern. When a database failure occurs, its impact is largely contained to the service directly connected to it, leaving other services unaffected and operational.

In this illustration, we see that the database linked to the Commenting service has gone offline. The immediate consequence is that the Commenting service loses its ability to retrieve or store data. Users might encounter error messages when trying to post new comments or view existing ones. However—and this is the critical difference—the ripple effect stops there.

The User Management service, with its own dedicated database, continues to authenticate users and manage account information unimpeded. The Messaging service still facilitates the exchange of direct messages between users. Even a hypothetical Analytics service could keep tracking user engagement metrics, perhaps with a temporary blind spot for comment-related events.

This resilience stems from the decoupling inherent in the Database per Service pattern. Each service not only has its own codebase and deployment pipeline but also its own data persistence layer. This autonomy means that a failure in one part of the system doesn't automatically cascade into a system-wide outage. Instead, the application gracefully degrades, maintaining core functionalities even as some features become temporarily unavailable.

Of course, in a well-designed microservices architecture, services rarely exist in total isolation. They often depend on each other for certain functionalities. In our example, the Messaging or User Management services might need to interact with each other -to perform users authentication and authorization, so the failure of any core services might still be critical to the overall functionality, however the rest of the services have to analyze the error response and notify customers together with system administrators about failure. Moreover, they can also narrow down the issue failure point, which makes debugging easier.

Services should be built to handle errors or timeouts from their dependencies gracefully. If the Messaging service tries to fetch recent comments for a conversation and receives an error response from the Commenting service, it shouldn't crash. Instead, it might display a message to the user that comments are temporarily unavailable, while still showing the rest of the conversation history. Similarly, the User Management service could queue moderation actions to be processed once the Commenting service is back online, rather than blocking user management tasks.

Greater Development Autonomy

One of the challenges in large software projects is coordinating changes, especially when it comes to the database schema. In a shared database, a schema change for one part of the application might require updates and testing across many components, slowing down development cycles as it has direct impact on the whole components.

By giving each service its own database, development teams gain more autonomy. They can evolve their service's data model, optimize queries, or even change database technologies without impacting other services. This autonomy can significantly speed up development and deployment cycles, allowing teams to work more independently and deliver value more quickly.

Figure 11. Access data through API only

This diagram illustrates a fundamental tenet of the Database per Service pattern: data encapsulation through well-defined APIs. Here, we see that any modifications to the Messaging Database's model have a direct impact solely on the Messaging Service. The changes remain invisible to other components within the broader system ecosystem. This encapsulation is achieved by adhering to a strict rule: all data access must flow through the service's API.

The power of this approach becomes evident when we consider the implications for schema evolution. Let's say the team responsible for the Messaging Service decides to denormalize their data model to improve query performance, perhaps by embedding frequently accessed user information directly within message records. In a shared database scenario, such a change could wreak havoc, breaking queries and data access patterns across multiple services. However, with the Database per Service pattern, the change is contained. As long as the Messaging Service continues to honor its API contract—returning the expected data structures and fields—other services remain blissfully unaware of the underlying data reshuffling.

This principle dovetails neatly with the Open-Closed Principle from S.O.L.I.D., which states that software entities should be open for extension but closed for modification. In our context, the Messaging Service's API represents the "closed" part—a stable interface that other services can rely upon. When new requirements emerge or optimizations are needed, the service can "extend" its capabilities by evolving its internal data model or adding new API endpoints, without disrupting existing consumers.

For instance, if there's a need to support richer metadata for messages, the Messaging Service team might add new fields to their database schema and expose this additional information through new API methods or optional parameters on existing methods. Services that don't need this metadata continue to function unperturbed, while those that do can opt-in to the enhanced functionality.

The benefits of this approach extend beyond just ease of change. By forcing all data access through a versioned API, we introduce a layer of abstraction that can smooth over differences between the service's internal data representation and the view of that data presented to the outside world. This can be particularly valuable during incremental migrations or when supporting legacy clients.

Consider a scenario where the Messaging Service is transitioning from storing timestamps as Unix epochs to ISO 8601 strings. Rather than forcing all consumers to adapt simultaneously—a herculean task in large systems—the service's API can continue to accept and return epochs for v1 of the API, while offering the new format in v2. Internally, the service handles the conversion, allowing for a phased migration that doesn't leave any part of the system behind.

Flexible Technology Choices

This benefit is a direct outgrowth of two fundamental patterns: separate data ownership and encapsulation. When data is tucked behind a well-defined API contract, the specifics of the underlying storage mechanism become an implementation detail—opaque and, to a large extent, irrelevant to consumers of that API. Does it matter to the rest of the system whether the Messaging Service persists its data in MySQL, PostgreSQL, or ventures into NoSQL territory with MongoDB? As long as the service honors its API commitments, the answer is a resounding no.

This abstraction opens up a world of possibilities, freeing teams from the "one size fits all" straitjacket often imposed by a shared, monolithic database. In reality, not all data is created equal, and the notion that a single database technology can optimally serve every type of data or access pattern across a complex system is increasingly recognized as a constraining myth.

The Database per Service pattern shatters this constraint, ushering in an era of polyglot persistence—where each service can select a database technology that aligns precisely with its unique data characteristics and operational demands. This isn't just about having choices for the sake of variety; it's about empowering teams to make nuanced, strategic decisions that can significantly impact performance, scalability, and even the fundamental way they model their domain.

The implications of this technological flexibility ripple beyond just query performance or data model fit. Different database technologies bring different operational characteristics—scaling models, backup and recovery procedures, monitoring and management tools. By aligning these with service-specific needs, teams can optimize not just for runtime performance but for the entire lifecycle of managing data in production.

Easier Maintenance and Updates

This capacity for streamlined maintenance and evolution is yet another dividend paid by the twin principles of separate data ownership and encapsulation within the Database per Service pattern. In a landscape where data stores are siloed and accessible only through well-defined service interfaces, routine database upkeep and even transformative changes can be orchestrated with surgical precision, minimizing system-wide disruption.

Consider the cyclic necessities of database maintenance—software updates to patch security vulnerabilities, backup procedures to safeguard against data loss, or performance optimizations to keep pace with growing loads. In a monolithic architecture with a shared database, each of these tasks casts a long shadow. A version upgrade might require extensive regression testing across all application components. A backup process could impose a system-wide performance penalty. An ill-conceived index might accelerate queries for one module while grinding another's write operations to a halt.

The Database per Service pattern redraws these boundaries of impact. When each service lays claim to its own database, maintenance windows can be staggered, tailored to the specific needs and tolerances of individual services. The Customer Support team might schedule their database patching for the wee hours of Sunday morning when ticket volume is lowest, while the E-commerce platform waits until Tuesday afternoon post-sales rush. Backups for a write-heavy Logging Service could run hourly, while a relatively static Product Catalog Service might only need daily snapshots.

This granularity extends to performance tuning. DBAs can optimize with abandon, knowing that an experimental indexing strategy or a shift in isolation levels will reverberate only within the confines of a single service. The blast radius of any potential misstep is contained, fostering an environment where teams can be bolder in pursuing optimizations that might be deemed too risky in a shared-database context.

But the true power of this pattern shines through when we zoom out from day-to-day maintenance to consider more seismic shifts in data architecture. In the life of any sufficiently long-lived system, there comes a time when incremental tweaks no longer suffice—when data models need comprehensive restructuring, or when the limitations of the current database technology become insurmountable barriers to growth or feature development.

Traditionally, such changes have been the stuff of CTO nightmares: "big bang" migrations where months of preparation culminate in a high-stakes cutover weekend, with the entire engineering team on high alert and business stakeholders watching the status dashboard with bated breath. The risks are manifold—data corruption, performance regressions, or the dreaded Monday morning realization that a critical legacy feature was overlooked in testing. Rollback plans for such migrations often boil down to "pray we don't need it," because reversing course once the switch is flipped can be nearly impossible.

Conclusion

As we draw our exploration of the Database per Service pattern to a close, it's clear that this architectural approach offers a compelling vision for data management in the age of microservices. By championing separate data ownership, encapsulation, and the autonomy of individual services, the pattern unlocks a cascade of benefits that ripple through every facet of system development and operation.

We've seen how this decoupling can accelerate development cycles, freeing teams to evolve their data models and optimize performance without fear of collateral damage. We've explored the flexibility it affords in choosing the right database for the job, eschewing one-size-fits-all solutions in favor of technologies tailored to each service's unique needs. And we've witnessed how it transforms maintenance and migration from high-wire acts to managed, incremental processes, reducing risk and increasing the system's capacity for change.

These advantages paint a picture of systems that are not just more scalable and resilient, but more adaptable—better equipped to keep pace with the relentless evolution of business requirements, user expectations, and technological capabilities. In a digital landscape where agility is currency, the ability to pivot quickly, to experiment boldly, and to embrace change without fear can spell the difference between market leaders and those left behind.

Yet, as with any architectural choice, the Database per Service pattern is not without its complexities and trade-offs. The very decoupling that grants such flexibility also introduces new challenges around data consistency, distributed transactions, and holistic data views. The autonomy that empowers individual teams can, if not carefully managed, lead to a fragmented technology landscape that strains operational resources. And while service interfaces provide a powerful abstraction, designing them to be both flexible and stable requires foresight and discipline.

These challenges are not insurmountable, but they do demand thoughtful strategies and sometimes novel solutions. In our next article, we'll pull back the curtain on the potential drawbacks of the Database per Service pattern. We'll confront head-on the questions of how to maintain data integrity across service boundaries, how to handle workflows that span multiple data stores, and how to balance service independence with the need for system-wide governance.

We'll also examine the organizational and operational implications of this pattern. What does it take to successfully manage a polyglot persistence environment? How do we ensure that our decentralized data doesn't become siloed data, invisible to the analytics and insights that drive business decisions? And how might this pattern influence—or be influenced by—emerging trends like serverless architectures, edge computing, or AI-driven operations?

By exploring these drawbacks and challenges, our aim is not to diminish the power of the Database per Service pattern, but to equip us with a complete picture—to ensure that our journey toward decentralized data ownership is undertaken with eyes wide open. After all, true architectural wisdom lies not in dogmatic adherence to any pattern, but in understanding deeply both its strengths and its limitations.

So consider this not an ending, but an intermission. We've celebrated the elegance and potential of a world where each service is master of its own data domain. When next we meet, we'll roll up our sleeves and dig into the gritty realities of bringing that vision to life—the obstacles to be navigated, the pitfalls to be sidestepped, and the trade-offs to be weighed.

The path to robust, evolvable systems is seldom straightforward. But armed with patterns like Database per Service—and a clear-eyed view of both their power and their demands—we can chart a course through the complexity. So stay tuned, as we prepare to balance today's optimism with pragmatism, and turn our gaze from the heights of possibility to the solid ground of practice.

Configuring SSH to Access Remote Server

Eugene Zimin — Sun, 07 Apr 2024 08:55:10 +0000

Few Words About SSH

Configuring SSH on Local Machine

Linux and macOS

Most Linux distributions and macOS come pre-installed with OpenSSH, a free open source SSH client and server utility. To check if it's installed, we can open a terminal and run:

ssh -V

# Response should be something like the line below
# OpenSSH_9.6p1, LibreSSL 3.3.6

Windows

Windows 11 has pre-installed SSH client. To check it we need to open cmd application (a.k.a Command Prompt) and run the same command:

ssh -V

If we will see a similar output we saw for Linux and macOS (some names may vary), it means we have installed SSH client and we are good to go.

A little bit more complicated situation happens if Windows doesn't have preinstalled client. In that case we may use one of the following 3rd party applications to run it:

PuTTY for Windows
Git Bash for Windows

Generating Key Pairs

We can run ssh-keygen in a terminal/command prompt and follow the prompts. Some common options include:

-t to specify the key type (e.g. rsa, ecdsa, ed25519)
-b to set the key length in bits (e.g. 2048, 4096)
-C to add a comment to identify the key

For example:

ssh-keygen -t ecdsa -b 521

will give us the following output:

Generating public/private ecdsa key pair.
Enter file in which to save the key (~/.ssh/id_ecdsa): ~/.ssh/id_ecdsa_gce
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ~/.ssh/id_ecdsa_gce
Your public key has been saved in ~/.ssh/id_ecdsa_gce.pub
The key fingerprint is:
SHA256:Utwi6b8NzdJSFT5RlH6HOvr7y5HvNJiw08YB7wxAaQU user@laptop.local

There are various key types we may use. Here are the most popular ones we may want to generate:

RSA - One of the oldest and most widely used key types. Recommended minimum key length is 2048 bits, with 4096 bits being more secure.
ECDSA (Elliptic Curve Digital Signature Algorithm) - This key type is based on elliptic curve cryptography which provides equal security strength with smaller key sizes compared to RSA.
Ed25519 - A modern EdDSA (Edwards-curve Digital Signature Algorithm) key type that provides better performance and security than RSA/ECDSA. Ed25519 uses elliptic curve cryptography with a 256-bit key length.

While RSA is the most widely compatible, Ed25519 keys are considered more secure and efficient for modern systems. We can check which key types our SSH server and client support using ssh -Q key.

During key generation, we have the option to protect the private key with a passphrase. While providing a passphrase enhances security, we need to enter it every time the key is used.

NOTE FOR GCP
GCP requires to add at the end of the key userName and expireOn in form of JSON. We should do that manually to get the ready for pasting. Eventually we should have something like that:
ecdsa-sha2-nistp521 AAAAE2...== user@laptop.local {"userName":"<user@yourgcp.email>","expireOn":"2024-05-06T08:43:20+0000"}

At the same time private key (without .pub extension) remains securely stored on our local machine. We must keep the private key safe and never share it.

With our key pair ready, we can now proceed to configure server to use key pair only and disallow further access using username and password.

Configuring Server Side

That being said the advantages of disabling Password authentication for the remote server are as follows:

Enhanced Security: SSH keys utilize strong encryption algorithms and prevent brute-force and dictionary attacks that are common threats to password-based authentication. Keys are virtually impossible to guess or decrypt.
Eliminate Weak Passwords: Enforcing key-based authentication eliminates the risk of users setting weak or easily guessable passwords, which is a common vulnerability.
Audit Trail: SSH key pairs provide better audit trails and monitoring capabilities, as each key can be associated with a specific user or system.
Compliance: Many security standards and best practices, such as PCI-DSS, HIPAA, and NIST, recommend or mandate the use of key-based authentication over passwords for remote access to servers.

ssh -i ~/.ssh/id_ed25519 remote-user@server-ip-address

On the remote server open the SSH daemon configuration file as a superuser:

sudo nano /etc/ssh/sshd_config # may be changed to vim

In the file editor find the line #PasswordAuthentication yes and change it to PasswordAuthentication no and save the file and exit the editor.

Finally we need to restart the SSH daemon to make our changes take effect:

sudo systemctl restart sshd # (or command for your OS)

After making this change, the SSH server will only allow key-based authentication, and password logins will be disabled.

It's important to note that we should have at least one authorized SSH key added to the VM instance before disabling password authentication. Otherwise, we may get locked out of the system.

Simplifying Connections with SSH Client Configuration File

While we can directly specify options when invoking the ssh command, using a configuration file allows us to set persistent options, create aliases, and simplify the process of connecting to remote hosts. This is particularly useful when managing multiple SSH connections to different servers or cloud instances.

On Unix-based systems (Linux and macOS), the SSH client refers to the configuration file located at ~/.ssh/config location. On Windows systems using the OpenSSH client, the file is located at %UserProfile%.ssh\config.

The ~/.ssh/config file supports a wide range of configuration options and settings that can be applied globally or to specific hosts. Here are some common use cases:

Creating Aliases

We can define aliases or shorthand names for hosts, making it easier to connect without having to type out the full hostname or IP address every time.

Host remote_server
    HostName remote_host.com
    User ubuntu

Instead of remote_host.com it is possible to use IP address of the remote server directly. Now we can use:

ssh remote_server

to reach remote server instead of using its name remote_host.com or direct IP address. It's clear that aliases might be different and their usage makes it easier to set up connections.

Setting Default Options

We can set default options that will be applied to all SSH connections, such as the default user, identity file (private key), and forwarding settings.

Host *
    User ubuntu
    IdentityFile ~/.ssh/id_rsa
    ForwardAgent yes

In this configuration instead of doing aliasing, we set default configuration which will be applied to all connections and hosts. 1. The asterisk (*) is a wildcard that matches any hostname.

User ubuntu sets the default user account to be used for SSH connections. In this case, it will attempt to log in as the ubuntu user on remote hosts.

IdentityFile ~/.ssh/id_rsa specifies the path to the default SSH private key file that should be used for authentication. Here, it is set to ~/.ssh/id_rsa, which is the standard location for an SSH private key on Unix-based systems. That's convenient way to use previously generated key pair.

ForwardAgent yes option enables SSH agent forwarding. When enabled, it allows authentication keys loaded into the local SSH agent to be automatically forwarded to the remote host, enabling single sign-on (SSO) functionality. This can be useful when you need to connect to additional hosts from the initial remote host without re-authenticating.

Host-Specific Settings

We can define settings specific to a particular host or group of hosts, such as using a different private key, enabling compression, or specifying a non-standard port.

Host *.example.com
    IdentityFile ~/.ssh/id_rsa
    Compression yes

Host gcp-instance
    HostName 34.123.45.67
    User user2
    Port 2222

In this configurations we used 2 different values for Host key. In the first configuration option host entry applies settings to any host whose name matches the pattern *.example.com. The * acts as a wildcard, so this would include hosts like server1.example.com, web.example.com, or any other subdomain under the example.com domain.

IdentityFile ~/.ssh/id_rsa specifies that the private SSH key located at ~/.ssh/id_rsa should be used for authentication when connecting to hosts matching the *.example.com pattern.

Second configuration behaves differently and creates an alias for the only single IP address of 34.123.45.67 in form of the gcp-instance.

Proxying Through Bastion Hosts

If we need to connect to instances that are not directly accessible (e.g., behind a bastion host or a NAT gateway), we can configure SSH tunneling through a jump host.

Host gcp-instance
    HostName 10.0.0.5
    User user2
    ProxyJump bastion.example.com

By leveraging the ~/.ssh/config file, we can streamline our SSH workflow, ensure consistent settings across connections, and simplify the management of multiple remote hosts, including our Google Cloud VM instances.

To get started, we can create or edit the file using a text editor, and refer to the ssh_config man pages (man ssh_config) for a comprehensive list of available options and their descriptions.

Continuance

Functional, Action, Sequence: 3 Diagram Types for Multi-Dimensional Software Modeling

Eugene Zimin — Mon, 04 Mar 2024 09:28:23 +0000

When developing a new software system, diagrams are invaluable tools for documenting the design and architecture. Three diagram types that are especially useful for clear communication and mapping out software structure are functional diagrams, sequential diagrams, and action diagrams. In this article, I will provide an overview of each of these diagram types, explain what kind of information they are best for conveying, and provide examples to demonstrate how to put together functional, sequential, and action diagrams for a software project. Whether we're a software architect creating new design docs or a developer trying to understand existing systems, having a grasp of these three diagramming techniques can improve understanding and communication around complex software systems. Read on to learn how functional, sequential, and action diagrams can help us visualize and document software designs.

Different Angles - One System

Functional, action, and sequence diagrams serve distinct yet complementary purposes in software design. Each diagram type looks at the system from a different angle, providing unique perspectives that together contribute to a comprehensive understanding.

Functional diagrams offer a static, structural perspective by breaking down the software into functional components and displaying their logical organization and relationships. This static view focuses on what elements make up the system and how they architecturally connect.

In contrast, action and sequence diagrams take a dynamic, behavioral view of the runtime aspects of the system. Action diagrams display the procedural flow through key processes while sequence diagrams reveal the detailed object interactions during execution. These dynamics views center on how the system operates.

Using both static and dynamic views in combination enables tracing from architecture to implementation. The functional diagrams provide the scaffolding to be fleshed out by behavioral logic in action and sequence diagrams. The multiple angles combine to tell a complete story.

In other words, functional and action diagrams give high-level overviews of the system, simplifying complexity with abstraction. Meanwhile, sequence diagrams offer a detailed, tactical focus ideal for drilling into specifics. These complementary zoomed out and zoomed in perspectives are invaluable.

By leveraging all three standard diagram types, software systems can be thoroughly visualized from multiple standpoints, enabling superior comprehension, communication and documentation.

Understanding Action Diagrams

Action diagrams or flowcharts are a simple yet powerful way to visualize the flow of actions in a software system. They depict how a set of actions are sequenced together to accomplish a specific task or workflow from different points of view - from user prospective experience, or describe from the architecture point of view. Speaking much generic, they are graphical representations of the steps and decision points in a process.

Unlike sequence diagrams, which focus on object interactions, action diagrams illustrate the flow of procedural logic through a process. That is very important thing, explaining the difference between two of them. In action diagrams we usually do not review such level of technical details like application objects interactions, because even though it gives us a very comprehensive view, it makes it much harder to abstract the general logic of the software and lose the line.

The steps are arranged sequentially from top to bottom, guiding the reader through the flow of the actions. Action diagrams typically contain the following elements:

Actions - These are represented in boxes or circles and use active verbs or short phrases to describe steps. For example, "Validate Form", "Call Payment API", "Send Notification Email".
Connections - Arrows connect the actions to indicate flow. Conditional logic can be shown using branches or decision nodes.
Swimlanes - Horizontal swimlanes can group related actions for different components or actors.
Annotations - Additional text can explain inputs, outputs, etc.

Some key benefits of using action diagrams include:

Clarifying complex logic flows at a high level

Action diagrams allow us to understand a complex workflow or business process at a high level before diving into the implementation details. The sequential flow and branching logic help clarify what steps occur and in what order.

Illustrating different branches in conditional logic

Complex conditional logic is common in real-world software workflows. Action diagrams enable us to visualize different branches that occur based on conditions or decisions in the flow. The diagram makes it easy to see how the happy path and edge cases are handled.

Visualizing steps for workflows, user stories, or test cases

Action diagrams are useful for documenting workflows tied to user stories or requirements. They provide a visualization of the steps to be coded (see the reference to the high level logic description) . Action diagrams can also aid in test case planning since they illustrate the key flows to be tested.

Serve as a base for lower-level sequence diagrams

The high-level action diagram can be a starting point for drilling down into more detailed sequence diagrams. The sequence diagrams can focus on expanding specific actions from the action diagram into the object interactions needed to implement them. Using the action diagram as a guide, multiple complementary sequence diagrams can be created to cover all the steps

Improving documentation through diagrams

Diagrams are much more effective at conveying complexity than pure text. Action diagrams ensure the documentation captures the logic visually, making it easier for new team members to understand.

Let's look at an example action diagram for a user login/signup flow:

This diagram shows the sequential steps the system will walk through to complete the use case of a user signing up for an account. Key flows like handling invalid input are called out.

Action diagrams help summarize complex workflows at a high level while still illustrating logical flow, branches, conditions and edge cases. They provide an intuitive way to understand how a software system accomplishes its goals. Developers can rely on the action diagram as a base or skeleton, and fill in the implementation details in granular sequence diagrams for each step. This helps ensure the lower-level diagrams trace back and support the overall workflow represented in the action diagram.

Stepping in With Flowcharts

Here are few some simple steps to follow with a good action diagram.

Identify the Process

Start by clearly defining the process or workflow which is needed to be diagramed. We need to keep in memory, that flowcharts are called so because the describe flow in system design, where every flow has begin and end points. That is why it is very important to identify them and describe them first.
For example, if we are diagramming the process for a customer trying to login to the web portal, the start point could be the customer loading the initial page or trying to reach out any page in the web portal and get redirected into the login page. As the end point there could be considered a showing to the user his profile page or some other default page.

Determine the Steps

Break down the process into individual steps. Each step should represent a single action or decision point. For instance, in the user login process, steps might include "Display form", "Fill in the form", etc.

Use Standard Symbols

Always try to utilize standard flowchart symbols to represent different elements of the process. For example, use rectangles for actions, diamonds for decision points, and arrows for the flow direction. This standardization helps in making the diagram universally understandable. It is possible to use UML as a design standard.

Arrange the Steps Sequentially

Place the steps in the order they occur, from top to bottom or left to right. Ensure that the flow is logical and easy to follow. We should always ask ourselves - if we perform this action, what should happen next?

Label the Steps

Clearly label each step with a brief description of the action or decision. Use concise and clear language. For example, instead of just "Login," label the step as "Enter Login Information."

Indicate Decision Points

Use decision symbols to represent points in the process where a choice must be made. Label each outgoing arrow from a decision point to indicate the criteria for each path. For example, at a decision point like "User exists?" the outgoing arrows could be labeled "Yes, proceed with login" and "No, register user first".

Review for Completeness and Clarity

Once the diagram is complete, review it to ensure that it accurately represents the process and is easy to understand. Make any necessary adjustments. This might involve rearranging steps, adding missing steps, or clarifying labels.

Diving Into Functional (Block) Diagrams

Functional diagrams, also known as block diagrams, are a valuable type of diagram used in software design. They illustrate the high-level functions and interactions in a system by breaking down the overall system into its main functional components and showing how they connect and share data.

It might be easier to understand them comparing their place in system design and architecture with other types of diagrams.

Unlike sequence diagrams or action diagrams (flowcharts) which focus on the step-by-step logic, functional diagrams aim to provide a bird's eye view of the system architecture. The functional elements that make up the overall system are represented as blocks or bubbles on the diagram.

Sequence diagrams display detailed interactions between objects and the order of messages passed between them. Flowcharts illustrate the tactical progression of steps through a process.

In contrast, functional diagrams zooms out to look at the system from a wider lens. Functional diagrams are focused on the structural decomposition of the system into major building blocks. So instead of drilling down into the tactical flow, they partition the system into high-level functional components.

These functional components are depicted as abstract blocks, not necessarily tied to specific classes or objects. The blocks represent logical groupings by broad areas of functionality. For example, there may be a block for the UI functions, the business logic functions, the persistence functions, and so on.

The goal is to break down the overall system into major functional pieces, show how they connect together, and provide a conceptual overview. Functional diagrams help crystallize the architectural vision and big picture before diving into implementation details with other diagram types.

Why to Use Functional Diagrams?

Functional diagrams provide valuable high-level visualization of a system's architecture and design. Some specific benefits include:

Logical overview of system architecture

Functional diagrams allow us to logically group major system capabilities into modular blocks that abstractly represent the overall structure. This bird's eye view establishes the architectural vision.

Illustrating relationships between elements

By connecting functional blocks with annotated arrows, functional diagrams clearly display the relationships and dependencies between components. This reveals how the pieces interact at a high level.

Capturing architectural details

Functional diagrams permit capturing a degree of technical details related to the architecture. For example, labeled arrows can represent messaging protocols used between components or specific APIs exposed by a block.

Data flows

The connections between blocks can indicate data flows between functional elements, highlighting where information is produced, consumed and transformed.

Complementing UML

While UML provides implementation specifics, functional diagrams effectively communicate high-level system design intent and architecture. The diagrams are complementary.

Communication

Functional diagrams create a common visual language for technical and non-technical stakeholders to understand system organization and interactions.

Foundation for other models

Functional diagrams provide the 20,000 foot view of the landscape to ground more detailed models like UML, flowcharts, etc.

By leveraging functional diagrams, architects and designers can effectively model and communicate key architectural aspects and relationships to set the stage for detailed implementation.

Preparing Functional (Block) Diagrams

When determining what functional blocks to include on a functional diagram, it is important to use nouns to label the blocks, not verbs. For example, correct block names would be "Inventory System", "Payment Gateway", "User Profile", not "Validate User", "Process Order", "Verify Payment".

Functional diagrams contain the following key elements:

Blocks - Represent major functions or components of the system. Blocks can be systems, subsystems, classes, services, etc.
Connections - Arrows between blocks show how they interact and communicate with each other.
Annotations - Additional text can label connections to explain the nature of interactions.

Example of the functional diagram is below.

It is easy to begin with diagraming while architecting a system or application, following simple rules below:

Start with High-Level Functions

Begin by identifying the major functionality of the system. These should represent the core capabilities or services provided by the system. For example, in an e-commerce platform, high-level functionality might include Product Catalog, Shopping Cart, Payment Processing, and Order Fulfillment. This step sets the foundation for the functional diagram by establishing the key areas of functionality within the system.

Break Down Complex Functions

If the chosen functionality is too complex, consider breaking it down into smaller, more manageable sub-functionalities. This helps in maintaining clarity and focus in the diagram. For instance, the Payment Processing function could be broken down into sub-functions like Credit Card Validation, Payment Authorization, and Payment Confirmation. This granularity ensures that each block in the diagram is focused and understandable.

Use Consistent Labeling

Ensure that all blocks are labeled consistently, using nouns that clearly describe the functionality or component. It is recommended even though not mandatory to avoid using technical jargon that might be confusing to non-technical stakeholders. For example, use clear and descriptive labels like "User Authentication" instead of vague or technical terms like "Auth Module". In all cases consistent labeling helps in making the diagram more accessible and easier to understand for all stakeholders.

Indicate Data Flow

Use arrows to show the flow of data or control between different functional components. This helps in understanding how information is processed and transferred within the system. For example, an arrow from the Shopping Cart function to the Payment Processing function indicates that the shopping cart details are passed to the payment process for transaction completion. Clearly indicating data flow provides insight into the interactions between different functions and how they work together to achieve the system's objectives.

Validate with Stakeholders

Regularly review the diagram with stakeholders to ensure that it accurately represents their understanding of the system. This is crucial for aligning the technical architecture with business requirements. Engaging stakeholders in the validation process ensures that their perspectives and needs are reflected in the diagram, leading to a more accurate and relevant representation of the system.

Modelling Detailed Logic with Sequence Diagrams

Sequence diagrams are a third important type of diagram used when designing software systems. While functional and action diagrams provide high-level overviews, sequence diagrams drill down to illustrate detailed logic and object interactions.

Sequence diagrams work by representing software objects as vertical boxes or "lifelines" over a timeline. Messages passed between objects are shown sequentially down the diagram to visualize the procedural flow.

Sequence diagrams are an essential tool for visualizing detailed software logic and object interactions. The granular focus of sequence diagrams allows us to zoom in on the flow for a distinct use case rather than trying to capture all flows in one diagram. This makes them ideal for detailing individual stories or features.

The core purpose of sequence diagrams is showing how objects collaborate and exchange data during a scenario, which matches nicely to modeling story requirements. The chronological visualization provided by sequence diagrams also aligns well with displaying the progression of steps through a workflow or user path.

Sequence diagrams work best for reasonably complex scenarios with around 5-15 steps. This level of abstraction is in the ideal sweet spot for capturing coherent stories or workflows, neither too high-level nor too detailed.

The objects and messages also correspond well to classes and methods needed to implement the scenario, providing traceability from requirements to code. Sequence diagrams allow clear visual communication of detailed logic to explain stories to stakeholders and developers.

The focused scope, object interaction modeling, logical sequential flows, ideal abstraction level, traceability, and communication strengths make sequence diagrams a great tool specifically for breaking down and displaying software scenarios at the granularity of user stories. The techniques are very complementary.

Key elements in a sequence diagram include:

Objects - Shown as lifelines labeled with class names or components
Messages - Arrows between lifelines indicate messages communicating data
Time sequence - Messages are ordered sequentially down diagram
Returns - Dotted arrows back to caller lifeline for return messages
Controls and conditions - Text notations for loops, conditionals, etc

Some benefits of using sequence diagrams:

Reveal detailed logic and execution flow through use case
Visualize complex object collaborations and data flows
Model software logic for documentation and testing
Complement higher-level action diagrams

Sequence diagrams are essential for understanding and communicating how objects interact to implement software behavior and use cases. They provide a dynamic view of detailed system execution.

In the next sections we'll walk through an example to see how to build a sequence diagram.

Sequence Diagram Example

Let's look at a simple example sequence diagram for a user login flow:

The diagram would start with two lifelines - one labeled "User" and one labeled "LoginController".

The first arrow goes from User to LoginController labeled submitLogin(username, password) indicating the user sends login credentials.

LoginController sends an arrow to the Database lifeline with the message query(username) to look up the user.

The Database returns user data back to LoginController.

LoginController sends a message to itself validate(user, password) to validate against the retrieved data.

If valid, LoginController sends loginSuccess(user) back to User.

If invalid, it sends loginFailed(error) back to User instead.

That covers the basic flow. Additional logic like forgotten password handling could also be added.

The key is using object lifelines and chronological messages to reveal the step-by-step interactions. This brings low-level logic to life visually.

Creating Sequence Diagrams

Here are some steps to follow when creating a sequence diagram:

Identify the scenario or use case to diagram. For example, this could be a user signup flow, payment processing, order fulfillment, etc. Focus on a specific workflow.
Determine the key objects involved in the scenario. The nouns in the use case often provide clues. For a user signup flow, the objects might be User, SignupController, AuthenticationService, etc. These objects become the vertical lifelines.
Map out the major steps sequentially from top to bottom. Focus on the overarching logical flow first before details. For example, the signup steps could be: Load registration form -> Submit credentials -> Validate inputs -> Create account -> Send welcome email.
Represent actions as messages passed between object lifelines. Use clear imperative phrases for message names like submitRegistration(user), validateCredentials(username, password).
Include returns back to the calling object for response messages. For example, after calling validateCredentials() the AuthenticationService would return either credentialsValid() or credentialsInvalid().
Number or label messages if needed to indicate ordering for complex flows with conditionals or loops. This clarifies sequence.
Use swimlanes to optionally group related lifelines, such as putting all UI objects in one lane and service objects in another.
Refine the diagram iteratively, adding more steps or reordering as logic is clarified. Diagrams evolve incrementally.
Review the completed diagram to validate the end-to-end flow makes sense. Update as needed.

Following these tips will allow us to leverage sequence diagrams to bring complex software executions to life on the page. The step-by-step visualization enables clearer communication and shared understanding.

Bringing the Diagrams Together

Functional, action, and sequence diagrams all play important roles in documenting software systems, but provide unique perspectives. Understanding how the diagrams complement each other is key.

Static vs Dynamic Views

Functional diagrams provide a static structural view by breaking the system into functional components. The diagrams display logical organization and relationships between elements.

Action and sequence diagrams offer dynamic behavioral views of the system. Action diagrams illustrate procedural flow through a process. Sequence diagrams reveal detailed object interactions.

High-level vs Detailed Focus

Functional and action diagrams give a high-level overview of the system architecture and workflows. They simplify complex systems with an abstracted view.

Sequence diagrams take a detailed focus to unfold specific use cases and object collaborations during execution. The diagrams get tactical.

Architectural vs Implementation Views

Functional diagrams align closely to architectural modeling to design the overarching system structure. They can inform high-level technical decisions.

Action and sequence diagrams tend towards implementation modeling to plan detailed logic, steps, and object interactions during coding.

Traceability

When created together, the diagrams form crucial traceability between architectural design and implementation specifics. The perspectives combine to tell a complete story.

By leveraging these diagram types together, software systems can be thoroughly visualized to improve understanding and communication.