Forem: Abdulazeez Abdulazeez

Introducing Mongo Broadcaster: A Multi-Channel MongoDB Change Stream Processor

Abdulazeez Abdulazeez — Tue, 22 Apr 2025 08:06:47 +0000

About a month ago, I wrote a blog post on streaming changes on certain collections in Mercury to build a real-time update system.

Over the weekend and due to boredom, I built a package that'll allow you and me to stream changes from a collection and broadcast them via a defined channel such as Websocket, Redis, HTTP streaming, or saved in a collection.

For now, I have decided not to implement the delete operation, even if it was a few LOCs. The package focuses solely on insert and update operations.

In this blog post, I'll discuss the implementation of the package. It's called broadcaster and can be downloaded from PyPi.

TL;DR

Install the library from PyPi:

pip install mongo-broadcaster

Abstract classes

In Mercury, the current implementation broadcasts only to a WebSocket client. However, my technical partner & I were discussing building activity logs and whatnot, and only then did it occur that I can extend this to Redis or even store the streams in another collection.

As a result, I designed a BaseChannel class that allows other channels to inherit its abstract methods to allow them to design their own implementation:

from abc import ABC, abstractmethod
from typing import Any, Dict


class BaseChannel(ABC):
    @abstractmethod
    async def connect(self):
        """Initialize connection"""
        pass

    @abstractmethod
    async def disconnect(self):
        """Close connection"""
        pass

    @abstractmethod
    async def send(self, recipient: str, message: Dict[str, Any]):
        """Send message to recipient"""
        pass

The BaseChannel is the progenitor of the Websocket, Redis, HTTP, and Database channel,s which will be discussed later on.

The BaseChannel class has three methods:

connect(): This initializes the connection to the channel. e.g., connecting redis server instance.
disconnect(): This closes the connection to the channel.
send(recipient, message): This handles the sending of messages to the registered recipient.

Let's take a look at the models defined for this package.

Models

There are four Pydantic models defined; three for configuration and one for standardizing the response from the MongoDB stream. Let's take a look at the solo standardization model first.

ChangeEvent

The ChangeEvent model class returns the log from the MongoDB stream into a presentable format for use in other functions/methods. It is defined as:

class ChangeEvent(BaseModel):
    """Standardized change event model"""
    operation: str  # 'insert'|'update'
    collection: str
    document_id: str
    data: dict
    timestamp: float
    namespace: Optional[str] = None
    recipient: Optional[str] = Field(
        None,
        description="Target recipient ID if available (for directed messaging)"
    )

For example, an update operation returns the following JSON:

{
    '_id': {
        '_data': '82680684FF000000012B042C0100296E5A1004D3EE9A783E1646B481F4358CF62D4A8D463C6F7065726174696F6E54797065003C7570646174650046646F63756D656E744B657900463C5F6964003C757365725F31323300000004'
    },
    'operationType': 'update',
    'clusterTime': Timestamp(1745257727,
    1),
    'wallTime': datetime.datetime(2025,
    4,
    21,
    17,
    48,
    47,
    405000),
    'fullDocument': {
        '_id': 'user_123',
        'name': 'Updated to test the newly added field to watch.'
    },
    'ns': {
        'db': 'test',
        'coll': 'users'
    },
    'documentKey': {
        '_id': 'user_123'
    },
    'updateDescription': {
        'updatedFields': {
            'name': 'Updated to test the newly added field to watch.'
        },
        'removedFields': [],
        'truncatedArrays': []
    },
    'fullDocumentBeforeChange': None
}

Now, that doesn't look all pretty. It can be formatted to:

operation = 'update'
collection = 'users'
document_id = 'user_123'
data = {}
timestamp = 1745257958.0
namespace = 'test'
recipient = user_123

The code above is from a function format_change_event defined in the utils.py file. We'll get there, but for now, let's take a look at the other models for configuration.

...Config

The three configuration models are:

ChangeStreamConfig
CollectionConfig
BroadcasterConfig

ChangeStreamConfig

This model defines a pipeline and options to be utilized by the .watch() method. That is, the configuration defined here will determine the behaviour of the change stream. The model comes predefined with a default pipeline and options field:

class ChangeStreamConfig(BaseModel):
    """Configuration for MongoDB change stream"""
    pipeline: List[Dict[str, Any]] = Field(
        default_factory=lambda: [
            {'$match': {'operationType': {'$in': ['insert', 'update']}}}
            # add support for delete later -> this requires some other changes
        ]
    )
    options: Dict[str, Any] = Field(
        default_factory=lambda: {
            'full_document': 'updateLookup',
            'full_document_before_change': 'whenAvailable'
        }
    )

The pipeline is your normal aggregation pipeline. Above, the default pipeline is to match the insert and update operations executed.

The options field comes with two options predefined:

full_document to return the full document on which the operation (insert|update) is performed on.
full_document_before_change to return the previous version of the document before a change when avaialble. e.g., the document before an update was carried out.

The document before change will not be available by default if the capture of pre-images on the collection is not enabled.

CollectionConfig

CollectionConfig is responsible for storing the details for a specific collection to be streamed. Let's take a look at the definition:

class CollectionConfig(BaseModel):
    """Configuration for a specific collection"""
    collection_name: str
    database_name: Optional[str] = None
    change_stream_config: ChangeStreamConfig = Field(default_factory=ChangeStreamConfig)
    fields_to_watch: List[str] = Field(default_factory=list)
    recipient_identifier: Optional[str] = None  # Field path to identify recipient (e.g., "owner.id")

The CollectionConfig model takes:

the collection name
the database name (optional)
the change stream configuration
a list of fields to watch for changes in the collection. If the list is empty, it watches for all fields in the collection
an identifier for the recipient of any change. If this is left out, every connected recipient or subscribe will be broadcasted to (e.g., a general service announcement).

BroadcasterConfig

This is the main configuration for the tool. Here, the database URI, a list of collections and optional default database is configured. It is defined simply as:

class BroadcasterConfig(BaseModel):
    """Main configuration for the broadcaster"""
    mongo_uri: str
    collections: List[CollectionConfig]
    default_database: Optional[str] = None

Now that the structure for each configuration model has been laid out, let's take a look at utility functions.

Utilities

These are essential functions that help with the smooth running for the package. The utility functions are self-explainable:

Validate mongo connection

This function validates that the MongoDB connection is valid and live.

def validate_mongo_connection(uri: str) -> bool:
    """Verify MongoDB connection is available"""
    try:
        from motor.motor_asyncio import AsyncIOMotorClient
        client = AsyncIOMotorClient(uri)
        client.admin.command('ping')
        return True
    except Exception as e:
        logger.error(f"MongoDB connection failed: {str(e)}")
        return False

Format change event

This function makes use of the ChangeEvent model to purify the chunk of data sent by the stream.

def format_change_event(change: Dict[str, Any], config: CollectionConfig) -> ChangeEvent:
    """Transform raw MongoDB change stream event"""
    return ChangeEvent(
        operation=change['operationType'],
        collection=change['ns']['coll'],
        document_id=str(change['documentKey']['_id']),
        data=extract_fields(change, getattr(config, 'fields_to_watch', [])),
        timestamp=change['clusterTime'].time,
        namespace=change['ns']['db'],
        recipient=None
    )

Extract fields

The function filters the stream of document to pick items from the change dictionary registered in the fields lsit.

def extract_fields(change: Dict[str, Any], fields: list) -> Dict[str, Any]:
    """Extract specific fields from change stream data"""
    result = {}
    for field in fields:
        keys = field.split('.')
        value = change
        try:
            for key in keys:
                value = value.get(key, {})
            if value:  # Only add non-empty values
                result[field] = value
        except AttributeError:
            continue
    return result

Backoff handler

For retries:

def backoff_handler(details):
    """Exponential backoff for connection retries"""
    logger.warning(
        f"Retrying in {details['wait']:.1f} seconds after "
        f"{details['tries']} tries calling {details['target']}"
    )

Channels

Broadcaster currently supports four channels:

Websocket
Redis
HTTP
Database logging

All four channels inherit the base channel and contain an additional broadcast method. Each of these channels has the methods:

connect(client_id): registers a new connection
disconnect(client_id): disbands a connection
send(recipient, message): sends a message to a designated client
broadcast(message): sends message to all connected clients

Websocket

For Websocket, here's the implementation:

class WebSocketChannel(BaseChannel):
    def __init__(self):
        self.active_connections: Dict[str, WebSocket] = {}

    async def connect(self, client_id: str, websocket: WebSocket):
        """Register a new websocket connection"""
        await websocket.accept()
        self.active_connections[client_id] = websocket

    async def disconnect(self, client_id: str):
        """Remove a websocket connection"""
        if client_id in self.active_connections:
            del self.active_connections[client_id]

    async def send(self, recipient: str, message: Dict[str, Any]):
        """Send message to specific client"""
        if recipient in self.active_connections:
            try:
                await self.active_connections[recipient].send_json(message)
            except Exception as e:
                await self.disconnect(recipient)
                raise e

    async def broadcast(self, message: Dict[str, Any]):
        """Send message to all connected clients"""
        for connection in list(self.active_connections.values()):
            try:
                await connection.send_json(message)
            except:
                # Remove dead connections
                client_id = next(
                    (k for k, v in self.active_connections.items()
                        if v == connection), None)
                if client_id:
                    await self.disconnect(client_id)

Redis


class RedisPubSubChannel(BaseChannel):
    def __init__(self, redis_uri: str, channel_prefix: str = "mongo_change:"):
        self.redis_uri = redis_uri
        self.channel_prefix = channel_prefix
        self.redis = None

    async def connect(self):
        """Initialize Redis connection"""
        self.redis = await aioredis.from_url(self.redis_uri)

    async def disconnect(self):
        """Close Redis connection"""
        if self.redis:
            await self.redis.close()

    async def send(self, recipient: str, message: Dict[str, Any]):
        """Publish message to recipient-specific channel"""
        channel = f"{self.channel_prefix}{recipient}"
        await self.redis.publish(channel, json.dumps(message))

    async def broadcast(self, message: Dict[str, Any]):
        """Publish message to broadcast channel"""
        channel = f"{self.channel_prefix}broadcast"
        await self.redis.publish(channel, json.dumps(message))

For redis, the channel name can be set and if not set, the mongo_change will be subscribed to for stream changes.

HTTP

This is specially designed for webhooks.

class HTTPCallbackChannel(BaseChannel):
    def __init__(self, endpoint: str, headers: Dict[str, str] = None, timeout: int = 5):
        self.endpoint = endpoint
        self.headers = headers or {}
        self.timeout = aiohttp.ClientTimeout(total=timeout)
        self.session = None

    async def connect(self):
        """Create aiohttp session"""
        self.session = aiohttp.ClientSession(headers=self.headers, timeout=self.timeout)

    async def disconnect(self):
        """Close aiohttp session"""
        if self.session:
            await self.session.close()

    async def send(self, recipient: str, message: Dict[str, Any]):
        """Send HTTP POST request"""
        payload = {
            "recipient": recipient,
            "event": message
        }
        async with self.session.post(self.endpoint, json=payload) as response:
            if response.status >= 400:
                raise ValueError(f"HTTP request failed with status {response.status}")

    async def broadcast(self, message: Dict[str, Any]):
        """Send broadcast HTTP POST request"""
        async with self.session.post(self.endpoint, json={"event": message}) as response:
            if response.status >= 400:
                raise ValueError(f"HTTP request failed with status {response.status}")

It is important to note that the current implementation is very basic and can be fine tuned for secure webhook use cases. For example:

class WebhookChannel(HTTPCallbackChannel):
    async def send(self, recipient: str, message: dict):
        """Add webhook signing or retry logic"""
        message["signature"] = generate_hmac(message)
        await super().send(recipient, message)

Database

This channel essentially records the change back to the database, either tied to a single recipient or as a broadcast under the configured parameters.

I think anyone can do this without necessarily using this library, but yay to anything that makes life easy

class DatabaseChannel(BaseChannel):
    def __init__(self, mongo_uri: str, database: str, collection: str):
        self.mongo_uri = mongo_uri
        self.database_name = database
        self.collection_name = collection
        self.client = None
        self.collection = None

    async def connect(self):
        """Initialize MongoDB connection"""
        self.client = AsyncIOMotorClient(self.mongo_uri)
        self.collection = self.client[self.database_name][self.collection_name]

    async def disconnect(self):
        """Close MongoDB connection"""
        if self.client:
            self.client.close()

    async def send(self, recipient: str, message: Dict[str, Any]):
        """Save message to database"""
        if not self.collection:
            raise RuntimeError("Database connection not established")

        document = {
            "recipient": recipient,
            "message": message,
            "timestamp": datetime.datetime.utcnow()
        }
        await self.collection.insert_one(document)

    async def broadcast(self, message: Dict[str, Any]):
        """Save broadcast message to database"""
        document = {
            "message": message,
            "timestamp": datetime.datetime.utcnow(),
            "broadcast": True
        }
        await self.collection.insert_one(document)

Exceptions

The ChannelNotConnectedError exception is raised when there's no valid channel configured for broadcast.

class ChannelNotConnectedError(Exception):
    """Raised when no output channels are configured"""
    pass

There's another that hasn't been implemented yet - that's to check for invalid configuration. Pydantic type checking can handle that.

Broadcasting

All these channels need an outlet for subscribers. The MongoChangeBroadcaster class comes to our rescue:

class MongoChangeBroadcaster:
    def __init__(self, config: BroadcasterConfig):
        self.config = config
        self.mongo_client: Optional[AsyncIOMotorClient] = None
        self.channels: List[BaseChannel] = []
        self._running = False
        self._tasks: List[asyncio.Task] = []

The class has nine methods.

Initialize connection

This method validates and initalizes the MongoDB connection. Validation is done with the aid of the validate_mongo_connection utility function.

async def _initialize_connection(self):
    """Validate and establish MongoDB connection"""
    if not validate_mongo_connection(self.config.mongo_uri):
        raise ConnectionError("Invalid MongoDB connection URI")
    self.mongo_client = AsyncIOMotorClient(self.config.mongo_uri)

Add channel

A station owner needs to add a channel for subscribers to stream events. That's what this function does:

def add_channel(self, channel: BaseChannel):
    """Register an output channel"""
    self.channels.append(channel)

Start streaming

After initialization and channel addition, the stream is started. The method raises an exception if no channel is configured.

async def start(self):
    """Start all change stream watchers"""
    if not self.channels:
        raise ChannelNotConnectedError("No output channels configured")

    await self._initialize_connection()
    self._running = True

    for collection_config in self.config.collections:
        task = asyncio.create_task(
            self._watch_collection_with_retry(collection_config)
        )
        self._tasks.append(task)

Watch collection with retry

Hiccups can happen during transmission. Instead of stopping the entire stream, the broadcast station retries for a bit. The method that handles this is defined:

async def _watch_collection_with_retry(self, config: CollectionConfig):
    """Wrapper with exponential backoff for resiliency"""
    from tenacity import retry, stop_after_attempt, wait_exponential

    @retry(
        stop=stop_after_attempt(3),
        wait=wait_exponential(multiplier=1, min=4, max=10),
        before_sleep=backoff_handler
    )
    async def _watch():
        await self._watch_collection(config)

    await _watch()

Watch collection

Since the stream has started, we need to make sure our agents are on site to get us information. Thewatch_collection() method takes the data from the collection config and watches the changes:

async def _watch_collection(self, config: CollectionConfig):
    """Monitor a single collection for changes"""
    db_name = config.database_name or self.config.default_database
    if not db_name:
        raise ValueError("Database name must be specified")

    db = self.mongo_client[db_name]
    collection = db[config.collection_name]

    try:
        async with collection.watch(
            pipeline=config.change_stream_config.pipeline,
            **config.change_stream_config.options
        ) as change_stream:
            async for change in change_stream:
                if not self._running:
                    break
                await self._process_change(change, config)

    except asyncio.CancelledError:
        logger.info(f"Stopped watching {config.collection_name}")
    except Exception as e:
        logger.error(f"Error watching {config.collection_name}: {str(e)}")
        raise

Extract nested field

Some documents have nested fields, and the recipient identifier can be in a field such as order.customer.id. The helper method breaks these fields to get the recipient:

 def _extract_nested_field(self, doc: dict, field_path: str) -> Optional[Any]:
    """Safely extract nested fields like 'owner.id'"""
    value = doc
    for key in field_path.split('.'):
        value = value.get(key, {})
        if not value:
            return None
    return value

This may not work for really complex scenarios now that I think about it. To be safe, make use of simple identifiers like _id.

Process change

When the streams are logged into the broadcast station, they need to be processed to retrieve the important information. Here, the method also converts the change event into a neater ChangeEvent model instance.

 async def _process_change(self, change: dict, config: CollectionConfig):
    """Process and distribute a change event"""
    try:
        # Extract recipient if configured
        recipient = None
        if config.recipient_identifier:
            doc = change.get('fullDocument') or change.get('fullDocumentBeforeChange', {})
            recipient = self._extract_nested_field(doc, config.recipient_identifier)

        event = format_change_event(change, config)
        event.recipient = str(recipient) if recipient else None
        await self._send_to_channels(event)

    except Exception as e:
        logger.error(f"Error processing change: {str(e)}")

Send to channels

The method broadcasts to registered channels to send to recipients or broadcast to the whole network via this method:

 async def _send_to_channels(self, event: ChangeEvent):
    """Broadcast to all registered channels"""
    for channel in self.channels:
        try:
            if event.recipient:  # Targeted delivery
                await channel.send(event.recipient, event.dict())
            else:  # Broadcast
                await channel.broadcast(event.dict())
        except Exception as e:
            logger.error(f"Channel {type(channel).__name__} failed: {str(e)}")

Stop

Every beginning must have an end. The stop() method cancels all the asynchronous tasks deployed and closes the MongoDB client when invoked. This is great for shutdown() events.

 async def stop(self):
    """Gracefully shutdown all watchers"""
    self._running = False
    for task in self._tasks:
        task.cancel()
    await asyncio.gather(*self._tasks, return_exceptions=True)
    if self.mongo_client:
        self.mongo_client.close()

Example

Here's a WebSocket channel example:

from contextlib import asynccontextmanager

from fastapi import FastAPI, WebSocket
from starlette.websockets import WebSocketDisconnect

from broadcaster import MongoChangeBroadcaster, BroadcasterConfig, CollectionConfig, WebSocketChannel


@asynccontextmanager
async def lifespan(app: FastAPI):
    await broadcaster.start()

    yield
    await broadcaster.stop


app = FastAPI(lifespan=lifespan)
websocket_channel = WebSocketChannel()

config = BroadcasterConfig(
    mongo_uri="mongodb://localhost:27017",
    collections=[
        CollectionConfig(
            collection_name="users",
            recipient_identifier="fullDocument._id",  # Send to user who owns the document
            database_name="test",
        ),
    ]
)

broadcaster = MongoChangeBroadcaster(config)
broadcaster.add_channel(websocket_channel)


@app.websocket("/ws/{user_id}")
async def websocket_endpoint(websocket: WebSocket, user_id: str):
    await websocket_channel.connect(user_id, websocket)
    try:
        while True:
            await websocket.receive_text()  # Keep connection alive
    except WebSocketDisconnect:
        await websocket_channel.disconnect(user_id)


if __name__ == "__main__":
    import uvicorn

    uvicorn.run(app, host="0.0.0.0", port=8000)

To test:

Run: python websocket_example.py
Open a websocket connection with a user ID: websocat ws://0.0.0.0:8000/ws/ydev
Make a change into the collection. In the example, I have a users collection in my test database.
Make a change, for example: db.users.updateOne({_id: 'ydev'}, {"$set": {"name": "Updated to test the newly added field to watch changes"}})
Listen for changes in the WebSocket console

Conclusion

This is a lengthy blog post, so I must commend you if you read or skipped to this part. I'm going to try to write a full documentation for the library (amen).

It's open source on GitHub on https://github.com/Youngestdev/broadcaster.

10 Javascript Data Visualization Libraries Every Front-End Developer Should Know About

Abdulazeez Abdulazeez — Mon, 17 Apr 2023 10:38:02 +0000

Visuals are your best option for presenting data in the modern world. In reality, no one has the time to read through a large amount of information. You must be able to use data to tell your story in a way that is simple and quick for everyone to comprehend.

Data visualization, otherwise known as data viz., refers to the process of presenting data in a visual or graphical format. With data visualization tools, you can quickly present data and make it easier to digest.

As with every technical decision, choosing your tools is important based on their ease of integration, the number of features they provide, and their intuitiveness.

Throughout this article, we'll go over ten of the most commonly used Javascript data visualizations, some of which I've encountered the most frequently as a software engineer, as well as some I believe deserve more attention. We'll look at various libraries that offer data visualization techniques, such as the basic level chart (bar charts, pie charts, and so on), financial charts (candlesticks, trading indicators), map charts, scatter plot charts, and many more. We’ll also cover their capabilities and drawbacks, along with instructions and code samples on how to use them.

ApexChart

ApexChart is an open-source charting library that helps you create beautiful and interactive charts with limited code. ApexChart supports 16 chart types, including line, bar, area, pie chart, candlestick, heatmap, and many more.

In addition to providing beautiful charts, some of ApexChart's added advantages are that its charts are responsive and interactive—they are automatically rendered with customized icons that let you control the zoom of your charts, pan through datasets, and even let you export rendered charts as a PNG, SVG or CSV file. You can also create dynamic charts that allow you to load data on selections. They offer over 10+ different theme palettes for styling charts as well.

However, the Apexchart library is quite large, with its current version (v3.35.4) weighing in at around 496kb. As a result, when working with a large data set, you will begin to notice significant performance issues.

How to use

You can include ApexChart CDN in your project markup with:

<script src="https://cdn.jsdelivr.net/npm/apexcharts"></script>

Optionally, you can also install the library via npm with the command below:

$ npm install apexcharts

Once installed, you can get started with rendering charts by creating an option object specifying your chart style and dataset. Below is an example of an area chart with a random dataset:

<head>
    <script src="https://cdn.jsdelivr.net/npm/apexcharts"></script>
  </head>
  <body>
    <div id="chart"></div>

    <script>
      var options = {
        chart: {
          height: 280,
          type: "area",
        },
        dataLabels: {
          enabled: false,
        },
        series: [
          {
            name: "Series 1",
            data: [75, 82, 68, 75],
          },
        ],
        fill: {
          type: "gradient",
          gradient: {
            shadeIntensity: 1,
            opacityFrom: 0.7,
            opacityTo: 0.9,
            stops: [0, 90, 100],
          },
        },
        xaxis: {
          categories: ["01 Jan", "02 Jan", "03 Jan", "04 Jan"],
        },
      };
      var chart = new ApexCharts(document.querySelector("#chart"), options);
      chart.render();
    </script>

  </body>

Performance-wise, you can also explore this example with 1000 random datasets to see ApexChart performance with a large dataset.

To learn more: ApexChart Documentation

Chart.js

Chart.js is one of the most used JavaScript data visualization libraries and the second most starred on GitHub after D3.js, with over 57k stars as at the time of writing this article.
Chart.js provides simple, clean, and engaging charts that you can quickly incorporate into your application.

Some of its advantages are that it’s fairly easy to integrate this library into your application. It provides out-of-the-box support for nine different chart types, including Scatter Plot, Line, Bar, Pie, Donut, Bubble, Area, Radar, and Mixed Chart, all of which are also responsive and accessible by default. Additionally, its integration includes customizable labels, legends, hover pop-ups, and series toggling.

One of Chart.js' drawbacks is that its charts are rendered with a dull gray color by default, which can make styling different chart parts with different colors difficult when working with large datasets. You'll also need to put in extra effort to support basic chart features like zooming and panning.

How to use

Chart.js can be installed with npm via

$ npm install chart.js

Or include its script CDN link in your markup like below:

<script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/3.9.1/chart.min.js"></script>

Chart.js usage is pretty much similar to that of ApexChart. To create a chart, you need to create a new instance of the Chart object while adding the chart target, its dataset, and additional option as its parameters. Below is an example of a simple Pie Chart created with chart.js:

 <head>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/3.9.1/chart.min.js"></script>
  </head>

  <body>
    <div class="container">
      <div id="chart">
        <canvas id="myChart" style="height: 60vh"></canvas>
      </div>
    </div>
  </body>

  <script>
    const ctx = document.getElementById("myChart");
    const myChart = new Chart(ctx, {
      type: "bar",
      data: {
        labels: ["Red", "Blue", "Green"],
        datasets: [
          {
            label: "# Ratings",
            data: [8, 19, 12],
            backgroundColor: ["#f44336", "#3f51b5", "#009688"],
            borderWidth: 2,
          },
        ],
      },
      options: {
        scales: {
          y: {
            beginAtZero: true,
          },
        },
        responsive: false,
      },
    });
  </script>

To learn more: Chart.js documentation

D3.js

D3.js is a javascript data viz library for adding dynamic and interactive visualizations to web applications. It is one of the most used and currently the most starred javascript data viz library on GitHub (with over 102k stars currently). Its first version was released in 2011, which is partly why it is so ubiquitous

D3.js differs from some of the other libraries mentioned in this tutorial. It does not directly provide charting features, such as how ApexCharts and Chart.js allow you to render charts by calling a single function. D3.js is more of a building block for constructing data visualization, i.e., it provides methods that let you gradually build your chart from ground up.

For this reason, it’s much more difficult to quickly create a simple chart with D3.js, as you need to know how its functions and modules work. As we'll see below, it also takes some pretty long code to create a basic chart with the library itself. However, its integration with other JavaScript libraries like react and vue has made it fairly easier to use.

How to use

You can get started with using D3.js by including its CDN link (which export a global d3 object) in your markup:

<script src="https://cdn.jsdelivr.net/npm/d3@7"></script>

You can also install d3 in a node.js project with the command npm install d3. Or using ES6 import:

<script type="module">
   import * as d3 from "https://cdn.skypack.dev/d3@7";
</script>

Let's explore the following example of creating a simple line chart with d3.js:

<script src="https://d3js.org/d3.v4.js"></script>

<div id="my_dataviz"></div>
<script>
  var margin = { top: 10, right: 30, bottom: 30, left: 60 },
    width = 460 - margin.left - margin.right,
    height = 400 - margin.top - margin.bottom;

  var svg = d3
    .select("#my_dataviz")
    .append("svg")
    .attr("width", width + margin.left + margin.right)
    .attr("height", height + margin.top + margin.bottom)
    .append("g")
    .attr("transform", "translate(" + margin.left + "," + margin.top + ")");

  d3.csv(
    "https://raw.githubusercontent.com/holtzy/data_to_viz/master/Example_dataset/3_TwoNumOrdered_comma.csv",

    function (d) {
      return { date: d3.timeParse("%Y-%m-%d")(d.date), value: d.value };
    },

    function (data) {
      var x = d3
        .scaleTime()
        .domain(
          d3.extent(data, function (d) {
            return d.date;
          })
        )
        .range([0, width]);
      svg
        .append("g")
        .attr("transform", "translate(0," + height + ")")
        .call(d3.axisBottom(x));

      var y = d3
        .scaleLinear()
        .domain([
          0,
          d3.max(data, function (d) {
            return +d.value;
          }),
        ])
        .range([height, 0]);
      svg.append("g").call(d3.axisLeft(y));

      svg
        .append("path")
        .datum(data)
        .attr("fill", "none")
        .attr("stroke", "steelblue")
        .attr("stroke-width", 1.5)
        .attr(
          "d",
          d3
            .line()
            .x(function (d) {
              return x(d.date);
            })
            .y(function (d) {
              return y(d.value);
            })
        );
    }
  );
</script>

With the code above, you can see how we had to first create an SVG element, load our chart data set from a remote API, and also use d3 to append this data into different paths in our SVG element.

It’s also worth mentioning that there’s a D3.js graph gallery which is a collection of ready-made charts created with d3 like the example.

To learn more: D3.js documentation

Techan.js

Techan.js is a charting library built on top of D3.js for adding interactive financial charts to your application. Techan lets you add stock charts, including candlestick, OHCL, indicators, and further technical analysis, which is particularly useful for financial applications.

While this library is still admirably serving its purpose and actively being used, it has not been updated for over 6 years now. This might be a concern if you’d prefer a library that is being actively reviewed. Another drawback of Techan.js is that you’ll need background knowledge of how d3.js works before you can understand how the library itself works.

How to use

Before you can get started with using Techan.js in your web application, you’ll need to include it’s script CDN, along with d3.js CDN:

<script src="https://d3js.org/d3.v4.min.js"></script>
<script src="http://techanjs.org/techan.min.js"></script>

Explore the following example of creating a simple candlestick chart with techan and d3.js:

<style>
  path.candle {
    stroke: #000000;
  }

  path.candle.body {
    stroke-width: 0;
  }

  path.candle.up {
    fill: #00aa00;
    stroke: #00aa00;
  }

  path.candle.down {
    fill: #ff0000;
    stroke: #ff0000;
  }
</style>
<body>
  <button>Update</button>
  <script src="https://d3js.org/d3.v4.min.js"></script>
  <script src="http://techanjs.org/techan.min.js"></script>
  <script>
    var margin = { top: 20, right: 20, bottom: 30, left: 50 },
      width = 960 - margin.left - margin.right,
      height = 500 - margin.top - margin.bottom;

    var parseDate = d3.timeParse("%d-%b-%y");

    var x = techan.scale.financetime().range([0, width]);

    var y = d3.scaleLinear().range([height, 0]);

    var candlestick = techan.plot.candlestick().xScale(x).yScale(y);

    var xAxis = d3.axisBottom().scale(x);
    var yAxis = d3.axisLeft().scale(y);

    var svg = d3
      .select("body")
      .append("svg")
      .attr("width", width + margin.left + margin.right)
      .attr("height", height + margin.top + margin.bottom)
      .append("g")
      .attr("transform", "translate(" + margin.left + "," + margin.top + ")");

    d3.csv("data.csv", function (error, data) {
      var accessor = candlestick.accessor();

      data = data
        .slice(0, 200)
        .map(function (d) {
          return {
            date: parseDate(d.Date),
            open: +d.Open,
            high: +d.High,
            low: +d.Low,
            close: +d.Close,
            volume: +d.Volume,
          };
        })
        .sort(function (a, b) {
          return d3.ascending(accessor.d(a), accessor.d(b));
        });

      svg.append("g").attr("class", "candlestick");

      svg
        .append("g")
        .attr("class", "x axis")
        .attr("transform", "translate(0," + height + ")");

      svg
        .append("g")
        .attr("class", "y axis")
        .append("text")
        .attr("transform", "rotate(-90)")
        .attr("y", 6)
        .attr("dy", ".71em")
        .style("text-anchor", "end")
        .text("Price ($)");

      // Data to display initially
      draw(data.slice(0, data.length - 20));
      // Only want this button to be active if the data has loaded
      d3.select("button")
        .on("click", function () {
          draw(data);
        })
        .style("display", "inline");
    });

    function draw(data) {
      x.domain(data.map(candlestick.accessor().d));
      y.domain(techan.scale.plot.ohlc(data, candlestick.accessor()).domain());

      svg.selectAll("g.candlestick").datum(data).call(candlestick);
      svg.selectAll("g.x.axis").call(xAxis);
      svg.selectAll("g.y.axis").call(yAxis);
    }
  </script>
</body>

This example was recreated from techan.js gallery, which also includes code examples for the different types of financial charts the library supports.

To learn more: Techan.js documentation

AnyChart

AnyChart is another interesting charting library offering a different suite of chart types, including the basics charts (bar, pie, and others), different stock charts, and even PERT charts.

With AnyChart you can also integrate drawing tools into its stock charts, which is very useful in financial applications. AnyChart also offers chart localizations, with support for over 194 predefined locales, so you can visualize data for your users in different languages.

However, while AnyChart is open-source and free for non-profit use, you’ll need to purchase a license for use in commercial applications.

How to use

AnyChart offers integration with different languages, libraries, and frameworks; however, for this tutorial, we’ll be exploring its react plugin.

We can add AnyChart to our react application by installing with the following command:

$ npm install anychart-react

Once installed, we can then import its component and render a simple bar chart like below:

import AnyChart from "anychart-react/dist/anychart-react";

function App() {
  return (
    <div>
      <h1>Hello</h1>
      <AnyChart type="bar" data={[3, 6, 2, 7, 9]} title="Simple bar chart" />
    </div>
  );
}

export default App;

For integration with other languages and frameworks, you can refer to their documentation page here.

Taucharts

This is another interesting open-source chart library built on top of d3.js and mostly used for data exploration and analysis. However, its integration is independent of prior familiarity with d3.js, unlike Techan.js.

Taucharts has a nice framework and has lots of extensibility options. Its plugin infrastructure is adaptable, allowing you to write plugins quickly.

How to use

For a traditional HTML project, you’ll need to include d3.js, taurichart.js, along with the Tauri CSS file like below:

 <head>
    <link
      rel="stylesheet"
      href="https://cdn.jsdelivr.net/npm/taucharts@2/dist/taucharts.min.css"
    />
    <!-- .. -->
    <script src="https://cdn.jsdelivr.net/d3js/latest/d3.min.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/taucharts@2/dist/taucharts.min.js"></script>
  </head>

You can also install with npm via:

$ npm install taucharts

And once the installation is completed, you can get started with integrating it in your application by creating a new instance of the Taucharts object, including your dataset/data source, chart type, and other additional options to style your chart as its parameters.

Here’s what creating a simple line chart looks like with Taucharts:

<head>
    <link
      rel="stylesheet"
      href="https://cdn.jsdelivr.net/npm/taucharts@2/dist/taucharts.min.css"
    />
    <!-- .. -->
    <script src="https://cdn.jsdelivr.net/d3js/latest/d3.min.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/taucharts@2/dist/taucharts.min.js"></script>
  </head>

  <body>
    <div id="line"></div>
    <script>
      var datasource = [
        {
          type: "us",
          count: 0,
          date: "12-2013",
        },
        {
          type: "us",
          count: 10,
          date: "01-2014",
        },
        {
          type: "us",
          count: 15,
          date: "02-2014",
        },
        {
          type: "us",
          count: 12,
          date: "03-2014",
        },
        {
          type: "us",
          count: 16,
          date: "04-2014",
        },
        {
          type: "us",
          count: 13,
          date: "05-2014",
        },
        {
          type: "bug",
          count: 21,
          date: "01-2014",
        },
        {
          type: "bug",
          count: 19,
          date: "02-2014",
        },
        {
          type: "bug",
          count: 23,
          date: "03-2014",
        },
        {
          type: "bug",
          count: 26,
          date: "04-2014",
        },
        {
          type: "bug",
          count: 23,
          date: "05-2014",
        },
      ];
      var chart = new Taucharts.Chart({
        data: datasource,
        type: "line",
        x: "date",
        y: "count",
        color: "type",
      });

      chart.renderTo(document.getElementById("line"));
    </script>
  </body>
</html>

To learn more: Taucharts documentation

Zoomcharts

Zoomcharts helps you add interactive data visualizations to your application with minimal input and development time by using its ready-to-go charts library. Zoomcharts supports different chart types, notably network charts, matrix charts, facet charts, time charts, geo-map charts, and many more.

Zoomcharts is free for non-production use, however, you’ll also need to purchase a license and obtain an API key once your application is in production. Otherwise the library will start showing an error in your browser console.

How to use

You can add zoomcharts to your web project with its CDN link:

<script src="https://cdn.zoomcharts-cloud.com/1/stable/zoomcharts.js"></script>

And depending on the type of chart you’re trying to render, it's pretty much easy to do with a few lines of code. Below is an example of a time chart from the zoomcharts documentation:

    var t = new TimeChart({
        container: document.getElementById("demo"),
        data:
        {
            units:["h"],
            url: "https://zoomcharts.com/dvsl/data/time-chart/temperature-kuldiga-h.json"
        }
    });

To learn more: Zoomcharts documentation

Canvasjs

Canvasjs offers a lightweight library for creating performant and elegant looking charts, with over 30 different customizable charts to choose from. Canvasjs stock chart, in comparison to some of the other stock chart libraries (Techan.js), tends to be more intuitive and interactive. It also offers integration with the top javascript frameworks including react, vue, and angular, as well as with java, spring, and PHP.

Additionally, like AnyChart, Canvasjs also offers localization of charts, so that you’re able to render your chart elements in different languages.

However, while Canvasjs is free for development use, the development version will add a watermark to all your charts, and you’ll need to obtain a license to use it in production.

How to use

To get started, you first need to download the canvasjs compressed library here. The downloaded file once uncompressed includes a canvasjs.min.js file, which you can link to your website. Optionally, you could include the CDN directly in your markup like below:

<script src="https://canvasjs.com/assets/script/canvasjs.min.js"></script>

And it's pretty much easy to start creating any chart. Here’s an example of a bubble chart from the Canvasjs documentation:

<html>
  <head>
    <script>
      window.onload = function () {
        var chart = new CanvasJS.Chart("chartContainer", {
          title: {
            text: "Comparison among Countries on Fertility Rate Vs Life Expectancy in 2009",
          },
          axisX: {
            title: "Life Expectancy",
            maximum: 85,
          },
          axisY: {
            title: "Fertility Rate",
          },

          legend: {
            verticalAlign: "bottom",
            horizontalAlign: "left",
          },
          data: [
            {
              type: "bubble",
              legendText: "Size of Bubble Represents Population",
              showInLegend: true,
              legendMarkerType: "circle",
              toolTipContent:
                "<strong>{name}</strong> <br/> Fertility Rate: {y}<br/> Life Expectancy: {x} yrs<br/> Population: {z} mn",
              dataPoints: [
                { x: 78.1, y: 2.0, z: 306.77, name: "US" },
                { x: 68.5, y: 2.15, z: 237.414, name: "Indonesia" },
                { x: 72.5, y: 1.86, z: 193.24, name: "Brazil" },
                { x: 76.5, y: 2.36, z: 112.24, name: "Mexico" },
                { x: 50.9, y: 5.56, z: 154.48, name: "Nigeria" },
                { x: 68.6, y: 1.54, z: 141.91, name: "Russia" },

                { x: 82.9, y: 1.37, z: 127.55, name: "Japan" },
                { x: 79.8, y: 1.36, z: 81.9, name: "Australia" },
                { x: 72.7, y: 2.78, z: 79.71, name: "Egypt" },
                { x: 80.1, y: 1.94, z: 61.81, name: "UK" },
                { x: 55.8, y: 4.76, z: 39.24, name: "Kenya" },
                { x: 81.5, y: 1.93, z: 21.95, name: "Australia" },
                { x: 68.1, y: 4.77, z: 31.09, name: "Iraq" },
                { x: 47.9, y: 6.42, z: 33.42, name: "Afghanistan" },
                { x: 50.3, y: 5.58, z: 18.55, name: "Angola" },
              ],
            },
          ],
        });

        chart.render();
      };
    </script>
    <script src="https://canvasjs.com/assets/script/canvasjs.min.js"></script>
  </head>
  <body>
    <div id="chartContainer" style="height: 300px; width: 100%"></div>
  </body>
</html>

For more information, and integrating with other javascript frameworks, refer to their official documentation.

Highcharts

Highcharts is another great data visualization solution offering interactive charting capability. Compared to the other libraries in this article, Highcharts charts seem to offer more ready-made chart types.

Highcharts charts offer a lot of unique features, you can integrate your chart with different theme options, you can even create 3D charts, export charts in different formats, and convert your rendered charts to data tables on the fly. Another awesome feature is that you can create charts that are accessible to screen readers by adding descriptions to chart elements.

However, while highcharts is also free for non-commercial and personal use, you’ll need to purchase a license to integrate its libraries into commercial applications.

How to use

It's easy to get started with using highchart by adding its script CDN to your application:

<script src="https://code.highcharts.com/highcharts.js"></script>

And depending on the additional plugin you want to use, you’ll also need to import them independently.

It also requires less code to create high-level charts with this library. With this map projection example from their documentation, we can see how easy it is to visualize the world population density on a map.

<script src="https://code.highcharts.com/maps/highmaps.js"></script>
<script src="https://code.highcharts.com/maps/modules/exporting.js"></script>
<div id="container"></div>

<script>
  (async () => {
    const mapData = await fetch(
      "https://code.highcharts.com/mapdata/custom/world.topo.json"
    ).then((response) => response.json());

    const data = await fetch(
      "https://cdn.jsdelivr.net/gh/highcharts/highcharts@v7.0.0/samples/data/world-population-density.json"
    ).then((response) => response.json());

    // Initialize the chart
    Highcharts.mapChart("container", {
      title: {
        text: "Predefined zoomed area",
      },

      mapNavigation: {
        enabled: true,
        buttonOptions: {
          verticalAlign: "bottom",
        },
      },

      mapView: {
        projection: {
          name: "WebMercator",
        },
        center: [10, 58],
        zoom: 2.8,
      },

      colorAxis: {
        min: 1,
        max: 1000,
        type: "logarithmic",
      },

      legend: {
        title: {
          text: "Population density per km²",
        },
      },

      series: [
        {
          data,
          mapData,
          joinBy: ["iso-a2", "code"],
          name: "Population density",
          states: {
            hover: {
              color: "#a4edba",
            },
          },
          tooltip: {
            valueSuffix: "/km²",
          },
        },
      ],
    });
  })();
</script>

To learn more: Highcharts documentation

Toast UI Chart

Toast UI chart is another open-source chart solution. Toast UI isn’t a complete charting library per se; it’s more of a JavaScript library providing UI components similar to bootstrap. However, it has been included in this article since this type of thing is uncommon - UI libraries with support for charts.

Some of Toasts UI chart’s best features are that it’s lightweight, offers a wide range of chart types, and easily integrates charts via JavaScript components.

How to use

You can install Toast UI charts in your project with

$ npm install --save @toast-ui/chart

Or load its CDN via:

<link rel="stylesheet" href="https://uicdn.toast.com/chart/latest/toastui-chart.min.css" />
<script src="https://uicdn.toast.com/chart/latest/toastui-chart.min.js"></script>

Here's an interactive example of a live updating chart created with Toast UI chart:

<link
  rel="stylesheet"
  href="https://uicdn.toast.com/chart/latest/toastui-chart.min.css"
/>
<script src="https://uicdn.toast.com/chart/latest/toastui-chart.min.js"></script>
<div id="chart-area"></div>

<script>
  const el = document.getElementById("chart-area");
  const data = {
    categories: ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10"],
    series: [
      {
        name: "A",
        data: [10, 100, 50, 40, 70, 55, 33, 70, 90, 110],
      },
      {
        name: "B",
        data: [60, 40, 10, 33, 70, 90, 100, 17, 40, 80],
      },
    ],
  };
  const options = {
    chart: { title: "LiveUpdate", width: 900, height: 400 },
    xAxis: { pointOnColumn: false, title: { text: "X Title" } },
    yAxis: { title: "Y Title" },
    series: { shift: true },
  };

  const chart = toastui.Chart.areaChart({ el, data, options });
  let index = 11;
  const intervalId = setInterval(() => {
    const random = Math.round(Math.random() * 100);
    const random2 = Math.round(Math.random() * 100);
    chart.addData([random, random2], index.toString());
    index += 1;
    if (index === 30) {
      clearInterval(intervalId);
    }
  }, 1500);
</script>

To learn more: Toast UI documentation

Conclusion

And it’s a wrap! Throughout this article, we’ve explored ten different JavaScript data visualization libraries, the type of visualization tools they provide, additional features, their drawbacks, and how to get started with using them with interactive code samples.

We can’t definitively say which library is the best–that depends on how easily it integrates with your particular application and your intended use case–but hopefully, by now, you have some ideas on where to start.

This post was written by Elijah Asaolu, and I’m a software engineer and technical writer, actively sharing all I've learned through writing. You can also follow me on Twitter if you appreciate programming memes and tips.

A Guide To Svelte Stores

Abdulazeez Abdulazeez — Mon, 13 Mar 2023 16:36:42 +0000

State management is an important aspect of programming that helps to control the flow and behaviour of data within an application. By effectively managing the state of an application, it becomes easier to maintain and scale the application, resulting in improved efficiency and performance.

State management also makes it easier for developers to share data between different components of an application and to organize code in a way that is more efficient and effective. This can help to improve the overall functionality of the application.

In this article, we’ll discuss the svelte store, which is one of the ways we can manage our states in svelte. It is also regarded as the most efficient way to manage complex data.

Prerequisites

To follow through with this article, the following prerequisites are required:

Familiarity with the svelte framework
Understanding svelte components: Svelte components are reusable, self-contained UI elements that can be easily shared, imported, and composed to build complex user interfaces.
Reactivity: State management in svelte is based on the concept of reactive programming. It means that all the variables used by the application are tracked and can be modified by the user. When a variable is changed, the application will update its state accordingly. This makes it easier to track changes and keep the application running smoothly, as svelte stores are reactive.

Getting started

Let’s first create a basic application using the CoderPad Svelte sandbox to show how we can pass data between components.

In the sandbox, create a new file Child.svelte:

<!--- Child.svelte-->
<script>
  export let greeting
</script>
<span>{greeting}</span>

Replace the content of App.svelte with the code below:

<!--- App.svelte-->
<script>
import Child from "./Child.svelte"
</script>
<h1><Child  greeting ="hello world"/></h1>
<h1><Child  greeting ="hello john"/></h1>
<h1><Child  greeting ="hello micheal"/></h1>
<h1><Child  greeting ="hello doe"/></h1>
<h1><Child /></h1>

Here, we are defining the values of our variable greeting and passing them as props in our parent component, our App.svelte, to the child component, Child.svelte. Our child components then access these props and render them into the browser.

Component props are limited in how much data they can hold. Because of this, complex data sets would need to be broken down into multiple props, which can get messy and complicated.

We might also face a scenario whereby components do not need the props. We still have to send the props through them to be able to pass it further down in the component tree just for a particular component to access the data, which might not be a good thing if we are working with complex data in an application.

It would be nice to directly make the data available to the required component without manually drilling the props through every level of the component tree. One of the things we can use is the Context API. We then have the option of using a context.

The context API

The Svelte Context API is a feature that allows components to share states and functions with their children, grandchildren, and descendants. It provides a way of passing values and functions down the component tree without having to pass them at each level explicitly. This makes it easier to maintain and reason about the data flow in an application.

Svelte's context API is ideal for cross-component communication without complicating your codebase with props. The context API is made possible by two Svelte functions: getContext and setContext. Setting an object or value in the context and associating it with a key enables you to make it available anywhere within your app, as shown in the code below:

<script>
  import { setContext } from "svelte";
  const thisObject = {};
  setContext("thisKey", thisObject);
</script>

We want thisKey to be available inside a different component within our app, so we import it using the getContext function.

<script>
  import { getContext } from "svelte";
  const thisObject = getContext("thisKey");
</script>

For example, passing an array of todos between components:

<!-- App.svelte-->
<script>
  import { setContext } from "svelte";
  import Child from "./Child.svelte";
  let todo = ["do chores", "clean", "code", "play football"];
  setContext("key", todo);
</script>
<Child />

In the code above, we set the context key as key and pass todo object as the second argument.

We can now access the todo context from any other components like below:

<!-- TodoCard.svelte-->
<script>
  import { getContext } from "svelte";
  let todo = getContext("key");
</script>
<p>{todo}</p>

Although the Svelte context API is great for small applications, it becomes increasingly more work to manage the state simply and concisely as applications scale up.

Additionally, the context API does not provide any built-in methods for tracking changes to the data, making it difficult to track which pieces of data are outdated and need to be updated. For these reasons, it is better to use a store for managing and passing complex data between components.

Introducing svelte stores

A Svelte store is a JavaScript object that holds data similar to a variable. However, unlike a variable, Svelte stores can be observed, meaning that any part of the application can monitor changes to the store and respond accordingly.

It also makes it easier to keep track of changes to the data and makes it easier to modify the data when needed. The store is also the main way of sharing data between different parts of an application, making it easier to keep the code organized and maintainable.

The svelte store offers a solution that helps developers overcome these challenges by helping us manage our state. It detaches and manages it in an entirely different place so components can’t become bloated with unnecessary data. This makes Svelte Stores ideal for managing state in Svelte applications, as we can use them to store application data and allow components to reflect changes in the store.

Svelte has two primary types of stores, writable and readable stores. Svelte also offers a special type of store called a derived store.

Writable Stores

Writable stores are basically objects that hold values or attributes that various app components may access. They are the sources of information storage. Let's use the writable store to store a value we can later change or pass around in our application. We'll create our store as a JavaScript file, add some values and then export it:

<!-- weather.js -->
<script>
  import writable from 'svelte/store'
  export const weather = writable('sunny')
</script>

We can then import inside any other component where we'll need the data:

<script>
  import { weather } from './weather.js'
</script>

We can alter the value in a writable store, and when we need to change the value of weather in any component where we imported it, we can use the unset() method. The unset() method is used to remove a value that has been previously set in a component's state:

<!-- Newweather.svelte -->
<script>
  import { Weather } from './Weather.js'
  city.set('Rainy')
</script>

We can also use the update() method. This will help us invoke a callback in which the current value is passed as an argument:

<!-- Newweather.svelte -->
<script>
  import { Weather } from './Weather.js'
  const newWeather = 'Rainy'
  function changeWeather() {
    Weather.update(existing => newWeather)
  }
</script>

Alternatively, we could have components use Svelte's subscribe() function to monitor changes to the value we put in our store. The subscribe method in Svelte is used to add a reactive effect to a component:

<script>
  import { Weather } from './Weather.js'
  const watch = Weather.subscribe(value => {
    console.log(value);
  });
</script>

Readable stores

Readable stores, like writable stores, hold objects but cannot be modified by external components. You must set the value of a readable store when you create it. We use Readable stores to handle data that must be immutable. Below is an example of fetching the current date:

<!-- date.js -->
import { readable } from 'svelte/store'

export const date = readable(new Date())

<!-- app.svelte -->
<script>
import {date} from "./date.js"
</script>

<h1>Today's date is{$date}</h1>

Derived Stores

Derived stores, as the name implies, are derived from another store. The value of a derived store is updated when the store from which it was derived is updated. Derived stores are simply readable stores, but their values come from somewhere else, at least partially:

import { derived } from "svelte/store";
import { date } from "./store.js";
export const test = derived(date, ($date) => {
  return (
    $counter.getHours() +
    ":" +
    $counter.getMinutes() +
    ":" +
    $counter.getSeconds()
  );
});

Our derived store can then be imported into another component as follows:

<!-- App.svelte -->
<script>
  import { test } from "./derived.js";
</script>
<h1>
  {$test}
</h1>

Svelte store example: Building a student data list

To showcase how Svelte store could be used in real-time, we will be building an app where we’ll be able to manage state in svelte.

We can set up a new project folder and install svelte for this project example by following the steps mentioned earlier, or we can use the one created previously.

Project scope

For this project, we’ll be building a student data list that will display some student data, and we’ll do that with the help of a store.

Creating a store

We’ll be creating a store that will contain some student data. Inside the src folder, create a folder called stores and create a file called students.js. This file will define the methods for updating our store's values and resetting them to their default values when needed. Our students.js will contain the following:

import { writable } from "svelte/store";
const STUDENTS = [
  { name: "john", surname: "doe", age: 17 },
  { name: "micheal", surname: "angelo", age: 21 },
];

const { subscribe, set, update } = writable(STUDENTS);
const addStudent = (student) =>
  update((students) => {
    return [...students, student];
  });

const reset = () => {
  set(STUDENTS);
};

export default {
  subscribe,
  addStudent,
  reset,
};

Now we’ll import the exported methods, which are the subscribe, addStudent, and reset, into the component we’ll create shortly and use them to subscribe, add new values and reset our student list.

Creating our components

Next, we will create components that accept the methods exported from our store. We'll be creating two components: StudentPage.svelte, which will be used to add new student data, and StudentList, which will display the list of students that have been added.

In StudentPage.svelte, add the following:

<script>
  import students from "./stores/students";
</script>
<div class="studentPage">
  {#each $students as student}
    <p>{student.name} {student.surname}</p>
    <p>{student.age}</p>
    <hr />
  {/each}
</div>

We’ll make our StudentList a form so we can add new student data as well as reset them:

<script>
  import students from "./stores/students";
  // binding values
  let name = "";
  let surname = "";
  let age = "";
  // submit student
  const submitStudent = () => {
    students.addStudent({ name, surname, age });
    // reset values
    name = surname = age = "";
  };
</script>

<div class="studentList">
  <input type="text" bind:value={name} placeholder="Enter student make" />
  <input type="text" bind:value={surname} placeholder="Enter student surname" />
  <input type="year" bind:value={age} placeholder="Enter student age" />
  <input type="submit" on:click={submitStudent} value="Include Student" />
  <input type="submit" on:click={students.reset} value="Reset Student list" />
</div>

Let’s add some basic styling to our components. In our app.css, paste this in:

.studentList {
  margin: 0 auto;
  box-shadow: 0 0 30px 0;
  height: auto;
  max-width: 410px;
  background-color: rgb(231, 231, 231);
  padding: 20px;
}

input {
  margin: 30px 0 0;
  height: 30px;
  width: 100%;
  padding: 0 10px;
  outline: none;
  border: none;
  background-color: rgb(231, 231, 231);
  border-bottom: 1px solid #000;
  font-size: 1.1rem;
  transition: .3s;
}

button {
  border-radius: 6px;
  margin: 20px 0 0;
  height: 40px;
  width: 100%;
  color: #fff;
  padding: 0;
  outline: none;
  cursor: pointer;
  transition: .3s;
  border: none;
}

.button {
  background-color: rgba(26, 26, 204, 0.582);
}

.button:hover {
  background-color: rgb(26, 26, 204);
}

.reset {
  background-color: rgb(189, 42, 42);
}

.reset:hover {
  background-color: rgb(255, 0, 0);
}

/* studentpage styling */
.studentPage {
  padding-top: 30px;
  display: flex;
  justify-content: center;
  align-items: center;
}

.table-bordered {
  border-width: 1px 1px 1px 0px;
  border-style: solid solid solid none;
  border-color: #fff #fff #fff -moz-use-text-color;
  border-collapse: collapse;
  border-radius: 4px;
  font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
  max-width: 100%;
  overflow-x: auto;
}

.table-bordered th,
.table-bordered td {
  min-width: 150px;
  padding: 8px;
  line-height: 20px;
  text-align: left;
  vertical-align: top;
  border: 1px solid #fff;
}

.table-bordered th {
  background-color: rgb(88, 83, 83);
  color: #fff;
}

We’ll then import into each of our components using:

import "./app.css";

The last thing we’ll do is import both our components into our app.svelte:

<script>
  import StudentList from "./StudentList.svelte";
  import StudentPage from "./StudentPage.svelte";
</script>
<div>
  <StudentList />
  <StudentPage />
</div>

We just created an app and managed it using svelte stores! Play with the application in this sandbox.

Conclusion

Svelte stores are an effective way to manage and access data in your Svelte applications. They provide a centralized way to store data, making it easier to keep track of changes and make modifications when needed. Using a Svelte store is straightforward and can make your code cleaner and more maintainable.

Top 10 CSS Animation Libraries

Abdulazeez Abdulazeez — Wed, 22 Feb 2023 15:54:29 +0000

For the longest time, developers were limited to Flash players and gifs when they wanted to display animations on a web page.

However, the introduction of the keyframes, transition, and animation properties in CSS3 made it much easier to create and display abstract animations that were appealing to the eye. The CSS3 update not only allows for the creation of animations but also to conditionally render them for various pseudo states (i.e., hover, focus, etc.).

Creating animations with these properties is pretty straightforward. First, you define a keyframes rule containing the preferred animation sequence:

@keyframes float {
  0% {
    transform: translate(0, 0px);
  }
  50% {
    transform: translate(0, 15px);
  }
  100% {
    transform: translate(0, -0px);
  }
}

You can see that we define a keyframe named float, which at 0% of the animation flow, sets the element transform property to translate by 0%, at 50% to translate by 15px on the y axis, and at 100% translate back to 0%.

We can then apply this keyframe flow to an element via the animation property:

.element {
  animation: floating 3s ease-in-out infinite;
}

/* OR */

.element:hover {
  animation: floating 3s ease-in-out infinite;
}

You'll notice that it takes a lot of code to create a simple float animation, and the code gets even longer when working on more complex ones. Animation libraries essentially solve this problem by making the process of adding animations to webpages as simple as adding class names to the appropriate elements.

This post will look at ten common CSS animation libraries and platforms for various types of animations, such as simple motion, elements/page transitions, lightbox animations, loaders, animating pages on scroll, and many more. We'll look at their benefits, limitations, and code examples to get you started quickly. Here’s the full list of libraries that we’ll cover:

Animate.css
Animista
Animation Library
Magic CSS
lightGallery
Loading.io
Skeleton Element
Micron
AnimXYZ
AOS

Animate.css

Animate.css is one of the most popular CSS animation libraries, having over 76k stars on GitHub at the time of this writing. Animate.css allows you to effortlessly add several ranges of animation to your web application by just including their class names in the element you wish to animate. Animate.css is very handy for displaying on-page animations, animation on sliders, and overall attention-grabbing animations.

This library also includes extra utility classes that allow you to adjust animation duration, speed, and repeat animation directly from your markup. You can also integrate animations provided by this library with the native CSS keyframe property, and call them directly from your CSS code.

How to use

Animate.css is available as a npm/yarn package, as well as via CDN:

<head>
  <link
    rel="stylesheet"
    href="https://cdnjs.cloudflare.com/ajax/libs/animate.css/4.1.1/animate.min.css"
  />
</head>

After including its CDN URL in your markup head section, you may start calling from its list of available animations, as shown below:

<div class="animate__animated animate__pulse">A pulse animated element</div>
<!-- using utility class  -->
<div class="animate__animated animate__pulse animate__infinite">A pulse animated element</div>

In the first example, we created a div with a pulse animation that will animate for 1 second by default, and in the second example, we configured the pulse animation to run endlessly using the animate__infinite utility class (which is an equivalent of CSS animation-iteration-count: infinite;). You can learn more about Animate.css via the links below.

Documentation | Github

Animista

Animista is more of a CSS animation platform than a library in that they provide animation on demand, i.e., you go to the platform, select the type of animation you want, and the animation CSS keyframe code is instantaneously generated for you.

On the surface, the types of animations available on Animista are very similar to those offered on animate.css, but, when you delve deeper, Animista offers a lot more useful animation categories, particularly for animating words and background elements. And, when you're ready to export your animation code, you can optionally choose to download standard or minified code.

How to use

As mentioned previously, you do not need to download any packages or include any CDN in your website to begin utilizing Animista; simply visit their homepage, select your favorite animation, customize the animation sequence as desired, and the code will be generated for you instantly.

Here’s a GIF showing this process:

Documentation | Github

Animation library

Animation library could be considered an Animate.css alternative, as they offer similar animation categories. However, unlike the former, animation library does not provide the additional customization options that allow you to set a preferred animation duration, speed, or timing.

Furthermore, animation library source files are divided into distinct classes — for example, all variants of the fade animation are in a single source file, as are other animation classes.

How to use

After downloading the ZIP file containing all available animation categories, you can choose to link to any of them and begin calling their associated class names in your markup. Here's an example of linking rotate.css and using the rotateUpRight style:

<head>
  <link rel="stylesheet" href="path/to/rotate.css">
</head>

<body>
  <div class="rotateUpRight">
    This div will rotate up right.
  </div>
</body>

Documentation | https://github.com/kaustubhmenon/animation-library[Github](url)

Magic CSS

Magic CSS is another interesting animation library that offers even more engaging animations compared to the ones we’ve covered so far. The animations provided by this package are very handy for page transitions. However, one disadvantage of magic CSS is that it does not support the Opera mini-browser.

How to use

Magic CSS usage is also pretty straightforward. You can get started by downloading the library from npm/yarn or simply download its CSS source file, and include it in your webpage markup. You can then begin calling from its list of animations:

<head>
  <link rel="stylesheet" href="https://raw.githubusercontent.com/miniMAC/magic/master/dist/magic.min.css">
</head>

<body>
  <div class="magictime puffIn">
  </div>
</body>

Documentation | Github

lightGallery

lightGallery is unique from the other animation libraries we’ve mentioned so far, in that it is specifically designed for making lightbox images. Lightbox images, such as the one seen above, are images that, when clicked, overlay the current website in a modal form.

lightGallery is a versatile library in that it supports video files, allows you to render media assets in a carousel format, and lets you create custom plugins to extend or modify its functionality. It's also worth noting that lightGallery isn't entirely a pure CSS animation library because it relies on JavaScript to function.

How to use

lightGallery is available via npm/yarn, bower, and also served via a CDN. To get started on a traditional web page, you’ll first include its CSS and JavaScript CDN link in your web page like below:

<head>
  <link rel="stylesheet" href="/path/to/lightgallery-bundle.css" />
</head>

<body>
  <!-- . . .  -->
  <script src="/path/to/lightgallery.min.js"></script>
</body>

Following that, you create an element that will act as a container for all of the images you want to convert to lightbox:

<div id="lightbox-container">
  <a data-lg-size="1600-2400">
    <img src="path/to/img" />
  </a>
  <a data-lg-size="1024-800">
    <img src="path/to/img" />
  </a>
</div>

Finally, you initialize lightGallery via the previously created container, with the script below:

lightGallery(document.getElementById("lightbox-container"), {
  speed: 500
  //   ...
});

lightGallery also integrates with other JavaScript frameworks such as React, Angular, and Vue, making it even easier to use the library via components and props.

Documentation | Github

Loading.io

Loading.io is a platform that allows you to customize from a collection of rich loading animations and export your creation as a CSS keyframe animation, GIF, SVG, or PNG file. These animations are especially handy for building preloaders or depicting asynchronous activities’ loading states.

Loading.io offers more than just preloaders; the platform allows you to animate text and even background patterns. They also offer an additional loadingBar.js library for directly and interactively including its preloaders on a web page.

How to use

Loading.io usage is similar to Animista's in that no additional package is required to get started. You'd simply go to their website, choose a preferred loader, customize as desired, and then export.

Documentation | GitHub

Skeleton Elements

Skeleton loaders can also be categorized as preloaders. However, they differ from typical loaders in that skeleton loaders simulate the structure and appearance of loading content into gray blocks. It indicates what type of content is loading in each block, such as an image, text, and so on.

Skeleton elements is a popular library that provides this type of functionality, and it is also available as a component for several JavaScript frameworks.

How to use

The skeleton elements library is available via npm and is also served on a CDN. For a simple webpage, you can include its CDN in your markup head section like below:

<head>
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/skeleton-elements@3.4.0/skeleton-elements.css" />
</head>

You can then start creating skeleton loaders via the skeleton-text and skeleton-block class:

<body>
  <h3 class="skeleton-text">Placeholder Header</h3>
  <p class="skeleton-text">
    Lorem ipsum dolor sit amet consectetur adipiscing elit.
  </p>
  <div class="skeleton-block" style="width:30%; height:50px"></div>
</body>

You can also add loading effects via the skeleton-effect-EFFECT_NAME class. Here’s our previous example modified with a wave effect:

<body>
  <h3 class="skeleton-text skeleton-effect-wave">Placeholder Header</h3>
  <p class="skeleton-text skeleton-effect-wave">
    Lorem ipsum dolor sit amet consectetur adipiscing elit.
  </p>
  <div class="skeleton-block skeleton-effect-wave" style="width:30%; height:50px"></div>
</body>

Documentation | Github

Micron

Micron is a JavaScript-controlled microinteraction library created with CSS animations. Simply put, micron allows you to easily attach multiple animations to elements when they are clicked. There are also other setup options to help you regulate the animation flow.

Furthermore, you can bind and simply trigger animation from other elements - for example, you might have a button element activate an animation you'd attach to another div.

How to use

You get started with micron by including its CSS and Javascript source file CDN in your markup, as shown below:

<link href="htps://unpkg.com/webkul-micron@1.1.6/dist/css/micron.min.css" type="text/css" rel="stylesheet">
<script src="https://unpkg.com/webkul-micron@1.1.6/dist/script/micron.min.js" type="text/javascript"></script>

The library can then be used by adding a data-micron attribute to your elements. Here's an example of how to include the bounce interaction:

<div data-micron="bounce">Click to bounce</div>

Documentation | Github

Animxyz

Animxyz is another easy to use CSS interaction animation library that allows for customization via attributes. You can choose from a variety of animation options and easily integrate with other JavaScript frameworks including React and Vue.

This library was also created with SCSS, which allows for variable usage, so that you can easily customize the library source to your liking.

How to use

You can install this library using npm, import its core SASS mixins, and customize as desired. Or you can simply include its CDN link in the head section of your HTML file:

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@animxyz/core">

And you can start adding animations from its list by using the xyz attribute:

<div class="xyz-in" xyz="fade up big">I will animate in!</div>

Documentation | GitHub

AOS

Animate on scroll (AOS) is another fascinating library that allows you to animate your markup elements while they are scrolling. The library offers predefined animations such as fade, flip, zoom, and so on, and allows you to attach them to custom elements while users scroll through them. It also includes additional configuration options that allow you to set a preferred animation duration, delay, and number of times it should run.

Furthermore, you are not limited to using only the animations provided by the library; you may also attach your own native animations created with the keyframe property.

How to use

To get started you first include the AOS CSS and JavaScript CDN link in your web page and then initialize the library as shown below:

<link rel="stylesheet" href="https://unpkg.com/aos@next/dist/aos.css" />

<script src="https://unpkg.com/aos@next/dist/aos.js"></script>
<script>
  AOS.init();
</script>

You can now start animating markup elements on scroll using the data-aos attribute:

<div data-aos="fade-up">
  This div will fade up when scrolled to
</div>

Documentation | Github

Conclusion

While creating basic animations on the web is fairly simple, it becomes tiresome when working on complex ones. In this article, we've explored ten CSS libraries offering various ranges of animation to help save you the stress of reinventing the wheel.

This post was written by Elijah, a technical writer, and software developer, actively sharing all he has learned through writing. Follow me on Twitter if you enjoy programming tips and memes.

A Guide To Database Unit Testing with Pytest and SQLAlchemy

Abdulazeez Abdulazeez — Wed, 15 Feb 2023 11:56:03 +0000

Testing is a decisive phase in your systems development lifecycle. This is an important step to make your software reliable and maintainable in the future. In this tutorial, you will have a practical guide to unit testing, one type of software testing by which individual units of your code are tested to determine if they are fit for use.

We will discuss how to do unit testing using pytest, which is a Python testing tool. This tool has useful features to help write better programs. We will test a database created by SQLAlchemy ORM.

At the end of this tutorial, you will learn how to test SQLAlchemy ORM using fixtures and the classic way to implement fixtures (setup and teardown methods).

As always, to grasp the ideas introduced in this tutorial, you need to experiment with your code by yourself, whether in your local machine or here in the CoderPad sandbox. The Pad sandbox has a ready environment to try out your experiments.

You can select Python 3, click on the three dots in the top left corner, and then select SQLAlchemy (Postgres) from the adapters. You're free to choose a MySQL adapter, but in this tutorial, we will use the Postgres engine in SQLAlchemy.
Before you start, you need to know more about how to interact with databases using SQLAlchemy with Postgres and revise transactions in SQLAlchemy.

Creating the database models

We will use the simple database models introduced in this SQLAlchemy tutorial.

For this tutorial, we will dump all the code in the Pad on the left window. But for a production use case, you need to structure your files into two files, for example:

models.py to have the database models
test_blog.py to have the testing suite So the database models would have the following:

from sqlalchemy import create_engine, Column, Integer, String, DateTime, Text, ForeignKey
from sqlalchemy.engine import URL
from sqlalchemy.orm import declarative_base, relationship, sessionmaker
from datetime import datetime

Base = declarative_base()


class Author(Base):
    __tablename__ = 'authors'

    id = Column(Integer(), primary_key=True)
    firstname = Column(String(100))
    lastname = Column(String(100))
    email = Column(String(255), nullable=False)
    joined = Column(DateTime(), default=datetime.now)

    articles = relationship('Article', backref='author')


class Article(Base):
    __tablename__ = 'articles'

    id = Column(Integer(), primary_key=True)
    slug = Column(String(100), nullable=False)
    title = Column(String(100), nullable=False)
    created_on = Column(DateTime(), default=datetime.now)
    updated_on = Column(DateTime(), default=datetime.now, onupdate=datetime.now)
    content = Column(Text)
    author_id = Column(Integer(), ForeignKey('authors.id'))

url = URL.create(
    drivername="postgresql",
    username="coderpad",
    host="/tmp/postgresql/socket",
    database="coderpad"
    )

engine = create_engine(url)
Session = sessionmaker(bind=engine)

So we have two tables, authors and articles, and we want to test a few units of a transactional database in SQLAlchemy.

Testing SQLAlchemy with Pytest

In the previous section, we defined the Session class, which we will use in the testing class. You can define a class called TestBlog, which contains the test functions. Each test function should start with test_ so that Pytest can understand it's a function that contains test cases.

Fixtures classic way

Fixtures is a powerful feature in Pytest. While testing a database created in an SQLAlchemy ORM, you need to ensure that the session is open at each test function call with just one setup. You don't need to instantiate it at every test function. You'd also need to close the session after all unit tests are done.

The classic way of using Python database fixtures in Pytest is to use setup and teardown functions. These functions are useful to avoid repeating code at every test function. This is useful because hitting the database multiple times would be discouraged, especially if you're testing a large application.

Here is how to write these methods:

class TestBlog:
    def setup_class(self):
        Base.metadata.create_all(engine)
        self.session = Session()
        self.valid_author = Author(
            firstname="Ezzeddin",
            lastname="Aybak",
            email="aybak_email@gmail.com"
        )

    def teardown_class(self):
        self.session.rollback()
        self.session.close()

The setup_class is called before all test methods in the TestBlog, while the teardown_class is called after all test methods. Both are called at the class level.

The setup method creates all your tables; authors and articles tables. It then defines the session, which is the Session object in SQLAlchemy in which the conversation with the database occurs. It also has the valid_author object, which we would frequently use in the following test functions.

However, the teardown method is the clean-up phase after all test methods. It rolls back the changes to the database and then closes the session, so there wouldn't be any conversation channel between the SQLAlchemy and your database anymore.

Testing a success test case

Let's start with testing the content retrieved by the database of the valid author:

class TestBlog:
    # ...
    def test_author_valid(self):   
        self.session.add(self.valid_author)
        self.session.commit()
        aybak = self.session.query(Author).filter_by(lastname="Aybak").first()
        assert aybak.firstname == "Ezzeddin"
        assert aybak.lastname != "Abdullah"
        assert aybak.email == "aybak_email@gmail.com"

Here, we use the session to add the valid author object (defined in the setup function). We then commit that change to the database and query that author. At the end of the test function, we typically use an assert statement to verify test expectations.

Testing a failure test case

Let's add another test method under the TestBlog class to test an integrity error, which is an exception raised when there is relational data integrity is affected:

from sqlalchemy.exc import IntegrityError

# ...
class TestBlog:
    # ...
    @pytest.mark.xfail(raises=IntegrityError)
    def test_author_no_email(self):
        author = Author(
            firstname="James",
            lastname="Clear"
        )
        self.session.add(author)
        try:
            self.session.commit()
        except IntegrityError:
            self.session.rollback()

In this test function, we use a pytest.mark.xfail decorator to mark this test_author_no_email test function that can fail for some reason. The reason to fail here is to have an IntegrityError while committing an author object with no email. That's because when we defined the authors table, we made a restriction on the email attribute to not have a NULL record.

In this test function, we use a try/except statement to be able to roll back the transaction where the IntegrityError exception occurs while committing to the DB.

Testing a referenced table

To reference a foreign key, you need to have the referenced object defined in the same test function, or you can define that object in the setup phase, where you can import it to other test functions. In our case, we defined the valid_author object, in the setup function, which we referenced in the articles table:

class TestBlog:
    # ...
    def test_article_valid(self):
        valid_article = Article(
            slug="sample-slug",
            title="Title of the Valid Article",
            content="Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
            author=self.valid_author
            )
        self.session.add(valid_article)
        self.session.commit()
        sample_article = self.session.query(Article).filter_by(slug="sample-slug").first()
        assert sample_article.title == "Title of the Valid Article"
        assert len(sample_article.content.split(" ")) > 50

Here, we define an object for a valid article with the referenced valid author defined in the setup method. We then add and commit it to the database.
Finally, we verify our tests. For example, we verify the title and then verify the article's length to be greater than 50 words.

Running pytest

To invoke Pytest, you have some options mentioned in the Pytest documentation. In this tutorial, you can use the following line:

pytest.main(['-v'])

Place this line after the TestBlog class and outside of it to invoke Pytest in the same Python script.
We use the -v option here to increase verbosity and show each test function result.

Using fixtures

Instead of the classic way to use a fixture, let's look at how to define a fixture itself. But why do we need fixtures anyway?

In addition to the benefits of the classic way of defining fixtures, fixtures themselves are useful because they lead to dependency inversion. Dependency inversion is a design pattern useful when a test function receives other objects it depends on. It aims to separate the concerns which lead to loosely coupled programs.

Let's see fixtures in action and get rid of the setup_class and teardown_class methods. Now, define the following before the TestBlog class:

@pytest.fixture(scope="module")
def db_session():
    Base.metadata.create_all(engine)
    session = Session()
    yield session
    session.rollback()
    session.close()

@pytest.fixture(scope="module")
def valid_author():
    valid_author = Author(
        firstname="Ezzeddin",
        lastname="Aybak",
        email="aybak_email@gmail.com"
    )
    return valid_author

These fixtures are defined at the module level. That's why there is a scope option inside the pytest.fixture decorator. A fixture is called based on the scope.

ℹ️ A scope option is what you can use to share fixtures across classes, modules, packages, or sessions. In our case, the db_session and valid_author fixtures can only be called across this module.

The first fixture function creates all the tables metadata and then yields the session object whenever it's called. Each test function calling the session will receive the same session instance, thus saving time. So that fixture function is invoked once per the test module.

Using the yield statement is recommended. After the yield line, the session is rolled back and then closed, equivalent to the teardown code.

The second fixture function returns the valid author object whenever a test function calls it.

Note: The separation of concerns here is clear. Instead of the classic setup function, which contained both the session and the valid author objects, we now have two separate fixture functions for each.

Calling fixtures

To call a fixture inside a test function, you need to pass it as an argument to that function. Here is the new TestBlog class with the associated test functions:

class TestBlog:
    def test_author_valid(self, db_session, valid_author):
        db_session.add(valid_author)
        db_session.commit()
        aybak = db_session.query(Author).filter_by(lastname="Aybak").first()
        assert aybak.firstname == "Ezzeddin"
        assert aybak.lastname != "Abdullah"
        assert aybak.email == "aybak_email@gmail.com"

    @pytest.mark.xfail(raises=IntegrityError)
    def test_author_no_email(self, db_session):
        author = Author(
            firstname="James",
            lastname="Clear"
        )
        db_session.add(author)
        try:
            db_session.commit()
        except IntegrityError:
            db_session.rollback()

    def test_article_valid(self, db_session, valid_author):
        valid_article = Article(
            slug="sample-slug",
            title="Title of the Valid Article",
            content="Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
            author=valid_author
            )
        db_session.add(valid_article)
        db_session.commit()
        sample_article = db_session.query(Article).filter_by(slug="sample-slug").first()
        assert sample_article.title == "Title of the Valid Article"
        assert len(sample_article.content.split(" ")) > 50

As you can see, db_session and valid_author are passed into each test function as arguments. The db_session fixture replaces each self.session, and each self.valid_author is now replaced by the valid_author fixture.

Wrapping up

This tutorial has covered how to do unit testing for a transactional database in SQLAlchemy using Pytest. You started with creating the database models and then went through the classic way of using Pytest fixtures.
Finally, you learned how to use fixtures and why they are useful for writing efficient tests.

This post was written by Ezz. Ezz is an AWS Certified Machine Learning Specialist and a Data Platform Engineer. I help SaaS companies rank on Google. Check out his website for more.

A Guide to CSS Flexbox

Abdulazeez Abdulazeez — Thu, 19 Jan 2023 21:12:05 +0000

This article was originally published on the CoderPad blog

When you add HTML content to your web page, it stacks on top of itself, leaving a lot of space on the edges or making your content look odd. The float CSS property, along with position, was then used to define layouts and position elements, but it was complicated to use, especially when dealing with responsiveness.

You no longer need to use the float and position properties to organize your page's layout and style containers or content side by side. With the advancement of the web and technology, you now have two CSS layout systems: flexbox and CSS Grid. This guide focuses on CSS Flexbox, but you can bookmark our detailed guide on CSS Grid.

This guide will teach you everything you need to know about the CSS Flexbox layout system, including the Flexbox layout axis and all of its properties, justifying and aligning flex items, adding gaps and ordering these elements, and much more. You'll also learn when to use flexbox and when to use CSS grid by the end of this guide.

What is Flexbox?

CSS Flexbox is a one-dimensional layout system that you can use to create a row or column axis layout. It is a module rather than a single property because it includes several properties, some of which are for the flex container (parent element) and others for the flex items (children elements). This layout module allows the container to change its items' width/height (and order) to fit the available space best and support all display devices and screen sizes.

CSS Flexbox was introduced in 2009 as a new layout system to make it easier to build responsive web pages and organize your elements, and it has grown in popularity since then. Flex is designed to provide a more efficient way of laying out all items, aligning, and distributing space among each item in a container, even when each flex item's sizes are unknown and/or dynamic.

It simplifies designing and building responsive web pages by eliminating the need for tricky hacks and a plethora of float and position properties in your CSS code. Flexbox allows you to automatically scale elements (change their height or width) to fill the available space, shrink or grow elements to fit into the container, and prevent overflow. It can change the order of the items, create columns of the same height, design navigation panels, and much more.

Understand the Flexbox Layout Axis

The way items are laid out when you use flexbox is based on the flex directions. If the flex-direction is row which is the default value, your main axis is along the row, and your cross-axis is along the column. If the flex-direction is column, your main axis is along the column while your cross-axis is along the row.

The primary axis in which flex items are laid out in a flex container is usually referred to as the main axis. It is not necessarily horizontal because it depends on the flex-direction property. The cross-axis is perpendicular to the main axis. Its direction depends on the main-axis direction.

Flexbox properties

Flexbox is a module, meaning it comprises many properties that are either applied to the flexbox container or the flex items. Here is a table to show the list of elements that you can apply to either the flex container or items:

Flex container	Flex items
display	order
flex-direction	flex-grow
flex-wrap	flex-shrink
flex-flow	flex-basis
justify-content	flex
align-items	align-self
align-content
gap, row-gap, column-gap

How to make a flex container

The flex container is a container that holds all the flex items. For example, if you have a div that holds three other divs:

<div class="flex-container">
    <div class="flex-item">
        <p>Item One</p>
    </div>
    <div class="flex-item">
        <p>Item Two</p>
    </div>
    <div class="flex-item">
        <p>Item Three</p>
    </div>
</div>

To make the parent div a flex container, you would apply the display property and give it a value flex. This turns the container into a block element, allowing a flex layout for all of its direct children.

.flex-container {
    display: flex; 
}

You can also set the value to inline-flex, which works similarly. The only difference is that it creates a container as an inline element.

.flex-container {
    display: inline-flex; 
}

Note: An absolutely-positioned flex item does not participate in flex layout.

How to set flex-direction

You can set the direction of your flex items using the flex-direction property. You can set the items to flex vertically (column), horizontally (row), or even reverse the items in the specified direction using values like row-reverse and column-reverse.

.flex-container {
    display: flex;
    flex-direction: row | row-reverse | column | column-reverse;
}

The flex-direction property will only work on a flex container, meaning you must have used the display property and set the value to flex or inline-flex.

The default value of flex-direction is row which sets the flex items from left to right. In contrast, row-reverse does the opposite and sets flex items right to left. column will set flex items from top to bottom with items order similar to row, while column-reverse will reverse the flex items from bottom to top.

Wrapping Flex Items

When you create a flex container, all its items (flex items) will try to fit onto one line by default. When these items reach the max-width, they shrink to fit onto that single line.

You can change that and let the items wrap as needed with the flex-wrap property. This property has three values, but the wrapped value will help avoid your flex items from shrinking onto one line.

.flex-container {
    display: flex;
    flex-wrap: wrap;
}

There are two other values, nowrap and wrap-reverse. wrap will help you wrap your items onto multiple lines, from top to bottom, rather than compressing all flex items to one line. While the default value of nowrap causes all flex items to shrink onto one line, wrap-reverse causes the flex items to wrap across multiple lines from bottom to top.

So far, you have learned two significant properties: flex-direction and flex-wrap. If you need to use both properties, then you can use the flex-flow shorthand property.

.flex-container {
    display: flex;
    flex-flow: column wrap;
}

How to align items along the main axis

The justify-content property defines the horizontal alignment of flex items. This property has six major values that perform individual functions, all aimed at exerting some control over the alignment of flex items.

.flex-container {
    justify-content: flex-start | flex-end | center | space-between | space-around | space-evenly;
}

flex-start: This is the default value in which flex items are packed toward the start of the main axis or flex-direction.
flex-end: This is the opposite of flex-start, as flex items are packed toward the end of the main axis or flex-direction.
center: This perfectly fits the flex items within the flex container. Other options, such as start, end, left, and right, do what the value indicates. For example, when you set a value to right, the flex items will be packed toward the container's right edge.
space-between: This option evenly distributes flex items in the line starting and ending on the flex container border, except for padding or margin.
space-around: This option will evenly distribute flex items in line with equal space around them (left, right, bottom and top). It is important to know that between each element, the gap won't be the same as the top and bottom because this is the sum of the gap on the right and left of both elements.
space-evenly: This creates an exact space between each flex item, ensuring that the space between any two items and the edges is equal.

How to align items along the cross-axis

Asides from aligning items along the main axis, you can also do for the cross-axis using the align-items property. Just like the justify-content property, align-items also has about five values that can be used to align flex items.

.flex-container {
    display: flex;
    align-items: stretch | flex-start | flex-end | center | baseline;
}

stretch: By default, flex items are stretched to fill the flex container, still respecting min-width and max-width properties.
flex-start: This places all flex items at the start of the cross-axis based on the flex-direction. Other values similar to this are start and self-start.
flex-end: This is the opposite of flex-start, as flex items are placed at the end of the cross-axis. Other values similar to this are end and self-end.
center: This places flex items in the center along the cross-axis.
baseline: All flex items are aligned in the sense that their baselines are aligned.

align-items works with the one-line flex items, while you can use the align-content property to define the vertical alignment of multiple lines of items. This property has no effect when only one line of flex items exists.

Note: In multiple line flexible containers, the align-content property will not work unless flex-wrap is set to wrap or wrap-reverse.

.flex-container {
    display: flex;
    align-content: stretch | flex-start | flex-end | center | space-between | space-around;
}

The justify-content property is used to align individual items on the main axis, but you can also use align-content for the cross-axis when you want to space the items, move them to the start or end of the cross-axis, or stretch them.

flex-start: This can be used to move all your flex items to the start of the flex container based on the flex-direction. This is also similar to using start.
flex-end / end: This is the opposite of flex-start or start. The flex items are moved to the end of the flex container. This is also similar to using end.
center: All flex items are placed at the center of the flex container.
space-between: This will space all flex items evenly, starting and ending on the edge of the flex container.
space-around: This helps you evenly distribute equal space around each flex item. You can also make use of space-evenly for perfect spacing.
stretch: flex items are stretched to take up the entire space.

So far, you have learned how to align all flex items through the container element. But it is important to know that you can also align each item individually using the align-self property. The align-items property allows you to override the alignment specified for the flex container.

.flex-item {
    align-self: stretch | flex-start | flex-end | center | baseline;
}

The align-self property has the same values and meaning as align-items. Here's an example of what happens when you give each flex item its own align-self value.

This property is used when you need to add specific alignment to an individual flex item different from other flex items.

Gapping Flex items horizontally and vertically

At this point, one major aspect left for your flex container styling is adding gaps to your flex items. You will want to be able to add equal gaps between flex items either horizontally, vertically, or on both sides.

You can do this with the gap property. The gap property controls the amount of space between each flex item. The gap or spacing is only applied between the flex items and not on the outer edges. This property is for more than just flexbox, as it works in Grid and multi-column layouts.

.flex-container {
    display: flex;
    gap: 20px;
}

The gap property is a shorthand that you can use to specify one value for both the column and rows. When you want different values, you can use two values. The first is the row, while the second is for the column. Finally, you can use two properties: row-gap and column-gap.

.flex-container {
    display: flex;
    // ...
    gap: 20px;
    gap: 20px 30px; /* row-gap column gap */
    row-gap: 20px;
    column-gap: 30px;
}

Ordering Elements

You can use the order property to assign how flex items are ordered. You will use integer values to lay out these items from lowest to highest. When flex items have the same order, it will revert to the source order.

.flex-item {
    order: 5;
}

For example, if you have four items, you can specify how you want them to be ordered by giving them an order value:

flex item 1: order: 3
flex item 2: order: 1
flex item 3: order: 1
flex item 4: order: 2

The items above would be displayed in the following order, based on the value you set:

flex item 2: order: 1
flex item 3: order: 1
flex item 4: order: 2
flex item 1: order: 3

.item1 {
    order: 3;
}
.item2 {
    order: 1;
}
.item3 {
    order: 1;
}
.item4 {
    order: 2;
}

Sizing flex items - Grow or Shrink

Sizing flex items and making them flexible depends on three flex item properties: flex-basis, flex-grow, and flex-shrink.

flex-basis

This property specifies the length of flex items or individual flex items. This overrides any previously defined width for that element. This accepts length values (e.g., 10%, 4rem, etc.) and can also use auto, which uses the declared width value.

.flex-item {
    flex-basis: <length> | auto ; 
}

💡The flex-basis property has no effect if the element or item is not a flex item. This means that if it is not contained within a flex container.

The flex container in the image above is wrapped, which means that when it reaches the edge, the items will fall to the new line. The flex-basis property is then applied to the flex items and set to 30%, meaning each item will take up 30% of the total width, leaving 10%. You can use the flex-grow property to help the elements grow and cover up small spaces that cannot accept new flex items.

flex-grow

You can apply the flex-grow property to individual flex items or all flex items simultaneously. It defines a flex item's ability to grow. It ensures that flex items fill the available space inside the flex container by causing each item to grow following the supplied value. It has a default value of 0 and accepts a unitless value (e.g., 1, 2, 3 ...) as a proportion.

ℹ️ Negative numbers are invalid.

.flex-item {
    flex-grow: 1; 
}

If all of your flex items (perhaps with the same class name) have their flex-grow set to 1, the container's remaining space will be distributed evenly to all flex items.

If an item's flex-grow value is set to 2, it will gain twice the width of other items with flex-grow values of 1.

.item1 {
    flex-grow: 2;
}
.item2 {
    flex-grow: 1;
}
.item3 {
    flex-grow: 1;
}

This will work when the flex-basis is not set to any specific value.

flex-shrink

The flex-shrink property applies to flex items and specifies how much it will shrink each element if insufficient space exists. It has a default value of 1 and accepts a unitless value (e.g., 1, 2, 3) as a proportion.

Note: When set to 0, the flex item retains its original size. Negative numbers are invalid.

.flex-item {
    flex-shrink: 2;
}

When you set a specific width for each flex item, and the sum of the width plus the gap surpasses the flex container width, you can use flex-shrink to indicate how much each element will be reduced.

.item1 {
    flex-basis: 150px;
    flex-shrink: 1;
}
.item2 {
    flex-basis: 250px;
    flex-shrink: 2;
}
.item3 {
    flex-basis: 350px;
    flex-shrink: 3;
}

Note: flex-shrink won't work if the flex container has the flex-wrap property set to wrap.

The Flex shorthand for better syntax

Finally, you have learned that you can use these three properties to size your flex items. Instead of using three properties, you can combine its value using the flex property. It is a shorthand for flex-grow, flex-shrink, and flex-basis combined.

Note: flex-shrink, and flex-basis are optional.

.flex-item {
    flex: none | [ <'flex-grow'> <'flex-shrink'>? || <'flex-basis'> ];
}

The default value for flex property is 0 1 auto, but if you use a single number value:

.flex-item {
    flex: 2;
}

This changes the flex-basis to 0%, so it's like setting:

.flex-item {
    flex-grow: 2;
    flex-shrink: 1; 
    flex-basis: 0%;
}

It is always recommended that you use this shorthand property because it intelligently sets the other values.

When to use Flexbox over CSS Grid

So far, you've learned what flexbox is, what it does, how it works, and that it's a CSS layout system similar to Grid. Then you wonder, do I choose one, or is there a situation where you must use flexbox over Grid?

It would be best to use CSS flexbox when you have a small design to implement with a few rows or columns. Also, when you need to align elements, flexbox comes in handy.

You can certainly build your entire website or application using only flexbox and achieve the same results as if you were using CSS grid, which is perfectly fine. However, for a better CSS approach, you must write codes that are clean, maintainable, and easy for others to manage over time. CSS grid is the best way to create and fit your layout perfectly.

Note: The Flexbox layout is best suited for application components and small-scale layouts, whereas the Grid layout is best suited for larger scale layouts.

In this very in-depth guide can learn more about CSS Grid.

Wrapping up!

In this guide, you have learned what the CSS Flexbox is all about, how it works and the various elements that you can use to help you create an awesome layout for your small projects.

There are so many instances when you’d need to make use of the CSS Flexbox such as centering items, on your Navbar, basic horizontal layouts and lots more. Having a proper understanding of each CSS property and their values will help you.

Have fun coding!

Everything You Need to Know about All 11 CSS Filters

Abdulazeez Abdulazeez — Thu, 12 Jan 2023 19:06:35 +0000

CSS, the language for visually styling elements on the web, has evolved to include features that can change an element’s visual effects in a website’s source code – freeing you from the need for graphical and photo editing software. Enter CSS filters.

CSS filters allow developers to alter an HTML element (usually an image) appearance before it gets displayed in the browser. It provides various operations that can be performed on an element to change its visual properties — for example, increasing the brightness of an image, enhancing an element's color intensity, adding a blur effect, and much more.

There are 11 filters in CSS, which are:

blur
brightness
contrast
grayscale
hue-rotate
invert
opacity
saturate
sepia
drop-shadow
url

This guide will walk you through each CSS filter, what they are, how they work, and how you can use them in your code.

How Do CSS Filters Work?

We can apply CSS filters to an element by declaring the CSS filter property within the element's style block, passing one or more of the 11 CSS filters as its value.

Syntax:

element {
    filter: none | blur() | brightness() | contrast() | drop-shadow() | grayscale() | hue-rotate() | invert() | opacity() | saturate() | sepia() | url();
}

Notice that each filter has an opening and closing parenthesis after its name. That is because each filter is a CSS function that accepts either a length, percentage, or degree value, signifying how much of the filter should be applied to an element.

Now, let's look at how each CSS filter works in detail.

blur

.blurry {
    filter: blur( <length>);
}

The CSS blur filter applies a Gaussian blur visual effect to an element. It accepts a CSS length unit (px, rem, em, etc.) to determine how many pixels on the screen need to blend into each other to generate a blurred result. The larger the value of the CSS length unit passed to the filter, the more blur is applied to the element. If we do not provide a value, a default value of 0 is used.

For the blur filter (and all CSS filters covered in this article) we'll use the image below to demonstrate each filter's effect on an element.

The code below adds a blur effect of 3px to the image:

img {
    filter: blur(3px);
}

Here is the result:

brightness

.exposed {
    filter: brightness( <decimal> | <percentage>);
}

The brightness filter adjusts the brightness level of an image. The filter applies a linear multiplier to the input image, making it appear brighter or darker.

It accepts a decimal value between 0 and 1, or a percentage value. A value of 0 results in a completely black image. A value of 100% or 1 leaves the image as is with its original brightness level. Values greater or less than 100% or 1 determine how dark or bright the image will be.

Note that all filter functions that accept decimal values do not work with negative integers (-10, for example) but only accept positive numbers.

The brightness filter has a default percentage and numerical value of 100% and 1. Passing a percentage or numerical value lower than these numbers reduces the image brightness level, giving it the appearance of having a shadow cast over it. The lower the value the higher the intensity of the shadow cast over it is.

The code below applies the brightness filter with a percentage value of 35% to an image:

img {
    filter: brightness(35%);
}

The result:

The opposite effect occurs when you pass a percentage or numerical value greater than 100% and 1 to the filter function. It increases the image's brightness level to make it appear like a photo that has been over-exposed to too much light.

The code below applies the brightness filter with a value of 135% to an image.

img {
    filter: brightness(135%);
}

The result:

contrast

.sharpen {
    filter: contrast(<decimal> | <percentage>);
}

The contrast filter adjusts the brightness difference between the darkest and lightest parts of an image, giving the image a more or less enhanced look.

It accepts a percentage or decimal value to determine the contrast level of an image — a value of 0 results in a completely gray image. Values above 100% and 1 increase the contrast, and parameters below decrease the contrast of the image.

The code below adds a contrast level of 180% to an image:

img {
    filter: contrast(180%);       
}

The result:

grayscale

.black-and-white {
    filter: grayscale( <decimal> | <percentage>)
}

In digital photography, an image contains both color information and luminance (brightness) information. The grayscale filter removes all color information (a representation of the three light sources — red, green, and blue (RGB) used to produce colors on a computer screen) from an image. It leaves only the luminance information — a range of monochromic (gray) shades, with pure white as the lightest shade, pure black as the darkest shade, and every other shade in between being a shade of gray — known as the grayscale.

The grayscale filter removes all color information from an image and sets its color to a tone in the grayscale.

It accepts a decimal value between 0 and 1, or a percentage value up to 100%. A value of 100% or 1 results in a completely gray image. The image would remain unchanged with a value of 0% or 0, and values between 0% and 100% give it a tone between its original color and complete gray, making the image have a desaturated look.

The code below uses the grayscale filter to make an image completely gray:

img {
    filter: grayscale(100%);
}

The result:

Typical applications where the grayscale filter could come into use would
be applying it to images when dark mode is enabled to slightly reduce the overall vibrancy of colors, putting less eye strain on the user. You might also use it to make client logos gray when listing them on a website.

hue-rotate

.change-color { 
    filter: hue-rotate(<angle> | <zero>); 
}

Digital photography uses a concept known as the color wheel, an organization of colors around a circle, showing the relationships between the primary, secondary, and tertiary colors to produce colors on a computer screen.

Each color in the color wheel has a property called hue, the position of the color (in angle degrees) on the wheel.

The hue-rotate filter accepts a CSS angle value (deg, grad, rad, or turn) and shifts the hue value of each color present on an element relative to the amount you specify, updating all colors relative to their hue's starting point.

For example, red is at 0 degrees on the color wheel, and blue is at 240 degrees. If a portion of an image contains red, passing a value of 240deg to the hue-rotate filter will shift the hues of all the colors on the image by 240 degrees, turning that part of the image from red to blue.

The code below adds a hue rotation of 180 degrees to an image:

img {
    filter: hue-rotate(180deg)
}

The result:

Note, the unit identifier for the hue-rotate filter is optional for zero angles. For example, both rotate(0deg) and rotate(0) are valid, but hue-rotate(45) is not.

invert

.turn-color-negative { 
    filter: invert(<decimal> | <percentage>); 
}

Remember the color wheel mentioned in the section above? When applied to an element or image, the CSS invert filter flips the hue of every color present by 180 degrees, converting each color to the color directly opposite it on the color wheel.

Bright areas become dark, red becomes cyan, magenta turns green, blue becomes yellow, and vice-versa, giving the image a color-negative effect.

The invert filter accepts a decimal value between 0 and 1, or a percentage value up to 100% that controls the extent of the color-negative effect. A value of 0.5 or 50% turns an image completely gray.

The code below inverts the colors on an image by 80%:

img {
    filter: invert(80%);
}

The result:

opacity

.transparent {
    filter: opacity( <decimal> | <percentage>)
}

The opacity filter applies a transparency effect to an element's colors. It accepts a percentage or decimal value to decide how much transparency to apply to an image. An opacity of 0% or 0 will result in a completely transparent element. 100% opacity will show no transparency. Setting the opacity between 0% and 100% will give the element or image partial transparency.

The code below adds an opacity of 35% to an image:

img {
    filter: opacity(35%);
}

The result:

You might notice that the opacity filter works just like the CSS opacity property because both control how transparent an element should be. The difference and why you should prefer the opacity filter over the CSS opacity property is that CSS filters are hardware-accelerated.

When you use the opacity filter to add opacity to an element, the browser will offload the computing task onto the Graphics Processing Unit (GPU) — a specialized processor designed to accelerate graphics rendering within the system. This increases the browser's efficiency and frees up the CPU to perform other tasks.

saturate

.enhance {
    filter: saturate(<decimal> | <percentage>);
}

The CSS saturate filter enhances or decreases the intensity of a color, known as its saturation, making it appear more or less vivid.

It accepts a decimal value between 0 and 1, or a percentage value. A value of 100% or 1 keeps the color as is. A value above 100% or 1 amplifies the color's intensity, making it more vibrant (saturated), while values below that reduce the color's intensity, making the color grayed and dull (desaturated).

The code below applies a 172% saturation to an image:

img {
    filter: saturate(172%);
}

The result:

sepia

.nineties-effect {
    filter: sepia(<decimal> | <percentage>);
}

The sepia filter adds a soft brownish color tone to an image, giving it a warmer and vintage appearance that is calming to the eye. It is similar to using the grayscale filter but with a brownish color tone.

It accepts a decimal value between 0 and 1, or a percentage value up to 100%. A value of 0 leaves the image unchanged. A value of 100% or 1 changes the image entirely to sepia, and values between 0% and 100% give the image a tone between its original color and completely sepia.

The code below calls the sepia filter with a percentage value of 100%, converting the image to be completely sepia:

img {
    filter: sepia(100%);
}

The result:

drop-shadow

.shadow {
    filter: drop-shadow(<offset-x> <offset-y> <blur-radius> <color>);
}

The CSS drop-shadow filter applies an (often blurry) background shadow to an element, giving it a more three-dimensional look.

The drop-shadow filter accepts four parameters when applying a drop shadow to an element:

: A CSS length value that specifies the horizontal distance between the element and the drop shadow. Positive values place the shadow to the element's right, and negative values place the shadow to the left.
: A CSS length value that specifies the vertical distance between the element and the drop shadow. Positive values place the shadow below the element, and negative values place the shadow above it.
: The blur radius of the shadow is specified as a CSS length unit. The larger the value, the more blurred the shadow becomes. If left unspecified, it defaults to 0, producing a sharp and unblurred shadow. No negative values are permitted.
: The color of the shadow, specified as a CSS color. If left unspecified, it defaults to the color black.

Here's an example of setting a drop shadow to an element:

img {
    filter: drop-shadow(30px 30px 50px #000);
}

The code above adds a black shadow with a 50px blur to the image, setting it 30px to the right of the image and 30px below it.

Here's the result:

The drop-shadow filter looks much like the CSS box-shadow property at first glance. However, there's a difference between the two, and it's worth comparing those differences to know when to prefer one over the other.

One difference between the drop-shadow filter and the box-shadow property is the drop-shadow filter does not accept the inset keyword, which, when set, rather than aligning the shadow behind the element, inserts the shadow within the element, giving it a framed appearance.

For example:

div {
    height: 200px;
    width: 200px;
    background: #fff;
    border-radius: 20px;
    margin: 20px;
    box-shadow: inset 15px 12px 30px #000;
}

The result:

The result of using the insert keyword that's only available to the box-shadow property to insert the shadow into the div.

Another difference between the drop-shadow filter and the box-shadow property is the drop-shadow filter does not accept the parameter — a CSS length value that specifies how wide the shadow will span. Positive values cause the shadow to enlarge, and negative values cause the shadow to shrink. If not specified, it will default to a value of 0 (the shadow will be the same size as the element).

Example:

img {
    box-shadow: 30px 40px 50px 70px gray;
}

The code above adds a gray shadow with a 50px blur to an image, passing an additional length value of 70px that signifies the shadow will span 50px more than the image in width.

The result:

Lastly, to explain the significant difference between the drop-shadow filter and the box-shadow property, we'll be using the CoderPad logo below.

If the box-shadow property is used to add a shadow to the CoderPad logo above:

.coderpad {
    box-shadow: 10px 10px 30px #000; 
}

It results to the image below:

Notice that the shadow is applied to a rectangular box around the logo? This is because CSS uses a box model, where the edges of an element are bound in a rectangular box containing its padding, border, and margin properties. Even in cases where the element does not appear to have a rectangular shape, the box is still there, which is where the shadow will be applied.

CSS filters are not bound to the box model. That means that the drop-shadow filter will only consider the outline of the CoderPad logo, ignoring its box model so that only the contents of the logo receive the shadow.

For example:

.coderpad {
    filter: drop-shadow(15px 10px 28px #000);
}

The result:

Deciding which method to use between drop-shadow and box-shadow is totally up to you based on the effect you want to achieve.

url

.svg-filter { 
filter: url('#selector-of-your-svg-filter-here'); 

/* you can also load filters from external SVGs this way: */ 
filter: url('filters.svg#name-of-your-other-filter-here'); 
}

The url filter takes a string value pointing to an SVG filter element and applies its effects to an HTML element.

The SVG filter element <filter></filter> is used to add special visual effects to SVG graphics.

Syntax:

<svg> 
    <defs> 
        <filter id="name-your-filter-here">
             ... <!-- insert filters here --> ... 
        </filter> ... 
    </defs> 
</svg>

The <filter> tag takes a required id attribute to identify the filter. We can then use the url filter to apply the SVG filters to HTML elements using either the id selector of the filter or a link to it.

For example, the code below defines a blur filter using the
<feGaussianBlur> filter tag:

 <svg>
    <defs>
        <filter id="blur-filter" x="0" y="0">
          <feGaussianBlur in="SourceGraphic" stdDeviation="8" />
        </filter>
      </defs>
</svg>

Now, you can apply the SVG filter to an element by passing its id to the url filter:

img {
  filter: url("#blur-filter");
}

Here's the result.

Other Usage Notes

Here are a few more points to note when using CSS filters.

Using Multiple CSS Filters

Throughout this article, we have only looked at using a single CSS filter per section. However, you may combine two or more CSS filters to manipulate the rendering of an element.

Syntax:

.multiple-effects {
    filter: blur() | brightness() | contrast() | drop-shadow() | grayscale() |          hue-rotate() | invert() | opacity() | saturate() | sepia() | url();
}

We can pass two or more of the 11 CSS filters to the filter property, and CSS applies them to the element according to the order they appear.

For example:

img {
    filter: opacity(50%) drop-shadow(20px 10px 0px black);
}

The opacity filter comes first in the filter property in the code above before the drop-shadow. When rendering the image in the browser, CSS will apply an opacity of 50% to the image, making it partially transparent, before adding a drop-shadow.

Here's the result:

Reversing the order of the filter properties changes the result:

img {
    filter: drop-shadow(20px 10px 0px black) opacity(50%);
}

Usage with the initial and inherit keywords

In addition to the 11 CSS filters, the filter property accepts two extra values.
initial: The default value of the filter property — resolves to "none".
inherit: The computed value of the filter property from an element's direct parent.

Try using CSS filters yourself!

Hopefully, you now have a thorough understanding of all 11 CSS filters, how they work, and how you can use them to apply different visual effects to the elements on your websites.

CSS filters give you more independence from photo editing software when creating production assets and visual effects for your websites—helping save on both time and money. They enable you to apply these visual effects directly in the browser.

If you'd like to read more informative articles from our blog, check out the related posts below:

Victor is a frontend developer and technical writer interested in the JAMStack. and has a knack for breaking down developer concepts to understandable bits for others.

Publish Webhooks From Your FastAPI API With Convoy

Abdulazeez Abdulazeez — Mon, 21 Nov 2022 16:07:18 +0000

Webhooks are messages ( or payload ) sent from an application on the execution of an operation. They are also used to communicate between a chain of services; for example, a payment provider emits webhook events to an e-commerce application’s endpoint after an operation.

Convoy facilitates publishing webhook events from your application to your clients by serving as a reliable egress.

💡 Unlike traditional webhook servers, Convoy allows you to retry and replay your webhook events.

In this article, you will learn how to use Convoy’s Python SDK to publish webhooks events to multiple endpoints in your FastAPI application. You’ll start by setting up your Convoy instance on Convoy cloud ( or Convoy OSS ), create a todo API, and lastly, integrate the Convoy SDK to publish webhook events to endpoints.

Setup Convoy

To set up your Convoy instance, sign in to your dashboard and create a new project:

Create a new outgoing project from your dashboard. An outgoing project is a project for sending out webhook events.

A project API key will be generated for the newly created outgoing project. Store the key in a safe place as you’ll use it in the API to authenticate your Convoy instance to publish webhooks:

Configure your outgoing project. For this article, you’ll be configuring your outgoing project manually. Select the ***Setup Without SDK*** option to manually configure your outgoing project:

💡All actions performed manually in this article can be performed via our SDKs.

Create an application: application refers to your backend app with valid endpoints where webhooks are delivered to. An application can contain as many endpoints as necessary:

ℹ️ Applications are created per user to distinguish their webhook events.

Create an endpoint: an endpoint is a specific destination that can receive webhook events. The endpoint URL used in this article is generated from webhooks.site:

Create a subscription for the application: Subscriptions on Convoy connect events to their respective application endpoint.

You’re all set to publish events and monitor them from your dashboard:

With your Convoy instance up and running, you’ll build the API to publish your webhook events in the next section.

Project setup

The first step to publishing webhooks from your FastAPI-powered API is to build the API. Start by creating a new folder and creating a virtual environment:

$ mkdir fastapi-todo-convoy && cd fastapi-todo-convoy
$ python3 -m venv venv

ℹ️ You may use a different virtual environment manager like Pipenv or poetry.

API structure

The API folder structure is outlined:

fastapi-todo-convoy
 +- .env # store Convoy API key and application ID.
 +- api/
   +- __init__.py # module file
   |
   +- config.py # interact with the environment file
   | 
   +- api.py # todo API source file
   | 
   +- events.py # store for various event types for your webhook payload
 +- main.py # entrypoint to run the application
 +- requirements.txt # application requirements file

In the project folder, add the following files:

$ touch .env api/{config,api,events}.py main.py

Install dependencies

As a next step, activate the virtual environment and install the following dependencies:

$ source venv/bin/activate
$ pip install fastapi uvicorn pydantic[dotenv]
$ pip freeze > requirements.txt

Lastly, install the convoy-python package:

$ pip install git+ssh://git@github.com/frain-dev/convoy-python

Set environment variables

In the .env file, add the variables:

CONVOY_API_KEY=<your-api-key>
CONVOY_APP_ID=<your-app-id>

Replace <your-api-key> in the environment file above with the API key generated earlier on and retrieve your Convoy application ID from the dashboard:

To allow your API to retrieve the values stored in the environment file, define a BaseSettings child class in config.py:

from pydantic import BaseSettings
from typing import Optional

class Settings(BaseSettings):
    # convoy credentials
    CONVOY_API_KEY: Optional[str] = None
    CONVOY_APP_ID: Optional[str] = None

    class Config:
        env_file = ".env"

settings = Settings()

The Settings class defined above reads the variables defined in the environment file. The API can directly access the variables from the instance variable of the Settings class settings.

Define event types

The next step is to define the various event payload to be included in the webhook to be published from the APIs. In [events.py](http://events.py), add the code:

events = {
    "ping": {
        "event": "ping",
        "description": "Webhook test from application."
    },
    "created": {
        "event": "todo.created",
        "description": "Todo created successfully"
    },
    "retrieved": {
        "event": "todo.retrieved",
        "description": "Todo retrieved successfully"
    },
    "updated": {
        "event": "todo.updated",
        "description": "Todo updated successfully"
    },
    "deleted": {
        "event": "todo.deleted",
        "description": "Todo deleted successfully"
    },
    "failed": {
        "event": "todo.failure",
        "description": "Todo not found."
    }
}

In the code block above, you have six event types all prefixed with todo. accompanied by a description of the event.

Build the API

The API performs a Create, Read, Update and Delete ( CRUD ) operation on todos stored in an in-app database, an array variable todos.

In api.py, import the dependencies and create an instance of FastAPI:

from convoy import Convoy
from fastapi import FastAPI

from .events import events
from .config import settings

app = FastAPI()

Next, create the todos variable and an instance of Convoy:

todos = []

convoy = Convoy({"api_key": settings.CONVOY_API_KEY})
app_id = settings.CONVOY_APP_ID

In the code block above, you connect to your Convoy instance using the API key stored in the environment file using the convoy-python SDK installed. The app_id variable is also set to the value stored in the environment file.

Next, you’ll define a function send_webhook_event that takes an argument event_type and uses the convoy.event.create() method to publish events to your application and by extension, your application endpoints.

def send_webhook_event(event_type: str):
    event = {
        "app_id": app_id,
        "event_type": event_type,
        "data": events[event_type]
    }

    (res, err) = convoy.event.create({}, event)
    return res

The convoy.event.create() method takes an optional query and an event payload body. The event payload body is an object containing the application ID, the event type and payload. In your API, the event type and event payload have been defined in events.py.

With the webhook publisher function in place, add the code below to complete your API:

@app.get("/")
async def ping():
    send_webhook_event("ping")
    return {"message": "Wilkomen!"}

@app.get("/todo", tags=["todos"])
async def get_todos() -> dict:
    send_webhook_event("retrieved")
    return { "data": todos }

@app.post("/todo", tags=["todos"])
async def add_todo(todo: dict) -> dict:
    todos.append(todo)
    send_webhook_event("created")
    return {
        "data": { "Todo added." }
    }

@app.put("/todo/{id}", tags=["todos"])
async def update_todo(id: int, body: dict) -> dict:
    for todo in todos:
        if int(todo["id"]) == id:
            todo["item"] = body["item"]
            send_webhook_event("updated")
            return {
                "data": f"Todo with id {id} has been updated."
            }
    send_webhook_event("failed")
    return {
        "data": f"Todo with id {id} not found."
    }

@app.delete("/todo/{id}", tags=["todos"])
async def delete_todo(id: int) -> dict:
    for todo in todos:
        if int(todo["id"]) == id:
            todos.remove(todo)
            send_webhook_event("deleted")
            return {
                "data": f"Todo with id {id} has been removed."
            }

    send_webhook_event("failed")
    return {
        "data": f"Todo with id {id} not found."
    }

You have defined the routes to perform various CRUD operations in the code block above. Each successful operation emits a webhook to your Convoy application, and a failed operation emits a failed webhook event.

Lastly, in the API’s entry point file main.py, add the following:

import uvicorn

if __name__ == "__main__":
    uvicorn.run('api.api:app', host="0.0.0.0", port=8080, reload=True)

The code above starts the API on port 8080 on the localhost. Start your application with the command:

$ python main.py

Publishing webhooks

It’s time to publish your first webhook! To publish your first webhook event, send a cURL request to the API:

$ curl http://0.0.0.0:8080/

The cURL request returns a response:

`{"message":"Wilkomen!"}`

In your Convoy dashboard, the event is delivered and logged:

In the Event Deliveries tab, you can verify the status of the delivery as well as view the number of attempts:

If you click on the event, a detailed delivery page is displayed:

Let’s create a todo using a cURL request and verify if your API will publish a webhook event:

$ curl -X 'POST' \
  'http://0.0.0.0:8080/todo' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "id": 1,
  "item": "Create a Convoy outgoing project today"
}'

The API returns a successful response:

{
  "data": [
    "Todo added."
  ]
}

Was the webhook for a create action emitted? Verify from your dashboard:

You can verify that the API webhook successfully emitted the event type created from the dashboard. A detailed view of the event delivery can is seen from the event delivery tab:

A visit to the endpoint URL strengthens the webhook delivery status from the Convoy dashboard:

Go on, execute more operations on the todo API and publish webhooks today on Convoy!

Conclusion

Convoy provides the ability to publish webhooks to various endpoints in one request. In this article, you learned how to create and configure a Convoy outgoing project and publish webhooks from an API you built to your configured endpoint.

Convoy provides you with reliability and replayability out of the box. If this sounds suitable for your architecture,try it outand give us feedback on ourslackcommunity!

The Environmental Impact of Digital Technology and How Software Eco-Design Can Help

Abdulazeez Abdulazeez — Fri, 04 Nov 2022 11:26:00 +0000

NOTE: This article was originally published on the CoderPad blog.

We often have the impression that digital technology does not pollute.

This is usually because software pollution is mainly invisible to the general public thanks to dematerialization (the use of the cloud or online word processing services, for example), which is ultimately only the decentralized materialization of digital services.

However, when we use these digital services, we utilize the 3-tier hardware architecture composed of:

Devices or terminals: computers, screens, smartphones, smartwatches, etc.
Networks: all the cables that allow the circulation of data, routers, relay antennas, etc.
Servers: including data centers.
Of all the sectors – transportation, textile industry, agriculture, etc.– the environmental footprint of the digital industry is the one that is growing the fastest on a global scale, notably linked to :
4% of greenhouse gases (GHG) in 2020 1.
10% of the world’s electricity usage 2
A 9% annual increase in electricity consumption 2.
The increasing flow of electronic waste – growing at 4% per year and per person 3
Increasing data traffic (25% growth per year), leading to growing development of network and server infrastructures 5
An economic sector that is expected to double its GHG emissions by 2025 compared to 2020 2.

To further show the environmental impact of technology, it’s been shown that a 2G electronic chip found in your phone, printer, laptop, tablet, etc., requires 32kg of natural resources to produce. More and more of these chips (and others) are needed to power our electronic devices each year 4.

According to a study conducted by ARCEP and ADEME in January 2022 6, the carbon footprint of the digital sector in France would be 17 Mt CO2eq (mega tonne CO2 equivalent), i.e., 2.5% of the national footprint, distributed as follows:

79% for terminals
16% for data centers
5% for networks

Understanding the magnitude of the problem is essential to realizing the necessity of levers to reduce the environmental impact of digital technology.

Apart from environmental issues, digital technology poses other problems, particularly social ones. The disparity in access to digital tools is known as the “digital divide.”

To put numbers on the digital divide: 3.7 billion people do not have access to a broadband network, and 87% of people in developed countries have access to the Internet (compared to 19% in developing countries) [7][https://www.itu.int/dms_pub/itu-s/opb/pol/S-POL-BROADBAND.23-2021-PDF-E.pdf].

This is generally due to lacking infrastructure, which implies a lack of access to information, education, communication, etc.

These social problems are also exacerbated by the mineral resource requirements that feed our digital economy.

The mining for these minerals tends to be in countries where geopolitical tensions exploit local populations in disastrous conditions.

For example, 60% of cobalt mining is carried out by children in the Democratic Republic of the Congo 8. It also pollutes the clean surrounding water, rendering it undrinkable for anyone living near the mine 9.

The need to reduce digital technologies’ impact on the environment and humanity cannot be understated. This article will explore ways we can do just that through intelligent eco-design. We’ll also explore the benefits that it can bring to a business and its customers.

Fundamentals of software eco-design

Before we discuss implementing measures to reduce the impact of digital technology, we first need to go over some basic eco-design concepts.

Digital services

A digital service meets a specific need, has one or more functionalities, and a set of users. It is considered a combination of the following:

Equipment that enables data to be stored, manipulated, and displayed: servers, terminals, boxes, etc.
Infrastructures that host and connect the equipment: networks, data centers, etc.
Several software packages stacked on top of each other that run on the different devices.
Other possible third-party digital services.

Eco-design

The principle of eco-design is to consider the system’s environmental impact from the design phase, with the aim of reducing this impact throughout the product’s lifecycle.

The eco-design of a digital service necessarily takes into account the following points:

The definition of the functional scope: which features add value, avoid useless widgets, which data and processing are beneficial, etc.
All stages of the lifecycle and associated equipment: manufacturing, installation, distribution, use, and end-of-life (recycling, reuse, dismantling, etc.)
Several environmental indicators: resource depletion, pollutant emissions, water consumption, waste production, emission of substances contributing to climate change, etc.
The three tiers of the architecture: terminals, networks, and data centers
The implementation of a dialogue with stakeholders Progressive and continuous environmental improvement without transferring pollution or impact

In the digital domain, as we have seen previously, the primary source of impact is located at the terminal level – essentially in the manufacturing phase–representing about 80% of the impact.

It is therefore essential to optimize and size digital services as much as possible to increase the longevity of terminals. This is to avoid their renewal and to make infrastructures more profitable in terms of space, power, and consumption of resources in general (the weight of software has been multiplied by 171 in 20 years, according to the Green IT collective 10).

Levers of optimization

Optimization levers are those aspects of a digital service that we can change to reduce or contain the service’s environmental footprint. They include:

The definition of the functional scope: which features add value, avoid useless widgets, which data and processing are beneficial, etc.
The usage, sizing, and solicitation of the associated physical equipment.
The necessary equipment and infrastructure and their sizing.
The software layer and its optimization.
The use of software best practices and the use of measurement tools (which we will explore later).
Communication with stakeholders to better define functionalities and needs via the UX approach (for example).

ℹ️ No product is completely “green” or “clean.” We can reduce the impact of our products and activities, but that doesn’t make them suitable for the planet. It is mainly a question of making our tools more efficient (less consumption for the same use) and questioning our services to keep only the essential and remove the nonessential. The notion of eliminating the unnecessary is not a substitute for efficiency and vice versa.

Of course, the objective is not necessarily to deprive ourselves or to cut back on certain uses, which is up to each individual. The aim is to adapt our consumption to our actual needs.

Impact transfer

Impact transfer is when impact improvements are made at one stage of the life cycle, implying adverse effects at another phase of the life cycle.

For example, 5G has a lower carbon footprint (between 7 and 35 kWh/user/year) on paper than 4G (50 kWh/user/year) 11, but requires the installation of more antennas than 4G and allows more devices to be connected, thus leading to a rebound effect (Jevons paradox) by indirectly increasing the number of connected devices.

As humans, we have a cognitive bias that pushes us to use the resources offered by a system to the maximum. In this case, that puts additional pressure on data networks by increasing the data flows, which in turn requires more infrastructure to manage these flows, and so on, in a vicious cycle.

In an eco-design approach, it is necessary to conduct a life cycle analysis to consider the notions of impact transfer and rebound effect to make the best possible design choices.

Reasons to choose eco-design

There are many different reasons to start an eco-design approach:

It eliminates nonessential elements, making services more efficient
Simplifies the UX and makes it more fluid, improving accessibility
Saves time and money by reducing costs (energy, consumables, cost to society, etc.)
Encourages companies to develop a CSR (Corporate Social Responsibility) strategy
Improves brand image
Brings more meaning to an employee’s work, therefore reducing burn-out and improving well-being at work
Develops a longer-term vision for the products
Attracts new clientele, thus increasing sales
Creates a network of partners with the same desire to reduce their environmental impact and thus influence others to adopt a CSR strategy
Attracts potential investors (especially related to the establishment of a CSR strategy)

These reasons aren’t just musings of a philosopher – real numbers back them. According to the 2014 study presented by France’s Agence de la Transition Écologique (ADEME) 12, an eco-design approach:

Increased margins of eco-designed products by an average of 12%
Improved at least two functional aspects of the studied products (ease of use, cheaper, more sustainable, etc.)
May improve profitability. Some indicators of the intensity of the eco-design approach are associated with higher profits. Thus, the more the company takes into account a large number of stages of the product life cycle in its eco-design approach, the more monetary gain it may see
Increased employee motivation (58% in the EU)
Easier to recruit employees
Improved innovation capacity

According to the 2020 GreenConcept study 13, it would be possible to drastically reduce the environmental impact of digital services: 85% less energy consumption, 63% less water needed, 40% fewer material resources, and 58% less greenhouse gas emissions.

Case-in-point: LinkedIn and IBM have managed to reduce server processing time by a factor of 100 by eco-designing their digital services 14.

Finally, eco-design allows you to identify new axes of value creation for your products and offers a triple benefit to your company:

Increased assets by increasing the value of the company through improved brand image and product design
Enhanced customer relations by responding better to their expectations and offering more innovative products
Improved ethical standing by adding social responsibility as a company value (i.e., CSR)

Reminder: CSR is not based on “good deeds” or “philanthropy” but aims to transform companies’ core business models and processes.

Putting software eco-design into practice

You will see that from a technical point of view, eco-design only requires the common sense of the developer. The advantage is that we already have all the tools.

Hosting

The choice of your web host can be important in reducing your environmental footprint. Indeed, behind each hosting option, many servers need to be powered 24/7/365.

According to Science magazine 15, web hosting servers alone represent 1% of the world’s electricity consumption. This may not seem like much, but also consider the following:

10% of the world’s electricity is used for ICT (Information and Communication Technologies) 2.
Electricity usage for digital technology is increasing by 9% every year 2.
The digital services industry is expected to double its greenhouse gas emissions between 2020 and 2025! 2

A significant portion of this energy expenditure is the cooling needed to ensure that servers function optimally, protect them from failure, and thus ensure their longevity. Cooling consumes up to 40% of the total energy required to run a data center 16.

Calculating the exact environmental footprint of a server is very difficult, given the variety of criteria to be taken into account, the diversity of methodologies used to calculate it, and the lack of publicly available data 17.

We can nevertheless estimate that manufacturing a server emits about 1 t CO2eq. This doesn’t account for abiotic resources (non-living resources: air, water, minerals, etc.) and any environmental destruction that results from obtaining those resources.

What can be done about it?

There is a lot you can do to ensure your hosting service has minimal impact on the environment:

Use a “local” hosting company for your end users to limit the distance traveled by the data (15,000 km on average, according to ADEME).
Choose a hosting company that communicates its environmental impact and makes commitments like OVH (France) or Infomaniak (Switzerland).
Check the PUE* of your host. The closer it is to 1, the better (the European average is 1.46). OVH has a PUE between 1.1 and 1.3 (variable according to the sites); Infomaniak’s is slightly lower than 1.1. These two hosts have an excellent PUE.
Prefer hosting providers with passive cooling systems to limit energy usage.
Prefer hosting companies that use low-carbon electricity, but beware of “green” energy certificates (you can buy a “green” energy certificate).
Find out about your host’s hardware renewal practices (ex: Do they use reconditioned equipment, components, etc.?).
Utilize shared server offerings to avoid having a dedicated server. This way, you reduce the environmental footprint of your hosting, which is divided among the different users of this shared offer.
Eco-design your applications. This allows you to take up less space and resources on the servers. *PUE: for Power Usage Effectiveness, used to measure the energy efficiency of a data center, it is calculated by dividing the energy consumed by the data center by the energy consumed by the equipment (servers, air conditioning, network, etc.)

Use benchmarks

An excellent way to get started in software eco-design is using best practice frameworks, which save you time in finding ways to optimize your code or the choice of hosting.

In France, for example, we have many frameworks that have emerged:

The GR491 by the Institute of Responsible Digital (also exists in English): https://gr491.isit-europe.org/en/
The guide to eco-design of digital services by the Association of Ethical Designers (also in English): https://eco-conception.designersethiques.org/guide/en/
Ecometer, co-financed by ADEME (in English): http://www.ecometer.org/rules/
The French government’s Référentiel Général d’Écoconception de Services Numériques (RGESN) also has a general reference framework available (only available in French): https://ecoresponsable.numerique.gouv.fr/publications/referentiel-general-ecoconception/

Some of these best practice frameworks will only address the code aspect; others will also discuss hosting management and UX. Still, overall they present many of the same eco-design principles.

Measurement tools

As the saying goes: you can’t improve what you don’t measure. Indeed, how can we know the critical optimization levers when we have no metrics?

Fortunately for us, many measurement tools already exist. The list I’m going to give you is not exhaustive, but are the ones I use most often:

Bundlephobia to check the size of the NPM dependencies you want to add to your project, and their download time
PageSpeed Insight for the performance of your sites. It also lists the points to correct to improve the performance of your site
GTmetrix, like PageSpeed Insight, uses Google LightHouse metrics for performance indices but uses a non-emulated browser, unlike PageSpeed
Website Carbon Calculator to measure the carbon footprint of your site
GreenIT-Analysis (currently available only for Chromium browsers) works with an “EcoIndex” calculated from different parameters such as the number of HTTP requests launched, the resource compression rate, image optimization, etc. (this topic will be discussed in the next section)

All these tools are free, don’t hesitate to use them!

Some technical recommendations for software eco-design

Eco-design requires taking into account physical environmental constraints. This is primarily done in the project’s design phase by considering the user experience (UX).

UX and eco-design also require optimizing interfaces and taking into account accessibility (which also takes into account the loading time of your application on a 2G mobile phone, for example) and requesting as little data as possible. It is also necessary to eliminate superficial functionalities to avoid unnecessary processing.

Technologies are also chosen during the design phase. The 2017 study Energy Efficiency across Programming Languages (R. Pereira et al. 18) studies energy, time, and memory consumption according to the languages. Here is a summary table:

![table]https://d2h1bfu6zrdxog.cloudfront.net/wp-content/uploads/2022/10/table.png

The 2017 study Energy Efficiency across Programming Languages (R. Pereira et al. [18]) studies energy, time, and memory consumption according to the languages.

We can note that some languages are much less greedy than others in terms of resources, but this does not mean we should systematically turn to them.

Indeed, the choice of a language is not only based on its raw performance but requires the study of other parameters such as :

Maintainability: will someone be able to take over the project after it’s complete? Does the job market have competent developers in this language? What is the technical environment in which this project will run? Etc.
Language competencies of the project team: let’s say that Node.js is more adapted to an upcoming API project. If the rest of the team is only competent in PHP, it might be better to stay with PHP or opt for a PHP framework (Symfony with API Platform, for example) even though Node may be the “better” language.
Appropriateness to the project: Some languages are better than others at dealing with a particular task. For example, Python is more adapted than PHP for a Machine Learning project. More importantly, how technology is implemented can impact the environment more than how it is used in isolation.

There are a lot of simple tips you can use – many of them found in the references I’ve sprinkled throughout this article – to help you minimize the impact of your digital service implementations:

Reduce the size of the images, and avoid using them when they are unnecessary.
Avoid videos as much as possible or reduce their quality, and do not activate automatic playback when possible.
Think mobile-first. This approach avoids loading content for desktop browsers when unnecessary (because we tend to put more elements on desktop than mobile).

Moreover, a smartphone (which accounts for 60% of web browsing 12) has a power consumption of between 2 and 15W for charging, while a laptop has a power consumption of between 20 and 75W, and a desktop is between 70 and 250W 13.

Utilize for loops (with ++i) rather than forEach loops. While it ultimately depends on the context and readability constraints, it is better to opt for a for each loop if the code becomes too complicated to read or understand. Code that is difficult to understand is code that is difficult to maintain and evolve, which can contribute to technical debt, contrary to the principles of eco-design.

ℹ️ You can learn more about for and forEach loops in these blog posts about the JavaScript some() method and JavaScript array methods.

Reduce the size of the DOM as much as possible since it is cumbersome to handle. In web applications, avoid using JavaScript as much as possible, especially for animations – use CSS instead if you can.

ℹ️ Read more about CSS filters, CSS grid, CSS math functions, and CSS pseudo-classes.

Limit the number of HTTP requests and optimize/group them as much as possible
Limit the number of SQL JOIN operations and sub-queries. These are very greedy in terms of server resources and response time. Of course, this is often dependent on the architecture of your database.

ℹ️ How to Optimizie Query Performance in MySQL Databases.
Serve static pages rather than dynamic ones.
Use caching as much as possible, especially for assets (images, CSS, JS), and define a long enough caching time according to usage.
Enable server-side file compression (Gzip, minification).
Use tree shaking, available, for example, with Gulp or Webpack.
Avoid using non-standard fonts, or if you must use them, try to choose the lightest fonts possible, and use readable fonts for people with reading disorders (accessibility).
Use lazy loading.
Avoid SELECT * in a database. Specify the names of the columns to return instead.
Ask for as little data as possible to make the program work. Not only is this an eco-design concept, but it’s a privacy/data protection one as well. For example: do we need the user’s first AND last name?
For Content Management Systems (CMS), deactivate and uninstall useless modules.
Optimize your application’s algorithms as much as possible by evaluating their complexity to reduce them.

✅ If you couldn’t tell, eco-design is also compatible with the Agile methodology! By really focusing on what brings value to the customer and evaluating useful enhancements, agile development can help reduce the energy overhead associated with developing unnecessary product features.

Eco-design is the future of software development

Eco-design is nothing more and nothing less than the global application of good programming and design practices that allow you to improve the performance of your web applications, all the while keeping in mind that our world obeys physical laws with limited resources that are not necessarily renewable on a human life scale.

By applying these good practices, your company will see its image enhanced with your customers, potential investors, and – more importantly – your employees. The more your employees are trained in eco-design principles, the higher the intensity of implementing eco-design in products, the more quality will be present, the more your customers will be satisfied, and the higher your profits will be.

In addition, by eco-designing digital services, we can reduce the amount of processing required and extend devices’ lifespan, thus reducing our impact on the planet’s resources and the energy needed to operate them.

Nevertheless, it is essential to keep in mind that despite the positive indicators that emerge from various studies on eco-design, they may be biased, especially due to the relatively small study samples. For example, the 2014 study 14 is based on a sample of about 100 companies across the EU and Canada. Moreover, the concept of eco-design is relatively recent.

Adopting eco-design at scale cannot be done in a snap of the fingers. Even if it is not a magic formula to boost your business, it is of undeniable interest in your digital services’ environmental impact, user experience, and performance.

Here I’m going to risk paraphrasing Jean-Marc Jancovici 15, but a world of service jobs is not a world without hardware – quite the contrary. Digital service jobs depend on the presence of machines because there are a lot of flows to manage (notably of information and people), and the more the flows increase, the more people are needed to deal with them (thus more machines, and so on).

Thanks for reading!

This article was written by Thomas Laure, a French junior IT engineer who loves solving problems regardless of the technology used, not only in tech, and passionate about many scientific topics and philosophy.

Let's implement Stripe webhook signatures in Golang

Abdulazeez Abdulazeez — Fri, 04 Nov 2022 10:26:00 +0000

NOTE: This article originally appeared on the GetConvoy Blog

Building a webhook publishing infrastructure requires providing a way to validate the message’s integrity to enable consumers to validate the webhook event origin. Generating webhook signatures would require us to implement certain important features which include:

Prevent Replay Attacks
Forward Compatibility.
Zero Downtime Key Rotation.

More on each item later on. But these properties exist in Stripe’s webhook signature implementation, see below:

Stripe-Signature:
t=1492774577,
v1=5257a869e7ecebeda32affa62cdca3fa51cad7e77a0e56ff536d0ce8e108d8bd,
v1=5257a869e7ecebeda32affa62cdca3fa51cad7e77a0e56ff536d0ce8e108d8bd,
v0=6ffbb59b2300aae63f272406069a9788598b792a944a07aba816edb039989a39

In this article, we would replicate this implementation in Golang, with one additional requirement; We want our implementation to be backward compatible with the common implementation, like the below:

Stripe-Signature: 
5257a869e7ecebeda32affa62cdca3fa51cad7e77a0e56ff536d0ce8e108d8bd

This backward compatibility allows new API consumers to opt-in to this new system at their choosing. For the purpose of this article, we define the latter signature specification as simple signatures and the former as advanced signatures. Implementing simple signatures is pretty trivial and commonplace, but advanced signatures aren’t common.

We start by breaking down the requirements to discuss advanced signatures even further.

Prevent Replay Attacks

Stripe-Signature:
t=1492774577,
v1=5257a869e7ecebeda32affa62cdca3fa51cad7e77a0e56ff536d0ce8e108d8bd,
v1=5257a869e7ecebeda32affa62cdca3fa51cad7e77a0e56ff536d0ce8e108d8bd,
v0=6ffbb59b2300aae63f272406069a9788598b792a944a07aba816edb039989a39

A replay attack occurs when an attacker intercepts a valid payload, and its signature, then re-transmits them. The aim is to exploit unsuspecting webhook consumers to perform an action multiple times. Idempotent keys aren’t sufficient against attacks like this because, webhooks are transient data, assuming consumers purge their webhook log after a certain period, it means re-transmitting purged webhook events all of sudden becomes valid.

To mitigate against this, we generate a timestamp and include it in the signed payload, so it is verified alongside the signature, so the attacker cannot change the timestamp without invalidating the signature. This ensures that events after a given threshold are regarded invalid.

When retrying events each delivery attempt should re-generate the timestamp, this ensures the timestamp is fresh.

Forward Compatibility

Stripe-Signature:
t=1492774577,
v1=5257a869e7ecebeda32affa62cdca3fa51cad7e77a0e56ff536d0ce8e108d8bd,
v1=5257a869e7ecebeda32affa62cdca3fa51cad7e77a0e56ff536d0ce8e108d8bd,
v0=6ffbb59b2300aae63f272406069a9788598b792a944a07aba816edb039989a39

Webhooks implementation evolves over time, providers can determine whether to switch from hex to base64 for encoding or change the hash function or the template of the payload being signed. To enable smooth upgrades for consumers, we version signatures in the example above. This allows consumers to verify against at least just one signature and migrate to the newest version at their convenience.

Zero Downtime Key Rotation

Stripe-Signature:
t=1492774577,
v1=xdz+2j9aMVQUUjSy0KUz/CsjD4jaD6wHJGGf1c3eZzrWxHTf1cAjZ3aL07O9NZXMhg5gajfi+TYuBU1aoU18xA==,
v1=5257a869e7ecebeda32affa62cdca3fa51cad7e77a0e56ff536d0ce8e108d8bd,
v0=cvt+CsjD4jaD6wHJGGf1/2j9aMVQUUjSy0KUzc3eZzrWxHTf1cTYuBU1aoU18xAAjZ3aL07O9+NZXMhg5gajfi==
v0=df51a848684dac3901d2b8bd17e5c8d2d971b15c544fa923493232df1fe0fbad

Webhooks rely on a shared secret that needs to be rotated periodically to be kept safe. Building effective key rotation mechanisms is a major implementation of great webhooks implementations. In the above sample, you can see v1 and v0 appear twice this means, we used two secrets to generate two different schemes.

Core Implementation

The core implementation goes thus:

type Scheme struct {
    // Secret represents a list of active secrets used for
    // a scheme. It is used to implement rolled secrets.
    // Its order is irrelevant.
    Secret []string

    Hash     string
    Encoding string
}

type Signature struct {
    Payload json.RawMessage

    // The order of these Schemes is a core part of this API.
    // We use the index as the version number. That is:
    // Index 1 = v1, Index 2 = v2
    Schemes []Scheme

    // This flag allows for backward-compatible implementation
    // of this type. You're either generating a simple header
    // or a complex header.
    Advanced bool

    // This function is used to generate a timestamp for signing
    // your payload. It was only added to aid testing.
    generateTimestampFn func() string
}

func (s *Signature) ComputeHeaderValue() (string, error) {
    // Encode Payload
    tBuf, err := s.encodePayload()
    if err != nil {
        return "", err
    }

    // Generate Simple Signatures
    if !s.Advanced {
        sch := s.Schemes[len(s.Schemes)-1]
        sec := sch.Secret[len(sch.Secret)-1]

        sig, err := s.generateSignature(sch, sec, tBuf)
        if err != nil {
            return "", err
        }

        return sig, nil
    }

    // Generate Advanced Signatures
    var signedPayload strings.Builder
    var hStr strings.Builder
    var ts string

    // Add timestamp.
    if s.generateTimestampFn != nil {
        ts = s.generateTimestampFn()
    } else {
        ts = fmt.Sprintf("%d", time.Now().Unix())
    }

    // Generate Payload
    signedPayload.WriteString(ts)
    signedPayload.WriteString(",")
    signedPayload.WriteString(string(tBuf))

    // Generate Header
    tPrefix := fmt.Sprintf("t=%s", ts)
    hStr.WriteString(tPrefix)

    for k, sch := range s.Schemes {
        v := fmt.Sprintf(",v%d=", k+1)

        var hSig string
        for _, sec := range sch.Secret {
            sig, err := s.generateSignature(sch, sec, []byte(signedPayload.String()))
            if err != nil {
                return "", err
            }

            hSig = fmt.Sprintf("%s%s", v, sig)
            hStr.WriteString(hSig)
        }
    }

    return hStr.String(), nil
}

Let’s break down the above code listing:

We use the Advanced flag to determine what type of signature to generate.
We use the Scheme type to encapsulate all versions, and the order in which they’re passed in determines their version. We map to index 0 to v1 etc similar to how we don’t define API versioning as /api/v0.
ComputeHeaderValue will either generate a simple signature string or an advanced signature string based on the Advanced flag.

Other aspects of this library were removed for brevity, you can find the full code here.

SDK

To enable easy migration, we’ve added webhook verification logic to our Ruby, Python & Golang SDKs to parse and validate this format. This verification will automatically identify either simple or advanced signatures and validate them respectively.

Separate API Keys from Webhook Secrets

Another useful benefit of Advanced Signatures is we can stop using our API Keys as webhooks secrets with zero downtime. This is good because compromised webhook secrets do not equal compromised API Keys and vice-versa. This is also known as the principle of least privilege. To achieve this do the following:

Create endpoints, and set the webhook secret as the API Key.
Update your apps to verify Advanced Signatures.
Roll over the current webhook secret with an expiry time.
Set the new webhook secret in your apps. 🎉

Conclusion

In this article, we showed how to implement Stripe-Like webhooks and build them in a backward-compatible fashion with your current implementation. We shipped this feature to Convoy OSS and Cloud. You can sign up here to get started!

Developing and Deploying a Helm Chart Application in Okteto

Abdulazeez Abdulazeez — Mon, 07 Feb 2022 18:17:43 +0000

Deploying applications in Kubernetes can be complicated. Even the simplest application could require creating a series of interdependent components (e.g., namespaces, RBAC rules, ingress, services, deployments, pods, secrets, etc.), each with one or more YAML manifests.

Helm is the de-facto package manager for Kubernetes applications that allows developers and operators to easily package, configure, and deploy applications onto Kubernetes clusters. If you're building an application that will run in Kubernetes, you should really look into leveraging Helm.

In this article, we'll show you how you can deploy a Helm chart powered application to Okteto, a Kubernetes powered service. We'll also show you how to push your application images to the Okteto container registry, create and configure a helm chart, and then develop the deployed application using Okteto's remote development environment.

If you're a beginner to Kubernetes, check out our Kubernetes For Developers series aimed at beginners.

Prerequisites

In order to follow this tutorial, you should have:

An Okteto Cloud Account (It's free!)
Some familiarity with Kubernetes

Install Helm

We'll be needing the Helm command line tool to create and deploy our application.

For MacOS

$ brew install helm

For Linux

Debian/Ubuntu

$ curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
$ sudo apt-get install apt-transport-https --yes
$ echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
$ sudo apt-get update
$ sudo apt-get install helm

For Windows

$ choco install kubernetes-helm

Building a helm powered application

In this section, we'll create a helm chart to enable us to deploy an already built application. Unlike a docker-compose powered application, helm chart powered applications can not be deployed using the okteto stack deploy command. Helm powered applications also depend on prebuilt container images, so we'll have to build and push our images to a registry too.

Clone the application

In your terminal, clone the GitHub repository and make it the working directory:

$ git clone -b default https://github.com/okteto/recipe-app
$ cd recipe-app

In the folder, we have an application prebuilt alongside a Dockerfile. As it stands, there is no manifest readily available for deploying the application to Kubernetes. We are going to create a helm chart for the application but before then, let's build the image and push it to Okteto's build registry.

Building the application image

We will be building the image for this application from the Dockerfile to Okteto's container registry. Before building the image, set the Okteto context:

$ okteto context

Next, run the command to build the application:

$ okteto build -t okteto.dev/recipe-app:latest

The command above builds and pushes the application's image allowing us to pull this image from our namespace easily.

Creating a helm chart

To create a helm chart, we'll use the Helm command-line tool. In the application directory, run the command to create the helm chart:

$ helm create charts

The command above creates a new folder containing:

chart/
├── Chart.yaml
├── charts
├── templates
│ ├── NOTES.txt
│ ├── _helpers.tpl
│ ├── deployment.yaml
│ ├── hpa.yaml
│ ├── ingress.yaml
│ ├── service.yaml
│ ├── serviceaccount.yaml
│ └── tests
│     └── test-connection.yaml
└── values.yaml

The Chart.yaml contains basic information about the Helm Chart. Change the name and description to:

name: recipe-app
description:  A recipe application built on FastAPI

In the list of files above, we're majorly concerned with _helpers.tpl, service.yaml, deployment.yaml and values.yaml in this article.

The _helpers.tpl contains variables to be defined in our normal k8s manifest. Instead of repeating variables such as annotations, labels, etc., in multiple manifests, they are placed in the _helpers.tpl file.

Update the _helpers.tpl file to this:

{{/*
Expand the name of the chart.
*/}}
{{- define "chart.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "chart.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "chart.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Common labels
*/}}
{{- define "chart.labels" -}}
helm.sh/chart: {{ include "chart.chart" . }}
app.kubernetes.io/name: {{ include "chart.name" . }}
{{ include "chart.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

{{/*
Selector labels
*/}}
{{- define "chart.selectorLabels" -}}
app.kubernetes.io/name: {{ include "chart.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}

{{/*
Create the name of the service account to use
*/}}
{{- define "chart.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "chart.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}

The deployment.yaml manifest contains the instructions for deploying your application to Kubernetes. It comes prefilled and fetches the values from the _helpers.tpl and values.yaml file. Update the containerPort in the container section to 8080:

containers:
  - name: {{ .Chart.Name }}
    securityContext:
      {{- toYaml .Values.securityContext | nindent 12 }}
    image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
    imagePullPolicy: {{ .Values.image.pullPolicy }}
    ports:
      - name: http
        containerPort: 8080
        protocol: TCP

Lastly, let's look into the values.yaml file. This stores the image name, replica count, service configuration data, etc. Update the image repository and tag value from:

image:
  repository: nginx
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: ""

image:
  repository: $IMAGE_REGISTRY/recipe-app
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "latest"

In the code block above, replace $IMAGE_REGISTRY with:

okteto.dev/recipe-app if you used the Okteto registry
Your docker hub username if you pushed your image to docker hub.

Next, under the serviceAccount heading, set the create variable to false as we're not interested in creating a service account:

serviceAccount:
  # Specifies whether a service account should be created
  create: false
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""

Next, change the service type and port to:

service:
  type: LoadBalancer
  port: 8080

We have successfully configured our application's helm chart. The next step is to deploy it to Okteto.

Deploying to Okteto

To deploy our application to Okteto, return to the base directory in your terminal and run the command:

$ helm install recipe-app chart/

We get a response from the console:

NAME: recipe-app
LAST DEPLOYED: Mon Dec 13 19:09:29 2021
NAMESPACE: youngestdev
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export NODE_PORT=$(kubectl get --namespace youngestdev -o jsonpath="{.spec.ports[0].nodePort}" services recipe-app-chart)
  export NODE_IP=$(kubectl get nodes --namespace youngestdev -o jsonpath="{.items[0].status.addresses[0].address}")
  echo http://$NODE_IP:$NODE_PORT

This tells us our application has been deployed. In our Okteto dashboard, we can see the deployed application:

Development environments for Helm applications

Helm chart powered applications can also be remotely developed in Okteto. Let's create a development container to verify this:

$ okteto init

Select the deployed application:

 i  Using youngestdev @ cloud.okteto.com as context
This command walks you through creating an okteto manifest.
It only covers the most common items, and tries to guess sensible defaults.
See https://okteto.com/docs/reference/manifest/ for the official documentation about the okteto manifest.
Use the arrow keys to navigate: ↓ ↑ → ←
Select the resource you want to develop:
  ▸ recipe-app-chart

The okteto.yml manifest is created and we can start our development container by running the command:

$ okteto up

 i  Using youngestdev @ cloud.okteto.com as context
 ✓  Persistent volume successfully attached
 ✓  Images successfully pulled
 ✓  Files synchronized
    Context:   cloud.okteto.com
    Namespace: youngestdev
    Name:      recipe-app-chart
    Forward:   8080 -> 8080
    Reverse:   9000 <- 9000

root@recipe-app-chart-okteto-c88fff58f-pt8c8:/app#

We have verified that our application can indeed be developed locally. In our dashboard, the state of the application has changed to In Development:

Let's start our application remotely:

$ python3 main.py

INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:     Started reloader process [80] using statreload
INFO:     Started server process [82]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

The development environment updates the application on every file change. Let's change the welcome message in app/api.py:

@app.get("/", tags=["Root"])
def get_root() -> dict:
    return {
        "message": "Welcome to your Okteto app powered by Helm charts live in development mode!"
    }

Okteto automatically synchronizes the changes, and the application is now updated:

We have successfully deployed a helm chart powered application as well as developed it using Okteto's remote development environment.

Conclusion

In this article, you learned how to build an application's image and push it to Okteto's build registry and docker hub. You also learned how to create helm charts for your application for deployments to Kubernetes. The application deployed using helm charts was also developed remotely by creating a development environment in Okteto.

Sign up on Okteto today to improve your development process experience using development environments and preview environmemts!

The code used for this article can be found on GitHub.

Developing a Rasa chatbot in Okteto

Abdulazeez Abdulazeez — Mon, 07 Feb 2022 18:16:52 +0000

Rasa is a Python framework for training and building an AI-powered chatbot integrated into various platforms.

Training and building AI-powered applications such as chatbots can be expensive. It requires high computational power and supported firmware. Okteto provides you with a development environment where you can train, build and deploy your models remotely on the Cloud.

In this article, we’ll show you how you can build a simple rasa chatbot, Oktebot, remotely in Okteto.

Want to see this built live? Watch the recorded live-stream on our Youtube channel!

Prerequisites

To follow this tutorial, you should have:

An Okteto Cloud account (It’s free! )
Okteto CLI installed
Basic knowledge of Python

Setting up

We’ll start by creating a project directory to house our application:



$ mkdir oktebot
$ cd oktebot

Building Oktebot in Okteto’s development environment

Now that we have created the project directory, the next step is to sign in to our Okteto account via Okteto CLI from our local terminal:



$ okteto context use https://cloud.okteto.com

In the project directory, we’ll create an Okteto manifest as the first step in creating our development container:



$ okteto init

The first prompt asks us to select the language we'll be working with:

This is to enable Okteto provide the necessary toolings for our remote development session.

In the prompt from the command above, select Use default values:



Select the resource you want to develop:
  ▸ Use default values

In the newly created manifest file, okteto.yml, modify the image name:



image: rasa/rasa

We have successfully created the manifest file. The next step is to start our development container and select Python as the default language in the prompt:



$ okteto up

The okteto up command creates a new development container in your dashboard from the parameters included in the okteto.yml manifest file and then starts a remote development session:




i  Using youngestdev @ cloud.okteto.com as context
 ✓  Persistent volume successfully attached
 ✓  Images successfully pulled
 ✓  Files synchronized
    Context:   cloud.okteto.com
    Namespace: youngestdev
    Name:      oktebot
    Forward:   8080 -> 8080
    Reverse:   9000 <- 9000

root@oktebot-okteto-7bbddd5c59-d7bmx:~#

In the remote development container, we’ll create a new rasa bot in the project directory:



root@oktebot-okteto-7bbddd5c59-d7bmx:~# rasa init

In the prompt in the remote terminal, select the current directory and train the initial model:



? Please enter a path where the project will be created [default: current direct
ory] 
? Directory’/app’ is not empty. Continue? (Y/n) Y
? Do you want to train an initial model? Y

Once the training is complete, the rasa tool prompts us to ask if we’d like to speak to the trained assistant:



Do you want to speak to the trained assistant on the command line? Y

Let’s speak with the trained assistant:

Training Oktebot

The data folder contains our bot's training data and responses in our project folder. There are three files currently present:

nlu.yml: This file contains the sample data for Natural Language Understanding.
stories.yml: This file includes example conversations with the bot to train the assistant to respond correctly depending on the preceding text.
domain.yml: This file defines the scenario in which the bot acts. This file defines intents from our NLU files and accompanying responses.

Now that we have successfully created our chatbot, let’s train the bot with our data. In the data folder, create two new files from your remote development container terminal:



root@oktebot-okteto-7bbddd5c59-d7bmx:~# touch {chat,faq}.yml

The newly created files will contain the intent and examples for training our bot similar to nlu.yml. The chat.yml file will house the intent for when a user starts a conversation with the bot, and faq.yml will contain the training data for when a user asks a question.

In the chat.yml file, add the following:



version: "3.0"

nlu:
  - intent: chat/hello
    examples: |
      - How can we help
      - we're glad we can help
      - please hold on while I get you an engineer
      - glad you love okteto

  - intent: chat/bye
    examples: |
      - do come back next time

  - intent: chat/sales
    examples: |
      - the sales rep will respond in a moment
      - you can pay using your credit card too
      - schedule a meeting with our sales team

In the code block above, we have defined three conversation intents in our chatbot. The examples listed are used to train the bot to respond appropriatelys to a similar message.

In faq.yml, add the following:



version: "3.0"

nlu:
  - intent: faq/more_resources
    examples: |
      - how much does the developer pro plan cost
      - I'll like to increase my bandwidth
      - What plans can give more benefit than the free plan
      - I'm hitting my limits easily. What can I do

  - intent: faq/plan_expires
    examples: |
      - what happens after my plan expires
      - can i renew my subscription

  - intent: faq/application_status
    examples: |
      - what happens to my applications after 24 hours
      - how can i check my deployment

  - intent: faq/restart_application
    examples: |
      - how can i restart my application
      - my application is asleep

In the code block above, we have defined three frequently asked questions ( FAQ ) intents in our chatbot. The examples listed are used to train the bot to respond to a similar message properly.

With the natural learning understanding files populated, let’s update the rules.yml guiding the chatbot. In rules.yml, add the following:



version: "3.0"

rules:

- rule: Say goodbye anytime the user says goodbye
  steps:
  - intent: goodbye
  - action: utter_goodbye

- rule: faq
  steps:
    - intent: faq
    - action: utter_faq

- rule: chat
  steps:
    - intent: chat
    - action: utter_chat

Lastly, let’s update the stories in stories.yml:



version: "3.0"

stories:

- story: faq
  steps:
    - intent: greet
    - action: utter_greet
    - intent: faq
    - action: utter_faq

- story: chat
  steps:
    - intent: greet
    - action: utter_greet
    - intent: chat
    - action: utter_chat

To wrap up the training data, let’s update the domain.yml file in the project directory:



version: "3.0"

intents:
  - greet
  - chat
  - goodbye
  - affirm
  - deny
  - faq

responses:
  utter_greet:
  - text: "Hi! How can I help you today?"

  utter_chat/hello:
  - text: "Hi, welcome to Oktebot. How may I be of help?"

  utter_chat/bye:
  - text: "Thanks for stopping by!"

  utter_chat/sales:
  - text: "The sales rep will respond in a moment"

  utter_cheer_up:
  - text: "We hope you enjoy the Okteto platform!"

  utter_did_that_help:
  - text: "Did that help you?"

  utter_happy:
  - text: "Great, carry on!"

  utter_goodbye:
  - text: "Bye. See you soon!"

  utter_faq/more_resources:
  - text: "You can request for more resources by either changing your plan or contacting the team at support@okteto.com"

  utter_faq/plan_expires:
  - text: "Your account automatically reverts to the free developer plan once the previous plan expires. You can renew to avoid that"

  utter_faq/application_status:
  - text: "Your application automatically goes to sleep after 24 hours of inactivity on the free plan only"

  utter_faq/restart_application:
    - text: "You can restart your application from the dashboard or wake them by visiting the endpoint"

session_config:
  session_expiration_time: 60
  carry_over_slots_to_new_session: true

By updating the domain.yml, we have updated the intent and responses our bot returns.

Retraining our bot

After populating our sample training data, the next step is to train the bot. Run the command in your development environment shell:



root@oktebot-okteto-7bbddd5c59-d7bmx:~# rasa train
Your Rasa model is trained and saved at 'models/20220113-053154-formal-sequence.tar.gz'.

Now that we have retrained our bot, we can test it from the shell:



root@oktebot-okteto-7bbddd5c59-d7bmx:~# rasa shell

We have interacted with the bot in the screenshot above, and it works according to the trained data. We can now exit the development container and start it whenever we introduce a model before deploying it.

The chatbot files created in our development environment are now available locally. This is as a result of the synchronization feature.

Conclusion

This article taught you how to train, build, and test a Rasa chatbot remotely using Okteto’s development environment. You also learned how Okteto synchronizes from the development environment to your local environment and vice versa.

Now that you have learned how to train and develop a rasa chatbot in Okteto, create a personal virtual assistant using rasa in Okteto and share it with us on our slack channel.

Sign up on Okteto today to improve your development process experience using development environments and preview environmemts!

You can find the code used for this article on GitHub.