Forem: Timilehin Aliyu

Progressive Web Apps: New FE systems

Timilehin Aliyu — Mon, 23 Dec 2024 07:57:48 +0000

Sometimes Connectivity Becomes a Bottleneck.

In enterprise environments, we often take stable internet connectivity as something given. However, real world conditions frequently challenge this assumption, potentially disrupting critical business operations. This article details how we transformed a traditional online only ERP system into a more reliable system with resilient offline capable solution. By leveraging browser-based storage solutions like IndexedDB, employing synchronization mechanisms, and using Progressive Web Apps (PWA).

Initially, the system followed a traditional client server architecture where all business logic resided on the backend. While this architecture works well in environments with reliable connectivity, it presented several challenges:

Transaction failures during network interruptions
Lost sales opportunities during outages
Poor user experience with constant loading states
Risk of data loss during critical operations
And most importantly, loosing customers due to lack of swift service.

So defining this, we had to improvise and see how we can make things better and also do without connecrivity since it isnt available initially, we implemented an offline system with some internet required using Progressive Web Apps (PWA), effectively moving critical business logic to the frontend while maintaining data integrity and synchronization with the core ERP system.

Some Core Components:

IndexedDB: For offline data storage and caching, we used IndexedDB via the Dexie.js library to provide a robust, client-side database that supports structured data storage. below is a simple example of how to set up a database with Dexie

// Initialize IndexedDB using Dexie.js
import Dexie from 'dexie';

interface LocalUsers{
id:string;
name:string;
role:string;
dob:string;
phone_no:string
}

interface LocalTrx {
id: string;
syncId:string;
created_at:string;
amount:string;
isSynced:boolean;
modified:string;
}

export class ArticleDatabase extends Dexie {
  transaction!: Table<LocalTrx>;
  users!: Table<LocalUsers>;
  constructor(){
    super("articleDB")
    }

this.version(1).stores({
// define the fields you'll like to query by or find items by within the indexDB
    transactions: 'id, date, amount, isSynced',
    users: 'id, name, role'
 });
}
//create the db
export const db=new ArticleDatabase()

// Open the database
db.open().then(() => {
    console.log("Database opened successfully!");
  })
  .catch((error) => {
    console.error("Failed to open database:", error);
  });

// Adding a new transaction to IndexedDB
import db from ../db
async function addTransaction(transaction) {
try {
   const trx = await db.transactions.add(transaction)
   console.log("Trx added", trx)
} catch (err) {
    console.log("Failed creating trx", err)
}
}

Service Workers: These act as a proxy between the app and the network, enabling offline functionality by caching resources and intercepting requests to ensure that critical data remains accessible during disconnection.

//service workesr can be setup easily, recently by default nextJS apps do come with service workes with vite, you can use the vite-pwa plugin
Background Sync: This allows us to sync data once the network is available again, ensuring that transactions are not lost and updates are made automatically once connectivity is restored.

System Architecture Flow
The architecture was divided into three main phases: initialization, transaction processing, and synchronization. The flowchart below shows how data flows between these stages.

Initialization Phase

When the system starts, it checks the network connection:

If the device is online, it fetches the latest master data from the server and updates the local IndexedDB.

If the device is offline, it loads data from IndexedDB, ensuring that users can continue working without disruption.

Transaction Processing

When users perform a new transaction:

Local data is validated and stored in IndexedDB.

An optimistic UI update is used to immediately show the result to the user, providing a smooth and responsive experience.

Synchronization Phase

When connectivity is restored:

Data is synchronized in batches with the server, either by manually clicking the sync button or after some certain timeframe.

If the sync fails (e.g., due to a slow connection), the transaction is added to a list of failed transactions and retried later.

Since we manage everything on the frontend how reliant is our service against securing customers Information.

Authentication & Authorization
In any enterprise system, securing sensitive user information is critical. Our solution ensures that:

JWT-based Authentication is used for secure user sessions.

Role-based access control ensures that only authorized users can perform specific actions.

Secure token storage is handled using browser-based mechanisms such as localStorage for added security.

To mitigate the risks of using locally stored tokens, we:

Trigger safe removal of user tokens upon logout.

Ensure that sensitive data is deleted from IndexedDB when the session ends or when the user logout from the system. Note: if transactions are left unsynced we display that to the logged in user and enforce them to sync before they logout.

Data Integrity & Conflict Resolution

Syncing data between the client and server introduces potential issues with data integrity, especially if multiple devices or users are making changes to the same data offline. To address this:

We validate all transaction details (e.g., quantities, amounts) before syncing to ensure that there are no discrepancies.

We assign unique IDs to each transaction to prevent duplication during synchronization.

Conflict resolution strategies are employed to handle situations where data changes are made on multiple devices while offline. For example, we use a timestamp approach.

//we try to make sure the offline is considered first, because it's the important bit of the system.

async function resolveConflict(localTransaction, serverTransaction) {
    // Compare timestamps determine which transaction should prevail
    if (new Date(localTransaction.timestamp) > new Date(serverTransaction.timestamp)) {
        // Local transaction wins
        await syncTransactionWithServer(localTransaction);
    } else {
        // Server transaction wins
        await updateLocalTransaction(serverTransaction);
    }
}

async function updateLocalTransaction(transaction) {
    // Update the local transaction in IndexedDB to reflect the server's state
    await db.transactions.put(transaction);
}

Network Security
Given that data is transmitted over the network once connectivity is restored, we ensured:

Rate limiting to prevent abuse and ensure that too many requests do not overwhelm the server with a 429 response hence why we originally worked with batch updates.

Encryption of data during transit using SSL/TLS.

Token expiration and secure token management, ensuring that stale or expired tokens are automatically removed from the client-side storage.

Alternatives to PWA & IndexedDB

While IndexedDB is a solid choice for client-side data storage in PWAs, there are other options available depending on the application's complexity and requirements:

SQLite via WebAssembly (WASM): Some developers opt to use SQLite via WASM for more advanced data management, particularly when dealing with larger datasets or complex queries. However, integrating SQLite via WASM introduces additional complexity, such as performance concerns and browser compatibility (e.g How sqlite made Notion faster).

Web Storage API (localStorage/sessionStorage): For simpler applications that don’t require complex querying or large datasets, the Web Storage API may be a viable alternative. It is easier to implement but has limitations in terms of storage capacity and querying capabilities.

Looking Ahead: Future Trends in PWA

As web technologies continue to evolve, so do the possibilities for applications like this. Emerging trends include:

WebAssembly and SQLite
Edge Computing
Advanced Sync Protocols: Emerging protocols like CRDTs (Conflict-Free Replicated Data Types) and DeltaSync

I myself can’t wait to explore how these technologies will transform the landscape of offline and distributed applications. With the rapid advancements in powerful machines and laptops, we have the opportunity to harness this increased computing power to deliver even more sophisticated and efficient software for users. At the same time, we must not forget the importance of catering to mobile devices and less capable devices, ensuring that our solutions are accessible and optimized across all platforms. The potential is enormous, and I’m excited to continue pushing the boundaries of what's possible with PWAs.

Note: What's Next

We’ll be taking things up. Using Djuix.io as our backend and React / Angular for our frontend, we would implement a proper basic flow. Stay tuned for more updates as we continue to enhance our approach to building amazing apps.

Anyway, I hope you enjoyed this and learned something new. I certainly did. I'd also love to hear your thoughts and experiences.

Until then.

Handling SQLite DB in Lambda Functions Using Zappa

Timilehin Aliyu — Mon, 10 Jun 2024 13:41:07 +0000

Using SQLite is rather straightforward for managing quick services, but when it gets scalable and a point of data integrity, RDBMS should be the choice at hand. While Relational Database Management Systems (RDBMS) like PostgreSQL or MySQL are the go-to choices for production-grade applications, there are scenarios where SQLite can be a viable and convenient option, especially during development or for small-scale projects. Moreover, when deploying a Django application to a Lambda function, it gets tedious to add secret keys just to test a simple app. You'd agree that working with SQLite would be a better option.

Lambda functions are designed to be stateless and ephemeral, making it challenging to use SQLite out of the box. Maybe it's quite straightforward and does support older versions easily, but along the line, you'd possibly run into issues such as deterministic=True which requires a specific version of SQLite. However, with few additional steps, you can seamlessly integrate SQLite with your Django project in Lambda, allowing you to focus on developing and testing your application without the overhead of setting up a full-fledged database.

When deploying Django applications to AWS Lambda functions, the ephemeral nature of the Lambda environment can pose challenges for maintaining persistent data storage. While it's possible to use RDBMS solutions with Lambda, it often requires additional configuration and setup, including managing database credentials, connection strings, and potentially adding external database instances. By using SQLite, you can simplify the deployment process and avoid the complexities associated with traditional RDBMS setups

Issues I came across while trying to use SQLite in Lambda

loss of data once the function execution ends.

Version Requirements: Some advanced features of SQLite, such as deterministic=True in functions, require specific versions of SQLite which may not be available in the Lambda environment.

Using SQLite with Django in Lambda

We can deploy our project using the following

To overcome the limitations of SQLite in Lambda, we can use a package called django_s3_sqlite. This allows us to store our SQLite database in an S3 bucket, enabling persistent storage beyond the ephemeral life of a Lambda function.

Install the django_s3_sqlite package:

pip install django_s3_sqlite

Update your Django settings:

In your settings.py file, update the DATABASES setting to use the django_s3_sqlite engine, and specify the S3 bucket and file name where your SQLite database will be stored:

DATABASES = {
    'default': {
        'ENGINE': 'django_s3_sqlite',
        'NAME': f'{s3_file_name}.db',
        'BUCKET': f'{s3_bucket_name}',
    }
}

This configuration allows your Django application to use an SQLite database stored in an S3 bucket, ensuring persistent data storage across Lambda function invocations. Now let's set up Zappa

Setting Up Zappa

Zappa makes it easy to deploy Python applications, including Django projects, to AWS Lambda.

Install Zappa

pip install zappa

Create a zappa_settings.json file with the appropriate Zappa configuration for your project. Here's an example:

{
  "production": {
    "django_settings": "project_dir.settings",
    "aws_region": "us-east-1",
    "role_name": "zappa-control",
    "role_arn": "your_arn_with_needed_access",
    "manage_roles": false,
    "project_name": "project_name",
    "runtime": "python3.8", # note this version
    "s3_bucket": "deploy_bucket",
    "use_precompiled_packages": false
  }
}

Note: We use use_precompiled_packages set to false to avoid potential compatibility issues with Lambda's environment. This way we can include our own SQlite binary file.

Handling SQLite Binary in Lambda

Given the earlier identified condition of Lambda functions being short-lived, we need to download the SQLite binary to run in our Lambda function. This ensures the required SQLite version is available. You can download the appropriate binary file for your Python version from the official SQLite website or from a trusted source.

Once you have the binary file, upload it to your S3 bucket for easy access during deployment.

aws s3 cp s3://{S3_BUCKET}/_sqlite3.so /var/task/_sqlite3.so

Note: The path /var/task/ is the root of your Lambda function's execution environment. However, you can also use the project directory you will be pushing to Lambda:

aws s3 cp s3://{S3_BUCKET}/_sqlite3.so {LOCAL_PROJECT_ROOT}/_sqlite3.so

Note we are using the cli here, while it might pose some challenge to run cli commands from your code into the lamda function you can also utilize boto3 and write the code to do that, but this might require extra information. But an easier way to run cli commands in lambda function is using the AWS Lambda Layers function.

Also, here's a python function using the subprocess module

import subprocess
def download_sqlite_binary():
    subprocess.run(['aws', 's3', 'cp', 's3://{S3_BUCKET}/_sqlite3.so', '/var/task/_sqlite3.so'])

Then, deploy your Django application to AWS Lambda using Zappa:

zappa deploy production

Voila. Once successful you should see the url provided for the deployed function. And if you test your application you'd notice your data being persistent. Nice Right.

While using SQLite with Django on AWS Lambda offers convenience and simplicity, it's important to consider the limitations and potential trade-offs such as data integrity, scalability, and Lambda limitations. Integrating SQLite with Django on AWS Lambda using Zappa can be a convenient and efficient solution for development, testing, or small-scale projects. However, reading from an S3 file in your Lambda function incurs additional costs. AWS S3 currently offers a free tier that includes 5GB of storage and allows 200,000 GET requests and 2,000 PUT or PATCH requests. With these pricing considerations, for small projects, the overhead is typically minimal compared to the benefits of simplicity and ease of setup.

Polling in React

Timilehin Aliyu — Sun, 09 Jun 2024 22:37:18 +0000

In today's tech space of web development, real-time data updates are important for creating engaging and responsive user experiences. Whether you're building a dashboard, a chat application, a server interface, or a stock trading platform, you often need to check the status of a server or a process at regular intervals. This process is known as polling, and it's a common technique used over socket connections.

Just before we see how Polling works, we should identify Why Polling?

We can describe polling as a straightforward approach to fetching data from the server periodically. Unlike more advanced techniques like WebSocket or Server-Sent Events (SSE), polling doesn't require complex setup or server-side configurations which makes it a reliable way to get the job done, especially in scenarios where you need to retrieve data from a third-party API or a legacy system that doesn't support real-time updates.

A good sample where polling can be useful will be:

When building a weather application that displays the current temperature, humidity, and wind speed for a specific location. You might want to poll the weather API every 5 minutes to ensure that your users have access to the latest weather information.

We can also use polling when we want to support older browsers or have limited server resources, and we can't utilize WebSocket for real-time chat.

Let's briefly look at how we can use Polling in React:

In our problem statement let's assume we have a web application where users can start and stop a server (e.g. something like npm run dev or python manage.py runserver). We want to provide real-time feedback on the server status whether it's running, creating, or stopping. Polling is an effective way to achieve this.

const pollingFunc = async () => {
   //fetch server data
  const data = await fetchServerData("GET");

  switch (data?.status) {
    case "stopped":
      setStatus({ stopped: true });
      break;

    case "running":
      if (data?.url) {
        setData(data);
        setStatus({ running: true });
        toast.success("Server is running");
      }
      break;

    case "creating":
      setStatus({ creating: true });
      break;

    case "stopping":
      setStatus({ stopping: true });
      break;

    default:
      handleError(data)
    toast.error(`An error occured, ${data.error_message}`)
      break;
  }
};

Addressing the asynchronous pollingFunc above, it fetches data from the server using the fetchServerData function. Based on the server's response, different cases are handled, such as updating the application state, displaying toast notifications, or stopping the polling process.

The key to implementing polling in React is to use the setInterval function, which allows you to execute a function repeatedly at a specified interval. Here's an example of how you might use setInterval to call the pollingFunc every 5 seconds:

import React, { useEffect, useRef } from 'react';

const MyComponent = () => {
  const pollingRef = useRef(null);
  useEffect(() => {
    const startPolling = () => {
      pollingRef.current = setInterval(() => {
        pollingFunc();
      }, 5000); // Poll every 5 seconds
    };
    startPolling();

    return () => {
      clearInterval(pollingRef.current);
    };
  }, []);
};

We utilize the useRef hook to create a mutable(changeable) reference to the interval ID returned by setInterval. This allows us to clear the interval when the component unmounts, preventing memory leaks. The useEffect hook is used to start the polling process when the component mounts. The startPolling function initializes the interval and calls pollingFunc every 5 seconds. The cleanup function returned by useEffect ensures that the interval is cleared when the component unmounts.

And that's how polling works, quite straightforward no tweaks or headaches, but one important consideration when implementing polling in React is how to update the component's state. Updating states within the polling function would leave you wondering what's wrong with my code? and this is because our setInterval runs in a different event loop. The setInterval callback captures the state at the time it was created and does not reflect subsequent updates to the state, so at that instance, our state doesn't know about any update. To handle this situation effectively, one can use a Ref Hook to keep track of the latest state OR manage the state updates within a useEffect. Another approach is to pass the state and the setState function as parameters to the polling function.

Let's update our above example to include the stopping state:

const pollingFunc = async (stoppingState: boolean) => {

  switch (data?.status) {
    case "stopped":
      setStatus({ stopped: true });
      if (stoppingState) {
        toast.success("Server stopped");
      }
      setStatus({ ...defaultState });
// let's assume we have a defaultState of all possible 
// status to be false here, so we can just spread it
      break;
   // we have the same prev code as earlier
}

//Updating our UseEffect

const MyComponent = () => {
  const pollingRef = useRef(null);
  useEffect(() => {
    const startPolling = (stoppingState) => {
      pollingRef.current = setInterval(() => {
        pollingFunc(stoppingState);
      }, 5000); // Poll every 5 seconds
    };

    startPolling(false);

    return () => {
      clearInterval(pollingRef.current);
    };

  }, []);
};

Just to display this in action let's create a stopServer function

const stopServer = async () => {
  setStatus({ ...status, loading: true });
  await fetchServerData("DELETE");
  setStatus({ ...status, loading: false, stopping: true });
  setData(null);
  startPolling(true);
};

Implementing polling in React is a straightforward approach to fetching data from the server at regular intervals. However, it's essential to consider the appropriate scenarios and the potential implications of polling on performance, server load, and user experience.

Depending on the problem at hand, we can figure out the best and cleanest way to handle polling. In cases where real-time updates are crucial, and server resources are available, utilizing WebSocket might be a better choice as they establish a bidirectional communication channel between the client and the server, allowing for efficient and immediate data updates without the need for continuous polling.

It's important to remember that as software engineers, we need to take into account the best tool for the job at the right time. Polling is a valuable technique, but it's not a one-size-fits-all solution. Carefully evaluate your application's requirements, performance needs, and server resources before deciding on the appropriate approach for handling real-time data updates.

Please share your thoughts, experiences, and any additional insights you might have on implementing polling or alternative approaches to real-time data updates in React applications. Also, stay tuned to see how we would implement polling in Angular.

Notetaking with AI.

Timilehin Aliyu — Thu, 26 Oct 2023 08:44:31 +0000

In a rapidly evolving world where technology continually shapes our lives, the way we capture and manage information is an amazing transformation. As a response to this, I participated in the Hanko Auth hackathon, and the result of this efforts is an innovative notetaking application. This application aims to revolutionize your notetaking experience by leveraging AI and introducing features, including AI-driven auto-completion. In this blog post, we'll highlight how it combines cutting-edge AI technologies, such as OpenAI's DALL·E and ChatGPT, with the power of Hanko Auth to provide an unparalleled notetaking experience.
Link to preview of the NoteAI App

Powered by AI and Hanko Auth:
The application utilize the most recent trend in software development from using AI, Neon DB, Drizzle ORM and Hanko Auth. We employ advanced AI technologies, specifically OpenAI's DALL·E and ChatGPT, to create a notetaking experience. Imagine an application that can anticipate your thoughts, assist you in generating content, and even embellish your notes with AI-generated cover images—all secured through the robust authentication provided by Hanko Auth.

Features

1. AI-Generated Images with DALL·E:
The NotesAI application gives your note covers AI-generated images, making it easy to identify your story. The process is incredibly simple: just provide a title when creating a new note. Here's where DALL·E AI comes into play. Using your notebook's title, a context is generated as a visually appealing cover image. It's a straightforward process that delivers astonishing results. You don't need any artistic skills; let the AI handle it.

2. Auto-Completion for Text:
The Notes AI app redefines the way you write. It's not just a notetaking app; it's your intelligent writing assistant. As you type within your notebook, it is constantly at your service. Activating auto-completion is as simple as pressing Shift+A. This user-friendly functionality is powered by TipTap, ensuring a seamless writing experience.

The logic behind this feature is elegant: the app considers your previous words as prompts, guaranteeing a consistent tone throughout your text. The OpenAI gpt-model, takes over, maintaining a friendly, helpful, and inspiring tone while offering thoughtful responses. Writing has never been easier.

In Action:
Here's a glimpse of how NotesAI operates:

Start by authenticating yourself through hanko provided signup page
Click on create a new notebook and provide a title.
The application taps into DALL·E to generate an image that complements your notebook's name.
While you're typing, simply press Shift+A on your keyboard to activate auto-completion.
The application calls the completion endpoint, using the last 30 words as prompts to help you continue your text effortlessly.

Here's how we generate images using DALL·E:

// Code snippet for image generation
const response = await openai.createChatCompletion({
    model: 'gpt-3.5-turbo',
    messages: [
        {
            role: 'system',
            content: 'You are a creative and helpful AI assistant capable of generating interesting thumbnails for my notes. Your output will be fetched into my DALL·E API to generate the thumbnail. I want your description to be minimalistic and flat styled.'
        },
        {
            role: 'user',
            content: `Please generate a thumbnail description for my notebook titled ${prompt}.`
        }
    ]
});

And here's the code snippet for text auto-completion:

// Code snippet for text auto-completion
{
    role: 'system',
    content: `You are an AI embedded in a Notion text editor app, dedicated to providing concise sentence completions. 
    Your characteristics include expert knowledge, helpfulness, cleverness, intelligence, and articulateness. 
    You always maintain a friendly, kind, and inspiring demeanor while offering vivid and thoughtful responses.`
},
{
    role: "user",
    content: `I'm crafting content in Notion and need your assistance to finish this thought: ${prompt}. 
    Please maintain a consistent tone with the existing text and keep the response brief and on point.`
}

Introducing Hanko Auth:
We have enhanced our app's security by implementing Hanko Auth. The decision to integrate Hanko Auth was influenced by the ongoing hanko hackathon, where I had the opportunity to witness its effectiveness and the flexibility it provides for user authentication. It seamlessly connects with popular OAuth providers, including GitHub, Google, Apple, and more. Hanko Auth ensures that your application's authentication is both robust and user-friendly. Check here for more information Hanko Auth

Conclusion:

That's a brief walkthrough unveiling a revolutionary notetaking application that merges advanced AI technologies with traditional notetaking methods to create a productive and seamless experience. To have a preview of the code flow you can find it on my github

The application serves as your creative partner, your productivity booster, and your intelligent companion. Whether you're a student aiming to take your notes to the next level, a professional in need of a versatile writing assistant, or simply someone who values efficiency and innovation, NoteAI is your ideal notetaking app.
Also, Elliot Chong and his team has paved the way for us to redefine how we use open AI models. I'm honored to build upon their inspiration and contribute to a brighter future of AI models.

Creating course content with AI

Timilehin Aliyu — Thu, 26 Oct 2023 08:44:08 +0000

Have you ever found yourself stuck, struggling to create the perfect outline for a new context you'd like to learn? Learning can be challenging, but it doesn't have to be. With this in mind we've used the trending gpt models to provide solutions to this scenario

Using the power of state-of-the-art technologies to provide great user experience among them include:

Prisma DB
Hanko Auth: Your Security Hanko.
OpenAI
YouTube API
Unsplash API

How It Works
Here's a quick overview of how we utilize all these technologies together:

Step 1: Sign Up and Create Your Course Outline

With a simple sign-up process through Hanko Auth. Once you've successfully registered, you'll find yourself on the path to creating your customized course outline or on your navbar you can navigate to create course.

Step 2: Share Your Vision

User Input: You play a vital role by sharing the course title and specifying the units you'd like to include. This is where your course adventure truly takes shape.

Using the input as a prompt for OpenAI, which generates a search query and passes it to YouTube API locates matching videos and transcripts for your course content and then generates an outline for you with several units and chapters.

Step 4: Your Creative Space

Your input becomes a prompt for OpenAI's GPT model which generates a tailored search query, and the YouTube API steps in, locating videos and transcripts that perfectly align with your course content. The result? A fine tailored crafted course outline featuring multiple units and chapters.

Crafting your notes is a breeze. While it's a bit like an exclusive art form, sometimes the AI model needs a moment to respond. But don't worry – any unit that's not available immediately is highlighted in red. You have the power to regenerate or continue, ensuring a smooth and efficient units and chapters to work with.

Behind the scenes, Unsplash API gets to work, providing the perfect cover image for your course. It selects images that matches your course title, elevating the visual appeal of your educational journey.

Once you've saved your course, your personalized learning dashboard awaits. Dive into your course outline, access YouTube videos, review summaries generated from transcripts, and challenge yourself with quizzes designed to test your understanding.

The quiz(i.e concept check) includes instant feedback to let you know where you stand. Expect the screen to light up in red for incorrect answers and green for the correct ones.

We also have a streamlined profile management system with Hanko Auth, you have access to a user-friendly profile management system. Here, you can explore your profile, add an email, and generate a secure passkey that replaces the need for traditional passwords and codes.

Say goodbye to endless hours spent searching for specific topics on YouTube. Now you can achieve a tailored learning experience with AI.

Interested to see how it works? Visit here.

And lastly, it's essential to acknowledge the inspiration that fueled the creation of this project as it draws its inspiration from the remarkable work of Elliot Chong.