Forem: ogbotemi-2000

A Playground for SQL RDBMS, custom Ollama API Interaction with RAG on Timescale DB and download of query plus result

ogbotemi-2000 — Mon, 11 Nov 2024 04:41:46 +0000

This is a submission for the Open Source AI Challenge with pgai and Ollama

What I Built

Main solution

A simple but complete webapp that facilitates development and testing of Ollama AI models through Timescale or other supported databases via a Retrieval Augmented Generation (RAG) widget for storing and retrieving embeddings generated by a specified Ollama model interacted with via its own widget in the webapp as well.

Extra benefits

It also doubles as a playground for thoroughly interacting with MySQL, PostgreSQL databases hosted on popular services like Neon.tech, Timescale with connection pooling support. All errors with queries and connections are caught and shown on the UI as a learning aid.

Perks

It can correctly receive and execute large volumes of SQL queries on connected databases as tested with the script for Timescale DB at https://app.chartdb.io/?ref=github_readme_2 under File > Import Database > PostgreSQL.

A encoding type of multipart/form-data along with a custom, efficient solution of less than 20 lines of code 😊 for parsing the request body when popular packages failed, did the trick. Check it out here: https://github.com/ogbotemi-2000/sqlUI_pinata/blob/main/utils.js#L11

Finally, it provides the option of saving a

{ 
  query: <RDBMS query>,
  response: <Result of query execution>

}

object as a direct download as a query-result.js local file or upload to IPFS via Pinata

UX Features

Directly paste database connection string into input fields and configure web app immediately with the password protected.
Switch immediately from MySQL to PostgreSQL database without hassle or storage of user data for uniqueness.

Demo

Hosted version

Here is a hosted version of the webapp.
https://sql-ui-pinata.vercel.app/

Remember to ollama pull <model>, ollama serve <model>, etc., before interacting with Ollama REST API via the web app

Local version

Similar to the hosted version but stores the provided database connection string locally and automatically bootstraps itself with it next time. Clone the repository

ogbotemi-2000 / sqlUI_pinata

A production-ready SQL UI for both MySQL and PostgreSQL databases with crash-proof error handling and support for uploading data to IPFS via Pinata

sqlUI_pinata

A production-ready SQL RDBMS (MySQL or PostgreSQL) UI webapp that accepts and stores database connection strings and executes queries, edited in a basic IDE by the user, in a crash-free manner whereby errors are intuitively shown to the user for correction or learning Databases can be changed on the fly by configuring them with the webapp, with support for connection pooling and support for uploading SQL queries and responses to the IPFS via Pinata

neon-starter-kit.mp4

Features

A HTTP Node.js server with dynamic support for middlewares and routes to api/ endpoints
Dynamic switching between MySQL and/or PostgreSQL databses without mixups as configured by the user anytime
Compatibility with Vercel serverless environment
Custom configuration that gets persisted until changed
A user interface for complete interaction with database
Option to enable pooled connection for database operations
Thorough testing of database via queries, the server never crashes; it returns detailed errors instead
Ready…

View on GitHub

Retrieval Augmented Generation Widget

The widget above receives its embedding from the Ollama API widget below before applying editable queries on the embedding and its associated metadata which is the JSON sent to the Ollama API in this case

Ollama REST API Interaction widget

Tools Used

pgvector

Its functions are used in the SQL template queries provided by the RAG widget. Here is how it is used in a part of the backend


let { toSql }      = require('pgvector/pg'),
{ parseMultipart } = require('../utils'),
isMySQL;

module.exports = async function(request, response) {
  /** request.body is only undefined for enctype=multipart/form-data */
  let { local, pooled, query, setup, embedding, metadata } = request.body||=await new Promise(resolve=>{
    let buffer = [];
    request.on('data', chunk=>buffer.push(chunk)),
    request.on('end', function(data) {
      data = Buffer.concat(buffer).toString('utf-8'),
      resolve(parseMultipart(request, data))
    })
  }),
  config=fs.existsSync(file)&&require(file),
  stored = (config||{ }).CONNECTION_STRING, dB;

  embedding = toSql(embedding.replace(/\[|\]/g, '').split(','));

...

toSql is used to ensure that values in the embedding; either out of range or invalid are replaced with null.

Using a more advanced toSql available in pgvector directly in the insert query templates for the RAG to either convert or pad vector embeddings to the length specified in the CREATE query ran prior.

Here is its schema below

toSql(embedding, vector_type i.e vector(float8), vector(int4) and vector(text), normalization options - l2_normalize or cosine_normalize, padding_value - eg 0, padding_length - 128, 256, 1536...)

And it is used as follows:

INSERT INTO data(metadata, toSql(embedding, vector(int4), l2_normalize, 0, 1536)) VALUES ($1, $2)

Here is alternative that uses a direct array_pad function in the INSERT query above if toSql triggers an error

INSERT INTO data (metadata, embedding)
VALUES (
    $1, 
    l2_normalize(
        (array_pad($2, 1536, 0))::vector(1536)
    )
);

The approach used at https://sql-ui-pinata.vercel.app is to create a database with an embedding column of variable length as follws:


CREATE TABLE IF NOT EXISTS data (id SERIAL PRIMARY KEY, metadata text, embedding vector);

Ollama

Created a minimalist but effective widget to make any call on the Ollama local REST API and specify options such as model, prompt, messages as well for dynamic generation of embeddings that are then used with customizable queries for RAG.

...
 fetch(url, {
    method: 'POST',
    body: meta
  }).then(res=>res.json()).then(res=>{
      result = res, flag&&(/*store embedding to be used sent to the server later*/account.embedding.value = embed = res.embedding)
  }).catch(err=>{result=err, flag=0})
  .finally(_=>{
    window.metadata.textContent = both.format((page.metadata = meta).slice(0, 100)+('...'.repeat(meta.length>100))),
    window.embedding.textContent = (page.embedding = embed).slice(0, 100)+('...'.repeat(embed.length>100))
...
})
...

pgai & pgvectorscale

Both pgai and pgvectorscale were not used directly however, the environment in which they can be used to create or use vector embeddings was created and assorted with widgets for quick and clear development with RAG and custom Ollama models, endpoints and prompts.

Final Thoughts

Lack of documentation on packages

Using pgvector.toSql in SQL queries has little information - code samples, use cases, and this waters down its usefulness, on how to use it to convert or pad vector embeddings to a given length on both Timescale and its Github repo.

Issues with downloading Ollama

The link to download the Ollama software on their homepage redirects to a githubusercontent domain after which the connection terminates. I was unable to download and run Ollama locally but I managed to learn about its API and built a widget for interacting with Ollama API, it may be bug-free but is untested. ### Early compromise
I was going to create a chatbot that generates and refines SQL queries for vector embeddings before OpenAI put up a paywall that I couldn't jump over and had to compromise.

Curious results

Curiously pgvector is not present in the result of the SQL query below but vector is, are the same?

SELECT * FROM pg_available_extensions;

Running the CREATE query below leads to the error below because of this

"could not open extension control file "/usr/share/postgresql/16/extension/pgvector.control": No such file or directory"

CREATE EXTENSION IF NOT EXISTS pgvector;

Prize Categories

Since I built a platform for generating embeddings from Ollama's API and storing them as vector embeddings, this submission qualifies for all, particularly

pgvector
Ollama, among others

Given that vector embeddings are much larger, byte for byte, than the original text they were generated from (i.e "Hello World" generated a vector embedding of length 4096), working with them requires a database service that handles large volumes well and considers chronology such as Timescale along with its packages for practical development and testing of LLMS.

Thanks to Timescale for such an opportunity to learn the nitty-gritty of what I used to call AI magic.

MySQL&PostgreSQL UI client for testing and querying databases with the query+result uploaded to Pinata

ogbotemi-2000 — Mon, 14 Oct 2024 11:36:50 +0000

This is a submission for the The Pinata Challenge

What I Built

Project

A resilient webapp for executing queries on either MySQL or PostgreSQL databases with a UI for writing queries and for giving fairly detailed feedback of the errors they cause.
There is a toggle to turn on/off connection pooling for a query before it is sent for execution on the database.

Use of Pinata

The webapp sports a widget for receiving the gateway and JWT of a Pinata account to use them to upload queries and responses to the IPFS and show the uploads in the files section of the Pinata account dashboard

Demo

The app was built to run locally and has not been tweaked to run on a public server. Here are screenshots of some UI flows in the app

the result of pasting a database URL in the dialog

display of a few details queried on the database

UI of webapp as it awaits the result of a query applied to the database

successful execution of the query - the "show query" button shows the query response below

the result of the query after clicking on "show query"

display of the error due to malformed query

Providing details to the Pinata widget for uploading queries and results to the IPFS

display of JSON sent after a successful upload to Pinata

error sent by the server when retrieving upuloads by their CID

My Code

The codebase is written as a hybrid of Node.js and Vercel serverless environments - it can successfully run in either.

ogbotemi-2000 / sqlUI_pinata

A production-ready SQL UI for both MySQL and PostgreSQL databases with crash-proof error handling and support for uploading data to IPFS via Pinata

sqlUI_pinata

neon-starter-kit.mp4

Features

A HTTP Node.js server with dynamic support for middlewares and routes to api/ endpoints
Dynamic switching between MySQL and/or PostgreSQL databses without mixups as configured by the user anytime
Compatibility with Vercel serverless environment
Custom configuration that gets persisted until changed
A user interface for complete interaction with database
Option to enable pooled connection for database operations
Thorough testing of database via queries, the server never crashes; it returns detailed errors instead
Ready…

View on GitHub

Issues

Pinata

The documentation regarding the use of Pinata's API did not show, in code, exactly how to upload a file through it
The SDK for uploading json - await pinata.upload.json({}) crashes due to some unclear error regarding the absense of a File
The documentation regarding using pinata.upload.file() uses the new File constructor despite its absence in Node.js

Fixes

The code below, not present in the docs, was used after many errors and trials

pinata.upload.file(new Blob([string_data], type))

Source codes

Pinata

An API endpoint defined in api/pinata.js to handle uploading and retrieving data from Pinata. It equally writes the provided JWT and Gateway strings to a sql_ui_config.json filef

let { PinataSDK } = require('pinata'),
    both          = require('../js/both'),
    fs            = require('fs'),
    path          = require('path'),
    filePath          = path.join(require('../rootDir')(__dirname), './sql_ui_config.json'),
    config        = fs.existsSync(filePath)&&require(filePath),
    pinata;

function upload(file, name, buffer, date) {
  date = new Date, buffer = Buffer.from(file, 'utf-8'),
  /** File object in similar fashion to the one present in browsers */
  // file = { buffer, name, type: 'text/plain', size: buffer.length, lastModified: +date, lastModifiedDate: date.toISOString() },
  file = new Blob([file], { type: 'text/plain' });
  return pinata.upload.file(file)
}

module.exports = async function(request, response) {
  let { data } = request.body||request.query;

  /** hardcoded string for splittig both on the client and server */
  data=data.slice(data.indexOf('=')+1).split('<_8c23#_>'),
  pinata = new PinataSDK({
    pinataGateway: data[2],
    pinataJwt: data[3]
  });
  /** write the provided data into files */
  config&&(config.PINATA_GATEWAY = data[2], config.PINATA_JWT = data[3]), 
  config||={ PINATA_GATEWAY: data[2], PINATA_JWT: data[3] },

  fs.writeFile(filePath, both.format(JSON.stringify(config)), _=>_)

  // pinata.testAuthentication().then()

  if(!data[4]) {
    let res
    //if CID is not in sent in data
    upload(data[0], data[1])
    .then(json=>{ console.log('::JSON::', res = json) })
    .catch(err=>{ console.log('::ERROR::', res = err) })
    .finally(_=>response.json(res))
  } else {
    let res;
    pinata.gateways.get(data[4])
    .then(file=>console.log('::RETRIEVED::', res = file))
    .catch(error=>console.log('::RETRIEVED::ERRORED::', res = error))
    .finally(_=>response.json(res))
  }
}

Vercel

The server.js file is written in such a way that it behaves similarly to using Vercel to receive and respond to API endpoints

let fs     = require('fs');

/** write ENV variables to process.env if available */
fs.readFile('.env.development.local', (err, data)=>{
  if(err) { /*console.error(err); */return; }
  data.toString().replace(/\#[^\n]+\n/, '').split('\n').filter(e=>e)
  .forEach(el=>{
    let { 0:key, 1:value } = el.split('=');
    process.env[key] = value.replace(/"/g, '');
    // console.log(process.env[key])
  })
});

let http   = require('http'),
    path   = require('path'),
    config = fs.existsSync('./config.json')&&require('./config.json')||{PORT: process.env.PORT||3000},
    mime   = require('mime-types'),
    jobs   = {
      GET:function(req, res, parts, fxn) {
        /** middlewares that respond to GET requests are called here */
        fxn = fs.existsSync(fxn='.'+parts.url+'.js')&&require(fxn)
        if(parts.query) req.query = parts.params, fxn&&fxn(req, res);
        return !!fxn;
      },
      POST:function(req, res, parts, fxn, bool) {
        fxn = fs.existsSync(fxn='.'+parts.url+'.js')&&require(fxn),
        req.on('data', function(data) {
          /**create req.body and res.json because the invoked module requires them to be defined */
          req.body = /\{|\}/.test(data=data.toString()) ? { data }
          : (parts = urlParts('?'+data)).params,
          fxn&&fxn(req, res)
        });
        if(!fxn) res.end();
        /** decided to return true instead of !!fxn since POST requests are not expected to GET html resources */
        return !!fxn||bool;
      }
    },
    cache  = {}; /** to store the strings of data read from files */

http.createServer((req, res, url, parts, data, verb)=>{
  ({ url } = parts =  urlParts(req.url)),
  /** data expected to be sent to the client, this approach does away with res.write and res.send in the jobs */
  res.json=obj=>res.end(JSON.stringify(obj)), // for vercel functions
  data = jobs[verb=req.method](req, res, parts),

  url = url === '/' ? 'index.html' : url,
  /** the code below could be moved to a job but it is left here to prioritize it */
  data || new Promise((resolve, rej, cached)=>{
    if (data) { resolve(/*dynamic data, exit*/); return; }

    /*(cached=cache[req.url])?resolve(cached):*/fs.readFile(path.join('./', url), (err, buf)=>{
      if(err) rej(err);
      else resolve(cache[req.url]=buf)
    })
  }).then(cached=>{
    res.writeHead(200, {
      'Access-Control-Allow-Origin': '*',
      'Content-type': mime.lookup(url) || 'application/octet-stream'
   }),
   /** return dynamic data or static file that was read */
    // console.log("::PROMISE", [url]),
    res.end(cached)
  }).catch((err, str)=>{
    str='::ERROR:: '+err,
    // console.error(str='::ERROR:: '+err, [url])
    res.end(str)
  })
}).listen(config.PORT, _=>{
  console.log(`Server listening on PORT ${config.PORT}`)
})

function urlParts(url, params, query, is_html) {
    params = {}, query='',
    url.replace(/\?[^]*/, e=>((query=e.replace('?', '')).split('&').forEach(e=>params[(e=e.split('='))[0]]=decodeURIComponent(e[1])), '')),
    query &&= '?'+query,
    is_html = !/\.[^]+$/.test(is_html = (url = url.replace(query, '')).split('/').pop())||/\.html$/.test(is_html);
    return {
        params, query: decodeURIComponent(query), url, is_html
    }
}

Backend for webapp

The API endpoint defined at the route api/accountData.js handles everything regarding configuring, responding to, persisting provided data to the sql_ui_config.json for the webapp.

let data   = {},
fs         = require('fs'),
path       = require('path'),
file       = path.join(require('../rootDir')(__dirname), './sql_ui_config.json'),
both       = require('../js/both'),
dB,
stringErr  = (err, cause, fix)=>[`"${err.message}"<center>----------</center>${cause}, code: ${err.code} with severity: ${err.severity||'&lt;N/A&gt;'} for sqlState: \`${err.sqlState||'&lt;N/A&gt;'}\`, at position \`${err.position||'&lt;N/A&gt;'}\` for operation \`${err.routine||'&lt;N/A&gt;'}\``, fix],
isMySQL;

module.exports = function(request, response) {
  let { pooled, query, setup } = request.body||request.query, /** to accommodate get or post requests via this server or Vercel serverless */
  config=fs.existsSync(file)&&require(file),
  stored = (config||{ }).CONNECTION_STRING;

  data.result='', data.errors = [];
  // console.log('::SETUP::', [setup, config, __dirname]);

  if(setup||stored) {
    config&&(config.CONNECTION_STRING = setup||stored||''), config||={ CONNECTION_STRING: setup||stored },
    // console.log('::SETUP::', setup, config, file),
    fs.writeFileSync(file, both.format(JSON.stringify(config)));
    if(!query) dB = null; /** absent query and present setup implies re-configuration of the app 
      with a different(enforced by the client) connection string which requires reconnecting the database
      driver with the new URL
    */
    isMySQL = /^mysql/.test(setup||stored),


    data.configured =  setup!=stored
    ? setup||stored/**sent the stored db string the very first time to synchronize with client */
    : /*a truthy instead of stored db string for security*/1,

    /** settiing dB = null destroys the closure below due to ||=, in order to update connectionString:setup after errors or re-configurations */
    setup&&(dB ||= require('../db')({isMySQL, connectionString:setup})) /** avoid requring module and invoking its exported function from scratch until either set to null or destroyed in serverless functions*/
    .then(operate)
    .catch(err=>{
      data.errors[0] = stringErr(err, 'Cause: connection string provided for configuration contains a nonentity', 'Re-configure app and ensure that the provided database URL resolves to an actual database'),
      dB = null, response.json(data)
    });
    if(stored&&!setup) {
      /** provide pinata configs to sent data if available as UI gimmick that fills them there in the client */
      let res = { configured: setup||stored };
      ['JWT', 'GATEWAY'].forEach((env, value)=>{
        (value = config['PINATA_'+env])&&(res[env.toLowerCase()] = value)
      }),
      response.json(res)
    }
  } else return response.json({configured:0});


  /* section that actually applies custom queries to the database */
  function operate(db) {
    db.pooled = pooled;
    if(query = query.replace(/\++/g, '\t')) db.query(query).then(arr=>{
      data.result = isMySQL ? arr[0] : arr
    }).catch(err=>data.errors[0] = stringErr(err, `Query: \`${query.split(/\s/).shift()}\``, 'Write syntactically correct queries and only specify fields or tables present in the the database or operations supported by the provider'))
    .finally(_=>response.json(data));

    else {
      let count = 0;/**used outside of loop cause the promsified nature makes the index i unreliable */
      /** added condition to avoid errors from reading non-existent functions or fields*/
      ['version'].concat(isMySQL ? [] : ['inet_server_addr', 'inet_server_port']).forEach((query, i, a)=>{
        db.query(`select ${query}()`).then(arr=>{
          data[query.replace('inet_', '')] = (arr=arr.flat())[0][query]||arr[0][query+'()']
        })
        .catch(err=>data.errors[0] = ['::DATABASE CONNECTION:: '+(/*data.version=*/err.message), 'Connect to the internet and remove typos in the environment variables for connecting the database'])
        .finally(_=>{
          if(!a[++count]) data.version = "VERSION • " + data.version, response.json(data);
        })
      })
    }
  }
}

Thanks for reading.

neon-starter-kit: with UI for thorough database interaction, for both Node.js and Vercel webapps

ogbotemi-2000 — Sat, 31 Aug 2024 10:12:36 +0000

This is a submission for the Neon Open Source Starter Kit Challenge : Ultimate Starter Kit

My Kit

Preview with video here

ogbotemi-2000 / neon-starter-kit

A starter kit for users getting started with the database services of neon.tech

neon-starter-kit

A starter kit for users using Node.js and/or Vercel serverless for getting started with or extending connecting and interacting with their PostgreSQL or MySQL databases hosted by direct effort or abstracted away via serverless SAAS like https://neon.tech

neon-starter-kit.mp4

Features

A HTTP Node.js server with dynamic support for middlewares and routes to api/ endpoints
Compatibility with Vercel serverless functions
Custom configuration via config.json
A user interface for complete interaction with database
Option to enable pooled connection for database operation
Thorough testing of database via queries, the server never crashes; it returns detailed errors instead
Ready to use templates for directives SELECT, DROP, ALTER, CREATE, INSERT and UPDATE
Editing support for queries in a basic HTML editor with formatting and styling
Dropdown containing various datatypes available in SQL for input types that store the type of data for a field - VARCHAR(255), INT...
Automatic addition of ending…

View on GitHub

Overview

The starter kit has a basic but adequate folder structure and codebase that is quickly extensible and doesn't get in the way of understanding and further developing it

| - server.js  # contains concise routing for GET and POST requests
| - index.html # contains UI for database interaction
| - config.json # is expected to be provided in cloned repo by user
| - css
| - js
| - fonts
| - webfonts
| - api # contains endpoints invoked by the Node.js server or Vercel

Configurations

A config.json file is imported where needed and read for environment variable values as an improvement over the approach of creating properties loaded from an .env file on process.env as done by dotenv - The values are easily changeable as JSON and default linting marks erroneous JSON files to prevent gotchas.

{
"PGHOST": "<DATABASE_HOST_ADDRESS>",
"PGDATABASE": "<DATABASE_NAME>",
"PGUSER": "<DATABASE_OWNER>",
"PGPASSWORD": "******************",
"ENDPOINT_ID": "<DATABASE_ENDPOINT>"
}

The user is free to adopt using an .env file in similar fashion config.json, in fact, the purpose of config.json is to encourage local development and testing before deploying to production.

Plug-n-play

Clone the template repository at https://github.com/ogbotemi-2000/neon-starter-kit
Create a config.json file (nothing will work without one) with the above details specific to your account on neon.tech;
Run npm start or node server

Remember to add your config.json to .gitignore since it contains secrets that should only remain in your local workspace

Interacting with your database via the UI

The UI offers a simple yet complete editing environment to quickly edit templates for SELECT, UPDATE, ALTER, INSERT, CREATE and DELETE queries.

You can even maintain the styling of keywords by copy-pasting them and editing them without completely clearing them

Performance gains

You are free to write incorrect SQL queries, the backend catches all errors caused by your custom SQL query and sends detailed errors for display for you to iteratively improve on your queries.

🚀 It even features support for enabling pooled connections and thus improve performance!

Room for Extension

You can write more query schema beyond the common queries present on the UI

Left out the ending semi-colon? No worries, such minor bloopers are fixed in the query before it is sent to the backend

The service displays the details and fix for the network error which prevents its complete loading

Complete loading and connection of database to display database name and PostgreSQL version

Displayed result of the `select` query on a database used for testing and development

Link to Kit

It is available as a publicly available template repository here
https://github.com/ogbotemi-2000/neon-starter-kit.git

Your Journey

Putting the 'full' into the stack

I have been using Redis and a smattering of MySQL for backend data storage but I do not think I have grasped using MySQL as expected of a fullstack developer so I try to understand its origins which led me to PostgreSQL. Given its level of maturity as an old-but-still-in-great-use tech, I find myself enjoying a blend of PostgreSQL and MySQL.

@neondatabase driver helped a ton!

I had tried to implement connecting to my hosted database on neon using the postgre package all to no avail until I used @neondatabase/serverless, problem solved.

Using Vanilla JavaScript and HTML for the UI/UX

Native is the best, frameworks only come close in most cases except CSS - TailwindCSS utility classes aided my design sprints and my knowledge of JavaScript made me get away with the less is more philosophy such as abstracting away everything about fetch and/or XMLHttpRequest in only and iframe and a form.

😊😉 Try searching for either fetch(...) or _XMLHttpRequest in the code, there wiil be no matches cause they are not directly used. Simplicity - less is truly more

Optimizations

Further down the line, I will remove unused CSS styles in the used stylesheets via the webapp I had developed prior to this challenge: https://rmrf-css.vercel.app
It is production ready and very thorough in detecting likely used selectors - in inline event listeners, in inline and external scripts.
Do check it out.
Thanks and go towards goodness

A smart contract that accepts a URL and return a JSON of only used CSS styles

ogbotemi-2000 — Mon, 19 Aug 2024 03:51:56 +0000

This is a submission for the Build Better on Stellar: Smart Contract Challenge : Build a dApp

What I Built

A smart contract on Stellar testnet that will receive a website URL and return a JSON of the used styles on the page as an improvement over this webapp I had deployed months ago:https://rmrf-css.vercel.app/

Demo

With a progressive UI, the user in onboarded on the useful aspects of the webapp as seen at the URL above.

The final logic flow that offers a customizable speed of execution

My Code

Reboot via Stellar blockchain

An experiment at trying HTTP requests for my first time in rust while communicating directly with the API layer of the webapp

use reqwest;

// tokio let's us use "async" on our main function
#[tokio::main]
async fn main() {
    // chaining .await will yield our query result
    let result = reqwest::get("https://rmrf-css.vercel.app?url=https://google.com").await;

    println!("{:?}", result)
}

Journey

The algorithm to trim css files has a very good performance and rewriting it in Rust will offer unreal speeds.

Going further

The old algorithm for trimming, available at the provided URL has been rewritten first, in Javascript and eventually, it will be rewritten in Rust.

If the Javascript version of the rewritten algorithm can boast a speed of 1.5s for unminified TailwindCSS against over 600 unique classnames, its rust equivalent is expected to be in the range of 60ms and below.

The rewrite will be debuted on the webapp: https://rmrf-css.vercel.app for web enthusiasts to get a whiff of its faster execution.

Join in on the code here!
https://github.com/ogbotemi-2000/rmrf-css-client

Additional Prize Categories: Glorious Game and/or Super Sustainable

"blocktcha"; a micro-payment CAPTCHA alternative on Stellar testnet blockchain

ogbotemi-2000 — Mon, 19 Aug 2024 03:13:46 +0000

This is a submission for the Build Better on Stellar: Smart Contract Challenge : Build a dApp

What I Built

A CAPTCHA alternative that accepts micro-payments in XLM to authenticate users while using common bot checks and a sandboxed environment

Demo

https://blocktcha.vercel.app/

My Code

https://github.com/ogbotemi-2000/blocktcha

let http   = require('http'),
    fs     = require('fs'),
    path   = require('path'),
    config = require('./config.json'),
    jobs   = {
      GET:function(req, res, parts, fxn) {
        /** middlewares that respond to GET requests are called here */
        fxn = fs.existsSync(fxn='.'+parts.url+'.js')&&require(fxn)
        if(parts.query) console.log("::PARTS::", parts), req.query = parts.params, fxn&&fxn(req, res);
        return !!fxn;
      },
      POST:function(req, res, parts, fxn) {
        fxn = fs.existsSync(fxn='.'+parts.url+'.js')&&require(fxn),
        req.on('data', function(data) {
          /**create req.body and res.json because the invoked module requires them to be defined */
          req.body = (parts = urlParts('?'+(data=data.toString()))).params,
          console.log('::POST::', parts)
          fxn&&fxn(req, res)
        })
        return !!fxn;
      }
    },
    mime   = { js: 'application/javascript', css: 'text/css', html:'text/html' },
    cache  = {}; /** to store the strings of data read from files */

http.createServer((req, res, url, parts, data, verb)=>{
  ({ url } = parts =  urlParts(req.url)),
  /** data expected to be sent to the client, this approach does away with res.write and res.send in the jobs */
  res.json=obj=>res.end(JSON.stringify(obj)), // for vercel functions
  data = jobs[verb=req.method](req, res, parts),

  url = url === '/' ? 'index.html' : url,
  /** the code below could be moved to a job but it is left here to prioritize it */
  data || new Promise((resolve, rej, cached)=>{
    if (data) { resolve(/*dynamic data, exit*/); return; }

    /*(cached=cache[req.url])?resolve(cached):*/fs.readFile(path.join('./', url), (err, buf)=>{
      if(err) rej(err);
      else resolve(cache[req.url]=buf)
    })
  }).then(cached=>{
    res.writeHead(200, {
      'Access-Control-Allow-Origin': '*',
      'Content-type': mime[url.split('.').pop()]||''
   }),
   /** return dynamic data or static file that was read */
    // console.log("::PROMISE", [url]),
    res.end(cached)
  }).catch((err, str)=>{
    console.log(str='::ERROR::ENOENT '+err, [url])
    res.end("::ENOENT::An error occured, you may create and use an error page in lieu of this string")
  })
}).listen(config.PORT, _=>{
  console.log(`Server listening on PORT ${config.PORT}`)
})

function urlParts(url, params, query, is_html) {
    params = {}, query='',
    (url = decodeURIComponent(url)).replace(/\?[^]*/, e=>((query=e.replace('?', '')).split('&').forEach(e=>params[(e=e.split('='))[0]]=e[1]), '')),
    query &&= '?'+query,
    is_html = !/\.[^]+$/.test(is_html = (url = url.replace(query, '')).split('/').pop())||/\.html$/.test(is_html);
    return {
        params, query, url, is_html
    }
}

The code above is the hand-written pure Node.js http server I wrote to mimic Vercel serverless functionality while offering more resilency

Journey

Ideation

I read Greg Brockman's article on what to build on Stellar where he mentioned an oft-requested feature of a micro-payment backed application. Building it on a blockchain as accessible as Stellar made the project easier to implement

User and domain registration

Users are assigned a UUID on their first submission of a unique domain and offered a template that they can use to begin using the widget on their sites that are under their registered domains

Payments UI and UX

The actual widget to enable the verification is in an iframe which automatically connects to the Freighter wallet extension if it is available or proceeds as usual if otherwise

Waiting period

Users are prompted to send some crypto to the provided wallet address and await a usually rapid confirmation thanks to the integrity of the Stellar blockchain, testnet nonetheless

Smart contract design

Here is what I could come up with, the purpose of the contract is for archival storage of little info such as timestamps, hashes of transactions. A better database implementation may suffice later on but a trick of persisting data on Vercel serverless backends is adequate for now: https://dev.to/ogbotemi2000/persist-data-in-vercelnextjs-serverless-backends-1h70

lib.rs

#![no_std]

use soroban_sdk::{contracttype, contract, contractimpl, Env, Address, Vec, vec};
use crate::extend_ttl::{extend_high, extend};
use crate::store::{read_balance, write_balance, Store};

#[derive(Clone)]
#[contracttype]
pub struct AccountData {
    pub balance: i128,
    pub trxes: i128,
    pub t_stamps: Vec<u64>,
}

#[contract]
pub struct Contract;

#[contractimpl]
impl Contract {

    pub fn register(env: Env, addr: Address) -> (Address, Option<AccountData>) {
        let account : Option<AccountData> = env.storage().persistent().get(&addr);
        match &account {
          Some(_ac) =>{

          }
          None =>{
            env.storage().persistent().set(&addr, &AccountData {
              balance: 0,
              trxes: 0,
              t_stamps: vec![&env, env.ledger().timestamp()],
            });
          }
        };
        (addr, account)
    }
    pub fn receive(env: Env, addr: Address, amount: i128) { 
        if amount < 0 {
            panic!("Negative amounts are disallowed {}", amount)
        }
        addr.require_auth();
        extend(&env);
        read_balance(&env, &addr, &amount, |env: &Env, addr: &Address, balance: &i128| {
            write_balance(env, addr, balance);
            extend_high(env, addr)
        });
    }

    pub fn store(env: Env, addr: Address) -> bool {
        Store::has(&env, &addr)
    }
}

mod test;

mod extend_ttl;

mod store;

The application utilizes EventSource to listen to transaction events and validate users as senders thereof.

Soroban RPC


// ../node_modules/
import { Contract, SorobanRpc, TransactionBuilder, Networks, BASE_FEE, Keypair, nativeToScVal } from '../node_modules/@stellar/stellar-sdk';
import config from '../config.json';// assert {type: "json"};

let kP     = Keypair.fromSecret(config.SECRET_KEY),
    rpcUrl = 'https://soroban-testnet.stellar.org',
    caller = kP.publicKey(), 
    fxn    = 'register',
    params = {
        fee: BASE_FEE,
        networkPassphrase: Networks.TESTNET
    };

const provider      = new SorobanRpc.Server(rpcUrl, { allowHttp: true }),
      sourceAccount = await provider.getAccount(caller);

async function invoke(publicKey) {
    /**build transaction */
    let contract  = new Contract(config.CONTRACT_ID),
        buildTx   = new TransactionBuilder(sourceAccount, params)
            .addOperation(contract.call(fxn, ...toType(publicKey||config.PUBLIC_KEY, 'address')))
            .setTimeout(30)
            .build(),
        prepareTx = await provider.prepareTransaction(buildTx);

    // console.log('::XDR::', prepareTx.toXDR()),
    prepareTx.sign(kP);
    try {
      let sendTx = await provider.sendTransaction(prepareTx).catch(err=>err)
      if(sendTx.errorResult) throw new Error('Unable to send Tx');
      if (sendTx.status === "PENDING") {
        let txResponse = await provider.getTransaction(sendTx.hash);
        while(txResponse.status==="NOT_FOUND") {
            txResponse = await provider.getTransaction(sendTx.hash);
            await new Promise(resolve=>setTimeout(resolve, 100));
        }
        if(txResponse.status==="SUCCESS") {
            let res = txResponse.returnValue;
            console.log('::RESPONSE::', res._value[0])
        }
      }
    } catch (error) {

    }
}

console.log(toType(config.PUBLIC_KEY, 'address'), Networks.TESTNET);

function toType(value, type) {
    return [nativeToScVal(value, { type })]
}

export default invoke

The Stellar blockchain is mature enough for real-world apps to be built on top of it. It is only left to rethink just what a blockchain really offers and how existing solutions can be improved via it.

With a low barrier of entry - readiness of testnet Lumens, it is clear that Stellar has long since been more than ripe to ride the wave of smart contracts that have intrinsic value and not hype.

Additional Prize Categories: Glorious Game and/or Super Sustainable

CC-Shoes: An innovative ecommerce website for buying and personalizing sneakers with an engaging menu design

ogbotemi-2000 — Mon, 15 Jul 2024 06:59:22 +0000

This is a submission for the Wix Studio Challenge .

What I Built

An interactive eCommerce website for sneakers resale that has engaging visuals and better navigation design. It further allows for customization of ordered shoes.

It further provides a login functionality to store the session of the user for when he/she adds either wants to custom designs to the sneakers or visit the basic profile page.

Demo

https://oogbotemi.wixstudio.io/cc-shoes

Landing Page

Footer

Adaptive, grid-like navigation

Custom design section

Responsive section grid

Mobile
Tablet
Desktop

It can be observed that the menu in the header adapts in the three screenshots above, here is a fourth as regards the custom menu design

Adaptive menu design

Development Journey

1. To toggle the grid menu animation behaviour with custom CSS

The concise code below goes in the onReady callback in the masterPage.js file and it ensures different animation behaviours for the squares in the grid menu.

They initallly animate in like cards with the next being either twice the width or height of the previous two.



  let track_click=0;
  explore = $w('#explore-pane'),
  $w('#open-explore').onClick(_=>{
      // [1, 2].forEach(n=>$w('#wishliststat'+n).label = /*wishlist count*/+' items added'),
      ++track_click,
      explore.customClassList.remove('hide'),
      explore.customClassList.add('show', 'menu')
  }),
  $w('#close-explore').onClick(_=>{
      explore.customClassList[track_click===2
        ? (track_click=0, 'remove') : 'add']('hide'),
      explore.customClassList.remove('show', 'menu')
  })

The squares in the grid appear staggered when clicked a second time between clicks only to animate neatly into position

2. Smoothly show hidden links upon login

Some buttons such as 'logout' and 'profile' are hidden to avoid distractions and only show after successful login.
Velo's authentication.onLogin and authentication.onLogout
methods make this very easy, secure and sure to do
Also, obtaining and showing member login username or email is straightforward without needing to handle forms.



import { authentication, currentMember } from "wix-members-frontend";

const member =()=>currentMember

  .getMember({fieldsets: [ 'FULL' ]})

  .catch((error) => {

    console.error(error);

  });

function loggedIn(args, uS, elm) {

  isLoggedIn = !0, rmStasis(), uS=$w('#user'), 

  member().then(user=>{

    elm = [], ['loginLoader', 'profile', 'to-design', 'logout']

    .forEach((str, i, e)=>(elm.push(e=$w('#'+str)), e).customClassList'show'i?'add':'remove'),

    signIn.link = '/profile',

    console.log('::LOGGED IN::', args, signIn.link),

    signIn.label = 'view profile',

    uS.text = user.loginEmail

  })

}

function loggedOut(args) {

  isLoggedIn = void 0, rmStasis(),

  signIn.label = 'sign in',

  ['#logout', '#profile'].forEach(str=>$w(str).customClassList.remove('show'))

}

authentication.onLogin(loggedIn), authentication.onLogout(loggedOut)

Showing loginEmail of signed in users in the grid menu

To speed up development with Velo, I made my showing/hiding states of elements via a CSS class that can be added or removed

4. Automatic code-syncing between IDE and editor

This was a notable boost to my development of the site as I could write code once in the IDE and design, try prototypes of layouts almost immediately in the editor.

Kudos to the devs for idea to include that along with collaborations without collisions.

Wix Members area
Wix Forms
'wix-data'
'wix-members-frontend'
File upload API
Wix stores API
CMS

Challenges Faced

Making custom CSS work

I had to see what transformations Wix does to the custom classes to find a workaround as well as to know what works and what doesn't

Adding sections to the global area for reusability

This stumped me. Adding a section as global always makes it hard to stack at the top z-index

<!-- Thanks for participating! →

CC-Shoes: An innovative ecommerce website for buying and personalizing sneakers with an engaging menu design

ogbotemi-2000 — Mon, 15 Jul 2024 02:51:47 +0000

This is a submission for the Wix Studio Challenge .

What I Built

An interactive eCommerce website for sneakers resale that has engaging visuals and better navigation design. It further allows for customization of ordered shoes.

It further provides a login functionality to store the session of the user for when he/she adds either wants to custom designs to the sneakers or visit the basic profile page.

Demo

https://oogbotemi.wixstudio.io/cc-shoes

Landing Page

Footer

Adaptive, grid-like navigation

Custom design section

Responsive section grid

Mobile
Tablet
Desktop

It can be observed that the menu in the header adapts in the three screenshots above, here is a fourth as regards the custom menu design

Adaptive menu design

Development Journey

### To toggle the grid menu animation behaviour with custom CSS The concise code below goes in the onReady callback in the masterPage.js file and it ensures different animation behaviours for the squares in the grid menu.
They initallly animate in like cards with the next being either twice the width or height of the previous two.

  let track_click=0;
  explore = $w('#explore-pane'),
  $w('#open-explore').onClick(_=>{
      // [1, 2].forEach(n=>$w('#wishliststat'+n).label = /*wishlist count*/+' items added'),
      ++track_click,
      explore.customClassList.remove('hide'),
      explore.customClassList.add('show', 'menu')
  }),
  $w('#close-explore').onClick(_=>{
      explore.customClassList[track_click===2
        ? (track_click=0, 'remove') : 'add']('hide'),
      explore.customClassList.remove('show', 'menu')
  })

The squares in the grid appear staggered when clicked a second time between clicks only to animate neatly into position

Smoothly show hidden links upon login Some buttons such as 'logout' and 'profile' are hidden to avoid distractions and only show after successful login. Velo's authentication.onLogin and authentication.onLogout methods make this very easy, secure and sure to do Also, obtaining and showing member login username or email is straightforward without needing to handle forms.

import { authentication, currentMember } from "wix-members-frontend";

const member =()=>currentMember
  .getMember({fieldsets: [ 'FULL' ]})
  .catch((error) => {
    console.error(error);
  });

function loggedIn() {...}
function loggedOut() {...}

authentication.onLogin(loggedIn), authentication.onLogout(loggedOut)

Automatic code-syncing between IDE and editor This was a notable boost to my development of the site as I could write code once in the IDE and design, try prototypes of layouts almost immediately in the editor. > Kudos to the devs for idea to include that along with collaborations without collisions.

Wix Members area
Wix Forms
'wix-data'
'wix-members-frontend'
File upload API
Wix stores API
CMS

Challenges Faced

Making custom CSS work

I had to see what transformations Wix does to the custom classes to find a workaround as well as to know what works and what doesn't

Adding sections to the global area for reusability

This stumped me. Adding them always makes it hard to stack at the top z-index

<!-- Thanks for participating! →

Persist data in Vercel(NextJS) serverless backends

ogbotemi-2000 — Fri, 28 Jun 2024 08:28:38 +0000

Photo by Aron Visuals on Pexels, chosen for its similarity to Vercel's logo

TL:DR;

This is not an introduction however, here is a good intro to serverless backends on Vercel by Vishal Yadav
Persisting data by storage in global objects doesn't work in Vercel functions because they are ephemeral.
Writing files to the directory of the project throws a server crashing error while reading files doesn't
The workaround used involves writing and reading files with unique filenames to a temporary directory; /tmp, allowed by Vercel but absent in the docs

I recently migrated my web app from Railway to Vercel only to encounter bugs in my Node.js server due to the short-lived nature of Vercel functions.
I had used an object to store intermediate data for the url parameter of each GET request. The said object reverts back to {} everytime the serverless function is hit.

The following comprises the steps I took in working up a solution as well a link to the web app to see a working version of the code and logic below.

Figuring out likely problems and their solutions

Unique filenames, unique data

Filenames for stored data should be unique to avoid mixing them for every request.
Say the exported function in api/root.js is reached by requests to https://<unique-url-to-app>/api/root. It may at first suffice to generate a unique string from the parameters of the GET or POST requests however, serializing request URLs into unique filenames leaves room for duplication not to mention, frustrations with invalid strings for filenames.

One solution to generating UUIDs is require('crypto').randomUUID() if it is supported by the current node version you choose to work with.
Otherwise, here is a useful function for generating UUIDs gotten from the 1loc repo

const uuid = (a) =>a
  ? (a ^ ((Math.random() * 16) >> (a / 4))).toString(16)
  : ([1e7] + -1e3 + -4e3 + -8e3 + -1e11).replace(/[018]/g, uuid);

Making reading/writing data similar to using objects: {"data":"data"}

The code design is done in a way that is asynchronous yet ergonomic in the way getting and setting properties on objects are.

Writing data is done after which a callback is supplied the generated UUID wherein it may be sent to the client as part of the response

/* api/store.js */
let fs   = require('fs'),
    path = require('path'),
    dir  = '/tmp/',
    uuid = a =>a
      ? (a ^ ((Math.random() * 16) >> (a / 4))).toString(16)
      : ([1e7] + -1e3 + -4e3 + -8e3 + -1e11).replace(/[018]/g, uuid);

module.exports = {
  write: (data, cb, id)=>fs.writeFile(data, path.join(dir, id=uuid()),
    function(err) {
      if(err) throw err;
      else cb(id)
  })
}

Using it is as follows; reading the data comes after writing it

let {write} = require('./api/store'),
    data;

write(data, function(uuid){
  /*the uuid is sent to the client which then uses it as a sort of unique string to retrieve its associated data by sending it along with its requests at a later time to be read by store.read */
  response.send(JSON.stringify({uuid})/*may be sent as either JSON or text*/)
});

Reading of stored files is wrapped with a Promise to await the result of its callback as follows:

/* api/store.js */
let fs   = require('fs'),
    path = require('path'),
    dir  = '/tmp/';

module.exports = {
  read: uuid=>new Promise((resolve, reject)=>{
    fs.readFile(path.join(dir, uuid), (err, buffer)=>{
      if(err) reject(err);
       else resolve(buffer.toString('utf-8'))
    })
  }),
  rm: id=>fs.unlinkSync(path.join(dir, id))
}

Which can then be used as follows

let {read, rm} = require('./api/store'),
    data;

if(data = await store.read(/*uuid*/)) {

  response.send(data);
  rm(/*uuid*/) //may be removed or left alone
}

Final and working draft of the code in `api/store.js`

/* store.js */
let fs     = require('fs'),
    uuid   = (a) => (a ? (a ^ ((Math.random() * 16) >> (a / 4))).toString(16) : ([1e7] + -1e3 + -4e3 + -8e3 + -1e11).replace(/[018]/g, uuid)),
    /* may be made asynchronous */
    rm     = id=>fs.unlinkSync(path.join(dir, id)),
    dir    = '/tmp/',
    path   = require('path');

/*writes/reads data as text/plain, open to modifications*/
module.exports = {
  read: id =>new Promise((resolve, reject)=>fs.readFile(path.join(dir, id), (err, buffer)=>{
      if(err) reject(err);
      else resolve(buffer.toString())
    })
  ),
  write: function(data, cb, id) {
    fs.writeFile(path.join(dir, (id=uuid())), data, _=>cb(id))
  },
  rm 
}

Using it in the wild

Here is how I used it to store and retrieve results generated for each valid request in this webapp

let store  = require('./store');
module.exports = async function(request, response, uuid) {
  let {url, id} = request.query;
  if(uuid = await store.read(id)) {
    /* exists */
    response.send(/* data */),
    /* you may remove the file or leave it since the contents of the /tmp/ directory are likely temporary */
    store.rm(uuid)
  } else {
    store.write('data', function(id) {
       /* send the unique id for the stored data in some way as JSON or text */
       response.send(id)
    })
  }
}

Conclusion

The /tmp/ directory, from its name, may be susceptible to changes from other programs on the server and therefore is not advisable to persist data in it in the manner you would a database.

Vercel provides object blobs for persistent storage but it is not used in this code base since its need for storage is temporary and only throughout the short runtime of the serverless function which is expected to send a reponse within a maximum allowable time of 25s if its runtime is edge.

This maximum duration is configurable if the runtime of the function is set to something aside edge like nodejs as follows

export const runtime = 'nodejs';
export const maxDuration = 15;
/*rest of the code in the file goes here*/

Serveless functions are a great way to start building the backend for a website or webapp with less worrying over edge networks, scaling, server distribution, IP-based content delivery, etcetera