Forem: cubiclesocial

Writing completely offline web apps

cubiclesocial — Wed, 19 Oct 2022 15:23:42 +0000

It seems contradictory in nature to make a web application that runs entirely offline. That is, it only needs to be loaded one time and then never makes another web request again. This is different from online apps that can lose Internet connectivity for a short while but sync when Internet functionality is restored.

Building a fully offline application is a tad bit more difficult to do than it seems on the surface. A fully offline web application must:

Load all resources in advance (HTML, Javascript, CSS, and images).
Handle its own navigation non-recursively and never navigate to another page.
Validate inputs in the web browser client.
Handle its own data storage and retrieval.
Not attempt to make requests outside of the initial app load (i.e. no form submissions or API calls).
Try to maybe have some security theater.

The need to build a fully offline application is pretty rare. An example is Offline Forms, which can design forms and gather data of a non-personal/non-financial nature even when Internet connectivity is spotty or non-existent:

cubiclesoft / offline-forms

A form designer and data gathering tool for use in areas with spotty or unknown Internet connectivity powered by any standard web browser. MIT or LGPL.

Offline Forms

A form designer and data gathering tool for use in areas with spotty or unknown Internet connectivity. Powered by any standard web browser. Offline Forms is a useful solution to the problem of digitally gathering data even when Internet access is not functional or only partially functional. Choose from a MIT or LGPL license.

Live demo

Features

Can be loaded one time from your web server and then saved offline. See below for details.
Excellent for trade shows, expos, and conferences for gathering potential client information (e.g. newsletter signups).
Great for use in areas where Internet connectivity might be spotty or nonexistent (e.g. gathering door-to-door survey responses).
Blazing fast. Changes to what is displayed happen almost instantaneously.
Password protected access to manage forms. With caveats - see below.
Supports custom HTML and Javascript.
Export and import prepared offline forms. Create on a desktop and then export the form…

View on GitHub

Offline Forms is a quick-n-dirty app I threw together just to test the idea of building entirely offline applications. It works fine and forms can be designed in a matter of minutes with the point-and-click interface, but don't expect to be impressed. I ended up spending most of the time fixing a number of minor bugs in JS FlexForms.

Security Theater

The biggest downside to building an entirely offline app is security. The web browser itself isn't secure and neither is any data stored in it that Javascript has access to (cookies, Local Storage, Session Storage, etc). A relatively simple bookmarklet can exfiltrate data from the browser to any properly configured remote host. The only security in Offline Forms is a repeatedly hashed password but that's just a minor deterrence.

Offline Forms allows custom Javascript to be written, executed, and even exported/imported (mostly intended for validation purposes). The importer displays a warning when importing a form with custom Javascript but not everyone reads warnings or pays attention to what they are doing. Remember what your parents told you about stranger danger: Don't import forms from total strangers and stick to those forms that you created yourself.

Load All Resources

The entire app has to be loaded up front, including all libraries and images. Otherwise the web browser won't know what all it needs to download to the device when saving it for offline use. HTML doesn't really have a way to declare a list of resources required for offline use of a webpage. It's somewhat expected that a webpage is, in general, only used online. Offline use is an edge case that few people think about until they encounter the scenario.

If a complete application uses 10MB of libraries, all 10MB has to be loaded whether or not those libraries will all be used. So extra care has to be applied to limit the number of libraries and the total file size of all of the libraries.

Handling Navigation Without Recursion

When a web browser navigates to another page, it usually sends a request to a web server. If there is no Internet connectivity, the request will fail. An offline app generally should not navigate outside of the current page (changing window.location.hash is probably fine).

Closures in Javascript, while generally useful, hit a weird edge case when navigation is done entirely within the same webpage. Navigation between app states is accomplished using the same internal function. This effectively recursively calls the app state function. Eventually, after thousands of app state changes, the call stack will throw an exception for being too deep. The fix for this problem is to store the next app state in an object and then wait for a setInterval() that was started when the page loaded to trigger. The interval callback sees the next state and switches to it. This helps keep the Javascript call stack small.

Validating Inputs

The user of the application will inevitably enter information somewhere. Input validation is something many Javascript apps already do when fields are filled in. That is still the case.

However, input validation is now also involved when loading data from Local Storage. Local Storage is one major location where data for the app will be stored. Previous versions of the application may do things differently or the data could even be modified outside of the app. Failure to validate all inputs can lead to app level failures where simple reloading of the app won't fix the problems.

Data Storage/Retrieval

An offline app still needs to store and retrieve data. Local Storage (window.localStorage) is a fairly obvious choice. And storing JSON encoded data in Local Storage is also a fairly obvious choice.

As mentioned earlier, Local Storage is insecure. Other webpages on the same domain can access Local Storage as can generic bookmarklets.

Local Storage also has limits. The maximum size for a single domain is somewhere around 2.5MB to 5MB.

I didn't explore the possibility of using a file Blob as a backend data store. A real backend file would bypass the Local Storage limits and perhaps improve security a little. I wanted to target the broadest device support and the File API is a tad bit newer and less broadly supported than I needed.

No Network Calls

Many people consider that an offline app should never do anything like try to talk to online services. That includes gathering app usage metrics (e.g. Google Analytics), check for app updates, or anything else. If the app is offline, it should be completely offline!

The "Export Data" feature of Offline Forms was originally going to have an online data pushing option to SaaS solutions (e.g. Google Sheets) in addition to exporting data as CSV and JSON Lines. The data entry portion to craft a POST request was too clunky, so I dropped it altogether.

The only time there is a network request in Offline Forms is when the iframe loads the form Preview. The iframe URL is the same as the parent with a fragment, so it shouldn't be a problem.

Conclusion

Offline Forms was an interesting exercise to attempt to build a fully offline app in a web browser. Obviously, not every web app can be fully offline, but some apps perhaps could be.

The biggest issue with offline web apps is that web browsers don't offer much natively in the area of secure data storage. Local Storage is useful but is not particularly secure and has limitations. It's almost like web browsers are supposed to be used for browsing the web.

The Incredibly Flexible Data Storage (IFDS) file format

cubiclesocial — Tue, 04 Oct 2022 01:32:51 +0000

Have you ever wanted to create a custom file format but didn't know where to start or found the task to be somewhat daunting? I've been there. Knowing how most binary file formats are generally structured certainly helps:

However, when making a new binary file format, everyone seems to go through the same set of exercises over and over again. These steps are roughly:

Looking around to see what other people have done when making a file format. (Sometimes this is the last step.)
Choosing a magic string for the first few bytes of the file.
Deciding what the file header consists of.
Designing what each object/structure in the file looks like. The more compact each structure is, the smaller the final file size will be but there are tradeoffs.
Calculating the size of objects/structures and their placement in the final file, including any extra bits such as padding, CRC-32/hashes, etc.
Return in 6 months to several years later only to realize that the original plans didn't make it possible to expand the format in a way that it now needs to expand.
Repeatedly return to the implementation of the format over many years as a variety of security vulnerabilities are discovered.

Steps 2-4 are theoretically easy, but the hardest part to get right in structured binary files is position tracking (step 5). The size of each structure has to be known before it can be written (or read) and the header or something else in the file has to point at the location of at least one structure. It's generally a bad thing if one structure overwrites part or all of another structure, so the position of each structure in a file has to be planned out before actually writing anything. The more layers of structures that there are, the more of a nightmare that these "bookkeeping bits" wind up becoming. Also, making a file format is fraught with difficulties: Planning for possible future expansion without bloating the file format is tough to figure out in advance and bad actors can supply malformed files that can trigger security vulnerabilities such as buffer overflows, privilege escalation, etc.

However, in general, a structured binary file format is not all that different from data structures in RAM. In fact, most binary files share several common patterns such as "fixed arrays" where each entry in the array is of fixed size. If these patterns were implemented in a way that eliminated the difficult parts of file format design, then anyone could rapidly design their own file format.

Enter the Incredibly Flexible Data Storage (IFDS) file format:

cubiclesoft / ifds

Easily create your own custom file format with the Incredibly Flexible Data Storage (IFDS) file format. Repository contains: The IFDS specification (CC0) and the official PHP reference implementation of IFDS, a paging file cache class, and example usage classes (MIT or LGPL).

Incredibly Flexible Data Storage (IFDS) File Format

The Incredibly Flexible Data Storage (IFDS) file format enables the rapid creation of highly scalable and flexible custom file formats. Create your own customized binary file format with ease with IFDS.

Implementations of the IFDS file format specification internally handle all of the difficult bookkeeping bits that occur when inventing a brand new file format, which lets software developers focus on more important things like high level design and application development. See the use-cases and examples below that use the PHP reference implementation (MIT or LGPL, your choice) of the specification to understand what is possible with IFDS!

Features

Custom magic strings (up to 123 bytes) and semantic file versioning.
Included default data structures: Raw storage, binary key-value/key-ID map, fixed array, linked list.
Extensible. Add your own low level data structures (supports up to 31 custom structures such as trees) and format encoders/decoders…

View on GitHub

The primary purpose of IFDS is to provide data structure (and associated data) storage and retrieval while, at the same time, handling all of the bookkeeping bits internally. That is all it does.

IFDS is fairly unique. Almost every file format is single-purpose: JPEG, PDF, ZIP, SQLite database files, etc. Accessing each format's structure and data requires its own separate library. Any file format with IFDS as its base, on the other hand, only needs a single IFDS library to access the data structures and data, allowing such files to be verified and even optimized using any IFDS library. Understanding the data still needs a library layer to interpret the data but lower level structure/data storage and retrieval can be handled by any IFDS implementation.

IFDS is also fairly unique in that there is no preset "magic string." The official reference implementation defaults to "IFDS" but it is recommended to set your own magic string for your own file format.

IFDS is also fairly unique in that it handles tracking position information in a file mostly via object IDs. Each individual data structure is stored in an "object," which tracks the location of each data structure + either internal object data or DATA CHUNKS. With very few exceptions, every object has an object ID associated with it. The "object ID to position" lookup table allows the object to move freely within the file while the ID is used to reference the object itself. The closest comparison here would be a database table with an auto-incrementing ID. IFDS is not a database though.

IFDS is fairly unique in that it is extremely scalable. It generates file sizes as small as a couple hundred bytes all the way up to 18EB (2^64). Each object supports storing seekable (i.e. random access) data up to 280TB in size. IFDS also supports up to 4.2 billion objects. In general, maximum file size limits imposed by the operating system will be encountered long before reaching the limitations of IFDS...assuming you've even got that kind of storage available. Internally, IFDS is somewhat similar to how some file systems work. It even has a free space tracker to track free space as objects and data are created, deleted, moved, and resized. IFDS is not a file system though.

There are over a dozen use cases that are provided as ideas for what IFDS can be used for and three of those use-cases are explored in-depth in the main IFDS repository README. I won't rehash those here (you can read those yourself), but any file format that has ever existed could be redone in an IFDS-based universe and be better off in the long-term. To provide a possible example of this claim, let's take the average Javascript file. We all know that minification and compression are "important" for content delivery on the Internet. I'm not a fan of minifying Javascript as it makes debugging much more difficult (and sometimes impossible). However, I don't have any issues with lossless data compression.

So hypothetically speaking: What if your web server was IFDS TEXT-aware, your web browser was IFDS TEXT-aware, and your favorite text editor was also IFDS TEXT-aware? Javascript files could be setup to be transparently compressed by the text editor, the web server could extract the MIME type from the metadata section and also not waste a bunch of CPU cycles compressing content, and the web browser could transparently decompress the file. You, the developer, would see and edit and debug in plain text as you do now but the content would be transparently compressed when storing to disk. Any IFDS TEXT-enabled tools would also know to transparently decompress the file as it reads in the data.

Transparent compression/decompression and better metadata support for text files are just scratching the surface of what IFDS is capable of. The text editors we currently use are vastly inferior to what they could and should be because they are editing file formats stuck in the 1970's. As a direct result, many hacky workarounds have been invented in the last 45+ years instead of fixing the root problems.

Binary formats like IFDS, of course, have their caveats with the main argument against them being that "text editors can't read them and viewing them in hex editors is cumbersome." However, imagine a world, for just one moment, where IFDS (or something similar) was the primary baseline file format. We would have a couple of common tools for every OS that could read any file to determine if it is partially corrupt and offer the user visual options to repair the partially corrupted portions. The ability to open IFDS (or similar) files in a wide variety of tools would exist and those tools could interoperate with any software application. Hex editors would support IFDS natively and show structure and object breakdowns visually. Text editors might even do rational things with IFDS files that aren't even in the IFDS TEXT format. Applications would load configurations from IFDS CONF files and simple, small OS-level configuration tools could configure any application with a single, shared point-and-click interface. The list of benefits goes on and on and the world would be a better place. Having every binary file format do its own thing is fundamentally problematic. There are more significant problems to solve than producing and parsing approximately the same data structures on repeat for each individual file format. Okay, now we can return to reality.

The reality is that we can't replace everything overnight with IFDS or something similar. For example, replacing every JPEG image on Earth would be an impossible endeavor. New file formats, however, have an opportunity to start with something highly scalable and skip a lot of the frustrating steps when making a new file format.

Note that while I'm obviously fond of and waxing poetic about IFDS, my goal here is to get people thinking about the files we create, the hacky workarounds that we use due to format limitations, and maybe start solving those problems. I'm not saying IFDS is the "best" option for new file formats, but I did pour the past 6 months of my life into designing it: Three complete rewrites of the specification + countless minor revisions + the official reference implementation + more revisions to the spec + the test suite and example classes + the documentation. I'd just hate to see that significant effort go to waste.

IFDS is almost certainly unique. I haven't seen a file format before whose stated goal is to be scalable up to 18EB while being efficient on system resources, transparently handle the tough bookkeeping bits, and provide common data structures and data storage specifically for creating custom file formats. However, there are thousands of different file formats floating around out there, so it is of course possible that I overlooked something similar.

The Incredibly Flexible Data Storage (IFDS) specification is released under CC0 and the official IFDS reference implementation is released under MIT or LGPL. IFDS is not tied to CubicleSoft or me other than just being a place for responsible custodianship and is intended to be used by the larger community of software developers.

DO Hackathon Submission

cubiclesocial — Mon, 11 Jan 2021 05:27:00 +0000

What I built

A recreation of DigitalOcean App Platform called Server Instant Start.

cubiclesoft / server-instant-start

Spin up a fully configured Ubuntu/Debian-based web server in under 10 minutes with Nginx (w/ HTTPS), PHP FPM, Postfix, OpenDKIM, MySQL/MariaDB, PostgreSQL, and more. Deploy your web application too.

Category Submission:

Built for Business
Random Roulette

App Link

cubiclesoft / server-instant-start

Spin up a fully configured Ubuntu/Debian-based web server in under 10 minutes with Nginx (w/ HTTPS), PHP FPM, Postfix, OpenDKIM, MySQL/MariaDB, PostgreSQL, and more. Deploy your web application too.

Video

Description

Spin up a fully configured Ubuntu/Debian-based web server in under 10 minutes with Nginx (w/ HTTPS), PHP FPM, Postfix, OpenDKIM, MySQL/MariaDB, PostgreSQL, and more. Deploy your web application too.

Link to Source Code

cubiclesoft / server-instant-start

Spin up a fully configured Ubuntu/Debian-based web server in under 10 minutes with Nginx (w/ HTTPS), PHP FPM, Postfix, OpenDKIM, MySQL/MariaDB, PostgreSQL, and more. Deploy your web application too.

Permissive License

Choose from MIT or LGPL.

Background/How I built it/Misc

See the first post in the series.

Adding the Final Touches

cubiclesocial — Mon, 11 Jan 2021 05:16:39 +0000

Well, it is down to the final hours of the DO Hackathon. Time to release the final touches of the Server Instant Start project: Data Relay Center integration and the project's YouTube video.

cubiclesoft / server-instant-start

Spin up a fully configured Ubuntu/Debian-based web server in under 10 minutes with Nginx (w/ HTTPS), PHP FPM, Postfix, OpenDKIM, MySQL/MariaDB, PostgreSQL, and more. Deploy your web application too.

Data Relay Center, or DRC, is a multi-channel, scalable WebSocket server. There are two types of clients per channel: Authorities and non-authorities. DRC runs in its own isolated user account on Linux systems, starts at boot, and is quite flexible and scalable when configured properly. DRC is the missing WebSocket component for PHP environments. Server Instant Start leverages the full capabilities of DRC by installing PECL and PECL ev (ev exposes libev to PHP userland). I happen to like libev better than libevent and libuv, which all do the same basic things for scalable event-driven socket I/O operations.

Making a video requires going through a complete video production workflow. I write a script, rehearse, record, edit, render, and publish. It can take weeks of advance planning to get through the entire cycle. And every video project has flaws that crop up, which can require expensive revisits to earlier stages.

Here's the final project video for Server Instant Start:

I've found that making a video really helps boost people's confidence in a project. It's easier to digest a well-designed video than a wall of text that may or may not have images interspersed.

Now that all the pieces are put together, it is time to go get this project rejected in style!

Recreating the DigitalOcean App Platform

cubiclesocial — Mon, 04 Jan 2021 14:11:57 +0000

When I first saw the DO Hackathon show up on Dev, I wasn't really all that interested because App Platform isn't particularly useful to me and Hackathons generally aren't my cup o' tea. But then I remembered that I had a project stuck on the ol' backburner and decided to make the ultimate tongue-in-cheek project for this Hackathon: Recreate DigitalOcean App Platform so that businesses don't waste their money on DigitalOcean App Platform.

By the very nature of this project, I don't expect to win this Hackathon, but I will have a whole lot of fun in getting my submission rejected in style!

Before I begin, it may help to know in advance that just one of several systems I maintain moves a mere 3.5TB of outbound data transfer each month. And that's an average amount of transfer for any given system I work with. I don't like surprise monthly bills. Paying through the nose per-GB of storage and/or transfer is a bad place to be (e.g. $0.01/GB of overage can add up fast). As a result, I end up doing a lot of comparison shopping that no one should have to do.

Outside of the relatively pointless free Starter tier, here's the fundamental problem with DO App Platform in a nutshell:

Tier	Cost	Transfer	Storage	Est. Cost vs. Closest Droplet
Basic	$5/mo	40GB/mo	~1GB	~13x (Ouch!)
Pro	$12/mo	100GB/mo	~1GB	~40x (Wow!)
Database	$7/mo	N/A?	256MB	~120x (Boinnnnnng!)

The critical limitations of just the entry level pay-actual-money-for DO App Platform Basic tier are pretty wild: 40GB/mo of transfer when compared to the DO baseline VPS of 1TB/mo. Also, because GitHub generally has a 1GB repo size limit and App Platform is ephemeral (no permanent storage), DO clearly isn't particularly concerned about disk usage. Those two items combined results in something around 13 times the cost compared to a baseline Droplet! Ouch.

Users can get 256MB of storage for a database for $7/mo! That's crazy expensive at $28/GB/mo. I can't even begin to wrap my head around how insane the cost metrics are for that one. The math works out to maybe something around 120 times the cost of a baseline Droplet. Boinnnnnng!

I could probably go on but I'll stop there. Wow.

The estimates are just that: Estimates. I threw some ballpark numbers at the wall to see what the cost differential against a Droplet of roughly equivalent size might be. Given the number of variables involved, attempting to be accurate is not really the point. Anything over 2x the cost for network transfer and storage is pretty bad. From my perspective, DO App Platform is simply not price competitive with Droplets.

Besides cost, App Platform has two major functional limitations: No root/SSH access and local storage isn't permanent. I consider those to be vital features of any system I use. The whole point of a VPS is to get bandwidth, transfer, OS, CPU, RAM, storage, and electricity and have near-total control of the setup and management of the system.

Okay, so App Platform might work fine for some things (e.g. pure static sites that could be hosted anywhere) but it is pretty expensive for most things. My project for this Hackathon is to recreate the relevant portions of App Platform at a fraction of the cost and without the limitations.

Behold, Server Instant Start:

cubiclesoft / server-instant-start

Spin up a fully configured Ubuntu/Debian-based web server in under 10 minutes with Nginx (w/ HTTPS), PHP FPM, Postfix, OpenDKIM, MySQL/MariaDB, PostgreSQL, and more. Deploy your web application too.

Server Instant Start

Spin up a fully configured Ubuntu/Debian-based web server in under 10 minutes with Nginx (w/ HTTPS), PHP FPM, Postfix, OpenDKIM, MySQL/MariaDB, PostgreSQL, and more. Deploy your web application too.

Instant Start is useful for setting up an entire server with minimal effort. Quickly install all components of a server in just a couple of minutes: A well-rounded OS configuration plus optional configuration of web server, email sending capabilities, a scripting language, and database(s). The contents of and knowledge contained in this repository come from responsibly managing many Linux-based web servers for over a decade.

Only using Instant Start on a brand new server is highly recommended. Any Debian-based Linux distribution will probably work fine. Failure to use Instant Start on a newly created system may result in damage to existing configuration files and/or data loss.

Features

A simple set of scripts that automatically install and configure several…

View on GitHub

I've been working on this off and on for the past few weeks. Christmas slowed progress down quite a bit since I took some time to write a web application to allow multiple people to remotely control 6 different webcams for Christmas during this year's Zoom session. That's a story possibly for a different time and I digress.

Anyway, most of the code and data for Server Instant Start was already built thanks to Barebones CMS Instant Start. It just required some refactoring to remove all of the Barebones CMS specific stuff and make components optional, adding scripts to install and configure MariaDB/MySQL and PostgreSQL, and creating a few post-install scripts.

Server Instant Start is essentially the Basic tier of DO App Platform but uses a Droplet via the underutilized "User data" feature of Droplets. Oh, and it is much more cost effective too.

If you are thinking, "This Instant Start thing looks a bit like Chef, Puppet, Ansible, etc." Sure, Instant Start is a bit like those tools but it is specifically geared for spinning up a ready-made, Internet-facing, pre-configured web server from scratch, which allows it to come with complex cross-package integration and configuration (e.g. connecting Nginx to PHP FPM), has a spot in the cloud-init script to pull down and deploy your project as you see fit (e.g. from GitHub), and also dramatically simplifies complex post-install tasks such as setting up HTTPS and DKIM. Those other configuration management tools take effort to learn and I'd rather write my own stuff anyway.

Here is the feature set from DO which shows how Instant Start stacks up to the Basic tier of App Platform:

Feature	DO App Platform (Basic tier)	Instant Start
Build static sites	Yes	Yes
Build and deploy dynamic apps	Yes	Yes (PHP is specifically configured)
Deployment from GitHub	Yes	Yes
Automatic HTTPS	Yes (see Side Note)	Not automatic but very easy to do - or use Cloudflare
Bring your custom domain	Yes	Yes
Global CDN via Cloudflare	Yes	Just go set up Cloudflare
DDoS mitigation	Yes (see Side Note)	Just go set up Cloudflare
Unlimited team members (this is an account-level feature, so I'm not sure why this is even mentioned)	Yes	Yes
Application metrics	Yes	No? But how useful is this if your app is written efficiently in the first place?
CPU – Shared	Yes	Yes
Auto OS patching	Yes	Yes
Vertical scaling	Yes	Yes
Outbound transfer	40GB/mo	1TB/mo minimum
Build minutes	400/mo	Unlimited
SSH root access	No	Yes
Permanent local disk storage	None (ephemeral)	25GB minimum

Side Note: If the Anti-DDoS and Automatic HTTPS features of App Platform actually come from Cloudflare, then DigitalOcean is actively misrepresenting several Cloudflare services as their own, which is a direct violation of the Cloudflare Terms and Conditions and could get DigitalOcean kicked off of the Cloudflare platform. If anyone at DO sees this note, please double-check your marketing materials.

As can be seen in the feature comparison list, Instant Start checks most of the boxes and is therefore fairly close in terms of features/options of the DO App Platform. It's also a fraction of the cost in terms of network data transfer allowance and storage, has SSH root access, and has permanent local disk storage, which are things that matter to me.

Time permitting, I plan on adding Data Relay Center to the servers and producing a project video.

I'm not entirely sure which Hackathon category this fits under. Probably the business and/or silly/wacky/fun categories.

Using Discord for product support from GitHub

cubiclesocial — Thu, 03 Dec 2020 01:06:56 +0000

I am now running an interesting experiment: Use Discord for providing live chat product support from GitHub repositories. In this post, you'll learn how to create a customer support portal using PHP to control access to a Discord channel.

The problem, in a nutshell, is that I get various requests on the issue trackers for my 80+ GitHub repos that aren't actual issues with the product but folks who just need some assistance. Opening an issue on an issue tracker should be one of:

An actual bug in the product itself.
A "bug" in the documentation (e.g. missing documentation).
A feature request.

Asking for help with understanding something is not really compatible with an issue tracker although it might be a disguised documentation issue. The main reason is that there is no hard-and-fast "end of issue" with support requests. Such issues can remain open for years as a result of waiting for the person to respond in the affirmative that they understand how to use a product.

My previous solution was to use forum software. While that works, it can be days before anyone notices the request for assistance and receive a response. Most folks would prefer a solution where there can be back-and-forth discussion in real-time.

Most live chat technical support solutions out there are expensive and primarily text-based. Enter Discord: A free chat system for text, audio, and video. However, Discord, by itself, is not really prepared to handle the workflow of:

User enters channel and announces themselves and their problem.
User gets picked up by a support person for direct messaging.
Gently kick the user out of the system when the support request is complete.
Repeat steps 1-3 for the next user.

From my own experience with running multiple forums, it is important to do two things when something is publicly accessible to streamline support:

Precisely control access to the Discord channel to prevent bots and the like from spamming the support mechanism.
Gather important meta-information about the incoming user (source GitHub repo, license info, etc).

After creating a Discord account and a new "server" for the technical support channel, go through the settings and adjust what you want to allow anyone to be able to do (file uploads, view the chat history, time delays between messages, etc). Once satisfied, it is time to get coding the missing bits that control server/channel access.

We'll use the Discord SDK for PHP to create a basic portal webpage that sends information to a Discord channel, creates a temporary, one-time use invite, and finally redirects the user into Discord. This will form the basis of the live support system and can be expanded later to, for example, require a CAPTCHA to be solved or accept a payment via Stripe to manage access. No one can get in without a valid invite code and invites are only issued when all checks within the script pass.

<?php
    require_once "support/sdk_discord.php";
    require_once "support/http.php";

    function DisplayError($title, $msg)
    {
        // Output website header here.

?>
<div class="contentwrap">
<div class="contentwrapinner">
<h1><?=htmlspecialchars($title)?></h1>

<p><?=$msg?></p>
</div>
</div>
<?php

        // Output website footer here.

        exit();
    }

    $githubuser = "[YOUR_GITHUB_USER]";
    $webhookurl = "[CHANNEL_WEBHOOK_URL]";
    $bottoken = "[YOUR_BOT_TOKEN]";
    $channelid = "[INVITE_CHANNEL_ID]";

    if (!isset($_SERVER["HTTP_REFERER"]))  DisplayError("Missing Referrer", "This page was accessed directly.  Please enter via a <a href=\"https://github.com/" . $githubuser . "/\">valid repository on GitHub</a>.");

    // Check the HTTP referer header to verify the user is coming from a valid repository.
    $url = HTTP::ExtractURL($_SERVER["HTTP_REFERER"]);
    if ($url["host"] !== "github.com" || strncmp($url["path"], "/" . $githubuser . "/", strlen("/" . $githubuser . "/")) != 0)  DisplayError("Invalid Referrer", "This page was accessed incorrectly.  Please enter via the relevant <a href=\"https://github.com/" . $githubuser . "/\">repository on GitHub</a>.");

    // Send initial message to the main channel.
    $options = array(
        "content" => str_replace("https://", "", HTTP::CondenseURL($url))
    );

    $result = DiscordSDK::SendWebhookMessage($webhookurl, $options);
    if (!$result["success"])  DisplayError("Discord Error", "An error occurred while attempting to access Discord.  " . htmlspecialchars($result["error"] . " (" . $result["errorcode"] . ")"));

    // Create a temporary invite.
    $discord = new DiscordSDK();
    $discord->SetAccessInfo("Bot", $bottoken);

    $options = array(
        "max_age" => 1800,
        "max_uses" => 1,
        "unique" => true,
        "temporary" => true
    );

    $result = $discord->RunAPI("POST", "channels/" . $channelid . "/invites", $options);
    if (!$result["success"])  DisplayError("Discord Error", "An error occurred while attempting to setup Discord.  " . htmlspecialchars($result["error"] . " (" . $result["errorcode"] . ")"));

    // Redirect the user's browser to the invite.
    $url = "https://discord.gg/" . $result["data"]["code"];

    header("Location: " . $url);
?>

The above PHP script sits on a public web server you have access to. Modify the various string placeholders with valid tokens and information. There are step-by-step instructions on how to set up Discord webhooks and bots on the Discord SDK for PHP repository.

Once the above script is configured, then it is time to connect it to a repository and begin accepting visitors in the Discord support channel. In the README markdown for the repo, add a dynamic button:

[![Discord](https://img.shields.io/discord/INVITE_CHANNEL_ID?label=chat&logo=discord)](URL_OF_SCRIPT)

Substitute in the correct values for INVITE_CHANNEL_ID and URL_OF_SCRIPT. Commit the changes to the repository.

If all goes well, a shield/button with the Discord logo, the word "chat" and the number of online users will appear on the respository. Using a Private Web Browsing/Incognito window can be used to verify that clicking the new button lets a user into Discord.

Only those who come from the source repository will have access to the Discord channel. Everyone else will be rejected (e.g. someone visiting via a fork of the source repo).

Enjoy your new live chat product support!

I turned Amazon Prime Photos into an "unlimited" off-site data backup service via JPEG APP markers

cubiclesocial — Sat, 14 Nov 2020 18:44:03 +0000

It has been several years since the Amazon Cloud Drive service API went from "okay" to "completely non-functional." Since I haven't used the service for quite a while, I figured it would be an okay time to reveal how I used to use Amazon Prime Photos as an unlimited storage backup service.

It all starts with modifying the JPEG file format in ways not really conceived of before. The JPEG file format is a chunked, structured binary file format. What this means is that JPEG format reserves the 0xFF byte for "markers" for the start of each "chunk" and each chunk has a specific structure. For example, a JPEG image starts with the Start of Image (SOI) marker, which is 0xFF 0xD8, so we know that if we see that sequence, we have found the start of a JPEG image. Similarly, the End of Image (EOI) marker is 0xFF 0xD9, which means we have found the end of a JPEG image. There are many other markers for specific purposes that have been reserved for JPEG itself within its own file format.

The JPEG file format specification also allows for what it calls custom application markers. These are defined as APP0-APP15 and are 0xFF 0xEn. This is where things get interesting. While some APP markers are in fairly common use, a few are not used at all. For example, APP5 is relatively unused. But instead of just picking an APP marker and irresponsibly using it, we're going to allow the marker to be infinitely extended should someone else want to do something with APP5:

0xFF 0xD8 (SOI)
...
0xFF 0xE5  (APP5)
stor-cbdat\x00  (The data coming next is stored, encrypted data for Cloud Backup)
~10MB payload of compressed, encrypted data  (all 0xFF's get replaced with 0xFF 0x00's as per the JPEG spec)
0xFF 0xD9 (EOI)

Most JPEG libraries will generally ignore or pass through unknown custom application markers. That is, most, if not all, image libraries will still verify that the image is a real JPEG image. Therefore, correctly dealing with the situation of using APP markers for shoving 10MB of extra data into the image almost certainly requires writing a custom image processing library.

The first step was to check/verify two things: Did Amazon servers modify JPEG images that were uploaded in any way and, second, did they count the JPEG images as real images and flag them as valid "photos" for the "unlimited" side of the service or did they count against the 5GB data file limit? Those were pretty simple questions to answer that involved grabbing a JPEG, stuff some test data in an APP5 marker, verify that Photoshop still opened the file, upload the file, download the uploaded file, and check the result to see if the original file was identical to the downloaded file. Obviously, it worked or this post wouldn't exist:

The implementation used in Cloud Backup went a slight step further. The ApplyPhoto() function in the code went out of its way to make the uploaded images fairly unique but also be real images. First, a single image was retrieved from Unsplash (a free photo site with a simple API). For each payload, the previously retrieved image was then drawn on with the current timestamp so the actual image data portion of the file would be unique. Then it dumped the real compressed, encrypted data payload into the APP5 marker and uploaded the final ~10MB photo to Amazon Prime Photos. At the time, it worked fine.

Cloud Backup used to support Amazon Prime Photos via the Amazon Cloud Drive API until the API stopped working altogether. Let's just say that Amazon isn't all that great at designing APIs and that there are much better alternatives to using their services, especially for something as critical as data backups and I'll leave it at that. But for a couple of years, I was successfully storing ~150GB of compressed, encrypted data inside JPEG images within Amazon Prime Photos using CubicleSoft Cloud Backup because unlimited photo storage was, well, unlimited! In case you are excited about this, let me burst that bubble: This solution is actually fairly dead because the Amazon Cloud Drive API is broken (by design). For backup, you are better off using Cloud Backup with Remoted API Server to route backup data to a friend's house via a public Digital Ocean instance.

The real purpose of this post is two-fold: First, chunked and many structured binary file formats like JPEG and PNG can be extended and used in unexpected ways as can be seen in this 23 minute video on structured binary file formats:

Second, I'm hoping this gets your wheels of inspiration rolling. It is, for example, possible to embed audio into a JPEG image using the same APP5 marker like this:

0xFF 0xD8 (SOI)
...
0xFF 0xE5  (APP5)
audio-flac\x00  (The data coming next is audio in the FLAC file format)
Encoded audio data  (all 0xFF's get replaced with 0xFF 0x00's as per the JPEG spec)
0xFF 0xD9 (EOI)

No native software currently exists to play such audio stored inside a JPEG. But that doesn't mean Javascript inside a web browser couldn't load the file and extract those bits of audio data and then start playing them. Why would anyone want to do this? Well, it's likely the browser caches the image, so the audio is already available locally instead of being part of a separate HTTP request. It's also a single file instead of multiple files. I dunno. Pick your favorite reason and go with that. The point is that this idea is quite extensible.

Another idea I've had before is to use APP5 to embed a PNG or SVG inside a JPEG so that the JPEG image doesn't show visual artifacting around the edges of written text. That is, get the small file size of a JPEG for a photo with the clarity of text and vectors in a PNG/SVG all within one file. I actually came up with the "embed a PNG or some audio into a JPEG" idea BEFORE the "let's cram 10MB of encrypted data into a photo file and see what happens" idea. Really, though, the possibilities are endless.

This article was originally published to CubicSpot on Blogger

Folder and File Explorer Widget in Javascript

cubiclesocial — Mon, 22 Jun 2020 13:11:06 +0000

After over two months of design and development (and lots of debugging), the CubicleSoft File Explorer widget is ready!

cubiclesoft / js-fileexplorer

A zero dependencies, customizable, pure Javascript widget for navigating, managing, uploading, and downloading files and folders or other hierarchical object structures on any modern web browser.

Here's a live demo of the widget:

Things you can do with the widget (non-exhaustive list):

Create new folders and files.
Rename folders and files.
Upload files and entire folders. Supports drag-and-drop from the desktop.
Download files one by one OR a bunch of files and/or entire folders as streaming ZIP files.
Cut/copy/paste using the clipboard. Even works between different web browsers.
Drag-and-drop items to move them to another folder. Even works between different browser tabs.
Delete files and folders and/or move them to a recycling bin.
Select multiple folders and files. Many different ways to do this.
Open files. (Open being application-defined.)
Navigate back and forth through folder history.

If I were to pick one feature as my favorite, it would have to be the mouse click + drag selection box with variable speed scrolling. I sunk more time into that feature than I probably should have.

I look forward to reading what you think about this widget in the comments below! Or just hit the heart/unicorn button and/or star the repo or whatever. That's cool too.

When to actually use preventDefault(), stopPropagation(), and setTimeout() in Javascript event listeners

cubiclesocial — Sun, 31 May 2020 04:20:02 +0000

Unfortunately, a search for "when to use stopPropagation()" and "when to call stopPropagation()" on Google turns up few answers except a number of very and semi-flawed articles related to the topic, but none of which answer the question of when it is okay to use stopPropagation(). stopPropagation() exists and therefore is meant to be used...but when?

It's time to remedy both the misinformation and provide the correct answer on when to call preventDefault() and stopPropagation() as well as setTimeout(). I promise setTimeout() is semi-related.

Event handling in web browsers is quite difficult for most people to grasp...even apparently for the experts! There are 85+ events to consider when writing custom Javascript bits. Fortunately, there are only a few in that list that are commonly used:

keydown, keyup, keypress
mouseenter, mousedown, mousemove, mouseup, mouseleave, wheel
touchstart, touchmove, touchend
click, input, change
scroll, focus, blur
load, submit, resize

I tried to group those into various categories and most should be pretty obvious as to what they do (e.g. 'click' means something was clicked, 'mousemove' means the mouse moved). But they are organized by: Keyboard, mouse, touchscreen, input elements, focus and scrolling, and miscellaneous events.

Digging into browser events

The web browser fires events in a specific order: Capturing then bubbling. What exactly does that mean? Let's use a picture of what happens:

The above diagram will be referenced as I go along. When I mention, "Step 5" or "Step 2" or some such, I am referring to this specific diagram.

If code like the following is written:

<style type="text/css">
.otherclass { width: 50px; height: 50px; background-color: #000000; }
</style>

<div class="theclass"><div class="otherclass"></div></div>

<script>
(function() {
  var elem = document.getElementsByClassName('theclass')[0];

  var MyEventHandler = function(e) {
console.log(e);
console.log(e.target);
console.trace();
  };

  elem.addEventListener('click', MyEventHandler);
  window.addEventListener('click', MyEventHandler);
})();
</script>

That will set up two bubbling event handlers. In this case, a click handler is applied to the div with the class 'theclass' and the window. When a user clicks the div inside of it, the 'click' event arrives in MyEventHandler at step 7 and again in step 10 in the earlier graphic. The browser walks down the hierarchy toward the target in the capturing phase and then moves back up to the window in the bubbling phase, firing registered event listeners in that order and only stops if it reaches the end OR a function calls stopPropagation().

When an event arrives, the 'e.target' contains the element with the target node in the DOM that resulted in the event being created. The 'e.target' is the single most important piece of information as it contains the DOM node that triggered the event.

Useful tip: Instead of registering events on every single button, div, and doodad in the hierarchy, it can be far more efficient to register a single event on a parent element of a group of nodes that share similar characteristics. Using 'data-'/dataset attributes can then allow lookups to be performed in O(1) time even if there are 500+ children.

What can go wrong: An example

Before diving into preventDefault() and stopPropagation(), let's look at what happens if there's a lack of understanding of how events and event propagation work:

In the example above, Bootstrap is used to show a menu of options when the "Dropdown" button is clicked. The menu closes as expected when clicking the "Normal Button" but it does NOT close when clicking the "Remote Link" button. The "Remote Link" button is using another library to handle 'click' events, which calls stopPropagation() and there is a bubbling 'click' event handler somewhere on the document.

The author of The Dangers of Stopping Event Propagation blames the authors of 'jquery-ujs' for calling stopPropagation() but we'll see momentarily that there are actually TWO bugs - one in 'jquery-ujs' and the other in Twitter Bootstrap...both bugs happen because the authors of both libraries don't actually understand the browser event model and the two libraries therefore collide in spectacular fashion when presented with a common scenario. The author of the article also makes a recommendation toward the end of the article that leads to unfortunate situations. Mind you, that article is near the top of Google Search results!

Understanding preventDefault() and stopPropagation()

Let's look at preventDefault() as it causes some confusion as to what it is used for. preventDefault() prevents the default browser action. For example, pressing the 'Tab' key on the keyboard has a default action of moving to the next element in the DOM that has a 'tabIndex'. Calling preventDefault() in a 'keydown' event handler tells the browser you don't want the browser to do the default action. The browser is free to ignore that and do whatever it wants but it will usually take the hint.

When should you call preventDefault()? When you know that the browser will do something you don't want it to do if you do not call it. In other words, generally don't call it and see what happens. If the default browser behavior does something undesireable, then and only then figure out precisely when and where to call preventDefault(). Overriding the default behavior should always make sense to the end-user. For example, if preventDefault() is called in a 'keydown' handler and the user presses 'Tab', the handler should do something sensible to move the focus to the "next" element. If they press 'Shift + Tab', the handler should go to the "previous" element.

Now let's look at stopPropagation() as it causes even MORE confusion as to what it actually does. When 'e.stopPropagation()' is called, the browser finishes calling all the events at the current step of the process and then stops running event callbacks. There is one exception for the 'e.target' node, which processes both step 5 AND step 6 even if stopPropagation() is called in step 5. (These "steps" are referring to the diagram from earlier.)

The problem with calling stopPropagation() is it stops event handling dead in its tracks. This creates problems for listeners further along as events they are listening for aren't being delivered. For example, if 'mousedown' propagates to a parent that is listening for 'mousedown' in order to start doing something and then listens for a matching bubbling 'mouseup' event but something else calls stopPropagation() in its own 'mouseup' handler, then the 'mouseup' never arrives and the user interface breaks!

Some people have suggested to call preventDefault() and use 'e.defaultPrevented' to not handle an event instead of stopPropagation(). However, this idea is problematic because it also tells the browser to not perform its default action. That can introduce a lot of subtle bugs too when going to do more advanced stuff. For example, calling preventDefault() in a 'mousedown' handler on a node that has 'draggable' set to 'true' will cause a 'dragstart' to never being called leading to all kinds of frustration. It is also improper to simply look at 'e.defaultPrevented' and return to a caller without doing anything else.

Suffice it to say that using 'e.defaultPrevented' won't work either. So what works? The correct answer is to cautiously call preventDefault(), only occasionally look at 'e.defaultPrevented' in combination with looking at the DOM hierarchy (usually to break a loop), and extremely rarely, if ever call stopPropagation().

Answering the question

Now let's answer the original question, "When is it actually okay to use stopPropagation()?" The correct answer is to only call stopPropagation() in "modals." The modal in a web browser is a little bit more fluid of a definition than "a child window blocking access to a parent window until it is closed," but the concept is similar. In this case, it is something we want to trap in a sandbox where it makes no sense to allow events to continue to propagate down/up the DOM tree.

An example could be a dropdown menu that allows the user to navigate the menu with both the mouse and the keyboard. For the mouse, a 'mousedown' anywhere on the menu results in selecting an item while clicking off the menu elsewhere on the page closes the menu (cancels) and carries out a different action elsewhere. This is an example where calling stopPropagation() would be the wrong thing to do because doing so would block the mouse from acting normally, requiring extra clicks to do things.

For the keyboard though, it is a completely different story. The keyboard should have focus on the menu and the focus should remain trapped there in that sandbox until the user navigates away with the keyboard (or uses the mouse). This is expected behavior! Keyboard events (keydown/keyup/keypress) are involved with a totally different user experience than mouse events. Keyboard navigation always follows a sequential set of steps.

In the case of a dropdown menu, pressing 'Escape' or 'Tab' on the keyboard should exit the menu. However, if the event is allowed to propagate up the DOM tree, pressing the Escape key might also cancel a parent dialog (another modal!). stopPropagation() is the correct solution for keyboard events where the keyboard focus is in a modal. Mouse and touch events are almost never modal unless you are displaying a true modal on the screen. As such, the keyboard can wind up in modal-style situations much more frequently and therefore stopPropagation() is the correct solution.

Putting it all together

Okay, let's go back to the Bootstrap/jquery-ujs example from before and figure out how to solve the problem using our new understanding of the browser event model. We know that calling stopPropagation() in the "Remote Link" button handler was the wrong thing to do because it caused Bootstrap to not be able to close the popup. However, remember I said there were TWO bugs here? Bootstrap is incorrectly watching for a bubbling event to close the dropdown. If you look at both the earlier diagram and the list of events, can you figure out which event Bootstrap should be looking for and where in the steps it should be watching for that event?

.
.
.
.
.
.
.
.
.
.
.
.
.

If you guessed a capturing focus change event on the window (aka Step 1), then you would be correct! It would look something like:

  window.addEventListener('focus', CloseDropdownHandler, true);

The handler would have to make sure that the target element for the focus change event was still within the dropdown's popup but that's a simple matter of walking up the 'parentNode' list looking for the wrapper element for the popup. If the popup is not in the hierarchy from 'e.target' to the window, then the user went elsewhere and it is time to cancel the popup. This also avoids the situation where another library might interfere by incorrectly calling stopPropagation() and the number of events that have to be registered in the browser to catch all possible situations is also reduced!

On setTimeout()

While we are on the topic of element focus, handling element focus is a huge source of preventDefault()/stopPropagation() headaches. This can lead to some really ugly hacks involving setTimeout() that don't need to exist such as:

  var elem = origelem;

  // But somelem or one of its children has the focus!
  someelem.parentNode.removeChild(somelem);

  // Doesn't appear to work...
  elem.focus();

  // But this does work.
  setTimeout(function() {
    elem.focus();
  }, 0);

This happens when improper focus changes cause the 'document.body' element to be focused because the focused element was removed from the DOM too soon. Calling setTimeout() with 0 milliseconds in order to change focus after all of the events have settled is always a hack. setTimeout()/setInterval() only run after completing a UI update, which is why the second 'elem.focus()' inside the setTimeout() above "works." But for a brief moment, the focus is on the body element, which can wreak all kinds of havoc.

stopPropagation() is sometimes used in conjunction with this hack to prevent, say, CSS classes from being removed that affect visual appearance without those classes (e.g. resulting in visual flashing from the CSS class being removed and re-added a moment later). All of that results in a jarring mouse and keyboard user experience and lots of workarounds for workarounds. This hack can be resolved by first moving focus to another focusable element that won't be removed before removing the element from the DOM that currently has the focus:

  var elem = origelem;

  // Now elem has the focus.
  elem.focus();

  // somelem can be removed safely.
  someelem.parentNode.removeChild(somelem);

  // No hacky setTimeout()!

There are very few instances where calling setTimeout() is totally legit - maybe use it for just the occasional things that actually timeout? When setTimeout() is used for something other than a timeout, there is almost always something that has been overlooked and could be done differently that's better for everyone.

Conclusion

Hope you learned something here about capturing/bubbling events and how preventDefault() and stopPropagation() work in that context. The event model diagram from earlier is probably the cleanest, most accurate representation of the web browser capturing/bubbling event model I've ever seen. That diagram might even be printer-worthy! Maybe not "put it in a picture frame and hang it up on a wall"-worthy but possibly fine for a printed page.

This article was originally published to CubicSpot on Blogger