Anything Can be Used for File Storage if You Use it Wrongly Enough

Nick Maloney — Mon, 31 Jul 2023 19:32:43 +0000

I recently came across the most excellent blog post “Anything can be a message queue if you use it wrongly enough” (It is a fun read and I highly recommend checking it out). After reading it, it dawned on me that I recently created an abomination that essentially did the inverse, I used a “message queue” (if you would please indulge me in considering Redis one) as a semi-ephemeral file storage service. Please do not use this for production, you have been warned!

In building out a proof-of-concept for some fun with Machine Learning (ML) audio processing, I needed a quick, easy way to store processed files. One of the challenges was that files required multiple processing steps across different services so local file storage was not the best option. The ideal solution would be to use something like S3, but given this was purely a proof of concept (POC) and being a lazy developer, I was looking for something simple and easy. The processing was happening via the most excellent Dramatiq library, which uses Redis. This got me thinking that Redis is actually pretty well suited for this use case; it stores binary data well, is relatively fast, is quite stable AND by using expiration param in SET it would handle house cleaning of old files.

For the sake of this example, we’ll create a simple Flask application that handles POST requests for file uploads and saves them to Redis, using a sha256 of the payload as the key:

from flask import Flask, request, Response, send_file
import base64
import datetime
import hashlib
import logging
import magic
import mimetypes
import re
import redis

app = Flask(__name__)
app.logger.setLevel(logging.DEBUG)

r = redis.Redis(host='localhost', port=6379, db=0)

@app.route("/", methods=["POST"])
def create() -> Response:
  file = request.files['file']
  encoded_file = base64.b64encode(file.read())
  key = hashlib.sha256(encoded_file).hexdigest()
  expiration_time = datetime.timedelta(hours=24)
  r.set(key, encoded_file, ex=int(expiration_time.total_seconds()))
  return str(key)

The example above will save the data to Redis and return the key. It sets the file to expire in 24 hours.

To retrieve the files, we’ll use a few utilities for attempting to determine mime-type/extension of the saved files and use Flask’s send_file method for the response.

Using that endpoint, GET requests made to /sha_of_file will return the file, with the appropriate mime-type/extension.

@app.route("/<key>", methods=["GET"])
def show(key) -> Response:
  encoded_file = r.get(key)
  if encoded_file is None:
    return "File not found", 404
  decoded_file = base64.b64decode(encoded_file)
  mimetype = magic.from_buffer(decoded_file, mime=True)
  file_extension = mimetypes.guess_extension(mimetype)
  temp_filename = f'{key}.{file_extension}'

  with open(temp_filename, 'wb') as temp_file:
    temp_file.write(decoded_file)

  return send_file(temp_filename, mimetype=mimetype, as_attachment=True)

…and there you have it. This ended up working quite well for an audio processing pipeline, quickly and easily being able to send audio files exceeding 100mb. The Dramatiq library has the concept of storing results between tasks baked in, but doesn’t solve for sending results back over the wire to the browser. Again, do not use this in a production app, but I would absolutely use this technique again for future POC projects.

Gnarly Technology Predictions for 2022

Nick Maloney — Wed, 23 Mar 2022 02:02:31 +0000

When we first started The Gnar Company I expected to spend a significant amount of time researching and implementing new technologies each year. What we’ve generally found is while new frameworks and stacks are constantly evolving, clients typically seek expertise and stability within a given tech ecosystem. In our 6 years of operating The Gnar we haven’t witnessed any major new frameworks/platforms get released and subsequently dominate the marketplace. If anything, we’ve seen more fragmentation, particularly in the NodeJS ecosystem.

Below is a quick summary of the technologies our shop will focus on using in 2022:

React

React will continue to remain the dominant JS library for UI in 2022. While there are a lot of promising contenders in the space (Svelte, Vue, Preact) there is simply too much momentum behind React for many larger organizations and consultancies to pursue other JS UI libs. Our entire team is proficient in React and we have extensive expertise building very complicated applications in React and we’ve been pleased with the continuous improvement in tooling.

React Native

React Native (RN) will continue to be our go-to tool for building mobile apps. Given our team's strong proficiency in React, it allows us to build on that knowledge and produce some great apps that are indistinguishable from native mobile apps. We also understand its limitations and will select RN only when it makes sense and recommend native when it doesn’t. Overall, mobile tooling is fickle and it changes even within the native ecosystems (ex: objective C > Swift, Java > Kotlin). Given those changes, I still don’t see a contender to supplant RN in 2022 but we are excited about some newer tech on the horizon like Strada.

Rails

Rails continues to surprise and impress me. There still isn’t another platform that we’ve found that can deliver the amount of features Rails provides out of the box while being easy to develop in and stable from an operations/maintenance perspective. We have projects that are well past the 5 year mark and they are rock solid - requiring very little maintenance aside from the occasional gem updates. Client demand for skilled Ruby/Rails developers has actually increased for us over the last few quarters. The Rails community and ecosystem is still very active so it will remain a tool our team reaches for.

Stimulus

Stimulus is another technology that has surprised me. When it initially came out, its usefulness wasn’t immediately apparent to me as we were heavily entrenched in React. What we’ve discovered is how well it integrates with both existing and new Rails projects. We’ve been able to completely overhaul UI on older Rails applications without retooling and we’ve found ourselves just using vanilla Rails + Stimulus vs immediately gravitating towards React, especially for relatively simple UI’s that don’t require a lot of complex state management. We’ve recently been coupling Stimulus with Turbo and are often able to deliver single-page application (SPA) level interactivity significantly faster and with fewer moving parts.

Typescript

You can take Typescript out of our cold, dead hands. It has become an integral piece of React/Javascript projects and is our default lingua in the JS ecosystem. We appreciate all the work happening in the language and get excited by all the new tools that are getting released.

NextJS

My outlook on NextJS is more opaque than the other stacks we use. We’ve used it on a few projects and overall had a favorable opinion of it but have yet to find its sweet spot. For purely front-end we default to create-react-app but have had some success with Next. On full-stack/back-end it lacks all the out-of-the box utilities and conventions Rails provides resulting in a lot of external dependencies and bike-shedding on how to structure complex back-end apps. I also think Vercel (maintainer of NextJS) is missing the bus by not having a database option tightly integrated within their stack, which makes it a non-starter for us on most projects. We've found DigitalOcean or Heroku to deliver the best out-of-the-box experience when deploying full-stack NextJS apps that require server-side data persistence.

Webpack

On a personal note, I hope 2022 is the year we no longer need to use Webpack. In the Rails ecosystem, I’m quite excited about jsbundling/import-maps and the complete removal of Webpacker as a dependency. On the JS side, I hope recent advancements in build tooling with TypeScript further reduce the need and complexity of transpiling and webpacker.

Summary

I don’t anticipate any bold technology projections or big changes happening at The Gnar Company in 2022. We are attracted to tools that produce quality results for our clients while still allowing us to innovate. I'm excited for the incremental improvements and enhancements we'll see within the core tech we use and will pay close attention to emerging trends in open source tech stacks.

Learn more about how The Gnar builds software.

Forem: Nick Maloney