Forem: Garret

Database Choices. NoSQL vs SQL

Garret — Sun, 23 Aug 2020 07:42:04 +0000

I have seen a few posts about people saying their favorite database and I am curious now why you might choose a NoSQL database over an SQL database or vice versa.

Personally, for basically every project, I choose to use DyanmoDB. There are a few reasons why but they all in a way boil down to cost and performance.

As far as the cost side DynamoDB offers an extremely generous free tier and I have never on my side projects reached passed the free tier capacity which is really nice. One of the said side projects I had 10,000 users and even with doing roughly 150 reads a second it was free.

Another aspect I love about Dynamo is how good it works with serverless because you never make a constant connecting with the database, every change is done through HTTP requests. Dynamo is also extremely performant no matter how high you scale it. Currently working with a client that wanted to switch off of Postgres to Dynamo and performance has been an insane improvement for them.

I feel one of the main reasons I do not choose an SQL database is because I do not really have any analytical reads or have access patterns that I do not know about. I think SQL performs best when you need those analytical type queries but it seems a lot of people prefer SQL databases over NoSQL and was curious to see the reasoning.

Best password authentication strategy?

Garret — Mon, 20 Jul 2020 19:16:58 +0000

I was curious based on your personal/professional opinion what is the best way to do password authentication?

I see so many different ways of handling users' passwords from salts hashing etc. and I wanted to know what you guys would do.

The App Stores...

Garret — Tue, 16 Jun 2020 20:29:05 +0000

App Stores... we need to talk. Google and Apple both have monopolies in the world of app stores for mobile devices. Since they have this monopoly they charge an outrageous fee for in app purchases, 30%! On top of that Apple also has some extremely strange ruling on apps they decide to run down of their app store, unless you cave into paying them.

In App Purchases

Apple and Google, for years have been charging a 30% fee for all purchases made on their app stores. A fee which you would never see in credit processing from services such as Stripe, Braintree, or similar processors. Why? Apple and Google are the only two companies which have a monopoly in their markets. Apple even more so with the extremely strict regulations on their store. Not only does Apple take a fee for app purchases, they also charge $100/year. Google also charges $25, however as a one time fee.

Apples Regulations

There has been many cases where Apple will try to remove apps off of the app store who do not subject themselves to the Apple tax by only allowing charges through their own website. An example of this can be found in a recent Twitter thread which prompted me to write this. https://twitter.com/dhh/status/1272968382329942017

In that thread you will see one of the tweets included an email from Apple saying they will take down the Hey app if they do not offer in-app purchases.

Yet there is also many apps that do the same thing, linking off to external webpage to purchase subscriptions that are still allowed on the app store.

Why?

Apple is most likely targeting the Hey app, because Apple sells its own email service iCloud mail. Therefore Hey would be considered a direct competitor to Apple, and of course Apple wants to keep all the profits for services to themselves.

Things Need Change

Things need to change. Apple should not be allowed this monopolistic approach to the app store. If we keep allowing big tech companies like Apple and Google to go down this road, they will own everything about us, and there is almost nothing we can do to stop it unless we act now.

My question for you is, what do you think we need to do in order to make this change happen? Do you think anything will change anytime soon?

Git and words...

Garret — Mon, 15 Jun 2020 00:09:44 +0000

So, as many of you know there are some articles about "master" in git being "racist." Obviously master in the case of git, and github is not racist if you look at the connotation of the word.

What is Connotation?

Conotation is the most important thing in language. A single word can have many different definitions that depend on how the word is used in a sentence.

The dictionary definition: "an idea or feeling that a word invokes in addition to its literal or primary meaning."

Example of Connotation

"The man is very blue"

This statement would literally mean that the man is colored blue. However, if you use the connotation with how it is used, you would know this means "the man is very sad"

The Word Master

The word master does have connection to the fact the slavery happened and people were in fact "masters" to slaves. I am not denying that. However, if you look into the connotation of how "master" is used in github, it refers to the "master copy." The word "master" in the case of git has nothing to do with people owning slaves.

My thoughts

Going down this road of people getting offended over every single word because it has any ties to slavery is a very bad road to go down.

If that is the way we go down we would have to stop using literally every word in the dictionary, because at one point our world was not the best. We stemmed all of our words with some sort of relation to how our world was back then. Rather than getting offended we need to understand the actual meaning of the word in the sentence it is used.

Going Down This Path

If we go down this path we would have to stop using words such as, main, control, git. Here's why:

Main comes from the "power" or "military control." If we look at whats happening in the world right now there is use of military control in the riots for police brutatility against people of color. Therefor main is a poor word since it has to do with the militaries control and power of the people.

Control literally means "the power to influence or direct people's behavior" which is what slave owners did to their slaves. They held power over everything they did. Therefor control is a bad word to use to define how git is a version control system.

Git literally means "an unpleasant or contemptible person." That is a pretty rude thing to use. Therefor git should not be used, and it should not be a version control system!

We could go on for days about every single word used for almost anything in the world. It is just not a good idea, and we instead need to understand that words have a connotation!

Where do you go to learn more advanced concepts

Garret — Sun, 14 Jun 2020 06:10:37 +0000

Hello everyone,

I was curious where is your goto place to learn some more advanced concepts or learn something more than just the basic tutorials of like chat rooms or todo lists etc.

I would say I am currently at the intermediate level of programming, and I know how to do all of those basic apps, and I know some advanced concepts. I want to be able to read or watch how people solve more advanced problems that require some more thought than the basic apps I see all over youtube.

Any youtube channels or company articles you always look out for? Any more specific websites dedicated to more advanced concepts?

The Problem With Mixer Clips

Garret — Sun, 14 Jun 2020 01:31:45 +0000

This post is more so of a rant about Mixer, and some of the things they do wrong, this post focusing on Mixer Clips. For those of you who do not know what Mixer is, it is a streaming platform similar to Twitch.

TLDR

Mixer does not actually create clips. They save references to the VOD of the stream for clips. If you delete a VOD all your clips will be unwatchable that happened in that stream.

How Mixer Create Clips

To start I wanted to go over how Mixer creates their clips. In order to even have clips enabled on your channel you have to have VODs enabled (taken from their Knowledge Base for Clips). This in it of itself is not the worst part, however, should not be a requirement to create clips in my opinion.

Once you have VODs enabled you can now create clips. When you request to create a clip Mixer creates a clip m3u8 file that references back to the timestamps in the VODs. The file looks like so (shortened):

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-ALLOW-CACHE:YES
#EXT-X-TARGETDURATION:4
#EXTINF:2,
https://clips-content.mixer.com/vods/bc379e2c-7ab2-4941-8bb4-6a8cfddde647/0002244.ts
#EXTINF:2,
https://clips-content.mixer.com/vods/bc379e2c-7ab2-4941-8bb4-6a8cfddde647/0002245.ts
#EXTINF:2,
https://clips-content.mixer.com/vods/bc379e2c-7ab2-4941-8bb4-6a8cfddde647/0002246.ts
...

Note how the URLs point back to the vods folder. Rather than creating a clip, Mixer instead points back to the VOD. This means if you delete a VOD your clips will not be playable. This also means that the clips cannot last longer than the VODs (which they only save for 14 days for non-partners).

Why Does Mixer Do This?

This is a question that I still have myself. My only guess as to why Mixer would do this is maybe to save money on storage costs. This would be a pretty dumb reason for the limitations this brings in, however.

How Should Mixer Fix This?

This is such a simple fix that would take less than 3 seconds for any developer to make. They already know the exact file parts they need from the VOD. All they would need to do is copy that data over into a clips folder. This would allow user to delete their VODs and still keep their clips, and it would also allow Mixer to save clips for longer than the minuscule amount of 14 days.

The question here though is does Mixer care? My guess, probably not. Mixer clips have been in "beta" for 3 years, and they still just point references to the VOD. I have not seen a bigger joke than this. From my experience in development, I would be willing to bet the reason the developers do it like this is because of the higher ups at Mixer who just do not understand, and do not care.

Further on Clips

Since clips for Mixer has been my passion as I am working on SmartClips, which is real clips, I wanted to go a little bit further on how Mixer can actually make a better clip system.

It is my opinion that VODs should not have to even be enabled for users to create clips. While most streamers will have VODs enabled it is still a limitation that shouldn't need to be there.

The change Mixer would need to make is rather than copying over the video parts from the VODs they would just need to copy over the files from the HLS stream while the user is still live, before those files are deleted. This is exactly what I do on SmartClips, and is not a hard implementation by any means.

Conclusion

Do you think that the higher ups at Mixer will ever realize the changes they need to do? I hope they will figure it out eventually. I want Mixer to succeed, that's why I started making 3rd party tools for Mixer (which is also a pain to do, but that's a rant for another day).

Node Express and WebSockets (WS) Boiler Plate

Garret — Fri, 22 May 2020 15:43:19 +0000

In this post I will be sharing with you some of the practices I follow, and how I create my API's using Node, Express, and WebSockets. I have recently publicly published the boiler plate that I use at https://github.com/itsUnsmart/express-ws-boiler so I decided to go over it.

Project Structure

The main thing I think about when setting up my projects is readability. If I cannot read through the project structure easily to find what I need it can be a pain.

In order to make things easy I split everything into its own directory. The Express API goes into the web directory. The WebSocket goes into the sockets directory. Finally any helpers I made need go into the helpers directory. In this example I have only one index.js file in the helpers, however, in bigger projects I recommend making multiple files with a more specific name to the type of helper functions if needed.

Simplified:

structure: {
  "/helpers": "Any helpers needed.",
  "/sockets": "Anything related to the websocket server.",
  "/web":     "Anything related to the express web server."
}

Express Specifics

Specific to Express I have one main index.js file which starts the server, and sets up all the routing, as well as any other configuration needed. In order to make the routes in a clean structure I create a routes folder which contains an index.js that will parse out the routes into a single object for the main server file.
Example:

module.exports = {
  example: require('./example')
}

In the above example it takes a route file called example.js and maps it to the "/example" path. The way the mapping works in the main index.js file is it require the route index file, and runs over the object keys.

const routes = require('./routes')
Object.keys(routes).forEach(key => {
  app.use(`/${key}`, routes[key])
})

This means now any request which has example in the path will get routed to the code in the example.js file.
Meaning our simple hello world file:

const { Router } = require('express')
const router = Router()

router.get('/', (req, res) => {
  return res.status(200).json({
    hello: 'world'
  })
})

module.exports = router

Will run when we go to http://localhost:3000/example and display:
{"hello": "world"}.

WebSocket Specifics

Specific to WS I have one main index.js file which starts the server, and sets up all the methods, as well as any other configuration needed. In order to make the methods in a clean structure I create a methods folder which contains an index.js that will parse out the methods into a single object for the main server file. This file is identical to the Express equivalent in the routes folder.

module.exports = {
  example: require('./example')
}

The difference is now in order to call the method I look for the method to call in the WebSocket message inside of the method parameter (Note that I code all WebSocket message to be JSON).
Example:

const Methods = require('./methods')

// inside ws.on("message")
if (typeof data.method === 'string' && Methods[data.method]) {
  Methods[data.method](WSS, ws, data)
}

Now if we were to connect to the socket and send the following message:
{"method": "example"} it would run our example method and return a response containing {"hello": "world"}.

Full structure

After this out full project structure would look like so:

structure: {
  "helpers": {
    "index.js": "The main file combining all other helper files if any."
  },
  "sockets": {
    "index.js": "The main server and configuration file.",
    "methods": {
      "index.js": "The main methods file which combines all methods."
    }
  },
  "web": "Same as sockets replacing methods with routes."
}

Errors

After working on projects that integrate with 3rd party services I truly understand how useful good error messages can be. I always make sure to give as much detail into why an error happened so anyone using it can understand what went wrong.

I do this by giving my errors an easy to read error message, a machine readable error code useful for doing switch statements on. As well as context into the error(s) that occured.

Example of a 404 error:

{
  type: "error",
  message: "Not Found",
  code: "not_found",
  context_info: {
    errors: [
      {
        reason: "invalid_path",
        message: "The requested path could not be found",
        data: "/invalid/path",
        location: "path"
      }
    ]
  }
}

This error may be a bit overkill for a simple 404 error, however, it gives a user or even yourself a good insight into exactly what went wrong in the request. This is very useful, and makes debugging super easy.

Easy Install

Would you like to use this boiler plate in your projects? The easiest way to install it into your projects is by using degit.
Run the command degit itsUnsmart/express-ws-boiler, and it will clone the repository into your current directory.

Conclusion

That is the basics for how I structure my Node, Express, and WebSocket applications, and how I handle things such as errors. I am curious how do you structure your projects, and what do you like or dislike about how I structure mine?