Forem: Alex Jacobs

How to set up a mechanical keyboard with super light-touch switches.

Alex Jacobs — Tue, 18 Feb 2020 01:33:30 +0000

How to set up a mechanical keyboard with super light-touch switches.

My journey to set up the easiest touch and shortest travel possible on a keyboard led me to a Planck-EZ keyboard with switches with modified springs, but it won’t stop there.

This article is aimed primarily at people with unusual keyboarding needs, such as those dealing with RSI (repetitive stress injury ). There is a difficult learning curve to using super light touch switches, during which time you will likely see a lot of extra home row characters sprinkled throughout your documents, as you rest your hands on the keyboard and unintentionally actuate the keys. j

Background

My life as a keyboard-using person temporarily ended for a few years, starting in early 2016, when the cumulative effects of RSI caused typing to be too painful, and my arms too achy to comfortably carry out some basic household tasks, or even continue playing much guitar.

In order to continue working in my field as a software engineer, I then became a voice coder, using VoiceCode (extinct) and then Talon, and got good enough with the software to be able to work professionally, largely hands free.

Eventually due to my professional role switching to teaching and mentoring, my needs for keyboarding increased. I had healed quite a bit from my injuries, but knew that full time keyboarding would likely lead me down the same path of eventual physical injury.

Daydreaming about the possibility of being able to use a keyboard again, my research was inspired by the way I’ve always set up my guitars, with the lowest “action” (string height off of the neck) possible, leading to the least finger strain.

That search led me to a slack posting by a member of the Talon community who mentioned they had made a super light-touch keyboard by cutting the springs in their Cherry MX reds to bring the actuation force down.

My research that I’ve documented in this post included buying a small wire cutter and trying some experiments with spring cutting, but I found it lacking in precision, enough to want to try something else. It turns out that there are now precision springs with a huge variety of actuation forces easily available online. You can view a list at https://www.spritdesigns.com/mx

Another piece of the puzzle was researching key switches that would work well with reduced spring forces. I found a variety of low-actuation distance “speed” switches that triggered a key press around 1.1mm. For comparison, a normal Cherry MX red actuates around 2mm. If you have a ruler with millimeters on it and actually look at the difference here, it would seem to be ridiculously negligible. I thought the same. But… here is an important distinction that made a big difference in terms of my personal need. Not only did I want keys that actuated by basically breathing on them, I also wanted to lower the amount of total travel that was possible during a keypress, so that even if I had the hand strength to type relatively hard on a particular day, the design of the key would limit the amount of finger movement I was doing.

After using a bunch of super-light modified Gateron Clear switches, I found that their greater overall travel of 4.0mm was irritating my RSI. The Kailh Speed Silvers have an overall travel of 3.5mm which by itself is a ridiculously small difference, but I further lower the key travel with 2 stacked O-rings on the key cap (details below). Over hundreds or thousands of key presses over the day, these modifications make a big difference for my RSI.

The final piece of the puzzle was locating a keyboard that would allow me to experiment with different key/spring combinations. I landed on the Planck-EZ which seemed too good to be true. It’s a Planck keyboard with the ability to hot-swap your key switches. I ordered one with a set of Kailh Speed Silver switches installed. I also bought some different switches to experiment with: Kailh copper switches, and some Gateron clears.

The Kailh copper switches were immediately out for me. The tactile bump was totally counter to my goal of lightest touch. They were irritating to my RSI, and with the lighter springs I was using, didn’t fully rebound after pressing. They helped me realize that I should only focus my further research efforts on linear switches.

The Kailh Speed Silver switches have an actuation point around 1.1mm. They were the switches that I ordered with the board, and I swapped the springs with 35g Spirit MX springs. They actuate at 25g. The 30g Spirit MX springs I also bought were not springy enough to fully ‘reset’ the Kailh Speed Silver. If I were to lube the switch, it might be sufficient, but I have not tried that yet.

You can follow this link to my layout on the Ergodox EZ Configurator site.

Here is how I have arranged the modified switches on my Planck-ez

Here is the key for the colors above:

Gateron clear springs are sold with 35g springs installed. However, that is the actuation force, which means that the finger force curve goes up from there. The Spirit MX compatible spring that I swapped into the Gateron Clears are sold as 30g, but they have a 20g actuation force. The 30g Spirit springs were insufficient to reset the Kailh Speed Silvers, but work very well for the Gateron Clears, making them incredibly light to actuate.

The modified Gateron Clears, with their slightly longer actuation point serve as a very useful “guard row” against accidental key presses that come from just moving your hands anywhere near the board. You may not be aware of how much pressure you put on your keys while you are subconsciously locating the right key to press. When you remove virtually all resistance from the keys with the modified springs, and lower the actuation point (on the letter keys) to 1.1mm with a Speed series switch, you will learn just how sloppy most of us are when it comes to locating keys.

Swapping Springs

The Gateron Clears are easy to open up with an MX Switch Top Removal Tool.
The Kailh switches require some semi-dangerous knife work to open up. Here is a good set of photos showing the process. Instead of a large blade, I use a thin box cutter blade similar to the one shown below. Either way, you are at a good risk of blood letting.

O-rings

I was able to further reduce the total key travel distance an estimated 1 or 2 mm with O rings. They also make the keyboard much quieter. They also raise the height of the keys off the board by a couple millimeters. I bought a set. They sit on top of each other nicely. I put the large ring on first, push it down with an unscrewed ball point pen body, and then put the small one on top. I make sure not to push down too hard on the key caps when attaching them to the stem. Yes, this means that they keycaps fall off occasionally if I carelessly drag something across the edge of a key when moving things on my desk.

The modified Gateron Clears with O-rings actuate much closer to the end of their total key travel than the modified Kailh Speed Silvers. I can actuate the Speed Silver with a super light touch and sometimes not even bottom out. The Gaterons are almost bottomed out (with O-rings) when they actuate.

Here are the sizes of O-rings that I am using:

Large: OD 9 x 2.5mm
Small: OD 8 x 1.5mm

Larger O-ring on the bottom, smaller one layered right on top.

You can even see in the following photo, on the spacebar, which only has 1 large O-ring, that it sits lower than the other keys, which all have 2 O-rings. The O-rings significantly affect key height, overall travel, and sound. I tried removing them recently and found I’ve really gotten used to the feel. I found the travel and noise irritating. Many mechanical keyboard fans love the noise and feel of their keys, without O-rings. I probably would as well, but due to RSI, my goals unfortunately have to focus on ergonomic purpose over aesthetics.

This is the type of box knife I use to open the Kailh Speed Silver switches.

Here is my board with all the keycaps removed and placed on the table above the board and exposed switches. Red sharpie dots were added to the modified switches to be able to identify the ones with the swapped springs.

My weirdo layout was inspired by my TypeMatrix 2030, as an attempt to make more space for my wrists. It’s still not enough for me.

The following photos show the customized LED settings that correspond to the various programmable “modes”.

The “move” layer

The “adjust” layer

The “raise” layer

The “lower” layer

The “num” layer

Where I plan to go from here

So it turns out through researching for this post, you can go even lighter. Check out the gboards link below in Further Reading. They’re using low profile Kailh Choc switches with 12g (!!) switches.

I don’t think I’m ready yet to move further down the rabbit hole to even less keys than I’ve gotten skilled with using on the Planck. The Planck-EZ is an incredible keyboard, and has been my gateway into the world of mechanical keyboards. My persistent problem with it is that, similar to any non-split keyboard, my hands are stuck too close together, which is causing wrist pain for me. Even with my weirdo layout that puts an extra row of keys between the left and right hand keys, the keys feel too close together. I also compared the Planck-EZ to my mac laptop keyboard, and the Planck-EZ offers more width than the laptop keyboard. It basically seems that any solid keyboard is going to bother my RSI.

So, next, I want to purchase an ErgoDox-EZ, which is a split keyboard that will allow me to space the right and left halves as far apart as I need to to remove the wrist bend that is bothering me so much. And, it will of course allow me to make the same switch mods that I was so easily able to add to the Planck-EZ. My plans for setting up the ErgoDox-EZ are to minimize use of the thumb clusters. I imagine this will change, but I initially only plan to use the most easily accessible thumb buttons for a space bar, and continue to use the ErgoDox as if it is a Planck. The beauty of learning to type and code efficiently with the Planck is that I’ve found it really great to be able to use the programmability of the firmware to “move the keys to my fingers rather than the other way around”. I will initially try to keep using my super compact layout that I’ve mastered with the Planck, but with the ergonomic benefit of the ErgoDox split keyboard. I’ll likely purchase the tent kit as well to experiment with different tilting angles to see if I can provide even greater ease to my fingers.

You can check out my ErgoDox-EZ layout here. At the present moment, I’ve programmed it without even having the board, so I’m sure it will change a lot after actually having the board.

Acknowledgements and disclaimer

I’m grateful to the hundreds of keyboard enthusiasts that have shared their knowledge on online forums, and the manufacturers that created the parts that went together to make my board. Also, this is description of how to make the lightest touch and shortest travel keyboard that works for my personal needs. The world of mechanical keyboards is very wide, and there may be ways to go beyond what I’ve found in terms of shorter travel, or lighter keys. One method is to use a “chorded” keyboard. And finally, nothing in this article should be taken as medical advice. If you have, or think you have RSI, please seek professional medical help. It’s a serious condition that can take a frustrating amount of time and resources to figure out how to manage and / or heal.

The Infinite Coastline Paradox and the importance of working through technical challenges on your own

Alex Jacobs — Fri, 11 Oct 2019 20:04:00 +0000

The Infinite Coastline Paradox and the importance of working through technical challenges on your own

There are unknown worlds to explore, hidden within the seemingly simplest of tasks

https://www.pexels.com/photo/aerial-view-beach-beautiful-cliff-462162/

According to the Infinite Coastline Paradox, the length of the coastline increases as our measurements become more precise. Imagine measuring the length of the coastline around Great Britain. If you only had a meter stick to lay end to end around the entire coast, you would get a seemingly accurate measurement. Now imagine you were to re-measure the coastline with a tiny centimeter stick. Now you can fit your measuring device into many more nooks and crevices around the edge of the landmass, which would increase the length of your total measurement considerably. As you can imagine how this continues, the smaller your measuring device, the more accurately you can fit it into smaller and smaller details of the coastline. Which means the length of the coastline is theoretically infinite, as your measuring device becomes more and more accurate.

Now let’s apply this concept to the idea of solving technical challenges. Perhaps the best way to transfer to concept over to discrete challenges is in the context of “toy problems” which are the “short stories of coding challenges”. These are discrete algorithmic tasks that are generally meant to be solved in a single session, and are often used as gating assessments during technical interviews.

Most published toy problems also have published solutions and video lectures on how to solve them. A common pitfall would be to turn to a video or solution before spending time trying to solve it on your own.

When first looking at and estimating the steps required to solve a technical challenge, you might compare that to looking at a satellite or aerial photo of a large land mass. The relative size and area of the structure are apparent, and a rough estimate can be made as to what sort of effort it would take to circumnavigate the “landmass of the problem”.

If you then take the shortcut of looking at the solution, you might be able to memorize the steps and repeat a solution to the problem at hand. But that is kind of like measuring the coastline of the landmass with a large stick. So what was lost by not working through the problem on your own? Well, when you really get inside a problem that you thought required only 5 steps to solve, that’s when you start to look at it more closely, like bringing out a more accurate measurement device. At that point, you might find that the 5 steps each had 5 steps of their own hidden inside. And what if each of those 5 steps had multiple additional steps?

To work through all those steps on your own, you get to explore inside hidden layers of complexity and explore the potentially vast surface area, hiding treasures of discovery and learning. All of those hidden layers that you go through on your own will also inform your problem solving process going forward, informing all of the future challenges you might encounter as well.

This blog post is a transcription of a motivational talk I gave to one of the groups of students I currently work with as a full time Technical Mentor at Hack Reactor.

Installing Sendy as a replacement for Mailchimp — 2019 version 4 update, the complete guide, with…

Alex Jacobs — Fri, 03 May 2019 20:40:09 +0000

Installing Sendy as a replacement for Mailchimp — 2019 version 4 update, the complete guide, with automated backups to S3, and https configuration

This information is compiled from many sources and is the complete record of my successful install. Start here to save yourself a lot of time.

This install guide you are now reading is for Sendy version 4. It was tested on Sendy version 4.0.2 with the latest server config available on Digital Ocean

The version 3 install guide is outlined in my previous article. It is nearly identical to this install guide, except it was tested on version 3.1.2.3

Intro and motivation for the change

I had a new-music-release email list hosted on Mailchimp to send infrequent updates. I learned the hard way that if a basic account is unused for two years, it’s deleted and the old username is permanently blocked. Just vaporized. No notification sent before the deletion, and no easy way to get the unsubscribe data back either. According to Mailchimp’s compliance department, this is a “security measure”. I still had a backup of the original subscriber emails, but not the important records of who previously unsubscribed from my list over the years. So while I could easily reimport my list again with Mailchimp or another host, there is a “spammer blacklist” risk related to sending email to someone who had unsubscribed in the past.

After jumping through hoops with customer service, I was able to get an export of all my previous Mailchimp data, but I was convinced that it makes more sense to find an alternative to the industry heavies such as: Mailchimp, Sparkpost, ActiveCampaign, AWeber, or Emma.

Fortunately, I had recently scanned the archives of Wes Bos’ web-dev blog posts, and remembered the catchy title “From MailChimp to Sendy — how I saved $2,400/year on my email list”. With my relatively small email list, the change wasn’t motivated by saving money, but by hosting my own list to avoid another surprise account deletion or other issue related to not controlling my own data. Wes’ article convinced me to check out Sendy. It’s a self-hosted mailing list software solution. But his article didn’t describe how to install it.

After purchasing it and combing through the product support forums, it looked like DigitalOcean was a great choice for hosting based on both price and features. While reading many, many pages of documentation and stack overflow posts,

I took detailed notes on my installation process so I could present a complete and detailed record of the steps you can take to get a running instance of Sendy.

I will also cover how to secure your Sendy server traffic with HTTPS with a free Certbot certificate, and set up automated mysql database backups to an AWS S3 bucket at the frequency of your choosing.

How to — start here

Read about and purchase Sendy from the developer here.
If you don’t already have an account with DigitalOcean, signing up through this link will get you $100 in free credit to use over 60 days.
You will need an AWS account.

Since the developer of Sendy says “if it doesn’t work out, we’ll refund you”, you can try this whole experiment without financial risk. I set up my Sendy installation on a $5/month standard droplet. The only other costs are the Amazon SES e-mailing, which is $0.10 per thousand e-mails, and your Sendy software purchase which is a one time fee with no additional costs until major version updates.

Caveats

Nowhere again in this post will I use the word ‘simple’. But I do believe that following this guide will make your installation process as easy as possible.

I recommend browsing it first, and then following the steps in the order they are presented. Make sure you have extra hairs available in case you need to pull some out.

I reinstalled Sendy by following and validating all of the instructions that I’ve documented here.

The only constant is change, and you may find yourself needing to jump through some unexpected hoops, just as I did during my initial install process. Hopefully everything that I have documented will be current and useful for you. If you found something that has changed, please describe it in the comments and I will update accordingly.

I have made extra sure to implement a quality guide that is up to modern standards by making sure to use some instances of dry humor, at least one sci-fi movie reference, and occasional emoji insertions. As an added bonus, I used the example domain www.contoso.com throughout the guide. I also documented the actual keys/secrets, and ip addresses I used while setting this up so you won’t have the extra mental baggage of trying to parse the syntax of things like .

I’ve put in the extra effort to describe various scenarios you may encounter, and also to describe why something is done so you’ll be able to make informed choices if some step has changed since I documented my process.

This is totally doable , so read this guide, warm up your google chops, and I wish you the best on your installation process!

It may be helpful to breathe deeply, drink water, and walk around every once in a while.

Documentation conventions

Text inside a block like this represents something you should type or something visible on the browser page or terminal display.

Text inside a box like this represents the terminal display

$ This represents your local terminal (after $)

# This represents the remote shell (after #)

Sail the DigitalOcean: create your droplet (cloud based virtual machine)

Log into your new account, and verify that you have the free $100 credit. If you are a new customer but you did not sign up through a referral link, send an email to customer service with the referral link. In my experience, they will credit you. I signed up right after finding somebody else’s referral link, and when I e-mailed it to them, they gave me the credit. If you need a referral link, here is mine: https://m.do.co/c/c29fea8989aa

Create a new ‘project’

The next window will ask you: Move resources into the new project? click Skip for now

Next, create your droplet and I will detail the settings below:

Click the blue button, or the green drop down

Click Droplets and then follow the instructions below for settings

The next set of options is Choose an image . Select Marketplace

For ease, we’re going to install on a preinstalled LAMP stack on Ubuntu

At the time of writing, the LAMP Marketplace app was version 18.04. The number is the version of Ubuntu (flavor of linux distribution). Newer versions (higher numbers), if available, may present a similar installation experience. If you have some experience working with the command line and aren’t afraid to google things, go ahead and install with the newer version.

Latest release at time of writing

Select the Standard plan

Select the left arrow to see smaller droplet sizes

Choose a size. For my currently puny mailing list, I’m confident with the Standard plan and the smallest size virtual machine and I chose $5/mo. In the future, you can scale cpu/ram up and down as you wish. You can scale disk size up, but not down again.

Add backups? I chose not to, since you can make and store manual snapshots. However, automating backups on a $5/month container will only cost you $1/month, so you may opt for this in order to save having to remember to make snapshots.

Add block storage? Not required for this tutorial

Choose a data center region. I chose New York. You may have legal obligations to host your data in a particular region depending on the type of data you are storing, so choose accordingly.

Select additional options? I left all of these unchecked

Add your SSH keys. I did not do so, and therefore use a password when I sign in via SSH. The initial root password is sent via email. This tutorial discusses using the password, so skip that step for now. SSH keys allow you to log into the remote server without needing to authenticate with a password.

Finalize and create. Make sure that you are only deploying 1 droplet, and rename it something sane (such as sendy-droplet in my example below), and make sure it is in your newly created ‘project’ (which it will be, unless you already have multiple ‘projects’)

Click the “Create” button. You will then be redirected to your ‘project’ page, and see a status bar of your droplet creation progress. It takes a couple minutes.

Followed by its successful creation, your login details and password have been e-mailed to you.

Congratulations, your newly spawned creation is running in the cloud. Now perform a sanity check by entering the ip address into your browser and you should see something like the following:

The “Quick”start Guide button points to this document on the Digital Ocean website, which details what software your One-Click installation contains, as well as some configuration information.

Let’s log into the new droplet and set it up!

You’re going to need the email you just received from DigitalOcean with the root password for your droplet. We will login with ssh (secure shell).

$ ssh root@157.230.133.78

Which will be greeted by the following appropriately paranoid message

The authenticity of host '157.230.133.78 (157.230.133.78)' can't be established.
ECDSA key fingerprint is SHA256:0U9apoBN+7h1ajk2+KZoYQiL35X4Zjp+9MQLC71mX7y.
Are you sure you want to continue connecting (yes/no)?

You do want to continue connecting, so type yes and you should then see the following:

Warning: Permanently added '157.230.133.78' (ECDSA) to the list of known hosts.

root@157.230.133.78's password:

The terminal will not show the characters you enter when you type or paste the password. Enter the password and then press return and you should be logged in and see some boilerplate text describing some of the default configurations of your server.

You will be prompted immediately to reset the root password. First enter the current password that you just used. Then enter a new secure password that you immediately record somewhere safe. You will need to enter your new password again right away to confirm, and then you will be logged into your droplet, a Linux Ubuntu virtual machine. The normal pwd ls cd mkdir etc terminal commands work as you would expect.

Update the list of available software

# apt-get update

will update the list of available packages and their versions, but not install or upgrade any packages. It will create output similar to what is shown next:

Get:1 [http://security.ubuntu.com/ubuntu](http://security.ubuntu.com/ubuntu) bionic-security InRelease [83.2 kB]
Hit:2 [http://ppa.launchpad.net/certbot/certbot/ubuntu](http://ppa.launchpad.net/certbot/certbot/ubuntu) bionic InRelease

\*\*\*[Lots more stuff …]\*\*\*

Get:37 http://mirrors.digitalocean.com/ubuntu bionic-backports/main Translation-en [448 B]
Get:38 http://mirrors.digitalocean.com/ubuntu bionic-backports/universe amd64 Packages [3468 B]
Fetched 14.0 MB in 5s (3035 kB/s)
Reading package lists... Done

Upgrade available packages

apt-get upgrade

Accept the confirmation message and you will be updated to the latest version of all installed packages.

Install some new software needed by Sendy

Next we need to install a couple software packages that are not part of the default 1-click installation, but will be needed by Sendy. The first one is php-curl

figure out which major.minor version of curl you need

php --version

will return a bunch of output including the (major.minor.patch) version that is installed on your server. In my case it’s 7.2.10 which we can use in the next command, but we only need the [major.minor] numbers.

# apt-get install php7.2-curl

If in doubt what version to install, type # apt-get install php7. and then press tab a couple times to see an auto completion of the various software packages that are available for the major version of php that you are using. Mine auto-completed with # apt-get install php7.2-curl and then I clicked enter to install and also accepted the confirmation dialogue that may pop up.

Here’s the next one to install:

# apt-get install php-xml

accept the confirmation dialogue that may pop up and then restart the droplet’s web server with:

# systemctl restart apache2

Next, Create New Virtual Host Files

Virtual host files are the files that specify the actual configuration of our virtual hosts and dictate how the Apache web server will respond to various domain requests.

In English: we need to set up the server to respond to the domain name that you are going to assign to it when you create a DNS record with your domain registrar. Example: if you run the website www.contoso.com, and you plan to install Sendy at sendy.contoso.com, you need to configure the apache web server on the droplet to serve the appropriate files when you visit the sendy domain. These files are hosted in /var/www/

Create The Directory Structure for the files to be served

# mkdir -p /var/www/contoso.com/public\_html

Create New Virtual Host Files

These are the files that tell the Apache web server how to respond to web requests. We are going to copy and configure the default host file (named 000-default.conf). The following shell command is one line

# cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/contoso.com.conf

Now are going to edit this file using nano or vim. If you’ve never used vim, I would not start now. Nano is fairly user-friendly and responds to arrow key movements to move the cursor around the screen just like a familiar text editor. After editing, press control and x together to save the file and return to the terminal.

# nano /etc/apache2/sites-available/contoso.com.conf

The file that appears should begin with

<VirtualHost \*:80>

and end with

</VirtualHost>

Don’t worry too much about what’s in the middle. It may vary a bit from what you see below.

Leave everything else as it is. We are going to add one directive (line of configuration code) after the tag:

ServerName sendy.contoso.com

and modify two directives (lines of configuration code). First, change theDocument Root directive to:

DocumentRoot /var/www/contoso.com/public\_html

and then remove the word “Indexes” from the line that looks like this:

Options Indexes FollowSymLinks

so that after you edit it looks like this:

Options FollowSymLinks

And then also modify the text inside the tag to:

<Directory /var/www/contoso.com/public\_html/>

The final result should be something similar to the following. Don’t worry about all the other lines matching exactly:

<VirtualHost \*:80>
        ServerName sendy.contoso.com
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/contoso.com/public\_html

        <Directory /var/www/contoso.com/public\_html/>
            Options FollowSymLinks
            AllowOverride All
            Require all granted
        </Directory>

        ErrorLog ${APACHE\_LOG\_DIR}/error.log
        CustomLog ${APACHE\_LOG\_DIR}/access.log combined

        <IfModule mod\_dir.c>
            DirectoryIndex index.php index.pl index.cgi index.html index.xhtml index.htm
        </IfModule>

</VirtualHost>

This is what mine looks like, but I’m installing Sendy as a subdomain of my personal website domain. If you plan to run Sendy as a standalone server (something like www.contoso.com),,) then your final result should probably be: (notice the extra directive of ServerAlias)

################# WHEN NOT INSTALLING AS A SUBDOMAIN

<VirtualHost \*:80>
        ServerName contoso.com
        ServerAlias www.contoso.com
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/contoso.com/public\_html

        <Directory /var/www/contoso.com/public\_html/>
            Options FollowSymLinks
            AllowOverride All
            Require all granted
        </Directory>

        ErrorLog ${APACHE\_LOG\_DIR}/error.log
        CustomLog ${APACHE\_LOG\_DIR}/access.log combined

        <IfModule mod\_dir.c>
            DirectoryIndex index.php index.pl index.cgi index.html index.xhtml index.htm
        </IfModule>

</VirtualHost>

If you edited using nano, when you use control + x to write and exit, you will confirm Save modified buffer? with Y and then enter again to accept the File Name to Write.

Activate The Virtual Host File (and deactivate the default host file)

# a2ensite contoso.com.conf

this will show some confirmation message including a notification about restarting apache. We will do that in a moment.

First you need to deactivate the default virtual host file with:

# a2dissite 000-default.conf

this will also show some confirmation message.

And as the confirmation messages stated, it is now time to reload the Apache server to enable the new Apache configurations.

Reload the Apache server

# systemctl reload apache2

This reloads the Apache web server only , and you should remain logged into the shell.

If you visit the ip address now, you should see something like the following:

Which is what we want. This is the result of removing Indexes from the Options directive inside of the tag.

**If you see something like this:**…

…then the Options directive wasn’t properly updated. Check the syntax above. After you correct it, restart Apache with # systemctl reload apache2

When you reload the browser, it should show the correct Forbidden message. This is going to keep the public from peering into your /uploads folder after you install Sendy.

Regarding HTTPS traffic to the droplet

Since the 1-click droplet is preconfigured with Certbot, securing your servers web traffic with SSL can be done with just a few commands. However, before we generate a certificate for the server, we need to set up DNS routing to the server so that we make sure we’re generating the certificate for the right domain. Detailed HTTPS instructions will follow, at first we will cover how to get the proper DNS records configured for your droplet.

DNS Routing

Log into your domain name registrar so that we can create some records that will point to your new server.

In my case, for this example server, I would be adding a new record for the DNS records of my domain contoso.com. I would create an “A” record. The A name record would be: sendy, and the value: 157.230.133.78

I just tested this out with my registrar, and within a couple minutes my subdomain resolved to the server and I saw the same image as when accessing the ip address directly.

If you plan to install a standalone server without a subdomain, then the name of your “A” record would be www, and you would create a second “A” record for the bare root domain (contoso.com in this example). The syntax for this differs by registrar, and may be “@”, or sometimes just a blank field. The value of both of these “A” records would be the ip address of your droplet.

Ideally it will only take a few minutes for your DNS changes to propagate. Feeling impatient or wondering if it worked? Check this out (and update it with your own domain name):

[https://www.whatsmydns.net/#A/sendy.contoso.com](https://www.whatsmydns.net/#A/sendy.contoso.com)

This is a pretty cool resource that shows you the status of the DNS propagation. My changes made it around the world in just a few minutes.

Enable HTTPS with SSP certificate installation with Certbot

As part of your 1-click installation, there is an automated SSL certificate generator already installed called Certbot. Now that we have the virtual host files set up correctly, and DNS records correctly configured we can run it.

For my example server, I only need to generate a certificate for the sendy subdomain, and this is what I would run:

# certbot --apache -d sendy.contoso.com

If you are running it for a standalone Sendy installation, you will run it as follows (for the www subdomain, and root domain):

# certbot --apache -d contoso.com -d [www.contoso.com](http://www.contoso.com)

After you run the command, you will be asked

Enter email address (used for urgent renewal and security notices) (Enter 'c' to cancel):

I personally consider entering an email address that you actually check a very good idea so that you can receive notifications in case the automated SSL certificate renewal process doesn’t work. If your certificate expires, browsing to your formerly secure site will not show your site, but instead a “Your Connection Is Not Private” warning page.

If you choose to generate your certificate without the email address you can do so with the command that’s given after you press c at that prompt.

After this step, you will be given the terms of service link and asked to agree. If you agree, you will be asked whether or not you want to share your email address with EFF.

After you answer that, you will see a bunch of output related to the installer trying to access your domain. If your DNS propagation has not completed, or there was an error with how you entered your DNS records, this step will fail.

The output so far should be something like:

root@sendy-droplet:~# certbot --apache -d sendy.contoso.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for sendy.contoso.com
Enabled Apache rewrite module
Waiting for verification...
Cleaning up challenges
Created an SSL vhost at /etc/apache2/sites-available/contoso.com-le-ssl.conf
Enabled Apache socache\_shmcb module
Enabled Apache ssl module
Deploying Certificate to VirtualHost /etc/apache2/sites-available/contoso.com-le-ssl.conf
Enabling available site: /etc/apache2/sites-available/contoso.com-le-ssl.conf

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel):

At this point you are asked whether or not you also want to allow HTTP traffic to your server, or redirect it to HTTPS. I selected 2 and had the installer create a redirect from HTTP to HTTPS. When it comes time to configure Sendy, you can specify whether or not you prefer HTTP or HTTPS. Your Sendy configuration file must specify one or the other.

Here is the remaining output:

Enabled Apache rewrite module
Redirecting vhost in /etc/apache2/sites-enabled/contoso.com.conf to ssl vhost in /etc/apache2/sites-available/contoso.com-le-ssl.conf

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations! You have successfully enabled
https://sendy.contoso.com

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=sendy.contoso.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/sendy.contoso.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/sendy.contoso.com/privkey.pem
   Your cert will expire on 2019-04-12. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew \*all\* of
   your certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let's Encrypt: [https://letsencrypt.org/donate](https://letsencrypt.org/donate)
   Donating to EFF: [https://eff.org/donate-le](https://eff.org/donate-le)

If you want to feel extra secure that this worked, go ahead and follow the link to the test site that was just output after the You should test your configuration at: phrase. For this site it would be: https://www.ssllabs.com/ssltest/analyze.html?d=sendy.contoso.com

Now refresh or navigate to your server url, with your browser (go to your site, such as https://www.contoso.com, NOT the ip address), and you should see the:

💥

Your secure server is up and rolling!

Let’s take a look at the ssl site configuration .conf file and make sure it properly copied over the directives from the original non-ssl version.

# nano /etc/apache2/sites-available/contoso.com-le-ssl.conf

Should bring up a page that looks pretty much like this. Look to see that the ServerName / DocumentRoot and Directory directives look similar. And also that the Options directory doesn’t have the Indexes keyword. If this .conf file wasn’t copied properly from the http version, you are currently in the nano editor, and can make the necessary changes now. Exit out of the editor with ctrl + x and you will return to the ssh terminal.

<IfModule mod\_ssl.c>
<VirtualHost \*:443>
        ServerName sendy.contoso.com
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/contoso.com/public\_html

        <Directory /var/www/contoso.com/public\_html/>
            Options FollowSymLinks
            AllowOverride All
            Require all granted
        </Directory>

        ErrorLog ${APACHE\_LOG\_DIR}/error.log
        CustomLog ${APACHE\_LOG\_DIR}/access.log combined

        <IfModule mod\_dir.c>
            DirectoryIndex index.php index.pl index.cgi index.html index.xhtml index.htm
        </IfModule>

SSLCertificateFile /etc/letsencrypt/live/sendy.contoso.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/sendy.contoso.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

Next is the initialization of the mysql database and then the upload of Sendy to your server.

Installing Your Mysql Database

Surprise…it’s already installed. Welcome again to the joy of the 1-click droplet installation. We just need to create the Sendy database, a database user/password, and grant the new database user privileges to use the Sendy database.

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.26-0ubuntu0.18.04.1 (Ubuntu)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

and create your Sendy database (don’t forget the ; at the end)

mysql> create database sendy;

and you should see:

Query OK, 1 row affected (0.00 sec)

Now we need to create a database user with a password, and grant them permission to use the Sendy database (remember the ;):

Come up with a username and password and record it somewhere secure. You’re going to need it again soon.

mysql> CREATE USER 'sendy\_db\_admin'@'localhost' IDENTIFIED BY '2zrQYe5NC8z2sw87TpC4SWDK';

which should respond with:

Query OK, 0 rows affected (0.00 sec)

Now we grant them privileges (remember the ; at the end):

mysql> GRANT ALL PRIVILEGES ON \*.\* TO 'sendy\_db\_admin'@'localhost';

again which should respond with:

Query OK, 0 rows affected (0.00 sec)

Our work is done here. Exit the mysql command line interface with

mysql> exit;

and the terminal will say Bye. Now you are back at the prompt for the root user inside the droplet.

You now have your database name, authorized user and password, and you can set up your Sendy configuration file.

Setting Up Your Sendy Configuration File

Assuming you have purchased and downloaded Sendy, unzip the file and edit /includes/config.php in your code editor.

This is the part that we’re going to modify:

/\* Set the URL to your Sendy installation (without the trailing slash) \*/

define('APP\_PATH', 'http://your\_sendy\_installation\_url');

 /\* MySQL database connection credentials (please place values between the apostrophes) \*/

$dbHost = ''; //MySQL Hostname
 $dbUser = ''; //MySQL Username
 $dbPass = ''; //MySQL Password
 $dbName = ''; //MySQL Database Name

The relevant values for the install in this tutorial would be:

/\* Set the URL to your Sendy installation (without the trailing slash) \*/

define('APP\_PATH', 'https://sendy.contoso.com');

 /\* MySQL database connection credentials (please place values between the apostrophes) \*/

$dbHost = 'localhost'; //MySQL Hostname
$dbUser = 'sendy\_db\_admin'; //MySQL Username
$dbPass = '2zrQYe5NC8z2sw87TpC4SWDK'; //MySQL Password
$dbName = 'sendy'; //MySQL Database Name

'localhost'will be the Hostname regardless of the other values. Enter the username/password for the database user we just created in the last step. And enter the database name you entered in the previous step, which in the demo was'sendy'.

Notice: since we installed the SSL certificate, the ‘APP_PATH’ value has been set to https, and there is no trailing /after the .com

Did you enter a different domain when you purchased Sendy than what you are going to end up using? No problem, just update your domain at the “ Changing your licensed domain” link sent to you in your purchase confirmation email. I’ve done this a couple times and the update was immediate.

It’s now time to upload Sendy to your server.

Uploading Sendy To Your Server

You can use sftp or rsync. Sftp is probably more intuitive with relatively simple documentation, so I will show the command for rsync. The following command will be issued from a new terminal window, not the ssh shell we’ve been using. The command should be entered as one line. The -ahrvz flags mean “archive/human-readable/recursive/verbose/compressed”. The first folder path is for your local Sendy folder (remember to include the trailing /) and the second location is the DocumentRoot web content folder inside your droplet, which we specified in the virtual host file we edited previously. You will be prompted to supply your droplet root password after issuing this command:

IMPORTANT: the following command must be entered in a new terminal window, not in the SSH shell that we have been issuing commands to the server with, (as specified by the $ at the beginning of the command)

$ rsync -ahrvz -e ssh /path/to/your/sendy/folder/ root@157.230.133.78:/var/www/contoso.com/public\_html/

This will be followed by a confirmation such as:

root@157.230.133.78's password:

After you successfully enter the root password, you will see a massive file list of everything that will be transferred to the server. The listed order of the files may be different, but it should start with something like:

root@157.230.133.78's password:
building file list ... done
./
.htaccess
\_compatibility.php
\_install.php
app.php
autoresponders-create.php
autoresponders-edit.php
autoresponders-emails.php

and end with something like:

locale/en\_US/
locale/en\_US/LC\_MESSAGES/
locale/en\_US/LC\_MESSAGES/default.mo
locale/en\_US/LC\_MESSAGES/default.po
uploads/

sent 5.62M bytes received 18.78K bytes 1.00M bytes/sec
total size is 13.41M speedup is 2.38

$

IMPORTANT: now we’re going to issue commands to the SSH shell to the server again

After uploading, you need to make sure that the permissions of the uploads folder is correctly set to 0777 . This can be achieved with

# chmod 0777 /var/www/contoso.com/public\_html/uploads/

. And you can check if this worked with

# ls -al /var/www/contoso.com/public\_html/uploads/

You should see something like

total 8
drwxrwxrwx 2 501 staff 4096 Jan 12 00:25 .
drwxr-xr-x 10 501 staff 4096 Jan 12 04:31 ..

The important permission bits are the drwxrwxrwx on the same line as the .

The other line relates to the permissions of the parent folder.

If everything has gone according to plan so far, when you reload your browser that is pointing to your new server, you should be redirected to /_install.php.

If you see the following, you’re walking the happy path:

💥

If you don’t, don’t worry. It’s also possible that you might see something like the following:

Not to worry, no matter what is missing, it’s easy to correct. Following the instructions, we navigate to: /_compatibility.php?i=1 and see what needs to be installed. Your results may vary slightly, but likely not too much if you have followed along with this this guide from the beginning.

🤔

Whatever is missing, search the support forum at: sendy.co/forum/ by entering the error into the search box to figure out how to remedy it.

After hunting down any possible remaining packages, your server compatibility checklist should all be green, score 10/10 and navigating to your Sendy url should show you the login page…

…or possibly this:

😶

If this is what you see, you will need to make sure that the domain that you entered as your APP_PATH variable in /includes/config.php matches the domain that you are currently pointed to with your browser, as well as being the domain that you entered when you purchased the product. Domain changes can be accomplished easily. The link for that form is in the email you receive when purchasing the software. In the form, you enter your license key and in theChange domain to field, enter the updated domain. You do not need to preface it with http(s) when you enter the domain (see below). However you do need the http(s):// preface when defining the APP_PATHvariable in the Sendy config file.

What I would need to enter in that form to enable Sendy for the examples here would be:

For the rest of the setup of Sendy, I leave you to the capable hands of the official getting started documentation. You can pick up at step 5. There is very important information regarding getting your Amazon Web Services IAM credentials set up properly, as well as information on increasing your SES Sending Limits. This usually takes at least 2 days for a new account, so if you are browsing through this documentation and planning on installing this software in the near future:

I recommend you actually handle this part now so that when you are ready to install you don’t have to deal with the speed bump of waiting for AWS to approve your increased sending limits.

One related tip that I will share is that the first time you log into your Sendy dashboard, you might see this, even if you already had your SES limit raised :

If you know that you are no longer in “Sandbox mode”, the fix is to go into /settings and make sure that this is set properly:

3 available region choices at the time of this writing

So what’s left? Let’s set up automatic backups of your database, and automatic transfers of the backups to an AWS S3 container!

Things Go Wrong. Protect Your Valuable Mailing List Data By Setting Up Automatic Backups And Automating Cloud Storage Of Those Backups.

Setting Up Automatic Mysql Dumps

Before we set up the scheduler, let’s create a storage directory for the backups and backup script. This command:

# mkdir /srv/db\_bak/

will create the empty folder where we will have our database backups dumped into.

Let’s create our actual backup script with the nano editor again:

# nano /srv/dump\_script.sh

Paste the following into the editor and press Esc and then $ to get the lines to word-wrap so you can see everything on the screen at once. For the first chunk, you will substitute the database username and password you created previously. The second line copies the uploads folder to your backup directory. It’s good to have a backup of that, as any attachments or images that you upload via Sendy when creating campaigns will be stored in this folder, and the e-mails you send out will be served from your droplet. In case you are sending out massive email campaigns, you will probably want to link to images and uploads served via a CDN, or S3 bucket, and not put undue stress on your server.

/usr/bin/mysqldump -u'sendy\_db\_admin' -p'2zrQYe5NC8z2sw87TpC4SWDK' sendy > /srv/db\_bak/$(
date +%F)\_sendy\_backup.sql

cp -r /var/www/contoso.com/public\_html/uploads/ /srv/db\_bak/

now control + x to exit, and confirm you want to save the changes, and confirm the file name. Now to configure the scheduler.

No…Cron

There is a fantastic utility that lives in the Unix-like world, including your droplet, called Cron. We’re going to use it to set up two jobs. One will take periodic snapshots of your database. The other will periodically transfer those snapshots to a secure cloud storage bucket. I’m going to show you how to transfer those backups to an AWS S3 bucket.

Initially there is no user configuration file, known as a crontab, for the cron scheduling utility on your computer. Here’s how to create one:

# crontab -e

which will probably be followed by:

no crontab for root - using an empty one

Select an editor. To change later, run 'select-editor'.
  1. /bin/nano <---- easiest
  2. /usr/bin/vim.basic
  3. /usr/bin/vim.tiny
  4. /bin/ed

Choose 1-4 [1]:

Unless you already know basic vim commands, go ahead and select the default nano editor by pressing 1 and then enter. You will be greeted with this screen which shows the default file which is entirely commented out with “helpful” boilerplate

# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '\*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 \* \* 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command

Let’s erase that boilerplate and replace it with this:

# backing up to /srv/db\_bak/
# every day at 8:12 utc, dump database
12 8 \* \* \* . /srv/dump\_script.sh > /dev/null 2>&1

Crontable scheduling syntax gets no awards for readability. Check out this site if you want to see more details about what those obscure symbols mean. https://crontab.guru/#12_8_*_*_*

click on the underlined “next” to expand the display

This will run our database backup script every day at 08:12 UTC, which is currently just past midnight here on the West Coast of the US. I picked that time because it’s unlikely that there will be many people interacting with my site in a way that will access the database at that time, and I wanted it to not be exactly on a 5-minute interval since that’s when other scripts will run. Perhaps not necessary to schedule like that, but that’s how I was thinking about it.

You can feel free to set that interval to be more frequent than every day, since the way it’s written, it will overwrite the previous file as long as the current day of the system date is the same. It is going to be saving the filenames as…

db\_bak/2019–05–03\_sendy\_backup.sql

…auto-updating the file prefix as the date changes.

I’m guessing you might want to see this actually work and not have to wait a day. If you want to update the cron scheduling interval temporarily to see if it works, change the first part of the third line to */2 * * * * and it will run every two minutes. Also don’t forget the . after the last *. Without that, the dump script won’t run. When you save the file, it will ask you to confirm the save, and then confirm the file name. Confirm the temporary file name that it shows you, and after you accept, it will install a new crontab that you can access again for edits and updates with crontab -e. You can also inspect your crontab at any time with crontab -l. You can also change your preferred crontab editor with select-editor.

If you remove the > /dev/null 2>&1 part from the end of the cron script, the output of the job will generate a system mail message each time it runs. You can check it with cat /var/mail/root and delete the mail messages with rm /var/mail/root.

Since you’ll be back in your Crontab file soon after you set up Sendy, I will save you some time and give you the code for the cron jobs that you are going to be setting up in order to schedule campaigns, activate auto responders, and import subscription oriented csv files. Feel free to enter these now, or wait until you get into configuring your Sendy dashboard.

# Five minute cron job to send scheduled campaigns
\*/5 \* \* \* \* php /var/www/contoso.com/public\_html/scheduled.php > /dev/null 2>&1

# One minute interval job for importing csv
\*/1 \* \* \* \* php /var/www/contoso.com/public\_html/import-csv.php > /dev/null 2>&1

# One minute interval job for activating auto responders
\*/1 \* \* \* \* php /var/www/contoso.com/public\_html/autoresponders.php > /dev/null 2>&1

Part 1 of the automation tasks complete! Next is getting this all transferred automatically to your S3 bucket.

Setting Up Automatic Transfers Of The Mysql Backups To The S3 Bucket

First we need to create the AWS S3 bucket that we’re going to send the transfers into. From the S3 console, create a new bucket. The Region is arbitrary, unless you have a specific need to host your data in a particular region.

Name And Region Settings

Then click next

I didn’t need to change anything in the “configure options” panel.

Just click Next again

But you need to “Set permissions” appropriately so that you will be able to upload to the bucket

Uncheck all these options

Review your options and press:

Then select your bucket, select Permissions , and set the following Access Control List:

“write objects” enabled for everyone

Then select the “Overview” tab and create a folder

Keep the bucket and folder name handy, you will need them soon.

And that’s it for setting up the bucket. Now to create a user and give them appropriate permissions to store things in your bucket.

Setting Up An AWS User In IAM And Granting Them Permission To Access Your Buckets

Head over to the AWS IAM https://console.aws.amazon.com/iam/ console and click on the Users tab over on the left. Click on Add user and give them a sensible name and Programmatic access. And then click Next : Permissions.

Select Attached existing policies directly

and in the search box enter: s3ReadOnly and give the user AmazonS3ReadOnlyAccess

And then click on Next: Tags. I did not enter anything here. So next click on Next: Review and if everything looks appropriate click on Create user

Make note of the Access Key Id and Secret access key values. You will need those soon, and this is the only time that you can access the Secret access key so if you need that value later, you will have to create a new key for this user.

Heading into the home stretch!

Installing AWS S3 Utilities On Your Droplet To Handle The Transfer Of Your Database Backups To The S3 Bucket

Back in your root SSH terminal connection, install the S3 utilities with:

# apt-get install s3cmd

Accept the installation confirmation, and then you will have these tools installed. Now you need to enter your credentials.

# s3cmd --configure

will bring up a dialogue that looks something like:

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key:

You will enter your Access Key, and Secret Key and for the rest of the values, just click enter to accept the default values unless there is something you want to explicitly set. At the end of the process, make sure that when it tests your credentials that it passes, and then agree to Save settings.

Things will probably have looked something like this:

Default Region [US]:

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]:

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

New settings:
  Access Key: AKIAJ3O4AL4VMLGBEPAQ
  Secret Key: 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
  Default Region: US
  S3 Endpoint: s3.amazonaws.com
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com
  Encryption password:
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n]
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N]

Save the settings and you are finished with this.

You can test it out now with the following command, assuming you have a db backup in that folder. If you don’t already have a backup file, do this first:

# echo 'just a test file' > /srv/db_bak/testFile.txt and then:

s3cmd sync /srv/db\_bak/ s3://contoso-sendy-backups/sql-dumps/

which should result in some output like this:

root@sendy-droplet:/srv# s3cmd sync /srv/db\_bak/ s3://contoso-sendy-backups/sql-dumps/
WARNING: Empty object name on S3 found, ignoring.
upload: '/srv/db\_bak/2019-01-12\_sendy\_backup.sql' -> 's3://contoso-sendy-backups/sql-dumps/2019-01-12\_sendy\_backup.sql' [1 of 1]
 7945 of 7945 100% in 0s 85.26 kB/s done
Done. Uploaded 7945 bytes in 1.0 seconds, 7.76 kB/s.
root@sendy-droplet:/srv#

Now checking my S3 bucket, I see that the contents of that folder transferred successfully

Okay…let’s make this a job for Cron!

Add this to your crontab with crontab -e

#every day at 9:12 utc, synchronize .sql dumps with s3 bucket
12 9 \* \* \* s3cmd sync /srv/db\_bak/ s3://contoso-sendy-backups/sql-dumps/ > /dev/null 2>&1

This will send the db dump to S3 daily at 9:12 UTC. Again you might want to test this by setting the interval to something much lower (detailed in the first Cron section, above), don’t forget to set it back. I think a daily external cloud backup is a sane default. Your needs may differ.

Restoring Database From a Saved Backup

So what do you do if a database disaster happens? If you set up the S3 automated backups as described above, you have a backup!

Restoring your database can be achieved as follows:

sign into your S3 console
select the backup that you want to restore

When you download the file, it will likely be downloaded as a .txt file. Just rename the file extension to .sql
Send the backup to your droplet from the command line with (this is all one line):

$ rsync -ahrvz -e ssh /path/to/your/sql/backup/2019-01-14\_full\_sendy.sql root@157.230.133.78:/srv/db\_bak/backup\_to\_restore.sql

Now login to your droplet and login to mysql to drop the existing corrupt database. **THIS WILL IRREVERSIBLY DESTROY THE CURRENT DATABASE** So make sure you are only doing this if the db is already corrupt and needs to be restored from one of your backups.

# mysql

then ( this is the destructive part ):

mysql> drop database sendy;

then re-create the sendy database

mysql> create database sendy;

then exit out of my sql with:

mysql> exit;

and perform the database restoration from the remote shell with:

# mysql sendy < /srv/db\_bak/backup\_to\_restore.sql

Then refresh Sendy in your browser and you should be back where you started prior to the database corruption.

Let’s Create A Snapshot Of The Droplet

It’s nice to have an “undo” point. DigitalOcean gives you the ability to create snapshots of your virtual servers at any point in time. This would be a good time to create a snapshot, in case you ever need to restore the virtual server. If you did, you would only need to reimport your database to the latest saved version in your S3 bucket and you’d be back in business.

To do this, first we need to shut down the droplet from the shell with:

# shutdown

this should give you some output similar to:

Shutdown scheduled for Tue 2019-01-15 04:49:54 UTC, use 'shutdown -c' to cancel.

Your server is now going through the shutdown scripts, which is better than a hard power off. Now navigate to https://cloud.digitalocean.com/droplets and select your Sendy droplet. You should see that it is currently off.

Click on the Snapshots tab and give your snapshot a sane name. Then click on Take snapshot and it will take a couple minutes.

For the costs quoted at the time of this article, you’re looking at about $0.10 per month to store a snapshot.

After the snapshot is complete, head back to the Power menu and turn your droplet back on. Give it a couple minutes to fully reboot before you try to reload, or you may see errors related to connecting to the server or database.

You can take snapshots at any time like this, but of course be mindful of whether or not there are running jobs when you shut it down.

Congratulations, That’s it!

Please comment on how it went! I would love to hear how your process went, and if this guide helped you.

I hope this guide saves you a lot of the headaches that I went through while setting this up. I’m happy knowing that it may have helped you, and if you want to say thank you, please leave a comment, and / or feel free to buy me a beer! 🍻 Cheers!

Kyle Simpson describes a paradigm shift in programming

Alex Jacobs — Wed, 10 Apr 2019 06:17:18 +0000

this free video lecture is one of the best descriptions of coding I’ve ever seen. It also describes a fundamental shift in style we can make to improve software development as a whole.

// Detect dark theme var iframe = document.getElementById('tweet-1115858817688965122-948'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1115858817688965122&theme=dark" }

Calling out "Stunt JavaScript™️"

Alex Jacobs — Fri, 22 Mar 2019 03:11:26 +0000

Not only will we dissect the following function that is laden with es6+ syntax, but show how a humble es5 implementation is not only easier to read, but also more efficient.

Today, a student came to a mentor session wanting to dissect the following function:

function omit(obj, keyToOmit) {
  return Object.entries(obj).reduce(
    (newObj, [ key, value ]) => (key === keyToOmit ? newObj : { ...newObj, [key]: value }),
    {}
  );
}

You could say that it functions like filter for objects. The intended usage is to pass in an object as obj, and a string as keyToOmit, and to then return a new object with all but the omitted key/value pair.

Like this:

const dog = { plays: true, guards: true, barksAllNight: true };
let dogUpgrade = omit(dog, 'barksAllNight');
// ahh, now we can get some sleep.
// dogUpgrade => { plays: true, guards: true }

So to understand how it works, we went through the exercise of rewriting it into a less showy, but so much more clear, implementation of filtering out a key/value pair from the object.

You may be wondering why we didn't just do something like
delete dog[keyToOmit];
Good question! It's because we don't want to 'mutate' the object that was originally passed into the function. Similar to the original omit function, we wanted to return a fresh object that had all of the original key/value pairs, but not the key/value pair we want to filter out.

Simple Version

Here is the rewrite we came up with. If you were having any trouble following the original function, see how this one compares when it comes to readability:

function omit(obj, keyToOmit) {
  var result = {}; // start with an empty object (don't mutate the original obj)
  // iterate and only add key/val pairs that are not the keyToOmit
  for (let key in obj) {
    if (key !== keyToOmit) {
      result[key] = obj[key];
    }
  }
  return result;
}

Using a for in loop to iterate through the keys of the object, we simply compare the key to the keyToOmit, and if it doesn't match, extend our result object before returning it at the end.

So now that we reverse engineered the function and made a basic implementation, we set out to decode the syntactic sugar sprinkled inside the original omit function.

Stunt Version

Object.entries() is a fantastic method that returns nested arrays consisting of the key/value pairs of the object that is passed as an argument to the method.

Invoking Object.entries on our dog would evaluate to

var entries = Object.entries(dog);
// entries => [["plays", true], ["guards", true], ["barksAllNight", true]]

Now that the key / value pairs are serialized within an array, we can use the HOF (higher order function) reduce to boil the values down to a single final result.

The function signature of the callback in reduce is (accumulator, currentValue, currentIndex, array). The last 2 are optional, and not used in this function, so we'll ignore them for now, but it is good to know that they exist. You can read more about reduce on mdn here.

So, since we are passing the first two arguments to reduce, what are they in the context of the omit function?

[["plays", true], ["guards", true], ["barksAllNight", true]].reduce(
    (newObj, [ key, value ]) => (key === keyToOmit ? newObj : { ...newObj, [key]: value }),
    {}
  );

The first argument newObj corresponds to the accumulator. The second is written as [ key, value ]. This is a nice use of array destructuring assignment which allows for a concise assignment of 2 variables at once, corresponding to the array indices of the current value being passed to the reduce callback.

Here's an example:

var [a, b] = ['first', 'second'];
// a => 'first'
// b => 'second'

So, since the reduce callback function is receiving another array with 2 values (key / value from the object) about the dog with each iteration, we can use destructuring to assign the variables key and value at once. Nice.

Now, inside the => expression, we can refer to newObj key and value. The ternary operator is set up so that if key is exactly equal to keyToOmit, we just return the accumulator unchanged. Otherwise, we use the object spread operator to succinctly copy the key/value pairs from the accumulator, and amend it with a new key/value pair. The [key]: value syntax is yet another es6+ trick. It allows using a Computed Property Name for the key. It evaluates the expression between the brackets, which is the key variable. The string that evaluates to becomes the computed key that is added to the accumulator. The value of this key/value pair is the value of the value variable.

After the reduce function completes, the entire expression evaluates to the new object that is now missing the keyToOmit and it is returned from the function.

It's super clever coding, but if used at scale, is quite inefficient.

Let's take a look!

Our lowly es5 implementation is less likely to get us labelled as 1337 but let's consider its runtime complexity.

enter the function
set up the results object
iterate through the enumerable properties of the object once
- make a comparison of each value with the keyToOmit
return the results object

Now let's break down the steps required for the 'stunt JavaScript™️' version:

enter the function
derive Object.entries by:
- iterating through the enumerable properties and assigning the key/value pairs to nested arrays
set up the reduce function
iterate again through the derived nested array and
- make the comparison of each value with the keyToOmit
  - if it matches, return the accumulator unchanged
  - otherwise iterate again due to the object spread operator! (I think this pushes the time complexity closer to O(n^2)
return the evaluated result of the reduce function.

I think that both are technically O(n) run time complexity, but if you were to show a graph of both, they would both be linear, but the second version would be steeper in terms of operations per input size. And it is definitely less efficient in terms of space complexity (auxiliary space). As a careful reader pointed out, if the implementation of the object spread operator truly iterates through each key value pair of the accumulator again, then this is now a nested for-loop. This would be an efficiency downgrade, closer to O(n^2)

Takeaway

For a simple helper function, this time difference is immaterial, but I still think it's good to get in the habit of thinking about algorithmic complexity in order to build up intuition about runtimes in preparation for writing efficient code in the situations when it does matter.

Final words

As a teacher / mentor of newer engineers, I get a lot of input about what kinds of information is confusing to people who are earlier on in their programming journeys. I like to write about about these topics, to share these learnings with a wider community.

I hope you've enjoyed coming along on this code exploration 🔎. More to come, so + FOLLOW and I'll see you further on down the code road!

Both functions side-by-side:

Thanks to Web Donut for the royalty free cover image

Conway's Game Of Life with different rules

Alex Jacobs — Tue, 19 Mar 2019 20:11:58 +0000

Conway's Game Of Life

The Game of Life, also known simply as Life, is a cellular automaton devised by the British mathematician John Horton Conway in 1970

If you've spent any time exploring or playing with Conway's Game Of Life, you're probably very familiar with resulting pattern such as:

But what if you could play with the settings that determine the rules for each turn, and could create patterns such as this?

Now you can!

Here is a deployed version of the project that is up and running that you can play with

I've created a game of life explorer using JavaScript that lets you change the rules.

The default rules are:

Death by overpopulation if neighbors are greater than 3
Death by under population if neighbors are less than 2
Rebirth by reproduction if neighbors exactly equal 3

I've also created simple editing tools that let you add or delete individual cells, play and pause during the evolution of the game, move your cells around on the game board, and also select random starting patterns. You can also change your initial board size, and of the millisecond delay between turns.

This allows you to make interesting starting patterns such as the following:
(scroll to 30 seconds to see the pattern get much more complex and interesting)

You can explore the code at my repository at:

lexjacobs / conways-game-of-life-explorer

Exploring Conway's Game of Life with variable board sizes, and alternate rules.

Learn more about Conway's Game of Life at the wikipedia article here

Explore the Game at the website deployed here

View on GitHub

I hope you enjoy exploring this project and enjoy learning more about The Game Of Life.

Please let me know what you think!

I asked 12 software engineers how to get hired at their companies, and here’s what they said…

Alex Jacobs — Wed, 12 Dec 2018 20:33:20 +0000

Lessons from a software engineering back-end recruitment “speed dating” style event in San Francisco

Summary

I feel that each company had their own take on the ideal path that could lead to getting hired at their company. If there was one major take away from the evening, I would say that the Silicon Valley/ Bay Area startup space seems to emphasize prior experience for their back end positions. If you are at a loss for how to get that initial experience, the advice that I heard more than once last night was to perhaps approach a larger tech company for an internship or apprenticeship.

Don’t be discouraged though! Speaking from my own experience, and that of many of my colleagues, it is possible to get hired with a startup immediately after graduating from a boot camp, as I did in 2014.

I’m not certain what keyword searches my various online engineering activities overlapped with, but this was the second time that I had been offered a free ticket to the HackerX recruiting event.

HackerX logo

I didn’t attend the first time because I was not seeking additional employment, and also nearly declined on the event again until I read that they welcomed passive job seekers. Although I’m currently very happy with my engineering balance of teaching, mentoring, personal projects and study, I felt it would be valuable to learn more about the types of roles the industry is currently hiring for.

After some brief presentations by the sponsoring companies, the “speed dating” portion of the evening began. As I was next in line to begin my first 5- minute interview, I realized the absurdity of the situation and quickly scrambled to figure out what would make the evening both entertaining and informative. I thought about my students and wondered what I could bring back to my students and mentees that could be useful. Out of my brain came the following question which I proceeded to ask for the remainder of the evening:

“[…quick personal introduction]. If you could reverse engineer the ideal employee for the positions you are hiring for, what skills and experience would you guide that candidate to gather right from the start of their learning path? Both soft and hard skills.”

Here’s a summary of what I learned, organized by the companies that I spoke to. Many of the companies also shared their opinions about boot camp graduates, as I had introduced myself as a technical mentor/teaching assistant for current boot camp students.

PlayStation logo

PlayStation

Most useful skill prior to applying: learn how to architect for distributed computing. The main question somebody should keep in mind while learning that skill is: “Can it scale? To millions?” Learn how to make applications that are simple and fast.
The “unicorn” that this person was looking for would be an expert in databases who can also solve problems on a software and hardware level. Someone who can handle when stuff goes wrong. Also looking to hire someone who can remove busywork. Someone who is able to automate repetitive tasks.
Looking for people with experience in building resilient applications. Can the systems handle failure?
Most important soft skills: collaboration. This is useful in the industry in general. Can you comment effectively on others’ work? Conversely can you receive feedback gracefully?
Most useful soft skill: the ability to defend your design. If you can’t clearly and logically describe your design from start to end, it probably won’t work.

eatsa logo

Eatsa

Values candidates having diverse skills backgrounds, which is often the case of boot camp graduates since they come from a variety of not-necessarily-technical backgrounds. They find that fresh perspective is even more useful than someone coming in with a CS college degree.
Encourages generalists to apply, yet also wants to see evidence of depth. this could be as simple as being clear about what part of the stack you enjoy working on.
In order to practice the skills that could be useful for a backend position with this company, building APIs would be good preparation as it provides a good balance of difficulty, yet allows for quick iterative development. Wants to see engineers that can produce results relatively quickly.
Learn about professional engineering teams and how scrum works. They felt that many junior candidates without experience in engineering teams didn’t know how to leverage stand up as a chance to expose blockers and work as a team.
Ideal prior experience: working with startups.

boxbot logo

Boxbot

Useful hard skills: learn the popular frameworks/structures that are actively being developed in your major tech market, if any. For example: here in the Bay Area, they lean heavily on the Facebook React ecosystem.
A startup often doesn’t have time to reinvent the wheel, so they seek to find engineers that are skilled in the locally developed popular frameworks and libraries.
They mentioned another example that if you were in Seattle, learn to lean heavily on AWS infrastructure.
Framework/library recommendations: React/Rails.
Feelings about boot camp students: they have breath but no depth, yet.
Recommended intermediary step prior to applying to this company: learn hard skills including deploying database frameworks such as GraphQL

thredUp logo

ThredUp

Useful hard skills: mobile and full stack. Using Objective-C/swift for iOS. Using all Java for android, moving toward Kotlin next.
Seeking applicants with full stack experience including high-level skills in Ruby on Rails and React.
Soft skills: strong and cross collaborative. Ability to move fast. Ability to ship, even if fixes are required later. Not looking for super meticulous coders who unnecessarily delay pushing to production.
Recommended intermediary step prior to applying to this company: they have had excellent applicants who had spent some time working or interning at Netflix.
According to their Engineering101 document that they handed out, they value: People looking to make a quick impact / Elegant solutions that allow us to learn as a company / Paying off technical debt / People who can thrive in ambiguity / A drive to design and build solutions to meet customer needs / Cleaner implementations focusing on team efficiency, future speed, future capabilities, or scalability

epam logo

epam

Recommendations to all applicants: understand a company and its culture first.
How they vet their engineers: initial skills match → technical interview → company culture fit interview

hcl logo

HCL

Soft skills: ability to learn new skills and be a good team player.
Hard skills: considered very unimportant due to the number of different projects that they provide engineering services for.

eatsa logo

Eatsa (second conversation)

Hard skills: full stack. Knowing React is good, but they also want backend applicants that are skilled with Python/Ruby.
Thoughts about boot camp grads: 60% of boot camp grads aren’t strong beyond the latest front end libraries such as React.
Recommended intermediary step prior to replying to this company: internship experience. They mentioned some companies that offer internships or apprenticeship such as dropbox, lyft, or google. Go to a big company and spend some time developing your skills because training is not available at a small startup such as this one.
Recommended amount of industry experience prior to applying here: 1 year
Recommended portfolio items: evidence of shipping code. Boot camp projects might be impressive, but don’t show evidence of shipping production code.
Interview skills: data structures/algorithms/big O notation.

volta logo

Volta

Thoughts about boot camp graduates: they may be up-to-date with the latest tech, but this may not be so important. They want applicants who have experience with architecture and business logic.
Learn the tools that can allow a startup to remain small and lean as long as possible, such as learning faas (function as a service) such as aws lambda.
Learn how to avoid putting bottlenecks into servers, such as offloading static assets.
What are some other factors that make a great candidate for them: applicants that write technical blog posts on medium and actively contribute to open source.