Email automation with Node and Dropbox

David James — Sun, 11 Feb 2018 17:51:09 +0000

I choose a lazy person to do a hard job. Because a lazy person will find an easy way to do it. - Bill Gates

This quote, I feel, describes why I love being a programmer. We are generally lazy yet quite good at solving problems, including our own! Sometimes, this may not involve even writing code, but most of the time you can assume we did 🤓

So, my most recent problem: My Mum is required to email an invoice to her client fortnightly, and is fairly computer illiterate. This is handwritten and needs to be scanned as a PDF, and because I’m not around the house much anymore, my sister scans it for her, but she is too lazy to organise the email, so she adds it to my Dropbox so that, finally, I can email it to the client.

I hate this entire process... and receiving "Have you sent that invoice?" text messages.

The steps involved for me are:

Downloading the file from Dropbox
Logging into Mum's email account
Typing a very generic email to the client
Attaching downloaded file
Send the email

Solution: AUTOMATE ALL THE THINGS!

Javacript/Node to the rescue!

Javascript and Node seemed the most appropriate for my solution as I knew I would need to run a server-side application to check my Dropbox regularly to find the file. I am also in the process of trying to become more of a fullstack developer, so I knew this would be a great learning exercise.

Recently I had completed Wes Bos' Learn Node course which greatly assisted in the design choices for my final solution. This included but was not limited to: Node, ES6, Promises, Nodemailer, Node Cron, and shell scripting for continuous deployment (but I will go into this further in my next post - follow me on Twitter!).

I won't go into too much more detail about specifics of the application as you can just take a look at it here on GitHub. However, I would like to go on further to explain the issues I faced, how I could improve the application, and what utilities made writing this application a joy to create!

Promises and the Dropbox API

Previously, I had worked with the Dropbox API using PHP to create an application that would randomly pick a set of photos and display them onto a webpage. This was fairly basic and just didn't feel right because we were just calling the API with a curl function and I am trying to use less PHP where I can these days.

When it came round to building the invoice application, I found out that Dropbox had created a Javscript SDK to interact with the API. This was exciting, and even more exciting when I read the documentation to find out that it was promise based! Promises mean that you can easily chain a few API calls to get the data that you require or perform the actions you need to with little to no effort.

Here is an example of a promise chain to download a file. It assumes you are passing the path of the file, which you can get easily using another API call/promise.

const Dropbox = require('dropbox');
const dbx = new Dropbox({ accessToken: process.env.ACCESS_TOKEN });

exports.getFile = function (path) {

  const file = dbx.filesDownload({ path: path })
    .then(function (response) {
      return response;
    })
    .catch(function (error) {
      console.log('Error downloading the file ❎');
      return Promise.reject(error);
    });

  return file;
};

I can't believe it's not butter! So simple, much file. 🐕

Just to show you I'm not bluffing, I created another function that I called once the email was sent. This moves the file in Dropbox to another folder to indicate that this invoice has been sent.

exports.archiveFile = function (path, subFolderName) {
  const archivedFile = dbx.filesMove({
    from_path: path,
    to_path: '/sent/' + subFolderName + path,
    allow_shared_folder: true,
    autorename: true,
    allow_ownership_transfer: true
  })
    .then(function (fileMove) {
      console.log('File ' + fileMove.name + ' archived successfully! 🗳️');
      return fileMove;
    })
    .catch(function (error) {
      console.log('Error archiving the file 💥');
      return Promise.reject(error);
    });

  return archivedFile;
};

Here I pass the client name as the subFolderName which means that you get a well organised file path like /sent/client-name/INV0001.PDF

But what about the Email?

Oh right, so before we go archiving the file, we obviously send the email. The creation of this email involves a few small parts but the sending of it is very straightforward.

As my Mum has multiple clients, the solution needed to incorporate some form of reusability and scalability. I managed this by creating each client as a JSON file that would look something like this:

{
  "name": "Recipient",
  "email": "test@email.com",
  "subject": "An interesting Email Subject",
  "text": "Hi John Doe,\n\nInvoice attached.\n\nKind Regards,\nJane Doe",
  "file-prefix": "INV"
}

This ensured each file to be sent from Dropbox would be mailed out based on its filename prefix, allowing each client to have a different name, email, subject, or text within the email. This also means that if she ever get more clients, it is just a matter of creating new JSON files to also be a part of the automation train. 🚂

Using the data above and the calls to the Dropbox API we are able to build our email and send it using Nodemailer.

The code for sending an email through Nodemailer is a function call with a few option parameters (as seen below). These are passed then used in conjunction with a transport function, with most of its config set using environment variables (because you don't want people spamming you or knowing your SMTP credentials).

In this application, I added the file using a binary file stream/buffer which sounds far more complicated than it is or needs to be. In reality, it just means we get the binary version of the file from Dropbox, save it as a variable, pass it to the buffer, and then it becomes a file attachment.

 ...

 const sendInvoice = attachedFile.searchFilePath(filePrefix)
      .then(function (filePath){
        foundFilePath = filePath;
        const file = attachedFile.getFile(filePath);
        return file;
      })
      .then(function (file) {
        const mailPromise = mail.send({
          email: recipient.email,
          subject: recipient.subject,
          text: recipient.text,
          attachments: {   // binary buffer as an attachment
            filename: file.name,
            content: new Buffer(file.fileBinary, 'binary'),
            encoding: 'binary'
          }
        });
        return mailPromise;
      })

  ...

Voila! There is the majority of the application in just a few function calls. If mailPromise resolves, then our email will send.

To test email sending while in development, using a service such as Mailtrap is a lifesaver as it is free and doesn't fill up anyone’s inboxes 😆

Once I got to production, I changed it over to Mailgun as you can send up to 10,000 emails every month for free!

Automation 🤖

So it seems the application covers all your previously mentioned steps... but what makes it automatic?

Not much really, just run the function once every hour (or as much as you'd like) using a cron. A cron is "a command to an operating system or server for a job that is to be executed at a specified time". In this case, the application checks if there are any files to be sent. If there are, execute the rest of the application; if not, don't do anything. As previously mentioned, promise chains make this process a breeze.

Like everything, there is always room to improve. The cron could be removed by only running the function when a file has been uploaded, and obviously you can't just do this with the API but you smart cookies out there would've realised you can do this with the use of webhooks (but that’s for another time).

Conclusion

This project was super enjoyable! I learnt a multitude of things from Node to Shell scripting, from Cron jobs to Promises. Little side projects like these really push you forward as a developer. They allow for you to be the perfectionist you want to be and create something to improve your life (and sometimes others’ lives too) in more ways than one.

Site generating with the great GatsbyJS

David James — Sun, 01 Oct 2017 14:37:00 +0000

When someone mentions static site generation a lot of people think ofÂ Jekyll. I had heard some good things about it and noticed it would allow me to build a blog quite easily. I was eager to learn something new and get something up and running that was more than just my usual WordPress theme. I was on the verge of rebuilding my portfolio site until a certain Scott Tolinski released a video on GatsbyJS.

GatsbyJS is a static site generator, similar to Jekyll, however it is written using React and allows you to write your pages as React components! It is somewhat similar to create-react-app where almost all the scary Webpack config has been abstracted away from you and is setup ready to go, so you can get to the important stuff like...building the site! (If you are a fan of React and not convinced, the React website/docsÂ were just released using Gatsby!)

This caught my attention for multiple reasons. I wanted to:

Be able to build a new site with a blog relatively quickly
Have a site that loads quickly/is performant
Learn some more React and JavaScript

Another reason I ended up going with Gatsby was the promise that your project could be connected to various APIs or even a CMS of your choice.

The magic of GraphQL and Gatsby Plugins

Originally, following Scott's and the official Gatsby tutorial, I had it pulling my content from good ol' Markdown files. Then I saw it... too good to be true, the mention of connecting it to WordPress... I have been writing WordPress themes for almost 2 years now, so this got me super excited. To have my back-end as WordPress (including ACF) and the front-end in React, I had found the perfect place to test my front-end abilities.

I was skeptical at first: would I have to parse large amounts of JSON to get the data I needed? I have never even interacted with the WordPress REST API, how will I query it?

The answer... GraphQL. Gatsby ships with it and through an npm install of a gatsby-source plugin of your choice and a tiny bit of a config, you can start querying in no time. I was amazed with how simple queries using GraphQL. You look at them and you go "Huh, that's it? Really?". Gatsby even ships with a in-browser query tester so you can see exactly what data you are getting from your GraphQL queries. Wanna sort those blog posts by date? No problem, just add a flag.

Wrap up and future

In just a few weekends I managed to rebuild my portfolio site with the blog I wanted. I'd highly recommend GatsbyÂ for anyone who has started getting acquainted with React. Before I started this project I didn't know a lot about:

Static site generation
Creating a Progressive Web App (PWA) and what qualifies as one
React Router
GraphQL
Wordpress REST API

Moving forward I'd like to extend the project to include pagination within the blog, use Styled Components and ensure the site scores a 90+ overall on Lighthouse (Google performance auditing tool)

tl;dr - Side projects are great and you can learn a lot from them. Stay in the loop and listen to others whether that be watching tutorials, listening to podcasts, following devs on twitter, reading articles on dev.to, blogs etc. #neverstoplearning

If you are like me, learn something, build something, even if it's small, then write about it, then you'll have a first blog post, just like this one :D

Forem: David James