Forem: Matt Mascioni

Switching from Customer Success to Engineering: a year later

Matt Mascioni — Tue, 28 Feb 2023 21:22:37 +0000

A couple years ago, I joined PartnerStack in their Customer Success department, as an integration specialist. A lot has changed since then — I’ve since transitioned into our engineering department, and it’s been an incredible ride along the way. After over a year in engineering, I wanted to share a little bit about how I got there, and offer some advice to anyone looking to make a similar move at their organization.

When I joined PartnerStack on the integrations team back in 2020, I did with the intent of eventually becoming a software engineer, as the role felt like a good medium between being product facing and being customer facing. I've loved writing software for a long time, but at the time I didn't have any traditional software engineering education. My education was in chemical engineering, and a lot of what I learned about web development had been on my own, through side projects, online courses, or freelancing in the past. Not having traditional or bootcamp education made it trickier to break into a technical role in tech, even more in 2020 as the COVID-19 pandemic had just begun. After going through hundreds of rejected job applications and interviews, the PartnerStack interview process felt like a breath of fresh air and I was delighted to join the team.

The integrations team is an interesting one: we primarily helped during the onboarding phase of a customer’s journey, which included consulting on how best to use our APIs and SDKs with other engineering teams, using our integrations platform to build integrations with a customer’s systems, and more. You were really the main technical point of contact throughout the journey. We also interfaced with and supported other teams, both within CS and outside of it. You had exposure to product and engineering teams as well to relay client feedback and address bugs. Being on the team was an incredible way to learn both the product and the people who used it deeply, and some of the fondest memories I have at the company were while I was there. The team was brand new back in 2020, I onboarded with the only other team member as the company was beginning to build the role out.

A while after joining, I started working with my manager to flesh out my goals of transitioning and how I could go about achieving them eventually. Even though I had support to carve out this career path, execution was difficult. Often times the process felt paralyzing, because I wasn’t sure where to start. I was already confident in the primary language we used on our back-end (Python), but wanted to begin gaining exposure to the codebase and working on issues. At the same time I didn’t want to step on any toes and take up other team’s time / de-rail any processes I didn’t know about.

Where I started was following a bit of a “one step up” model — go “one step up” from what you’re currently doing to dip in to what another team is working on. For me, the target was our integrations platform — I already worked extensively with it (and the APIs backing it) as someone customer-facing. One step up would be working with the team who managed that platform from an engineering perspective, so I got in touch with the manager of that team and some of the engineers who worked there. We settled on a model where I could take on backlog tickets or bugs (i.e. nothing in a planned or active sprint, but where a user need was validated) from their team, pairing with their engineers where needed.

A lot of these tickets or bugs were “first time” type issues that would normally be handed off to new developers onboarding. The bugs forced me to understand the path of execution from a code perspective for features I was used to seeing from a user’s perspective, and even though the fixes were a couple lines long, they were monumental in helping me understand our back-end (largely a monolith). Over the coming months, I took on bigger tickets, eventually helping close out a whole feature. It felt amazing being able to share what I had built with my team but also the customers I was on call with — it became a really nice feedback loop.

The challenge became balancing these three sets of responsibilities:

Building out team processes and preparing the team to grow
The day-to-day responsibilities of being on the integrations team — being customer-facing can be demanding!
The responsibilities I was taking on as part of picking up and completing engineering tickets

The balancing act was a tough one, but a few things really helped out. Firstly, we began hiring on the team. This made building out processes much more important, but it really helped in the long run as team members onboarded faster. A nice side effect of this was that other teammates could now help take on some of the things we were working on if they were interested, since they were now public and not living in people’s brains. The other thing that helped was strictly not working on tickets within an engineering team’s sprint — this kept expectations low and didn’t affect their delivery as much while giving me a ton of space to learn. If I needed to de-prioritize a ticket, I totally could.

Eventually, after numerous conversations and plans being drawn up I was offered the position and started in late 2021. Along the way, I learned a few things that I thought might be useful to someone else wanting to make the switch down the line. None of these are meant to be prescriptive, every workplace and situation is unique and mine pertains more to a SaaS startup.

Keep notes of the journey: Keep track of the details as you go through all of this — what did you work on? What sucked? What conversations with which people did you have? These notes, a lot like a brag document, help you advocate for yourself as you discuss your goals with your supporter, and supporters within the engineering org. Adding some dates in will help you look back and feel like you made progress on days where you feel you didn’t. I presented my story linearly, but this process is anything but linear, making these useful.
Communicate often and advocate for yourself: Your manager/supporter is there to advocate for you, but they can’t do so until you make your goals known! Communicate with them from the beginning to build a plan, and communicate with managers/team leads on teams you’re interested in learning more about-- this isn’t something you do on your own. Also keep them up to date on your progress (this is where keeping notes comes in handy!) One misconception I had when I started was that wins would speak for themself, but they commonly don’t. By sharing your wins/losses, and speaking openly about the work you’d like to take on and where you want to orient your career, you’re advocating for yourself and making every conversation a little easier.
Immerse yourself a bit: One thing that helped me a ton was joining the public Slack channels of engineering teams I was interested in working with (if they were cool with it). This gave me insight into the “stream” of work that team was producing; if there were requests for PR reviews, I got to check out the PR and learn from how existing teammates organized their code. If there are regular, open-invite, cross-team meetings that take place for engineering teams at your company, look into attending them as a fly on the wall. These meetings might contain high level feature proposals, which may be easier to digest since they need to be created for a broader audience. The discussions are arguably the best part, and the kinds of questions raised can help prep you for code review opportunities you’ll have down the line.
Try to onboard on your own: Ask around and find whatever onboarding documentation you can on how an engineer would onboard onto the team you were interested in working with. Request access to and set up the repos locally yourself, keeping a list of questions for when you get stuck. The benefits of this are multi-fold:
1. You uncover holes in the onboarding documentation, which engineering leaders will appreciate (a fresh pair of eyes is underrated)
2. You’ll uncover frameworks you don’t know yet, adding to your list of things to start learning to prepare you to take on tickets
3. You’ll have your own copy of (at least a part of) your product running locally, which will benefit your current team immensely as eventually you’ll be able to replicate customer scenarios locally and answer customer/team questions even faster. This is extremely beneficial for “legacy” features few people have context on.
One foot in your comfort zone, one foot out: I don’t have a better name for this, but when looking at work to take on, first try engineering work thats as closely as possible related to what you work on in your role. This lets you use the context you have from your role to give you a leg to stand on. Some examples:
1. If you’re an integration/implementation specialist who regularly talks about specific features or integrations look to see if there are any reported bugs on those features, or “quick wins” improvements one of the engineering teams has planned.
2. If you’re on a support team, talk to an engineering team member looking into a bugfix for a bug you reported, as it might be a good opportunity for a pairing or shadowing session.
3. If you’re a customer success manager, try shadowing with a support / integrations team member as they solve an issue or solution for one of your customers.
Comb through engineering team backlogs: Every engineering team has a backlog of things that they haven’t been able to prioritize yet. If these backlogs are public to the company, or you talk to the team lead there to get access to see, you have an advantage. Ask questions on a ticket you’re interested in to see if it might be a good fit for your comfort level. If you’re in CS, bugs can be a great first class of issue to work on because there’s a chance you’ve experienced the bug yourself or one of your customers has, which will make it even easier to reproduce. Take on one thing at a time, and ask before taking anything on. If you put up pull/merge requests, ensure the summaries can speak for themselves as you won’t be as connected to the rest of the engineers on the team for a bit. As a general rule, especially in the beginning I considered anything within 2 sprints of work (assuming your workplace’s engineering team follows a sprint model) to be a no-fly zone.
Get information out of your head: If there are:
1. Processes only you know, that people come to you frequently for (especially in DMs)
2. Knowledge that you have on the product or platform that allow you to answer questions quickly
3. Context on specific accounts that lives exclusively in your head
Externalize these! Make use of whatever documenting tools your company has to offer and level up your teammates. This helps reduce the “risk” involved with team switches and can help ease your transition, and help take work off your plate in the meantime.
Don’t lose customer context once you go: Once you transition over, don’t be quick to lose your customer context — keep in touch with your CS pals, frequent the channels you were a part of before. On an engineering team, product managers are usually the people who have access to customers, and try to gauge priority based on their needs. This is something you’ll have baked in, and will be one of your greatest strengths in the beginning. With your customer context, don’t be shy about helping product managers with understanding needs and priorities of the customers you used to work with. It’ll help your entire team to have another person with customer context to bounce ideas off of.
The imposter syndrome: The imposter syndrome may feel strong while you’re trying to get your first pieces of work out there. You might not feel you deserved what you got, or that it could be taken away at any moment. Even after transitioning, it’ll be around (and may even be worse). A couple things that helped me to keep in mind:
1. You’re here for a reason — you’ve worked hard to get where you are, and have the work to prove it (pull/merge requests, or old conversation threads might help on days you’re feeling it hard)
2. Your access to customer context is one of your greatest strengths on your new engineering team, and will help everyone deliver solutions much faster
3. You’re setting the stage for other people at your organization who want to make a similar transition

I hope this helps! If you’re considering a career switch like this at your organization and want to talk, shoot me an email, I’m always happy to talk and dive into things further.

Build a Link Shortener with Cloudflare Workers: Deployment

Matt Mascioni — Fri, 23 Jul 2021 21:11:40 +0000

In this part of the tutorial, we'll be deploying your Worker to production, where users will be able to access it!

If you haven't gone through the front-end part of this tutorial yet, head back to that part first.

Before we continue, let's take another look at your wrangler.toml file to make sure everything's in order. Your file should look something like this:

type = "webpack"
name = "redirect"
account_id = "YOUR_ACCOUNT_ID"
workers_dev = true
route = ""
zone_id = ""

kv_namespaces = [ 
    { binding = "SHORTEN", id = "7654a938359f4f0e86b11afc7133166b", preview_id = "58f1a4c227534317817846d697f9ade7" }
]

[site]
bucket = "./public"
entry-point = "workers-site"

If it doesn't, please check in the back-end or front-end parts of this tutorial to ensure everything's set up properly. Once deployed, your Worker will be available for requests at <name>.<yoursubdomain>.workers.dev, where name is the project name you defined in the wrangler.toml name key, and yoursubdomain is your Workers subdomain, which you can see on your dashboard. Note that in production, your Worker will use the KV namespace available at id instead of preview_id automatically.

If everything looks good here, deploy your Worker with wrangler:

$ wrangler publish

And that's it! 🎉 Wrangler should return the URL your Worker's available at in the output, and you should be able to see it in your dashboard too. Go check it out and take it for a spin!

What we covered and what's next

Thanks for coming along this tutorial journey with me! We've covered:

What Cloudflare Workers are and some of the advantages of using them
How to use Workers KV to store key:value pairs of data, and access it in your code
How to use Wrangler to scaffold your project, interact with KV, test and deploy
Using Workers Sites to upload and serve static assets from your KV namespace
Creating API endpoints for your Worker

If you're looking to extend this project further, here are a few ideas:

Add basic reporting functionality: Make use of your KV namespace to store the number of clicks a link has received so far!
Use a custom domain: Make use of routes to deploy the worker on a custom domain! (as it stands right now, the shortener actually lengthens domains 😛)
Enable the ability to delete a link: Currently, links expire every 24 hours. Give users more control over when the link goes away.
Improving error handling: Currently, error handling is pretty basic. Improve it by adding a static 404 page when a slug isn't found/expired, or experiment with piping failures to a 3rd party log ingestion service!

Let me know if you add in any of these features or build whole new things on top of this. Hope you enjoyed the tutorial; have fun building with Cloudflare Workers!

Build a Link Shortener with Cloudflare Workers: The front-end

Matt Mascioni — Fri, 23 Jul 2021 21:11:10 +0000

In this part of the tutorial, we'll build a front-end atop our back-end to let users input a long link and get a short link back. Our front-end will be a simple Vue application that just interfaces with our POST /links endpoint, and will be served up using Workers Sites.

If you haven't gone through the back-end part of this tutorial yet to set up the routes this front-end depends on, head back to that part first!

How Workers Sites works

In the back-end part of this tutorial, we used Workers KV to store key-value pairs of slugs for long URLs. What Workers Sites allows you to do is automatically upload your site's static content to a KV namespace as well, where it can be retrieved and displayed by your Worker.

These assets are stored in another namespace that Wrangler can create for you, and your Worker can retrieve using the kv-asset-handler package. In our Worker code, we can grab the correct asset depending on the request it receives.

To get started, in your project directory, install the kv-asset-handler package: npm i @cloudflare/kv-asset-handler

A bit of home renovation first

To make this work, we'll need to re-structure our project folder a bit:

Move the index.js, package.json and package-lock.json from the project root into their own folder, which we'll call workers-site.
Create a public directory in your project root, with a static subdirectory in it.
Modify your wrangler.toml file to add this section at the bottom:

[site]
bucket = "./public"
entry-point = "workers-site"

Going forward, Wrangler will now upload your static assets in public to their own KV namespace.

At the end of these steps, your folder structure should look something like this (assuming the project root is called redirect):

redirect
| wrangler.toml
└───public
    └───static
└───workers-site
    └───index.js
    └───package.json
    └───package-lock.json

Creating the Vue application

First, copy the stylesheet from the finished project into your public/static directory.

Afterwards, copy the index.html file from the finished project directly into the public folder. This tutorial won't focus on the specifics of Vue too much, but let's explore what the code is doing. Looking at this section of the code (line 16-32):

<form @submit.prevent="handleUrlSubmit">
    <input
    type="text"
    id="input-url"
    name="url"
    size="40"
    placeholder="https://google.com"
    required
    v-model="longUrl"
    />
    <input type="submit" id="input-submit" value="Shorten" />
</form>

<div class="message" v-if="shortUrl">
    <p>Your new shortened URL is:</p>
    <h2>{{ shortUrl }}</h2>
</div>

First, we've created a data binding on our form inputs using the v-model directive. Whenever the input box for the URL is updated, the longUrl data property will be updated.

We register an event listener for the submit event on this form. The handleUrlSubmit method we define will take care of interacting with the endpoint we defined before, and will update the shortUrl data property. Once this is available, the URL will be displayed to the user (visibility toggled by the v-if directive).

Taking a look at the handleUrlSubmit method on the Vue app:

methods: {
    handleUrlSubmit() {
        fetch('/links', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
        },
        body: JSON.stringify({
            url: this.longUrl,
        }),
        })
        .then((response) => {
            if (response.status == 200) {
                this.error = false;
                return response.json();
            } else {
                throw new Error('Issue saving URL');
            }
        })
        .then((data) => {
            this.shortUrl = data.shortened;
        })
        .catch((error) => {
            this.error = true;
        });
    },
}

Here we're doing a fetch request (with very little error handling) to our /links endpoint, and retrieving the shortened value from the API response. The shortUrl data property gets set to this value, and the shortened URL is displayed to the user.

Using `kv-asset-handler` to render our application

There are two scenarios where we'd want to render static assets:

A user visits the homepage (i.e. the path is /)
A static asset is requested (e.g /static/style.css)

To intercept these requests, while still responding to requests to our API endpoints, we can define a middleware function. This would either pass the fetch event to the router or kv-asset-handler's getAssetFromKV function, which consumes a FetchEvent and returns a Response based on what's stored in the KV namespace for static assets.

Open up index.js. First, import the getAssetFromKV function:

import { getAssetFromKV } from '@cloudflare/kv-asset-handler';

Then, let's write our function to pass the event/request around:

async function handleEvent(event) {
  let requestUrl = new URL(event.request.url);
  if (requestUrl.pathname === '/' || requestUrl.pathname.includes('static')) {
    return await getAssetFromKV(event);
  } else {
    return await router.handle(event.request);
  }
}

Note that our route handlers currently take in a Request object, while the getAssetFromKV function takes in the whole FetchEvent. Next, let's hook into this function on our fetch event listener. Modify the listener from:

addEventListener('fetch', event => {
  event.respondWith(router.handle(event.request))
})

to:

addEventListener('fetch', event => {
  event.respondWith(handleEvent(event));
});

With these changes made, it's time to take the Worker for a test spin! Run wrangler dev. Notice you'll get a bit of extra output as your static assets get populated into their own KV namespace:

$ wrangler dev

🌀  Using namespace for Workers Site "__redirect-workers_sites_assets_preview"
✨  Success
👂  Listening on http://127.0.0.1:8787

And, you should be able to see it in action:

Note your URL might look a little different. If you now take this key and append it to the URL in your address bar (e.g 127.0.0.1:8787/nFXAau for me), it should redirect! Our routing has been set up properly.

If you peek at your KV namespace for the static assets in your dashboard, you should see them listed:

🎉 The front-end is ready to rock!

The front-end is ready to go and now it's time to deploy our application with Wrangler. In the next part of this tutorial we'll deploy the link shortener -- we're almost there!

➡️ Onward to deploying the application!

Build a Link Shortener with Cloudflare Workers: The back-end

Matt Mascioni — Fri, 23 Jul 2021 21:10:33 +0000

In this part of the tutorial, we'll use Cloudflare Workers to accomplish two things and build the back-end of our app:

Creating a new short link and returning the slug of that short link (e.g given a link like shorturl.at/gIUX4, the slug is gIUX4)
Redirecting from a short link to the full link, given a slug

Before we begin

If you haven't gone through the introduction to the project part of this tutorial yet to set up your environment, head back to that part first!

Let's install two packages this project will depend on. We'll make use of these two very lightweight packages:

itty-router: Will allow us to declare URL routes and parameters in those routes easily
nanoid: What we'll use to generate the random slugs to access the URLs at

Switch to the project directory we created in the last part of this tutorial, and use npm to install itty-router and nanoid:

npm i itty-router nanoid

Also, please check to ensure that you've populated your Account ID in your wrangler.toml file, and have set the type for your project to webpack to package up these dependencies.

Setting up our Workers KV Store

To store slugs and the URLs they point to, we'll use Workers KV. It's optimized for high-read applications where we'll need to access these keys frequently, and can be easily accessed from your Worker code. Each key will be a slug (e.g gIUX4), and the value the URL they point to (e.g https://www.youtube.com/watch?v=dQw4w9WgXcQ).

KV stores are organized in namespaces, and you can create them using Wrangler. Let's create one called SHORTEN:

$ wrangler kv:namespace create "SHORTEN"

You should get some output back from Wrangler on what to add to your wrangler.toml file:

🌀  Creating namespace with title "rd-test-SHORTEN"
✨  Success!
Add the following to your configuration file:
kv_namespaces = [ 
         { binding = "SHORTEN", id = "48ae98ff404a460a87d0348bb5197595" }
]

In my case, I'd add that kv_namespaces entry to the end of my wrangler.toml file. The binding key here lets you access the KV namespace by the SHORTEN variable in your Worker code. If you log in to the Cloudflare Workers dashboard, you should be able to see your namespace listed under KV too:

Clicking "View" lets you see all associated keys and values, though it should be empty now. In your Worker's code, you can now interface with this namespace through the SHORTEN variable, which we'll see in a moment.

Finally, create a preview namespace. This will be automatically used in development (i.e. when running wrangler dev) when previewing your Worker:

$ wrangler kv:namespace create "SHORTEN" --preview

Take the preview_id provided in the output of this command and add it to the wrangler.toml file, in the same entry as your SHORTEN KV namespace. For example, if my output was this:

🌀  Creating namespace with title "rd-test-SHORTEN_preview"
✨  Success!
Add the following to your configuration file:
kv_namespaces = [ 
         { binding = "SHORTEN", preview_id = "d7044b5c3dde494a9baf1d3803921f1c" }
]

Then my wrangler.toml file would now have this under kv_namespaces:

kv_namespaces = [ 
    { binding = "SHORTEN", id = "48ae98ff404a460a87d0348bb5197595", preview_id = "d7044b5c3dde494a9baf1d3803921f1c" }
]

If you check the Cloudflare Workers KV section of the dashboard, you should now see two namespaces, one appended with _preview and one not.

Generating slugs and creating short links

We'll define an API endpoint, POST /links, that takes in a JSON payload like { "url": "https://google.com" }, then:

Generates a random, alphanumeric slug using nanoid
Saves a new entry to the SHORTEN KV namespace with the key as the slug in (1) and the value as the URL in the request payload
Returns the slug, and the short URL

To do this, open up index.js in your project folder. Let's import some functions we'll need from itty-router and nanoid, create a router, and set up a custom alphabet to pick our URL slugs from (the 6 ensures they're 6 characters):

import { Router } from 'itty-router';
import { customAlphabet } from 'nanoid';

const router = Router();
const nanoid = customAlphabet(
  '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz',
  6,
);

Now, let's define our API endpoint and walk through what it's doing:

router.post('/links', async request => {
  let slug = nanoid();
  let requestBody = await request.json();
  if ('url' in requestBody) {
    // Add slug to our KV store so it can be retrieved later:
    await SHORTEN.put(slug, requestBody.url, { expirationTtl: 86400 });
    let shortenedURL = `${new URL(request.url).origin}/${slug}`;
    let responseBody = {
      message: 'Link shortened successfully',
      slug,
      shortened: shortenedURL,
    };
    return new Response(JSON.stringify(responseBody), {
      headers: { 'content-type': 'application/json' },
      status: 200,
    });
  } else {
    return new Response("Must provide a valid URL", { status: 400 });
  }
});

First, we're registering a route at /links to accept POST requests, and consuming the request object passed from our fetch event the worker listens for. We're using our custom alphabet nanoid to generate a 6-character random slug, and then pulling the URL out of the request body. We also add this to our KV namespace we generated earlier:

await SHORTEN.put(slug, requestBody.url, { expirationTtl: 86400 });

Notice the SHORTEN variable is bound to your Worker after adding it in wrangler.toml. Here we're setting the key to automatically expire in a day, but you can choose to set this to any value you want (or not make it expire at all!) If all goes well, we'll return the slug and the full shortened URL so the front-end can make use of it.

Go to this part of index.js that came with the template Wrangler used to create the project:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

This fetch event is what's received once an HTTP request is made to your Worker. We'll modify this handler to connect to your router function instead:

addEventListener('fetch', event => {
  event.respondWith(router.handle(event.request))
})

In this way, your router will be passed the Request object in the fetch event and can act on it, passing the request off to the correct route (like the one we defined above). Let's test it out! Run wrangler dev and make a test request to your Worker:

$ curl -X "POST" "http://127.0.0.1:8787/links" \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'{
  "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
}'

{"message":"Link shortened successfully","slug":"TCqff7","shortened":"https://redirect.mascioni.workers.dev/TCqff7"}

Note that the value in shortened depends on your Workers subdomain and project name you chose in the beginning. If you head over to your SHORTEN_preview KV namespace in the Workers dashboard, you should see the entry was added too!

You can also remove the handleRequest function the template comes with from index.js.

Redirecting from short links to full links

The implementation for this is similar to the one for generating the short link, except this time we're calling .get on our KV namespace and returning a redirect.

Let's register another route on our router, in index.js:

router.get('/:slug', async request => {
  let link = await SHORTEN.get(request.params.slug);

  if (link) {
    return new Response(null, {
      headers: { Location: link },
      status: 301,
    });
  } else {
    return new Response('Key not found', {
      status: 404,
    });
  }
});

This time, we're registering a GET /:slug route, and making use of the slug parameter to fetch a value from our KV namespace. If a value exists, we'll return it with a 301 status code and set the Location header appropriately to do the redirect, otherwise we'll throw a 404 error.

If wrangler dev isn't running anymore, run it again, and now make a GET request from your browser to try and retrieve the URL you stored earlier! In my case, my Worker is listening for requests on port 8787, and I saved a URL with slug TCqff7, so I'd go to http://127.0.0.1:8787/TCqff7.

🎉 Back-end is finished!

With that, the back-end of our app is basically done! We can now generate short URLs and redirect from them easily. In the next part of this tutorial, we'll use Workers Sites to build a front-end on top of this.

➡️ Onward to building the front-end!

Build a Link Shortener with Cloudflare Workers

Matt Mascioni — Fri, 23 Jul 2021 21:10:13 +0000

Cloudflare Workers is a serverless platform that allows you to run back-end code in response to an event (like an HTTP request). Similar to other serverless platforms, this means that you won't have to keep a server running to wait for requests, allowing you to save money by only paying for resources you need.

Workers run on Cloudflare's Edge Network and are incredibly fast, with a very generous free tier. In this 4-part tutorial series, we'll build one to power a link shortener that looks something like this:

The finished product is also available on GitHub if you want to dive in and use the code for your own projects! For this project, we'll be using JavaScript, but Workers support other languages too (e.g Rust)

What you'll need

A Cloudflare account. The project we're building should fall within the Workers Free plan.
Wrangler (the CLI for working with Workers) installed: e.g with NPM, npm i @cloudflare/wrangler -g: see install instructions

What we'll be building

Our link shortener has two main pieces that we'll build out:

A back-end that should be able to take a "long" URL, generate a "short" one, and return the shortened URL. It should also be able to return a redirect to the correct long URL, given a short URL.
A front-end to interact with the link-shortening back-end.

For (1), we can create a Worker to listen for HTTP requests and route accordingly:

POST /links: Generate a new short URL from a long one, returning a short slug to access it at
GET /:slug: Looks for a long URL with this slug, and redirects to it if it exists

In an application like this, the Redis in-memory database may be an awesome choice, as we could store the slugs as keys and quickly access a long URL by slug. One huge benefit with Cloudflare Workers is that a key-value store, Workers KV is built in and easily accessible from the Worker function. We'll be using Workers KV here.

For (2), our Worker can also serve static assets, and we'll store our HTML/CSS files using Workers KV too via Workers Sites. For fun, the front-end will use Vue too. More on this soon!

Getting started

Ensure you've installed Wrangler as described above. Afterwards, run wrangler login and you'll be prompted to sign-in to your Cloudflare account.
Generate a new project using a JavaScript template, since that's what we'll be using for this tutorial:
```
wrangler generate <project-name> 
https://github.com/cloudflare/worker-template
```
This will create a new folder at <project-name>. Open up wrangler.toml in that folder, and set account_id to the account ID the Wrangler CLI returns. Also, set type = webpack instead of javascript, to package some dependencies we'll be installing.

Your Worker's code is in index.js:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})
/**
 * Respond with hello worker text
 * @param {Request} request
 */
async function handleRequest(request) {
  return new Response('Hello worker!', {
    headers: { 'content-type': 'text/plain' },
  })
}

What's happening here? When an HTTP request hits Cloudflare's edge network, and there's a Worker mapped to the route that got accessed, a FetchEvent object will get passed to the fetch event listener. The FetchEvent object has a respondWith method that lets us return a response right away. The Worker uses this to return a Response object with the Hello worker! text. For other things you can do with the FetchEvent object, check out the FetchEvent lifecycle docs.

Run wrangler dev from your project directory. Behind the scenes, this creates a tunnel between your machine and the edge server that handles your requests.

You should get a URL to try sending a request to:

💁  watching "./"
👂  Listening on http://127.0.0.1:8787

Make a request to that URL. You should see Hello worker! returned. If all is working so far, it's time to dive into building the back-end!

➡️ Onward to building the back-end

Testing and debugging webhooks with ngrok

Matt Mascioni — Sun, 07 Feb 2021 20:51:52 +0000

Let's say you're building a web application that receives webhooks from external services like GitHub, Stripe or others. These webhooks work by sending HTTP POST requests to an endpoint on your server, allowing you to listen for events as they happen without having to continuously poll for new data. If you're developing locally though, you won't be able to receive webhooks at http://localhost:8000 (or any other local URL!) This is where ngrok comes in, allowing you to create a temporary public URL to tunnel HTTP traffic to your local application.

In this tutorial I'll walk through installing ngrok and using it to test out the Stripe webhook, but this can work for any webhook you need to work with!

Installing ngrok

Head over to ngrok and sign up for an account. You don't really need one for this tutorial so you can skip this step if you want, but having an account gives you a few extra features (like longer session times)
On macOS, ngrok can be installed via Homebrew: brew install --cask ngrok. You can also download an executable for macOS, Windows or Linux directly from their website. Keep track of where it's been downloaded so you'll know which path to access it at after.
Try running it from a terminal window to make sure it's working: ngrok help.

The syntax for tunneling HTTP traffic to a port on your local computer is ngrok http <port_to_tunnel_to>. For example, if I wanted to tunnel HTTP requests to a server listening for requests at port 5000 on my local machine, I would type ngrok http 5000 in a shell:

You'll notice the Forwarding: https://11d51c640b97.ngrok.io -> http://localhost:5000. This means all requests to https://11d51c640b97.ngrok.io will be forwarded to port 5000 on my local machine, where my application can receive them! You can also access a request inspector in a browser at http://127.0.0.1:4040, where you can see (and even replay) any HTTP requests that went through. Press Ctrl+C whenever you're done and the tunnel will be closed.

Using ngrok to test the Stripe webhook

In this example I'm going to use a Flask application that's listening for incoming webhooks from Stripe at /webhooks/stripe. Let's say I'm interested in charge.succeeded events, because these tell me someone has paid for something on my fictitious platform! If you want to follow along with this tutorial, all the code is available on GitHub. Set up your local environment and install a few dependencies (ensure Pipenv is installed first):

git clone https://github.com/mm/stripe-ngrok-example.git
cd stripe-ngrok-example
pipenv install

Let's take a look at the code in app.py, which defines the endpoint that's listening for events:

from flask import Flask, request, jsonify

app = Flask(__name__)


@app.route('/webhooks/stripe', methods=['POST'])
def receive_stripe_webhook():
    """Receives a webhook payload from Stripe.
    """

    # Try to parse a webhook payload, get upset if we couldn't
    # parse any JSON in the body:
    stripe_payload = request.json
    if not stripe_payload:
        return jsonify(message="Could not parse webhook payload"), 400

    event = stripe_payload.get('type')
    if not event:
        return jsonify(message="Could not determine event type"), 400
    if event == 'charge.succeeded':
        # Pull fields out of payload:
        data_object = stripe_payload.get('data').get('object')
        customer_id = data_object.get('customer')
        amount = data_object.get('amount')
        # Here we just log the transaction, but at this point we can do
        # anything! (Provision accounts, push to a database, etc.)
        print(f'Customer {customer_id} made a purchase of {amount} cents!')       

    return jsonify(message="Webhook received"), 200

Besides a bit of input data handling, all this code is doing is waiting for a POST request to /webhooks/stripe, checking for a charge.succeeded event (we successfully charged someone on our platform), and logging a bit of information about the charge to the console. You can start a local Flask server by running pipenv run flask run in a shell from the project directory. The server will let you know where it's listening for new requests -- in my case, http://127.0.0.1:5000/. Let's expose this via ngrok and have Stripe send us some data!

While the server's running, in a new terminal window type ngrok http 5000 (or whatever port the Flask app is listening at):

Notice the forwarding URL generated by ngrok. In my case, mine is https://07136e03dd33.ngrok.io. Now, let's have Stripe send data here:

Log in to your Stripe account (here I'm using a test account)
In the left sidebar, click on "Developers", then "Webhooks".
Under "Endpoints", click "+ Add Endpoint". The endpoint URL is the forwarding URL ngrok gave you (plus whatever path your server is listening at). In my case, it's https://07136e03dd33.ngrok.io/webhooks/stripe (this would be equivalent to http://localhost:5000/webhooks/stripe)

Here I set it up so I only receive charge.succeeded events, but there's all kinds you can listen for in the Stripe docs (my favourite docs ever)
Next, record a charge for a customer. I'm in a test environment, so I'll just record a charge for a test customer named after me:

If you've been checking your Flask server, you'll notice we received a request to our endpoint in the logs:

Customer cus_Itm097l6URXxaZ made a purchase of 400 cents!
127.0.0.1 - - [06/Feb/2021 16:39:42] "POST /webhooks/stripe HTTP/1.1" 200 -

Head over to http://localhost:4040 in your browser (or whatever URL ngrok gave you for your "Web Interface") Here, you'll see a log of the request made to your local app:
Let's say you make some changes to your code and want to test the same payload against the new code. You don't need to record another payment in Stripe! You can hit the "Replay" button in the ngrok inspector and replay the request (or even modify it before replaying). This can help you test code faster without adding huge amounts of data to your test environment (be it Stripe or any other)
Hit Ctrl+C on both your ngrok session and Flask server once you want to stop receiving requests. That's all there is to it.

Wrapping up

In this tutorial, we covered:

How to tunnel HTTP requests to your local server to test webhooks with ngrok
Using the ngrok inspector to inspect and replay requests to your server without needing to generate more events

Let me know if it was helpful in testing your own applications. Here we covered testing the Stripe webhook this way, but this strategy can work for other webhooks too!

Header photo by Paul Carmona on Unsplash

What Should We Play? - A DO Hackathon Submission

Matt Mascioni — Sat, 26 Dec 2020 21:08:09 +0000

What I built

I built a searchable index of online games people can play with each other (or on their own!) to feel connected while social distancing measures are in place. It can also quickly pick a game for you depending on how many people are in your online party, and anyone can suggest new games to help improve the index for others!

Category Submission:

Program for the People

App Link

https://whatshouldweplay.xyz

Screenshots

Description

This is a little website I built to showcase games you could play with friends online while on a Zoom/Meet/FaceTime/some other call together. The site breaks down games based on how many players can fit in a game and whether they're free or not. Besides searching, anyone can quickly have the website generate game suggestions for them depending on their party size! The site also features a suggestion form so that anyone can help add new games to the index. This submission is broken down into the back-end (main repository), front-end as well as an admin panel to quickly review and add in new suggestions captured on the form.

The entire project was written using Python for the backend and React in the frontend, with Auth0 managing authentication for the admin panel.

For those trying to use this data/functionality in their own applications, I've also exposed certain endpoints on the app's REST API that are documented here.

Link to Source Code

The back-end, front-end and admin panel code live in separate repositories. Each have a Deploy to DigitalOcean button to easily get them live on App Platform or a Docker Compose file to get up and running on your local machine!

Main repository and back-end: https://github.com/mm/wswp
Front-end: https://github.com/mm/wswp-frontend
Admin panel: https://github.com/mm/wswp-admin

Permissive License

All repositories are released under the MIT License:

Back-end: https://github.com/mm/wswp/blob/main/LICENSE
Front-end: https://github.com/mm/wswp-frontend/blob/main/LICENSE
Admin: https://github.com/mm/wswp-admin/blob/main/LICENSE

Background

Throughout the lockdown, a constant source of entertainment has been playing games with friends or co-workers. "What should we play?" ended up getting asked a lot on those calls, and I saw a lot of threads on Reddit with great games to play and news articles but no website dedicated to it, so this was a cool opportunity to try building one!

How I built it

The tech I used in this project was:

Back-end: Python (Flask, SQLAlchemy, Marshmallow), PostgreSQL
Front-end: React, Chakra UI
Auth/Identity: Auth0

I love working on back-end projects and APIs, but I hadn't dabbled in front-end work too much. I really wanted to learn more about building React apps, so this was a project that allowed me to dive into React in general and learn how to use Chakra UI, an amazing component library that helped me make this whole thing a reality. Their documentation is stellar, and I have a whole new level of appreciation for how tricky UI work can be. Also, building an admin panel and submission functionality gave me an excuse to try out authentication with JWTs and Auth0 as my identity platform. Their Flask and React documentation helped me get up and running quickly.

DigitalOcean's App Platform powered this project the whole way through. I couldn't believe how intuitive it was to get my API + database set up, and then add a static site atop it with environment variables set dynamically depending on the app's base URL or database URL. I deployed the admin panel as a separate static site to not complicate the main build too much. Some things I really appreciated about App Platform:

Easily adding a static site to a project that already has a service & database defined
Environment variables super easy to manage
Built-in console a huge plus for running CLI tasks on the fly (like running database migrations)
Painless domain management and component HTTP routing

I'm hoping to use this project to write a tutorial about deploying Python/Flask + React apps to App Platform in the future!

Additional Resources/Info

Component library powering the front-end (and the awesome dark mode): Chakra UI
Icons courtesy of Octicons, Hero Icons, and Font Awesome via react-icons
If you see something that could be improved on the site, please let me know or open up an issue on GitHub! I had so much fun developing this and would love to build it into something the community could get use out of.

Using external Python packages with AWS Lambda layers

Matt Mascioni — Sun, 06 Dec 2020 17:56:00 +0000

Amazon Lambda is an awesome service that lets you run code serverlessly: rather than keeping a server running to execute your code anytime, a server is spun up only when necessary in response to some event. This event could be from an HTTP request, a message posted to a queueing service like Amazon SQS, or even when a file is uploaded to an S3 bucket!

Lambda supplies different runtimes depending on what language you want to write your function in. For Python programming, this runtime includes the Python standard library, but what if you want to use external packages from PyPI or elsewhere? This is something you can do with Lambda layers! Layers provide dependencies (or even a custom runtime) for your function, and in this quick tutorial we'll walk through how to use them!

What you'll need

An AWS account and a little experience using Lambda functions. If you haven't used them before, Amazon has great tutorials!
Docker installed on your computer, as we'll use that to install dependencies in a similar environment to the AWS Lambda Python runtime

Our demo project will be a very simple Lambda function that accesses the PokéAPI's /api/v2/pokemon endpoint to return information about a Pokémon, given a name passed in the event that triggers the function.

Setting up our project locally

First we'll create a virtual environment for our project and work in there, to keep any project dependencies separate from others on your computer:



$ mkdir poke-lambda 
$ cd poke-lambda
$ python -m venv venv
$ source venv/bin/activate

We're using the requests library to do our API calls, so install that with pip install requests. Our one file, lambda_function.py looks like this:



import json
import requests

def lambda_handler(event, context):
    # Make sure a Pokemon name (or ID) was passed in the event object:
    pokemon = event.get('pokemon')
    if pokemon is None:
        return {
            'statusCode': 400,
            'body': json.dumps('Missing pokemon attribute in the event object')
        }

    # If we have a pokemon name/ID passed in, try to get info for it:
    r = requests.get(f"https://pokeapi.co/api/v2/pokemon/{pokemon}")

    if r.status_code == 200:
        status_code = 200
        body = json.dumps(r.json())
    else:
        status_code = r.status_code
        body = json.dumps(f"Could not load information for Pokemon {pokemon}")

    # Return what we got from the API:
    return {
        'statusCode': status_code,
        'body': body
    }

What this code does is check for the pokemon attribute in the event data the Lambda function receives, queries the PokéAPI for information about the Pokémon passed in, and returns the API's response back (with basic error handling). At this point our project structure looks like this:



poke-lambda
|   lambda_function.py
└───venv

Building our Lambda layer

In order to put this code on AWS Lambda, we need a way to include the requests library first! Lambda layers are .zip files containing libraries (and a custom runtime if you need) your function requires. For Python lambda layers, AWS requires libraries to be placed in either the python or python/lib/python-3.x/site-packages folders.

While the virtual environment is activated for your project, save a list of its dependencies to a requirements.txt file: pip freeze > requirements.txt.
Create a python folder. This will house all the libraries to go in our layer. At this point, your folder structure should be:
```
poke-lambda
| lambda_function.py
| requirements.txt
└───venv
└───python
```


3. From the project root, run the following command to build your dependencies in a container similar to the Lambda execution environment Amazon provides (courtesy of [lambci](https://hub.docker.com/u/lambci)):

    ```shell


    $ docker run --rm \
    --volume=$(pwd):/lambda-build \
    -w=/lambda-build \
    lambci/lambda:build-python3.8 \
    pip install -r requirements.txt --target python

Let's break down what this command does, argument-by-argument:
- `--rm`: Makes sure that once the Docker container exits, it's removed (keeps things clean)
- `--volume`: [Bind mounts](https://docs.docker.com/storage/bind-mounts/) the current working directory to the `/lambda-build` directory on the container (this allows the container to access the requirements.txt file we've just generated, and add files to the `python` directory). If you're on Windows, you can also paste in a full path to your project root instead of `$(pwd)`.
- `-w`: Sets the working directory in the container (in this case, to our project root)
- `lambci/lambda:build-python3.8`: The docker image to run this container from -- make sure it matches the Python version you're working with! For reference, Amazon currently provides runtimes for Python 2.7, 3.6, 3.7 and 3.8 (I'm using 3.8 here)
- `pip install -r requirements.txt --target python`: Installs the project dependencies as normal, from the requirements.txt file to the `python` folder (more on that [here](https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-t))

You should see some output from pip. This is running in the Docker container. If successful, you should have dependencies installed to the python folder you created earlier! Mine looks like this:



poke-lambda
| lambda_function.py
| requirements.txt
└───venv
└───python
    └───bin
    └───certifi
    └───certifi-2020.11.8.dist-info
    └───chardet
    └───chardet-3.0.4.dist-info
    └───idna
    └───...


5. At this point, all we have to do is zip our `python` folder: `zip -r layer python/`. This will create a `layer.zip` file in your project's root directory. 

6. Next, upload the .zip file to Lambda! Sign into your AWS Console and head over to Services > Lambda > Layers (it should be under "Additional resources"). Click "Create layer" and give your layer a name! Pick the runtime that corresponds to the version of Python you're trying to use in your function, and upload the .zip file you created earlier:

![Filling out the name, runtime and layer .zip file in the Create Layer form](https://dev-to-uploads.s3.amazonaws.com/i/cgv94fjghy0tfy3vf7y0.png)

## Adding the layer to our Lambda function

We'll walk through this using the front-end console, but you can totally do this all over the AWS CLI!

1. While in your AWS Lambda console, click "Create function". Set it up however you want (pick a Python runtime though)

2. In the Lambda function designer, click on the "Layers" button:

    ![Clicking on the Layers button in the designer](https://dev-to-uploads.s3.amazonaws.com/i/gecow286pxsz8eqebxi0.png)

3. A table full of layers your function is using should appear below the designer. There aren't any so far, so lets add ours! Click "Add a layer" and choose "Custom layers". You should be able to see your layer you created earlier in the dropdown. Select and add it!

4. Once added, you'll be taken to the Designer again. Scroll down and paste your function code in (you can alternatively upload a .zip file containing just the `lambda_function.py` file since that's all we need here)

    ![Pasting in our function code from before](https://dev-to-uploads.s3.amazonaws.com/i/6inl9dz4julagqrpe8ei.png)

5. Save and click "Deploy" to update your function's code. Let's send our code a test event! At the top of the page, click on "Configure test events" under "Select a test event". We'll create an event payload to fetch details about Charizard:

    ![Configuring the test event for the Lambda function. This is a JSON payload with only one key (pokemon) equal to Charizard](https://dev-to-uploads.s3.amazonaws.com/i/4fl59jxnbpzol9uj8aki.png)

6. Save your test event, pick it from the dropdown and click "Test". You can now see your function returning results!

    ![Testing the Lambda function with the test event configured in step 5 should result in a success](https://dev-to-uploads.s3.amazonaws.com/i/w1pzxxndhl85kwxl5dtn.png)

That's all there is to it! Keep in mind functions can have multiple Lambda layers (up to 5 right now) and functions can share layers too.

In this tutorial we:

- Wrote a simple Lambda function that calls an external API and returns the results
- Created a Lambda layer containing all dependencies for our project that we can reuse in other projects later, using the [lambci/lambda](https://hub.docker.com/r/lambci/lambda) images and the Lambda GUI

Keep in mind all of this can also be accomplished over the AWS CLI (uploading the function, creating the layer and so on) if you prefer this method. There are lots of [AWS docs](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-using) which document this pretty well!

Hope you enjoyed this tutorial! Let me know if it was helpful in your project 😄

Tutorial: VPN on Demand with Siri, Shortcuts, Python, AWS EC2 & Lambda

Matt Mascioni — Wed, 29 Jul 2020 14:42:33 +0000

Sometimes you need a VPN for a short period of time (like when you'll be on public Wi-Fi for a little while). Amazon EC2 makes deploying servers to be used as VPNs for this purpose pretty simple and cost-effective since you're only charged from the time you start until the instance's termination. Open-source projects like hwdsl2/setup-ipsec-vpn make this even easier. However, this can require you to get onto the AWS Console to get everything set up, which can sometimes be a pain. The motivation for this project was to be able to do this on the go:

(It works on iOS 13 too 😊)

To make this work, we're going to be deploying some Python code as a Lambda function on AWS. This code will start/stop/list EC2 instances for us based on an EC2 template we'll create. This template will run the setup-ipsec-vpn script on new instances automatically. We'll use AWS API Gateway to create a RESTful API that will trigger the Lambda function. Finally, we'll use Apple's Shortcuts app to make an HTTP request to the API endpoint we create. Visually, this is what's going on:

(Icons used above can be found at AWS)

What you need:

An Amazon AWS account. If this is your first time signing up, you'll be eligible for the free tier.
A computer with Python 3 and the AWS CLI installed and configured
An iOS device with the Shortcuts app installed

Limitations/Disclaimer

You'll be limited to deploying only in one AWS region at a time, since EC2 Launch Templates are tied to a region. In this tutorial, we use us-east-1 as our region.
About AWS charges: Some of the services used in this post are intended to fall within the free tier usage, however some do not. Lambda offers 1 million Lambda requests every month even once your free-tier access expires, while API Gateway only offers 1 million requests per month for the first year, and EC2 offers 750 hours per month on t2.micro instances (what we're using) for the first year. Make sure to review pricing for Lambda, API Gateway and EC2 before continuing. Also, as we test things out in this tutorial, keep an eye on your AWS console to make sure anything you created to test with gets deleted to avoid incurring extra charges.
This is more of a demo so far than anything else! I'd love any feedback on how to make it more secure or user-friendly in the future.

Let's get started!

Creating the EC2 Launch Template

EC2 Launch Templates are a great way to save a frequently used EC2 launch configuration (i.e. instance type, security groups, firewall rules, storage, etc.). In our case our launch template will have all the necessary configuration necessary to deploy an EC2 instance as a VPN for us.

Sign into your AWS Console and ensure you're in the region you want to deploy your VPNs to (you can see this at the top right of the screen). In this guide we use us-east-1 (North Virginia). Head over to Services > EC2.
First, we'll set up a Security Group, which contains the firewall rules for our EC2 instance. On the left side of the page, click on "Security Groups" under "Network and Security. Click "Create Security Group".
Give your group a name, leave the VPC dropdown at its default, and set the following inbound UDP rules (leave Outbound at its default): this will allow you to connect to your server.

(Note: For greater security, you can restrict Source further here: perhaps to a range of IP addresses)
Save the security group. Next we'll create a key pair, useful if you want to SSH into your instance in the future. Under "Network & Security", click on "Key Pairs".
Click "Create Key Pair" and follow the instructions (if you already have set up one, you can use that key pair instead for the remainder of this guide)
Now we'll create our launch template. Under "Instances", click on "Launch Templates", and then "Create launch template". Give the template a name, and begin filling out the form. For the AMI, I chose one that corresponded to Ubuntu 18.04 (you can check what OS's are currently being supported by hwdsl2/setup-ipsec-vpn). I filled out the following details:
```
Instance Type: t2.micro (Free Tier eligible)
Key pair: The key pair you created in step (5)
Security groups: The security group you created in step (3)
Storage (volumes): If a volume wasn't added automatically once you picked the AMI, click "Add new volume". Make sure "Delete on termination" is set to Yes, and leave everything else at default.
```
Under "Resource tags", click "Add tag" and set a key of instance_type to vpn. Leave "Resource types" at its default (should have "Instances" selected). This will cause EC2 instances deployed with this template to be tagged, which lets us keep them separate from other instances and makes sure our Lambda function terminates the right instance when asked to.
Under "Advanced details", scroll to "User data". This is where we'll put the script from hwdsl2/setup-ipsec-vpn in. Make sure to put #!/bin/bash at the top of the script since this is being executed in a Bash shell. It's executed when your instance is booted for the first time. To find out more about User Data, check the Amazon docs on it.

Fill out VPN_IPSEC_PSK, VPN_USER and VPN_PASSWORD with an IPSec PSK (should be >20 random characters), a VPN username and password. Generate these randomly and store them in a password manager. Every time an instance is created, these credentials will be used and only the IP address will change.

Note: More secure than using user data here may be to have this shell script in a secured S3 bucket accessed at launch time or pass the user data at launch time with Lambda, pulling credentials from Lambda environment variables.
Save the launch template and note the name you gave to it. Also note the launch template ID. If you'd like to test and see if it works, you can launch an EC2 instance based on it! Follow the connection instructions here after it's been up and running for a few minutes (it can take a little bit as the script is installing/updating packages)

Otherwise, let's move on to the Lambda part of the project!

Setting up our Lambda function

Amazon Lambda is a service that allows you to write code and execute it using various triggers (here we use an HTTP request). What's so neat about it is you're only charged for the time and resources your code uses while it runs, saving quite a bit. Here, we're using some Python code to automatically launch an EC2 instance based on the template we defined above (or terminate/list instances).

I've already written the code and you can check it out on GitHub here. It uses the Flask framework to define a tiny API that can receive HTTP requests and act on them accordingly. We'll package this up and deploy it as a Lambda function with the popular Zappa package a little later. Before that, we need to ensure our function will have the right permissions and make sure it works locally!

Creating the necessary IAM policy

For Lambda to be able to perform EC2 actions on your behalf (starting servers up, shutting them down...) it needs permissions to do so. When using AWS, we define these permissions by an IAM policy. We'll set one up here:

Log in to your AWS Console > Services > IAM. Select "Policies" from the sidebar, and then "Create policy". Switch from the visual editor to JSON mode and replace the policy in the text editor with the following:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "ec2:Describe*",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:RunInstances",
                "ec2:CreateTags"
            ],
            "Resource": "*",
            "Condition": {
                "ArnLike": {
                    "ec2:LaunchTemplate": "arn:aws:ec2:AWS_REGION:ACCOUNT_ID:launch-template/LAUNCH_TEMPLATE_ID"
                }
            }
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "ec2:CreateTags",
            "Resource": "arn:aws:ec2:*:ACCOUNT_ID:instance/*",
            "Condition": {
                "StringEquals": {
                    "ec2:CreateAction": "RunInstances"
                }
            }
        },
        {
            "Sid": "VisualEditor3",
            "Effect": "Allow",
            "Action": [
                "ec2:TerminateInstances",
                "ec2:StartInstances",
                "ec2:StopInstances"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "ec2:ResourceTag/instance_type": "vpn"
                }
            }
        }
    ]
}

Fill in the AWS_REGION, ACCOUNT_ID and LAUNCH_TEMPLATE_ID with the AWS region you made your EC2 launch template in (for example, I used us-east-1), your AWS account ID (numeric) and your launch template ID from the last section.

This policy allows your function to describe details about any EC2 instance, create new instances when your launch template was specified, create tags on an instance during its creation, and only start/stop/terminate instances tagged as VPNs. For more examples of policies if you're interested (as they relate to EC2 instances), Amazon's docs have a wealth of info. With this policy, any parameters in your launch template can be overridden at launch time (ideally you'd want to lock this down further).

Click "Review policy" and give your policy a name. Then click "Create policy" to save the policy.

Creating a test IAM user with the IAM policy

This step allows us to test out the code locally before deploying. Here we'll create an IAM user and attach our policy to it.

Log in to your AWS Console > Services > IAM. Select "Users" from the sidebar, and click "Add user".
Give the user a username (doesn't matter) and make sure Programmatic access is checked.
Click Next, and click "Attach existing policies directly". Search for your policy, ensure it's checked and click Next again. You can leave tags blank. Finally, click Create user. Make note of the Access key ID and Secret access key -- we will be using these shortly.

Working with the code locally

With all of the policy setup out of the way, we're ready to go! Now, we'll deploy a web service I wrote using Flask to Lambda. First, we'll test the code locally, and once it's working, we'll use Zappa to automate deployment. These instructions are also on my GitHub repo for this project.

Ensure you've installed the AWS CLI, configured your environment and have Python 3 and pip ready to go for local development.
Clone this repository in a directory of your choosing: git clone git@github.com:mm/siri-shortcuts-vpn.git
I recommend setting up a virtual environment for this project to keep its dependencies/versions of those dependencies separate from other projects.
Once in the project folder, with your virtual environment activated, install the packages this code relies on: pip install -r requirements.txt

Create an .env file in the root folder of the project and fill in the following environment variables. This will allow you to simulate using the IAM policy you created in tutorial to ensure it works before deploying.

AWS_ACCESS_KEY_ID=access_key_id_you_just_created
AWS_SECRET_ACCESS_KEY=secret_access_key_you_just_created
AWS_DEFAULT_REGION=fill_in_your_aws_region_here_(e.g us-east-1)
LAUNCH_TEMPLATE_NAME=fill_in_your_launch_template_name_here

If everything is in order, you should be able to run python3 app.py in the root directory to start up the Flask development server. Look out for the command's output. By default, the server will be listening for connections on port 5000.
Try making HTTP requests to your server! You can use cURL, httpie, Paw, Postman or any API testing tool you'd like. For example, if I was trying to deploy instances in the US-East-1 region (and that's where my launch template was stored), with cURL I'd make this request to try starting up:
```
$ curl -X POST http://localhost:5000/instances/us-east-1
```
If successful I should get a response like this:
```
{"instance_id":"i-some-instance-id","ip":"<some ip address>","region":"us-east-1"}
```

Getting running instances can be done via a GET request (i.e.

curl -X GET http://localhost:5000/instances/us-east-1

) and terminating instances in a region can be done via a DELETE request (

curl -X DELETE http://localhost:5000/instances/us-east-1

). The code attempts to make sure only 1 instance is running at a time (to protect against accidentally starting many instances at once) -- so DELETE works as intended (deletes the whole collection of instances, but that collection is only supposed to contain 1 instance at a time).

Once you're sure everything is working okay, let's deploy to Lambda!

Deploying to Lambda and API Gateway

For deployment, we could totally wrap up our Flask app ourselves and set up API Gateway to send requests to it. However, this would take a little time and is a bit beyond the scope of this tutorial. Zappa is an open-source Python tool I've been using for a while now. It can:

Wrap your code up (including all dependencies) in a .zip package that is automatically uploaded to Lambda
Create an AWS API gateway endpoint (and help you secure it with an API key/another form of authorization). This endpoint is set up as a Lambda proxy integration, where API Gateway passes all request data (method, headers, etc.) to your Lambda function transparently. This means you can set up routing in your Flask, Django or (insert framework here) app and not have to fiddle too much with routing in API Gateway.

... among other things! I highly recommend checking out the docs for Zappa, it's an extremely useful tool for serverless apps like this.

While in the project directory from before (and in your virtual environment), run zappa init in your terminal. You should get something like this:

Go through the setup steps and keep the default options as you go through. Open up your zappa_settings.json file. Mine looks like this:

{
    "dev": {
        "app_function": "app.app",
        "aws_region": "us-east-1",
        "profile_name": "default",
        "project_name": "siri-shortcuts-",
        "runtime": "python3.8",
        "s3_bucket": "zappa-9bh21r44r"
    }
}

First, ensure your aws_region value matches the AWS region your EC2 template is located in. If not, change that (I've been using us-east-1 throughout this tutorial). Next, we're going to add two keys: api_key_required (to secure our API with an API key) and another key aws_environment_variables to set environment variables in the Lambda environment (currently we only need one for our launch template name):

{
    "dev": {
        "app_function": "app.app",
        "aws_region": "us-east-1",
        "profile_name": "default",
        "project_name": "siri-shortcuts-",
        "runtime": "python3.8",
        "s3_bucket": "zappa-9bh21r44r",
        "api_key_required": true,
        "aws_environment_variables": {
            "LAUNCH_TEMPLATE_NAME": "Your launch template name here (not ID)"
        }
    }
}

Save zappa_settings.json. We should be good to go! Assuming you named your stage dev (as in Zappa setup), you can run zappa deploy dev in your terminal! (Side note: Zappa supports deploying multiple stages! i.e. You could have totally different environment variables/security settings for the dev vs. prod vs. staging vs. whatever stage you want. Very cool!) This could take a few minutes while dependencies are packaged and uploaded. There should be a bunch of output. At the end, you should see something like this:
```
Deploying API Gateway..
Created a new x-api-key: zgjcmuieo1
Deployment complete!: https://bsy9qfjo7k.execute-api.us-east-1.amazonaws.com/dev
```
That URL in the Zappa output is actually part of our endpoint URL! We'll be making requests to it soon. As you can see from the last step, an API Key was created for our project, but it hasn't been associated with our endpoint yet. Let's secure that now. Sign into your AWS Console > API Gateway. You should see your project listed as an API! If not, ensure your region is set to the one you used in zappa_settings.json. Click on it, and then click on Usage Plans.
Create a new usage plan (it can be named whatever you want), and enable throttling. I keep mine restrictive since I'm the only one supposed to be using it anyway: at 1 request per second and 1000 per month. Hit "Next."
Add your API there! Click "Add API Stage" and then choose your API from the list, as well as the dev stage. Click the grey checkmark and hit "Next". Click "Add API Key to Usage Plan".
For the "Name", search dev_ plus the beginning of your endpoint URL. For example, Zappa outputted https://bsy9qfjo7k.execute-api.us-east-1.amazonaws.com/dev in step 4, so I chose dev_bsy9qfjo7k:
Hit the grey checkmark, click "Done" and you're good to go! You should see a list of Usage Plans on your account. Click the one you just created then "API Keys". Click the API key you added, and then click "Show" where it says API key. Make note of this key as it's the key you'll use to make requests!
Head to Services > Lambda in your console. Again, you should see the Zappa-created Lambda function there. Click on it, then click on "Permissions". You should see an execution role listed there. Click on it. This is the role your Lambda function uses, and to add extra permissions to it (the ability to launch and shut down EC2 instances), we'll attach the policy we created earlier.
Click on the "Attach policies" button, and search for the IAM policy you created a few sections back. Select it, and click "Attach policy". That's all!

After all this work, it's finally time to test it! Here I'm using cURL, but you can use whatever API testing tool you want:

$ curl -H "x-api-key: api_key_here" -X GET https://bsy9qfjo7k.execute-api.us-east-1.amazonaws.com/dev/instances/us-east-1

{"region":"us-east-1","running_instances":[]}

If we want to start up an instance, we POST to the endpoint instead. We can see we get back the server's IP in the response (we can use this to connect!)

$ curl -H "x-api-key: api_key_here" -X POST https://bsy9qfjo7k.execute-api.us-east-1.amazonaws.com/dev/instances/us-east-1

{"instance_id":"i-09912cc4f30e020d7","ip":"54.236.255.219","region":"us-east-1"}

Note that if you try and run this when a VPN is already running, the function will return a 429 - Too Many Requests response instead. This is to prevent accidentally starting too many instances up and incurring additional charges.

To connect to your instance, wait a few minutes and then follow this guide. Use the IP address your function outputs, as well as the key, username and password you set way back when you made the EC2 launch template! You should have an up and running VPN.

If we want to delete the instance, we can send a DELETE request and get some info about how many instances we terminated and where (or, we could go to our EC2 dashboard manually as always)

$ curl -H "x-api-key: api_key_here" -X DELETE https://bsy9qfjo7k.execute-api.us-east-1.amazonaws.com/dev/instances/us-east-1

{"instances_terminated":1,"region":"us-east-1"}

If ever you want to remove everything (delete the endpoint, Lambda function and so on), you can run zappa undeploy dev from the project directory (or just remove from the AWS Console directly).

Making use of our API with Shortcuts and Siri

Since we've created a web service, technically anything capable of making an HTTP request (and passing headers for our API key) should be able to trigger our function. For this guide, we'll use Shortcuts, an iOS app by Apple. Of the many actions Shortcuts has, "Get Contents of URL" is the primary workhorse we'll be using as it can make arbitrary HTTP requests and output the results. I've made 3 shortcuts for you to use and look through how they work. Click on them to install the shortcut on your iOS device (it'll ask a few questions on import like what your endpoint URL and API key are) and review what they do before running them. You might have to change your device's settings to allow untrusted shortcuts to run.

Start EC2 VPN: This will make a POST request to your endpoint to start a new server up. If successful, it will copy the server's IP to your clipboard.
List running VPNs: This will make a GET request to your endpoint, and afterwards return the number of servers running (should at most be 1) and copy the IP address to your clipboard.
Terminate EC2 VPN: This will terminate all VPN instances (with a DELETE request) running in a given region, and tell you how many instances were terminated.

To get Siri up and running, simply tell Siri to "Run Shortcut Name", for example "Run Start EC2 VPN" to launch a new VPN. I'd give the server some time before trying to connect to it (in my tests one took close to 5 minutes once). You can follow the instructions at hwdsl2/setup-ipsec-vpn to get connected!

That's all there is to it! Thanks for checking out this tutorial/demonstration; it's been helpful for me in the past and was fun to try out so I hope it helps you too! This is one of the first technical tutorials I've written publicly, so I'd really appreciate any feedback you have!

Huge thanks to Lin Song for writing the setup-ipsec-vpn setup script our template uses and documentation for connecting! 😊

Heartbridge: A command-line tool to transfer iOS Health heart rate data to your computer

Matt Mascioni — Thu, 21 May 2020 21:22:57 +0000

My Final Project: Heartbridge

This was a small project I put together this year (final year of engineering woo!) I wanted an easy way to transfer heart rate readings from my iPhone/Apple Watch to my Mac so I could play around with the data using something like pandas or visualize it differently. This command-line tool is the result! It can export data as a CSV or JSON file depending on what arguments get passed in.

Link to Code

mm / heartbridge

Using the iOS Shortcuts app and Python together to export iOS Health data to a JSON/CSV file.

Heartbridge: iOS Health Data Export

Heartbridge is a command-line tool that exports health data from your iOS device to your local computer, with the help of an iOS Shortcut. It supports exporting many types of data from the Health app, including:

Heart Rate
Resting Heart Rate
Heart Rate Variability
Steps
Flights Climbed
Cycling Distance

Heartbridge receives data from the Shortcuts app (via HTTP), automatically exports it to the directory of your choosing (in CSV or JSON format) and automatically names files according to the health data type and date range they cover. Exported files contain a time stamp ("Start Date" in Health) and reading ("Value" in Health). To read more about how the file exports look depending on the data type, check out Data Type Support.

If you don't want to use the built-in CLI or server, you can also use Heartbridge to parse data from Shortcuts directly-- for…

View on GitHub

How I built it/how it works

This was built in Python (as of now it only depends on what comes with the Python standard library), and makes use of an iOS shortcut (using the native Shortcuts app) I wrote as well. The Python package, when run will start accepting HTTP requests at a specified port (defaults to 8888).

When the shortcut is run on the user's iPhone, they'll be asked to select a date range of heart rate data they want to export. Readings will be sent in the body of an HTTP POST request to that same port on their computer, provided they're on the same network. The JSON in the request body is processed further by Heartbridge once it is received, and the readings are exported to either a CSV or JSON file for further exploration.

This project was a great way to learn about using a continuous integration tool for unit testing (to make sure my unit tests pass with every commit I make). For this I used Travis CI to hook into my GitHub repo and run code in my commits against the unit tests I had written. This was offered as part of the GitHub Student Developer Pack and I loved using it!

This was also my first time submitting my code as a package on PyPI, which allows anyone with pip to install it at the command line and run it easily. This required me to structure my code as an overall package, which I've always wanted to learn and finally got a chance to with this project. I'm excited to create more of them in the future!

Forem: Matt Mascioni

Switching from Customer Success to Engineering: a year later

Build a Link Shortener with Cloudflare Workers: Deployment

What we covered and what's next

Build a Link Shortener with Cloudflare Workers: The front-end

How Workers Sites works

A bit of home renovation first

Creating the Vue application

Using kv-asset-handler to render our application

🎉 The front-end is ready to rock!

Build a Link Shortener with Cloudflare Workers: The back-end

Before we begin

Setting up our Workers KV Store

Generating slugs and creating short links

Redirecting from short links to full links

🎉 Back-end is finished!

Build a Link Shortener with Cloudflare Workers

What you'll need

What we'll be building

Getting started

Testing and debugging webhooks with ngrok

Installing ngrok

Using ngrok to test the Stripe webhook

Wrapping up

What Should We Play? - A DO Hackathon Submission

What I built

Category Submission:

App Link

Screenshots

Description

Link to Source Code

Permissive License

Background

How I built it

Additional Resources/Info

Using external Python packages with AWS Lambda layers

What you'll need

Setting up our project locally

Building our Lambda layer

Tutorial: VPN on Demand with Siri, Shortcuts, Python, AWS EC2 & Lambda

What you need:

Limitations/Disclaimer

Creating the EC2 Launch Template

Setting up our Lambda function

Creating the necessary IAM policy

Creating a test IAM user with the IAM policy

Working with the code locally

Deploying to Lambda and API Gateway

Making use of our API with Shortcuts and Siri

Heartbridge: A command-line tool to transfer iOS Health heart rate data to your computer

My Final Project: Heartbridge

Link to Code

mm / heartbridge

Using the iOS Shortcuts app and Python together to export iOS Health data to a JSON/CSV file.

Heartbridge: iOS Health Data Export

How I built it/how it works

Using `kv-asset-handler` to render our application