Forem: Marc Nevin

Testing out Serverless Stack

Marc Nevin — Thu, 18 Aug 2022 21:41:00 +0000

A serverless approach can be really powerful. Developing software with serverless architectures and services can speed up development time, make managing infrastructure pain-free and typically are cheaper, more flexible solutions. But it's not without its challenges local testing and mocking can be a pain, debugging systems is more trawling logs than anything else, and monitoring and maintenance of dozens of small runtimes can be overly complex.

To alleviate these pains, people have been building a range of platforms and tools to help, with Serverless Framework being one of the best known, they allow you to build on the serverless promise and avoid some of the challenges. Some newer platforms are being developed leaning on lessons from earlier ones and leveraging some developments from cloud providers since.

A new challenger appears...

Serverless Stack (SST), is one of the newer players that pitches itself to be the easiest way for your projects "to go from idea to IPO", selling itself as an end-to-end serverless development, deployment, and monitoring tool for building on AWS. It's pretty similar to the Serverless Framework but instead of relying on terraform, or specialised languages, it’s standing on the shoulders of CDK; allowing you to write and maintain your infrastructure in the same languages you're developing with and push them to AWS as configured stacks.

Key Features

Serverless Stack has a few key features worth digging into.

Constructs

Serverless Stack uses CDK, one of AWS’ better IaC tools, which allows you to write and persist your cloud infrastructure in familiar languages; a key feature of CDK and SST is its use of high level components called constructs. These are preconfigured infrastructure components and structures that allow you to build faster by leveraging predefined configurations, boilerplates, and pre-written glue logic. SST then builds on these constructs refining them even further, although currently it only supports a few services, into an even more concise version. Allowing for building out of several components that go into making something like an API in a few lines. They currently have a load of constructs supporting A few key patterns, you can see the full list here.

Live Lambdas

Normally, when developing serverless functions, you've got to either deploy and test your changes with live invocations, or locally run the lambda with heavy mocking, using libraries or manual stubs and time; SST proposes a novel solution to this, Live Lambdas. For testing and debugging, they've built functionality that creates a local dev environment that you can invoke from a testing stack the CLI deploys, meaning invocations to a live stack are directed to your local environment rather than a hosted lambda. This allows for local testing, debugging and live reloads, by creating a stub in debug stack that's deployed which redirects invocations of the stub to a local WebSocket and then our Lambda code. This allows us to edit, watch and debug it live, with native support for a few major editors, such as VSC, allowing you to set and observe breakpoints in your editor. There’s a diagram from their docs that helps explain the flow.

SST Dashboard

Another feature, one you see with similar tools, is a browser based console for observation and monitoring of your local and deployed stacks. It allows you to inspect various aspects of your application, seeing real-time logs in your app, shows you all resources in your stacks and allows you to manually invoke and replay invocations to your services.

It also allows you to explore different resources, such as Databases and Cognito pools, allowing you to see and query the data, users, and configuration of components in your app. It can be connected to local and live remote environments allowing you to see test and debug data, or live data from CloudWatch and other services. Making it pretty useful in developing your application, preloading any data for testing and teasing out any issues when connecting services together.

Spiking the Service

SST sounded promising, so to test it out, I spiked those key features, aiming for a quick test, I aimed to build out a serverless 3 tier application using it; building a simple React frontend, an API accepting form data, and a table storing the form data.

Since SST is built around a series of NPM packages, the only prerequisites it has is NPM and making sure you’ve the AWS CLI already configured. From there we can use SST and its templates to create our project structure, navigate into the created folder and kick off the project.


$ npx create-sst@latest --template=starters/typescript-starter react-app 

$ cd react-app 

$ npm install

This uses an SST template to generate the structure we need for a TypeScript React app in seconds, from there we can start our stack from NPX and it’ll deploy the basic structure for the first time, within 5-10 minutes.


npx sst start

From this point, we’ve deployed a basic app that’s observable from the SST console and we can build on towards our goal, it’s fairly straight forward. Rather than repeating SST’s tutorials and “getting started” content here, it makes more sense to direct you to their a great guide covering a simple counter React App using the same services I used, that you can check that out here.

Rather than abridging their guides, I’ve pulled out a sample to show just how concisely it allows you to build. The stacks/MyStack.ts file allows you to implement the new resources in one line, so at the end of that guide the top of that file has;


import {  

    StackContext, 

    Api, 

    ReactStaticSite, 

    Table  

} from "@serverless-stack/resources";

With a few lines for defining each, for example an API with a POST endpoint is defined with;


const api = new Api(stack, "API", { 

    defaults: { 

      function: { 

        // Allow the API to access the table 

        permissions: [table], 

        // Pass in the table name to the API 

        environment: { 

          tableName: table.tableName, 

        }, 

      }, 

    }, 

    routes: { 

      "POST /": "functions/lambda.handler", 

    }, 

  });

This defined most of the bare bones of the API in a few lines, an API gateway with an endpoint directed at a Lambda Handler, the environment variables setup for the table you want to use accessible for it. It's easily extended allowing you to add more routes or features to get it production ready fairly easily.

Overall, its surprisingly painless to work with, the constructs enabled rapid standing-up of services, the live debugging was useful for testing and trying different requests and the console allowed us to monitor the debug stack and check invocations locally without having to constantly redeploy the stacks. It had a surprising amount of utility in the time we used it, living up to the simplicity and speed promises of the website.

Limitations

After spiking it there's still some broader benefits and limitations worth addressing.

NPM - It's the devil

Package management was a challenge, there was a torrent of version clashes between libraries and dependencies that had no easy solution without diving into the weeds which was beyond the scope for me. I imagine with some time you could untangle the knots of dependencies, but a major issue or vulnerability logged with an old version could throw this all up again creating a maintenance overhead.

Live Lambdas ... are Live?

Live lambdas are pretty effective, but they are still live and how easy they were to deploy seems like we could easily accidentally leave some resources running and lead to a wave of security confusion or unplanned cloud spend without any awareness. Security given you might have unprotected and poorly configured endpoints live and spend from leaving resources running accidentally. There's very clearly easy remediations here but in reading the docs it wasn't something jumping out, seems like an easy way to burn yourself on the tool.

Console Sync

The console and the CLI tool are pretty great, fairly easy to use and straightforward. Unless you manage to have a change in the stack that isn't reflected in the SST Console; when they end up out of sync, fixing them wasn't as simple as turning them off and on, it lead to a lot of manual stack task killing, and console reviews until finally killing off enough that the console accepted something had went wrong and it was able to successfully tear down its own provisioned stack. Recommendation here would be making sure whatever IAM role you setup to use with SST is configured that you can manage its resources and add separate monitoring.

Benefits

Frustratingly, a lot of the places SST struggled, where also the places with the most utility and hope.

NPM - Package management for the whole team

As much as dependency issues are a nightmare, when working in a team, a lot of people have had some interactions with NPM, so they're often somewhat familiar with how it works, how to use it and working with it in a team. It feels "lightweight" and familiar to a lot of people and familiarity would make adopting something like SST into your team a lot more straightforward.

Local Debugging and Testing

Working in VSC and setting breakpoints without having to worry about stubs and mocks was a dream, it sped up the process of chasing down any issues in the course of this spike and it seems like scaling this out to a full-scale project it'd pay dividends for larger features.

Detailed Dashboards

A lot of similar tools have UIs and consoles like this, and much like those, it has a fair amount of utility, being able to inspect and review different aspects of requests to the API, replay them and trace locally would be invaluable. Being able to inspect and review Cognito setups or data in a DB locally would save time trawling through the AWS console, even to those well acquainted. It was a big-time saver and generally just improved the Developer Experience (DevEx).

Summary

SST is an interesting tool; it feels like it’s still maturing but it's got a lot of potential and if you can afford / make an early adoption of it on a project I think you'll be able to see fairly big payoff for DevEx and time saving down the line at the cost of some granular control and power. Overall, I think a small team who want to move fast and build an early version of an AWS backed application could do some great things with it early.

Personally, I’m going to try it out on something larger scoped next and see about working with it more in a collaborative setting before looking at a larger adoption.

Cover Image Credit - Photo by Ryan Stone on Unsplash

Image Credit - ServerlessStack

A Guide to Giving Better Feedback

Marc Nevin — Wed, 21 Oct 2020 09:48:29 +0000

Feedback is essential for growth, and by encouraging the right behaviours, you'll see your team develop their skills and an increase in their overall performance. Feedback doesn't need to wait for an annual review; frequent focused feedback and communication helps you build a better relationship with your team based on mutual understanding and candour.

Taking the time to give and get regular feedback can be a building block towards psychological safety within your team that you'll see real benefits from. But giving good feedback is a skill, and like every skill, it gets better with practice; this post is written to help you focus, structure and deliver your feedback in a better way, today.

First, it's important to think about why you want to give someone feedback - what's your motivation for it? The goal should be to improve their performance and help develop your team, not to vent some grievances with the individual. Giving good feedback can feel easy, and bad feedback can be hard to handle objectively, focusing on what makes it constructive and an asset to the individual is where we start.

What makes good feedback?

There are a few aspects you should consider when planning feedback you want to give someone;

Specific: Feedback should contain specific, factual examples rather than a generalisation, examples help them tie it back to their experience.
Usable: Try to tie the feedback to clear actions and goals, don't make it esoteric, enable the individual to improve their performance.
Clear: Clarify understanding, make sure they are getting the most out of the feedback.
Punctual: Feedback should be given as soon as possible after the situation, giving it closer to the event helps reinforce good behaviour or action bad.
Desired: Feedback can still be effective even in those who don’t actively seek it, but it can also affect trust and relationships if demonstrated inappropriately. This extends to where you deliver it, pick a safe environment, don't air dirty laundry in public.
Objective: Feedback should be unbiased and unprejudiced; don't let your personal relationship with the person get in the way, especially if it's formal feedback that'll be stuck with them.

Structuring with Feedback Models

There are a bunch of models you can use to better structure feedback and help focus on concise points, rather than dancing around it. We'll cover the three most popular here;

"The Feedback Sandwich" (aka "s*** sandwich")

Give positive feedback;
Constructive or negative feedback;
End with targeted feedback to build trust and comfort

This is a fairly weak model, it's common and easy to recognise; often feels like a cheap move as its often used really to convey negatives wrapped to some placid ones. People see through it. It develops into a culture built on saving face and evasion, not of candour and respect. Avoid it, we can do better.

Situation, Behaviour, Impact

Identify the situation the feedback refers to;
Define the specific behaviours you want to address, for good or for bad;
End by describing how their behaviours impacted you or others.

A much better model, and one I find myself using the most; it's a simple structure that lends itself to a more concise piece, often of written feedback, that feels real and direct. The idea of focusing on specific situations and impacts helps keep emotion out of the feedback and allows for a truer reflection on their actions.

Pendleton’s Model of Feedback

Ask them to highlight positive behaviours, reinforce what went well;
Then ask to highlight areas for improvement, reinforce what could be improved;
Discussion of an improvement plan going forwards;
Agreeing on the plan and summarising key points.

A structure that only works when feedback is sought out, it allows the individual receiving the feedback to be more open and start the conversation; It helps them feel like they have a voice. It's good for building trust and mutual understanding and works well in a longer-term ongoing system of feedback.

Next Steps

Giving feedback can feel daunting, but with an understanding of the components of good feedback and some structure, it should be easy to start compiling some feedback for your colleagues when they come looking for it!

Think about who would appreciate some feedback from you today, who can you start preparing feedback for and who will really appreciate it?

This post was compiled from my notes on the topic over the last few months, if you have a good method or any tips I've missed, please share it below or reach out with any questions you might have!

Cover photo by Charles Deluvio on Unsplash

Wisdom of the Crowds, Using Twitter API v2

Marc Nevin — Thu, 24 Sep 2020 21:33:35 +0000

The Wisdom of the Crowds is a concept that states that averaging a large group's opinions, or guesses, smoothes out their bias and noise to give a measured opinion or the right answer. Using this concept and the new Twitter API - can we start winning some competitions?

The Idea

It's a pretty interesting concept and reading about it again recently in Thinking, Fast and Slow got me thinking about a short project to try out the new Twitter API. The author points specifically at challenges to guess the number of 'things' in a jar, individuals are often wrong but in theory, averaging the values should give you the winning answer!

So, there's the odd competition like this on Twitter, let's see if we can weaponise the new Twitter API to win some prizes?

Twitter API

The new version of Twitter's API is in early access with a limited number of endpoints but it already looks interesting; it can't be worse than the restricted original right?

To test out this idea, we'll need a developer account to get started, you can apply for access here. Once approved there are a few steps for setting up a new project that are covered well in their new guides and far quicker than the original. I already had an account and more or less right away had a project set-up with keys generated. No more long waiting just to test out some ideas in a throwaway project.

Before building something out, we'll need to see if we can access the tweets for the competitions and prototype the calls we'll need to get the guesses. The docs link off to a prebuilt collection for Postman that lets us get started right away! (+1 for API V2)

Postman Prototyping

We'll pick this competition as our target since it's already over and we have the answer;
// Detect dark theme var iframe = document.getElementById('tweet-1293279536294105088-459'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1293279536294105088&theme=dark" }

Then rushing in without reading the docs, we can use the Tweet's ID (from the URL) and the Tweet Lookup Request we can get the conversationID, which all replies and retweets will include, so we can use to filter tweets for it. We need to add in our Bearer Token before we can get the response, we can generate it through the Auth page for the project on the Twitter Developer Portal. When that's in we can send the request and get something like this;

Great - now we have the conversation ID, we can use the Filtered Stream endpoint to get the guesses!

Adding a rule to the stream to filter for the conversation ID, starting to stream in... nothing? Wait, what?

Back to the docs... ohhhh, ohhhh, oh.... oh.

So that's not how it works.

So, streaming - very cool but;
1) doesn't work on Postman,
2) It's a stream... it's new, not historic...

The docs quite clearly state that it's for monitoring an event in real-time, streaming tweets in when the competition starts we'd be able to calculate the average as it changes, but for proving out our idea it's not much use. I'll be revisiting the streaming endpoint soon though - looks interesting.

Recent Search

The correct endpoint is Recent Search, here we can search for tweets that meet our filters up to one week old.
Why one week? Good question - seems like it's because the API is in early access, historic tweets are on their platform roadmap but not something we can use today.

Note: Due to the time restriction, we can't recreate the response for running this on the competition above. It was well over a week when I initially drafted this. Picking some trending topic today will show it working. Luckily we have the data from the initial run after the competition ended.

Running Recent Search with some parameters;

query=conversationID:<id> - the competition tweet to filter for,
max_results=100 - the maximum tweets per response,
tweet.fields=text - just giving us the text of the tweets

We get a response with a data section that has the tweets with that conversation ID and a next_token parameter in the meta section. We can use this as a parameter in the next request, to get the next 'page' of tweets. As long as there's a next_token parameter in the response, we know there are more pages to parse!

But we can't do this paging or the logic we'll need to find our answer in Postman; it does allow us to convert the request to a bunch of popular tools and languages though. So, we can output a Python 3 (and requests as we don't hate ourselves) snippet and copy directly to a separate script.

import requests

url = "https://api.twitter.com/2/tweets/search/recent?max_results=100&tweet.fields=text&query=conversation_id:<id>"

payload = {}
headers = {
  'Authorization': '<Bearer Token>'
}

response = requests.request("GET", url, headers=headers, data = payload)

print(response.text.encode('utf8'))

Paging and Parsing

Using the sample as a basis we need to add a little logic to handle the paging and parsing the tweets for the guesses. With a recursive call to handle the paging and a little regex to filter the numbers out we end left with something like this;

url = ENDPOINT + str(conv_id) + POSTFIX

    if next_token:
        url = ENDPOINT + str(conv_id) + POSTFIX + TOKEN_PREFIX + next_token

    response = requests.request("GET", url, headers=headers, data = payload)

    json = response.json()

    for tweetObj in json['data']:
        tweet = tweetObj["text"]
        # Regex - maybe not the most pythonic
        guess = re.findall(r'\b\d+\b', tweet)

        if guess:
            guesses += guess

    if 'next_token' in json['meta']:
        token = json['meta']['next_token']
        build_data(token)
    else:
        token = []

The Wisdom

This leaves us with the list of 206 guesses, which some python builtins make short work of and give us an average of 820 beans in the jar.

Which is exactly 100 out, the correct answer was 720 beans.

Giving the theory the benefit of the doubt and a little visualisation later, I discovered there's a lot of outlier guesses. Some people entered things like '1' or values far too big to be a real guess; well that or they think the jar is massive.

We can use SciPy's trimmed mean to take out around 10% of the values out from the ends gives us an answer of 761 beans - A much closer guess!

With this relatively small sample of 206 guesses, I thought it was pretty impressive; unfortunately for us, someone guessed exactly right. But this theory was still considerably closer than my initial guess of 460. It'll be interesting to try it with a larger sample on the next competition I see!

Overall, a quick intro to the Twitter API makes it look really interesting and I'll have to revisit it in future. You can read about more of my Python-related posts here!

Cover photo by Morning Brew on Unsplash - They actually write a good newsletter - sign up here

Facial Analysis with Python and Amazon Rekognition

Marc Nevin — Tue, 19 May 2020 14:31:22 +0000

Cameras are getting smarter, they're changing how we manufacture goods, how we drive our cars and even, for good or bad, how surveillance is being carried out.

And now it's getting even easier than ever to add this functionality to your solutions; it's becoming more democratised, allowing anyone to add features like facial analysis to their projects. One of the big players driving this is AWS, their goal is to put machine learning into the hands of every developer. For vision, they're doing it with Amazon Rekognition.

Amazon Rekognition

Rekognition makes it easy to add image and video analysis to your applications using pre-trained models that require no machine learning know-how. It unlocks the ability to identify objects, people, text, and actions in images and videos, as well as detecting inappropriate content.

One of the most impressive features is the highly accurate facial detection; you can use it to detect, analyze, and compare faces for a whole range of verification, behaviour and public safety use cases. This post will focus on those facial detection features and I'll cover some of the others in the future.

So, let's get started!

Pre-requisites

To try out Rekogniton you'll need to an AWS account, the team at AWS have a great guide on getting your account set-up properly with roles for developers; if you're starting fresh, you'll also need to install and configure the CLI - it'll handle storing credentials for the SDKs to use with Rekognion. For experimenting with Rekognition the full access role from the docs will work well, but for anything further, stick to granting the least possible permissions.

The examples covered in this post should fall under the AWS Free Tier but it's worth checking out the pricing page for any service before getting started.

Set-up

We'll be using Python3 so we'll need the Python AWS SDK, called boto3, to access Rekognition as well as any other AWS services,

First, you need to install boto3 using PIP;

pip install boto3

Then we can import boto into a new python file and set up a client, which acts as an interface to AWS to start using the facial detection functionality;

import boto3

# Create a Rekognition client
client=boto3.client('rekognition')

Detection

Once you've got a client set up you can call the detect_faces method with an image, the response will contain a full batch of info on any faces it detects. There's two ways to load images to Rekognition: byte arrays and S3.

S3

The way to store blobs on AWS - to use with Rekognition we just specify the bucket and file, pass it to Rekognition through our client and get our response.

response = client.detect_faces(
    Image={
         'S3Object':{
             'Bucket':bucket-name,
             'Name':photo-file-name
          }
     }
)

Using S3 allows easier integration to other services and images can be uploaded to S3 anyway you normally would.

Byte Array

We can also send images straight to Rekognition as a byte array. You can open the files as you would load any file in Python, using the built-ins;

client=boto3.client('rekognition')

with open("photo.jpg", 'rb') as image:
    response = client.detect_faces(Image={
        'Bytes': image.read()
     })

You can also try using webcams and directly firing images to Rekognition, you can access them with OpenCV, this Gist shows how you can use your Webcam with Rekognition;

Attribute Option

There's another argument you can add to the request; Attributes, you have two options ALL or Default. Default returns BoundingBox, Confidence, Pose, Quality, and Landmarks for any faces in the image, so it just detects the faces. All, takes a fraction longer and returns a fuller set of information for any faces, everything from Default and more interesting aspects like predicted emotions, age and gender.

response = client.detect_faces(
    Image={
        'S3Object':{
            'Bucket':bucket-name,
            'Name':photo-file-name
        }
    },
    Attributes=['ALL']
)

Response

After making the request, the response contains a set of "face details" for each face it detects in the image (up to around 40 faces). Here's a sample of an ALL response;

{
   "FaceDetails": [ 
      { 
         "AgeRange": { 
            "High": number,
            "Low": number
         },
         "Beard": { 
            "Confidence": number,
            "Value": boolean
         },
         "BoundingBox": { 
            "Height": number,
            "Left": number,
            "Top": number,
            "Width": number
         },
         "Confidence": number,
         "Emotions": [ 
            { 
               "Confidence": number,
               "Type": "string"
            }
         ],
         "Eyeglasses": { 
            "Confidence": number,
            "Value": boolean
         },
         "EyesOpen": { 
            "Confidence": number,
            "Value": boolean
         },
         "Gender": { 
            "Confidence": number,
            "Value": "string"
         },
         "Landmarks": [ 
            { 
               "Type": "string",
               "X": number,
               "Y": number
            }
         ],
         "MouthOpen": { 
            "Confidence": number,
            "Value": boolean
         },
         "Mustache": { 
            "Confidence": number,
            "Value": boolean
         },
         "Pose": { 
            "Pitch": number,
            "Roll": number,
            "Yaw": number
         },
         "Quality": { 
            "Brightness": number,
            "Sharpness": number
         },
         "Smile": { 
            "Confidence": number,
            "Value": boolean
         },
         "Sunglasses": { 
            "Confidence": number,
            "Value": boolean
         }
      }
   ],
   "OrientationCorrection": "string"
}

A full breakdown of the response can be found in the docs but let's take a look at some of the most interesting aspects like landmarks and emotions.

Landmarks

Landmarks are coordinates for points on the face, they're used for detecting the face itself and are points like different spots on the nose, eyes and jawline. This image from the docs shows roughly where they all are on the face:

From AWS docs

Landmarks can provide a different way of understanding detected faces, to spark some ideas of interesting ways of using landmarks; here's a rough demo to work out what a male's face shape might be;

Nevin243 / face-shaper

PoC to test if you can use CV to get someones "face-shape"

Face Details

The details returned with the ALL attributes call can be really useful, these are aspects like estimated age range, predicted gender, if they've facial hair and if the person is wearing glasses.

Rekognition returns a dict that contains a boolean for the aspect and the confidence in the analysis. Here's a snippet for parsing the response to show the dict for detecting glasses on the faces in the scene:

for faceDetail in response['FaceDetails']:
    print(faceDetail['Eyeglasses'])

# Example Output
# {'Value': False, 'Confidence': 98.8663558959961}

Detecting aspects like glasses or age allows us to do better understand demographics to tailor services during testing or verify information about who our users could be!

Emotions

One of the most interesting aspects of Rekognition is the emotional analysis it can do. Understanding your user with the other details is useful but being able to see and understand exactly how they're reacting can create open up some really interesting opportunities.

This snippet parses the emotions of the response and displays them all, confidence indicates how likely the face is displaying that emotion:

for faceDetail in response['FaceDetails']:
    print('Emotions: \t Confidence\n')
    for emotion in faceDetail['Emotions']:

    print(str(emotion['Type']) + '\t\t' + str(emotion['Confidence']))
    print('\n')

# Sample Output
# Emotions:     Confidence
# HAPPY     0.051588401198387146
# ANGRY     7.0355730056762695
# SAD       0.04484181851148605
# DISGUSTED 0.11442316323518753
# CONFUSED  0.2994600534439087
# FEAR      0.013996988534927368
# CALM      90.5924301147461
# SURPRISED 1.8476886749267578

This on can be quite fun to play around with trying to see what gets a 'Disgusted' response rather than 'Calm' but it can be really useful, being able to even get a rough sentiment read on someone interacting with your application can be really valuable.

Streaming, Comparing and Videos

Rekognition features are a lot broader than what's been covered here but when it comes to faces, you can stream pictures in with kinesis, pass batches of images or create collections of images to sample and analyse.
You can use compare facesamong these data collections, check if the faces are celebrities, or even begin to track movement across images and video.

To take any of these further you'll want to take a proper look at the docs!

Hopefully, this has shown you that when used responsibly it can be a fantastic tool and how to start experimenting with facial detection and analysis!

Want to see some really impressive Rekognition example projects to check out? There's a bunch at the official AWS samples repo.

If you want to learn more about AWS in general? A great place to start is their Certs here's a post I wrote after getting my first!

A Journey Through the Amazon

Marc Nevin ・ Mar 2 '20

#aws #beginners #career #devops

And don't forget to take a look at other posts with the AWS Tag:

#aws Follow

Amazon Web Services (AWS) is a collection of web services for computing, storage, machine learning, security, and more There are over 200+ AWS services as of 2023.

How Are You Staying Healthy During Lockdown?

Marc Nevin — Wed, 22 Apr 2020 12:07:47 +0000

How is everyone handling staying healthy and happy during your lockdown? any tips/tricks? What's working for you and what's not?

It's made me take running up again, and I'm starting to actually enjoy it! After work, if I've not been on a run, I've been going on a pretty big walk - try and make up for sitting in all day.

Been really conscious of what I'm eating too and keeping a good record of my weight too (I actually have a dataset which is kinda weird) since I'm generally being less active

How's everyone else coping?

What Makes A Great Tutorial?

Marc Nevin — Fri, 17 Apr 2020 11:17:17 +0000

Everyone learns in different ways but something you usually can't escape is tutorials for new concepts, but what makes a tutorial great for you?

My team and I spend our time jumping around a load of different stacks or pieces of tech to test concepts for the business, build out some prototypes and try and integrate some of the newer ideas into our existing solutions!

To build these ideas out, we end up reading a load of docs, academic papers and combing search results for what the best tutorials or breadcrumbs to start out - over time we've got this down to a fine art. Most of the time, we can quickly pick up a post and decide to move on or keep reading,

There are some standout points that push one a tutorial one way or the other for me are;

Clarity & communication - how easy it is to read
Length & depth - is it a simple vertical or a step by step to a specific goal
Explained code snippets as a % of the content
Links to further resources & repos

This is my personal preference, and I'm sure everyone has something different, so what makes a great tutorial for you?

Learn a New Language: Ruby or Go?

Marc Nevin — Thu, 09 Apr 2020 11:25:22 +0000

I'm lucky enough that I can take my time on lockdown to build some backlogged projects, draft some posts and develop some new skills, one thing I've been thinking about is learning some core concepts of a new language.

After looking at what I enjoy, what's popular and what's actually used, I've narrowed it down to Ruby and Go,

The case for Ruby

So many tools I love to use are built with Ruby and loads of great websites are written in Rails. Makes me think there's some real utility there I could start leveraging in my own solutions, not to mention Rails opens a new avenue for webdev!

The only hold-ups I have, it's not got a popular reputation locally, there are a few product teams that use it and talk regularly about how there's not many jobs in it and that they hope to migrate to something else...

The case for Go

Go has a certain appeal to me - most of my stack is Python-based with some JavaScript, using AWS; I used to work in a C/C++ stack and the promises of efficiency, combing my old domain with my current and "easy" integration to my current stack make it hard to not notice.

And again a lot of great tools and services are being built in it or things that were once Python (still my fav) are being ported to Go now, it seems a little less broad spectrum of services built with it?

Decisions, decisions

I'm leaning towards Go, but want some input, I know very few people who actively use either to draw from, anything I've found normally has a strong agenda, so I want to ask here, see what people think and crowdsource some opinions!

Which would you recommend starting with and why?

or if you've any strong experiences working on either, think one is easier to learn etc etc

When's the Best Time to Post on Dev.to?

Marc Nevin — Sat, 04 Apr 2020 14:53:45 +0000

Using Python to work out visualise the best times and topics on Dev.to

Having an international community on Dev.to means that there are users active all around the clock, that makes it hard to understand when the community is the most active. Knowing this helps understand your audience more and helps you get the most engagement with the content you're posting.

With some simple Python, we can work out as much as we can about the users of Dev.to and their behaviour, like when we should be posting to get it in front of as many people as we can. Not to mention, its a good chance to learn something about data prep and manipulation.

Last year, I worked on a project to understand what users were posting on my company's Yammer, think company Facebook; we were using python to track engagement on topics my team were interested in, how they were being perceived and when they were being interacted with. While looking around for some visualisation ideas, I saw Pierre's post about the best time to post here,

What is the best time to post on dev.to? a data-backed answer 🕰🦄🤷‍♂️

Pierre ・ Mar 24 '19

#python #webdev #beginners #meta

Now that I've started posting here more regularly, I thought I'd try something similar, so I decided to try a short project to understand more about the users here when they are reading, are there any trends and what topics they interact with the most.

To try to understand the reader, we need to work out what determines a successful post - normally that's read numbers, comments and reactions.

Since we can't get reads numbers for every post, we'll look at reactions as the two are strongly linked; more reacts, more reads and more reads, more reacts. So, reactions will be the metric we use to determine the users behaviour or response to a post!

Getting Data

First, we need data.

We need to know how the previous posts have performed, luckily the team here have a great API for us to use!

Using Python and the requests library, we can call the API and build up a JSON lines file of every post:

# To get the first page on articles on Dev.to
URL = "https://dev.to/api/articles"
payload = {"page":1}
r = requests.get(URL, params=payload)
r.raise_for_status()
f.write(r.text)

Splitting out the payload lets us iterate up the page value and get all posts up to a set page number, using this we can grab every post or aim for a rough date.

Running this we can build up a dataset, but it needs some cleaning up, to get it into a more parsable format for the next library we're going to use - Pandas, alongside Numpy it's the backbone of data manipulation in Python.

Using Pandas and a simple generator we can load the data into a DataFrame;

# Generator for data
def json_line_gen(file_name):
    for row in open(file_name, "r"):
        yield row

json_response = json_line_gen('./data.json')

for json in json_response:
    df = df.append(pd.read_json(json), sort=False)

Data frames are really useful, basically, it's a table with labelled rows and columns, that are highly flexible with a lot of functionality built-in. They're easily scaled and manipulated enabling a whole world of data manipulation with one library.

Prepping our Data

Getting the data frame set-up, we can do some initial exploration and see we need to do the same thing Pierre did; split the date and time into their own columns to make it easier to manipulate. While we're here we can also drop the columns we're not interested in, like the cover image or canonical URL;

df = df.drop(columns=unwanted_columns)

# Splitting Timestamp into hour and day of week
df['hour'] = pd.to_datetime(df['published_at']).dt.hour
df['day_of_week'] = pd.to_datetime(df['published_at']).dt.strftime('%A')

When's the Best Time to Post?

One of the first things we'll want to know about users is when they read posts and we can see this from when they are reacting to the posts. Part of the reason we broke it down into hours and not minutes is so we can make a generalisation, trying to estimate the time down to the minutes is too granular and won't really aid us any more than knowing the hour will.

The best way to visualise this then, will be as a heatmap - we start by grouping the data we need, reaction count and timing, before pivoting the table so that its columns will be the days of the week:

# Get the average reactions per post at a given timeslot
reaction_df = df.groupby(["day_of_week", "hour"]) ["positive_reactions_count"].mean()

# Pivot the dataframe & reorganise the columns
reaction_df = reaction_df.reset_index().pivot('hour', 'day_of_week', 'positive_reactions_count')
reaction_df = reaction_df[["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]]

Then we use Seaborn, a data visualisation library, to generate a heatmap:

plt.figure(figsize=(16, 16))
sns.heatmap(reaction_df , cmap="coolwarm")

And we hit a problem, there's no clear trend, sometimes that's just the case but this one hour on Sunday seems like an outlier. That time is a lot darker than the others - let's check, we can use a boxplot as a simple way of checking, again from Seaborn:

sns.boxplot(reaction_df)

Definitely an outlier, since we're making a generalisation, let's filter out the outliers and see how it affects our visualisation. Using z score we can remove some of the outlier posts from our original data frame;

z = np.abs(stats.zscore(reaction_df))
reaction_df = reaction_df[(z < 3).all(axis=1)]

This filtering removes some of the top-performing posts of all time, but enables a far more general view of how most posts perform when we generate our heatmap again!

So that leaves us with a clear band; around midday UTC during the week!

This is similar to Pierre's findings on time, but on a tighter band, could be down to the site's presence growing and having an even broader user base or just having another year worth of data!

Writing for a Specific Tag

The heatmap was a very general, what if you only are interested in producing content around some specific tags, let's try out one of my favourites - Discuss;

# Tag to find map for
tag = 'discuss'

tag_df = tag_df.loc[tag_df['tags'].str.contains(tag, case=False, na=False)]

tag_df = tag_df.groupby(["day_of_week", "hour"]) ["positive_reactions_count"].mean()
tag_df = tag_df.reset_index().pivot('hour', 'day_of_week', 'positive_reactions_count')
tag_df = tag_df[["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]]

Less clear, the problem is we're cutting out so much of our dataset, that we'll just start missing data that will begin skewing results. That said some of the most popular tags might have enough data to look into.

Engagement and Comments

The discuss tag made me try something else, can we check for the link between how many people read and react to a post and how many people comment on it? This would be especially relevant for tags like discuss but generally if you want readers to interact with your post beyond just a react.

We can use a regression plot to compare the reactions with comments and see if there's a correlation:

sns.regplot(comment_df["comments_count"], comment_df["positive_reactions_count"])

That's showing a moderate correlation, now that doesn't imply causation, but it's worth investing and trying out to investigate the link in the future.

If we replace the reaction count in our previous heatmaps, with the comment count column and generate a new heatmap:

comment_df = comment_df.groupby(["day_of_week", "hour"]) ["comments_count"].mean()
comment_df = comment_df.reset_index().pivot('hour', 'day_of_week', 'comments_count')
comment_df = comment_df[["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]]

plt.figure(figsize=(16, 16))
sns.heatmap(comment_df , cmap="coolwarm")

We can see there are two big clusters we can test in the future:

Very early mornings UTC on Saturday and Sunday
And early evening 5/6 PM UTC Tuesday - Sunday

Using these trends could lead to a more engagement on your posts through comments and discussion, worth a shot to see if there is causation or if you want to engage more with your users.

What topics should we write about?

Let's jump back to tags then, what if someone wants to go full SEO? What're the most popular topics and does this tell us anything about the community?

To start we need to split the tag list column out into individual items, in Pandas, this used to be a massive pain but now its just a method call:

popular_df = popular_df.explode('tag_list')

# Fill in any gaps
popular_df["tag_list"] = popular_df["tag_list"].fillna('None')

From there, we could sum by occurrences, but turns out this has already been done for us! Some big ones you'd expect, Javascript etc, so definitely what users like to post about, but is this what the community wants? are these the most interacting with?

If we first get the average reactions for comparison:

popular_df['positive_reactions_count'].mean()

We can then use our dataset, with outliers removed, to generate a list of the top tags with the highest average reaction count:

popular_sum_df = popular_df.groupby(['tag_list'])["positive_reactions_count"].mean()

# Get top 50 average posts
popular_sum_df.sort_values(ascending=False).head(50)

After giving some of the tags a check most are still fairly skewed by a couple highly reacted posts, compared to the average but some of the top tags aren't one-offs. They are tags that consistently performing well with good interaction, some at a quick look are;

Career
Beginners
Hooks
SQL
Learned

(The full list of tags is available in the repo, take a look at them there)

Some I expected (Careers), some I didn't (SQL) but it allows us to look at what content our users are really interested in and what's not. This means will can filter content that would work best on this site, playing off trends or topics that people care about here.

Summary

Understanding your audience through data is only part of the puzzle, just posting at these times won't just instantly increase your read count. You still need to focus on understanding the end reader and producing high-quality content first!

There's scope to take this further; what length or type of content performs best, is there any sentiment or structural aspects of a post that engages better and what topics are trending at a given time.

To do any of this, we'd need to build a richer dataset, the API looks like it could be manipulated to do that but that's content for another post.

If you want to try and start building something bigger, do this yourself or look at how quickly something like this can be done in python, the Jupyter notebook and scripts are available here;

Nevin243 / data-driven-posting

For working out the best times to post on Dev.to

Now go spend some time trying to understand your reader, happy posting!

Creating a Shared Vision

Marc Nevin — Thu, 26 Mar 2020 15:58:51 +0000

It can be tough to keep teams motivated.

One thing we know works is making sure everyone understands and appreciates the impact of the teams work, that their working on something that's not just interesting but really valuable for the company as a whole.

We're noticing now that we're remote, some of the newer rotations into the team are struggling to see the value of their work and we're trying to be a bit proactive here; talking about the team's previous work and impact.

But we're thinking longer-term how can we help create and foster a shared vision for the teams work going forwards?

Currently, we're thinking;

Making the team's guiding strategy more open and accessible
Having the team's short and long term goals more visible
Getting the whole team more involved in helping review the current mission statement

We know there's always room to improve so, I'm interested in what else we can do to help the team really understand their value? How we can help encourage a shared vision? or what else we can do to make sure everyone's on the same page!

If you've got any ideas or just want to share your thoughts please share below!

How Do You Support Remote Juniors?

Marc Nevin — Mon, 16 Mar 2020 14:45:35 +0000

With the recent news, a few of us are trying to encourage more remote working across the team and company, we've broad support and every day we're seeing more people moving remote!

A few of us have been reading up on some good practices, reading some people's experiences and a pretty good books on it!

But something we're trying to understand is how we best support Juniors. Our team is currently pretty junior; from apprentices to graduates, this can mean they need a fair degree of support, from advice and direction to whiteboarding out problems with them.

We want to make sure they continue to develop when we're not as easily accessible for just grabbing for 5 minutes while walking past.

What can we do to better support Juniors? Are there tools and techniques you have found that work well? What's your experience working with junior staff remotely?

Photo by Bongkarn Thanyakij from Pexels

What's your Conference Plan?

Marc Nevin — Mon, 09 Mar 2020 11:26:01 +0000

Recently, my job has had me going to a lot of conferences and events, current news aside, conferences are going to eventually start back up,

During this conference lull, I've been thinking about how to get the most out of them when they're back and some people have been giving me advice on "How to conference"

Some of its been pretty solid advice;

Don't try to plan more than two things you want to do a day
Don't feel you need to "get your money's worth"
Try and attend one thing a day outside normal events you'd attend
Conferences are the best time to catch local people you've been struggling to find time with

But now, it's time to crowdsource some wisdom!

How do you plan for conferences? Do you have any tips? How do you get the most out of the opportunities and networking?

Cover Photo by Jakob Dalbjörn on Unsplash

A Journey Through the Amazon

Marc Nevin — Mon, 02 Mar 2020 14:06:02 +0000

A guide to understanding and getting started with AWS Certifications

AWS and the Cloud can be a tough subject when you’re starting out.

The variety and different types of certifications can all seem a bit much, but cutting through it all with some training you can come out with a better developer with a broader skillset. Not to mention some in-depth knowledge about one of the widest used cloud platforms, here’s how to get started.

Why Get Certified?

It’s a choice to make and not always a cheap one, but if you can afford it or your company are willing to sponsor you, it's a great way to highlight your technical skills and can help show off what you can do, plus AWS themselves list a load of benefits including communities and stores!

But if you've not worked in the space much the education working towards a cert can really develop your knowledge and skillset. The journey to getting exam ready will teach you a lot about the underlying concepts, the services and standards when building solutions on AWS.

Personally, my first cert was to address a skill gap in my team, we were building on top of AWS but no one had taken the time to learn the best practice and its starting to really help the team having more people understand the base concepts. I didn't know how little I knew, it was the difference between thinking I knew how to build things on AWS and knowing how to build.

What Cert Is Best For Me?

From AWS

It really comes down to what you want to learn and what you find interesting, the certs are broken down into three levels and a set of specialities;

Cloud Practitioner for those just starting out, teaching basic concepts of AWS and cloud best practice
Associate Level (SysOps, Architecture and Developer) which are fairly broad and focus mostly on 'core' AWS services
Professional Level (DevOps and Architecture) built on the Associate skills area but far more in-depth focus into the specifics of AWS
Speciality (5 and growing!) in-depth focus in a set of services focusing on one area

If you’re not sure but are keen to learn, you can check out the AWS guide to help scope out your learning path but I’d recommend starting with one of the general certs in the first two tiers.

I got my first cert recently and decided Solution Architect best met my interests and needs. Going out and reading the descriptions of what each cert covers will roughly help you scope out which will line up directly with what you want. Looking at the learning paths I think I might look at going towards the ML Speciality.

It's worth mentioning, the certs are constantly being developed and it's worth keeping an eye on what's coming up before committing to a learning path!

How Do I Get Started?

Unless you’re already an expert and are just out for the recognition, you’ll need to do some training to get exam ready.

AWS offers some solid advice in preparing for their exams, it breaks down into, exam outlines, sample questions and links to relevant whitepapers and FAQs for each different certification. The prep guides will also cover any of the online AWS training they recommend.

But sometimes it’s best to leave this to someone who knows what they’re talking about, there's a bunch of training providers who will cover everything you need, covering theory, resources and labs.

There’s a variety of providers, here's some crowd favourites to start looking at:

ACloudGuru (Actually used them myself in the end)
Linux Academy (They were recently acquired by ACG and will be merging eventually)
Cloud Academy
WizzLabs

They all have benefits and drawbacks, check out some reviews and decide which best lines up with your learning style. I’d recommend picking a practical heavy course and one that goes beyond just the exam, teaching you the topics with optional courses for topics you're interested in, this makes it go so much further even if it seems tough.

The best advice I can give is to practice what you’re learning, AWS has free tier accounts available that are great for testing out the services and working through labs that come with a lot of the training providers.

After coving of some training and some practice there's a few more things for getting ready - it's worth taking some time and reading;

After all this, you should be ready to sit an exam and come out the other side with a certification!

What Did I Get From It?

Besides a sheet of paper that says I can AWS, the training has changed how I build out services and gives me loads more confidence when I open up the AWS console.

Not to mention, it's got me reading more and more about AWS and it's changing how the future of Development and where my career might go!

——

Good luck if you decide to start your journey,

If you’ve any strong feelings pro or anti-cert, I’d love to hear or let me know what you’ve got from certs over the years;

If you want to read more about AWS check out the tag here;

#aws Follow

Amazon Web Services (AWS) is a collection of web services for computing, storage, machine learning, security, and more There are over 200+ AWS services as of 2023.

Original cover by David Riaño Cortés, Edits by Me