Forem: Steve Ziegler

🚌 How big is the bus factor for Forem's most complicated files?

Steve Ziegler — Fri, 18 Jun 2021 19:33:01 +0000

Ever since watching Adam Tornhill's excellent talk on prioritizing technical debt as if time and money matters, I've wanted a simple way to analyze my own repos. I wanted to replicate Adam's process of identifying code hotspots based on each file's code complexity, change frequency, and recency of the last change. I also wanted to see if I could recreate his "bus factor" analysis to see which files would be most adversely affected if a primary developer was hit by a bus, won the lottery, or just plain quit.

I was able to create a simplified Python version that generated the hotspot data along with other basic git log metrics. I expected some high bus factor files knowing how many initial file commits Ben made to Forem. Surely there were plenty of complicated files that only he touched!

The hotspot list shows some intuitive hotspots for a Rails app like schema.rb and routes.rb which are used to manage the database and url routing. It also shows how user.rb and article.rb are hotspots for Forem which make sense given its focus on content publishing.

🔥 Hotspots 
Commits  Comp.   Age    Score  File
  388    63589     1    100.0  db/schema.rb
  427    29318     1     50.7  config/routes.rb
  157    65260     2     41.5  app/javascript/chat/chat.jsx
  359    24357     1     35.4  app/models/user.rb
   98    72604    20     28.8  app/views/admin/configs/show.html.erb
  290    24093     6     28.3  app/models/article.rb
  169    34455     3     23.6  spec/models/user_spec.rb
  154    37107     7     23.2  spec/models/article_spec.rb
   69    48449    59     13.5  spec/requests/api/v0/articles_spec.rb
  177    17235     2     12.4  app/controllers/stories_controller.rb

So how susceptible are these hotspots to losing a team member? To analyze the bus factor, you can see how many different authors touched the file in the last 365 days.

Luckily, most hotspots have had at least 10 unique authors touch the file. This might be another big benefit of taking Forem open: many people have touched the code and that collective body of knowledge now has 20+ unique authors committing per month.

The good news is that the bus factor with Forem is low and the project should be able to weather changes in contributors.

If you want to learn more, you can see all of the analytics on the repo and view the source code on GitHub.

Happy Coding!

TIL that git can checkout commits, not just branches

Steve Ziegler — Sun, 11 Oct 2020 17:13:33 +0000

You can pass a commit hash to checkout to quickly see a previous version of code. I like to commit often as I get new chunks of partial code working. If I advance to the point of breaking everything, it's an easy way to see what used to work.

git checkout <commit hash>

Creating my own mixed news website free from "echo chamber" content algorithms 📰

Steve Ziegler — Sat, 12 Sep 2020 14:03:58 +0000

I've been frustrated with my various news feeds for showing the same type of content, with sensational headlines, re-enforcing my echo chamber. Also, I often read a headline and wonder how "the other side" would report the same story. I wanted to have a simple website with a sampling of headlines representing liberal, conservative, and some international perspectives.

Given my fear of all things frontend, I created a simple site with Hugo, a static site generator. I used Python to read a set of RSS feeds and create content markdown files based on the first few entries. The code is in GitHub and the site is at https://onlineheadlines.net

Here are a few highlights:

Reading the feeds

My original vision was to visit each news site and effectively scrape the H1 as the top headline. But, assuming that sites would be too different to do that effectively, I decided to focus on RSS feeds (which are still alive and kicking!).

I used the feedparser module to read the feeds. I iterate over an array of feeds and read them.

    for feed in FEEDS:
        count = count + 1
        NewsFeed = feedparser.parse(feed)

Like most things in Python, the data is returned in beautiful dictionary objects.

{
    "title": "The warning isn't new. Experts have long cautioned the months ahead will be challenging. Here's why.",
    "link": "http://rss.cnn.com/~r/rss/cnn_topstories/~3/b-gJezqWSTM/index.html",
    "summary": "Nearly 30 US states are reporting downward trends in...",
    "media_content": [
        {
            "url": "https://cdn.cnn.com/cnnnext/dam/assets/200911014935-us-coronavirus-friday-0906-restricted-super-169.jpg"
        }
    ],

    ...

}

Creating the content markdown files

To create the markdown files, I grabbed key fields from the RSS JSON and put them into a formatted string. Python has an object called f-strings that allow you to format multiline strings and easily insert variable values.

template = f"""---
title: "{title}"
date: {published}
{image_url_front}
target_link: {link}
type: {content_type}
categories:
    - {content_type}
---
{summary}"""

The resulting string looks like standard front matter and markdown format for website content, like here on Dev.to.

---
title: "The warning isn't new. Experts have long cautioned the months ahead will be challenging. Here's why."
date: 

target_link: http://rss.cnn.com/~r/rss/cnn_topstories/~3/b-gJezqWSTM/index.html
type: first_headline
categories:
    - first_headline
---
Nearly 30 US states are reporting downward trends in...

I just save the files into the appropriate Hugo subdirectory.

Generating the site

Hugo is a typical static site generator that pre-generates all of the HTML and folders to provide a simple, pretty-url navigation experience. Building the site generates the full website in the public/ folder.

Hugo has lots of community-created themes. I picked hugo-xmag because it looked like a newspaper.

The theme creator has an interesting perspective on how he created the theme to avoid the clutter of modern-websites and focus on the articles.

Building the site

This is where the fun turned into self-inflicted pain. There are simple options for CI and hosting, like Netlify, that would build and deploy based on changes to master. But, I wanted to try to get a static S3 site running, with a custom domain name and SSL cert. I had failed at doing this once before and wanted to see if I could get it to work.

I usually use AWS CodePipeline for builds since it's so easy to set up. But, I have a process that sends build progress emails for every CodePipeline build and I didn't want a flood of emails if I was going to build the site every hour. Since CodePipeline and CodeBuild are essentially just short lived containers to execute your build step, I created a custom container with Python, Hugo, my Hugo site, and my build script.

...

RUN apt-get install -y curl wget

# Install Hugo
RUN curl -L -o hugo.deb https://github.com/gohugoio/hugo/releases/download/v0.70.0/hugo_0.70.0_Linux-64bit.deb
RUN dpkg -i hugo.deb


# Install AWS CLI
RUN pip install awscli

# Copy directory files for Hugo site
COPY ./ ./
RUN ls -la headlines_site/

...

I also created a script to build the container locally and push it to AWS ECR as my container registry. I scheduled a Fargate task to run the container every hour. I knew Fargate was a "serverless" (read: "managed") service to run containers and thought it would be easy to set up. But, it took me a while to get the services, tasks, and container all working together.

Hosting the site

This is where most of the frustrations occurred, which surprised me since I'm just hosting a static S3 site. Here are the issues I ran into:

Using a custom domain name with an S3 website requires CloudFront (as far as I could tell)
CloudFront doesn't automatically automatically respect index.html in subfolders which prevented the Hugo pretty URLs from working (like https://onlineheadlines.net/about). Luckily, digital-sailors created standard-redirects-for-cloudfront to give the URLs the respect they deserve.
Using Route 53 and CloudFront for the domain name and SSL cert requires that you know what you're doing with DNS (A and CNAME records, with and without "www."). Unfortunately, my fear of networking is just as healthy as my fear of frontend.

Even with all of the frustrations, it was a good learning experience with some lessons learned that I can apply at work when I'm helping other teams.

Another Reason to Blog: Your Words Might Be Better SEO

Steve Ziegler — Mon, 25 May 2020 22:00:56 +0000

I was recently struggling with getting a CloudWatch alarm on a log filter metric to work. I was refactoring the creation from a CloudFormation template into just Lambda Python code. I was able to get both the filter metric (looking for errors and exceptions in logs) and alarm created.

But, when I tested the alarm, it never went into an alarm state, even when then metric filter showed active data. It was driving me crazy. I was able to manually create a new alarm, pointing to the same metric filter and see it work. This is the type of bug that developers hate: a completely illogical bug (as viewed by the bug's creator).

Here is the metric showing data:

Here is what the empty alarm looked like:

Searching online for an answer was not easy. My heart sunk when Stack Overflow didn't have the exact answer to my question with my first search. :) Searches for "alarms not working for filter metrics" weren't leading anywhere.

I slept on it and tried different searches the next day. Luckily, Marta Tatiana blogged her solution once she finally found it:
Insufficient data: CloudWatch alarm based on custom metric filter.. It turns out that you can't specify the period units (seconds, minutes, ...) when creating the alarm programmatically.

Marta demonstrated one of the best reasons for blogging: showing other people your solution to a problem using your words. Your words may end up being better SEO for some lucky developer down the road.

Bash alias for GMT

Steve Ziegler — Mon, 17 Feb 2020 14:21:19 +0000

I always struggle trying to convert GMT time to EST or EDT especially in 24 hour format. It's even harder when I'm staring at a log file trying to find the root cause of an issue.

I wrote this bash alias to output the last 24 hours of time in both local and GMT. I find I'm using it all of the time now. Since it's an alias, I can easily run it from a Terminal window or within VSCode.

Here is the script:

gmt(){

for i in {-24..-1}
do
    est=$(date -v${i}H)
    gmt=$(date -u -v${i}H)
    echo "$i    $est    $gmt"
done
est=$(date)
gmt=$(date -u)
echo "0 $est    $gmt"

}

Here is sample output:

-24     Sun Feb 16 08:49:05 EST 2020    Sun Feb 16 13:49:05 UTC 2020
-23     Sun Feb 16 09:49:05 EST 2020    Sun Feb 16 14:49:05 UTC 2020
-22     Sun Feb 16 10:49:05 EST 2020    Sun Feb 16 15:49:05 UTC 2020
-21     Sun Feb 16 11:49:05 EST 2020    Sun Feb 16 16:49:05 UTC 2020
-20     Sun Feb 16 12:49:05 EST 2020    Sun Feb 16 17:49:05 UTC 2020
-19     Sun Feb 16 13:49:05 EST 2020    Sun Feb 16 18:49:05 UTC 2020
-18     Sun Feb 16 14:49:05 EST 2020    Sun Feb 16 19:49:05 UTC 2020
-17     Sun Feb 16 15:49:05 EST 2020    Sun Feb 16 20:49:05 UTC 2020
-16     Sun Feb 16 16:49:05 EST 2020    Sun Feb 16 21:49:05 UTC 2020
-15     Sun Feb 16 17:49:05 EST 2020    Sun Feb 16 22:49:05 UTC 2020
-14     Sun Feb 16 18:49:05 EST 2020    Sun Feb 16 23:49:05 UTC 2020
-13     Sun Feb 16 19:49:05 EST 2020    Mon Feb 17 00:49:05 UTC 2020
-12     Sun Feb 16 20:49:05 EST 2020    Mon Feb 17 01:49:05 UTC 2020
-11     Sun Feb 16 21:49:05 EST 2020    Mon Feb 17 02:49:05 UTC 2020
-10     Sun Feb 16 22:49:05 EST 2020    Mon Feb 17 03:49:05 UTC 2020
-9      Sun Feb 16 23:49:05 EST 2020    Mon Feb 17 04:49:05 UTC 2020
-8      Mon Feb 17 00:49:05 EST 2020    Mon Feb 17 05:49:05 UTC 2020
-7      Mon Feb 17 01:49:05 EST 2020    Mon Feb 17 06:49:05 UTC 2020
-6      Mon Feb 17 02:49:05 EST 2020    Mon Feb 17 07:49:05 UTC 2020
-5      Mon Feb 17 03:49:05 EST 2020    Mon Feb 17 08:49:05 UTC 2020
-4      Mon Feb 17 04:49:05 EST 2020    Mon Feb 17 09:49:05 UTC 2020
-3      Mon Feb 17 05:49:05 EST 2020    Mon Feb 17 10:49:05 UTC 2020
-2      Mon Feb 17 06:49:05 EST 2020    Mon Feb 17 11:49:05 UTC 2020
-1      Mon Feb 17 07:49:05 EST 2020    Mon Feb 17 12:49:05 UTC 2020
0       Mon Feb 17 08:49:05 EST 2020    Mon Feb 17 13:49:05 UTC 2020

How DEV's job listings changed my perspective on hiring

Steve Ziegler — Sat, 06 Jul 2019 14:54:21 +0000

I recently used DEV's listings to help make a hire. We'd been struggling to get candidates for a mid-level position located near our main company locations in the DC area. I remembered that DEV created a new Listings section and decided to look there for possible candidates. The process opened my eyes to a few new perspectives:

DEV members already pass my first level of vetting
DEV is a wonderful community. In my opinion, members already have a desire to continually learn and realize the value of others in their learning/career growth. Those that contribute by blogging themselves also show good organizational and communication skills.
Junior is OK
We were originally looking for a mid-level developer. But, as a consulting company, ICF is very dynamic environment with a wide variety of projects and opportunities for junior devs to learn on the job. I checked with our PMs to make sure they'd be able to use a junior dev before moving to far. Even if her first project isn't using the latest, cutting-edge technology, a junior dev could learn a lot about software development, real-world projects and challenges, and the soft skills required to work in a team.
Starting remote is OK
ICF has always been remote friendly, but usually with experienced developers that had previously worked in one of the main offices (including myself!). I've always thought that being in a local office provided some comfort to new hires. Even though your colleagues might be ecologists, cancer researchers, or social workers, they could still help a new hire navigate the company organization and processes. While that could be helpful, I don't think it's required any more.

Overall, it was a good experience and I'm glad DEV has this new capability. I'm really like that the Contact via Connect is checked by default now to make it easier for people to connect.

Happy Coding!

Script To List AWS Services In Use

Steve Ziegler — Sun, 02 Jun 2019 13:45:06 +0000

I recenty created aws-account-summary to list out basic information on the AWS services in use by an account. It comes in handy when trying to confirm current or newly created infrastructure. I went with quick-and-dirty approach of just finding text in AWS CLI output. It's suprisingly useful for such little code.

Sample Output

Sun 06/02/2019  8:47:59.33
*
*
* API Gateway
            "name": "TestAPI",
*
*
* Cloudfront
                            "DomainName": "jekyll-site-1.s3.amazonaws.com",
                "DomainName": "d1xau5ri19gn0n.cloudfront.net",
                            "DomainName": "www.nerdthoughts.net.s3.amazonaws.com",
                "DomainName": "d20ip71gqm0ltw.cloudfront.net",
*
*
* Cloudwatch
            "logGroupName": "/aws/aes/domains/ziegler-es/application-logs",
            "logGroupName": "/aws/codebuild/DocumentLambdas",
            "logGroupName": "/aws/codebuild/NCStateFair",
            "logGroupName": "/aws/codebuild/nerdthoughts",
            "logGroupName": "/aws/lambda/aws-code-index-bulk-load",
            "logGroupName": "/aws/lambda/aws-code-index-escape-files",
            "logGroupName": "/aws/lambda/aws-code-index-escape-files-2",
            "logGroupName": "/aws/lambda/aws-code-index-format-files",
            "logGroupName": "/aws/lambda/aws-code-index-stream-bulk-load",
            "logGroupName": "/aws/lambda/aws-dynamodb-to-es-bulk",
            "logGroupName": "/aws/lambda/aws-lnkchk-cache-stream",
            "logGroupName": "/aws/lambda/aws-lnkchk-dynamodb-stream-to-es",
            "logGroupName": "/aws/lambda/aws-lnkchk-extract-links",
            "logGroupName": "/aws/lambda/aws-lnkchk-log-stream-to-es",
            "logGroupName": "/aws/lambda/aws-lnkchk-queue-stream",
            "logGroupName": "/aws/lambda/aws-lnkchk-start",
            "logGroupName": "/aws/lambda/aws-read-s3-es-events-in-chunks",
            "logGroupName": "/aws/lambda/aws-s3-to-es",
            "logGroupName": "/aws/lambda/aws-tika-app",
*
*
* Running EC2 instance
                    "KeyName": "linux",
*
*
* Elasticsearch Domains
    "DomainNames": [
            "DomainName": "my-es"
*
*
* Firehose
{
    "DeliveryStreamNames": [
        "code-index-files-csv",
        "code-index-files-es-bulk",
        "code-index-logs"
    ],
    "HasMoreDeliveryStreams": false
}
*
*
* Lambdas
            "FunctionName": "aws-lnkchk-start",
            "FunctionName": "aws-code-index-escape-files",
            "FunctionName": "BlogBuild",
            "FunctionName": "nc-state-fair-food-finder",
            "FunctionName": "aws-read-s3-es-events-in-chunks",
            "FunctionName": "aws-lnkchk-dynamodb-stream-to-es",
            "FunctionName": "aws-dynamodb-to-es-bulk",
            "FunctionName": "ephemeral",
            "FunctionName": "aws-code-index-format-files",
                    "type": "code-index"
            "FunctionName": "aws-s3-to-es",
            "FunctionName": "BingSpellcheck",
            "FunctionName": "aws-lnkchk-extract-links",
            "FunctionName": "aws-code-index-bulk-load",
            "FunctionName": "aws-code-index-stream-bulk-load",
            "FunctionName": "aws-lnkchk-log-stream-to-es",
            "FunctionName": "aws-tika-app",
                    "type": "tika"
*
*
* RDS
*
*
* S3 Buckets
            "Name": "aws-s3-to-es",
            "Name": "aws-tika-app-code",
            "Name": "code-index",
            "Name": "code-index-2",
            "Name": "code-index-3",
            "Name": "link-checker",
            "Name": "lnkchk",
            "Name": "lnkchk-100-pages",
            "Name": "lnkchk-complex-site-integration",
            "Name": "lnkchk-simple-site-integration",
*
*
* Sagemaker
*
*
* Secretsmanager
*
*
* SQS
{
    "QueueUrls": [
        "https://queue.amazonaws.com/9999/code-index",
        "https://queue.amazonaws.com/9999/code-index.fifo",
        "https://queue.amazonaws.com/9999/link-checker"
    ]
}
*
*
* Users
            "UserName": "joesmith"

You get two professional apologies per year. That's it.

Steve Ziegler — Tue, 19 Mar 2019 13:10:08 +0000

Jessica Hagy wrote a great article on Forbes about things to never apologize for at work. While the apology theme is great, there are lots of hidden career-advice gems as well. I love her simple, hard-hitting graphs!

It reminds me how I've coached some people that they are only allowed two professional apologies at work per year, so they should use them wisely.

How To Use A Meeting To Elevate a Team Member

Steve Ziegler — Sat, 22 Dec 2018 14:41:21 +0000

I've been fortunate in my career to have very strong advocates. My advocates would mentor me and provide good opportunities for work and career advancement. A few years into my first job, my advocates positioned me for a leadership role on a new, large IT modernization project. They used a kickoff meeting with the client to elevate my status into a leadership role. The client, ExxonMobil, had just completed their merger to become the largest company in the world and was going to replace dozens of legacy systems with a new global, health and safety system for all of their business units.

For the kickoff meeting, ExxonMobil was going to fly in senior leaders from around the world to hear our company's plan for the project. To my surprise, my advocates assigned me a speaking role on a data conversion tool that I had never used. I had no business telling 30-year petrochemical industry veterans how I was going to use this tool to convert their complex legacy data. I had to scramble to talk to the tool developer and learn everything I could from her before the meeting.

On the day of the kickoff meeting, I was able to get through the presentation (with no questions luckily!). After the meeting, my advocates told me that the clients were happy that I was on the project. Knowing how inexperienced I was, I realized how important perceptions, not reality, are. I also realized that my advocates did several specific things to change perceptions about me and elevate my status.

As I've been an advocate for people, I try to use those same techniques to elevate people (usually developers) around key project or company meetings. Here are some of the things I do:

Have the developer send the meeting invite - This shows some ownership over the meeting and topic, even if it is just a perception.
Make sure she has a speaking role - She should have a defined speaking role during the meeting, even if just to review the agenda. You can use subsequent meetings to increase the speaking role.
Coach him on how to confidently introduce himself - A major factor in the perception of the developer's status is his confidence. Coach the developer on how to introduce himself with strong, active language: "I'm John Doe and I will be ..." Be sure to avoid phrases like "I'm here because Steve invited me" or "I'll be helping Steve to ..."

You don't have to learn everything

Steve Ziegler — Sun, 04 Mar 2018 15:37:05 +0000

I read Abisekh's post on not knowing what to study next while trying to learn programming and data science.

Article No Longer Available

I really felt for him. He's obviously passionate about wanting to get into the field and learning software development. Many of us in the development community have the same nagging feelings that we haven't learned the right things or haven't kept up. It's the tough part about being in an industry where everything changes constantly. But, as I thought about his post, I reflected on my career and how some things have stayed the same since I was building client server apps in PowerBuilder. These building blocks helped with my career when I wan't able to learn everything.

User Interface

There will always be a user interface. Given Siri and Alexa these days, it's not always a graphical interface. But, there will be a need to get input from the user and take action on it. Spending time to understand the basics of input data types (string, number, currency, ...), input controls (text box, dropdown, checkbox, radio button, ...), and when to use them will serve you well.

Logic

Regardless of language, basic logic and flow control is essential. Every language will have mechanisms to:

Iterate over sequences or lists with for loops, while loops, and list iterators
Create and manipulate lists whether they are arrays, linked lists, hashes, or dictionaries
Branch on conditions using if, if else (spelled a hundred different ways!), and else

As many people recommended in the comments, going deep in a single language first will allow you to master these basics making it easier to apply to other languages.

Data Storage

Most apps will require data storage at some point, whether just in memory or written to disk. Understanding the basics of writing data for future retrieval is major building block for a career as a developer. I would start with a good ole relational database and learn SQL. SQL is a beautiful language. You'll be amazed with what you can do joins, group bys, and sub-selects.

Grey Boxes

Scott Hanselman has a great post saying that the internet is not a black box. There really aren't many true black boxes in the software world. When things go wrong in these grey boxes, regardless of tech stack, there are lots of tools to help:

monitor file or directory access (Process Monitor on Windows)
read logs of systems
check server performance for CPU or memory hogs
review DB processes for slow queries or locks
see web client behavior (F12 as Scott explains)
understand how a web resource responds to different HTTP verbs and headers (curl or my favorite HTTPie)
see network traffic (Wireshark)

Testing

One constant throughout my career is that people expected my code to work. How rude! Learning to never trust your own code is another valuable career skill. Hammer your inputs with nulls, empty strings, bad data types, wrong combinations, invalid file formats, and bad urls. Luckily, doing this is much easier these days with the prevalence of unit testing tools.

Soft skills

Chances are very high that you will be writing code as a part of team in a organization that contains people. When learning how to code, don't forget to learn how to deal with people. You will need to win friends and influence people. You might find Jon Sonmez's Soft Skills: The software developer's life manual book useful.

While the desire to learn everything may pull at you, focusing on the basics first will provide a great foundation for a development career.

Cover Image: Abbey Bookshop by Mandolan Greene