Forem: Andy McKenna

Software Engineering in the Real World -or- How to Use Agile to Mow Your Yard

Andy McKenna — Fri, 31 Jan 2020 21:20:01 +0000

Intro

One of the really fascinating things about writing software is that after you've done it for a while, you realize how the processes and mindset that you use can apply to just about everything around you. A perfect example is Bryan Watts' article on finding humanity in software. Sometimes the software process helps you understand the non-software task and other times it's just the opposite. Today I'm going to highlight how the Agile methodology fits in with cutting grass. Before we go any further, the answer is yes, I wrote the rough draft in my head during multiple summer sessions of lawn mowing.

There's a lot of high-tech stuff going on here

Architecture Decisions

Before you begin on any project you have to make a few upfront decisions that will set the tone for the entire process. How big is my yard? Do I need a riding mower or can I be just as successful with a walk-behind? It's tempting to just say you need the best and most expensive tools but you'll have wasted a lot of money and storage space if you bought a rider to mow 100 sq. ft. of grass.

Some decisions can be swapped out relatively cheap and easy while others would require a massive effort to change. If you decide later that you want to mulch the grass as you cut, most of the time you can just swap the blade out. If you were too conservative with your budget up front and ignored the self-propelled mowers then you're out of luck and have to replace the whole thing.

This same discussion happens at the start of any new software project. What platform and data storage should we use? Do we need a whole suite of cloud products or can we get by with lower level hardware? It depends on how much grass you're planning to cut. The analogy of our mulching blades applies here, as well: if we start with SQL Server Express we can eventually graduate to Azure SQL databases that we can keep scaling up. On the other hand if we learn that we should have gone with a NoSQL datastore, that's a larger effort.

Scoping/Refinement

You've purchased and assembled the mower and you're rarin' to go. Don't just drive out into the grass and start hacking at it. You need to plan out your overall sense of attack and set your scope. Are you also using the string trimmer and edging in this session? Are you only committing to mowing the front yard first and saving the back yard as a stretch goal? Setting your scope ahead of time helps you avoid making a mistake by over committing and cutting a straight line from the front yard to the back, requiring you to finish the entire yard.

When you're ready to start writing code in a new project it's tempting to write a grand framework that handles everything you can think of but that creates a huge upfront investment with nothing shippable for multiple iterations. Agile is focused on being able to ship a working version of the project at the end of each iteration and being able to pivot your priorities if they change over time. If you only mow the front yard first, you could put the backyard on hold and deal with some other hot issue that springs up.

Iterations/Retro

When I'm cutting the grass, I like to think of each line as a hyperspeed version of an agile iteration. When you start the line, you look across to where the line will finish and aim the mower before you start moving. This is sort of like sprint planning: you've started adjacent from the previous line, you've made a very short term goal, you've decided how you're going to get there, and you walk the line with your eyes only on the end of that iteration.

Let's stretch the analogy further and talk about how you committed to only cutting the grass in that line and you're not going to veer off in some random direction. You might spot other things that need to be done later in the task but you're just mentally filing those away in your backlog. Everyone loves the happy path but there is also the possibility of your iteration getting blown up when you discover that your kids left a toy in the yard.

When you get to the end of your line and turn around, it's a great time to have a little retrospective on how that previous line went. Was it straight? Did you miss a spot? If you followed the line before it and you still were curved, you might be a victim of grass debt and entitled to compensation.

Grass Debt

Grass Debt isn't a real thing but I think of it as the analog of Tech Debt. If you introduce a curve into your lines and all subsequent lines follow each other, you'll soon have a yard full of mismatched curves. This is like introducing tech debt into your code that you just work around but never correct the core problem. In both scenarios, it's generally preferred to pay back debt as soon as you can. If your next line is perfectly straight but you have to drift back into the previous one, you've corrected the problem for all future iterations even if you didn't cut as much grass as possible. On the other hand, some debt is acceptable. If you can tell it's going to rain soon or your software has a hard deadline approaching, it's better to finish the job and accept that not all of the lines are straight. I'm a fan of the .NET Rocks podcast and one of their tenants is that "shipping is a feature".

Conclusion

As you might have guessed, mowing the lawn is generally a boring task if you forget to charge your headphones. If you ever find yourself in a similar situation, think of how you would model that thing with software and then write an article about it. It seems like a silly exercise but will only improve your skills at modeling real processes in the workplace.

The Courteous Consumer

Andy McKenna — Fri, 30 Aug 2019 16:25:23 +0000

As a developer on the DealerOn Reporting Team, I've gone through a lot of the pain points with importing data from other services. We automatically download data daily for thousands of accounts from some of the largest providers. The classic mistake that someone makes when designing a system like this is to build out a Task for each report/account and Task.WhenAll. Eventually you realize that too many requests could flood the provider so you add in a semaphore and set it to something like 20. Done and done, right?

This is dangerous because for the most part it will work. You'll get a lot of data and think you're safe until you need to backfill a single account for more than a few days. You'll watch the logs and after a few days worth you'll notice that your requests time out or have errors that don't make sense. What's wrong?

Most of the large providers implement rate protections on their APIs that will warn a consumer when they're getting close to a limit and outright block them when they cross it. If you don't know how to recognize and heed these warnings, you'll just end up cursing the provider's name in vain. In this article, we'll examine how Facebook, Microsoft, and Google tell you when it's time to slow down so that you can be a courteous consumer.

Facebook

Facebook's Graph API has many different throttling limits based on what you're accessing. Depending on when you're reading this, Facebook is also phasing in a change to their rate limiting called Business Use Case Usage. The basic premise of the old version and the new version is that every response to a call you make will have a header that tells you the current status of the account you are accessing.

x-ad-account-usage: {
  'acc_id_util_pct':9.67     //Percentage of calls made for this ad account.
}

When that percentage hits 100, further calls will result in an error. We decided to pause at 90% because we could have multiple requests running at once. This older version will reset to 0% after 5 minutes. The new response header:

x-business-use-case-usage: {
  '{business-id}':  [{
     'call_count': 100,       //Percentage of calls made for this business ad account.
     'total_cputime': 16,     //Percentage of the total cpu time has been used.
     'total_time': 45,       //Percentage of the total time has been used.
     'type': 'ads_insights',                      //Type of rate limit logic being applied.
     'estimated_time_to_regain_access': 10       //Time in minutes to resume calls.
  }]
}

If any of the first 3 numbers hit 100%, the account is restricted. Facebook API v3.3 will use either response depending on what endpoint you are accessing. Version 4.0 will just be the Business Use Case Usage style.

We check every response and use a Dictionary of SemaphoreSlims so that we can lock individual accounts until they are ready to resume. Any other calls that we build for that account will check _accountLocks and find it already in use. The method below only handles the old style and will need to be updated for the new version (including incorporating the nice estimated_time_to_regain_access field).

public static async Task CheckResponse(this HttpResponseMessage httpResponse, 
           BaseFacebookResponse dataResponse, string url, string accountId)
{
  var accountLimit = JsonConvert.DeserializeObject<FacebookAccountLimit> 
                     (httpResponse.GetHeaderValue("x-ad-account-usage") ?? "");
  var appLimit = JsonConvert.DeserializeObject<FacebookAccountLimit> 
                 (httpResponse.GetHeaderValue("x-app-usage") ?? "");

  if (appLimit != null && appLimit.Usage > MAX_USAGE_PERCENT)
  {
    await _appLock.WaitAsync();
    await Task.Delay(APP_SLEEP_MINUTES);
    _appLock.Release();
  }

  if (accountLimit != null 
   && accountLimit.Usage > WARNING_USAGE_PERCENT 
   && accountLimit.Usage <= MAX_USAGE_PERCENT)
  {
    await Task.Delay(ACCOUNT_SLOWDOWN);
  }

  if (accountLimit != null 
   && accountLimit.Usage > MAX_USAGE_PERCENT)
  {
    await _accountLocks[accountId].WaitAsync();
    await Task.Delay(ACCOUNT_SLEEP_MINUTES);
    _accountLocks[accountId].Release();
  }

  if (dataResponse.Error != null)
  {
    throw new Exception($@"Facebook API Error: {dataResponse.Error.Message}
                           URL: {url}
                           Full Error: 
                           {JsonConvert.SerializeObject(dataResponse.Error)}");
  }
}

Google Adwords

Adwords' rate limiting is based on watching for the RateExceededError in the response.

We're using the googleads-dotnet-lib nuget package which has an example in the source on how to handle RateExceededError:

try
{
  //make your request
}
catch (AdWordsApiException e)
{
  // Handle API errors.
  var innerException = e.ApiException as ApiException;
  if (innerException == null)
  {
    throw new Exception(
      "Failed to retrieve ApiError. See inner exception for more details.", e);
  }

  foreach (ApiError apiError in innerException.errors)
  {
    if (!(apiError is RateExceededError))
    {
      // Rethrow any errors other than RateExceededError.
      throw;
    }

    // Handle rate exceeded errors.
    var rateExceededError = (RateExceededError)apiError;
    //lock out other requests for this account
    await _accountLocks[accountId].WaitAsync();
    await Task.Delay(rateExceededError.retryAfterSeconds * 1000);
    _accountLocks[accountId].Release();
    //we should retry this request now
  }
}

On the other hand, the best offense is a good defense and we typically avoid the RateExceededError by using a semaphore to limit ourselves to 10 open Adwords report calls at any one time.

Google Analytics

This product is made by the same company as Adwords so the APIs must be similar, right? Once we're all done laughing, take a look at the documentation.

Each account has a hard daily limit as well as a real time Queries Per Second limit. You can detect the daily or QPS limit getting exceeded by the status code 403 or 429 but even better would be limiting your requests so that you don't hit it.

public async Task<T> MakeRateLimitedRequest<T>(Func<T> action)
{
  await _semaphore.WaitAsync();
  Task.Delay(_timeSpan).ContinueWith(_ => _semaphore.Release());
  return action();
}

If your semaphore has 10 slots and the timespan is 1 second, you shouldn't be able to make more than 10 requests per second. If you're handling multiple accounts, you'll want a dictionary of semaphores so that each can run at it's maximum allotment.

Bing Ads

Microsoft has probably the simplest method to detect and handle throttled requests:

To ensure resources for everyone, the API limits the number of requests a customer ID may make per minute. The limit is not documented and is subject to change. If you exceed the request per minute limit, the API returns HTTP status code 429. When you receive status code 429, you must wait 60 seconds before resubmitting the request.

A modified version of our Facebook method will handle this easily enough.

Conclusion

Something that might have come through in each section is that the documentation usually will explain how the provider will limit access to their servers. This is necessary to prevent a malicious user or someone that accidentally wrote an infinite loop from crashing everything. No matter what third party data you are accessing, make sure you fully test your code by running it until your rate limiters kick in so you can see them work. By following some of the suggestions here you'll soon learn to implement rate limiting before you notice a problem in production.

So you want to work from home…

Andy McKenna — Fri, 04 Jan 2019 14:01:00 +0000

Anyone who has worked in an office for a significant amount of time has had the same thought at some point, “How great would it be to work from home every day?” It can be incredibly rewarding as long as you go into it with the right expectations and avoid some classic pitfalls.

Cutting this out of your life is usually the #1 reason to work remotely

One of the first things you need to consider is if your company is prepared to handle remote workers. Check out our previous article to see what the company can do to make things easier for everyone. The other major hurdle is if your job absolutely requires you to be on-site. It would be hard to be a construction worker that worked from home, I’d imagine. The Development team at DealerOn really only requires that we have a computer and an internet connection, so let’s dive in!

This is the Best Decision I’ve Ever Made!

Let’s start by listing some of the main benefits to working remotely. The carrot comes before the stick.

The Commute — the average commute time in the United States is now 26.4 minutes each way. In larger/dense urban areas this can be as high as 60–90 minutes. A long commute can be stressful and doubly so when it’s due to traffic that can be frustrating and unpredictable. For most of the country that time is wasted by driving your own car due to lack of public transportation options. I’m sure you’d agree that you could find something better to do with an extra hour or three in your day!

Comfort — even the nicest office probably isn’t as comfortable to you as your own place. You’re in sole control of the thermostat, you can blast your music as loud as you want, and no one will make a face if you skip the shoes all day. You also have the final say in your furniture options. Do make sure you still get dressed for work though. Nobody wants to see you on camera in your robe.

Peace and Quiet — some offices can be noisy and chaotic. Background conversations, phones, and printers can all provide a din that hangs over you like a cloud, sapping your concentration. One solution is to wear headphones to drown everything else out but nothing beats the silence of your own private space.

Multitasking/Flexibility —if you’re home all day it becomes much easier to stay on top of certain chores. Laundry and dishes are two tasks that can be started before work and then finished on your lunch break. You’re already there if an important package needs a signature or is at risk of being stolen. If you have pets, they’ll love just having you around and the more frequent bathroom breaks. You can also be there if you have a child staying home sick and still be productive, depending on their age.

What Could Go Wrong?

Telecommuting sounds amazing, why aren’t we all doing this right now?!?! Besides resistance by management, there are a few reasons why it doesn’t work for everyone.

Kids and dogs love to sneak in when you’re on-camera

Distractions — in the multitasking section, I made sure to only mention before/after work and lunch breaks because it can also be easy to get distracted by all of the things to do in your house. I’ve got an amazing library of video games, movies, and TV shows. I also have more housework than I have desire to do said housework at night. Don’t fall into the trap of neglecting your work for home projects and entertainment just because it’s so convenient now.

Children are another giant distraction that can tank your concentration. It’s easy to think of working from home as a great way to save money on babysitting but kids under a certain age need constant supervision and mental stimulation. They don’t understand that you’re “at work” and you don’t have time to hear a full recap of the latest episode of PJ Masks. If they’re usually in school or daycare but have to stay home sick it can be less of an issue because they’ll spend most of the day in bed. There’s no hard and fast rule on what the minimum age is for kids to stay home but understand that you have work to do and be able to entertain themselves.

Communication —if your coworkers are all in the same office and you work remotely, it can be hard to have the same quality of communication as your teammates. We covered some of the bigger roadblocks in the previous article, like using Slack and video conferencing, but just having the tools isn’t enough. Since your manager/teammates can’t physically see you at any given time, it’s easier for them to forget about you or not know what you’re working on. Combat that by being almost overly-communicative. Be extra talkative in the chat rooms, direct message teammates with questions, have your camera on whenever possible for the meetings. You’re a member of the team so make sure you’re constantly accessible and in the conversation (during work hours).

Another wrinkle that can affect communication with your team is time zone differences. Maybe you’re working remotely in another part of the country or on the other side of the planet. Your two options are to adjust to the home office times or work your local time. If you’re working your own times, make sure to get things you need from your teammates during the overlap and save items you can do during your alone time for the offset hours. Nothing is more frustrating than needing someone else to answer something but knowing they won’t be available for half a day.

Loneliness — this one affects people differently but it’s worth noting because many new remote workers don’t consider it before making the change. When you’re working in an office, there’s almost always someone to talk to about work or just what’s going on in the news or pop culture. Too much of this might be why you want to telecommute in the first place. The problem is that too much isolation can have negative effects on you. If your company isn’t using chat/video conferencing, you might go an entire day without saying anything out loud. Make sure to break this up by meeting someone for lunch or doing social things after hours. It’s hard to know if this will be an issue for you until you go through it. Don’t brush this one off, there’s a reason solitary confinement is seen as a punishment in prison.

Blurred lines — when your home and office are the same place, it can be easy to never “leave” work. Having a room dedicated as The Office can help (and might be a nice tax write off**) but even if you don’t have space for that, just holding yourself to normal working hours is important. Take a break every hour or two to get out of your seat and stretch your legs, get away from your desk to eat lunch, and know when you’re going to stop for the night. Avoiding mental burnout is better for your health than the short term gains of the extra work.

Do I Still Want to Do This?

I know, it’s not an easy decision

Working from home can be a pretty sweet gig if your company is open to it and you know what you’re getting in to. All of the complications that I listed are manageable as long as you set some ground rules for yourself. If you’re still unsure, try working from home one or two days a week as a trial. Once you adjust, the hardest obstacle you’ll face is not responding with a Nelson Muntz image anytime somebody complains about their commute.

** Consult a real life CPA on how to deduct a home office from your taxes

Remote Workers Are Not That Far Away

Andy McKenna — Fri, 28 Dec 2018 14:01:00 +0000

As your company grows, you might find yourself struggling to fill your open positions in the local job market. If you already have offices in other regions, hiring remote employees that work out of those offices can be a huge win. The key to making it work is having the right culture and communication structure. I’m a developer in DealerOn’s Grand Rapids, MI office while the majority of my team is based in the Maryland home office. I’ll give an overview of what steps DealerOn has taken as a company to set the stage and soon we’ll look at what it takes to be a remote worker.

DealerOn has satellite offices in Texas and Michigan as well as individual remote workers all over the world.

Embrace the chat client

A big hurdle that remote workers struggle with is feeling like they’re out of the loop. They’re not there for any of the informal discussions that happen at people’s desks or impromptu meetings that can pop up while a problem is being solved. The old adage stills applies here — out of sight, out of mind.

Fortunately, you can preempt this by integrating a chat client like Slack into your day-to-day operations. Slack moves your discussions and collaboration to a digital audience rather than just to who is in earshot at the time. It also removes the necessity for someone to physically be there to answer a question or offer feedback. If your employees grow accustomed to having discussions in a team specific chat room then all of the team is on equal footing. Remote workers can observe and comment without the friction or interruption of a phone call.

Slack moves your discussions and collaboration to a digital audience rather than just to who is in earshot at the time.

Most people know that when they see two people talking, they should not interrupt and should either wait for the conversation to end or come back later. The problem here is this is inefficient for everyone involved. If you really need an answer you’ll be put in the tough spot of feeling terrible about interrupting or you’re stuck and need to find something else to do while you wait. With Slack, that 1 on 1 conversation could be one of the 4 or 5 I’m having simultaneously. If someone needs to know when my current ticket will be completed, I can give them a quick answer without the social stigma of someone “butting in.” This might sound intimidating but it’s actually very freeing. If someone walks up to your desk, they’re demanding your attention at that moment. If someone sends me a direct message on Slack, I can finish my thought before addressing them.

Another benefit is you’ll naturally reduce the number of meetings where you think to yourself, “this could have been replaced with an email.” A team lead can say something in their channel with an @ tag and everyone in that list will get a notification. If someone has followups to that, it’s instant and visible to everyone.

At this point you might be thinking, “We already have email, isn’t that good enough?” Email and chat are two tools on your workbench but they aren’t interchangeable. Email is more formal by having a subject line, usually a signature, and each one is weighted the same in your inbox. Chat is typically informal and I can add a 👍response with one click and nothing added to your inbox. There’s also no need to curate your chat, whereas my blood pressure rises proportionally with the number I see in Outlook.

Video Conferencing

A stock image of photogenic people

While chat can cut down on your number of meetings, you’ll never get rid of them entirely. Sometimes you need to get people in a room and discuss something face to face. This is where the difference between local and remote workers is apparent, but there are things you can do to smooth out the burrs.

#1: Use video if at all possible — Humans are visual creatures and it’s easier to understand someone if you can see the facial cues that go along with what they’re saying. It’s also easier sometimes to just nod or throw a thumbs up without breaking the speaker’s momentum.

#2: Rock the mic(s) — Each conference room involved will need a different microphone strategy based on the size of the room. A good starting point is a conferencing phone that includes wireless satellite mics. These can be placed around the table but only pick up people actually sitting at the table. Anyone standing away from the table will sound like one of the adults from the Peanuts cartoons. Pass the mic to them or look at adding hanging microphones around the perimeter of your room. On-table mics are also especially good at picking up taps and bumps so aspiring drummers will have to keep it in check.

#3: Limit the background noise —If the remote workers are dialing in from their desk, they can bring a lot of extra noise whenever they unmute their microphone. A headset/mic combo helps but doesn’t totally eliminate it. One solution that we built into our brand new Michigan office is the inclusion of single person “phone booths.” These soundproofed rooms are great to take your laptop into when you have a meeting with the team and you want to eliminate any background noise. It’s also perfect for a 1-on-1 with your boss when you don’t want to broadcast your business to everyone else.

Our new Michigan office is being built with private video conferencing in mind

Change is hard

If you’ve read through all this and still worry that this sounds like a big disruption, fear not! Everything gets easier with repetition. Dialing in to a web conference, learning mic discipline, and adjusting to people working in different time zones will all become second nature after a while. You can also start with a few fake meetings so that everyone can play with their sound levels and get configured without the pressure of trying to actually discuss something work related.

An extra benefit of this setup and preparation is that you’re now prepared if any of your local employees need to work from home. Remote workers can be a huge asset to your team as long as you properly include them. They still can’t participate in the company potluck but I’m confident a research team exists that is focusing on making that a reality.