Forem: Adam McKerlie

Improving the sharing experience in Astro

Adam McKerlie — Thu, 02 Nov 2023 10:00:00 +0000

A couple months ago I migrated my blog from Hugo to Astro because I was more familiar with the underlying language (Javascript) and I wanted to learn this relatively new framework. Overall, the process was extremely easy but I was missing something that came included previously, sharing links to allow people to share my blog posts on social media. That's where Astro Social Share comes in, a fully configurable set of social media buttons for your website. Here's how I went about designing, building and releasing my first NPM package.

Building v0

The first step in making something that others could use was to make it work for myself. I started by opening up my editor and adding in a couple stubs for the sites I knew I'd want to support, Twitter, Reddit, Hackernews and LinkedIn.

<a href="#">Twitter</a>
<a href="#">Reddit</a>
<a href="#">Hackernews</a>
<a href="#">LinkedIn</a>

Next, I had to figure out which URLs to actually send people to and what variables they each take. This was the most difficult step since each site is different. Reddit, for example, would try to load an invalid subreddit unless I encoded the sharing URL before sending people to it. Twitter is the only site that allows tagging the author using the via tag.

<script>
  const encoded_url = encodeURIComponent(url);
</script>

<a
  href="https://twitter.com/intent/tweet?url={url}&text={description}&via={username}"
  >Twitter</a
>
<a href="https://www.reddit.com/submit?url=${encoded_url}&title=${title}"
  >Reddit</a
>
<a href="http://news.ycombinator.com/submitlink?u=${url}&t=${title}"
  >Hackernews</a
>
<a href="https://www.linkedin.com/sharing/share-offsite/?url=${url}"
  >LinkedIn</a
>

Once I had the links working I just had to find icons to use. One way to do this is to Google <company> logo svg but that would lead to styles that don't match. I ended up finding a website that had all of the company logos as well as their company colour to use when I got around to styling the hover state.

<a
  href="https://twitter.com/intent/tweet?url={url}&text={description}&via={username}"
>
  <svg role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">
    <title>X</title>
    <path
      d="M18.901 1.153h3.68l-8.04 9.19L24 22.846h-7.406l-5.8-7.584-6.638 7.584H.474l8.6-9.83L0 1.154h7.594l5.243 6.932ZM17.61 20.644h2.039L6.486 3.24H4.298Z"
    />
  </svg>
</a>
...

With this, I had a working social media sharing bar for this website. Now I just needed to make it easy to use and publish it.

Making it Configurable

Armed with a working MVP I was now challenged to make it easy to use and configurable. Just because I like a certain set of websites to share or a given order doesn't mean everyone will.

The first iteration of this was moving each component into its own Astro file (.astro) and showing people how to import these components. This allows people to include the components they want, in the order they want.

---
// reddit.astro
const { url = Astro.request.url, title } = Astro.props;

const encoded_url = encodeURIComponent(url);
let URL = `https://www.reddit.com/submit?url=${encoded_url}&title=${title}`;
---

<a href={URL}>
  <svg></svg>
</a>

---
// post.astro
import { RedditShareButton } from "./reddit.astro";
---

<RedditShareButton title={title} />

Next up was figuring out how to allow people to only use the single share component while also giving them the flexibility around which share buttons to include and the order of those buttons. At first, I thought about using Astro's integration framework and storing the configuration there. My main concern with this was around usability, the settings would live in a different file than the component and I worried that it'd be confusing to figure out.

After thinking about it for a while I settled on a different approach, creating an optional array of components that could be passed in. This would allow the user to define the order and show buttons alongside the actual component.

---
// SocialShare.astro

import HackerNewsShareButton from "./HackerNews.astro";
import LinkedInShareButton from "./LinkedIn.astro";
import RedditShareButton from "./Reddit.astro";
import TwitterShareButton from "./Twitter.astro";

const DEFAULT_COMPONENTS = [
  TwitterShareButton,
  HackerNewsShareButton,
  LinkedInShareButton,
  RedditShareButton,
];

const {
  buttons = DEFAULT_COMPONENTS,
  url = Astro.request.url,
  title,
  description,
  via,
} = Astro.props;
---

<div>
  {
    buttons.map(Button => (
      <Button url={url} description={description} via={via} title={title} />
    ))
  }
</div>

---
// post.astro
import {
  LinkedInShareButton,
  TwitterShareButton,
  SocialShare,
} from "./SocialShare.astro";

// Optional
const BUTTONS = [TwitterShareButton, LinkedInShareButton];
---

<SocialShare
  buttons={BUTTONS}
  description="Description of the page/post"
  via="YourTwitterAccount"
  title="Page Title"
/>

This final solution felt simple enough to explain and with that, I moved on to turning it into its own package.

Creating an NPM package

With everything ready to go, I now needed to learn how to turn this into an NPM package that people could install and use. The first step is to create a new directory and initialize both git and NPM.


mkdir astro-social-share && cd $_
git init
npm init

Running npm init will walk you through a few basic questions to help you create your new package. There are also a few optional fields you can include to help others find your new package or help them know where to go for bug reports.

package name: (astro-social-share) <enter>
version: (1.0.0) <enter>
description: Social media share buttons for your Astro site <enter>
entry point: (index.js) <enter>
test command: <enter>
git repository: https://github.com/silent1mezzo/astro-social-share
keywords: <enter>
author: Adam McKerlie
license: (ISC) <I chose MIT, up to you.>

If you're wanting to test out your package locally you can do so through npm link which allows you to install a local version of your package and use it in another project. To do this you first call npm link in your package's directory. Then you run npm link <package name> in your other projects directory to install it. Now any changes to your package will be reflected without having to publish a new version first. This is really helpful while building out new features.

Publishing

The last step to all of this was actually publishing this to NPM. The first thing I needed to do was login to npm via the command line npm login. This opens up a browser window to authenticate with npm. After that, I ran npm publish --dry-run to see if everything was working before publishing the package. Once I was ready, I removed the --dry-run flag and published the package directly to npm. With the package published, I installed the package like you would any other and updated this website to pull from the package vs the local copy.

npm install astro-social-share # put in the package name you created

---
// post.astro
import { SocialShare } from "astro-social-share";
---

<SocialShare
  description="Description of the page/post"
  via="YourTwitterAccount"
  title="Page Title"
/>

Finally, I added a postpublish script to my package.json file to also tag a GitHub release

// package.json
...
  "scripts": {
    "postpublish" : "PACKAGE_VERSION=$(cat package.json | grep \\\"version\\\" | head -1 | awk -F: '{ print $2 }' | sed 's/[\",]//g' | tr -d '[[:space:]]') && git tag v$PACKAGE_VERSION && git push --tags"
  },
...

The first feature request

It took approximately one week for the first feature request to come in. While a fairly simple request, just asking for rel="noopener noreferrer" to be added, it was a great milestone for me. It's the first time someone cared enough about something I built to ask for something.

Overall, it's been a fun project to get to know Astro better. It solved a need I had, helped me better understand how Astro works and seems useful for the community. There isn't a lot more I need to do it, other than add more providers and allow users to use their own logos, it's feature complete. If there's something you'd like to see feel free to add a feature request!

The best advice for shipping software

Adam McKerlie — Mon, 30 Oct 2023 00:00:00 +0000

This is the best piece of advice I’ve ever received about building software.

If you’re not embarrassed by the thing you shipped, you waited too long

When people first hear this, they think it means shipping buggy code. It isn’t.

It’s also not about shipping broken experiences.

The mindset revolves around the idea that you’ll learn what you need to build when your customers first start using it. If you wait until you have a full-featured, polished application there’s a good chance you’ve waisted time building the wrong thing.

Figure out what your MVP is and ship it two weeks early.

Talk with customers and see if it resonates. Get comfortable shipping early and often.

Iterate and repeat. Happy shipping! 🚢

As a manager, what is your ideal time spent on coding?

Adam McKerlie — Sat, 27 Aug 2022 12:04:48 +0000

This question was recently asked in a local engineering slack organization and it got me thinking, what would the ideal amount of time spent on technical contributions be for me? When I first transitioned from a developer to a manager I didn't know how I provided value. I assumed that being a manager was largely the same as being a developer except now I had to talk to other developers and help them become better somehow. As meetings, writing more and more documentation, project planning, people planning and other managerial responsibilities started to pile up I started blocking the team on all of my technical tasks. What used to take me a day started taking three or four. Projects that I would have finished in a month stretched on for a quarter and the more technical work that I took on the less happy both my team and I was. I started to burn out and my team started to get frustrated with the way I blocked them on their own work. It took me over 18 months in this first management role before I figured out how I provided value as an Engineering Manager (EM) and started reducing my technical contributions.

So, how much time should you spend?

It depends! Your job as an Engineering Manager is to unblock your team and help your developers grow. Depending on the size and makeup of your team this can mean very different things. It can also mean different things at different times in your career.

If you have a really junior team, you may have to be more hands-on, mentoring them, and teaching them how to build software and grow as an engineer. This could take the form of pair-programming through different features and projects. You may spend more time reviewing pull requests or sitting down and walking them through architecting a solution. Time spent here growing their own engineering skills will hopefully help them down the road as they grow into more senior engineers.

If you've got a more senior team or a team of really strong developers your technical contributions may fall to nothing. Sure, you may want to review a PR or two to make sure you have a handle on how your team is doing. You may also want to unblock your team by taking small, one-off requests that come up but you shouldn't be taking on large features or complex tasks. Taking on this work will only block your team and take away opportunities for your team to shine.

But I love coding!

Me too! Being a great developer is often what gets us to the manager position in the first place and it can be difficult to give up something you enjoy doing. This is where it's important to know what you want to get out of your career and to find a mentor. Going from an IC to a Manager is a career change. This change can be difficult, especially if you weren't expecting it. Having someone to support you during this change and taking the time to really understand what you want from your career can help you let go of coding.

It's also important to find a creative outlet. Coding is a creative career and as you become a manager and your day-to-day changes you'll often find yourself yearning. Lots of Engineering Managers build side projects or contribute to open source projects after hours. This provides a creative release and also allows you to stay technical. Others take up a hobby like woodworking or photography. These kinds of hobbies provide a creative outlet and fill the hole left by coding.

How will my team know what I'm doing if I'm not coding?

One common fear of new managers, especially those that are now managing their peers, is the feeling that they're not doing real work. Because you're no longer hands-on with the code base and often have your schedules chock-full with meetings it's easy to worry that you'll be judged. One thing you can do is be transparent about what you're working on. This not only helps you stay accountable for what you're thinking about and working on but it also helps give your team insight.

Every Monday morning I like to post a quick message to the team about what I'm focused on for the week. These can be presentations that I'm preparing, meetings that I'm having, strategy documents, and conversations that I'm having. You do have to be careful to not talk about confidential things you're working on or things that may be sensitive to some people. These messages help inform the team of what you're doing and help provide transparency. They can also help if you're mentoring any of your engineers into a management position.

Final Thoughts

It can be difficult and scary to give up programming as a manager, especially if you've been doing it for a long time. As a manager, your job is to help your team grow into better engineers and build a team that delivers great software. Depending on your time size and makeup sometimes this means that you still need to code regularly, helping unblock your team. More often than not, your technical contributions will go down as you grow as a manager so make sure you find another creative outlet!

2020 review of the books I read

Adam McKerlie — Mon, 04 Jan 2021 17:37:38 +0000

In 2020 I read 25 books. This was 5 less than the goal I set in 2019, but, as everyone knows, 2020 was a doozy. I often use my commute to read and with Covid happening, I started working from home which reduced my reading time significantly. In June I also switched companies which reduced the time I had to read even further. Overall I’m happy with the number of books that I finished, including some really good ones.

Favourite Books

Resilient Management – Lara Hogan

This was the most useful book I read last year, probably because I led a book club on it which forced me to really pay attention and write down questions for each chapter. Lara Hogan has always been a source of inspiration and actionable insights into how I can be a better manager and Resilient Management was no different. Whether you’re new to management or a veteran I strongly suggest reading this book.

I Am C-3PO: The Inside Story – Anthony Daniels

I love Star Wars and a good number of the books I read are related to it. Getting an inside look into Anthony Daniels’ experience playing C-3PO was a part of the story I hadn’t heard yet. I listened to this on Audible and hearing it from Anthony Daniels directly made it even better.

The Better Allies Approach to Hiring – Karen Catlin

This book opened my eyes to areas that I can improve upon while hiring and building more inclusive teams. Examples such as tagging photos for accessibility on your career website, rethinking the “dude wall” and understanding the importance of retention definitely resonated with me.

A Christmas Carol – Charles Dickens

Growing up The Muppet Christmas Carol was my favourite movie but until this year I had never actually read the original. I was surprised at how close the Muppets version was to the original and really enjoyed reading it for the nostalgia it brought on. It’s a pretty quick read and one I’ll be doing every Christmas now.

Statistics

Last year I read/listened to 25 books. You can find the full list here. I listened to 18 audiobooks (at 1.5x speed) for a total of 98 hours of listening time (4.1 days). This is one of the reasons why I was able to get through so many books. On average it took me 5.2 days to get through an audiobook vs 43.3 days to get through a hard copy. Of the 6 physical books, I read a total of 1182 pages.

Of the 25 books, only 10 were written by women (40%). This was an increase over 2019 where 35% were from female authors and a significant increase over 2018 where only 14% of the books I read were written by women. I plan on continuing this trend into 2021.

The trend continues where I rated books slightly more positive than the rating on Good Reads. In 18 cases I rated the book more favourably than the community, though my average rating was 4.48/5 vs 4.05/5 so the difference is very minimal. I rated books pretty evenly regardless of the gender of the author but I did notice that I rated physical books much higher than audiobooks in general (4.83 vs 4.33). This makes sense since I tend to only buy books that I think I’ll be interested in.

This year I was much more consistent in the genres of books I read. Last year 23% of the books I read fell under the business category. This year Children, Business, Science Fiction, Politics, Self Help and Fantasy all tied for 11.5%

Onwards

I already have a couple of goals for 2021. I want to continue being mindful of the books I’m reading and who’s writing them and I want to make sure that I’m reading books from a diverse set of authors.

The second goal is to read at least 30 books. This is the same as my goal for 2020 but more than I actually read last year. As with last year, I want to be more consistent with the number of books I read in a month. I often get on a roll and then burn out.

My final goal is to read more physical books. I love audiobooks because I can listen to them while I’m walking and driving but I’d really like to get to more physical books. There’s something about flipping through the pages and finally getting to the end that is so much more satisfying than finishing an audiobook.

Here’s to 2021 and more hopefully many more books!

The post 2020 review of the books I read appeared first on Procrastinating Manager.

Meetings can be Poisonous

Adam McKerlie — Tue, 22 Dec 2020 21:03:00 +0000

Eight years ago I wrote about how meetings are poisonous. At the time I was the senior developer on my team and I was getting pulled into 4-5 meetings a day. This killed my productivity and made it difficult to get any of my IC work done. I struggled for many months until I realized that I could just say no to meetings. Since then I’ve transitioned from developer to manager, manager to director and then back to manager. My feelings on meetings have changed over the years but the general advice from eight years ago still stays the same. If you’re interested in reading more about meetings Leigh Espy has a great book called “Bad Meetings Happe To Good People” which is worth a read.

Have a Purpose

The single most important thing you can do to make sure your meeting is successful is to have a purpose. People often set up meetings without having a good understanding of why they need the meeting in the first place. Pure status updates can be done more efficiently over email, slack or project management tools. Meetings where the tone is important or decisions need to be made are good candidates for a successful meeting.

Have an Agenda

Once you’ve figured out the purpose of the meeting the next most important step is having an agenda. Agendas inform participants whether they need to attend or not, help keep the meeting on track and are the starting point for keeping notes. They can be used to summarize the talking points and show the required outcomes from a meeting.

50 or 25 minutes meetings

Parkinson’s Law is as true for writing code as it is for meetings. The meeting expands so as to fill the time available for its completion. If you schedule an hour the meeting will more likely take an hour, even if you only need 50 minutes or less. Google Calendar even has a setting to help you called Speedy Meetings.

Enabling this setting means your 30-minutes meetings will be by default 25-minutes long, 45-minutes meetings will be 40-minutes long, and 60-minutes meetings will be 50-minutes long. This also helps give people a small break between meetings to get a glass of water, use the restroom or catch-up on notifications.

No Laptops or Phones

Eight years ago when I first wrote about this most of my meetings were in person. Putting your laptop and phone away helped reduce distractions and kept meetings on track. Now that most of the world has gone remote and will continue to do so for the foreseeable future it’s all about minimizing distractions. Close everything except for the video call and wherever you take notes, put your laptop and phone into do not disturb and resist the temptation to open a browser window and fall down the rabbit hole that is the internet.

Take Charge

The person who calls the meeting should take charge or talk to someone beforehand that will lead it. The goal of this moderator is to make sure the meeting sticks to the agenda, allows the conversation to wander when productive but brings it back as soon as people get off track. Depending on the personalities in the meeting this can be a very difficult job. It can be a fine line between a productive conversation and one that’s completely out of control. Your job as a moderator is to also make sure everyone has a chance to talk. That means calling on the quiet ones and giving them a space to talk.

As few people as possible

Meetings are one of the most costly parts of your business. Research shows that across the United States, meetings that are not even necessary waste over $25 million every single day, or about $37 billion every year. Smaller meetings not only reduce the cost of the meeting they also make it easier to stick to the agenda, make sure everyone has a say and make decisions. If you’re finding your meetings are starting to bloat try using the RACI framework to determine who needs to be there. RACI stands for:

Who are the ones who will be Responsible for implementing decisions made (Developers, ICs)
Who is Accountable for decisions made (Management generally)
Who needs to be Consulted before decisions are made (Experts, Stakeholders)
Who simply needs to be Informed of the outcome if it impacts them (everyone else)

Often only those Responsible and Accountable need to be in the meeting. They should consult with the experts and stakeholders beforehand and a summary with the meeting notes and decisions made should be sent out to those that need to be informed afterwards. This helps keep meetings to a manageable amount.

Action Plan

Every meeting should end with an action plan. This is a list of meeting notes, a list of the decisions made in the meeting and a list of action items with due dates and assignees. If a follow-up meeting is needed this should be done in the meeting as well. This list of action items should be clear and concise and people should walk away knowing what’s required of them and when it’s due.

There’s a lot that needs to go into making a meeting productive and valuable but if you spend a few minutes on each meeting everyone involved will benefit. What are your favourite tips for running good meetings?

The post Meetings can be Poisonous appeared first on Procrastinating Manager.

Running an Engineering Management Book Club

Adam McKerlie — Wed, 16 Dec 2020 16:26:58 +0000

Being a manager can be extremely hard, it can feel isolating at times especially if you don’t have a good support network of peers. But how do you cultivate this team of peers when your days are filled with 1:1s, meetings, code reviews and other things you need to get done?

Earlier this year I was struggling with this question. I needed to expand and nurture my peer group but was too busy with the day to day tasks of leading multiple teams to do anything regularly. That’s when I found this excellent post on starting an engineering manager book club by Daniel Na. I asked my peers if they’d be interested in starting a book club for engineering managers. They said yes and I got started thinking about what it could look like.

I wanted the book club to, first and foremost, be a place where we could help one another with challenges in our day to day jobs. Sometimes helping someone is as simple as listening to their problems. Sometimes you have experience and can offer more targeted advice and sometimes people just need to vent. I wanted a place where this could happen freely. I also wanted us to grow as a team, learning new ways to manage our engineers, new techniques to motivate and help grow people’s careers.

Having now led a few of these I thought I’d share some tips that have worked well.

1. Keep it small, 8 people or less

One of the goals is to allow everyone a chance to discuss the book and talk about how that’s impacting their day to day challenges. If the group is too big it can make it difficult for a couple of reasons. One reason, logistically speaking, is that it’ll be hard to get everyone a chance to talk in the allotted time. The other problem is a little more nuanced. As the group grows people can be worried about sharing their opinion with a large audience. You may also have someone who dominates the conversation. It’s the job of the moderator to keep the loud ones from talking too much and to get the quiet ones to share their opinions. Keeping the group smaller helps with both of these.

2. Choose the book together

It can be easier to want to read your favourite management book. You know it’s great and will help others just like it helped you. It’s important to choose the book as a group though because it broadens the potential list of books and gives everyone an opportunity to suggest their favourite book. I asked for people’s favourite management books and then did a Slack poll allowing people to vote for their top three. We then read the most voted on the book. This helped ensure the majority of people were actually interested in the book club and helped with attendance.

3. Schedule every meeting from the start

One of the things that worked really well was scheduling every meeting in advance. I planned out every chapter that we’d be reading and scheduled it for the same time every week. We all have busy calendars and planning in advance made it easier to carve out a chunk in everyone’s day. It also helped me plan for days that didn’t work for the majority and allowed me to not have to skip a week because I couldn’t find a time slot that worked.

4. Have the meeting early in the morning

This one is very team and timezone dependent. We found that early mornings before the rush of the day started worked best. People were less likely to have conflicts or meetings that went long and it helped set the tone for the day. I always felt refreshed after the book club, feeling like I could be a better manager to my team.

5. Keep it manageable each week (1 or 2 chapters)

This one is very book dependent. Some books throw a lot of discussable content into each chapter. Some books have very short chapters with themes that span multiple chapters. Plan the book to read 30-60 pages a week.

6. Have some prepared questions but allow the conversation to flow

The goal of a moderator is to guide the conversation. Sometimes that means letting people go where they need to go and sometimes that means guiding the conversation with specific questions. I usually find 3 or 4 questions is the right number for an hour-long discussion with <8 people.

Icebreakers are a good way to get everyone comfortable talking. I like asking questions like “What’s the best piece of advice you’ve ever been given?” or “What’s the most pressing decision you need to make right now”. Sometimes these icebreakers take the entire meeting and that’s ok! If you’re looking for some good icebreakers Vanessa Van Edwards has a great article on them.

It’s also ok to have periods of silence. Silence is a tactic you can use to empower others to participate, to drive home an important answer or just to figure out what you want to say next.

7. Call on the quiet people

Not everyone is comfortable sharing. Some people take longer to warm up than others and some people feel like they have nothing valuable to share. Every manager has a unique perspective on things and when they contribute the whole group grows. It’s the job of the moderator to know who’s participating and who isn’t and call on the quiet ones to provide their opinion.

8. Vegas Rules

What happens in the book club stays in the book club. One of the most important rules of running a book club is creating psychological safety. If people don’t feel safe in sharing their management challenges then the discussion will never move past superficial conversation. If the group commits to open and honest conversation that stays within the walls of the book club you can really make something great.

9. Know when to quit a book

Not every book is a winner. You and your book club don’t have to power through a book that isn’t working for you. After the first few meetings, it’s a good idea to check in on everyone and see how they’re enjoying the book. If it feels like you’re trying to pull teeth getting people to share it may be time to move on to another book.

These are the things that helped our group have a successful book club. I hope you all can level up your peer group with your own book club and I’d love to hear how it goes.

The post Running an Engineering Management Book Club appeared first on Procrastinating Manager.

What's the worst piece of advice you've ever received?

Adam McKerlie — Sun, 31 May 2020 12:58:21 +0000

I've had a few throughout my career:

You should really focus on C, don't bother learning Python.

Focus on whatever makes you happy. You'll be way more likely to succeed.

Slow down

There are definitely times where it makes sense to slow down, especially if you're burning out. But momentum can be extremely useful throughout your career.

You shouldn't try to go to university, you don't have the attention span for it

This was my grade 11 music teacher. If I had listened I wouldn't have gone into Computer Science. Don't listen to people who try to limit what you can accomplish.

What's the one thing you always need to google?

Adam McKerlie — Sat, 30 May 2020 01:42:41 +0000

For me, it's reading/writing CSV's in Python. I've done it a thousand times but I can never remember the syntax.

7 Musings on Minion Management

Adam McKerlie — Tue, 19 May 2020 13:35:57 +0000

So, you’ve just been promoted to the dark side of management but you don’t know how to lead your minions? Here are a few tips on keeping your plebeians in line.

Make Your Minions Do the Dirty Work

You shouldn’t worry yourself about work that’s underneath you, you get paid way to much. You’ve hired coder monkeys to handle everything from downed servers at 3 am to documentation to those pesky clients. All you have to do is sit back and make sure they don’t lose focus.

Don’t Give Any Praise

Your minions just put in 100 hour weeks to get the feature launched on time? They get paid to do that work so there’s really no need to tell them they’re doing their job. Work horses don’t need to be told they’re doing a good job every time so why should your employees? As soon as you’re done you should give them more work since that’s what they need.

Belittle Your Minions

You’re the manager so that makes you better than everyone else. If they were as good as you, they’d be managing people themselves. This means you’ve earned the right to make the others know that they aren’t as good as you.

Always Leave Before Your Minions

Your time is way more valuable and you don’t want to wear yourself out. This means you should come to work later and leave earlier than your team. Besides, as long as you’re doing a good job guiding your newbies they should be able to carry on their work when you’re gone.

Avoid All Responsibility

If your staff messes something up it’s their fault, not yours. Why should you be held responsible for something they’ve done. When the top brass asks you what happened make sure they know exactly who screwed up. This also secures your position because you’ll never do anything wrong.

Believe That Your Job Is More Valuable

Obviously the people underneath you wouldn’t be able to do their job without you. Who would lead them? You get paid more so that means you’re way more valuable than them. Make sure that they know you’re more valuable to the company.

Always Think You Are Right

Again, you’ve probably been there longer and you get paid more so that means you’re always right. When someone tries to tell you otherwise make sure you make them look bad to their co-workers; this will show them who’s boss.

If you’ve gotten to this point and haven’t realized you shouldn’t do any of these things you probably shouldn’t be managing people. If you realized that you should be doing exactly the opposite than go forth and manage!

The post 7 Musings on Minion Management appeared first on Procrastinating Developer.

Using Python to generate over 10,000 unique 8-bit lightsabers

Adam McKerlie — Mon, 04 May 2020 10:44:00 +0000

Python is great for a number of things. It powers 1.4% of the internet, Nasa uses it a lot! and you can use it to create art. In honour of Star Wars Day, I wanted to create a program that dynamically generated lightsabers and tweeted them out once a day.

TL;DR

I created a computer program that generates a unique lightsaber made out of four different parts (blade, hilt, pommel and button) and tweets it out once a day along with some statistics about the lightsaber.

Luke briefly had a red lightsaber

Getting Started

The overall premise of this script is fairly simple. It randomly selects four pieces of a lightsaber and then pieces them together. To achieve this I used an online tool called Piskel to generate all of the 8-bit parts. While Piskel is normally used to create animated sprites for websites it’s one of the better 8-bit art editors. My favourite feature from it is the ability to mirror lines. This simplifies the process of making symmetrical lightsabers.

Once I created a base set of parts put them in an images directory. This directory has 5 folders, one for each part and an output folder (lightsabers). This allows me to keep everything organized as well as helps later on when generating lightsabers.

images/
    blades/
        b1.png
        b2.png
    hilts/
        h1.png
        h2.png
    buttons/
        u1.png
    pommels/
        p1.png
    lightsabers/
        h1b2u1p1.png

To do all of this I installed three python packages. I use Pillow (version 7.1.1) for all of the image processing, Tweepy (version 3.8.0) to send out the tweets and numpy (version 1.18.3) to do some colour processing that I’ll discuss later on. Overall it’s a very simple requirements.txt

# requirements.txt
numpy==1.18.3
Pillow==7.1.1
tweepy==3.8.0

The first version of this script was really this simple. Call generate_lightsaber and that’s it. Up next I’ll talk about how I generated the initial lightsaber (just the blade and hilt), then how I added buttons and pommels and finally how I tweeted the final image.

# lightsaber.py
import os, random
from pathlib import Path, PurePath
# This is actually Pillow but it's API is backwards compatible with the older PIL
from PIL import Image

def generate_lightsaber():
    # Will talk about this more next

if __name__ == " __main__":
    generate_lightsaber()

Adding Blades and Hilts

In my initial design, I only had blades and hilts so that’s where I started. My first goal was to get a single blade and a single hilt merged and properly lined up on a single image. I first added code to fetch a unique lightsaber part based on the directory layout above.

# lightsaber.py
# Set some constants to keep the code clean and consistent
IMAGE_PATH = Path('../images')
BLADE_PATH = IMAGE_PATH / 'blades'
HILT_PATH = IMAGE_PATH / 'hilts'
OUTPUT_PATH = IMAGE_PATH / 'lightsabers'

def fetch_lightsaber_parts():
    hilt = Path(f"{HILT_PATH}/{random.choice(os.listdir(HILT_PATH))}")
    blade = Path(f"{BLADE_PATH}/{random.choice(os.listdir(BLADE_PATH))}")

    return (hilt, blade)

First, we have os.listdir(), which returns a list of filenames in the directory given. The list is in arbitrary order and it does not include the special entries ‘.’ and ‘..’ even if they are present in the directory. It does, however, include any folders or special files (like .DS_store in OS X) so make sure your folder only includes the image files. We pass that filename into random.choice which will choose a random filename. Finally, we create the full file path and return that for both the hilt and the blade.

Once we’ve gotten the image paths we can start piecing them together into a final image.

# lightsaber.py
def generate_lightsaber():
    blade_path, hilt_path = fetch_lightsaber_parts()

    # Open the images using Pillow
    blade = Image.open(blade_path, 'r')
    hilt = Image.open(hilt_path, 'r')

    # Open the output image. Twitter displays the entire image if it's 1024x512
    output = Image.new("RGB", (1024, 512), (255, 255, 255))

    # Paste the blade and hilt onto the output image. 
    output.paste(blade, blade_offset, mask=blade)
    output.paste(hilt, hilt_offset, mask=hilt)

    # Save the output image to disk
    img.save("{}/{}.png".format(OUTPUT_PATH, output_filename))

First, we’ll fetch the paths as described above. After that, we’ll use Pillow to open the images as well as create a new output image with the width and height specified by Twitter to allow us to show the entire image as a preview. Finally we paste the two original images and save the output.

When pasting the images onto the output you’ll see two arguments that I haven’t shown yet, <part>_offset and mask. The offset is used to determine where the parts are placed on the output image. When you get this wrong, funny things can happen.

The tiniest of blades

To calculate the offset, where the image gets placed, I need to get and store all of the dimensions of the images. Then we calculate the middle position for the width (output_w - blade_w) // 2 this aligns it horizontally on the image. This is true for both the blade and the hilt.

To position the blade and hilt vertically it requires a little more math. For the hilt, we take the height of the output image (512px) and subtract the height of the hilt image (between 70 and 120px depending on the design). This makes the hilt start at the bottom of the image.

# lightsaber.py
output_w, output_h = output.size
blade_w, blade_h = blade.size
hilt_w, hilt_h = hilt.size

blade_offset = ((output_h - blade_h - hilt_h + hilt_offset), (output_w - blade_w) // 2)
hilt_offset = (output_h - hilt_h, (output_w - hilt_w) // 2)

You can think of the image like an array

Using the array above as an example, if we had a lightsaber 1px wide and 2px (1px hilt, 1px blade) tall the math would be.


blade_offset = ((4 - 1 - 1), (5 - 1) // 2 ) # (2, 2)
hilt_offset = ((4 - 1), (5 - 1) // 2) # (3, 2)

Finally, we have to pass in the original image as a mask, otherwise, Pillow will default the transparent backgrounds into black. By using a mask Pillow only writes data to the output image for the pixels that have colour. The image below on the left has a mask applied, the one on the right does not.

Adding Hilt Offsets and Extra Information

When calculating the blade offset you may have noticed a variable hilt_offset. In some designs, the part where the blade comes out of the hilt isn’t the start of the lightsaber. In these cases I needed the blade to start “below” where the hilt starts. Knowing that I needed to do this but also store metadata on hilts, blades, buttons and pommels down the road I created a manifest.py file to store these defaults.

# manifest.py
MANIFEST = {
    "hilt": {
        "h1": {
            "offsets": {
                "blade": 5
            }
        }
    }
}

# lightsaber.py
from manifest import MANIFEST

def get_hilt_offset(hilt):
    return MANIFEST['hilt'][hilt]['offsets']['blade']

def generate_lightsaber():
    ...
    hilt_offset = get_hilt_offset(hilt)
    blade_offset = ((output_h - blade_h - hilt_h + hilt_offset), (output_w - blade_w) // 2)

Without this offset, I was originally getting a floating blade for certain designs. You can see below that the bottom of the blade lines up with the top of the rightmost side of the lightsaber but it still looks funny.

The force must be keeping this thing together

This manifest file also holds information on button offsets, colours and tweet information but I’ll go more in-depth into that later on.

Adding Buttons

After getting the blades and hilts lined up I realized that do hit 10,000 unique lightsabers with only blades and hilts I would need to design 100 of each (200 total). My 8-bit art skills just aren’t up to that. By adding buttons I could cut down the unique designs needed only 22 each (66 total). Buttons were added to the final image in exactly the same way as hilts and blades.

# lightsaber.py
# Set some constants to keep the code clean and consistent
IMAGE_PATH = Path('../images')
BLADE_PATH = IMAGE_PATH / 'blades'
HILT_PATH = IMAGE_PATH / 'hilts'
BUTTON_PATH = IMAGE_PATH / 'buttons'
OUTPUT_PATH = IMAGE_PATH / 'lightsabers'

def fetch_lightsaber_parts():
    hilt = Path(f"{HILT_PATH}/{random.choice(os.listdir(HILT_PATH))}")
    blade = Path(f"{BLADE_PATH}/{random.choice(os.listdir(BLADE_PATH))}")
    button = Path(f"{BUTTON_PATH}/{random.choice(os.listdir(BUTTON_PATH))}")

    return (hilt, blade, button)

def generate_lightsaber():
    blade_path, hilt_path, button_path = fetch_lightsaber_parts()

    # Open the images using Pillow
    blade = Image.open(blade_path, 'r')
    hilt = Image.open(hilt_path, 'r')
    button = Image.open(button_path, 'r')

    # Open the output image. Twitter displays the entire image if it's 1024x512
    output = Image.new("RGB", (1024, 512), (255, 255, 255))

    output_w, output_h = output.size
    blade_w, blade_h = blade.size
    hilt_w, hilt_h = hilt.size
    button_w, button_h = button.size

    hilt_offset = get_hilt_offset(hilt)
    blade_offset = ((output_h - blade_h - hilt_h + hilt_offset), (output_w - blade_w) // 2)
    hilt_offset = (output_h - hilt_h, (output_w - hilt_w) // 2)

    # Paste the blade and hilt onto the output image. 
    output.paste(blade, blade_offset, mask=blade)
    output.paste(hilt, hilt_offset, mask=hilt)
    output.paste(button, get_button_offset(hilt), mask=button)

    # Save the output image to disk
    img.save("{}/{}.png".format(OUTPUT_PATH, output_filename))

The main complication with buttons is that they can be placed in a square on the hilt and this square is different for each hilt. There are a few different ways we could do this automatically (detect edges, detect width and height of the blade, etc…) but for simplicity’s sake, I just decided to manually calculate it for each hilt and store it in the manifest file. The x

# manifest.py
"hilt": {
    "h1": {
        "offsets": {
            "blade": 0,
            "button": {
                "x": (8, 9),
                "y": (110, 111)
            },
        },
    },
}

# lightsaber.py
def get_button_offset(hilt):
    between_x = MANIFEST['hilt'][hilt]['offsets']['button']['x']
    between_y = MANIFEST['hilt'][hilt]['offsets']['button']['y']

    return (random.randint(between_x[0], between_x[1]), random.randint(between_y[0], between_y[1]))

Adding Pommels

Once again, pommels were added to the image in much the same way as the others only this time I needed to update the height of both the blade and the hilt to accommodate the pommel. This didn’t go well at first.

This is what happens when you get your +’s and -‘s mixed up

# lightsaber.py
# Set some constants to keep the code clean and consistent
IMAGE_PATH = Path('../images')
BLADE_PATH = IMAGE_PATH / 'blades'
HILT_PATH = IMAGE_PATH / 'hilts'
BUTTON_PATH = IMAGE_PATH / 'buttons'
POMMEL_PATH = IMAGE_PATH / 'pommels'
OUTPUT_PATH = IMAGE_PATH / 'lightsabers'

def fetch_lightsaber_parts():
    hilt = Path(f"{HILT_PATH}/{random.choice(os.listdir(HILT_PATH))}")
    blade = Path(f"{BLADE_PATH}/{random.choice(os.listdir(BLADE_PATH))}")
    button = Path(f"{BUTTON_PATH}/{random.choice(os.listdir(BUTTON_PATH))}")
    pommel = Path(f"{POMMEL_PATH}/{random.choice(os.listdir(POMMEL_PATH))}")

    return (hilt, blade, button, pommel)

def generate_lightsaber():
    blade_path, hilt_path, button_path, pommel_path = fetch_lightsaber_parts()

    # Open the images using Pillow
    blade = Image.open(blade_path, 'r')
    hilt = Image.open(hilt_path, 'r')
    button = Image.open(button_path, 'r')
    pommel = Image.open(pommel_path, 'r')

    # Open the output image. Twitter displays the entire image if it's 1024x512
    output = Image.new("RGB", (1024, 512), (255, 255, 255))

    output_w, output_h = output.size
    blade_w, blade_h = blade.size
    hilt_w, hilt_h = hilt.size
    button_w, button_h = button.size
    pommel_w, pommel_h = pommel.size

    pommel_offset = pommel_h
    hilt_offset = get_hilt_offset(hilt_name) - pommel_offset
    button_offset = get_button_offset(hilt_name)  

    blade_offset = ((output_h - blade_h - hilt_h + hilt_offset), (output_w - blade_w) // 2)
    hilt_offset = (output_h - hilt_h - pommel_offset, (output_w - hilt_w) // 2)
    pommel_offset = (output_h - pommel_h, (output_w - pommel_w) // 2)

    # Paste the blade and hilt onto the output image. 
    output.paste(blade, blade_offset, mask=blade)
    output.paste(hilt, hilt_offset, mask=hilt)
    output.paste(button, get_button_offset(hilt), mask=button)

    # Save the output image to disk
    img.save("{}/{}.png".format(OUTPUT_PATH, output_filename))

The next challenge I encountered with pommels was designing them so they looked nice across all hilts. When I first designed the lightsaber I included both the hilt and the pommel. When I realized I needed to split them out to allow for more unique combinations I simply cut the old 8-bit images in two. The problem with this is that the colour schemes between hilts and pommels just didn’t match.

After a lot of thinking I decided that I could do something similar to how green screens work. If I designed the images with specific colours (in this case Red = Primary, Blue = Secondary and Green = Tertiary) I could pull them out using Pillow and Numpy and substitute them for the colours I want.

# manifest.py
"hilt": {
    "h1": {
        "offsets": {
            "blade": 0,
            "button": {
                "x": (8, 9),
                "y": (110, 111)
            },
        },
        "colours": {
            "primary": (216,216,216), #d8d8d8
            "secondary": (141,141,141), #8d8d8d
            "tertiary": (180, 97, 19), #b46113
        },
    },
}

# lightsaber.py
import numpy as np

def convert_colours(img, hilt):
    img = img.convert('RGBA')
    data = np.array(img)

    # Grab the pixels which are 100% red, 100% blue and 100% green
    red, green, blue, alpha = data.T
    primary = (red == 255) & (blue == 0) & (green == 0)
    secondary = (red == 0) & (blue == 255) & (green == 0)
    tertiary = (red == 0) & (blue == 0) & (green == 255)

    # Substitute out the colours for the hilt colour scheme
    data[..., :-1][primary.T] = MANIFEST['hilt'][hilt_name]['colours']['primary']
    data[..., :-1][secondary.T] = MANIFEST['hilt'][hilt_name]['colours']['secondary']
    data[..., :-1][tertiary.T] = MANIFEST['hilt'][hilt_name]['colours']['tertiary']

    return Image.fromarray(data)

def generate_lightsaber():
    ... 
    pommel = Image.open(pommel_path, 'r')
    pommel = convert_colours(pommel, hilt_name)
    ...

Once I did that everything was all set. The pommels now have the correct colour scheme as the lightsaber.

Generating Random Tweet Text

The final part of all of this is generating the actual tweet. I added a number of fields to the manifest file including the hilt length and material, the blade colour, crystal and who used it and finally the pommel length. After that a new function, generate_tweet_text, that pulls all of this new information together and generates the text to tweet out.

# manifest.py
"hilt": {
    "h1": {
        "offsets": {
            "blade": 0,
            "button": {
                "x": (8, 9),
                "y": (110, 111)
            },
        },
        "colours": {
            "primary": (216,216,216), #d8d8d8
            "secondary": (141,141,141), #8d8d8d
            "tertiary": (180, 97, 19), #b46113
        },
        "length": 24,
        "materials": "Alloy metal/Salvaged materials"
    },
},
"blade": {
    "b1": {
        "colour": "Red",
        "crystal": ["Ilum crystal", "Ultima Pearl"],
        "type": "Sith"
    },
},
"pommel": {
    "p1": {
        "length": 5,
    },
}

# lightsaber.py
AVERAGE_HILT_LENGTH = 25
AVERAGE_POMMEL_LENGTH = 3
AVERAGE_BLADE_LENGTH = 90

NAMES = ['List', 'of', 'generated', 'names']

def generate_tweet_text(hilt, blade, pommel):
    hilt_details = MANIFEST['hilt'][hilt]
    blade_details = MANIFEST['blade'][blade]
    pommel_details = MANIFEST['pommel'][pommel]

    hilt_length = hilt_details['length']
    pommel_length = pommel_details['length']

    total_length = hilt_length + pommel_length
    average_length = AVERAGE_HILT_LENGTH + AVERAGE_POMMEL_LENGTH
    blade_length = int(AVERAGE_BLADE_LENGTH * (total_length / average_length))

    title = blade_details['type']
    if type(title) is list:
        title = random.choice(title)

    crystal = MANIFEST['blade'][blade]['crystal']
    if type(crystal) is list:
        crystal = random.choice(crystal)

    name = f"{title} {random.choice(NAMES)}"

    tweet = f'''Owner: {name}
Hilt Length: {total_length} cm
Blade Length: {blade_length} cm
Blade Colour: {MANIFEST['blade'][blade]['colour']}
Kyber Crystal: {crystal}

#StarWars
'''

    return tweet

To generate the unique names I used this Star Wars name generator to generate a number of names. I then randomly choose one to use.

To tweet out the final tweet I use the python library Tweepy to do all of the heavy lifting for me. Since I already have the image saved on file all I need to do is grab the credentials from an environment variable, upload the media and then post the tweet.

consumer_key = os.getenv('CONSUMER_KEY')
consumer_secret = os.getenv('CONSUMER_SECRET')

access_token = os.getenv('ACCESS_TOKEN')
access_token_secret = os.getenv('ACCESS_TOKEN_SECRET')

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)

tweet_text = generate_tweet_text(hilt, blade, pommel)

media = api.media_upload(path)
api.update_status(status=tweet_text, media_ids=[media.media_id,])

If this is your first time setting up a twitter bot you’ll need to create an application first, grab your credentials and then store them in an environment variable.

Wrapping it all up

Now that everything’s running correctly, I’ve got it generating a unique lightsaber and posting it to Twitter I just have to add it to a Cron to run every day.

12 21 * * * python3 lightsaber.py >> lightsaber.log

This will run the program, every day at 9:12 pm. And with that, we’re done! We’ve got a Python program that generates a unique lightsaber, every single day, and tweets it out. Which Jedi/Sith owns your favourite lightsaber?

The post Using Python to generate over 10,000 unique 8-bit lightsabers appeared first on Procrastinating Developer.

Migrating our API Gateway from Flask to Django

Adam McKerlie — Tue, 17 Apr 2018 16:12:56 +0000

“Fresh pineapple sitting on the ocean shoreline” by Pineapple Supply Co. on Unsplash

At G Adventures, we’ve been building a publicly accessible API for the past three years. We’ve been pushing hard to make the G Adventures API a standard within the Adventure Travel industry, and we believe we’re doing so by providing superb documentation & support, as well as enabling our customers with a standards-compliant JSON/XML API, powered by a suite of services and tools.

When we first tackled the construction of our API, we wanted to keep things simple. We have many systems at G Adventures which own data, and we wanted to ensure that the API would not be an owner of data, simply focused on delivering data.

Thus, we looked towards frameworks that kept things simple. In the Python ecosystem, one of the most popular frameworks for keeping things simple is Flask.

Flask is a micro-framework that truly does not get in your way. It ensures you can write plain Python, with additional batteries included. This was great for our needs, as our mandate was:

Build a system that can capture a single request, and transform that into potentially multiple requests to source systems. Effectively, an API Gateway
Ensure we don’t re-invent the wheel when it comes to HTTP Standards
Keep it lean and fast. Speed is important. Both from development ramp-up and data delivery perspectives

Thus we began iterating on our API prototype. Flask truly did not get in our way. In a couple of days, we had a prototype that would proxy a single client request into multiple requests behind-the-scenes. We iterated on this prototype until we came closer to a reasonable product. Building out our resource definitions in pure Python, we were able to define common patterns, and store them as simple dictionaries. We’ll touch a bit more on what this looks like today shortly.

Eventually, we launched our API. It was a success! We had many internal, and external clients consuming G Adventures data. We began expanding our system by offering more resources, tools like OAuth, payload validation, and so on. As we began to add these tools, we introduced concepts like an ORM (Via SQLAlchemy) and migrations (Alembic) to our infrastructure.

These aren’t foreign concepts, and the tools mentioned are superb, but we’d began to slide away from our core fundamentals. Our API was becoming larger in scope, we began re-inventing the wheel, and our simple approach of dictionaries for resource definition worked, but didn’t. Essentially, we’ve hit a point where we could identify that if we continue to build upon this micro-framework, we’d build upon our technical debt. This was an opportunity to rethink

We began to think of how we move forward. We want to keep the API layer simple, and allow other microservices to provide the various features of the G Adventures API, but we wanted to have a consistent, well-structured set of ideas we could leverage and build from.

And thus, we waved hello to Django Rest Framework. Django Rest Framework is a perfect companion to a system where you have Django Models, and you want to expose an API to them. It’s so simple that most of our internal systems that expose an API to our API Gateway use it. However, we were in a different situation than a typical Django project.

We were actively running an API service which was receiving millions of hits per week, we wanted to leverage Django Rest Framework, but would need to pivot it to fit our needs of multiple HTTP backend services. Essentially, we had to make Django Rest Framework work in the ways we’ve structured our API. Let’s take a look at what that looked like.

Migrating while the machine is running

On the API, we support 90 resources, each with a different degree of complexity. A read-only resource on one backend is easier to test than a read-write resource using many backends. At the end of the day, migrating more than 5 simple resources at once is asking for trouble. Our decision was to migrate a few at a time and proxy the requests to the appropriate process (Flask or DRF). This allowed us to migrate slowly, communicate with our clients in case of errors and easily route back to Flask if needed.

Being able to revert to Flask served us well. You can migrate resources with precise respect to the documentation, but that does not guarantee success. Django Rest Framework could reject a request while the Flask framework wouldn’t. For example, Flask will happily process:

POST to /bookings POST to /bookings/resource

While Django Rest Framework will only accept:

POST to /bookings

Our documentation always said to use bookings. bookings/resource was not prohibited in our documentation, it simply wasn’t mentioned. One of our clients used bookings/resource, it went through and they continued to use it. When we migrated bookings to DRF, the client messaged us about it. Flask overlooked this error but DRF did not. Well, we were surprised bookings/resource was a valid request on our own API. I told the client to remove resource for all our resources. Now imagine we migrated all 90 resources at once — all integrations using <resource>/resource would be sad.

Customizing our endpoints

With Django Rest Framework, a resource is served by binding an endpoint to a view. For example, to serve bookings on the API, we simply had to register it:

router.register(bookings, BookingView)

Using DRF generic router, this will support requests on:

bookings/\<booking\_id\>

Our Flask supported other URL variations, and one of our challenges was to properly route all existing requests. Here is what we needed to serve:

\<resource\>/\<id\>/ \<resource\>/\<id\>/\<resource\_2\> \<resource\>/\<id\>/\<variation\_id\> \<resource\>/\<id\>/\<variation\_id\>/\<resource\_2\>

First, we found how DRF parses URL: https://github.com/encode/django-rest-framework/blob/1c53fd32125b4742cdb95246523f1cd0c41c497c/rest_framework/routers.py #L241.

This means our bookings/<id> will be valid through DRF’s default regex:

base\_regex=’(?P\<{lookup\_prefix}{looking\_url\_kwarg}\>{lookup\_value})’

URLs are constructed in get_urls, we can easily overwrite this function:

def get\_urls(self): urls = super().get\_urls() return urls + \<our custom urls\>

Our custom URLs will take care of:

\<resource\>/\<id\>/\<resource\_2\> \<resource\>/\<id\>/\<variation\_id\> \<resource\>/\<id\>/\<variation\_id\>/\<resource\_2\>

Let’s say we had an itineraries resource with ID 2222, a possible variation ID of 3333 and also a nested resource of itinerary_maps. These are all valid requests:

Variation 1: itineraries/2222/3333 Variation 2: itineraries/2222/itinerary\_maps Variation 3: itineraries/2222/3333/itinerary\_maps

First, we overwrote this method https://github.com/encode/django-rest-framework/blob/1c53fd32125b4742cdb95246523f1cd0c41c497c/rest_framework/routers.py#L230 to add extra regexes, such as:

\<base\> + "/\*(?P\<variation\_id\>[^/]\*)"

Notice in our get_urls method, we call super() before we create our nested URLs. super().get_urls() calls self.get_lookup_regex(viewset) which constructs our variation_ids before constructing our nested URLs (i.e. those ending with <resource_2>).

The call to super() creates:

itineraries/2222 itineraries/2222/3333

While making our nested URLs, we append these urls, also creating:

itineraries/2222/itinerary\_maps itineraries/2222/3333/itinerary\_maps

By constructing our regex orderly, we are able to create these custom URLs and bind them to the appropriate views.

Routing our backends

I mentioned backends a few times in this article, but never formally defined them. A backend is what we call one of our internal systems that provides data into the API. Our reservation system provides bookings, our Salesforce piece provides contact data, our operations tool provides tour data and so on. This means our API must know which backend(s) to use for which resource. We did this by defining a set of backends and binding resources (example):

class Backend: config\_variable = 'B1'

def request: \<initializes a request with the system\> def get\_object: \<uses request to get an object\> def update\_object: \<uses request to update an object\> def create\_object: \<uses request to create an object\>

class Resource: object\_backends = [(READ\_WRITE, 'B1'), (READ\_ONLY, 'B2')]

The above declares Resource to read and write from B1 and then use B2. Notice the order, B2 will actually override data from B2 if fields collide. Now we just need to patch DRF to use this structure, and this is easy. Here is just the function to override: https://github.com/encode/django-rest-framework/blob/7078afa42c1916823227287f6a6f60c104ebe3cd/rest_framework/generics.py#L77 . We did the same for update_object and create_object, allowing us to select backends for resources. Our Flask application operated the same way, but the code for this functionality was much more complex.

Now and future of the API

Our DRF framework has been serving all resources for over a year now and we could not be happier. While Flask served us well initially, we eventually found ourselves reinventing the wheel. With Flask, all the API basics had to be done from scratch. With DRF, we get basic functionalities and an easy way to add our own. A structured framework allowed us to focus on the actual API while allowing us to slightly customize the framework. DRF is just flexible enough to give us endless possibilities. We are currently customizing the API to route different requests to different reservation systems, stay tuned!

Keep following us on Medium and Twitter for more snippets of the Tech world according to G!

Want to help G Adventures change people’s lives and travel the world? Check out all our jobs and apply today.

Why we created a REST API at G Adventures

Adam McKerlie — Wed, 11 Apr 2018 13:43:55 +0000

Photo by Artem Sapegin on Unsplash

As a senior dev at G Adventures, I’ve had the great opportunity to work on the REST API here. This post is the first of many detailing how we got to where we are.

Our core business at G Adventures is small group travel, a social enterprise that strives to show the world to our travellers and in turn improve both the lives of our travellers and the lives of those who live at the destinations we visit.

As one can imagine lots goes into planning a tour on the other side of the planet to ensure everything goes smoothly from a day one. As our business grew so did our various systems. Eventually, it was necessary to have one common interface (or one common language) that our systems could use to talk to each other. We chose REST API which stands for Representational State Transfer and API stands for Application Programming Interface.

We use this interface to expose our business systems to developers both internal and external. As an example

contains all kinds of useful details about our Peru Panorama tour, including things such as tour’s itinerary or links to specific departures.

This means of communication has been a huge factor for our business growth as it allows our partners to integrate with us in a much more cost-effective way (in the past booking a tour with us required the use of a specialized website and often even a phone call). With REST API it’s a few HTTP requests instead.

Internally it allowed us to integrate our various systems and hide implementation details so that application developers no longer have to have intimate knowledge of every single system that we have. Once a developer becomes familiar with the REST API they can use any of our systems in uniform fashion whether they are interacting with our inventory management system or our customer management system (CRM), or anything else we expose via API.

The existence of this uniform interface even allowed whole new applications previously not possible. Such as our automated email engine or universal search functionality.

Not long after we launched the API did we discover the need to also be aware of when things change. E.g. if the above Peru Panorama trip no longer starts in Lima, we need to notify the website, our booking engine, all our API clients, who often display the trip on their own websites.

One way to do this is for every developer to check the above URL at specific intervals (polling). But this isn’t really a good strategy because we run around a thousand different tours with most of them having at least one departure every 10 days. That is a lot of URLs to poll.

Webhooks (don’t call us we will call you)

Webhooks can be compared to push notifications and are a simple mechanism to inform our API users when something in our ecosystem changes. In other words, rather than our users always checking the above URL to see if anything changed we will let everyone subscribed know if there are changes to the Peru Panorama tour.

These webhooks enable a universal mechanism of notifications that various business units use to obtain relevant info as soon as possible. Side effects included us being able to keep a replica of our data that we used for uniform searching and caching to increase functionality and responsiveness of our API.

Our external partners can use the webhooks to observe our inventory, watch for special deals or be immediately informed when anything happens on a booking they made with us.

I am proud of what we were able to accomplish with our API, but in my next installment I’ll get more technical as the story of REST API at G Adventures is far from finished.

Stay tuned.

Keep following us on Medium and Twitter for more snippets of the Tech world according to G!

Want to help G Adventures change people’s lives and travel the world? Check out all our jobs and apply today.