Forem: Josh Stillman

Embrace Your Laziness: Automatically Convert Word Documents into Terms & Conditions Pages

Josh Stillman — Fri, 07 Jan 2022 15:02:30 +0000

Modern Single Page Applications (SPAs) often embed terms and conditions pages into the app itself for a slick and modern feel. While this makes for a great user experience, it can be tedious and time consuming for developers to convert long Microsoft Word documents of legal copy into HTML/JSX that can be embedded into a terms and conditions component or modal. But fret not, fellow developer! With some macOS and shell utilities, you can let the computer handle the drudgery, so you can focus on something more important.

Simply using some built-in command line programs in macOS will do the trick, converting the Word document into clean HTML you can paste into your terms and conditions component! Let's see how—and why.

Laziness Is a Virtue

Larry Wall, the creator of the Perl programming language, argued that laziness is one of the primary virtues of a good programmer. It's what makes coders "write labor-saving programs that other people will find useful." And indeed, a good engineer will "go to great effort to reduce overall energy expenditure" by finding opportunities to make processes more efficient.

Along these lines, I'd say that a primary coding virtue is the ability to identify which tasks are rote, repetitive, and best delegated to a computer, and which tasks instead require human creativity, problem solving, and ingenuity. We'll only ever have time and energy for the latter category if we find a way to let the computers handle the boring, repetitive stuff.

The Inevitable Terms & Conditions Ticket

It's inevitable. When developing a new SPA, there will come a day that you or your teammate will be assigned a ticket to create an embedded terms and conditions page or modal. (That's our litigious modern society. Sigh...) Typically, a developer is handed a Word document from the legal department and some fancy designs, and left to figure out the rest.

The most painstaking approach would be to manually copy each paragraph, add any bold and italic formatting, and wrap it in appropriate HTML tags. This can take a while if it's a long Word document! And it won't be pleasant. Our laziness instincts should be kicking in about now.

We can make this process a little less manual through this nifty VS Code extension. It will let us wrap each paragraph or sentence of text in the appropriate , /, or / tags. But it's still a pretty manual process of copying, pasting, and formatting. How can we fully automate this?

There's a CLI for That

Good news! macOS ships with a command line tool called textutil that excels at converting documents into different formats. It can convert a Word document into HTML in a single terminal command: textutil -convert html -strip terms.docx. This will take your Word document, strip out all the metadata, and convert it into basic HTML markup. Paragraphs will be wrapped in  tags, and bold and italic formatting tags will be added as well. No more need to go through the document paragraph by paragraph yourself to create the markup. And it even works on other document formats, such as .txt and .rtf files. Joy!

Much Too Classy

One problem! textutil creates some basic CSS styles for you based on the source Word document and attaches very generic class names such as p2 and Apple-converted-space to seemingly every tag it creates. But you probably don't want these generated class names polluting your markup. Not only does it just look ugly and hard to read, but these highly generic class names could clash with other classes in your app, leading to unintended consequences.

Sadly, textutil lacks any built-in option to suppress these class names. Sure, we could manually remove all the classes from the generated markup, but we don't want to do that either.

Right Sed Fred

Fear not—we can clean up the HTML that textutil gives us using sed, a shell tool for text manipulation that comes built into Bash and Zsh. We'll pipe the HTML that textutil generates into sed, strip out all the class names, and save the result to a file.

The sed command we'll use to delete the class names is sed 's/class="[^"]*"//g'. Let's break that down. The leading s in the argument means we'll substitute text matching the pattern between the first and second / characters with the text between the second and third /'s. The regex pattern we'll match is class="[^"]*" (explained below). Then, we'll replace the text matching that pattern with the text between the last two slashes—here, an empty string. And we'll do it for every occurrence with the global modifier, /g. That is, we'll simply delete the text matching the pattern throughout the document.

About that funky-looking regex... sed doesn't have the same regex capabilities you're familiar with in modern languages such as JavaScript. It doesn't have lazy matching, meaning that if you try to match class=".*", sed will greedily match far more text than you intended, well beyond the end of the HTML tag.

Instead, we can mock lazy matching in sed with this technique: we can match the opening ", followed by any character except a ", then the closing ". So /class="[^"]*"/ will get us the lazy matching we need—effectively /class=".*?"/ in JavaScript's regex dialect. Lazy matching for lazy programmers!

After running textutil's output through this sed command, we'll have nice, clean markup without all the random class names.

Building on this technique, we could even take it a step further and strip out unnecessary  tags, and anything else we wanted to get rid of from textutil's output.

Putting It All Together

Last, we'll save the cleaned HTML to a file. The final command line script is textutil -convert html -strip -stdout terms.docx | sed 's/ class="[^"]*"//g' > output.html, which (1) converts the Word document to HTML with textutil, (2) strips out the class names that textutil adds to each tag with sed, and (3) saves the cleaned HTML to a file. From there, we can simply paste the HTML into our terms and conditions component in our SPA, style it, and call it a day.

Conclusion

If a development task is manual, repetitive, time-consuming, and boring, that's a sign. As developers, we should hone a keen awareness of this feeling, which is usually a clear sign that it's time to automate the task and move on to more creative, higher-value problem solving. It's a unique privilege of being software engineers that we can (and should!) automate these annoying parts of our jobs. So, embrace your laziness, fellow devs! It's the virtuous thing to do.

TL;DR

Convert your Word document to clean HTML on macOS by running this command in your shell: textutil -convert html -strip -stdout terms.docx | sed 's/ class="[^"]*"//g' > output.html

Instantly Create Gmail Addresses for Testing with a Keyboard Shortcut on Mac OS

Josh Stillman — Mon, 08 Mar 2021 15:00:34 +0000

When developing new features, it can be useful to be able to create new, unique, working email addresses for testing. For example, say you want to repeatedly test a sign-up flow end-to-end, and you need a new, unique email address each time through the flow. Furthermore, you'd like to use a real, working email address to verify the behavior of a confirmation email that gets sent after signing up. How can we easily accomplish this?

If you have a Gmail or Google Workspace (formerly G Suite) email address, you're in luck! One of Gmail's coolest under-the-radar features is the ability to create task-specific email addresses by adding a plus sign followed by additional characters to your email address. So, if your email address is bob.smith@gmail.com, you can sign up with bob.smith+hello-world@gmail.com, and emails to bob.smith+hello-world@gmail.com will reach your inbox. This makes it easy to repeatedly create new, unique, working addresses for testing!

But if you do this a lot, it becomes mentally draining to keep track of which email addresses you've already used and come up with new addresses on the fly. You'll have to keep track of whether you've already used bob.smith+test15@gmail.com or should instead be on bob.smith+test16@gmail.com, for example. Instead, let's automate it!

With some built-in Mac OS tools, it's possible to automatically create new, unique Gmail addresses with a handy keyboard shortcut to speed up our smoke testing. First, we'll use the Mac OS Automator application to create a script to generate new email addresses using the current system time. Then we'll attach that script to a keyboard shortcut in our system keyboard preferences.

Automator Quick Action

Mac OS ships with the Automator program that allows you to create scripts using JavaScript or your shell, and these scripts can output text. In Automator, select New, then choose Quick Action.

On the left hand pane, select Utilities, then Run JavaScript or Run Shell Script.

On the top pane, select Workflow receives no input in any application. Also select the Output replaces selected text checkbox.

In the Run JavaScript dialog below, create a script that will interpolate the system time in milliseconds into your Gmail address after the + sign. Using ES6 template literals, you can have the function:

return `bob.smith+${new Date().getTime()}@gmail.com`;

This will return an email in the format bob.smith+1613507754883@gmail.com. Make sure not to check Show this action when the workflow runs, otherwise Automator will show a pop-up each time. Then, save and name your Automator action.

You can also use Automator to execute shell scripts, which opens up all kinds of possibilities. The equivalent shell script here (in seconds) would be printf "bob.smith+%s@gmail.com" $(date +%s).

Keyboard Shortcut

Next, open System Preferences, choose Keyboard, then Shortcuts. In the left-hand pane, choose Services. You should see the Quick Action you created with Automator as an option in the right-hand pane. Add a memorable keyboard shortcut, then click the checkbox to activate. (I chose Control + Option + Command + C, but you might want to investigate setting up a Hyper Key for things like this instead.)

Now, give your keyboard shortcut a try!

This is one simple example of how to use scripting, Mac OS Automator, and keyboard shortcut mappings to automate simple tasks and speed up your development experience. But these tools open up tons of other possibilities. Got any cool ideas of your own? Let us know in the comments!

Slackbot to the Rescue: Keeping GitLab Branches in Sync with a Slackbot

Josh Stillman — Thu, 28 Jan 2021 16:12:25 +0000

If you've ever had to deal with horrific merge conflicts from infrequently synced branches, I've felt your pain. Our team develops off of the Dev branch, promotes code to a QA branch for testing, and merges bug fixes into the QA branch before releasing to production. As we get closer to a release and do more work in QA, Dev can quickly get out of date. And it's easy to forget to down-merge QA to Dev when you're focused on getting a release out. But there's a price to pay: when it comes time to down-merge after the release, resolving the merge conflicts can become a tedious, time-consuming, and error-prone nightmare.

There's got to be a better way! To get us into the habit of down-merging QA to Dev every day, I wanted to make it as easy and frictionless as possible to keep the branches in sync. So I created a slackbot to open a new GitLab merge request ("MR") for us and post it to our team Slack channel each morning.

Each weekday morning, the slackbot checks whether QA and Dev are in sync in our three main repos. If so, it notifies us via Slack that the branches are in sync. If not, it opens a new MR from QA to Dev and posts the link to Slack. It lets us know if there are merge conflicts, and if so, it includes resolution instructions.

Avoiding Gnarly Merge Conflicts Through Continuous Integration

Ultimately, the slackbot is a tool to help us maintain good Continuous Integration ("CI") habits. CI, in its broadest definition, is the practice of frequently integrating code changes into a shared branch, kept in a working state, so that everyone on the team has the most recent code. With respect to branch management, as Martin Fowler puts it, "[t]he over-arching theme is that branches should be integrated frequently and efforts focused on a healthy mainline that can be deployed into production with minimal effort."

One of the biggest benefits of good CI habits is minimizing Git merge conflicts. Git is a fantastic tool for collaboration, and it can often integrate code changes automatically. But the less frequently branches are integrated and the further the code has diverged, the more likely it is that Git won't be able to merge the code without human intervention.

Small merge conflicts from recent changes aren't such a big deal. It's often clear which side of the conflict is correct, even to another developer on the team that didn't author the change. But infrequent integration can lead to "nasty" merge conflicts that quickly spiral out of control.

These "nasty" merges can "generate a considerable amount of work", since the developer resolving the conflicts must first get context on other developers' changes, then figure out how they should be synchronized, then ensure the code is still in working order. They're also dangerous, since human error in resolving complex conflicts can introduce hard-to-trace bugs.

CI is a Habit

To avoid these nasty merge conflicts, we need to get into the habit of frequently integrating our branches. Behavioral psychology teaches that habits are easiest to form when they are made easier. If we set out our running shoes ahead of time, there will be less friction in forming the habit of running. It's also easiest to form habits when they are preceded by a consistent cue to perform the behavior, at the same time and place. We can set an alarm on our phone for the same time every day, which will provide a consistent cue to put on our running shoes and hit the road.

The slackbot helps us form good CI habits by making integration as frictionless as possible. It creates a branch and MR for us and sends it right to where we're already looking. All we need to do is click and review. And integrating daily makes conflicts infrequent and easy to resolve. The slackbot also provides a strong, consistent cue at the same time and place: it sends the message at the same time every work day, right as we're sitting down to our desks.

Creating the Slackbot with Bash Scripting and the GitLab API

To create the slackbot, I used a Bash script, GitLab, and the GitLab API.

For my initial attempt, I decided to write a Bash shell script so I could use Git's CLI to diff QA and Dev and skip the GitLab API calls if the branches are already synced. (I used this command: DIFFCOUNT=$(git diff origin/dev..origin/qa | wc -m | tr -d " ")). That ended up being more trouble than it was worth, both because it's easier to manage multiple repos using only GitLab's API, and because GitLab's diffing algorithm is different than the diff command above, leading to unmergeable empty MRs. (See these issues.)

While I kept the script in Bash after refactoring away the use of the Git CLI, the script could easily be converted to Node or another language.

Bash Tooling

If you're relatively new to Bash scripting like me, I think you'll find that it becomes more straightforward once you get past the terse, outdated syntax and get some experience. But it does require a somewhat different way of thinking about how to solve problems than you might be used to.

Using some modern tooling in VS Code helps. I'd recommend installing the Bash IDE for syntax highlighting and code completion, Shellcheck for code formatting and linting, and Shellman for code snippets.

Explainshell is also helpful for breaking down dense and confusing lines of Bash code. Try pasting in some of the lines of code in this blog for more detailed explanations.

The Frontend Masters course Complete Intro to Linux and the Command-Line is an excellent place to start learning more. And if you'd like more depth, Learning the Bash Shell is a very thorough primer.

Once you've got your tooling set up, follow these steps.

1. Gather Credentials

Gitlab Token

To use the Gitlab API, you'll need to generate a personal access token. Follow the steps here, and make sure to choose the api scope. Save the token somewhere safe.

Gitlab Project ID(s)

For each repo in which you'd like to use the slackbot, navigate to the project homepage and copy down the "Project ID", located directly under the project name.

Slack App

You'll need to create a slack app by clicking "create app" at https://api.slack.com/apps. On your app's homepage, click "add features and functionality", then "incoming webhooks", then click activate.

Create two webhooks: one that sends you a DM for testing (search for your name), and the "production" webhook that posts to your team's Slack channel. Keep the webhook URLs secret and safe.

On your app's home page, you can add a name, image, and description that will be shown when the slackbot posts messages.

2. Setup a Slackbot Branch in one of your Repos

In one of the repos you want to manage with the slackbot, checkout a new branch (e.g., js/slackbot). You can have the slackbot create MRs for multiple repos from this single branch.

Create a .env file

Create a .env file (make sure it's git-ignored!) and paste in your GitLab variables and Slack webhook url for testing locally. (The variables will be set in the GitLab console for the deployed job). Make sure they are exported so they are available to subshells.

Because Bash lacks complex data structures, we'll use string matching to map each repo's display name to its GitLab ID in our environment variables. First, create a comma-separated string listing the single-word display names of the repos you will manage with the slackbot. For example, export REPO_DISPLAY_NAMES="Frontend, Services, Personalization". For each repo in that list, create a matching ID variable with the repo name in all caps followed by _ID, such as FRONTEND_ID, SERVICES_ID, and PERSONALIZATION_ID. The display name variable in the list and the ID variable name before the _ID must match exactly (aside from case). Here's an example:

export REPO_DISPLAY_NAMES="Frontend, Services, Personalization"

export FRONTEND_ID=
export SERVICES_ID=
export PERSONALIZATION_ID=

export GITLAB_TOKEN=
export SLACK_URL=

The benefit of this approach is that adding another repo for the slackbot to manage is as easy as adding another repo name to the list and another repo ID variable.

Create the script file

Create a file called slackbot.sh in your repo, perhaps in a ci folder. Open it and add the Bash shebang on the first line: #!/usr/bin/env bash

Make the script executable. On the command line, run chmod u+x slackbot.sh

Edit your gitlab-ci.yml

Since the slackbot doesn't really fit into the traditional build, test, and deploy stages, add a slackbot stage to the stages key.

stages:
  - slackbot

And add a slackbot job that will execute the script.

slackbot:
  stage: slackbot
  only:
    - schedules
  script:
    - ci/slackbot.sh

You'll want to ensure that only the slackbot gets triggered when the scheduled job runs. For safety, I keep the slackbot in its own branch, the scheduled job targets that branch, and all the other jobs explicitly do not run on schedules. For each other job, you could add:

  except:
    - schedules

3. Create the Slackbot Bash Script

Our slackbot will use another dedicated branch for creating the MRs from QA to Dev -- slackbot/qa-to-dev. This will help later with merge conflicts.

Generally, our script will:

Read in the array of repos to be managed and call the handle_repo function for each repo.
For each repo, start with a clean slate by deleting the slackbot/qa-to-dev branch (if it exists) and creating a fresh slackbot/qa-to-dev branch off of QA.
Create an MR from that branch to Dev.
See if merge conflicts exists.
Send the link to the MR and any merge conflict info to Slack.

A. Load environment variables.

First, add the following code to load your .env when testing the script locally.

if [ -f "./.env" ]; then
    source ./.env
    echo "Importing env vars from .env file"
fi

We'll exit with an error code if the necessary variables aren't present.

if [ -z "$SLACK_URL" ] || [ -z "$GITLAB_TOKEN" ] || [ -z "$REPO_DISPLAY_NAMES" ];  then
  echo "Missing environment variables: SLACK_URL: $SLACK_URL, GITLAB_TOKEN: $GITLAB_TOKEN, REPO_DISPLAY_NAMES: $REPO_DISPLAY_NAMES"
  exit 1
fi

B. Call the handle_repo function for each repo.

First, transform the comma-separated list of repo names in REPO_DISPLAY_NAMES environment variable into an array:

IFS=', ' read -r -a REPO_ARRAY <<< "$REPO_DISPLAY_NAMES"

Then, for each repo, we'll call our handle_repo function by looping over the array of repo display names, capitalizing the name and adding _ID to get the name of that repo's ID variable, then passing in the repo display name and the repo ID as function arguments.

for REPO in "${REPO_ARRAY[@]}"; do
  REPO_ID="$(echo "$REPO" | tr '[:lower:]' '[:upper:]')_ID" # Capitalize repo display name and add "_ID" to get project id's variable name

  handle_repo "$REPO" "${!REPO_ID}" # Indirect variable. REPO_ID's value is another variable's name, such as FRONTEND_ID, and we access that second variable's value.
  # https://stackoverflow.com/questions/16553089/dynamic-variable-names-in-bash
done

C. Setup the function and variables.

Create a handle_repo function and setup the variables you'll need. Note that the slackbot branch name must be url-escaped for the GitLab API calls:

handle_repo () {

  #1. Set up variables.
  local REPO=$1 # first function argument
  local PROJECT_ID=$2 # second function argument
  local SOURCE_BRANCH_DISPLAY=${3:-"QA"} # default value of QA
  local TARGET_BRANCH_DISPLAY=${4:-"Dev"} # default value of Dev

  # Use exact (lowercase) branch name for Gitlab API calls
  local SOURCE_BRANCH=$(echo "$SOURCE_BRANCH_DISPLAY" | tr '[:upper:]' '[:lower:]' )
  local TARGET_BRANCH=$(echo "$TARGET_BRANCH_DISPLAY" | tr '[:upper:]' '[:lower:]' )

  local MR_TITLE="${SOURCE_BRANCH_DISPLAY} => ${TARGET_BRANCH_DISPLAY}"

  local SLACKBOT_BRANCH="slackbot%2fqa-to-dev" # URL escaped for API calls
  local SLACKBOT_BRANCH_DISPLAY="slackbot/qa-to-dev"

  if [ -z "$REPO" ] || [ -z "$SOURCE_BRANCH" ] || [ -z "$TARGET_BRANCH" ] || [ -z "$PROJECT_ID" ];  then
    echo "Missing environment variables: REPO: $REPO, SOURCE_BRANCH: $SOURCE_BRANCH, TARGET_BRANCH: $TARGET_BRANCH, PROJECT_ID: $PROJECT_ID"
    return 1
  fi

   echo "Running handle_repo for $REPO: From $SOURCE_BRANCH to $TARGET_BRANCH for project $PROJECT_ID"

D. Start with a clean slate.

So that we can follow the same steps every time, even if we forgot to merge in yesterday's MR, we'll delete the current slackbot/qa-to-dev branch if it exists. This (usually) will delete the old MR too. Then we'll create a new slackbot/qa-to-dev branch off of QA.

#2. Delete existing slackbot branch
DELETE_BRANCH_RESP=$(curl -X DELETE --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT_ID/repository/branches/$SLACKBOT_BRANCH")
echo "delete branch resp is $DELETE_BRANCH_RESP"

#3. Create remote slackbot branch off of source branch
CREATE_BRANCH_RESP=$(curl -X POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT_ID/repository/branches?branch=$SLACKBOT_BRANCH&ref=$SOURCE_BRANCH")
echo "create branch resp is $CREATE_BRANCH_RESP"

E. Create the MR and get MR info.

We'll create a new MR from slackbot/qa-to-dev to Dev using the GitLab API, and get the MR ID from the response.

#4. Create merge request to target branch
CREATE_MR_RESP=$(curl -X POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT_ID/merge_requests?source_branch=$SLACKBOT_BRANCH&target_branch=$TARGET_BRANCH&simple=true" --data-urlencode "title=$MR_TITLE")
echo "create MR resp is $CREATE_MR_RESP"

# Handle Gitlab occasionally retaining yesterday's unmerged MR after delete/create
ALREADY_EXISTS=$(echo "$CREATE_MR_RESP" | grep 'already exists')

# 5. Get MR ID
if [ -n "$ALREADY_EXISTS" ]; then
  MR_ID="${CREATE_MR_RESP//[^0-9]/}" # extract numbers from response string
  echo "MR already exists $MR_ID"
else
  MR_ID=$(echo "$CREATE_MR_RESP" | python3 -c "import sys, json; print(json.load(sys.stdin)['iid'])")
fi

Then, we'll get the MR's info from the GitLab API. I've found we need to sleep for 10 seconds before fetching the MR info to let GitLab finish calculating merge conflict info.

We'll use Python to parse the JSON response and extract the MR's URL, then we'll add the URL to our Slack message.

sleep 10  # Wait for GitLab to calculate merge conflicts info

MR_INFO_RESP=$(curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT_ID/merge_requests/$MR_ID")
echo "MR info resp is $MR_INFO_RESP"

# 7. Get MR URL
URL=$(echo "$MR_INFO_RESP" | python3 -c "import sys, json; print(json.load(sys.stdin)['web_url'])")

# 8. Set initial slack message
MSG="$REPO: $MR_TITLE: $URL"
echo "New MR: $MSG"

F. Report on merge conflicts.

The MR info API response tells us whether there are merge conflicts. If so, we'll append instructions for resolving them to our Slack message. Since we're using a separate branch, anyone on the team can just pull down slackbot/qa-to-dev, merge Dev into it locally, resolve the conflicts, then push back up. That way, no one has to create a new branch.

Note that we can use Slack text formatting and emojis in our message. 🎉

Also note that when using Python to parse JSON, true values become uppercase True, and null values become None.

# 9. Report on merge conflicts
CONFLICTS=$(echo "$MR_INFO_RESP" | python3 -c "import sys, json; print(json.load(sys.stdin)['has_conflicts'])")

if [ "$CONFLICTS" == 'True' ]; then
  MSG="$MSG. *Merge Conflicts!* 🙀 _Pull down $SLACKBOT_BRANCH_DISPLAY, merge ${TARGET_BRANCH} into it locally, resolve conflicts, then push back up._ 😸"
fi

G. Close empty MRs.

GitLab will still create an "empty" MR even if the branches are in sync, so we'll need to close the MR if it comes back without any changes. If so, we'll set our MSG variable to let the team know the branches are already synced for that repo.

# 10. If MR is empty, close it
CHANGES=$(echo "$MR_INFO_RESP" | python3 -c "import sys, json; print(json.load(sys.stdin)['changes_count'])")
echo "changes count is $CHANGES"

if [ "$CHANGES" == "None" ]; then
  echo "No changes, closing MR"
  MSG="$REPO: $SOURCE_BRANCH_DISPLAY is synced with $TARGET_BRANCH_DISPLAY! 🎉"
  curl -X PUT --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT_ID/merge_requests/$MR_ID?state_event=close"
  fi

H. Send to Slack!

Finally, we're ready to send our message to Slack. We'll simply send JSON to our webhook, with a key of text pointing to our MSG variable that we set above, escaping the interior quotation marks.

# 11. Send to Slack
curl -X POST -H 'Content-type: application/json' --data "{\"text\":\"$MSG\"}" "$SLACK_URL"

I. Test Locally

That's the script! We can try running it locally to send a test DM to ourselves and make sure everything works.

4. Setup the Scheduled Job

Almost there! All that's left is to configure the scheduled job on GitLab.

On the GitLab menu in your project, navigate to CI/CD -> Schedules, and click "Create New Schedule".

Set the cron signature and timezone. I set the job to run every weekday at 9am Eastern with this cron signature: 0 9 * * 1-5. (Checkout crontab.guru to test out cron signatures.)

Set the target branch to the branch in which you wrote the slackbot script. (E.g., js/slackbot, not the $SLACKBOT_BRANCH that the slackbot will use to create MRs!)

Enter in your environment variables, set the job to activated, and click save.

I'd recommend testing it out first by setting the SLACK_URL variable to send you a DM, and clicking the play button on the job. If all goes well, switch the SLACK_URL to send the message to your team's channel.

And TADA! You've got yourself a slackbot.

Further Enhancements

The current version of the script assumes that each repo has the same source/target branch names (qa and dev), but that could be made configurable for different branch names or for syncing multiple branches within a repo. And if the script's complexity grows much further in future editions, it's probably time to move over to Node.

Conclusion

And that's how you set up your very own slackbot! It'll help you and your team establish good branch integration habits and avoid frustrating, time-consuming, and dangerous merge conflicts. It's also a fun way to get experience with Bash scripting, GitLab, and Slack Apps. These building blocks open up lots of possibilities for automating away pain-points and improving your team's development experience.

The coolest thing about learning skills like these is that you're no longer limited to the productivity tools handed down to you. You're empowered to create tools that fit your team's unique needs and improve your team's experience.

Happy coding!

Full Slackbot Script

Also available at this gist.

#!/usr/bin/env bash

# Function that will be called for each repo.  Script exectuion begins below.
handle_repo () {

  #1. Set up variables.
  local REPO=$1 # first function argument
  local PROJECT_ID=$2 # second function argument
  local SOURCE_BRANCH_DISPLAY=${3:-"QA"} # default value
  local TARGET_BRANCH_DISPLAY=${4:-"Dev"} # default value

  # Use exact (lowercase) branch name for Gitlab API calls
  local SOURCE_BRANCH=$(echo "$SOURCE_BRANCH_DISPLAY" | tr '[:upper:]' '[:lower:]' )
  local TARGET_BRANCH=$(echo "$TARGET_BRANCH_DISPLAY" | tr '[:upper:]' '[:lower:]' )

  local MR_TITLE="${SOURCE_BRANCH_DISPLAY} => ${TARGET_BRANCH_DISPLAY}"

  local SLACKBOT_BRANCH="slackbot%2fqa-to-dev" # URL escaped for API calls
  local SLACKBOT_BRANCH_DISPLAY="slackbot/qa-to-dev"

  if [ -z "$REPO" ] || [ -z "$SOURCE_BRANCH" ] || [ -z "$TARGET_BRANCH" ] || [ -z "$PROJECT_ID" ];  then
    echo "Missing environment variables: REPO: $REPO, SOURCE_BRANCH: $SOURCE_BRANCH, TARGET_BRANCH: $TARGET_BRANCH, PROJECT_ID: $PROJECT_ID"
    return 1
  fi

  echo "Running handle_repo for $REPO: From $SOURCE_BRANCH to $TARGET_BRANCH for project $PROJECT_ID"

  #2. Delete existing slackbot branch
  DELETE_BRANCH_RESP=$(curl -X DELETE --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT_ID/repository/branches/$SLACKBOT_BRANCH")
  echo "delete branch resp is $DELETE_BRANCH_RESP"

  #3. Create remote slackbot branch off of source branch
  CREATE_BRANCH_RESP=$(curl -X POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT_ID/repository/branches?branch=$SLACKBOT_BRANCH&ref=$SOURCE_BRANCH")
  echo "create branch resp is $CREATE_BRANCH_RESP"

  #4. Create merge request to target branch
  CREATE_MR_RESP=$(curl -X POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT_ID/merge_requests?source_branch=$SLACKBOT_BRANCH&target_branch=$TARGET_BRANCH&simple=true" --data-urlencode "title=$MR_TITLE")
  echo "create MR resp is $CREATE_MR_RESP"

  # Handle Gitlab occasionally retaining yesterday's unmerged MR after delete/create
  ALREADY_EXISTS=$(echo "$CREATE_MR_RESP" | grep 'already exists')

  # 5. Get MR ID
  if [ -n "$ALREADY_EXISTS" ]; then
    MR_ID="${CREATE_MR_RESP//[^0-9]/}" # extract numbers from response string
    echo "MR already exists $MR_ID"
  else
    MR_ID=$(echo "$CREATE_MR_RESP" | python3 -c "import sys, json; print(json.load(sys.stdin)['iid'])")
  fi

  # 6. Get MR info from GitLab API
  sleep 10  # Wait for GitLab to calculate merge conflicts info
  MR_INFO_RESP=$(curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT_ID/merge_requests/$MR_ID")
  echo "MR info resp is $MR_INFO_RESP"

  # 7. Get MR URL
  URL=$(echo "$MR_INFO_RESP" | python3 -c "import sys, json; print(json.load(sys.stdin)['web_url'])")

  # 8. Set initial slack message
  MSG="$REPO: $MR_TITLE: $URL"
  echo "New MR: $MSG"

  # 9. Report on merge conflicts
  CONFLICTS=$(echo "$MR_INFO_RESP" | python3 -c "import sys, json; print(json.load(sys.stdin)['has_conflicts'])")

  if [ "$CONFLICTS" == 'True' ]; then
    MSG="$MSG. *Merge Conflicts!* 🙀 _Pull down $SLACKBOT_BRANCH_DISPLAY, merge ${TARGET_BRANCH} into it locally, resolve conflicts, then push back up._ 😸"
  fi

  # 10. If MR is empty, close it
  CHANGES=$(echo "$MR_INFO_RESP" | python3 -c "import sys, json; print(json.load(sys.stdin)['changes_count'])")
  echo "changes count is $CHANGES"

  if [ "$CHANGES" == "None" ]; then
    echo "No changes, closing MR"
    MSG="$REPO: $SOURCE_BRANCH_DISPLAY is synced with $TARGET_BRANCH_DISPLAY! 🎉"
    curl -X PUT --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT_ID/merge_requests/$MR_ID?state_event=close"
  fi

  # 11. Send to Slack
  curl -X POST -H 'Content-type: application/json' --data "{\"text\":\"$MSG\"}" "$SLACK_URL"
}

# Begin script execution here:

# Setup env vars.  Located in the Gitlab project settings console.  For running locally, use a .env file at project root and export the vars.
if [ -f "./.env" ]; then
    source ./.env
    echo "Importing env vars from .env file"
fi

if [ -z "$SLACK_URL" ] || [ -z "$GITLAB_TOKEN" ] || [ -z "$REPO_DISPLAY_NAMES" ];  then
  echo "Missing environment Variables: SLACK_URL: $SLACK_URL, GITLAB_TOKEN: $GITLAB_TOKEN, REPO_DISPLAY_NAMES: $REPO_DISPLAY_NAMES"
  exit 1
fi

# Create array from comma-separated string of repo names
IFS=', ' read -r -a REPO_ARRAY <<< "$REPO_DISPLAY_NAMES"

# Call function for each repo.  For each repo name in the REPO_ARRAY, a matching variable with the GitLab project ID named with the repo name in all caps followed by _ID is required
for REPO in "${REPO_ARRAY[@]}"; do
  REPO_ID="$(echo "$REPO" | tr '[:lower:]' '[:upper:]')_ID" # Capitalize repo display name and add "_ID" to get project id's variable name

  handle_repo "$REPO" "${!REPO_ID}" # Indirect variable. REPO_ID's value is another variable's name, such as FRONTEND_ID, and we access that second variable's value.
  # https://stackoverflow.com/questions/16553089/dynamic-variable-names-in-bash
done

(First published on Medium)