Forem: Scotty

Powerful regex for the practical dev #2

Scotty — Sat, 12 Sep 2020 00:00:00 +0000

As a developer you’re constantly working with large amounts of text like source code, logs, and data files. Often you need to extract, replace, or manipulate that text and regex can help you.

Today I present variants of a real case study where I personally used regex. These examples build upon my last post so check that out too.

Powerful regex for the practical dev

Scotty ・ Sep 1 ・ 6 min read

#regex #search

Case study example

This is a snippet of this csv file which you can also play with here.

reddit_id,colorblind_comment,score,title,url,created
...
8cwcbu,False,101457,"Cause of Death - Reality vs. Google vs. Media [OC]",https://i.imgur.com/GtIzEok.gif,1523970172
8bzdr8,False,99626,"Gaze and foot placement when walking over rough terrain (article link in comments) [OC]",https://v.redd.it/h0f0m4v5nor01,1523628194
fpga3f,False,99488,"[OC] To show just how insane this week's unemployment numbers are, I animated initial unemployment insurance claims from 1967 until now. These numbers are just astonishing.",https://i.redd.it/tch0t0is32p41.gif,1585245693
i2vx78,True,98638,"The environmental impact of Beyond Meat and a beef patty [OC]",https://i.redd.it/jskjkodg3se51.png,1596456703
fxoxti,False,98067,"Coronavirus Deaths vs Other Epidemics From Day of First Death (Since 2000) [OC]",https://v.redd.it/yemjrb1p9rr41,1586422082
...

(Top 100 posts from r/dataisbeautiful on Reddit where 'colorblind' is mentioned in the comments)

Text extraction: Extract urls from data

There are a ton of things you could extract here but I’ll go with this:

Extract image urls from posts with colorblind comments

You could use excel or write a script here but this is why I would use regex:

Quick and easily done from your IDE
Not limited to CSVs - this could be source code which isn’t so easy via excel/scripts
Took me under a minute to do (with prior regex knowledge)

Step #1: Select all lines where the colorblind column is True

.*,True,.*

True match the word ‘True’
, match commas (so we don’t match ‘True’ in the title text)
.* match everything before and after

Step #2: Limit to lines that have an image url

.*,True,.*http.*\.(png|jpg).*

http.* match start of url (not necessary but needed for the next step)
\.(png|jpg) match .png or .jpg extensions
http.*\.(png|jpg) combine to match urls with an image extension
.*,True,.* + http.*\.(png|jpg) + .* combine step 1 regex, the image url, and everything after. Now it matches lines with colorblind comments AND image urls

Step #3: Match just the url:

(?<=.*,True,.*,)http.*\.(png|jpg)(?=.*)

Lets break up our regex so far into parts:

We want to isolate part_2 so must exclude part_1 and part_3 like this:

(?<= ... ) ... exclude stuff before a match
... (?= ... ) exclude stuff after a match
(?<= part_1 ) + part_2 + (?= part_3 ) exclude part_1 and part_3
(?<=.*,True,.*)http.*\.(png|jpg)(?=.*) substitute in our actual regex

Step #4: Extract the text

A modern IDE should now let you select your matches. In VS Code you can select and copy your matches like this:

Alt + Enter to select matches
Ctrl + c to copy matches
Ctrl + v to paste matches (in a separate file)

Urls successfully extracted! You could take this further by running this over your entire repo using global search (Alt + Shift + f). Show me an excel script that does that!

Text replace: Reformat url structure

Sometimes we don’t want to extract text but rather update it. For this example we’ll update all image urls in the file like so:

from this: https://i.redd.it/jskjkodg3se51.png
to this: https://i.redd.it/png/jskjkodg3se51

Step #1: Match image urls (with ID match)

http.*/.*\.(png|jpg)

This the url regex we made before but with an image ID match that we’ll use in the next step
http.* match start of url and everything after
/.* match forward slash and everything after (matches the image ID)
\.(png|jpg) match .png or .jpg extensions

Step #2: Create regex groups

(http.*)/(.*)\.(png|jpg)

Everything wrapped with () creates a regex group which we can reference in our replace command.

Given the url https://i.redd.it/jskjkodg3se51.png we must create groups for the following parts to transform them as desired:

https://i.redd.it/ the domain part
jskjkodg3se51 the image ID part
png the image extension part

Let’s group those parts of our regex:

(http.*) groups the domain part
/(.*) groups the image ID part (note the slash is excluded)
\.(png|jpg) groups the image extension part (note this was already grouped and that . is excluded)

Step #3: Replace

(http.*)/(.*)\.(png|jpg) + $1/$3/$2

Open the replace UI (Ctrl + h in most IDEs)

We can now reference groups in replace commands as follows:

$0 is the entire match
$1, $2, $3 is the first, second, third group etc

Our search and replace commands will therefore look like this:

Search: (http.*)/(.*)\.(png|jpg)
Replace: $1/$3/$2

That regex replace will change urls like so:

https://i.redd.it/jskjkodg3se51.png -> https://i.redd.it/png/jskjkodg3se51
https://i.redd.it/mkdikcce8gh31.jpg -> https://i.redd.it/jpg/mkdikcce8gh31

Nice huh? You can also do this globally via Alt + shift + h to search and replace over your whole repo. Very useful.

Powerful stuff eh? Once you learn how to build up a regex like this (hint: practice makes perfect) you can do this kind of stuff in seconds. Searching, extracting, and transforming text will become second nature.

Well that’s it for now! I’ve got 2 more sections of this case study ready to go but this article was getting long. So next week we’ll learn how to use multi-cursors to augment regex even further.

It’s seriously powerful stuff!

Found this useful? Tweet about it or follow me on Twitter. Until next time 👋

Powerful regex for the practical dev

Scotty — Tue, 01 Sep 2020 18:07:08 +0000

Summary

Regex is search on steroids. This post demonstrates how to create powerful searches by example.

Examples start basic and build up to more complex expressions. They are designed for devs who search via their IDE.

We'll use this text throughout (demo):

Alabama       (AL)  Birmingham  (Dec 14, 1819)  4,903,185
Hawaii*       (HI)  Honolulu    (Aug 21, 1959)  1,415,872
Michigan      (MI)  Detroit     (Jan 26, 1837)  9,986,857
North Dakota  (ND)  Fargo       (Nov  2, 1889)    762,062
Wyoming*      (WY)  Cheyenne    (Jul 10, 1890)    578,759

(state, id, largest city, founding date, population)

How to use this guide

Browse quickly by skimming the picture/title. There are many examples.
If you're a beginner start at the top
Find more advanced/interesting examples further down
Examples next to each other are related
Experiment by testing the examples yourself
Find a cheat sheet at the bottom

Setup

Follow along here or via your IDE:

Open a modern IDE (I use VSCode)
Paste the example
Open search (ctrl + f or cmd + f)
Enable regex (usually a .* icon)

Series

This is the first part in a short series I'll be making. Following guides will cover real case studies and more useful concepts like regex replace.

Follow the newsletter or hit me up on Twitter for more.

Basic matches

Letters [a-zA-Z]

[a-z] lowercase letters
[A-Z] uppercase letters
Casing only matters when the 'match case' (Aa) option is enabled in VSCode
102 matches found because it's matching a-z characters

Words [a-zA-Z]+

[a-zA-Z] letters
+ repeats the match for consecutive characters
21 matches found because it's matching a-z words

Specific words (Jan|Jul|Dec)

(Jan|Jul|Dec) matches Jan, Jul, or Dec specifically

2 numbers [0-9]{2}

[0-9] numbers
{2} match twice
Note long numbers contain multiple matches

4 numbers [0-9]{4}

[0-9] numbers
{4} match 4 times

2-3 letters [a-z]{2,3}

[a-z] letters
{2,3} match between 2 to 3 times (inclusive)
Note long words contain multiple matches

6+ letters [a-z]{6,}

[a-z] letters
{6,} match 6 or more times (inclusive)

3 letters/numbers \w{3}

\w letters and numbers (see special chars)
{3} match 3 times
Note long words contain multiple matches

3 whole letters/numbers \b\w{3}\b

\w{3} match 3 letters and numbers
\b word boundaries (see special chars)
Note long words don't contain multiple matches

3 whole letter words \b[a-z]{3}\b

[a-z]{3} match 3 letters
\b word boundaries

Two words [a-zA-Z]+\s[a-zA-Z]+

Looks scarier than it is. The form is word space word
[a-zA-Z]+ word
\s space (see special chars)

One or two words [a-zA-Z]+(\s[a-zA-Z]+)?

Looks scarier than it is. The form is word (space word)?
[a-zA-Z]+ word
\s space
( ... )? optional
Note North Dakota is considered one match now

Wildcards

Everything in brackets (greedy) $.*$

$ and $ match brackets (see special chars)
.* greedy wildcard
This greedy wildcard will match up to the last ) bracket

Everything in brackets (non-greedy) $.*?$

$ and $ match brackets
.*? non-greedy wildcard
This non-greedy wildcard will match up to the first ) bracket

Lines with the * character ^.*\*.*$

^ and $ match the start/end of the line (optional)
.* wildcard
\* the star * character (see special chars)

Lines without the * character ^[^\*]+$

^ and $ match the start/end of the line
[^ ... ] matches anything not in the brackets
- \* the star * character
- [^\*] matches anything not a * character
+ repeats the match for consecutive characters

All lines with the e character ^.*[e].*$

^ and $ match the start/end of the line
.* wildcard
[e] the letter e

All lines without the e character ^[^e]+$

^ and $ match the start/end of the line
[^ ... ] matches anything not in the brackets
- [^e] matches anything not an e character
+ repeats the match for consecutive characters

Brackets starting with certain words $(Jan|Jul|Dec).*$

$ and $ match brackets
(Jan|Jul|Dec) matches Jan, Jul, or Dec words
.* wildcard

Mixed matches

The short date in brackets [a-z]{3}\s+[0-9]+

[a-z]{3} 3 letters exactly
\s+ one or more spaces
[0-9]+ one or more numbers

The date in brackets [a-z]{3}\s+[0-9]+,\s[0-9]+

Looks scarier than it is. The form is word number, number
[a-z]{3} 3 letters exactly
\s+ one or more spaces
, comma
[0-9]+ one or more numbers

Words with m (in the middle) [a-z]+[m][a-z]+

[a-z]+ one or more letters
[m] the letter m
Note this doesn't match Michigan because m is at the start of the word

Words with m (anywhere) ([a-z]+)?[m]([a-z]+)?

Looks scarier than it is. The form is (word)? m (word)?
( ... )? optional
- [a-z]+ a word
- ([a-z]+)? an optional word
[m] the letter m
Note m can be anywhere in the word so Michigan is matched now

Exclusive matches

Match expressions but exclude them from the result. Officially known as 'look arounds'.

Word in brackets (inclusive) $[a-z]+$

Note the word is matched with the brackets
$ and $ match brackets
[a-z]+ a word

Word in brackets (exclusive) (?<=$)[a-z]+(?=$)

Note the word is matched without the brackets
[a-z]+ a word
(?<= ... ) starts a match but excludes it from the result
- \( the bracket ( character
- (?<=\() matches from bracket ( without including it
(?= ... ) ends a match but excludes it from the result
- \) the bracket ) character
- (?=\)) matches up to bracket ) without including it

Everything in brackets (exclusive) (?<=$).*?(?=$)

(?<=\() matches from bracket ( without including it
.*? non-greedy wildcard
(?=\)) matches up to bracket ) without including it

Everything in brackets on lines with * (exclusive)

(?<=\*.*$).*?(?=$)

(?<= ... ) starts a match but excludes it
- \* the star * character
- .* wildcard
- \( the bracket ( character
- (?<=\*.*\() wildcard from * to ( without including them
.*? non-greedy wildcard
(?=)) matches up to ) without including it

Everything up to * (exclusive) ^.*(?=\*)

^ start of a line
.* wildcard
(?=\*) matches up to * without including it

Cheat sheet

. ^ $ * + ? ( ) [ { \ | reserved characters

Escape with \
- (abc) matches abc (in a regex group)
- $abc$ matches (abc) (with brackets)

[a-zA-Z] letters (case-sensitive)

[0-9] or \d match numbers

[a-c1-3#] matches characters a b c 1 2 3 #

.* greedy wildcard. .*? non-greedy wildcard.

^ start of line. $ end of line.

\s space. \t tab. \n new line.

\w letters and numbers. \W not letters and numbers.

\b word break. \B not word break.

+ repeat matches

{3} repeat match exactly thrice

{1,3} repeat match 1, 2, or 3 times

{3,} repeat match 3+ times

[^ ... ] match all but given characters

(?<= ... ) start match with given characters and exclude them (look behind)

(?= ... ) end match with given characters and exclude them (look ahead)