Forem: Anthony Slater

Setting up MongoDB

Anthony Slater — Mon, 01 Feb 2021 22:03:25 +0000

It's been a while since I fired up MongoDB, so I opened up a terminal and typed mongo thinking everything would be fine. Instead macOS Catalina gave me some grief about the developer being unrecognized...

The solution: Homebrew

>> brew tap mongodb/brew
>> brew install mongodb-community

Next I added the /data/db folder in System/Volumes/Data

>> sudo mkdir -p /System/Volumes/Data/data/db
>> sudo chown -R `id -un` /System/Volumes/Data/data/db

The first command creates the folders and the second changes its permissions.

Finally

Here are the commands to start and stop MongoDB as a background service:

>> brew services run mongodb-community
>> brew services stop mongodb-community

Unless you love typing things, edit your .bash_profile and set some aliases for the above commands.

Once MongoDB is running, you can type mongo in the terminal to start the mongo shell or run a Node.js program.

var MongoClient = require('mongodb').MongoClient;

// Connect to the db
(async () => {
  const client = await MongoClient.connect('mongodb://localhost:27017', {useUnifiedTopology: true });
  const dbs = await client.db().admin().listDatabases();
  console.log(dbs)
  client.close();
})();

Gustavo looks through a Telescope

Anthony Slater — Mon, 16 Nov 2020 16:00:22 +0000

I'm going to be integrating Gustavo, a HTTP status checker, with Telescope

Seneca-CDOT / telescope

A tool for tracking blogs in orbit around Seneca's open source involvement

Telescope uses many technologies, so it's best to get started by reading through and following all the steps in its Environment Setup Documentation.
Since I already had npm installed, I used brew to install the two other necessary technologies: redis and elasticsearch

>>> brew install npm
>>> brew install redis
>>> brew tap elastic/tap
>>> brew install elastic/tap/elasticsearch-full

Now you can open three terminal windows to get your local Telescope server running:

>>> redis-server
>>> elasticsearch
>>> npm start

If everything is working correctly, opening a browser to http://localhost:3000/posts should display an array of 10 json objects.

[{"id":"979c6d3eb5","url":"/posts/979c6d3eb5"},
{"id":"f1f56132fd","url":"/posts/f1f56132fd"},
{"id":"6413f59523","url":"/posts/6413f59523"},
{"id":"a0d10ce9e7","url":"/posts/a0d10ce9e7"},
{"id":"476491f37d","url":"/posts/476491f37d"},
{"id":"d5eed3b8ec","url":"/posts/d5eed3b8ec"},
{"id":"b1f2991c27","url":"/posts/b1f2991c27"},
{"id":"6fc08da083","url":"/posts/6fc08da083"},
{"id":"1aca420d0f","url":"/posts/1aca420d0f"},
{"id":"6ba1ae431a","url":"/posts/6ba1ae431a"}]

Next I wanted to update Gustavo to check the HTTP status of the above urls. Gustavo works by searching through a file for any string that starts with HTTP:// or HTTPS:// and creates a list of all matches. This list is then processed. I knew the program would work, if I could produce a list of fully formed URLs from the above json.

I updated args.py to handle additional flags -t or --telescope.

parser.add_argument('-f', '--file', action='store', dest='source', default='', help='location of source file')
parser.add_argument('-t', '--telescope', action='store_const', dest='source', const='TELESCOPE', help='check recent posts indexed by Telescope')

Instead of specifying a file path, using the -t or --telescope flags will save a string TELESCOPE to the source variable. The program runs normally until the get_list() function is called. At this point, if the value of the source variable equals TELESCOPE, the following code executes.

posts = requests.get('http://localhost:3000/posts')
urls = re.findall('/posts/[a-zA-Z0-9]{10}', posts.text)
return ['http://localhost:3000' + url for url in urls]

What does it do?

First a query is made to the server's REST API for posts.
Next a list of URLs if created by using a regular expression to find all unique 10-digit IDs.
Using a list comprehension, the domain is appended to each URL.

Now heading back to the terminal, running the command

>>> python gus.py -t

produces the following output:

[GOOD] [200] http://localhost:3000/posts/78e5ab8438
[GOOD] [200] http://localhost:3000/posts/2da60bf766
[GOOD] [200] http://localhost:3000/posts/4a8a76df4a
[GOOD] [200] http://localhost:3000/posts/e5f703c004
[GOOD] [200] http://localhost:3000/posts/168346fd63
[GOOD] [200] http://localhost:3000/posts/db7e046128
[GOOD] [200] http://localhost:3000/posts/f87957048e
[GOOD] [200] http://localhost:3000/posts/35b33ca432
[GOOD] [200] http://localhost:3000/posts/3555e6f683
[GOOD] [200] http://localhost:3000/posts/868b94d523

The complete summary of changes I made can be seen below:

Concurrent Futures and Showing Progress

Anthony Slater — Fri, 13 Nov 2020 23:32:11 +0000

I made a few updates to Gustavo: one major; one minor; and some refactoring.

slaterslater / gustavo

Print a colourized report of all HTTP urls in a file

The main issue I wanted to address was performance. I found that Gus took far too long to make HTTP requests compared to other similar tools. I didn't implement a time object to get precise measurements, but it felt like a list of 800 URLs could take anywhere from 5-8 minutes to process. Gus would check a list of URLs one after the other and I wanted them checked (practically) all at once.

This project is one of my first times working with Python and I was really impressed with how easy it was to implement threading. After importing concurrent.futures I found that using a list comprehension really made for an elegant solution.

with concurrent.futures.ThreadPoolExecutor() as executor:
  results = [executor.submit(get_status, url) for url in urlist]

That's all it took to exponentially improve the performance. The first line creates a ThreadPoolExecutor which creates, starts and joins threads. The second line is a list comprehension which creates a list of threads executing get_status() for each url in the list.

During the processing, the iterative version of Gus would update the console with a print statement ie: "Checking URL 123 of 456." This didn't really make much sense in the threaded version of Gus as it would print "Checking URL 456 of 456." in a fraction of a second and then waits for the get_status() functions to complete. I thought a progress bar would be a nice touch.

verigak / progress

Easy to use progress bars for Python

In the end I used a Spinner which advances the animation each time a thread completes. I noticed that once the Spinner finishes, it would leave itself on the console line and the program would continue. I read through the source code and found that the Spinner class had a writeln() which takes a string. Thank you very much Stack Overflow for answering the question: Is there go up line character?

The updated function looks like this:

def process_list(urlist, wanted, output):
  processed = []
  formatted = out.rtf_format if output  == 'rtf' else out.json_format if output  == 'json' else out.std_format
  spinner = Spinner('Checking URLs ')
  with concurrent.futures.ThreadPoolExecutor() as executor:
    results = [executor.submit(get_status, url) for url in urlist]
    for connection in concurrent.futures.as_completed(results):
      status = connection.result()
      if status['desc'] in wanted:
        processed.append(formatted(status))
      spinner.next()  
  spinner.writeln("\033[F")  # move cursor to the beginning of previous line
  spinner.finish()
  return processed

The remaining changes I made were to refactor gus.py into smaller related files:

File	Purpose
gus.py	creates a Gus class
const.py	any constant values needed for processing
args.py	handles all command line argument parsing
urls.py	all HTTP checking functions
out.py	output formatting functions

Using Numpy to build WAV files

Anthony Slater — Sun, 01 Nov 2020 23:26:01 +0000

I found a fun little tool that converts text into Morse Code with audio.

nimit2801 / MorseCodeMessenger

This Project will currently let you convert your text into Encrypted Morse Code with audio!

It works by first declaring a dictionary of letters, numbers and symbols with their corresponding definitions of dots and dashes. The program asks the user for input; iterates through the provided string one character at a time; looks up the character's definition; and saves the result to a new encoded string. The program includes two wav files: a dot and a dash. The program iterates through the saved encoded string of dots and dashes and plays the appropriate wav file. So it can encode text into sound, but the issue was can it decode sound into text?

I have a little bit of experience making command line tools, but I have never worked with sound before. My plan to tackle this issue was:

figure out how to read a wav file
create a wav file from multiple smaller ones
figure out how to find patterns in a wav file

After a little bit of research I found that a straightforward way to read a wav file was to use scipy.io.wavfile.read. The method takes a file location to a wav file and returns a sample rate as an integer and a numpy array. The opposite method scipy.io.wavfile.write takes a file location, a sample rate and a numpy array and creates a wav file. After playing around with these functions for a while, I was able to read the dot and dash wav files and create a large numpy array by appending copies of the dot and dash arrays.

dt = np.dtype('uint8')
blank = np.array([0] * 800, dt)   # 0.1s
padding = np.array([0] * 240, dt) # 0.03s
samplerate, dot = wavfile.read(DOT)
samplerate, dash = wavfile.read(DASH)
dot = np.append(dot, padding.copy())
dash = np.append(dash, padding.copy())
wav_data = np.empty((0,0), dt)
for tune in self.encoded:
  np_data = dot.copy() if tune == '.' else dash.copy() if tune == '-' else blank.copy()
  wav_data = np.append(wav_data.copy(), np_data) 
location = f'{self.filename}.wav'
wavfile.write(location, samplerate, wav_data)

The last thing I needed the tool to do was read a wav file and recognize patterns, specifically a dot, a dash, or blank silence. Perhaps there is a better method, but I decided that I would have the program read the wav file and look for specific values in the returned array at specific locations. This isn't a particularly elegant solution as it mainly involved checking an arbitrary location.

samplerate, wav = wavfile.read(f'{self.filename}.wav')
end = wav.shape[0]
self.encoded = ''
ptr = 0
while ptr < end: 
  if wav[ptr] == 0:
    ptr += 800      # size of blank array
    self.encoded += ' '
  elif wav[ptr + 800] == 128:
    ptr += 1120     # size of dot array
    self.encoded += '.'
  else:
    ptr += 2080     # size of dash array
    self.encoded += '-'

The result is a string of dots, dashes and spaces which could be passed to the original functions of the tool.
The last thing I wanted to do, was give the user a way to specify which functions to call from the command line. So I turned to argparse and implemented the following flags:

Optional flags	Purpose	Example
`-w` `--wav`	Create a Morse Code audio file	At the prompts, enter a message Hello World and a name hi. An audio file will be created and saved as `hi.wav`
`-p` `--play`	Play a Morse Code audio file	At the prompt, enter hi and if `hi.wav` exists, the audio will be played
`-r` `--read`	Decipher a Morse Code audio file	At the prompt, enter hi and if `hi.wav` exists, the text Hello World will be displayed

This was a fun little project and I was really happy to have my PR merged.

Adding Form Validation to Eztrackr

Anthony Slater — Sun, 01 Nov 2020 19:14:31 +0000

I use Trello - maybe you do too. I came across Eztrackr, a chrome extension to save notes to Trello. I noticed that Eztrackr had an open issue on github to
validate the input form before sending an API call. The first thing I did was install the extension and follow the documented steps to reproduce the bug.

With all fields filled out, the tool works great and your note will appear as a new card in your Trello board. If the user doesn't select a board, the post fails, all entered data is lost and must be re-entered. What I needed to do was run a check to make sure a board was selected before making the API call. An error message should be displayed with a close button, so that all form data fields remain untouched. All the user should have to do is fix the error by selecting the desired Trello board, and resubmit.

I walked through popup.js and found the API call and since all data fields could potentially be empty except for the idList, I just added a simple condition before its execution.

if(idList != 'Choose list'){
  Trello.post(`/cards?key=${APP_KEY}&token=${token}&idList=${idList}&name=${data_company}&desc=${description}`)

If the condition fails, an error message should prompt the user to select a Trello board. I noticed that popup.html contained two hidden divs that could be conditionally shown for (un)successful API calls and display a hard-coded message. Since I was attempting to fix a bug, I knew I needed show an error message. Instead of adding a third div, I decided I would modify the existing one.

<!-- failed posted alert -->
<div class="alert alert-danger" id="post_fail" role="alert" style="display: none;">
  <h4 class="alert-heading">Error</h4>
  <p>There was an issue posting this card to Trello. Please try again.</p>
</div>

I thought what the most efficient solution would be a generic error that could receive text from javascript instead of hard coded HTML. I added a button and the new code looked like this:

<!-- alert box -->
<div class="alert alert-danger" id="alert_box" role="alert" style="display: none;">
  <h4 class="alert-heading">Error</h4>
  <p id="alert_msg"></p>
  <button class="btn btn-danger btn-sm" id="close_alert">Close</button>
</div>

I went back to the popup.js added a function that would display #alert_box and pass text into #alert_msg. I updated the existing code to call this function if the Trello board isn't selected or after a failed API call.

After a few tests to confirm everything was working as intended, I made a pull request. Evidently the work checked all the necessary boxes, as it was merged the next day.

Fixes #28: Form validation #29

slaterslater posted on Oct 21, 2020

alert box displays if Trello list not chosen
close button added to alert box
choosing a Trello list will close alert box

View on GitHub

Contributing to a RPG with CSV

Anthony Slater — Thu, 22 Oct 2020 23:56:34 +0000

Recent events << looks over at covid19 >> have forced many of us to isolate indoors. My own DnD campaign has been slowly advancing through Zoom for the past six months. So this year, I figured I would start my Hacktoberfest by making a perception check (with advantage) and looking for game-related issues. I pretty quickly came across Mimic Tools:

Mimic-Tools / name-generation

Generation of character names

The issue they wanted addressed was how to display a report of tabular data from their github repo. The script region-checker.py could produce the desired report as HTML, but if you click on an HTML file in github... you see HTML. The solution was CSV. If you click on a CSV file in github, it's displayed as a nice, easy-to-read, zebra-striped table.

The region-checker.py already includes functions to iterate through all the important objects and builds strings of HTML where appropriate. I thought it would be easy enough to make a copy of said function and swap HTML syntax with CSV... But, as it turns out, CSV doesn't mean a string with commas in it; it means values separated by commas. After a little research, I found the python csv library and was almost ready to go.

If a table cell required a positive value, the ✓ character u'\u2713' was written. An X was written for negative values. One of the things requested was a cross symbol instead of a literal X.

I browsed through the Unicode Block “Dingbats” and it turns out the
✕ u'\u2715' and ✓ u'\u2713' are effectively neighbours and should work well together.

I set these variables at the top of the document and I think if I were to do this over again I would move them down closer to where the reports are being built. There is also some code repetition during the loop of dictionary items.

html_output = """<html><table border="1"><tr><th>Region</th>"""
csv_output = [['Region'] +  report_headers]
for items in report_headers:
    html_output += "<th>{}</th>".format(items)
html_output += """</tr>"""
for report_item in report_dictionary:
    region = splitext(report_item)[0].capitalize()
    csv_row = [region]
    html_output += "<tr><td>{}</td>".format(region)
    for location in report_dictionary[report_item]: 
        if report_dictionary[report_item][location]:
            html_output += "<td>{}</td>".format(check)
            csv_row += [check]
        else:
            html_output += "<td>{}</td>".format(cross)
            csv_row += [cross]    
    csv_output.append(csv_row)        
    html_output += "</tr>"
html_output += "</table></html>"

csv_file = open(csv_name, 'w', newline='', encoding='utf-8')
csv_writer = csv.writer(csv_file)
csv_writer.writerows(csv_output)
csv_file.close()

I didn't want to disturb the existing code any more than was necessary, but I think I should have modified the loop to look like this:

for location in report_dictionary[report_item]: 
  symbol = check if report_dictionary[report_item][location] else cross
  html_output += "<td>{}</td>".format(symbol)
  csv_row += [symbol]

Overall fixing this issue was pretty straightforward. I learned a little bit about building/writing CSV files in python and how they're displayed in github.

Fixes #52: Change report to produce CSV instead of HTML #55

slaterslater posted on Oct 20, 2020

region-checker.py creates a csv file in addition to the html report
added the ✕ to compliment the ✓

View on GitHub

Checking links & ignoring domains

Anthony Slater — Sun, 18 Oct 2020 19:05:28 +0000

For the past several weeks, I've been working on Gustavo: a URL checking command line tool written in python. I wanted to gain some experience forking repos; setting up remotes; and merging branches, so I got permission to work on a new feature for checkThatLink

TDDR / checkThatLink

The feature I wanted to add would allow for a command line argument
-i or --ignore to be passed along with a path to a file. The program should open the file; read the contents; and if any URLs are present, omit their domain from the HTTP status checking process.

To begin, I needed to understand how this program worked before I could add any features. It just so happens, that Gustavo and checkThatLink are very similar in purpose and language. I'm by no means and expert in Python, but I felt pretty comfortable reading through the source code. As I walked through the program i determined it followed these steps:

Create a parsed command line arguments object and pass it into a new checkFile object checkFile(args)
checkFile initializes all of its member variables from the args object
checkFile runs its main function checkThatFile()
Each line of the source file is inspected to see if it contains a URL
A HTTP connection is made for each URL requesting the HEAD and the URL and its response status are appended in a list of dictionaries allLinks
A series of conditional statements determine the correct function should generate the output from allLinks

I didn't want the program to waste time checking a link that would later be omitted from the output, so I figured I should insert my feature/check in between step 4. & 5.

I first created a function that would open the provided file with urls to ignore; use a regular expression to pick out all the valid urls; and return the ignoreList of domains.

Next I added a condition before step 5. above, so that each url's domain would be checked against the domains in the ignoreList

At this point, I thought I was finished. I had the author review my work before merging and I was informed that some updates were needed. The issue requirements stated that if the program receives a file containing domains to ignore, but no comments or urls are present, then the program should exit. It was a pretty straight-forward update to make:

I added another regular expression to find comments.
If both regular expressions (domains and comments) are empty, the program exits.

def getIgnoreList(self, ignoreFile):    
  found = []
  try: 
    if ignoreFile:
      with open(ignoreFile) as src:
        text = src.read() 
        found = re.findall('^https?://.*[^\s/]', text, flags=re.MULTILINE)
        comment = re.search('^#.*', text, flags=re.MULTILINE)
        return found if comment or found else sys.exit(1)
    return found
  except:
    print(f'Error with {ignoreFile}')
    sys.exit(1)

The above update passed the test and the code was merged upstream!

On the other side of the ball, I had a volunteer to implement the same ignore-feature in Gustavo. We worked through the bugs together. I found it pretty easy to make a new branch from the contributor's remote and make some fixes. I hope I wasn't too overbearing in this regard; I didn't just point out the existence of a bug, I made suggestions and provided the code to fix them.

I did learn the hard way that it is a good idea to run git status or git branch to know where you are in your git tree before fetching or pulling. It wasn't too serious, as I got issue-8 and issue-8-fix mixed up. I'm getting more comfortable with git each week, but clearly I need more practice.

Adding flag support to Gustavo

Anthony Slater — Tue, 13 Oct 2020 23:35:49 +0000

I've been tinkering with a command line tool that searches a source file for URLs and generates a report of their status.

slaterslater / gustavo

Print a colourized report of all HTTP urls in a file

Gus received a recent PR to add windows & unix style command line arguments. Once this PR was merged, Gus could accept and do the following:

$ python gus.py -f some.file  // checks the URLs in some.file
$ python gus.py -v            // shows version info 
$ python gus.py -h            // shows usage instructions

--file, --version and --help work too.

A great start, but why stop there? I knew I wanted Gus to handle many more arguments, so it no longer made sense to check sys.argv against strings through a series of conditional statements. I decided to import argparse to do this heavy lifting for me.

The first thing I did was change the default output of the program. Instead of automatically producing a Rich Text File, output to the console would be standard. If the user wants a Rich Text File as the output, they can pass Gus
-r or --rtf from the command line.

Next I updated the checker function by swapping the http.client library for requests. I renamed get_status_code() to get_status() since it would be returning the code and description as a dictionary. I figured this would be helpful down the road when determining the output format (console or RTF).

def get_status(url):
  try:
    conn = requests.head(url, timeout=2.5)
    code = conn.status_code
    series = str(code)[0]
    desc = 'UNKN'
    if series == '2':
      desc = 'GOOD'
    elif series == '4':
      desc = 'FAIL'
    return {'code':code, 'desc':desc}
  except:       # all exceptions default to status == 400
    return {'code':400, 'desc':'FAIL'}

One thing I want to point out before moving on... Notice how all exceptions default to status == 400. I'm not really happy with this, so I opened a new issue to improve exception handling. Hopefully in the near future I (or someone else 🤞) will have the time to work on this.

Whichever output format the user chooses, I decided it will be displayed or produced once all URLs have been checked. Since Gus doesn't yet leverage threading, this can take a while. In the event that Gus is taking a long time, I wanted to show the user that everything is ok by showing the progress. After a little research I came up with the following:

print(f'\r Checking URL {list.index(string)} of {len(list)}', end="\r")

By putting the above print statement in my for loop, each iteration over-writes itself and provides a single line progress update to the user.

Once everything worked as expected, I created two new branches to develop two new features:

At this point I was very happy to have argparse on board. Argparse was already sending a string parameter output to the main function. The value of output could either be "std" or "rtf" and adding "json" into the mix was easy peasy.

In order to support --all, --good, and --bad flags, argparse needed to send a list parameter wanted to the main function. By default, the wanted list would be equal to:

['GOOD','FAIL', 'UNKN']

Passing --good or --bad restrict the values of the above list accordingly. In all cases, the values in wanted get checked against the returned dictionary from get_status() and if there is a match, the formatted string is saved for later output.

Since many functions need to know what the user's output choice is, I decided to make it a global variable instead of passing it from function to function. I'm not sure if this is bad form, but the code looks pretty clean and everything worked as intended.

To date, these are all the options for Gus:

Command line arguments	Description
-f FILENAME, --file FILENAME	location of source file to be checked
-a, --all	output includes all results (this is default)
-g, --good	output includes only [GOOD] results
-b, --bad	output includes only [FAIL] results
-r, --rtf	output as colourized rich text file
-j, --json	output as JSON
-h, --help	show information on how to use the tool and exit
-v, --version	show program's version number and exit

The hard part was the merging of two branches. It was supposed to be a recursive three-way merge, but it ended up a dog's breakfast... I'm certain I screwed up the merging, but the code I want is up on Github so I guess I can't complain. I have two more features planned for Gus, so I'll try merging branches again soon.

Refactoring Verify-URL

Anthony Slater — Sat, 03 Oct 2020 22:10:39 +0000

abhaseen / Verify-URL

A Python script that verifies the return code of URLs.

The first thing I noticed was the entire program was written inside the main() function. To help me understand how the program worked, I decided I would refactor Verify-URL into smaller functions.

I reordered main() into the following:

get_args()
check_args()
get_urls()
verify_url()
print_status()

Everything looked good and worked as expected... except when I supplied a filename.

>>> python verify_url.py src.html
Invalid URL '': No schema supplied. Perhaps you meant http://?

I traced the error to get_urls() as it looked as though a NoneType object was being appended into what was supposed to be a list of strings.

for url in soup.find_all('a'):
  urls.append(url.get('href'))

To fix, I cast every item as a string and checked to see if it started with 'http'

for url in soup.find_all("a"):
  href = str(url.get('href'))
  if href.startswith('http'):   
    urls.append(href)

I ran the tool again from the command line and supplied a file to check. This time it ran smoothly and printed a colourful list.

I submitted a Pull Request, so hopefully these changes pass the Author's test! That wasn't supposed to rhyme...

Reviewing URLChecker

Anthony Slater — Fri, 02 Oct 2020 04:09:03 +0000

Since I recently started development of a command line tool that searches a source file for URLs and check their status, I decided to review an existing one while I'm at it. My tool is written in Python, but I'm interested to see how the same task could be done using JavaScript.

danishalim / URLChecker

Bulk check URL status codes

First thing I noticed was just how fast async functions are! I ran this tool against the same source file I've been using to develop my own work and it took maybe 15 seconds. Maybe. Usually when I run my Python URL checking tool, I get up to walk around and check back in a few minutes...

My one criticism of URLChecker is the regular expression it uses to find URLs in a file. I ran into the same issue myself during development and fixing it ended up opening up a whole other can of worms. Once the regex includes characters like [] and () then:

"http://some(web).site" // matches http://some(web).site
"(http://someweb.site)" // matches http://someweb.site)

Although URLChecker will need to update its regex so it includes all valid URL characters, I am extremely envious of how fast it produces its results.

Introducing Gustavo

Anthony Slater — Thu, 01 Oct 2020 20:48:45 +0000

I needed to search a file for URLs and generate a report of their status. So I created Gustavo to do the work for me.

Get-Url-Status (GUS) Text-As-Visual-Output (TAVO)

Sure, the name might be a little corny, but it makes me happy so I'm sticking with it. I also like to imagine Gus as an old mustachioed plumber that checks all the URL connections in a document...

Gus can be called from the command line

python gus.py example.html

or from the python shell

>>> import gus
>>> gus.tavo('example.html')

How does Gus work?

The file at the provided location is opened and the contents are saved as a string.
A regular expression matches every instance of HTTP or HTTPS and the results are saved as a list.
For each item in the list, an HTTP connection is made - requesting just the header - and the HTTP response code is returned.
The returned code corresponds to a status: 2xx is labeled as [GOOD]; 4xx is labeled [WARN]; and everything else defaults to [UNKN].
The entire list of URLs, codes and their status are printed to the console and also written to a file.

I knew I wanted the written output file to be colourized, so I did some research into producing Rich Text Files. There is lots of great info at www.pindari.com but unfortunately my solution ended up being a little hackier than anticipated.

First I opened TextEdit and created output.rtf with four lines of text. Each line was given a different colour (default, grey, green, & red). Then I had python open the file, read the contents and print it all to the console. Next I copied everything up to the first line of text and saved it to a constant. What remained were the lines of text and the colour-codes I needed! I copied those into a function which would append the appropriate code to the URL string depending on the returned HTTP status.

for url in list:
    url = check_nested(url)
    code = check_status_code(url)
    color = r'\cf2' #grey
    status = 'UNKN'
    if code[0] == '2':
      color = r'\cf4' #green
      status = 'GOOD'
    elif code[0] == '4':
      color = r'\cf3' #red
      status = 'WARN'
    checked.append(f"{color} [{code}] [{status}] {url}")

No doubt there is a lot of room for future improvement, but overall I'm pretty happy with results so far.

If you're at all interested, check out the repo:

slaterslater / gustavo

Print a colourized report of all HTTP urls in a file

Getting started in Open Source development

Anthony Slater — Fri, 18 Sep 2020 22:01:38 +0000

I'm Anthony - a computer programming student living in Toronto, Canada. During my first semester, I remember circling the Open Source Development courses as ones I knew I wanted to take. Each semester I would take all the requisite classes in C++, SQL, Java & Javascript as stepping stones towards the goal of Open Source Development. Now - here I am in my final semester and I feel wholly unprepared and out of my depth.

It's my understanding that the imposter syndrome is pretty normal for beginner programmers and it's my hope that over the next few months I'm able to shake it.

I've been taking a look at RxDB

pubkey / rxdb

💻 🔄 📱 A realtime Database for JavaScript Applications

It looks really interesting, as I've had some aspirations to build some mobile apps that could leverage a feature like state change subscriptions. I don't know if I'll be able to contribute anything meaningful to this repo, but it looks to be something I could use in future project of mine.