Forem: Barrett Otte

Jupyter Notebooks + IBMi

Barrett Otte — Wed, 06 May 2020 16:28:16 +0000

This is going to be a short post on calling basic IBMi DB2 statements from Jupyter Notebooks. It is really light on explanation, so I encourage googling terms you are not familiar with.

Briefest Introduction to Jupyter

Jupyter provides an environment for executing live code in a notebook like format. I find these especially useful for keeping "living notes" for mathematics and production support notes.

A notebook consists of cells, which allow you to mix and match SQL, Markdown, Python, Javascript, and much more in the same notebook.

Each cell can be executed independent of the other cells. So you can execute whatever cell you want.

This is a really basic example, but you can get the full power of pandas, numpy, matplotlib, etc to create some cool data visualizations. I won't be covering that because I'm pretty new to these modules still.

Notebooks can be shared easily to other developers.
When the notebook is committed to a repository, it includes all of the output for each cell (if they were executed).
So, in a GitHub repository you can actually preview the notebook frozen in time.
Its pretty neat, you'll see some cooler stuff here soon.

I encourage you to go out and read more on Jupyter Notebooks.

Setup

Install dependencies - pip3 install jupyter pandas matplotlib numpy pyodbc (Pyodbc is for my IBMi example)
Start Jupyter server -

jupyter notebook
Hosted locally at http://localhost:8888/notebooks by default

Why Pyodbc?

So, I made this post because calling IBMi is a little funky to do from Python and Jupyter. Most SQL Jupyter examples you'll come across use a Python module called SQL Alchemy. This module provides some convenience functions for interacting with a database.

The driver for IBMi access does not work with SQL Alchemy. There is a repository in progress to get it working, but I don't think its ready yet - https://github.com/IBM/sqlalchemy-ibmi

But, that's okay we can still use the pyodbc module.
I won't go into a full example, but here is a link to my little eBook with an example program - https://barrettotte.github.io/IBMi-Book/#/additional/call/python

The most important bit is in the connection string:

conn = pyodbc.connect(driver='{IBM i Access ODBC Driver}', system=host, uid=user, pwd=pwd)

It's slightly different than a lot of the examples you'll find floating around for connecting to DB2 and was slightly annoying to figure out at first.

Small Utility Notebook

To make it even easier to work with IBMi on Jupyter I made a little utility notebook at https://github.com/barrettotte/IBMi-Jupyter

It includes some utilities for connecting and displaying result sets.
The main thing the utility notebook adds is some cell magic.
I won't go into too much detail, but it allows us to mark a cell to perform certain actions.

I added cell magic for DB2 for i. To mark a cell as DB2 for i, you include %%ibmi, which pretty much tells Jupyter to treat this cell special with whatever is defined for ibmi magic. The cell magic I defined takes in a single flag and a series of SQL statements delimited by a semicolon.

The SQL result set is returned as a pandas dataframe. I encourage you to read up on them. To summarize, they are convenient data structures used in a lot of data science packages.

Here is a summary of the flags I have currently implemented, they cause the cell magic to do things slightly different behind the scenes.

flag	description
-csv	result set as CSV
-html	result set as HTML string
-json	result set as JSON string
-tex	result set as LaTex string
-str	result set as simple string
-xml	result set as XML string
-plotbar	result set as simple bar graph
-plotpie	result set as simple pie graph
-plotline	result set as simple line graph

Take a closer look at the cell magic in https://github.com/barrettotte/IBMi-Jupyter/blob/master/IBMi.ipynb

Examples (finally)

To test out my utility notebook, I made a simple test notebook with a few queries.

To include the utility notebook in another notebook, invoke it with

%run IBMi.ipynb

(or whatever path you put it).
This will trigger the login, where it prompts for hostname, user, and password (kept as secret) using python inputs (getpass module for password).

Enter host: DEV400
Enter user: OTTEB
Enter password: ········
Connecting to DEV400 as OTTEB ...
Successfully connected!

Now every cell afterwards will have access to the connection.
So let me show some example query screenshots, they are pretty self explanatory.
I skipped the JSON, XML, etc string examples because they weren't as cool to show.

All examples can be found in https://github.com/barrettotte/IBMi-Jupyter/blob/master/tests/Test.ipynb

Conclusion

This utility notebook is pretty barebones right now. But, maybe it will give somebody some ideas on a fun project to start on. As I get bored over some lunches at work I intend to add in more flags and features to make this more powerful.

The next thing I want to add is called Line Magic so we can use the result set we get back from a query. At the same time, I will add in prepared statements to the existing magic, so more complex visualizations can be made.

Feel free to yank the code for this and make something awesome.

EDIT:
I forgot to mention one of the other cool things. You can do all of this from VS Code as well. Jupyter Notebook support is bundled with the Python extension.
This means you can also have a basic DB2 for i SQL client in the comfort of VS Code.

TRS80-Twilio, Sending Texts like its 1983!

Barrett Otte — Tue, 28 Apr 2020 14:03:41 +0000

What I built

Category Submission: Exciting X-Factors

Send a text message from a TRS-80 Model 100 using the Twilio SMS API and an ESP8266.

The basic goal was to use a vintage computer as a frontend and an ESP8266 as a backend to send a text message.

As I state in my repository, In no way is this practical or should be recreated. This was solely for fun; don't take this too seriously.

I wanted to put the "HACK" in HACKATHON.

Demo Link

An awful phone video showing this project in action with test credentials and "magic" test phone number. I figured it was better than nothing in case you wanted to see it actually running.
https://www.youtube.com/watch?v=C2nngeuc3Oc

Link to Code

Check out some of the additional documentation I included, as well as some small test programs. https://github.com/barrettotte/TRS80-Twilio

How I built it (what's the stack? did I run into issues or discover something new along the way?)

Frontend: TRS-80 Model 100, BASIC
Backend: ESP8266, Arduino platform, C++

After I decided I was going to participate in the Twilio hackathon I stumbled across documentation for using an ESP8266 to send an SMS. Naturally, I decided to take it a step further and make a complete coding abomination.
The basic goal was to use a vintage computer as a frontend and an ESP8266 as a backend to send a text message.
This probably won't get judged (I can't blame them), but this was one of the funnest projects I've ever done.

I started out making the "frontend" on a TRS-80 Model 4P using BASIC and Z80 assembly. In order to write to the Model 4P's serial port, I had to embed Z80 machine language in BASIC. Example of embedding machine language:

1   REM TEST EMBEDDED MACHINE LANGUAGE PROGRAM IN BASIC
10  DIM A%(12)
20  B = VARPTR(A%(0))
30  FOR I = 0 TO 11
40  READ C,D
50  E = D * 256 + C
60  IF E > 32767 THEN A%(I) = E - 65536 ELSE A%(I) = E
70  NEXT I
80  IF B > 32767 THEN DEFUSR0 = B - 65536 ELSE DEFUSR0 = B
90  X = USR0(0)
100 GOTO 90
105 REM ************* MACHINE LANGUAGE PGM *******************
110 DATA 33,64,60,17,0,60,1,192,3,237,176,33,192,63,62,32,6,64
120 DATA 119,35,16,252,201,0
125 REM ******************************************************

I woke up one morning and my old computer no longer worked...Read more in my repo.
I had no time to fix it.

So, I had to start from scratch 5 days before the due date of this hackathon.
I recently bought a cheap TRS-80 model 100, so I switched over to learning how it worked.

To my surprise, it was magnitudes easier to work with than the TRS-80 model 4P.
With the model 4P, I was forced to embed Z80 machine code into BASIC in order to write to its serial port.
With the model 100, all I had to do was open the serial port like a file and print to it...amazing.
The primary frontend file is TWILIO.BAS.
The frontend's job was to:

provide a simple interface to input a phone number and message
build a buffer of 145 bytes from user input
basic phone number validation
send buffer to the ESP8266 via RS232 (serial).

The "backend" was pretty easy to build. I recently refreshed on Arduino and learned how to use PlatformIO.
The biggest challenge with the backend was waiting for the incredibly slow serial port send all of the data. There was no good way to figure out when it was done. To solve this challenge, I decided to send a flat 145 bytes from the TRS-80 frontend each time. That way, I always knew that after reading 145 bytes I was done reading from the serial port.

The primary backend file is src/main.cpp
The backend's job was to:

establish connection to WiFi
output helpful messages to OLED
read serial input and parse buffer from TRS80
establish connection to Twilio API
build and invoke HTTP request to Twilio SMS API

In summary, this was one of the funnest projects I've ever worked on.
Truthfully, I wasn't sure if I was going to be able to pull this project off.
The reward of hacking away and completing this was huge.

Not to mention, I got to learn a lot about vintage computing and a bit about embedded development.

Project Layout

### Basic Wiring ###

TRS80 DB25 --> DB25-DB9 Null Modem --> DB9 Male-Male --> TTL-RS232 ...

TTL-RS232   ESP8266      Mini OLED
              GND <----> GND
              3V3 <----> VCC
              D1  <----> SCL
              D2  <----> SDA
GND <-------> GND
VCC <-------> 3V3
TXD <-------> D6 (RXD)
RXD <-------> D5 (TXD)

Additional Resources/Info

Sending a Text Message with SQL (IBMi)

Barrett Otte — Thu, 23 Apr 2020 12:13:38 +0000

This is going to be a pretty short post, but I thought it would be fun to share this in light of the Twilio + DEV hackathon this month. Honestly, I'm not sure if either project I planned out will be finished in time. This month went by way too fast, so the outlook is not good.

If you are unfamiliar with IBMi and have a few minutes to read, I encourage you to glance over a previous subpar post. If not, it doesn't really matter; its not critical to looking at a couple blocks of SQL.

This image is IBMi in a nutshell. It either disgusts or fascinates a younger developer; it is what it is.

Since I'm still a pretty new IBMi developer, I found that the best way to learn this new world was to take a fun idea and see if it was possible to do with RPG and/or DB2 SQL.
Now I have a handful of arguably useless code snippets for my day job lol...

Example HTTP Request in DB2 for i

One of the major things that made me fall in love with IBMi was that DB2 was really versatile. The fact that I could easily send HTTP requests with SQL was mind blowing.

For a really basic example, I'm going to do a basic HTTP GET to the GitHub REST API v3 for my user using only SQL. The equivalent CURL command will be

#!/bin/bash
curl -i https://api.github.com/users/barrettotte

This HTTP request is super simple because we don't need any HTTP headers.

values SysTools.HttpGetClob(
  cast('https://api.github.com/users/barrettotte' as varchar(128)), -- url
  cast(null as clob) -- http headers
);

The request returns

00001
'{"login":"barrettotte","id":15623775,"node_id"...}'

Ok. That's cool because we can parse the JSON string in another program or something. But, actually we can do some way cooler stuff. We can return the HTTP response as a result set.

Here is a really basic example, I mostly just used varchars for simplicity, but any field could be mapped to any appropriate data type.

select *
from json_table(
  SysTools.HttpGetClob(
    cast('https://api.github.com/users/barrettotte' as varchar(128)),
    cast(null as clob)
  ),
  '$' columns(
    login          varchar(64)   path 'lax $.login',
    id             varchar(16)   path 'lax $.id',
    node_id        varchar(32)   path 'lax $.node_id',
    avatar_url     varchar(64)   path 'lax $.avatar_url',
    url            varchar(64)   path 'lax $.url',
    html_url       varchar(64)   path 'lax $.html_url',
    followers_url  varchar(64)   path 'lax $.followers_url',
    following_url  varchar(64)   path 'lax $.following_url',
    gists_url      varchar(64)   path 'lax $.gists_url',
    starred_url    varchar(64)   path 'lax $.starred_url',
    subs_url       varchar(64)   path 'lax $.subscriptions_url',
    orgs_url       varchar(64)   path 'lax $.organizations_url',
    repos_url      varchar(64)   path 'lax $.repos_url',
    events_url     varchar(64)   path 'lax $.events_url',
    rcv_events_url varchar(64)   path 'lax $.received_events_url',
    type           varchar(16)   path 'lax $.type',
    site_admin     varchar(8)    path 'lax $.site_admin',
    name           varchar(64)   path 'lax $.name',
    company        varchar(64)   path 'lax $.company',
    blog           varchar(64)   path 'lax $.blog',
    location       varchar(64)   path 'lax $.location',
    email          varchar(64)   path 'lax $.email',
    hireable       varchar(8)    path 'lax $.hireable',
    bio            varchar(512)  path 'lax $.bio',
    public_repos   int           path 'lax $.public_repos',
    public_gists   int           path 'lax $.public_gists',
    followers      int           path 'lax $.followers',
    following      int           path 'lax $.following',
    created_at     varchar(32)   path 'lax $.created_at',
    updated_at     varchar(32)   path 'lax $.updated_at'
  )
);

The request returns

LOGIN	ID	NODE_ID	...
barrettotte	15623775	MDQ6VXN...	...

Personally I think this is really neat.

Calling Twilio from DB2 SQL

I figured I should say I know that this is not an ideal solution for probably a majority of cases. But, this is just a fun thing I'm showing.

The HTTP request I'm going to make looks like this

POST {{base}}/2010-04-01/Accounts/{{account}}/Messages.json HTTP/1.1
Authorization: Basic {{account}}:{{auth}}
Accept: application/json
Content-Type: application/x-www-form-urlencoded

To={{to}}&From={{from}}&Body=Hello+World

Calling Twilio takes a few more steps than the previous example. In DB2 for i, you have to pass HTTP headers as XML (I assume its a legacy thing and I wish it was JSON, but its still easy to work with).

The authorization header has to be passed as a Base64 encoded string. There is a scalar function called Base64Encode in SysTools that seems to work fine.

Additionally, Twilio wants the request body as a url-encoded string. To my surprise, I learned that SysTools also has a scalar function called UrlEncode. All of the hard work is done with scalar functions, you just have to piece them together.

I replaced all secrets in this query with 'variables' surrounded by double curly braces.

select *
from json_table(
  SysTools.HttpPostClob(
    'https://api.twilio.com/2010-04-01/Accounts/{{account}}/Messages.json',
    cast((
      '<httpHeader>
        <header name="Authorization" value="Basic ' || trim(SysTools.Base64Encode(
          cast('{{account}}:{{auth}}' as varchar(256) ccsid 1208))) ||
        '"/>
        <header name="Accept" value="application/json"/>
        <header name="Content-Type" value="application/x-www-form-urlencoded"/>
      </httpHeader>'
    ) as clob),
    cast((
      'To='    || SysTools.UrlEncode('{{to}}', 'UTF-8') ||
      '&From=' || SysTools.UrlEncode('{{from}}', 'UTF-8') ||
      '&Body=' || SysTools.UrlEncode('Hello World', 'UTF-8')
    ) as clob)
  ),
  '$' columns(
    sid           varchar(64)   path 'lax $.sid',
    date_created  varchar(64)   path 'lax $.date_created',
    date_updated  varchar(64)   path 'lax $.date_updated',
    date_sent     varchar(64)   path 'lax $.date_sent',
    account_sid   varchar(64)   path 'lax $.account_sid',
    phone_to      varchar(32)   path 'lax $.to',
    phone_from    varchar(32)   path 'lax $.from',
    msg_srv_sid   varchar(64)   path 'lax $.messaging_service_sid',
    body          varchar(1600) path 'lax $.body',
    status        varchar(16)   path 'lax $.status',
    num_segments  varchar(8)    path 'lax $.num_segments',
    num_media     varchar(8)    path 'lax $.num_media',
    direction     varchar(32)   path 'lax $.direction',
    api_version   varchar(16)   path 'lax $.api_version',
    price         varchar(8)    path 'lax $.price',
    price_unit    varchar(4)    path 'lax $.price_unit',
    error_code    varchar(8)    path 'lax $.error_code',
    error_message varchar(512)  path 'lax $.error_message',
    uri           varchar(256)  path 'lax $.uri',
    nested '$.subresource_uris[*]' columns(
      media       varchar(256)  path 'lax $.media'
    )
  )
);

The request returns

SID	DATE_CREATED	DATE_UPDATED	DATE_SENT	ACCOUNT_SID	PHONE_TO	...
...	Wed, 22 Apr 2020 23:35:26 +0000	...	...	...	...	...

Conclusion

That's kind of it, I thought it was pretty cool and figured I'd share it for anyone browsing around this morning.
The gist can be found here

Realistically, I don't know what you would do with this. But, knowing that it exists "might" do some good in some situation.

Writing an IBMi Syntax Highlighter for VS Code

Barrett Otte — Mon, 30 Mar 2020 13:22:18 +0000

Introduction

I love VS Code and really want it to be the "all in one" editor.
Unfortunately, there are still plenty of fringe cases that just don't work yet.

To make a long stort short, I recently started working as an IBMi developer. Its a midrange system (the middle ground between standard server and mainframe) that uses an old programming language called RPG (https://en.wikipedia.org/wiki/IBM_RPG, the same era as COBOL and FORTRAN).
So, I decided I wanted to be able to use VS Code to read IBMi code.

Time to Learn Regular Expressions

A couple months ago I didn't know the black magic behind regular expressions.
In a prior side project (https://github.com/barrettotte/Ranger-Lang) I learned about compiler theory and finite automata.
Which cleared the fog on the reasons why regular expressions exist and a bit of the syntax. I highly suggest taking a bit of time to read through some basic examples of finite automata to get a sense of what I'm talking about.

The rest of my learning was messing around for literal hours using https://regexr.com/ . This tool saved my life, it made things so easy to test.

Yes, it took me that long to get the hang of things; you have to be willing to put in the time when you're as dumb as me.

Regular Expression Lookaround

One of the most valuable things to know how to do in regular expressions is lookaround.
A key concept to understand in regular expressions is that once a pattern is matched, the characters matched are "eaten" by the regex engine (they cannot be reused).
A lookaround allows you to match a pattern ahead or behind and does not consume the characters matched.

Regular Expression Example

For example,
I had to make sure there was an 'F' in column 6, an 'F' or 'E' in column 14, and a certain set of words in column 41.

     FPRINTER O   F     132     OF     PRINTER

Using positive lookbehind, I can check the 'F', the 'F' or 'E', and 'PRINTER' while ensuring that all conditions are matched.

REGEX:

(?i)(?<=(?<=(?<=^[\s]{5}F).{8}).{4}(F|E).{20})(WORKSTN|DISK|PRINTER|SPECIAL|SEQ)

This regex will lookbehind for two conditions and match a word (WORKSTN...SEQ) if the prior conditions are matched.

Broken down this is:

(?i)               // case insensitive switch
(?<=               // start outer positive lookbehind
  (?<=             // start middle positive lookbehind 
    (?<=           // start inner positive lookbehind
      ^            // beginning of string
      [\s]{5}      // any non-whitespace character 5 times
      F            // 'F'
    )              // end inner positive lookbehind
    .{8}           // any character 8 times
  )                // end middle positive lookbehind
.{4}               // any character 4 times
(F|E)              // match either 'F' or 'E'
.{20}              // any character 20 times
)                  // end outer positive lookbehind
(WORKSTN|...|SEQ)  // match any word in list

At least for me, this didn't make much sense until I got my hands dirty.

Introduction to TextMate

Atom and VS Code both use a thing called TextMate to handle all syntax highlighting. It provides a convenient way to list out sets of regular expressions that correspond to parts of a language such as constants, functions, keywords, operators, etc.

Googling around it seems that the TextMate grammars can be written in XML, JSON, and YML. I decided on JSON after screwing around with the YML version and not getting it working correctly (I think).

The previous regular expression example as a TextMate Grammar; I make (WORKSTN|...|SEQ) highlight as
a constant.

"constants": {
  "patterns": [
    {
      "name": "constant.language.rpg.f.device",
      "match": "(?i)(?<=(?<=(?<=^[\\s]{5}F).{8}).{4}(F|E).{20})(WORKSTN|DISK|PRINTER|SPECIAL|SEQ)"
    }
  ]
}

There's also a lot more fancy things you can do with nested patterns, but I won't get into it here. If you're curious read more at https://www.apeth.com/nonblog/stories/textmatebundle.html

For fun, here's a link to Dart's syntax highlighter in TextMate https://github.com/Dart-Code/Dart-Code/blob/master/syntaxes/dart.json

VS Code Language Extension

Another reason why I love VS Code is that they provide just enough documentation to get you started and let you go off on your own.

Language extensions are no exception, I just used this documentation page to get started https://code.visualstudio.com/api/language-extensions/overview

Everything else was achieved by looking around at existing extensions and screwing around with it.

Putting It All Together

Using the basic knowledge I wrote above, I sat down one week and just kept hacking away.

For my particular extension, I sat down reading IBM's language documentation and making regex one piece at a time.
By the end of it I made my language extension support DDS, RPG, RPGLE (fixed and free formats), Control Language (CL), and Machine Interface (MI).
You can check out my extension at https://github.com/barrettotte/vscode-ibmi-languages

Summary

Honestly, its not that hard of work its just insanely tedious and time consuming. But if you don't know how regular expressions really work, here is the perfect excuse to learn.

This is also the perfect weekend project to put on some music, drink some beer, and be a tiny bit productive while not working too hard.

IBM i "Book" Chapters 1 and 2

Barrett Otte — Thu, 10 Oct 2019 14:02:39 +0000

This is a post to say that I have "completed" the first two chapters of my small "book" Learning IBM i as a Lowly Web Developer. (Most of the core content is in an acceptable state, but may still be tweaked when I read it over for the millionth time).

The work in progress can be found at https://barrettotte.github.io/IBMi-Book/#/

Preface

It should be obvious, but I'm not a writer and barely a programmer sometimes.
This book is lazy and heavily relies on "learning by example". Therefore, it might be tough for some people to follow. But, I do try to briefly explain and walkthrough writing each program.

I also decided to use SEU throughout this book just to get the full "green screen" experience. If you can code in SEU, you can code in something better be default.

I do mention ILEditor and plan to include a brief overview of this awesome IDE so that people have the option to use it instead of ancient SEU.

Chapter 1

https://barrettotte.github.io/IBMi-Book/#/core/ibmi/index

In chapter 1, I mainly go over operating system concepts and Access Client Solutions (ACS). This includes some Source Entry Utility (SEU) and Program Development Manager (PDM).

For practice, I walkthrough creating a library, source physical file, and a CL source member. Additionally, I step through compiling and running a three line CL program.

Chapter 2

https://barrettotte.github.io/IBMi-Book/#/core/cl/index

In chapter 2, I give an overview of the syntax/structure of CL and start writing some basic CL. The example CL program I provided accepts user input via inquiry and echos it back to the user as a message.

For a more beefy example, I included a subpar implementation of fizzbuzz in CL to go over subroutines, conditionals, and loops. It could definitely be a better implementation, but I think it serves its purpose just fine.

Conclusion

I hope to keep making progress on this book as I feel inspired. The next chapter is going to be on the basics of some old RPG and newer RPGLE. Additionally, I hope to add more content to the CL chapter on the differences between newer CLLE vs CL.

If you happen to read the book and have a suggestion on an improvement, please open an issue in my repository here https://barrettotte.github.io/IBMi-Book/#/ and I'll definitely consider it. After all, I'm still a complete noob to the IBMi and could use all the help I can get.

On a related note, my company allowed me to attend RPG & DB2 Summit in Minneapolis. I'm 23 so I'll be sticking out like a sore thumb around all the seasoned IBM i developers. I hope to learn a bunch of cool content to include/mention in this book.

Coding Abomination: Conway's Game of Life in TSQL

Barrett Otte — Sun, 22 Sep 2019 21:08:53 +0000

GIF can be seen here
https://github.com/barrettotte/SQL-Game-of-Life/blob/master/demo.gif

One morning during my commute I was zoning out and realized I wanted to make a really dumb side project in SQL. One of my favorite things to do is use a language unconventionally and I thought implementing Game of Life was the perfect little project to do.

Game of Life

If you are unfamiliar with Conway's Game of Life, I encourage to read more about it here https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life

The "algorithm" it uses is very simple. For each cell (pixel) in the environment (board), there are 4 rules for determining the next generation.

Any live cell with fewer than two live neighbours dies, as if by underpopulation.
Any live cell with two or three live neighbours lives on to the next generation.
Any live cell with more than three live neighbours dies, as if by overpopulation.
Any dead cell with exactly three live neighbours becomes a live cell, as if by reproduction.

(Taken from the previously linked wiki page)

Through these simple rules, "organisms" can appear that exhibit different behavior. In my example, I implement a "blinker" and a "glider". They can also be found on the wiki page I linked.

Implementation

For this project I could have used TSQL's built in PRINT function, but I wanted actual animation. To do this, I would have to leverage a little bit of Python to display the data that my stored procedure produced using the stdout buffer.

The stored procedure I made stores the cell data of each generation. If I take the cell data for each generation I can feed it into a drawing function to represent each frame in an animation.

Now, I will say the stored procedure runs terribly. But, the goal was to get a basic glider to travel across the screen. I am 100% sure I could have done some fancy JOINs or something. But, I just ended up using SQL like a general purpose language instead.

Source

The project consisted of a single stored procedure and a python display script.
Everything can be found here https://github.com/barrettotte/SQL-Game-of-Life

Coding Abominations

Its all well and good to spend your time programming useful things or learning new subjects. But, I think one of the funnest things to do is make really obscure and useless projects.

You would be surprised on how much you learn when making something super bizarre that has been implemented by only a few people. It forces you to think outside the box and touch parts of a language that you don't normally handle.

I find that making a coding abomination every once in a while is a good anti-burnout exercise. After finishing a fun coding abomination, I feel refreshed to tackle another side project. The important thing is to pick something you're relatively interested in, otherwise you're not going to have a good time.

Another fun abomination I made a few months back was a 2D perlin noise SVG generator using only XSLT https://github.com/barrettotte/XSLT-Perlin2D .
I ended up learning a bunch of neat tricks in XSLT that I didn't even know were possible.

Write a Book with Docsify

Barrett Otte — Mon, 16 Sep 2019 12:24:23 +0000

If you ever thought about writing a small book in the comfort of markdown/html then I would suggest using Docsify; https://github.com/docsifyjs/docsify

Docsify is pretty intuitive and has sufficient documentation to get you through the basics of putting together a book. Each directory is a section
and each markdown file is a page in the book. It also has other features such as navigation, cover pages, and sidebars that I haven't dove too deep into.

Since it uses markdown, you can also use standard HTML and CSS to style everything. I've had mixed results, but it seems like you can also use basic JavaScript as well. The only catch I've found so far is that you cannot use any asynchronous content. For example, I was unable to use embedded code snippets from https://www.cacher.io/ since it is an asynchronous request to cacher to get the content.

Commands

Install docsify cli through npm - npm i -g docsify-cli
Start a new project - docsify init mybook
Serve book locally on http://localhost:3000 - docsify serve mybook

Sample Page

As expected, you can mix and match markdown and HTML depending on what you need to do. For all of my cases, this was perfect.

<!-- Sample page from my book -->

## Introduction to the 5250 Emulator

This is an emulator of the IBM 5250 terminal originally used to interact with the IBMi and its ancestors. A bit more information on it here https://en.wikipedia.org/wiki/IBM_5250

There's a lot of features in this emulator that I haven't messed around with.

One important thing I learned is using the **popup keypad** located in **Actions** > **Popup Keypad...**
Sometimes when things go wrong, you have to use **SysReq** to bail yourself out (more on this later).
<figure align="center">
    <img src="./core/ibmi/_assets/5250-01.PNG" alt="Popup Keypad" />
</figure>

Code Blocks

One feature I wanted in my book was basic code blocks with a dark theme. Docsify uses a syntax highlighting package called Prism JS. For the most part, Prism JS works pretty well out of the box. To include syntax highlighting for PHP you would throw this into your index.html

<!-- index.html -->
<script src="//unpkg.com/prismjs/components/prism-php.min.js"></script>

Unfortunately, I wasn't smart enough to figure out how to use the Prism plugin with Docsify (if you even can). I also tried to get line numbers working through a couple different CSS snippets, but I think my frontend abilities are severely lacking.

Code Blocks with Monokai Theme

Additionally, I wanted to implement a dark theme like Monokai in VS Code.
I found a theme called Xonokai that was close enough to what I wanted
https://github.com/PrismJS/prism-themes . I also created a basic styles.css to do some additional tweaking on the Xonokai theme.

/*styles.css*/

/* Hide php tag in code block */
#main > pre::after {
  display: none;
} 
#main > pre, #main > pre > code {
  background-color: #2e2e2e; 
}
#main > pre > code {
  color: white;
}
.sidebar, .sidebar-toggle {
  background-color: rgb(228, 228, 228);
}

As expected, you can link stylesheets in your index.html. I put all of my css in a root directory _assets/css.

<!-- index.html -->
<link rel="stylesheet" href="./_assets/css/prism-xonokai.css">
<link rel="stylesheet" href="./_assets/css/styles.css">

Currently, this is what it looks like when its all said and done

Additional Setup

I haven't gone super deep into docsify, but I've found a few cool things.
There's a pretty sweet plugin for pagination within chapters that can be added using

<!-- index.html -->
<script src="//unpkg.com/docsify-pagination/dist/docsify-pagination.min.js">

A lot of customizing can be done within the window.$docsify element, here is my configuration that includes a homepage, repo link, and sidebar.

<!-- index.html -->
<script>
    window.$docsify = {
      name: '',
      repo: 'https://github.com/barrettotte/IBMi-Book',
      loadSidebar: true,
      alias: {
        '/.*/_sidebar.md': '/_sidebar.md'
      },
      homepage: 'README.md',
      search: 'auto',
      auto2top: true
    }
  </script>

This is really just scratching the surface, but you can find much more at https://docsify.js.org/#/?id=docsify

Deploying to GitHub Pages

If you put all of your book content within a docs directory of your repository, you can use your repository as a GitHub.io page with ease. To make sure that your repository isn't mistaken for a Jekyll site, create an empty file named .nojekyll.

To enable the GitHub.io page go to your GitHub repository and select master branch /docs folder in Settings > GitHub Pages > Source

My "Book"

I hesitate to call what I'm making a book, but you can find my project Learning the IBMi as a Lowly Web Developer at https://barrettotte.github.io/IBMi-Book/#/

It will be a beginner's guide to programming on the IBMi with RPGLE, CL, and more. I hope to stay motivated/focused and keep chipping away at it.

Thanks for reading this post.

Simple Util to Pull Code From IBM i

Barrett Otte — Mon, 19 Aug 2019 15:38:42 +0000

In one of my really dumb side projects I'm making, I have some IBM i code that I'd like to keep in my git repository with some other stuff. I could use the Integrated File System (IFS), but truthfully I don't know enough about IBM i yet to use it correctly.

So, in the meantime I have a quick and dirty python script that lets me pull multiple source members based on a config file, repo.json. This isn't the cleanest, but it works pretty well.

My repo.json is very basic, but lays out a basic IBM i file structure to loop over below with FTP

{
  "library": "BOLIB",
  "spfs": [
    {
      "name": "QRPGLESRC",
      "extension": "RPGLE",
      "members": [
        {
          "name": "FIZZBUZZ"
        }
      ]
    }
  ],
  "output": "./"
}

Basic Idea

If you connect to IBM i over SSH you can navigate and find your source members. For example, I'm going to drill down into my QRPGLESRC file within my BOLIB library.

As you can see, there are my QRPGLESRC members hanging out. All my script does is automate grabbing the source members.

The Code

To start, I use a few basic modules found in the python standard library.

import ftplib  # easily setup FTP connection
import json    # read in repo.json config file
import getpass # used to discretely get password
import os      # used to safely make directories for output

I load my config file, instantiate my FTP client, and get the hostname, username, and password from the console

config = {}
with open("./repo.json", 'r') as f: 
  config = json.load(f)

ftp_client = ftplib.FTP()
host = input("Enter Host: ")
user   = input("Enter User: ")
password = getpass.getpass("Enter Password: ")
# FTP logic below...

Next, I setup my FTP connection shell with very basic error handling

try:
  ftp_client.connect(host, timeout=10000)
  ftp_client.login(user, password)
  # The meat of the code here ...
except ftplib.all_errors as e:
  print("Error occurred with FTP.\n" + str(e))
  exit(1)
except Exception as e:
  print("Some other error occurred\n" + str(e))
  exit(1)
finally:
  ftp_client.quit()

Now the fun stuff, sending FTP RETR command to get data from IBM i

lib = config['library'] # my example only has one library, so no looping

for spf in config['spfs']:
  print("Fetching member(s) from {}/{}".format(lib, spf['name']))

  if not os.path.exists('./'+spf['name']): 
    os.makedirs(spf['name']) # make a directory based on source physical file name

    for mbr in spf['members']:
      resp = []

      # The magic command to get data  ex: BOLIB/QRPGLESRC/FIZZBUZZ.mbr
      cmd = "RETR {}".format("/QSYS.lib/{}.lib/{}.file/{}.mbr").format(
        lib, spf['name'], mbr['name'])

      ftp_client.retrlines(cmd, resp.append) # run the command

      # Create file based on specified extension ex: RPGLE
      filepath = spf['name'] + '/' + mbr['name'] + '.' + spf['extension']

      # Finally, write data to file
      with open(filepath, 'w+') as f:
        for line in resp: f.write(str(line) + '\n')

      print("  Saved " + filepath)

Final Script

import ftplib  # easily setup FTP connection
import json    # read in repo.json config file
import getpass # used to discretely get password
import os      # used to safely make directories for output

config = {}
with open("./repo.json", 'r') as f: 
  config = json.load(f)

ftp_client = ftplib.FTP()
host = input("Enter Host: ")
user   = input("Enter User: ")
password = getpass.getpass("Enter Password: ")

try:
  ftp_client.connect(host, timeout=10000)
  ftp_client.login(user, password)

  lib = config['library'] # my example only has one library, so no looping

  for spf in config['spfs']:
    print("Fetching member(s) from {}/{}".format(lib, spf['name']))

    if not os.path.exists('./'+spf['name']): 
      os.makedirs(spf['name']) # make a directory based on source physical file name

    for mbr in spf['members']:
      resp = []

      # The magic command to get data  ex: BOLIB/QRPGLESRC/FIZZBUZZ.mbr
      cmd = "RETR {}".format("/QSYS.lib/{}.lib/{}.file/{}.mbr").format(
        lib, spf['name'], mbr['name'])

      ftp_client.retrlines(cmd, resp.append) # run the command

      # Create file based on specified extension ex: RPGLE
      filepath = spf['name'] + '/' + mbr['name'] + '.' + spf['extension']

      # Finally, write data to file
      with open(filepath, 'w+') as f:
        for line in resp: f.write(str(line) + '\n')

      print("  Saved " + filepath)

except ftplib.all_errors as e:
  print("Error occurred with FTP.\n" + str(e))
  exit(1)
except Exception as e:
  print("Some other error occurred\n" + str(e))
  exit(1)
finally:
  ftp_client.quit()

The File was Fetched!

This is stored in ./QRPGLESRC/FIZZBUZZ.RPGLE

       /free
       // The classic fizzbuzz program in RPGLE Free
       dcl-s num int(10);

       for num = 1 to 100;
           if %REM(num:3) = 0 and %REM(num:5) = 0;
               dsply ('num - ' + %char(num) + ' FIZZBUZZ');
           elseif %rem(num:3) = 0;
               dsply ('num - ' + %char(num) + ' FIZZ');
           elseif %rem(num:5) = 0;
               dsply ('num - ' + %char(num) + ' BUZZ');
           else;
               dsply ('num - ' + %char(num));
           endif;
       endfor;
       *INLR = *ON;

Simple Batch script

In my repository, I made a little batch script so I could pull IBM i code and commit it with the rest of my repository in one call.

@ECHO OFF
IF [%1] == [] GOTO NOMSG
python ibmi-pull.py && git add . && git commit -m "%~1" && git push origin master
GOTO END
:NOMSG
  ECHO "Enter the commit message!"
:END
PAUSE

Again, there's probably some better "tools" you could make involving the IFS, but I'm just not there yet knowledge-wise.

As an experiment, I expanded upon this simple script to grab an entire library and generate a basic git repository :
https://github.com/barrettotte/IBMi-Lib-Repo

Thanks for reading.

RPG and IBM i

Barrett Otte — Fri, 02 Aug 2019 16:43:46 +0000

This is my first post to DEV to see if there are any developers on here who have messed around with IBM i before or are interested in learning what it is. Primarily, I will be focusing on the history of IBM i and the RPG programming language that has evolved with it.

At my current workplace, I noticed this ominous green and black screen on numerous older developers' monitors. Shortly after, I found out that my company develops and maintains an IBM i system with an RPG(LE) codebase. I was immediately intrigued and set out to learn what was happening. The problem is that learning a new paradigm of thinking is difficult.

Since I was a web developer and expressed a bit of interest in learning IBM i, I was given a user account and library that I could mess around with. I searched around various sites and some IBM documentation for a "tutorial". But, I quickly learned that there really isn't too great of a starting point compared to learning every other programming language and/or OS.

An Important Lesson

If you ever find yourself wanting to learn RPG by purchasing a used AS/400 on eBay, don't do it (unless you have an additional $20K). IBM is cool, but their software is expensive and closed source. So, I have a pretty sweet 500lb bar table in my basement now.

Brief Introduction

IBM i is an OS that works very well with business applications, reporting, data warehousing, and more. It also comes with a built in database called DB2 which uses its own little flavor of SQL.

IBM i also has the ability to use many languages such as RPG, CL, DDS, COBOL, BASIC, C/C++, Java, Python, DB2 SQL, and REXX. There is a lot to this machine, but at a high level this amount of information should be sufficient.

A big selling point of IBM i is its stability. According to developers at my workplace, our IBM i system has only needed to be rebooted once in its 30+ years of operation.

Brief History

IBM i evolved a lot over time and often gets confusing trying to google things. The easiest way to trace its evolution is to follow the RPG programming language.

1959ish IBM 1400 series - RPG I
- Punch cards and magnetic tape
- Also supported COBOL, FORTRAN, and some ancient ASM languages.
1969-1977 IBM System/3 - RPG II
- Also supported COBOL, FORTRAN, BASIC
1978 IBM System/38 - RPG III
- Disk storage, diskette drive, etc.
- Also supported COBOL, BASIC, PL/I
1988 Application System/400 - AS/400 - RPG 400/IV
- Also supported ASM, C/C++, COBOL, Python, REXX, and more.
2000 - renamed to iSeries
2006 - renamed to System i

I glossed over a lot of information, but feel free to read more.

RPG

So, wait...RPG has been around since around 1960?

Yeah.

RPG (Report Program Generator) started out on punch cards as a programming language that was used for generating reports. Over its life span, RPG received new features/changes and can currently be used like a general purpose programming language.

RPG fixed-format looks so ugly because of its origin. Therefore, RPG's grammar is dependent on certain columns needing specific data.

Gradually, RPG evolved out of its fixed-format ancestry and took a free-format appearance. IBM made sure that there was a decent amount of backwards compatibility between version of RPG. For example, on the AS/400 you could have a mix of RPG III and RPG IV programs. With RPG's evolution into free-format

RPG is sometimes referred to as RPGLE. Without getting into too much detail: On IBM i, there is the concept of Integrated Language Environment (ILE) languages. ILE languages can talk to other ILE languages. This includes CL, COBOL, C/C++, RPG, and more. Modern RPG development is typically using RPGLE.

Hello World in fixed-format RPG (RPG III)



      C                     MOVEL'HELLO'   HELLO  11
      C                     MOVE 'WORLD'   HELLO
      C           HELLO     DSPLY          WAIT    1
      C                     SETON                     LR

Hello World in fixed-format RPGLE (RPG IV)



      C     'Hello World' DSPLY
      C                   SETON                                        LR

Hello World in free-format RPGLE (RPG IV)



       /free
       dsply ('Hello World');
       *inlr = *on

With the use of the /free and /end-free you can mix and match fixed and free RPGLE.

Conclusion

This is just a small glimpse at IBM i. There is a mountain of information to learn about the machine, most of which I haven't learned yet. I know this was probably confusing, but I hope my writing was coherent enough to enjoy reading. If anyone would like to read and learn more about IBM i let me know in the comments. I'd love to keep making posts like this.

Possible future posts:

What is Control Language (CL) ?
What is Data Description Specification (DDS) ?
IBM i file structure, Physical Files, and Logical Files
Calling IBM i from Java, Python, and more
Writing a basic RPGLE program

My IBM i "Projects"

To start learning I made a small repository to dump random files and notes into; Located at https://github.com/barrettotte/IBM-RPG

I am writing a small "book" called "Learning IBM i as a Lowly Web Developer" (obviously the title is poking fun). This book will be a casual guide to IBM i , introductions to various obscure languages (RPGLE, CL, DDS, REXX, COBOL, etc), and a short course to wrap-up concepts.

To go along with my book, I am casually developing an IBM i VS Code extension and editor API to enable viewing, editing, and compiling IBM i code from the comfort of VS Code.

Thank you for reading.