Forem: namuan

Bash/Zsh function to export SVG diagrams using PlantUML

namuan — Sun, 07 Jun 2020 11:56:52 +0000

In this short clip, we'll look at creating a simple Bash/Zsh function to export SVG diagrams using PlantUML.

Requirements:

PlantUML: Direct Download

Place it where you can easily refer to it.

nmn$ ls -l $TOOLS_HOME/plantuml/plantuml.jar
-rw-r--r--@ 1 nmn  staff  7985304  9 Nov  2019 /Users/nmn/tools/plantuml/plantuml.jar

Next, we'll create a function and put that in bash/zsh profile so that it can be loaded automatically on each session.

function plantsvg() {
    NAME="$1"
    cat ${NAME} | java -DPLANTUML_LIMIT_SIZE=8192 -jar $TOOLS_HOME/plantuml/plantuml.jar -tsvg -pipe > ${NAME}.svg
    open -a Firefox.app ${NAME}.svg
}

Once you have it setup, this will export the provided plantuml file to an SVG image and open it up using Firefox.
You can change the application used to open this SVG by replacing Firefox.app with the another application.

Here it is in action

📝:

👉 Tested on MacOS but should work on earlier versions and all flavours of Linux

Handy Bash/Zsh function to generate PlantUML diagrams

namuan — Sat, 06 Jun 2020 20:56:11 +0000

In this short clip, we'll look at creating a simple Bash/Zsh function wrapping PlantUML so that we can use it from anywhere.

Requirements:

PlantUML: Direct Download

Download it where you can easily refer to it.

nmn$ ls -l $TOOLS_HOME/plantuml/plantuml.jar
-rw-r--r--@ 1 nmn  staff  7985304  9 Nov  2019 /Users/nmn/tools/plantuml/plantuml.jar

Next, we'll create a function and put that in bash/zsh profile so that it can be loaded automatically on each session.

function plantuml() {
    NAME=$1
    cat $NAME | java -DPLANTUML_LIMIT_SIZE=8192 -jar $TOOLS_HOME/plantuml/plantuml.jar -pipe > $NAME.png
    open $NAME.png
}

Then you can call this function with the file name.

plantuml sample.png

Once you have it setup, this will convert the provided plantuml file to a PNG image and open it up using the default PNG viewer.

Here it is in action

📝:

👉 Tested on MacOS but should work on earlier versions and all flavours of Linux

Using EasyRandom with Spring Framework

namuan — Sat, 06 Jun 2020 20:52:08 +0000

Tired of creating objects in tests. Try using EasyRandom library. Here is how easily you can add it to a Spring Java project

Step 1: Add gradle/maven dependency

https://mvnrepository.com/artifact/org.jeasy/easy-random

Step 2: Declare it in test

Step 3: Use it to generate random data objects

Step 4: 😍

Tips:

Random initial seed

In the default setup, EasyRandom uses a default seed value to generate randam data.
Although it is quite simple to initiate EasyRandom with a random seed.

Enforcing javax.validation rules:

If an object is using javax.validation then there is an extension which enforces those rules when generating random objects.

Include https://github.com/j-easy/easy-random/wiki/bean-validation-support

Github: https://github.com/namuan/dev-rider-codesnippets/tree/master/spring-boot-snippets

January 2020 Progress Report

namuan — Mon, 03 Feb 2020 17:59:42 +0000

Hopefully first of many as January was the most productive month in terms of Github commits.

Here is a brief summary of the things

🚀 Started DevRider. Please consider contributing if you are interested in Python and PyQt. Contributions for other languages needed as well. See this post on my motivation behind building it.

🚀 Created Windows Release Package for HttpRider

🚀 Started MindBooks - A collection of thought provoking books.

🚀 Started new series on Tooltips (Currently posting on twitter but website coming soon)

February Goals:

📝 Add links to MindBook to buy books

📝 Promote Mindbooks on ProductHunt, IndieHackers and other channels

📝 Publish around 30 tooltips

📝 Move tooltips to its own website

I usually post more frequently on https://twitter.com/deskriders_twt so please consider following there for updates 🙏

Simple setup to remember one-liners/code snippets

namuan — Mon, 27 Jan 2020 10:43:04 +0000

Here is a simple setup explaining a comment I made on Reddit.

All I have is a notes.txt under my home directory where the snippets (one-liners) are stored in the following format.
Note that I don't store block of code or longer text here as they'll become difficult to search.

find . -type f -name '*.txt' -exec sed -i '' s/this/that/g {} + # search and replace sed find

It starts with the command then # and a brief comment on what it does.

The notes.txt file is just a soft link to a file stored in a Dropbox shared folder so that it can be synced across devices.

Here are a couple of bash helpers to work with the notes file.

fnote

fnote=find note

A simple function defined in a bash profile to search within the notes file.

function fnote() { grep "$@" ~/notes.txt }

Then I can do the following to filter out any lines.

$ fnote epoch
*  date +%s # epoch time

en=edit notes

A bash alias to open up notes.txt file in your favourite editor.

alias en='code ~/notes.txt'

Then just typing en will open up the notes.txt file for me to edit.

So, that's it. Pretty simple but very effective.

Let me know what you think and share if you have anything similar in your daily workflow.

Introducing DevRider :: One Desktop App for Common Development Tools

namuan — Sun, 26 Jan 2020 10:14:04 +0000

There is a serious problem with the state of simple development tools on the internet.

Most of the websites providing simple encoders/decoders/formatters/prettifiers are filled with ads and popups which makes it annoying and risky to use.

And then there is also a huge problem with privacy and leaking any private data that is used on any of these websites.

Although it is impossible to replace them altogether, let me introduce you to DevRider, a new open source to provide some of the common tools in an extensible desktop application.

Which means no tracking and it guarantees data safety as everything stays on your machine.

Along with these tools, there is also a curated code snippets repository with multiple supported languages.

Please checkout the project page to download application package for Mac or Windows.

🚨 Call for Help 🚨

The list of things that we can add is huge so appreciate anyone willing to contribute. I'm available to guide you through the application architecture and what is involved in making appropriate changes. Hopefully we'll learn something useful in the process 🙌.

⚠️ Add more languages for code snippets

DevRider supports curated code snippets for a tool. For eg, if you are using Base64 Encoder then it'll show you the source code to do base64 encoding in different languages.

So if you are looking to learn a new language then please don't hesitate to provide code samples for some of the tools already exists.

See https://github.com/namuan/dev-rider/issues for picking up a task as a first time contributor.

⚠️ Add more tools

If you are thinking about learning Python/PyQt then consider adding a new tool. Here is a page where I described the process of adding a Markdown Viewer https://github.com/namuan/dev-rider/blob/master/docs/devrider-adding-tool.md.

Also see the current list here but you are welcome to add anything missing from that list.

Please like the Github Project and follow https://twitter.com/deskriders_twt/ for development updates and spreading the word out.

Percent/URI Encoding - Step by Step example

namuan — Tue, 21 Jan 2020 12:10:22 +0000

Percent encoding is used to encode some reserved characters in the URI. See Wikipedia for more details.

As part of adding URL encoder in DevRider, I put together some notes which may be useful for others so publishing it here.

Here we'll take a single character : and convert it to %3A which is the equivalent encoded character. This is a very simple example and don't cover unicode characters but it is similar in principle.

We start with finding the ASCII value of : which is 58.
See ASCII table for the mapping table.

Convert 58 to binary using Short division by Two with Reminder.
i.e. Divide the answer by 2 and keeping the reminder aside which will become the binary form.

58/2 = 29 => Reminder 0
29/2 = 14 => Reminder 1
14/2 = 7  => Reminder 0
7/2  = 3  => Reminder 1
3/2  = 1  => Reminder 1
1/2  = na => Reminder 1

So the binary form is 111010 if we line up all the reminders.
Padding the output will make it 00111010.

Next, we split the binary into two parts of 4 bits each.

0011 1010

Then using the following Hex table conversion, we map 0011 => 3 and 1010 to A which becomes 3A.

Decimal	Binary	Hex
1	0001	1
2	0010	2
3	0011	3
4	0100	4
5	0101	5
6	0110	6
7	0111	7
8	1000	8
9	1001	9
10	1010	A
11	1011	B
12	1100	C
13	1101	D
14	1110	E
15	1111	F

The final part is to prefix it with % (which is the escape character) making it %3A which is the output we are expecting. 🎉

Overlay button on a PyQt5 widget

namuan — Thu, 02 Jan 2020 10:33:51 +0000

This is a short post on how to add a floating button on top of a widget in PyQt5.

First, we'll define the button and the relative positioning.

class FloatingButtonWidget(QtWidgets.QPushButton): #1

    def __init__(self, parent):
        super().__init__(parent)
        self.paddingLeft = 5
        self.paddingTop = 5

    def update_position(self): 
        if hasattr(self.parent(), 'viewport'):
            parent_rect = self.parent().viewport().rect()
        else:
            parent_rect = self.parent().rect()

        if not parent_rect:
            return

        x = parent_rect.width() - self.width() - self.paddingLeft
        y = self.paddingTop #3
        self.setGeometry(x, y, self.width(), self.height())

    def resizeEvent(self, event): #2
        super().resizeEvent(event)
        self.update_position()

    def mousePressEvent(self, event): #4
        self.parent().floatingButtonClicked.emit()

Inheriting from QPushButton.
Overriding resizeEvent and updating position of the this button based on the parent position.
This will add the button to the top. If you need to place it at the button then use something like parent_rect.height() - self.paddingLeft.
Catching the mouse pressed event and calling an signal on the parent widget.

With the button defined, we'll embed it in a custom QPlainTextEdit and define appropriate interfaces for the caller to work with the floating button.

class OverlayedPlainTextEdit(QtWidgets.QPlainTextEdit): #1
    floatingButtonClicked = QtCore.pyqtSignal() #2

    def __init__(self, parent=None):
        super().__init__(parent)
        self.floating_button = FloatingButtonWidget(parent=self)

    def update_floating_button_text(self, txt):
        self.floating_button.setText(txt) #3

    def resizeEvent(self, event):
        super().resizeEvent(event)
        self.floating_button.update_position() #4

Inheriting from QPlainTextEdit although this could be any widget.
Defining a custom signal which is triggered when a mouse pressed event is captured in the floating button as we saw earlier.
Updating the text of the button which we'll see later.
Overriding resizeEvent so that we can update the position of the floating button.

Now we can use this class instead of QPlainTextEdit and update the floating button.

self.plainTextEdit.floatingButtonClicked.connect(self.on_test)
self.plainTextEdit.update_floating_button_text("Edit")`

That is it. Here is the full source code. Github Gist

import sys

from PyQt5 import QtCore
from PyQt5 import QtWidgets


class FloatingButtonWidget(QtWidgets.QPushButton):

    def __init__(self, parent):
        super().__init__(parent)
        self.paddingLeft = 5
        self.paddingTop = 5

    def update_position(self):
        if hasattr(self.parent(), 'viewport'):
            parent_rect = self.parent().viewport().rect()
        else:
            parent_rect = self.parent().rect()

        if not parent_rect:
            return

        x = parent_rect.width() - self.width() - self.paddingLeft
        y = self.paddingTop
        self.setGeometry(x, y, self.width(), self.height())

    def resizeEvent(self, event):
        super().resizeEvent(event)
        self.update_position()

    def mousePressEvent(self, event):
        self.parent().floatingButtonClicked.emit()


class OverlayedPlainTextEdit(QtWidgets.QPlainTextEdit):
    floatingButtonClicked = QtCore.pyqtSignal()

    def __init__(self, parent=None):
        super().__init__(parent)
        self.floating_button = FloatingButtonWidget(parent=self)

    def update_floating_button_text(self, txt):
        self.floating_button.setText(txt)

    def resizeEvent(self, event):
        super().resizeEvent(event)
        self.floating_button.update_position()


# NOTE: This class is generated from QtDesigner
class Ui_DebugWindow(object):
    def setupUi(self, DebugWindow):
        DebugWindow.setObjectName("DebugWindow")
        DebugWindow.resize(622, 498)
        self.centralwidget = QtWidgets.QWidget(DebugWindow)
        self.centralwidget.setObjectName("centralwidget")
        self.verticalLayout = QtWidgets.QVBoxLayout(self.centralwidget)
        self.verticalLayout.setObjectName("verticalLayout")
        self.plainTextEdit = OverlayedPlainTextEdit(self.centralwidget)
        self.plainTextEdit.setObjectName("plainTextEdit")
        self.verticalLayout.addWidget(self.plainTextEdit)
        DebugWindow.setCentralWidget(self.centralwidget)
        self.menubar = QtWidgets.QMenuBar(DebugWindow)
        self.menubar.setGeometry(QtCore.QRect(0, 0, 622, 22))
        self.menubar.setObjectName("menubar")
        DebugWindow.setMenuBar(self.menubar)
        self.statusbar = QtWidgets.QStatusBar(DebugWindow)
        self.statusbar.setObjectName("statusbar")
        DebugWindow.setStatusBar(self.statusbar)

        self.retranslateUi(DebugWindow)
        QtCore.QMetaObject.connectSlotsByName(DebugWindow)

    def retranslateUi(self, DebugWindow):
        _translate = QtCore.QCoreApplication.translate
        DebugWindow.setWindowTitle(_translate("DebugWindow", "MainWindow"))


class DebugWindow(QtWidgets.QMainWindow, Ui_DebugWindow):
    edit_mode = True

    def __init__(self, parent=None):
        super(DebugWindow, self).__init__(parent)
        self.setupUi(self)
        # ui events
        self.plainTextEdit.floatingButtonClicked.connect(self.on_test)
        self.update_floating_button_text()

    def on_test(self):
        self.edit_mode = not self.edit_mode
        self.plainTextEdit.appendPlainText("Button Clicked - Mode: {}".format("Edit" if self.edit_mode else "Preview"))
        self.update_floating_button_text()

    def update_floating_button_text(self):
        if self.edit_mode:
            self.plainTextEdit.update_floating_button_text("Edit")
        else:
            self.plainTextEdit.update_floating_button_text("Preview")


if __name__ == "__main__":
    app = QtWidgets.QApplication([])
    window = DebugWindow()
    window.show()
    sys.exit(app.exec_())

Developing API exchange sharing service - Part II

namuan — Wed, 01 Jan 2020 14:07:21 +0000

Please check the previous part to know the background of this project.

In this part, we'll dive into code and deploy a simple API to AWS Lambda and Api Gateway.

I'll be using Python Flask framework for defining the APIs and Serverless framework to manage the infrastructure and deployment.

Please make sure you have Python3 and relatively newer version of nodejs installed and working to follow along.

The first step is to create a new project folder and initialise package.json file which will be used to define and install Serverless plugins.

$ mkdir api-exchange-server
$ cd api-exchange-server

$ npm init
$ # Enter all the way to generate package.json file

Installing Serverless CLI.

$ npm install -D serverless # Although better to install this globally

If Serverless CLI is installed globally then you don't need to prefix the serverless commands with ./nodes_modules/.bin/.

Let's create a new project

$ ./node_modules/.bin/serverless
Name the project serverless-project. Ignore the project name as we'll move the files to the current folder.

We'll just copy the files that we need from the generated project.

$ mv serverless-project/serverless.yml .
$ mv serverless-project/.gitignore .
$ rm -rf serverless-project

After removing the folder generated by the Serverless CLI, this is how the structure looks like

$ ls -la
total 336
drwxr-xr-x    7 nmn  staff     224  1 Jan 09:00 .
drwxr-xr-x   15 nmn  staff     480  1 Jan 08:53 ..
-rw-r--r--    1 nmn  staff     192  1 Jan 08:58 .gitignore
drwxr-xr-x  386 nmn  staff   12352  1 Jan 08:54 node_modules
-rw-r--r--    1 nmn  staff  156725  1 Jan 08:54 package-lock.json
-rw-r--r--    1 nmn  staff     271  1 Jan 08:54 package.json
-rw-r--r--    1 nmn  staff    3206  1 Jan 08:58 serverless.yml

Let's open the folder in your favourite text editor.

To connect AWS Lambda to Flask handlers, we'll need to install serverless-wsgi plugin to do the necessary transformation.

$ ./node_modules/.bin/serverless plugin search -q wsgi
1 plugin(s) found for your search query "wsgi"

serverless-wsgi - Serverless plugin to deploy WSGI applications (Flask/Django/Pyramid etc.) and bundle Python package

To install a plugin run 'serverless plugin install --name plugin-name-here'

It will be automatically downloaded and added to your package.json and serverless.yml file

As stated by the really useful help message, we'll install this plugin.

$ ./node_modules/.bin/serverless plugin install --name serverless-wsgi

Next, we'll create a simple flask Api which will return the environment variables in json. As this code will run on Lambda, you'll be able to see all the environment variables available for the function.

File -> New -> "api.py"

import os
from flask import Flask, jsonify

app = Flask(__name__)


@app.route("/", methods=["GET"])
def environ():
    return jsonify({k:v for k, v in os.environ.items()})

Just noticed that we haven't installed Flask yet so let's create a requirements file and install it in a new virtualenv.

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install flask
$ pip freeze > requirements.txt

With the project completed, we can test the Api by running it locally.

$ FLASK_APP=api.py flask run

Next we need to tell/configure Serverless framework to connect Lambda to Flask App. There are few ways to setup the integration.
Here we'll be using AWS Lambda Proxy integration for the API handler.

functions:
  api:
    handler: wsgi_handler.handler
    events:
      - http: ANY /
      - http: ANY {proxy+}

We also need to tell wsgi plugin where to find the Flask API, so let define a custom section for this.

custom:
  wsgi:
    app: api.app

Although the wsgi plugin can manage python dependencies, I had better luck with using a different plugin serverless-python-requirements so we'll install it as well

$ ./node_modules/.bin/serverless plugin install --name serverless-python-requirements

Now let's try packaging up the app.

$ ./node_modules/.bin/serverless package

If successful, the generated zip file will be placed in .serverless folder. If we examine that file, we'll find that it contains all the files in the current folder including the giant venv and node_modules folder 😱.

I'm sure python requirements folder is right up there as well.

Luckily, we can configure what to include in the zip file so we'll exclude everything and just add the file that we need to run the application. Note that the plugin will still take care of all the python dependencies defined in requirements.txt file.

package:
  exclude:
    - "**"
  include:
    - "api.py"

This is what it looks like with the changes that we made in the serverless.yml file.

You can also check the complete file.

That should be everything to deploy on AWS Lambda. The next step requires an AWS profile with appropriate permissions to deploy and create Lambda resources.

Note that this is where you may be charged.

$ AWS_PROFILE=myprofile ./node_modules/.bin/serverless deploy -v

It will take some time to create all the resources before printing out the endpoints that it generated for the Api Gateway.

You should be able to hit the above endpoint via curl or open it up directly in the browser. There you can see all the environment variables available to your lambda function.

curl -v https://mcb1lbc2ti.execute-api.us-east-1.amazonaws.com/dev
# or 
open https://mcb1lbc2ti.execute-api.us-east-1.amazonaws.com/dev

Once you are done with it, don't forget to remove the setup.

$ AWS_PROFILE=myprofile ./node_modules/.bin/serverless remove -v

Developing API exchange sharing service - Part I

namuan — Wed, 01 Jan 2020 13:55:36 +0000

In this series of articles, I'll go through my experience of building a simple Serverless API using Flask python framework and deploying it on AWS Lambda and API gateway.

This API is used from HttpRider to save API exchanges (request+response).

Here is a sample page generated from HttpRider.

Demo

The service itself consists of two simple APIs.

POST /prints

Saves the HTML encoded in base64 in dynamo database. Once it is saved, it returns the URL of the shared document in the Location header of the response.

POST /prints

Headers

Content-Type: application/json

Body

{
    "document": "SHR0cFJpZGVy"
}

Response

HTTP 201

Headers

location: http://localhost:8080/prints/025000cf-14ba-421b-a000-d2d043d4d90b

GET /prints/{print-id}

This GET request uses the URL from the location header in the previous request and receives the HTML response with the document.

GET /prints/025000cf-14ba-421b-a000-d2d043d4d90b

Headers

Accept: text/html

Response

HTTP 200

Headers

content-type: text/html; charset=utf-8

Body

<!DOCTYPE html>
<html>
<head>
    <title>Deskriders :: API Print</title>
    <style>
       .... removed to save space ...
    </style>
</head>
<body>
<div>HttpRider</div>
</body>
</html>

In the next part, we will look at setting up a new Flask project and serverless framework configuration to deploy it to AWS Lambda.

HttpRider :: Using GitHub APIs

namuan — Sun, 08 Dec 2019 18:22:19 +0000

Here is a sneak preview of HttpRider, a Rest API client that I'm working on and using it as my primary Rest API client for a while.

In future, I'm planning to write more about the different feature set and what makes it unique across the different products in the same space.

Interested? Have a look at the README file for feature list and give it a try.

Let me know if you have any issues or any suggestions for improvement.

HttpRider is open source and built using Python and PyQT5 so should work across platforms ¹.

I'm mainly using it on Mac but there is nothing which should make it platform dependent. ↩

Protecting applications with OAuth2 Proxy

namuan — Sat, 23 Nov 2019 21:30:44 +0000

Here is a simple guide on protecting a website with social logins supporting OAuth2. It uses an open source OAuth2 Proxy (which is a fork from Bitly OAuth2 Proxy) to secure private applications without adding any authentication logic in the application itself.

Here, we'll see how to secure the open source version of https://httpbin.org. Httpbin application itself is open and can be accessed without any authentication.

In this guide, we will setup a Github OAuth application but the same setup can be easily configured with any other provider. See Providers for different config options.

Setting up Github Application:

I'll call this application SocialBin and set the homepage as http://localtest.me. Localtest.me is a DNS configured to point to 127.0.0.1 and makes it easy to do local testing.

We will be running the application on port 8080 so the callback url is set as http://localtest.me:8080/oauth2/callback.

That is it for setting up the application. Note down the Client ID and Client Secret which will be used later.

Running application with proxy:

Clone this Github project and then copy or move the example configuration file.

cp .env.example .env

You can also run the following command to generate a random value which will be used for cookie secret.

head -c32 /dev/urandom | base64

Edit the .env file and update the variables.

OAUTH2_PROXY_COOKIE_SECRET=<generated random value>
OAUTH2_PROXY_COOKIE_DOMAIN=http://localtest.me
OAUTH2_PROXY_CLIENT_ID=<github client id>
OAUTH2_PROXY_CLIENT_SECRET=<github client secret>

Now, switch over to docker-compose.yml which is setting up two docker containers. The httpbin container is the service we are trying to protect so you can see that it is not exposing any ports. The "--upstream=http://httpbin:80", configuration is telling the oauth2 proxy where to send the traffic once the login is successful.

With everything in its place, we can bring up the containers using docker-compose.

docker-compose up

It'll take some time if it is running for the first time to pull any missing docker images. Once everything is running, you can visit http://localtest.me:8080/ in the browser which should show you a screen with a button to "Sign in with Github". Here is a brief sequence of the login flow and accessing httpbin.

Conclusion:

As we saw, OAuth2Proxy is a useful tool to protect any application without changing the code. The source code for this setup is available here https://github.com/namuan/oauth2-proxy-httpbin.

References:

https://github.com/pusher/oauth2_proxy
https://readme.localtest.me/
https://hub.docker.com/r/kennethreitz/httpbin
https://hub.docker.com/r/bitnami/oauth2-proxy