Forem: Muhammad Syuqri

saferm - A tool to safely remove files on CLI

Muhammad Syuqri — Fri, 28 Aug 2020 09:15:49 +0000

Introduction

I started this toy project to learn about Golang and have some experience using it. Test-Driven Development and GitHub Actions were side learning points for this project.

I have been doing development in the past which did not prioritise automated testing. As a result, extensive manual testing at the end of each development iteration was required and it definitely was a pain. I wanted to take a different approach and learn the methodology behind TDD so that each iteration of the development process can go through automated testing and can be published with confidence.

I decided to go into GitHub Actions by submitting my first GitHub Action - Newbiefy (Wacky Wildcards Category) - as part of the DEV x GitHub Actions Hackathon. This toy project builds on the knowledge from that experience.

What is `saferm`?

You know when you want to delete a repository from GitHub, they ask for the name of the repository as confirmation of the deletion so that they know it is by choice, not accident? saferm does exactly that when you are trying to remove files from a directory using the command-line. Each file it discovers will prompt the user to input the filename before deletion.

Why `saferm`?

Apart from the learning reasons as mentioned before, there is one thing that I found in the rm command.

The conventional rm command does have an interactive mode using the -i flag. However, this only prompts the user for a Y/y + Enter response.

The purpose of saferm is to make the deletion process more painstaking so that the user will take ownership of the deletion.

Check it out here

Dansyuqri / saferm

This repository is a project for me to explore Go.

saferm

A command-line tool for removing files from a directory in a safe fashion. Written in Go.

Installation

First, install or update the saferm binary into you $GOPATH like so:

$ go get -u github.com/dansyuqri/saferm

Usage

You can immediately use the saferm command on your cli.

$ saferm

By default, the saferm works on the current working directory. In order to specify another directory, you will need to supplement the -p flag like so:

$ saferm -p /home/user/Downloads

The above command immediately starts giving you a filename for you to input in order to delete the file:

$ saferm -p /home/user/Downloads
Enter filename to confirm deletion
testfile.txt

Once you have entered the exact filename, it will proceed to delete the file. This painstaking process of typing the exact name of the file will assure you that the deletion is by choice, not accident (similar to how deletion of a repository…

View on GitHub

Conclusion

Definitely open to feedback and suggestions, be it on the code or TDD methodology that I was trying to adopt. Leave them below thank you!

Why Python Generators?

Muhammad Syuqri — Fri, 28 Aug 2020 05:33:24 +0000

In the usual use case of iterating over an iterable object such as a list, string, or dict, we can simply use a for loop to go through each element.

my_numbers = [ x for x in range(5) ]

for num in my_numbers:
    print(num)  # prints numbers from 0 to 4

We can also achieve a similar result using generators. To instantiate a generator, one can simply change the [] to () like so:

my_numbers_gen = ( x for x in range(5) )  # this is now a generator

print(my_numbers_gen)
# ouput >>> <generator object my_numbers_gen.<locals>.<genexpr> at 0x7f53ebcb1f20>

for num in my_numbers_gen:
    print(num)  # prints numbers from 0 to 4

We can see that my_numbers_gen is of the type generator object. When we iterate through the generator object and the list, we can see that there is no difference in the output. Why would you then want to use a generator?

Reasons to use a generator

Reduce memory usage when iterating over large objects

When we iterate over a large list, say with 1 million elements, we can use memory profiler to see that the memory usage jumps from 13.992 MiB to 52.895 MiB.

syuqri@pop-os:~/Documents$ python3 -m memory_profiler test.py 
Filename: test.py

Line #    Mem usage    Increment   Line Contents
================================================
     2   13.992 MiB   13.992 MiB   @profile
     3                             def test_func():
     4   52.895 MiB    0.309 MiB       test_list = [x for x in range(10**6)]
     5   52.895 MiB    0.000 MiB       for i in test_list:
     6   52.895 MiB    0.000 MiB           pass

On the other hand, the generator version of this sees no increment in the memory usage. This is because an iterator does not load all of its values into memory at once. Instead, it loads one value and then proceeds to the next.

syuqri@pop-os:~/Documents$ python3 -m memory_profiler test_gen.py 
Filename: test_gen.py

Line #    Mem usage    Increment   Line Contents
================================================
     1   14.000 MiB   14.000 MiB   @profile
     2                             def test_func_gen():
     3   14.000 MiB    0.000 MiB       test_gen = (x for x in range(10**6))
     4   14.000 MiB    0.000 MiB       for i in test_gen:
     5   14.000 MiB    0.000 MiB           pass

You need to use intermediate values from a function

In a typical function, you can have multiple return statements, but ultimately, only one of those return statements holds true.

def normal_func():
    if True:
        return "hello"
    return "goodbye"

test = normal_func()
print(test)
>>> hello

When we run the above snippet, we can see the output is always "hello". What if we want both strings to be printed out?

Well, that is where generators can come in handy. The major difference change needed to the above function to make it a generator is to use yield instead of return.

def gen_func():
    if True:
        yield "hello"
    yield "goodbye"
test = gen_func()
print(next(test))
print(next(test))
>>> hello
>>> goodbye

Since the function is a generator, we can iterate through the different yield statements of the function using next().

References

https://www.programiz.com/python-programming/generator

Newbiefy - Make Your Repo Beginner Friendly Again

Muhammad Syuqri — Fri, 21 Aug 2020 05:51:46 +0000

My Workflow

Makes your repository beginner friendly by removing everything in place of a hello_world file in the mostly used language of the repo.

Submission Category:

Wacky Wildcards

Yaml File or Link to Code

Dansyuqri / newbiefy

Makes your repository beginner friendly by removing everything in place of a hello_world.<ext> in the mostly used language of the repo.

Newbiefy

The Action you never knew you needed.

What

Makes your repository beginner friendly by removing all source code and putting a hello world file (in the most used language of the repo) in its place.

Why

Why not

Usage

- uses: dansyuqri/newbiefy@latest
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}

Additional Info

Currently it supports the following languages:

Java
Python
C
C++
C#
Visual Basic .NET
JavaScript
PHP
Objective-C
SQL
Ruby
MATLAB

The codes have been referred from this page

Demo

Head over to the test page for a demo.

View on GitHub

Additional Resources / Info

https://github.com/Dansyuqri/newbiefy-test
source codes for the different languages were from here

Pub/sub with PyZMQ: Part 2

Muhammad Syuqri — Sun, 12 Apr 2020 10:33:18 +0000

Hello everyone! I hope everyone is keeping healthy and staying safe. I hope that you have had a good meal before this, as I will be enticing you with a couple of breakfast analogies for this PyZMQ tutorial :P

This post is a continuation in the "Pub/sub with PyZMQ" series. Part 1 can be found here. This part will build upon what we have discussed in the previous one. Do feel free to revisit Part 1 as a recap or if you are new to this series.

The overview for this post will be as follows:

Multiple Subscribers to One Publisher
Single Subscriber to Multiple Publishers
Using Poller to Prevent Code Blocking

Multiple Subscribers to One Publisher

Multiple subscribers to one publisher diagram

This pattern is extremely useful for when the same data is required by multiple subscribers from a common publisher. An example use case would be a home automation system that consists of appliances connected to smart plugs. Let's say you wake up in the morning and would like your breakfast to be prepared automatically while you have your shower. You would like for your toaster and coffee maker to be started as soon as you turn on the lights in your bathroom.

Instead of using the server/client pattern in which both the toaster and coffee maker constantly requests for the status of the light switch in your bathroom, the light switch instead tells both the toaster and coffee maker when it is time to turn on.

Toaster and coffee maker analogy

Using the Thread class from the threading module in Python, we can simulate this scenario. First, we create the two subscribers like so:

We create two threads which simulate the coffee maker and toaster listeners. The listeners are subscribed to the 'light' topic and await for any messages coming from the 'light' publisher.

We can then fire this script up and continue to the next step - creating the single 'light' switch publisher. As you will notice again, the threads will be blocked due to the recv_string() function in both the coffee make and toaster threads.

Once the single_pub.py script has also been run, it will have a sleep for 1 second (reason for sleep here) before the publisher sends the message to the coffee maker and toaster threads. The results can then be seen:

>>> python multi_sub.py
COFFEE MAKER received 'light is ON' from light.
TOASTER received 'light is ON' from light.

NOTE: It is better to create a new socket instance in each thread as demonstrated. This is because the socket objects are not thread-safe. The context however, can be shared. More info here.

Single Subscriber to Multiple Publishers

Single subscriber to multiple publishers diagram

We can bring in a new element into our coffee maker and toaster analogy. This time, let's include a smart mirror in your bathroom. After you are done showering, you get ready for your day ahead and prepare yourself using the smart mirror. The mirror tells you if your coffee and toast are ready or not.

Smart mirror analogy

For this set of code, we shall utilise the multipart messaging of PyZMQ. This way, we can separate the topic and the actual message that we want. Recap on the multipart messaging can be found here.

Let's start off with the single subscriber code:

The main difference between single_sub.py and single_pub.py is the multipart message implementation and that single_sub.py is receiving messages instead of sending.

The multi_pub.py code now utilises the Thread class and instantiates two threads to send messages concurrently.

We now have the topic and status of each appliance sent in separate parts of the messages. Again, the flag zmq.SNDMORE is the one that establishes that there are more parts to the message, and the receiver needs to prepare to receive the remaining messages. The topic is always the first part of the message, which makes it easier for us to distinguish topic from message. This results in the following:

>>>python single_sub.py 
Topic: COFFEE MAKER => {'status': True}
Topic: TOASTER => {'status': True}

Using Poller to Prevent Code Blocking

As you might have noticed in previous examples, all variants of the recv() function blocks the code execution. As such, it can be annoying to have to force close your terminal each time you run your code and the recv() statement is within an infinite loop.

We can overcome this by using the zmq.Poller() object. It mirrors the built-in Python poll interface. Essentially, it checks whether the file descriptor created during the socket creation in the context.socket() function call, has any pending I/O events.

We can instantiate a Poller object like so:

poller = zmq.Poller()

We then have to register a socket to the poller object so that it knows which file descriptors to check for I/O events.

poller.register(socket, zmq.POLLIN)

zmq.POLLIN indicates that the check whether any data is ready to be read or not at the socket's file descriptor.

We can then use our previous single_sub.py and build upon it. Let's call the new script simple_poller.py. We can introduce a check for events with a timeout in the while loop.

Each iteration of the loop, once the timeout limit has been reached for the poll event, the if code block is executed, before continuing into the next iteration of the loop. This way, break/interrupt events can cause the script to exit within this small time frame.

Future Guides

I hope that this second segment of the series has served as a good build on Part 1. As a recap, this post has covered:

Multiple subscriber to single publisher
Single subscriber to multiple publisher
Using poller to prevent code blocking

With these small parts, I hope that you will be able to apply some of these concepts into your own personal projects. For the next post, I do hope to cover the following:

N-to-N publisher/subscriber messaging
Implementing a Web Socket forwarder

Here are the links for the codes:

Thanks for reading! I do hope to learn from your feedback and comments. Also, do share if you have implemented any of these patterns to your projects! I would love to see them in action :D

Pub/sub with PyZMQ: Part 1

Muhammad Syuqri — Sun, 08 Mar 2020 08:54:59 +0000

This series of guides will go into the usage of the publish-subscribe pattern with PyZMQ. I will start off with simple examples and then go into some of the use cases that I have applied using this pattern. You will find Gists with sample code to get you started with.

Here are the topics I will go through for this guide:

What is publish/subscribe?
What is PyZMQ?
Getting started
Specifying topics
Multipart messages
Future Guides

What is publish/subscribe?

It is a messaging pattern that is meant for allowing senders to send their messages to recipients without having to know who their recipients are. The senders are known as publishers while the recipients are known as subscribers.

A simple analogy for this pattern would be a convention. A convention can be held in a large hall. In the hall there may be multiple sections for speakers to give their talks at. A speaker can give his/her talk regardless of whether or not anyone is listening. As attendees, we can roam around and drop in at any of these sections to listen to any speaker. When we sit in at a particular section, we are only listening to a single speaker. The speakers in this analogy refer to the publishers, while attendees refer to the subscribers.

In this guide, I will be covering a one-to-one publish/subscribe pattern, which simplifies the explanation process for now.

What is PyZMQ?

PyZMQ is the Python-based binding for the popular open-source ZeroMQ (ØMQ/ZMQ) messaging library. ZMQ comes with no messaging brokers on the side, meaning that there is no intermediary server/module which is needed to translate the messages sent between sender and recipient to a uniform messaging protocol. This makes ZMQ lightweight and easy to get started with.

ZMQ also comes with support for multiple messaging patterns apart from pubsub and programming languages, allowing cross-language communication. However, this guide will only be focusing on the pubsub pattern in Python.

Getting started

Install the pyzmq package

pip install pyzmq

We can then proceed to write a simple publisher to start sending out messages.

# simple_pub.py
import zmq

host = "127.0.0.1"
port = "5001"

# Creates a socket instance
context = zmq.Context()
socket = context.socket(zmq.PUB)

# Binds the socket to a predefined port on localhost
socket.bind("tcp://{}:{}".format(host, port))

# Sends a string message
socket.send_string("hello")

The publisher script will run and send a single message with the string "hello" and exit.

The subscriber code can be written as such:

# simple_sub.py
import zmq

host = "127.0.0.1"
port = "5001"

# Creates a socket instance
context = zmq.Context()
socket = context.socket(zmq.SUB)

# Connects to a bound socket
socket.connect("tcp://{}:{}".format(host, port))

# Subscribes to all topics
socket.subscribe("")

# Receives a string format message
socket.recv_string()

The subscriber code is very similar to the publisher one. The only difference is that the subscriber will now connect to the publisher socket to receive messages from the publisher. The subscriber can also subscribe to certain topics, but for the example above, we will subscribe to all topics indicated by the empty string.

Open up a terminal. If you try to run the simple_sub code now, you will see that the code execution is blocked. This is because socket.recv_string() is blocking the execution, waiting for a message to be received. You cannot interrupt the execution using the ctrl+Z command as well. As such, it can be quite troublesome to always have to close the terminal window to force the code to exit or run the publisher code to ensure the completion of the subscriber code. Let's use either of these simple solutions for now.

Open up a separate terminal. When you run simple_pub, you might expect that the subscriber would receive the message. However, this will not be the case as bind() executes in an asynchronous manner. This means that the entire script actually finishes even before the bind() statement manages to complete.

Let's edit simple_pub to include a sleep statement, which is a simplified solution to this problem. This way, the asynchronous bind() should run to completion before the publishing of the message.

# simple_pub.py

# ...

# Binds the socket to a predefined port on localhost
socket.bind("tcp://{}:{}".format(host, port))

time.sleep(1) # new sleep statement

# Receives a string format message
socket.send_string("hello")

Let's run the publisher code again.

Now we can see that the code execution in simple_sub has run to completion and printed out the word hello in string format. If we use the generic socket.recv(), we will end up with a bytes type.

>>> socket.recv_string()
hello
>>> socket.recv()
b'hello'

It really depends on the use case as PyZMQ provides a few functions to send and receive different serialized/encoded messages. These include:

send_json/recv_json
send_multipart/recv_multipart
send_pyobj/recv_pyobj
send_serialized/recv_serialized

More details on each of these can be found in the PyZMQ Socket docs.

Specifying topics

In the pubsub pattern, subscribing to a topic means that you only listen in for particular messages that you are interested in and disregard the rest.

Using the convention analogy again, let's say you would like to only listen to certain segments of speaker A's presentation. You are only interested in Machine Learning and Blockchain topics, but find other topics a bore. Whenever the speaker mentions Machine Learning, you take down the notes related to that topic. Whenever the speaker mentions Blockchain, you take down the notes related to that topic. Whenever the speaker mentions Dev.to, you close your notebook and ignore whatever he/she says. This is what subscribing to a topic in the pubsub pattern equates to.

To do this with PyZMQ, you can use socket.subscribe(). Previously, we have subscribed to all topics by using .subscribe(""). From the high-level perspective, the function allows us to subscribe to a topic. What we do not see are the inner workings of ZMQ, which actually creates a filter to sieve the messages that we are interested in. I will not elaborate too much on this as it can be found here for your leisurely read.

Let us now only subscribe to the topic "ML".

# simple_sub.py

# ...

# Subscribes to all topics
socket.subscribe("ML")

# Receives a string format message
socket.recv_string()

This means that any message published starting with "ML" will be received by the subscriber. So, we have to alter the messages sent by the publisher to achieve this.

# simple_pub.py

# ...

# Sends a string message
socket.send_string("ML hello")

When we run the code, we can see that the message is received by the subscriber. To demonstrate that all other topics are filtered out, you can try changing the "Dev.to" in the subscriber code to any other characters and you will see that the code execution for the subscriber will be blocked again, indicating that it has not received messages based on the topics it has subscribed to.

So now, we have the topic and the actual message that we want combined together into a single message that is sent from the publisher. We can use .split() to separate the topic and actual message in the subscriber.

# simple_sub.py

# ...

whole_message = socket.recv_string()
topic, actual_message = whole_message.split(" ")

Alternatively, we can use send_multipart() and recv_multipart() accordingly, which will be explained next.

Multipart messages

Multipart messages are sent from the publisher by including all of your message parts together with a flag which indicates more message parts are to follow. When we were using send_string(), we were only sending a single message disguised as two parts separated by a space.

Now, we shall attempt to use send_multipart() in the publisher to send two separate parts so that the topic and message can be distinguished without using any split().

# multipart_pub.py
import zmq
import time

host = "127.0.0.1"
port = "5001"

# Creates a socket instance
context = zmq.Context()
socket = context.socket(zmq.PUB)

# Binds the socket to a predefined port on localhost
socket.bind("tcp://{}:{}".format(host, port))

time.sleep(1)

# Sends a multipart message
socket.send_multipart([b"ML", b"hello"])

Note: When using .send() and .send_multipart(), the message/message parts need to be bytes type respectively. Previously when we were using send_string(), the function helped us encode the string into bytes.

# multipart_sub.py
import zmq

host = "127.0.0.1"
port = "5001"

# Creates a socket instance
context = zmq.Context()
socket = context.socket(zmq.SUB)

# Connects to a bound socket
socket.connect("tcp://{}:{}".format(host, port))

# Subscribes to all topics
socket.subscribe("ML")

# Receives a multipart message
print(socket.recv_multipart())

After running both scripts together, we can see the output of the execution in multipart_sub:

>>> print(socket.recv_multipart())
[b'ML', b'hello']

The first element in the list is the topic while the second one is the message part which we are interested in.

You can also craft your own multipart message by using the zmq.SNDMORE flag.

# multipart_pub.py

# ...

# Sends the first part
socket.send_string("ML", flags=zmq.SNDMORE)
# Sends the second part
socket.send_string("hello")

There are no flags applied to the second message because the default flag is 0, which indicates that there are now more parts following the message part being sent. The subscriber will still receive the same multipart message as shown previously.

In a more realistic scenario, you might want to send a JSON instead so that key/value retrieval can be done. To do this, simply replace send_string() in the second part with send_json(). send_json() will automatically serialize the dictionary that you pass to it into JSON format.

# multipart_pub.py

# ...

# Sends the first part
socket.send_string("ML", flags=zmq.SNDMORE)
# Sends the second part
socket.send_json({"message": "hello"})

The end result on the subscriber end would be as follows:

>>> print(socket.recv_multipart())
[b'ML', b'{"message":"hello"}']

To ease the deserialization of the JSON back into a Python dictionary, we can use the recv_json() function instead. We can replace the recv_multipart() with two lines of code as follows:

# print(socket.recv_multipart()) 

print(socket.recv_string())
print(socket.recv_json())

Which will then yield the following results:

ML
{'message': 'hello'}

Future Guides

I hope that this post serves as a good starting point for anyone who might be keen on using the pubsub pattern with PyZMQ. For this post, I have mainly focused on the following:

Single publisher to single subscriber messaging
Subscribing to topics
Sending/receiving multipart messages

For the next post, I do hope to cover the following to come up with solutions with some of the limitations touched on in this post:

1-to-N/N-to-1 publisher/subscriber messaging
N-to-N publisher/subscriber messaging
Using Poller to prevent code execution blocking

Here are the links for the codes:

Thanks for reading! I do hope to learn from your feedback and comments, so feel free do drop them below.

How to sprinkle some exercise into your developer lifestyle

Muhammad Syuqri — Fri, 01 Mar 2019 14:51:19 +0000

We are all too familiar with the excuses we give ourselves.

"Maybe I will do some exercise after work later on"

And when you reach home...

"Maybe tomorrow morning before work"

And when the next morning comes

"Oh my, I am going to be late for work"

And the cycle continues. After work, we would probably want to wind down, or take on some personal challenges and explore new technologies. We tend to have little to no time for taking care of our bodies. However, we need to remember that when we take our bodies for granted, it may affect our minds - the very tool we use every day.

This isn't a post on why you should do it. I am in no way a fitness expert. But I would like to share what has worked for me and how you might also be able to achieve little bits of exercise throughout the day, considering the hectic lifestyle you might have.

How to add mini exercises to your daily routine

Let me share with you a couple of tricks I have been able to come up with just to make myself walk more. You might think that walking doesn't do much, but if you do it on a collective and daily basis, it works wonders.

Come hooman let's walk

Take the stairs

You take the public/personal transport and arrive at your office building. Say your cubicle is on the 2nd/3rd/4th floor of the building. Why not take the stairs? Imagine heading up to the 4th floor by stairs everyday. Assuming you have to take the stairs 2 times daily (one just before work, and one after lunch), you would have scaled the total height equivalent to that of Mount Everest in just a year! Who needs to climb the real mountain when you can climb it everyday in your own city :P

ohhhh yeaaahhhh

If the stairs at work are too cumbersome to access, you would probably have access to some stairs on the way to work. At the train station, up that grand staircase at the park near your home. After work, take the stairs back up to your apartment. Whenever you get the chance, just go for it!

At first it might seem a bit overwhelming and extremely tiring. The first few times are probably the hardest, both physically and mentally. After a while, it will naturally become a habit and you will want to climb every staircase that you find! (might be exaggerating here a little) You just need to get over that initial bump, and the rest will follow suit :D

Go to that toilet which is further away

We all have to go for toilet breaks every now and then (pardon me if you do not). Instead of heading down to the nearest toilet, you could instead head on to the one further away. If there is only one on your floor, head up to the one on the upper or lower floor. This ties closely to the previous point about taking the stairs.

let me help you with that

Adding more steps to and fro not only helps burn extra calories, but allows for you to give your body a good stretch - something it needs after a long day seated at your desk. Personally, during these walks, I tend to have little sparks ideas as well. Be it finding a new angle to tackle the issue I am currently facing or thinking of new features to add to my work.

Firstly, because the walk takes away my current form of thinking which is embedded on my screen. Secondly, because it allows a good stretch on my strained back and neck and relaxes my mind a little when I return to my desk. A freshened mind always helps. Perhaps these walks will help you too!

(Pro tip: Keep yourself hydrated as much as you can. The more water you drink, the more walking you will do)

For those who have 10 minutes to spare

10 minutes does not sound like much but if you can focus this time to get your heart pumping, you will break out in a sweat in no time. I usually put 5-10 minutes of my busy days to working out. This usually comes in the form static exercises so that 10 minutes of my time is purely focused on sweating, nothing less. Mobile apps such as Home Workout (not affiliated to them whatsoever) provide short exercise routines which usually take less than 10 minutes.

We all can jump can't we?

You can even do these during your lunch breaks or any other breaks you might have. The shortest routine on the app consists of just a 3 minute routine (minus the break intervals between each exercise). However, if you manage to get more time, why not attempt a chain of exercise routines after work? Sweat it all out and have a good night's sleep after.

TL;DR

Climb
- Conquer Mount Everest in your own backyard by climbing all the stairs you can find
Walk
- Find the further toilet, cover more distance by walking to it
Sweat
- Do static exercises in 10 minutes or less -> Home Workout

Live in the now

Muhammad Syuqri — Sat, 20 Oct 2018 04:40:07 +0000

I used to feel extremely exhausted everyday after work, which leaves me with barely any form of motivation to do anything. As a junior developer in the industry, I felt that I need to change the way I do things, so at least when I get home, I still have some energy left to practice my programming because I really do enjoy it.

I treat programming as my hobby and it makes me happy to be able to solve small challenges through code. However, it was obvious that the mental fatigue was not allowing me to spend time on my hobby. So I decided to change things a little.

Why I was mentally exhausted?

I was constantly indulging myself in worry at work and at home. It came to me subconsciously and I did not do anything about it other than thinking more about the worries.

What if I cannot finish this task?
What if there is an email request from work tonight?
What if? What if...?

There were lots of "what ifs" taking up my brain capacity to the point that it was in a constant loop of worry. After work, the work worries subsided a little, but not too much. It was on repeat over and over again.

I knew I needed some kind of help or advice.

Living in the NOW

I listened to some advices from the audiobook
How to Stop Worrying and Start Living by Dale Carnegie which says that in order to stop worrying, you just have to be in the now. Not promoting this book, but I feel it may be of some value to others out there who might be in the same situation I was.

Worries do not help in any day activities at all, apart from wasting my time. So, each time a worry came to mind, I tell myself "Hey there Mr. Syuqri, do not worry about it NOW. Put it in your to-do list and think about it LATER". My worries slowly vanished and they appeared in the form of To-Dos. I schedule each To-Do for a designated time if need be, or leave them floating if they aren't important.

Although my schedule each day is filled now from day (work) to night (side projects), I feel a sense of ease at each point in time as I am only focused at the task at hand, and nothing else. This certainly helps in productivity and ultimately frees more time for myself - something that worry has never done for me.

TLDR

Think and live in the now. Worries should be transformed into To-Dos (scheduled or free floating) and be taken care of later.

What are you struggles in your busy schedule? How do you overcome them?

How I used Google Cloud Platform to start investing in stocks

Muhammad Syuqri — Sat, 06 Oct 2018 13:38:48 +0000

I got interested in investing after attending a short talk recommended by a friend of mine. I decided to do some research and started reading The Little Book That Still Beats The Market by Joel Greenblatt. From the book, I found some formulas that could be useful to me when making decisions on whether or not to invest in the stocks of companies in Singapore. This post isn't to promote the book or its investment strategies, but more to showcase the following and how I did it:

Interacting with Firestore through Python
Running a Python script at specific time intervals on the Compute Engine
Using Cloud Functions to retrieve data from Firestore

At first, I created a Python script for populating a Google Sheet with the financial details and self-calculated ratios from companies listed on the Singapore Exchange website. I found this a hassle as I had to do run the Python script everyday to get the updated prices of the stocks. I then decided to move this everyday process to the Google Cloud Platform so that I no longer have to do the everyday process myself, and leave it to the cloud to do it for me :D

The following will explain how I did what I did in hopes to help anyone else out there who might want to use the Google Cloud Platform in a similar fashion as I did.

Prerequisites

Before proceeding any further, I would like to note that the following have to been done first to keep this post short and simple. I have included links to get you started as well.

Overview

From the above diagram, the only thing I have to do is to make a GET request through the Cloud Functions HTTP API which will return all the already calculated formulas and values stored in the Firestore. Essentially, steps 1, 2 and 3 involve the Python script I have created. Steps 1 and 2 are done simply by using the Requests library.

Interacting with Firestore through Python

Firestore uses the concept of collections, documents and fields to store the data you want it to. So for instance, using the analogy of a book library, if you have a shelf of books, that is a collection in Firestore's viewpoint. The books themselves are documents, and each page in the book is a field on its own. Each document can have its own collection as well, but I will not get into that.

shelf [collection]
|--book1 [document]
  |-- page1 [field]
  |-- page2 [field]
|--book2 [document]
  |-- page1 [field]

To interact and update data on the Cloud Firestore from your Python script, you first have to install the Google Cloud Firestore library via pip install google-cloud-firestore. The following is the code snippet to initialize Firestore with your service account key that you have previously retrieved.

from google.cloud import firestore
db = firestore.Client.from_service_account_json('/path/to/service/key')

Well that is it actually! To write data to Firestore, simply do the following:

doc_ref = db.collection(u'name_of_collection').document(u'name_of_document')
doc_ref.set(data_to_update)

data_to_update is a Python dictionary which holds the keys and respective values you would want the Firestore document to hold. The .set() allows you to update or insert new fields into the document. For myself, I was putting the company name, stock prices, financial ratios and other fields here.

A point to note here is that even if the document or collection does not exist yet, .set() function automatically creates the collection and document for you and populates the document with the fields as mentioned before.

Running a Python script on Compute Engine

There are a few ways of pushing your Python script to your VM Instance. How I did it was to create a repository in my Google Cloud project and pushed it there. The reason I created the repository was because I still wanted some form of version control as, knowing myself, I like to make changes and explore different ways to do things in my code and end up confusing myself in the end. Even though it is a small project, I felt it was a good practice for me personally. I then remotely accessed the VM Instance via SSH and cloned the repository into the instance.

Now for the scheduling of the Python script. Initially, I thought calling the Python script every 30 minutes was a good idea. However, after some consideration, I felt scheduling the script to run at 6pm (GMT +0800) was the ideal case because the Singapore Exchange opens at 9am and closes at 5pm, and I really only have time to view the stock prices after work anyway.

To schedule your Python script to run either at certain time intervals or at specific timings, you can use Cron jobs as I did. In the SSH session of your VM Instance, edit your user's Crontab using the crontab -e command. At the end of the file, at your schedules in the following format

# m h  dom mon dow   command
0 10 * * 1-5 cd /path/to/python/folder && python main.py

The above snippet runs the Python script at 10am UTC (aka 6pm SGT), every weekday of the day, indicated by the 1-5 segment. If you would like your script to run after every time interval, you can do the following instead:

# Runs the command every hour at the 0th minute
0 */1 * * * <some command>

# Runs the command at the 0th minute every day
0 * */1 * * <some command>

Note: A mistake that I made during my first few times using Crontab in the VM Instance is the following:

# Runs the command every minute after every hour
* */1 * * * <some command>

My intention was to run it at every hour. But I missed the 0 at the minute mark of the cron job. So it was running the script at EVERY MINUTE AFTER EVERY HOUR. My script was taking around 3 minutes to run each time it was called. I did not mind the relatively long run time. However, since the script is being run every minute, and each takes 3 minutes to complete... Well, you can do the math. And silly me was trying to figure out why the CPU usage on my VM Instance was constantly at 150-200% and I could not even access it via SSH. That was a funny lesson :P

Using Cloud Functions to retrieve data from Firestore

For this step, I linked the Google Cloud project to Firebase. The reason I did this was for possible future versions in which I could host a website on Firebase Hosting, which taps on the data from the Cloud Firestore, allowing anyone to view the financial details at a glance. Another reason is also because I am much more familiar with Firebase and the requirements for Cloud Functions there.

I installed Express.js into my Cloud Functions folder via npm install --save express. Express.js allows me to easily create web APIs as I needed multiple end-points for retrieving various company information from the Firestore I have.

var  db  =  admin.firestore();

const  express  =  require("express");
const  app  =  express();

app.get('/:nameOfDocument',( req, res)=>{
    const  nameOfDocument  =  req.params.nameOfDocument;
    var  firestoreRef  =  db.collection("name_of_collection").doc(nameOfDocument);
    res.setHeader('Content-Type', 'application/json');
    firestoreRef.get().then((snapshot) => {
    if (snapshot.exists) {
        var  returnObj  =  snapshot.data();
        return  res.status(200).json(returnObj);
    }
    else {
        return  res.status(422).json({error:"Invalid document name"});
    }
    }).catch(errorObject  => {
        return  res.status(500).json({error:"Internal Server Error"});
    });
})
exports.api  =  functions.https.onRequest(app);

Here is a step by step explanation of what is happening is the snippet above. Firstly, access to Firestore is initialized by var db = admin.firestore();.

app.get('/:nameOfDocument',( req, res)=>{
...
}

The above tells the Express that we would like to create a GET request with the '/:nameOfDocument' end-point, where :nameOfDocument is a parameter in the URL. req and res are request and response objects which are received and going to be sent respectively. Currently, only the res is being used, but more on that later.

const nameOfDocument = req.params.nameOfDocument;

This line takes the parameter from the URL, that is :nameOfDocument in this case, and stores it as a variable called nameOfDocument, which will be used in the next line.

var firestoreRef = db.collection("name_of_collection").doc(nameOfDocument);

This line essentially creates a reference to the document nameOfDocument. The collection name is currently not a variable. You can also use include the name of collection as a parameter as such:

app.get('/:nameOfCollection/:nameOfDocument',( req, res)=>{
    const nameOfDocument = req.params.nameOfDocument;
    const nameOfCollection= req.params.nameOfCollection;
    var firestoreRef = db.collection(nameOfCollection).doc(nameOfDocument);
    ...
}

This way, you can specify it in the URL without having to alter the code.

firestoreRef.get().then((snapshot)  =>  {
    if  (snapshot.exists)  {  
    var returnObj = snapshot.data();  
    return res.status(200).json(returnObj);  
    }
    ...
}

The above segment takes the reference mentioned earlier and checks if it exists. This is essential as a user might accidentally type a wrong document or collection name, and we would want to return the appropriate response. snapshot.data() retrieves all the field key-value pairs and puts it in the object called returnObj We then return this as a JSON object with a status code of 200.

exports.api  =  functions.https.onRequest(app);

This line tells Cloud Functions that when a request is made to <cloudfunctions.net url>/api should be passed to the Express object called app and handled accordingly based on the end-points specified in the app object itself.

And that is it! You can now call your Cloud Functions from the link provided on the Firebase Cloud Functions page which will retrieve the relevant data that you want to work on from your Firestore.

P.S. This is my first tutorial/personal experience post. Kindly do let me know what can be improved and how I can be a better programmer as well. All constructive feedback are welcome. Thank you for reading through my post! :D

What is YOUR ultimate goal that you want to achieve as a developer?

Muhammad Syuqri — Wed, 15 Aug 2018 04:09:52 +0000

We all know that the journey of a developer is one that requires constant learning, no matter how long you have been on the road. However, what is that one thing that you want to achieve, and when you do achieve it, you can sit back and say, "Ahhh.. I have finally fulfilled my purpose as a developer!"

(It can be directly or indirectly related to being a developer)

How did you get out of the habit of copying/pasting code?

Muhammad Syuqri — Fri, 29 Jun 2018 14:13:14 +0000

Like the title describes, I would like to know how you got out of the (bad) habit, particularly when it saves time and does the job.

Is being a master Android developer feasible today considering the availability of UI frameworks such as Flutter, React etc.?

Muhammad Syuqri — Wed, 28 Feb 2018 09:12:45 +0000

Forem: Muhammad Syuqri

saferm - A tool to safely remove files on CLI

Introduction

What is saferm?

Why saferm?

Check it out here

Dansyuqri / saferm

This repository is a project for me to explore Go.

saferm

Installation

Usage

Conclusion

Why Python Generators?

Reasons to use a generator

Reduce memory usage when iterating over large objects

You need to use intermediate values from a function

References

Newbiefy - Make Your Repo Beginner Friendly Again

My Workflow

Submission Category:

Yaml File or Link to Code

Dansyuqri / newbiefy

Makes your repository beginner friendly by removing everything in place of a hello_world.<ext> in the mostly used language of the repo.

Newbiefy

What

Why

Usage

Additional Info

Demo

Additional Resources / Info

Pub/sub with PyZMQ: Part 2

Multiple Subscribers to One Publisher

Single Subscriber to Multiple Publishers

Using Poller to Prevent Code Blocking

Future Guides

Pub/sub with PyZMQ: Part 1

What is publish/subscribe?

What is PyZMQ?

Getting started

Specifying topics

Multipart messages

Future Guides

How to sprinkle some exercise into your developer lifestyle

How to add mini exercises to your daily routine

Take the stairs

Go to that toilet which is further away

For those who have 10 minutes to spare

TL;DR

Live in the now

Why I was mentally exhausted?

Living in the NOW

TLDR

How I used Google Cloud Platform to start investing in stocks

Prerequisites

Overview

Interacting with Firestore through Python

Running a Python script on Compute Engine

Using Cloud Functions to retrieve data from Firestore

What is YOUR ultimate goal that you want to achieve as a developer?

How did you get out of the habit of copying/pasting code?

Is being a master Android developer feasible today considering the availability of UI frameworks such as Flutter, React etc.?

What is `saferm`?

Why `saferm`?