Forem: Zhang Zeyu

Blind XPath Injections: The Path Less Travelled

Zhang Zeyu — Wed, 31 Mar 2021 02:07:10 +0000

This article is inspired by the “X marks the spot” challenge in picoCTF 2021. For the solution to the challenge, skip to the ‘Exploitation’ section.

While SQL injections are one of the most common web application vulnerabilities, its less notorious twin can be equally, if not more dangerous.

XPath?

XPath is a query language that locates elements in an XML document. Conceptually, it is similar to SQL. Most web applications use relational databases and SQL to store and query large amounts of data. Yet, in some use cases, especially those where data needs to be extracted and transferred between systems easily, XML databases have become much more appealing.

It is thus increasingly common for web applications to use XML data on the backend, using XPath the same way as SQL is traditionally used.

XML Documents

We can think of XML documents as a tree structure.

The above tree would correspond to the following XML document:

<bookstore>
  <book category="cooking">
    <title lang="en">Everyday Italian</title>
    <author>Giada De Laurentiis</author>
    <year>2005</year>
    <price>30.00</price>
 </book>

 ...

</bookstore>

XPath Syntax

Basic XPath queries consist of path expressions. / will select from the root node, while // will select nodes no matter where they are in the document.

For instance, bookstore/book will select all book elements that are children of bookstore. //book on the other hand, will select all book elements no matter where they are in the document.

Same Same, But Different

Much like SQL injections, XPath injections occur when user-supplied data is embedded in the XPath query in an unsafe manner.

In SQL, access control is implemented with user-level security — each user is restricted to certain resources. However, when using XPath, there are no access controls and it is possible to access any part of the XML document.

Therefore, an XPath injection attack can be much more dangerous and devastating than an SQL injection attack.

Exploitation

In this challenge, we are given a simple login page. There are two POST parameters, name and pass.

The Basics

First, we can try to imagine the source code that constructs the query. It would look something like this:

String FindUserXPath;
FindUserXPath = "//user[username/text()='" + Request("name") + "' And password/text()='" + Request("pass") + "']";

A basic payload would be:

name=' or 1=1 or 'a'='a&pass=test

which would translate to the following query:

//user[username/text()='' or 1=1 or 'a'='a' And password/text()='test']

Importantly, note the order of operations in boolean algebra: AND comes before OR. Therefore, as long as the first part of the query

username/text()='' or 1=1

evaluates to True, the entire query is True. This will be useful later on.

Using this payload, we get the message “You’re on the right path”. Note that this is a blind injection since we do not get any actual data from the XML document (we only have a boolean indicator telling us whether or not our query evaluated to True or False). This mirrors the real-world scenario where a successful login means our query returned True, while a failed login means our query returned False.

Booleanization

In blind injection attacks, the key is to focus on getting one piece of information at a time by using a series of boolean conditions.

Remember the order of operations we discussed above? We can tweak our previous query to the following:

//user[username/text()='' or BOOLEAN_CONDITION or 'a'='a' And password/text()='test']

This will only evaluate to True if BOOLEAN_CONDITION is True, allowing us to test any condition.

Using XPath Functions

We can use XPath functions with booleanization to extract information about the XML document. For instance, count() returns the number of nodes in a node-set.

The following payload evaluates to False, telling us that the number of user nodes in the XML document is not 1.

name=' or count(//user)=1  or '1'='1&pass=test

Then, we change the count to 2, then 3, and so on… until we get a payload that evaluates to True:

name=' or count(//user)=3  or '1'='1&pass=test

This tells us that there are 3 user nodes in the document.

The same logic can be applied to getting the number of child nodes.

name=' or count(//user[position()=1]/child::node())=5  or '1'='1&pass=test

evaluates to True, telling us that for the first user node, there are 5 child nodes. The same can be checked for all 3 users.

Getting Node Values

To get the node values, there are two steps:

Get the value length using string-length(). The payload for this would be something like:

name=' or string-length(//user[position()=USER_POSITION]/child::node()[position()=NODE_POSITION])=LENGTH or ''='&pass=test

where USER_POSITION and NODE_POSITION refer to the position of the user and child node respectively (if USER_POSITION = n, the nth user is selected) and LENGTH refers to the length we want to test for.

Get the value, character by character using substring(). The payload for this would be something like:

name=' or substring((//user[position()=USER_POSITION]/child::node()[position()=NODE_POSITION]),INDEX,1)=CHARACTER or ''='&pass=test

This will test for CHARACTER at index INDEX (starting from 1) of the string.

There are a few ways to automate this. Burpsuite’s Intruder allows us to load a list of payloads, which can be a list of numbers in this instance. Alternatively, we could write a simple script.

The following script will take two arguments, USER_POSITION and NODE_POSITION, find the length of the node value, then finds the ASCII characters at each position.

To get child node 2 of user 1:

We can repeat this process for the other two users, and find their usernames (“bob” and “admin” respectively).

Let’s continue getting the other child node values. After some trial and error, child node 4 looked promising:

If we run the script with arguments 3 and 4 (to get the admin user’s 4th child node value), we are handed the flag.

Prevention

Now that we know how XPath injections work, how can we prevent them? The solutions are quite similar to those of preventing SQL injections but may be overlooked due to the lack of built-in APIs.

Parameterized XPath

Similar to SQL prepared statements, the idea is to ensure that user-specified data is never interpreted as executable content (and always interpreted as only a parameter).

However, this likely requires the use of an XQuery processor such as Saxon, and the use of the corresponding external APIs. Bare-bones implementations of parameterized XPath queries can be rather cumbersome and tricky for platforms such as .NET and Java SE.

Input Validation / Sanitization

Never trust user-provided data. Input validation / sanitization should be treated as the bare minimum, not a panacea.

It may be impractical to implement an overly-strict filter. For instance, is a password that consists of letters only secure?

// Restrict the username and password to letters only
if (!Regex.IsMatch(user, "^[a-zA-Z]+$") || !Regex.IsMatch(pass, "^[a-zA-Z]+$"))
{
    return BadRequest();
}

String expression = "/users/user[@name='" + user + "' and @pass='" + pass + "']";
return Content(doc.SelectSingleNode(expression) != null ? "success" : "fail");

Yet, an overly-loose filter will leave gaps (filter-bypass techniques can be borrowed from SQL injections).

Conclusion

Yay, you made it to the end! We’ve looked at how XPath injections work, and how they can be prevented. I hope you’ve learnt something new.

Thanks for reading!

Build a Simple Chess AI in JavaScript

Zhang Zeyu — Sun, 20 Dec 2020 12:10:01 +0000

Chess is a great game. It’s even better if you’re good at it. Regrettably, I’ve never taken the time to learn chess strategy, so I decided to rely on the power of computation and game theory instead! As a fun side project, I have implemented a simple chess AI using JavaScript.

You can find the full source code for this tutorial in my GitHub repository.

zeyu2001 / chess-ai

Simple chess AI in JavaScript. Uses the chess.js and chessboard.js libraries.

The final product is playable at https://zeyu2001.github.io/chess-ai/.

Prerequisites

You should know basic programming and the general concept of a tree data structure. Everything else will be covered as part of this tutorial.

The two main algorithms involved are the minimax algorithm and alpha-beta pruning. These will be explained in-depth later on, and should be relatively simple to grasp if you have experience in programming.

First Things First…

Getting the GUI and game mechanics out of the way. This allows us to direct our focus towards only the most fascinating aspect of the application: the decision-making (AI) part! For this, we will be using external libraries:

chessboard.js handles the graphical interface, i.e. the chess board itself.
chess.js handles the game mechanics, such as move generation / validation.

With these libraries, you should be able to create a working chess game by following the examples (5000 through 5005 in particular) on the chessboard.js website.

Evaluation Function

Great! We have a functioning chessboard. But how do we implement an AI that plays (reasonably) good chess? Well, we’re going to need an evaluation function. Basically, we want to assign a ‘score’ to each chessboard instance (i.e. each set of positions of pieces on the board) so that our AI can make decisions on which positions are more favourable than other positions.

A Zero-Sum Game

Chess is a zero-sum game. Any advantages gained by Player A implies disadvantages for Player B. Advantages can come in the form of capturing opponent pieces, or having pieces in favourable positions. Therefore, when assigning a score from our AI’s perspective, a positive score implies an overall advantage for our AI and disadvantage for its opponent, while a negative score implies an overall disadvantage for our AI and advantage for its opponent.

A Simple Example

For instance, the score for the starting position is 0, indicating that neither side has an advantage yet. Later on into the game, we are faced with a decision between two moves: Move A and Move B. Let’s say Move A captures a queen, putting our score at 900, while Move B captures a pawn, putting our score at 100.

The AI will be able to compare between the two potential scenarios, and decide that Move A is the better move. Of course, this does not consider future ramifications — what if Move A gives our opponent the opportunity to attack? We will overcome this hurdle in the following sections by performing lookahead to anticipate subsequent moves.

Piece Weights

The first aspect of our evaluation involves assigning weights to each piece type. If our AI plays from black’s perspective, any black pieces will add to our score, while any white pieces will subtract from our score, according to the following weights:

Pawn: 100
Knight: 280
Bishop: 320
Rook: 479
Queen: 929
King: 60,000

Piece Square Tables

We now have a score based on which pieces exist on the board, but some positions are more favourable than others. For instance, positions that grant higher mobility should be more favourable. For this, we use *piece square tables *(PSTs), which assign an additional score delta to each piece based on its position on the board.

For instance, the PST for knights encourages moving to the center:

This is from white’s perspective, so it would have to be reflected for black.

I certainly am not a chess expert, so the piece weights and PST values are adapted from Sunfish.py. The following is my implementation of the evaluation function. Note that instead of iterating over 64 squares for each evaluation, we simply start from 0 and add or subtract from the score according to the latest move, keeping track of the previous score.

Minimax

Now that we have an evaluation algorithm, we can start making intelligent decisions! We will use the minimax algorithm for this, and I highly recommend reading up on the Wikipedia article to better understand this decision strategy.

Game Tree

We can represent chessboard positions as nodes in a *game tree. *Each node is a chessboard instance, and has children corresponding to the possible moves that can be taken from the parent node.

Minimizing Losses

Essentially, minimax aims to minimize the possible losses, assuming both players are rational decision makers. We can represent the possible moves as a game tree, where each layer alternates between the maximizing and minimizing player. We are the maximizing player, attempting to maximize our score, while the opponent is the minimizing player, attempting to minimize our score.

At the leaf nodes, the evaluated score is backtracked. Positive and negative infinity are wins and losses respectively. At each recursive layer, the maximizing and minimizing roles are alternated. Layer 0 is the current game state, and the goal is to maximize our score.

Alternate Moves

The question our AI has to answer is: “Out of all the possible moves at Layer 0, which guarantees the maximum score?”

This is the same as asking, “Assuming my opponent is always making the most optimal decisions, which move leads to the possibility of attaining the best possible score?”

If we want our AI to be any decent at chess, we would have to perform a lookahead to anticipate our opponent’s subsequent moves. Of course, we can only anticipate a couple turns in advance — it’s not computationally feasible to look ahead as far as the final winning or losing states. We will have to introduce a depth limit that corresponds to the number of turns we are willing to look ahead, and use our evaluation function to determine the favorability of game states once we reach the depth limit.

The Algorithm

This is a fun recursion problem, and I recommend trying to implement it yourself, although my implementation can be found below. If you're stuck, here’s the general idea:

We decide on a predetermined depth limit, k.
At Layer 0, we consider each of our possible moves, i.e. child nodes.
For each child node, we consider the minimum score that our opponent can force us to receive. Then, we choose the maximum node.
But to know the minimum score that our opponent can force us to receive, we must go to Layer 1. For each node in Layer 1, we consider their child nodes.
For each child node (possible move by our opponent), we consider the maximum score that we can achieve subsequently. Then, the minimum score that our opponent can force us to receive is the minimum node.
But to know the maximum score that we can achieve subsequently, we must go to Layer 2.
And so on…
At Layer k, the final board state is evaluated and backtracked to Layer k - 1, and this continues until we reach Layer 0, at which point we can finally answer: “What is the optimal move at this point?”

Here’s my implementation. Note that I used a slightly modified version of chess.js, which allows me to use game.ugly_moves() and game.ugly_move() to generate and make moves without converting them to a human-readable format, improving the efficiency of the algorithm. The modified version can be found here, but using the normal game.moves() and game.move() will work just fine too.

Alpha-beta Pruning

Our AI should now be able to make reasonably good decisions. The higher the search depth, the better it will play. However, increasing the search depth drastically increases the execution time. Alpha-beta pruning helps to improve the algorithm’s efficiency by ‘pruning’ branches that we don’t need to evaluate. An additional reading resource can be found here.

Core Idea

The core idea of alpha-beta pruning is that we can stop evaluating a move when at least one possibility has been found that proves the move to be worse than a previously examined move.

Suppose that the game tree is as follows:

For brevity, let’s consider the following subtree:

The maximizing player first considers the left child, and determines that it has a value of 5. Other paths will only be chosen if their value is x > 5.

Next, the right child is considered. The minimizing player, at the right child, has found the values 7 and 4 so far. But then this means that regardless of what the remaining value is, the minimizing player would end up with a minimum value of at most 4. We know the final value of this subtree would be x <= 4, regardless of the remaining value.

In order for this path to be relevant, x > 5. But we know that x <= 4. This is a contradiction, so the maximizing player wouldn’t choose this path and there is no point evaluating this path further.

The Algorithm

The same idea can then be extended to the rest of the game tree. We use two variables, alpha and beta, to keep track of the maximizing and minimizing values (5 and 4 in the previous example) respectively. This only requires minor modifications to the previous minimax function — see if you can implement it yourself!

Here’s my implementation:

Conclusion

That’s all! I hope you have enjoyed reading this article as much as I have enjoyed writing it. I’ve explained how I implemented my AI, and hopefully introduced several new and interesting concepts to you.

I’ve also implemented some other features, including pitting the AI against itself. You can play it at https://zeyu2001.github.io/chess-ai/, and refer to my GitHub repository for the implementation.

Network Scanning with Scapy in Python

Zhang Zeyu — Sat, 01 Aug 2020 08:49:54 +0000

What is Network Scanning?

Network Scanning is the process of gathering information about devices in a computer network, through employing network protocols. It is especially relevant for network administrators, penetration testers and, well… hackers.

Prerequisites

You should know basic Python. Other than that, not much! I will be writing on some basic network theory before getting into the actual code, so if you already know this stuff, feel free to skip ahead!

All code for this tutorial can be found at my GitHub repository.

zeyu2001 / network-scanner

Simple network scanner built with Scapy for Python

Protocols, Protocols, Protocols

Communications over networks use what we call a protocol stack — building higher-level, more sophisticated conversations on top of simpler, more rudimentary conversations. Network layers can be represented by the OSI model and the TCP/IP model. Each network layer corresponds to a group of layer-specific network protocols.

For the purpose of this tutorial, we will only concern ourselves with the ARP protocol and the TCP protocol.

Address Resolution Protocol (ARP)

ARP maps IP addresses to MAC addresses. IP addresses are logical addresses, while MAC addresses are physical addresses. When computers communicate with each other over the network, they will specify a target IP address. However, switches (which act as packet forwarders) don’t understand IP addresses — they can only make forwarding decisions based on MAC addresses. Hence, computers need to determine the MAC address of their intended recipient before sending out a packet.

If a computer wishes to send a packet to 192.168.52.2, it will first send an ARP request, asking all devices in the network “who is IP address 192.168.52.2?” The computer with IP address 192.168.52.2 will respond with “Hi, I am 192.168.52.2. My MAC address is 03-CA-4B-2C-13–8A.”

As you might have noticed, as ARP is a standalone protocol, anyone can send an ARP request at any time to learn about the devices on the network through ARP replies. We will use Scapy to scan the network using ARP requests and create a list of IP address to MAC address mappings.

Transmission Control Protocol (TCP)

TCP is a transport layer protocol that most services run on. It is a connection-oriented protocol, meaning that two devices will need to set up a TCP connection before exchanging data. This is achieved using a 3-way handshake.

TCP uses port numbers to differentiate between different applications on the same device. For example, if I am running both Firefox and Chrome on my computer, the OS uses port numbers to distinguish between the two applications so that webpages meant for Chrome don’t show up on Firefox.

When Host P wishes to connect to Host Q, it will send a SYN packet to Host Q. If Host Q is listening on the target port and willing to accept a new connection, it will reply with a SYN+ACK packet. To establish the connection, Host P sends a final ACK packet.

Using Scapy, we will send SYN packets to a range of port numbers, listen for SYN+ACK replies, and hence determine which ports are open.

Scapy

Scapy is a packet manipulation tool written in Python. If you haven’t already, you need to install Scapy with pip.

$ pip install scapy

Now, we can start trying out the basic features of Scapy.



$ python3
...
>>> from scapy.all import *

We can create a packet like so:



>>> Ether()
<Ether  |>

We have created an Ethernet frame. This corresponds to the data link layer (Layer 2) of the OSI model. If we don’t pass in any parameters, default values will be used.

We can specify parameters, such as the destination MAC address:



>>> p = Ether(dst='ff:ff:ff:ff:ff:ff')
>>> p.show()
###[ Ethernet ]###
  dst       = ff:ff:ff:ff:ff:ff
  src       = d0:81:7a:b0:bb:0c
  type      = 0x9000

Here, we specified dst='ff:ff:ff:ff:ff:ff', which is the broadcast address. The packet is addressed to all devices within the local network.

We can stack higher layer protocols on top of the Ethernet protocol, like so

>>> p = Ether() / IP()
>>> p.show()
###[ Ethernet ]###
  dst       = ff:ff:ff:ff:ff:ff
  src       = 00:00:00:00:00:00
  type      = IPv4
###[ IP ]###
     version   = 4
     ihl       = None
     tos       = 0x0
     len       = None
     id        = 1
     flags     =
     frag      = 0
     ttl       = 64
     proto     = ip
     chksum    = None
     src       = 127.0.0.1
     dst       = 127.0.0.1
     \options   \

Here, we stacked IP, a network layer (Layer 3) protocol on top of Ethernet, a data link layer (Layer 2) protocol.

ARP Scanner



from scapy.all import *

def arp_scan(ip):

    request = Ether(dst="ff:ff:ff:ff:ff:ff") / ARP(pdst=ip)

    ans, unans = srp(request, timeout=2, retry=1)
    result = []

    for sent, received in ans:
        result.append({'IP': received.psrc, 'MAC': received.hwsrc})

    return result

Creating the ARP Request

Wow, what did we do here? First, we stacked ARP on top of Ethernet, and set the Ethernet destination address to the broadcast address so that all devices on the local network receive this ARP request.

Sending and Receiving

Next, we called srp() to send the ARP request, and wait for a response. We specified the timeout to be 2 seconds, and if we do not receive an ARP reply within 2 seconds, we retry 1 time before giving up.

Analysing the Results

srp() returns two lists: a list of answered requests and a list of unanswered requests. Within the list of answered requests are nested lists: [<sent_packet>, <received_packet>].

Information contained within the packets is stored as attributes. We use received.psrc to get the source IP address of the reply and received.hwsrc to get the source MAC address of the reply.

TCP Scanner



from scapy.all import *

def tcp_scan(ip, ports):
    try:
        syn = IP(dst=ip) / TCP(dport=ports, flags="S")
    except socket.gaierror:
        raise ValueError('Hostname {} could not be resolved.'.format(ip))

    ans, unans = sr(syn, timeout=2, retry=1)
    result = []

    for sent, received in ans:
        if received[TCP].flags == "SA":
            result.append(received[TCP].sport)

    return result

Creating the SYN Packet

Here, we create an IP packet and specify the destination IP address, then stack TCP on top of it, specifying the destination ports and setting the SYN flag.

Note that dport can be either a single port or a range of ports.

If dport=80, the TCP packet will only be sent to port 80 (HTTP).
If dport=[80, 443], the TCP packet will be sent to both port 80 (HTTP) and port 443 (HTTPS)
If dport=(0, 1000), the TCP packet will be sent to all ports from port 0 to port 1000.

Hence, the ports parameter of our function can be either an integer, a list or a tuple.

flags="S" sets the SYN flag in the TCP packet. If the receiving port is open, it should reply with a packet with flags set to "SA" (for SYN+ACK).

socket.gaierror

socket.gaierror is raised when either the IP address provided is invalid, or a hostname provided could not be resolved by the DNS service. We catch this exception and raise a more meaningful exception to the user instead.

Sending and Receiving

Again, we send the packet and wait for a response. We use sr() instead of srp() because we are dealing with a Layer 3 packet. Both methods return the same results but srp() is for Layer 2 packets.

Analysing the Results

Not all replies are SYN+ACK packets! If a port is not open, the target device may respond with an RST (reset) packet to inform our OS that it does not want to establish a TCP connection.

We use if received[TCP].flags == “SA” to check the TCP flags of the received packet. Note that we can use <packet>[<protocol>] to access the protocol-specific information of the packet. Again, information is stored as attributes and we use received[TCP].flags to obtain the flags of the received packet. received[TCP].sport is the source port of the received packet.

Full Code

I used argparse to turn our scanner into a command-line application, and added some documentation. The full code is embedded below. You can also find it at my GitHub repository.

Conclusion

That’s all! I hope that you’ve enjoyed reading this as much as I have enjoyed writing it. If you have any questions, please feel free to let me know in the comments.

Build a Chatroom App with Python

Zhang Zeyu — Thu, 18 Jun 2020 14:57:53 +0000

Socket programming + Tkinter GUI

Photo by Volodymyr Hryshchenko on Unsplash

I’ve recently delved into the wonderful world of computer networking. One of the fun projects I’ve created is a simple chatroom application that facilitates real-time messaging between different clients.

zeyu2001 / pychat

A real-time Python chatroom application with Tkinter GUI

At any point in this tutorial, you may refer to my source code in GitHub. The aim of this tutorial is to introduce some basic networking theory while providing practical socket programming experience. If, at any point, you find that you are already comfortable with the relevant theory, please feel free to skip ahead!

Prerequisites

You should know basic Python. Other than that, nothing! In the process of creating this application, you will also learn basic computer networking and client-server architecture.

We are going to use Tkinter and sockets, both of which are available in the Python standard library. There is nothing you need to install!

Client-Server Architecture

Photo by İsmail Enes Ayhan on Unsplash

The client-server architecture is a basic computing model where the client requests services from the server, while the server processes these requests and provides services to the clients.

In fact, your web browser is a client requesting web services from Medium’s web server through the HyperText Transfer Protocol (HTTP) right now! Medium’s web server processes your request and returns HTML documents, CSS stylesheets, JavaScript files and images so that your web browser can display the website the way it is.

Client-Server Architecture

Furthermore, a server can serve multiple clients! In our application, we want many clients to be talking in real-time. The client software will send a message to the chatroom server, and the chatroom server will broadcast our message to all other connected clients.

Protocols, Protocols, Protocols

Network Protocol Stack

For the purpose of this application, we need not concern ourselves with many of the lower-level protocols. But we do need to know that we are using something called the Transmission Control Protocol(TCP).

TCP and UDP are transport-layer protocols — they govern how data is sent from one point to another. We are building on top of TCP, which means we do not need to care about how data is sent, only what and where data is sent.

The main difference between TCP and UDP is that TCP guarantees reliable delivery, without any information lost, duplicated, or out-of-order. UDP does not guarantee the same and leaves it up to the application layer to handle dropped packets. It requires the server to acknowledge receipt of data.

TCP vs UDP

UDP is normally used for time-sensitive transmissions where a dropped packet is preferred to waiting for lost packets to be re-transmitted. Imagine a real-time voice call, for example. If a packet has been lost, it won’t make sense to wait for that piece of audio to arrive because, by the time it does, it would already be too old. Furthermore, you would probably still be able to piece the conversation together without that packet of audio.

TCP, on the other hand, would have kept stubbornly resending that piece of lost audio even though it is already too old to be of any use. However, for most other applications, this aspect of TCP is extremely valuable. In our chat application, we don’t want to have to deal with lost packets because we always want to receive complete messages without any errors.

Chatroom Server

Let’s begin with our server script. We start by defining a Server class in server.py.

#!/usr/bin/env python3

import threading
import socket
import argparse
import os

class Server(threading.Thread):

    def __init__(self, host, port):
        super().__init__()
        self.connections = []
        self.host = host
        self.port = port

    def run(self):
        pass

Threading Basics

Note that we are using multithreading to allow multiple pieces of code to run concurrently. Our Server class inherits from Python’s threading.Thread class, thus creating a thread. We need to define the logic for our Server thread in the run() method. When start() is called on the Server object, run() will be executed in parallel to the main thread.

Socket Basics

A socket is an IP address + port number pair. An IP address identifies a host. However, a host can have many different applications running at the same time. How does your operating system (OS) know that an HTTP response is meant for your web browser, and not Call of Duty: Modern Warfare? Port numbers are used to identify the particular application that should receive the data. Call of Duty: Modern Warfare uses the TCP port range 27014–27050, for example.

Thus, any communication between two network devices needs a socket pair:

Source (IP: Port Number) → Destination (IP: Port Number)

Creating a Socket

Let’s start defining our thread logic. Add the following to the run() method.

def run(self):

    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind((self.host, self.port))

    sock.listen(1)
    print('Listening at', sock.getsockname())

First, we created a socket.socket object. socket takes two arguments: the address family and the socket type. The AF_INET address family is used for IP networking. The SOCK_STREAM socket type is used for reliable flow-controlled data streams, such as those provided by TCP. UDP, on the other hand, requires a packet-based socket type, SOCK_DGRAM.

Next, we set the SO_REUSEADDR option. You can read more about socket options in the Linux manual page socket(7). This option allows the server to use the same port after an old connection was closed (normally, you would have to wait for a few minutes).

Then, we use bind() to bind the socket object to a socket address on the server machine. bind() takes in a tuple, in the format

(IP Address (str), Port Number (int))

Note that a machine can have many external IP interfaces: you can find yours by using the ifconfig command at the terminal (or ipconfig for Windows).

Running ifconfig command from the terminal

You should know that the very first result I got, the lo0 interface, is a very special one. This is the loopback interface, which is only reachable by other programs running on the same machine. It has an IP address of 127.0.0.1 and is only locally significant within your own machine. It has the hostname ‘localhost’ and provides a safe testing environment.

Within the bind() command, we can specify any IP interface (even the loopback interface!), or we can use an empty string '' as a wildcard to indicate that we are willing to receive packets arriving at the server via any of its network interfaces. In our program, we will leave this up to the user, as you will see later.

Finally, we use the listen() to indicate that this is a listening socket. TCP uses two types of sockets: listening sockets and connected sockets. After calling listen() on the socket, this becomes a listening socket and can only facilitate establishing TCP connections through handshakes, but not the actual transfer of data. We will need to create an entirely new socket whenever a client connects, in order to send and receive data. Let’s continue:

Accepting Connections

def run(self):

    ...

    while True:

        # Accept new connection
        sc, sockname = sock.accept()
        print('Accepted a new connection from {} to {}'.format(sc.getpeername(), sc.getsockname()))

        # Create new thread
        server_socket = ServerSocket(sc, sockname, self)

        # Start new thread
        server_socket.start()

        # Add thread to active connections
        self.connections.append(server_socket)
        print('Ready to receive messages from', sc.getpeername())

We create an infinite loop to listen for new client connections. The accept() call will wait for a new client to connect and when it does, returns a new connected socket along with the socket address of the connected client.

A few more methods to introduce: getpeername() returns the socket address on the other end of the connection (in this case, the client) while getsockname() returns the socket address to which the socket object is bound.

We need a way to communicate with each individual client, but at the same time, we need to be listening for new connections from other potential clients. The way we do this is by creating a new ServerSocket thread (we will define this later) every time a new client connects, and this thread runs alongside the Server thread. The server also needs a way to manage all active client connections, so it stores the active connections as ServerSocket objects in self.connections.

‘Broadcasting’

How our little chatroom app will work is:

A client sends a message to the server from the command line or GUI.
The server receives and processes the message.
The server sends the message to all other connected clients.
The clients will display the message in the command line or GUI.

Let’s add the ‘broadcasting’ functionality from step 3 to the Server class. I use the word ‘broadcasting’ with caution because a broadcast refers to a completely different concept in the context of computer networking. We are really sending many unicasts, which are one-to-one transmissions to each individual connected client.

class Server(threading.Thread):

    ...

    def run(self):

        ...

    def broadcast(self, message, source):

        for connection in self.connections:

            # Send to all connected clients except the source client
            if connection.sockname != source:
                connection.send(message)

Note that self.connections is a list of ServerSocket objects representing active client connections. We will define the ServerSocket class below.

Sending and Receiving

The ServerSocket class will facilitate communications with individual clients.

#!/usr/bin/env python3

...

class ServerSocket(threading.Thread):

    def __init__(self, sc, sockname, server):
        super().__init__()
        self.sc = sc
        self.sockname = sockname
        self.server = server

    def run(self):

        while True:
            message = self.sc.recv(1024).decode('ascii')
            if message:
                print('{} says {!r}'.format(self.sockname, message))
                self.server.broadcast(message, self.sockname)
            else:
                # Client has closed the socket, exit the thread
                print('{} has closed the connection'.format(self.sockname))
                self.sc.close()
                server.remove_connection(self)
                return

    def send(self, message):
        self.sc.sendall(message.encode('ascii'))

Again, we create an infinite loop. This time, instead of listening for new connections, we are listening for data sent by the client.

When recv() is called, it will wait for data to arrive. If no data is available, recv() will not return (it ‘blocks’) and the program pauses until data arrives. Calls like accept() and recv() that make the program wait until some new data has arrived, allowing it to return, are called blocking calls. Data is sent and received over the network as bytestrings, and hence need to be encoded and decoded using encode() and decode() respectively.

recv() takes in one argument, bufsize, which specifies the maximum amount of data to be received at once.

send(), on the other hand, sends data from the socket to its connected peer.

One problem with both send() and recv() is that there is a slight possibility that only part of the data is sent or received, because the outgoing or incoming buffers are almost full, so it queues whatever data it could, while leaving the rest of the data unprocessed. This becomes problematic when send() returns, but in fact, some of the data is still left unsent. We could put send() in a loop, or we could use sendall() as a simple way to say “I want to send all of the data”.

Closed Sockets

One thing you would notice in the above code is that we are able to tell if the client has closed its end of the connection. When the client socket is closed, recv() returns an empty string '' immediately, similarly to how read() behaves when an end-of-file is reached.

Thus, when we see that recv() returns an empty string (in the else statement), we close() our side of the connection as well, remove the ServerSocket thread from the list of active connections, and end the thread.

Final Touches

#!/usr/bin/env python3

...

def exit(server):

    while True:
        ipt = input('')
        if ipt == 'q':
            print('Closing all connections...')
            for connection in server.connections:
                connection.sc.close()
            print('Shutting down the server...')
            os._exit(0)

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Chatroom Server')
    parser.add_argument('host', help='Interface the server listens at')
    parser.add_argument('-p', metavar='PORT', type=int, default=1060,
                        help='TCP port (default 1060)')
    args = parser.parse_args()

    # Create and start server thread
    server = Server(args.host, args.p)
    server.start()

    exit = threading.Thread(target = exit, args = (server,))
    exit.start()

We allow the user to specify the host address and port number at the command line and enter ‘q’ at any point from the command line to terminate the program.

You can now run your server script!

Running server.py from the terminal

Chatroom Client

The server doesn’t really do much without a client connecting to it. Let’s create a client.py file:

#!/usr/bin/env python3

import threading
import socket
import argparse
import os

class Send(threading.Thread):

    def __init__(self, sock, name):
        super().__init__()
        self.sock = sock
        self.name = name

    def run(self):

        while True:
            message = input('{}: '.format(self.name))

            # Type 'QUIT' to leave the chatroom
            if message == 'QUIT':
                self.sock.sendall('Server: {} has left the chat.'.format(self.name).encode('ascii'))
                break

            # Send message to server for broadcasting
            else:
                self.sock.sendall('{}: {}'.format(self.name, message).encode('ascii'))

        print('\nQuitting...')
        self.sock.close()
        os._exit(0)

class Receive(threading.Thread):

    def __init__(self, sock, name):
        super().__init__()
        self.sock = sock
        self.name = name

    def run(self):

        while True:
            message = self.sock.recv(1024)
            if message:
                print('\r{}\n{}: '.format(message.decode('ascii'), self.name), end = '')
            else:
                # Server has closed the socket, exit the program
                print('\nOh no, we have lost connection to the server!')
                print('\nQuitting...')
                self.sock.close()
                os._exit(0)

While creating the server script, I hope I have made much of the theory clear to you. The same rules apply here, except we have a Send thread that is always listening for user input from the command line. We will add a GUI later, but it follows much of the same logic. Any data received will be displayed on the client interface, and any data sent will be processed by the server to be broadcast to other connected clients.

Again, we make use of multithreading. This time, it is to let the sending and receiving operations run alongside each other. This way, our chatroom is real-time (instead of alternating between send() and recv() calls).

Connecting to the Server

#!/usr/bin/env python3

...

class Client:

    def __init__(self, host, port):
        self.host = host
        self.port = port
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    def start(self):
        print('Trying to connect to {}:{}...'.format(self.host, self.port))
        self.sock.connect((self.host, self.port))
        print('Successfully connected to {}:{}'.format(self.host, self.port))

        print()
        name = input('Your name: ')

        print()
        print('Welcome, {}! Getting ready to send and receive messages...'.format(name))

        # Create send and receive threads
        send = Send(self.sock, name)
        receive = Receive(self.sock, name)

        # Start send and receive threads
        send.start()
        receive.start()

        self.sock.sendall('Server: {} has joined the chat. Say hi!'.format(name).encode('ascii'))
        print("\rAll set! Leave the chatroom anytime by typing 'QUIT'\n")
        print('{}: '.format(name), end = '')

The client calls the connect() method to connect to a specified socket address (again, in the form of a tuple) of the server. If the specified socket is not ready to receive connections (e.g. you did not run the server script yet), the connect() call will fail.

connect() fails with ConnectionRefusedError

Final Touches

#!/usr/bin/env python3

...

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Chatroom Server')
    parser.add_argument('host', help='Interface the server listens at')
    parser.add_argument('-p', metavar='PORT', type=int, default=1060,
                        help='TCP port (default 1060)')
    args = parser.parse_args()

client = Client(args.host, args.p)
    client.start()

Testing our Command-Line App

What we have done is built a fully-functioning command-line chatroom.

You can run your server script by giving it any of your machine’s IP interfaces (you can also use your hostname!). If you like, you can just use ‘127.0.0.1’ or ‘localhost’ if you are only going to be testing this on one machine.

Running server.py with an IP address

Running server.py with a hostname

Same rules apply for the client script. Open a separate terminal window and run client.py.

Running client.py with a server IP address

Running client.py with a server hostname

Open up three or more terminal windows, and you can have multiple clients! Alternatively, grab another machine connected to the same LAN and run client.py from there. If it is in a different LAN, your firewall might block it.

Running Multiple Clients in iTerm

In the above screenshot, I am running three terminal windows, with two clients connected to the same server.

Creating a GUI

We are not quite done yet. A command-line app is nice and all, but how about making a GUI? We are going to use Tkinter, which is included in the Python standard library.

We just need to make a few adjustments to our client.py, and add some more code to create the GUI. You can learn more about Tkinter by reading its documentation, but that is beyond the scope of this tutorial.

The complete client.py code is available below:

and for reference, the complete server.py code:

The Final Product

Run server.py and client.py the same way as you did before. After entering your name, the GUI will pop up!

I have designed this so that you can use either the command line or the GUI to interact with the application so that it is easier to debug and see what is going on behind the scenes.

Conclusion

That’s all! I hope that you’ve enjoyed reading this as much as I have enjoyed writing it. The world of computer networking is really quite interesting and this was only the tip of the iceberg.

If you have any questions, please feel free to let me know in the comments.

Replace Loops, map() and filter() With List Comprehensions in Python

Zhang Zeyu — Fri, 08 May 2020 09:10:10 +0000

I’m a fan of The Zen of Python, which says

There should be one — and preferably only one — obvious way to do it.

But in Python, there are in fact many ways to achieve the same objective. Of course, some ways are more elegant than others and in most cases, it should be obvious which way is better.

We are going to look at list comprehensions, and how they can replace for loops, map() and filter() to create powerful functionality within a single line of Python code.

Basic List Comprehension

Say I want to create a list of numbers from 1 to 10. I could do

numbers = []
for i in range(1, 11):
    numbers.append(i)

and I would get

>>> numbers
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

But using list comprehension, this could be accomplished within a single line

>>> numbers = [i for i in range(1, 11)]
>>> numbers
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

This is the basic syntax of list comprehension: [expression for element in iterable]. Here, the iterable is range(1, 11), the element is i and the expression is i. This is equivalent to the for loop we used earlier: we add i to the list where i is a number from 1 to 11.

map()

The map() function is often used to apply a function on each element in an iterable. Pass in a function and an iterable, and map() will create an object containing the results of passing each element into the function.

For example, say I want to create a list of squares from the numbers list we created earlier. We could do

squares = []
for num in numbers:
    squares.append(num ** 2)

and we will get

>>> squares
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Or we could use map() like so

>>> squares = list(map(lambda x: x ** 2, numbers))
>>> squares
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

Here, we pass each of the elements in numbers into the lambda function (which is just an easy way to create functions if you are lazy). The output of putting each number x into the function lambda x: x ** 2will be the square of the number. Using list() we turn the map object into a list.

Replacing map() With List Comprehension

Using list comprehension, we could simply do

>>> squares = [num ** 2 for num in numbers]
>>> squares
[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]

This will pass each number num into the expression num ** 2 and create a new list where the elements are simply the squares of each number in numbers.

filter()

The filter() function is used to create a subset of an existing list, based on a condition. Pass in a function and an iterable, and filter() will create an object containing all elements where the function evaluates to True.

For example, say I want to get a list of all odd numbers and a list of all even numbers from the numbers list we created earlier. We could do

odd_numbers = []
even_numbers = []

for num in numbers:

    if num % 2 == 1:
        odd_numbers.append(num)

    elif num % 2 == 0:
        even_numbers.append(num)

and we would get

>>> odd_numbers
[1, 3, 5, 7, 9]
>>> even_numbers
[2, 4, 6, 8, 10]

Or we could use filter() like so

odd_numbers = list(filter(lambda x: x % 2 == 1, numbers))
even_numbers = list(filter(lambda x: x % 2 == 0, numbers))

Here, all the numbers will be passed into the lambda function and if x % 2 == 1 is True, the number will be included in odd_numbers. Likewise, if x % 2 == 0 is True, the number will be included in even_numbers.

Replacing filter() With List Comprehension

Using list comprehension, we could simply do

odd_numbers = [num for num in numbers if num % 2 == 1]
even_numbers = [num for num in numbers if num % 2 == 0]

Here, we are using a conditional. The syntax for this is [expression for element in iterable (if conditional)]. This is equivalent to the for loop we used earlier — if the condition num % 2 == 1 is met, num will be added to odd_numbers, and num is an element in the numbers iterable.

Nested Comprehensions

Say we want to create a matrix. This would involve creating nested lists. Using a for loop, we can do the following

matrix = []
for i in range(5):
    row = []
    for j in range(5):
        row.append(i * 5 + j)
    matrix.append(row)

and we would get

>>> matrix
[
    [0, 1, 2, 3, 4],
    [5, 6, 7, 8, 9],
    [10, 11, 12, 13, 14],
    [15, 16, 17, 18, 19],
    [20, 21, 22, 23, 24]
]

Using a nested comprehension, we could simply do

matrix = [[i * 5 + j for j in range(5)] for i in range(5)]

The outer list comprehension [... for i in range(5)] creates 5 rows, while the inner list comprehension [... for j in range(5)] creates 5 columns.

Dictionary Comprehensions

You could use dictionary comprehensions too. For example, if I want to create a dictionary that maps each number in numbers to their corresponding square, we could do

num_to_square = {}
for num in numbers:
    num_to_square[num] = num ** 2

and we would get

>>> num_to_square
{
    1: 1,
    2: 4,
    3: 9,
    4: 16,
    5: 25,
    6: 36,
    7: 49,
    8: 64,
    9: 81,
    10: 100
}

Or we could use a dictionary comprehension

>>> num_to_square = {num: num ** 2 for num in numbers}
>>> num_to_square
{
    1: 1,
    2: 4,
    3: 9,
    4: 16,
    5: 25,
    6: 36,
    7: 49,
    8: 64, 
    9: 81,
    10: 100
}

You could even use a dictionary comprehension within a list comprehension or vice versa.

>>> num_to_square = [{num: num ** 2 for num in numbers} for numbers in
[odd_numbers, even_numbers]]
>>> num_to_square
[
    {1: 1, 3: 9, 5: 25, 7: 49, 9: 81},
    {2: 4, 4: 16, 6: 36, 8: 64, 10: 100}
]

Why Use List Comprehension?

Cleaner, clearer code
Slightly faster than map() and filter()
Generally considered more ‘pythonic’

But hey, at the end of the day, the choice is yours. List comprehension is just another way to do the same things, and whether something is ‘cleaner’ and ‘clearer’ is largely subjective. However, most people would agree that list comprehension is a more standard way to do things.

Conclusion

That’s it! Now you know how to use list comprehensions in Python.

If you have any questions, feel free to comment down below.

Thanks for reading, and follow me for more stories like this in the future.

Sorting Algorithms in Python

Zhang Zeyu — Mon, 04 May 2020 03:34:11 +0000

We are going to look at 4 different sorting algorithms and their implementation in Python:

Bubble Sort
Selection Sort
Insertion Sort
Quicksort

1. Bubble Sort

Time complexity: O(n²)

Implementation

def bubble(lst):
    no_swaps = False
    while no_swaps == False:
        no_swaps = True
        n = 0
        for i in range(len(lst) - 1 - n):
            if lst[i] > lst[i + 1]:
                lst[i], lst[i + 1] = lst[i + 1], lst[i]              
                no_swaps = False
        n += 1

How It Works

Iterate through the elements in the array
If there are adjacent elements in the wrong order, swap them
If we have reached the end of the array and there have been no swaps in this iteration, then the array is sorted. Else, repeat from step 1.

For example, suppose we have the following array:

In the first pass, when i = 0, lst[0] = 54 and lst[1] = 26 . Since 26 < 54, and we want to sort the array in ascending order, we will swap the positions of 54 and 26 so that 26 comes before 54. This goes on until the end of the list.

Note that the largest element (93) always gets passed to the end of the array during the first iteration. This is the reason for the variable n which increases after each iteration and tells the program to exclude the last element after each iteration over the (sub)array.

The time complexity is O(n²) because it takes n iterations to guarantee a sorted array and every iteration iterates over n elements.

2. Selection Sort

Time complexity: O(n²)

Implementation

def selection(lst):
    for i in range(len(lst)):
        iMin = i
        for j in range(i + 1, len(lst)):
            if lst[j] < lst[iMin]:
                iMin = j
        lst[i], lst[iMin] = lst[iMin], lst[i]

How It Works

Iterate through the elements in the array
At each position, iterate through the unsorted elements to find the minimum element
Swap the positions of this minimum element and the element at the current position

For example, suppose we have the following array:

When i = 0, the minimum value within the unsorted array is 1, at position j = 3. Hence, the elements at position 0 and position 3 are swapped.

When i = 1, the minimum value within the unsorted array (the unsorted array is [5, 7, 8, 9, 3] since the first position is already sorted) is 3, at position j = 5. Hence, the elements in position 1 and 5 are swapped. The unsorted array is now [7, 8, 9, 5].

This goes on until the minimum value is selected (hence the name) for all positions. The time complexity is O(n²) since we have to iterate through the unsorted array n times, once for each current position, and the unsorted array consists of n elements on average.

3. Insertion Sort

Time complexity: O(n²)

Implementation

def insertion(lst):
    for i in range(1, len(lst)):
        temp = lst[i]
        j = i - 1
        while j >= 0 and temp < lst[j]:
            lst[j + 1] = lst[j]
            j -= 1
        lst[j + 1] = temp

How It Works

Iterate through the elements in the array
For each element, iterate through the sorted array to find a suitable position (where it is larger than the element right before it, and smaller than the element right after it)

For example, when i = 5 and lst[i] = 31, I want to insert (hence the name) 31 into the sorted array. The while loop will ‘copy’ the element at j to the position j + 1 and decrease the value of j, until 31 > lst[j]. This occurs at j = 1, where lst[j] = 26. Since 26 < 31 < 54, the right position to insert 31 would be at j + 1 = 2.
Insert the element at its suitable position. From step 2, all the elements from this position onwards are ‘pushed back’ to the right by one position to make space for the inserted element

For example, suppose we have the following array:

When i = 0, there is no sorted array and we simply move on to i = 1. Note that this case is captured under one of the while loop exit conditions; since j >= 0 is False, the while loop will not be run.

When i = 1, the sorted array consists of [54,], and the current element is 26. The correct position to insert 26 is before 54, so 54 is ‘pushed back’ to the right (it is now at position 1) and 26 is inserted at position 0.

This goes on until all the elements are inserted at their suitable positions and the whole array is sorted. The time complexity is O(n²) because we have to iterate through the sorted list n times to find a suitable position for all n elements, and the sorted list consists of n elements on average.

4. Quicksort

Time complexity: O(n log(n))

Implementation (out-of-place)

def quicksort(lst):

    if len(lst) == 0:
        return lst

    else:
        pivot = lst[0]
        left = []
        right = []

        for ele in lst:

            if ele < pivot:
                left.append(ele)

            if ele > pivot:
                right.append(ele)

        return quicksort(left) + [pivot] + quicksort(right)

Implementation (in-place)

def partition(lst, start, end):

    # partition list into left (< pivot), middle (pivot) and right (> pivot), and return position of pivot

    pivot = lst[start]
    left = start + 1
    right = end

    done = False    
    while not done:        

        # find left position where element > pivot
        while left <= right and lst[left] < pivot:
            left += 1        

        # find right position where element < pivot
        while right >= left and lst[right] > pivot:
            right -= 1

        if left > right:
            # stop sorting
            done = True

        else:
            # lst[left] now < pivot, lst[right] now > pivot            
            lst[left], lst[right] = lst[right], lst[left]

    # place pivot in the correct position
    lst[start], lst[right] = lst[right], lst[start]

    return right        

def quicksort_in_place(lst, start, end):

    # sort list recursively with the help of partition()

    if start < end:

        pivot = partition(lst, start, end)    

        quicksort_in_place(lst, start, pivot - 1)        
        quicksort_in_place(lst, pivot + 1, end)

How It Works

This is a recursive algorithm. For each recursive call,

The first element of the array is taken as the ‘pivot’
Partition the array so that you have a subarray of elements that are smaller than the pivot, and another subarray of elements that are larger than the pivot. Position the pivot in the middle of the two subarrays.
Recursively sort both subarrays

The base case: when the subarray is empty (or contains only one element), it cannot be sorted any further.

The In-Place Algorithm

The in-place algorithm modifies the original array, whereas the out-of-place algorithm creates and returns a new array that is a sorted version of the original array. The recursion logic is pretty much the same, except the partition() function can be a bit confusing.

The partition function involves left and right pointers. The left pointer moves to the right until it finds an element that is larger than the pivot. The right pointer moves to the left until it finds an element that is smaller than the pivot. These two elements are then swapped because we want everything on the left to be smaller than the pivot, and everything on the right to be larger than the pivot.

This continues until the left and right pointers cross positions. At this point, we have (almost) successfully partitioned the array. Everything before the left pointer is smaller than the pivot, and everything after the right pointer is larger than the pivot. The last step involves exchanging the positions of the pivot and the element at the right pointer so that the pivot now lies between the left and right subarrays.

The left subarray consists of everything before the pivot, and everything in this subarray is smaller than the pivot. The right subarray consists of everything after the pivot, and everything in this subarray is larger than the pivot.

We have now successfully partitioned the array. This process is repeated on each subarray until the entire array is sorted.

Time Complexity

Working from the bottom-up, 1×2ʰ = n where h is the height of the recursive tree and n is the size of the array. This is because starting from a ‘leaf’ (a one-element subarray), every layer above it merges two subarrays to form a new array that is twice the length of each individual subarray. The size of the subarray, therefore, roughly doubles every time it goes ‘up’ a layer. Therefore, h, the height of the recursive tree, is log(n).

At each layer of the recursive tree, a total of n comparisons are made to partition each subarray (since the total number of elements is still n).

Hence, the overall time complexity is O(n log(n)).

Conclusion

Thanks for reading! If you have any questions, feel free to comment on this post.

Follow me for more stories like this one in the future.

Managing Python Virtual Environments With virtualenvwrapper

Zhang Zeyu — Sat, 02 May 2020 15:41:29 +0000

What Is a Virtual Environment?

A virtual environment is an isolated Python environment. Working on a project in an isolated Python environment ensures that project dependencies are kept separate, and allows you to manage Python packages for different projects without breaking system tools or other projects.
For example, if both projects A and B depend on the same library, project C, but use different versions of it, Python would not be able to serve both versions of the library.

We can use virtual environments for projects A and B, and each virtual environment would be able to use their own version of project C without interfering with other virtual environments.

Why Use virtualenvwrapper?

At one point or another, any programmer would run into the problem of managing multiple virtual environments. virtualenvwrapper allows you to store all your virtual environments in one convenient location and provides methods to easily create, delete and switch between virtual environments. You can even specify different versions of Python for each virtual environment.

virtualenvwrapper is simply a set of extensions to virtualenv, making it easier to work with Python virtual environments.

Installing and Configuring virtualenvwrapper

Install virtualenv and virtualenvwrapper with pip.

$ pip install virtualenv

For macOS and Linux:

$ pip install virtualenvwrapper

For Windows:

$ pip install virtualenvwrapper-win

Now, we need to add a few lines to the shell’s startup file. First, find the exact location of the installed virtualenvwrapper.sh script.

$ which virtualenvwrapper.sh
/usr/local/bin/virtualenvwrapper.sh

Now, find your shell’s startup file. For Bash shell, it would be the ~/.bashrc file. Add the following lines into the file:

export WORKON_HOME=$HOME/.virtualenvs  
export PROJECT_HOME=$HOME/projects      
source /usr/local/bin/virtualenvwrapper.sh

Note that you should change /usr/local/bin/virtualenvwrapper.sh to the path you got from $ which virtualenvwrapper.sh if they are different.

Reload the startup file:

$ source ~/.bashrc

Check that it works; there should now be a directory at $WORKON_HOME that contains all your virtualenvwrapper files:

$ echo $WORKON_HOME
/Users/zhangzeyu/.virtualenvs

Using virtualenvwrapper

To create a new virtual environment, use the mkvirtualenv command.

$ mkvirtualenv my-project
(my-project) $

The new virtual environment is stored at the directory at $WORKON_HOME. This is a convenient location where all virtualenvwrapper environments are stored.

To stop using the environment, use the deactivate command.

(my-project) $ deactivate
$

This is where the power of virtualenvwrapper comes in. To list all your virtual environments, use the workon function.

$ workon
my-project
my-other-project
i-have-many-projects
2048-game

Now, select the virtual environment that you want to use:

$ workon 2048-game
(2048-game) $

To remove an environment, use the rmvirtualenv command.

$ rmvirtualenv 2048-game
Removing 2048-game...

Let’s say you want to use different versions of Python. mkvirtualenv has a -p parameter that allows you to select which version of Python to use.

$ mkvirtualenv python2-env -p python2
(python2-env) $ python -V
Python 2.7.17

Conclusion

That’s it! virtualenvwrapper allows you to easily manage your Python virtual environments, so that you can work on different projects without breaking things.

I Built a Python WhatsApp Bot to Keep Me Sane During Quarantine

Zhang Zeyu — Sat, 02 May 2020 15:26:20 +0000

This pandemic has taken a huge toll on my mental and emotional health. In order to keep me occupied and brighten up the lives of those around me, I started on yet another Python project — this time, a WhatsApp bot that sends me random cat pictures, trending memes, the best cooking recipes, and of course, the latest world news and COVID19 statistics.

The full project can be found on my GitHub repository, and my webhook is live on https://zeyu2001.pythonanywhere.com/bot/.

Prerequisites

We will be using Python, the Django web framework, ngrok and Twilio to create this chatbot. I will show you how to install the required packages, but you need to have Python (3.6 or newer) and a smartphone with an active phone number and WhatsApp installed.

Following Python best practices, we will create a virtual environment for our project, and install the required packages.

First, create the project directory.

$ mkdir whatsapp-bot
$ cd whatsapp-bot

Now, create a virtual environment and install the required packages.

For macOS and Unix systems:

$ python3 -m venv whatsapp-bot-venv
$ source whatsapp-bot-venv/bin/activate
(whatsapp-bot-venv) $ pip install twilio django requests

For Windows:

$ python3 -m venv whatsapp-bot-venv
$ whatsapp-bot-venv\Scripts\activate
(whatsapp-bot-venv) $ pip install twilio django requests

Configuring Twilio

You will need a free Twilio account, which allows you to use a Twilio number as your WhatsApp bot. A free account comes with a trial balance that will be enough to send and receive messages for weeks to come. If you wish to continue using your bot after your trial balance is up, you can top up your account.

You won’t be able to use your own number unless you obtain permission from WhatsApp, but the Twilio number would be good enough for this project. You will need to set up your Twilio sandbox here by sending a WhatsApp message to the Twilio number. This has to be done once and only once.

Setting Up Your Webhook

Twilio uses what is called a webhook to communicate with our application. Our chatbot application would need to define an endpoint to be configured as this webhook so that Twilio can communicate with our application.

Django is a web framework that allows us to do just that. Although the Django vs. Flask debate can go on for eternity, I chose to use Django simply because I have just started using it a few weeks ago and I wanted to get used to using it. You can use Flask to achieve the same thing, but the code would be different.

First, navigate to your whatsapp-bot directory and establish a Django project.

(whatsapp-bot-venv) $ django-admin startproject bot

This will auto-generate some files for your project skeleton:

bot/
    manage.py
    bot/
        __init__.py
        settings.py
        urls.py
        asgi.py
        wsgi.py

Now, navigate to the directory you just created (make sure you are in the same directory as manage.py) and create your app directory.

(whatsapp-bot-venv) $ python manage.py startapp bot_app

This will create the following:

bot_app/
    __init__.py
    admin.py
    apps.py
    migrations/
        __init__.py
    models.py
    tests.py
    views.py

For the sake of this chatbot alone, we won’t need most of these files. They will only be relevant if you decide to expand your project into a full website.

What we need to do is to define a webhook for Twilio. Your views.py file processes HTTP requests and responses for your web application. Twilio will send a POST request to your specified URL, which will map to a view function, which will return a response to Twilio.

from twilio.twiml.messaging_response import MessagingResponse
from django.views.decorators.csrf import csrf_exempt

@csrf_exempt
def index(request):
    if request.method == 'POST':
        # retrieve incoming message from POST request in lowercase
        incoming_msg = request.POST['Body'].lower()

        # create Twilio XML response
        resp = MessagingResponse()
        msg = resp.message()

This creates an index view, which will process the Twilio POST requests. We retrieve the message sent by the user to the chatbot and turn it into lowercase so that we do not need to worry about whether the user capitalizes his message.
Twilio expects a TwiML (an XML-based language) response from our webhook. MessagingResponse() creates a response object for this purpose.

resp = MessagingResponse()
msg = resp.message()
msg.body('My Response')
msg.media('https://example.com/path/image.jpg')

Doing this would create a response consisting of both text and media. Note that the media has to be in the form of a URL, and must be publicly accessible.

from twilio.twiml.messaging_response import MessagingResponse
from django.views.decorators.csrf import csrf_exempt
from django.http import HttpResponse

@csrf_exempt
def index(request):
    if request.method == 'POST':
        # retrieve incoming message from POST request in lowercase
        incoming_msg = request.POST['Body'].lower()

        # create Twilio XML response
        resp = MessagingResponse()
        msg = resp.message()

        if incoming_msg == 'hello':
            response = "*Hi! I am the Quarantine Bot*"
            msg.body(response)

        return HttpResponse(str(resp))

With this knowledge, we can now return a HttpResponse that tells Twilio to send the message “Hi! I am the Quarantine Bot” back to the user. The asterisks (*) are for text formatting — WhatsApp will bold our message.

This won’t work unless we link it to a URL. In the bot_app directory, create a file urls.py. Include the following code:

from django.urls import path
from . import views

urlpatterns = [
    path('', views.index, name='index'),
]

Now, we need the root URLconf to point to our bot_app/urls.py. In bot/urls.py, add the following code:

from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path('bot/', include('bot_app.urls')),
    path('admin/', admin.site.urls),
]

The include() function allows referencing other URLconfs. Whenever Django encounters include(), it chops off whatever part of the URL matched up to that point and sends the remaining string to the included URLconf for further processing.
When Twilio sends a POST request to bot/, it will reference bot_app.urls, which references views.index, where the request will be processed.

Testing It Works

Make sure you are in the directory with manage.py, and run

(whatsapp-bot-venv) $ python manage.py runserver

You should see the port that your Django application is running on. In this screenshot, it is port 8000. But this is still running from your computer. To make this service reachable from the Internet we need to use ngrok.

Open a second terminal window, and run

$ ngrok http 8000

The lines beginning with forwarding tell you the public URL ngrok uses to redirect requests to your computer. In this screenshot, https://c21d2af6.ngrok.io is redirecting requests to my computer on port 8000. Copy this URL, and go back to your Twilio console.

Paste the URL into the “When a message comes in” field. Set the request method to HTTP post.

If you want to use my app, use https://zeyu2001.pythonanywhere.com/bot/ instead for the “When a message comes in” field.

In your settings.py, you also need to add your ngrok URL as one of the ALLOWED_HOSTS.

Now you can start sending messages to the chatbot from your smartphone that you connected to the sandbox at the start of this tutorial. Try sending ‘hello’.

Adding Third-Party APIs

In order to accomplish most of our features, we have to use third-party APIs. For instance, I used Dog CEO’s Dog API to get a random dog image every time the user sends the word ‘dog’.

import requests

...

elif incoming_msg == 'dog':
    # return a dog pic
    r = requests.get('https://dog.ceo/api/breeds/image/random')
    data = r.json()
    msg.media(data['message'])

...

requests.get(url) sends a GET request to the specified URL and returns the response. Since the response is in JSON, we can use r.json() to convert it into a Python dictionary. We then use msg.media() to add the dog picture to the response.

My Final Code

My final chatbot includes the following features:

Random cat image
Random dog image
Random motivational quote
Fetching recipes from Allrecipes
Latest world news from various sources
Latest COVID19 statistics for each country
Trending memes from r/memes subreddit

Deploying Your Webhook

Once you’re done coding, you probably want to deploy your webhook somewhere so that it runs 24/7. PythonAnywhere provides a free Django-friendly hosting service.

In order to deploy your project on PythonAnywhere, upload your project to GitHub and follow the documentation to set up your web app.

Once deployed, update your Twilio sandbox configuration so that the webhook is set to your new URL (such as https://zeyu2001.pythonanywhere.com/bot/).

Thanks for Reading!

Now you know how to set up your very own WhatsApp chatbot with Twilio. If you have any questions, please feel free to comment on this post.