Forem: David Newberry

Pygame Snake, Pt. 3

David Newberry — Tue, 21 Apr 2026 23:25:18 +0000

Part 2 left off with a square moving on a grid. Now, let's take control of it by consuming KEYDOWN events.

One simple approach would be to have a variable that represents one of the four directions the snake can go. It might use values 0-3, or it could use string literals like "up" and "left".

A slightly more advanced technique would be to use the Vector2 class. We use pygame's Vector2 to keep track of the position of out snake part. We could also use a Vector2 to represent the snake's speed and direction. For example, the direction right would be represented by Vector2(1, 0), because if we add that vector to the current position, it will increase the x value (the first coordinate) by 1.

Left would be Vector2(-1, 0). Up and down would be Vector2(0, -1) and Vector2(0, 1), respectively. If up and down seem backwards, it's because in pygame (which uses a common computer drawing convention) the Y-axis increases as you go down, instead of up. The origin is the canvas's upper-left corner.

Add a new variable under dot:

vel = pygame.Vector2(1, 0)

Inside the for loop that consumes events, add the following under the first if statement:

        if event.type == pygame.KEYDOWN:
            if event.key == pygame.K_RIGHT:
                vel = pygame.Vector2(1, 0)
            if event.key == pygame.K_DOWN:
                vel = pygame.Vector2(0, 1)
            if event.key == pygame.K_LEFT:
                vel = pygame.Vector2(-1, 0)
            if event.key == pygame.K_UP:
                vel = pygame.Vector2(0, -1)

Finally, update the line of code that moves dot:

dot = dot + vel # or dot += vel

When you run your program now, you should be able to control the dot's movement with the arrow keys.

Full code at this point:

import pygame

W = 30
H = 30
S = 20

# pygame setup
pygame.init()
screen = pygame.display.set_mode((W * S, H * S))
clock = pygame.time.Clock()
running = True

dot = pygame.Vector2(W / 2, H / 2)
vel = pygame.Vector2(1, 0)

while running:
    # poll for events
    for event in pygame.event.get():
        # pygame.QUIT = user closed window
        if event.type == pygame.QUIT:
            running = False
        if event.type == pygame.KEYDOWN:
            if event.key == pygame.K_RIGHT:
                vel = pygame.Vector2(1, 0)
            if event.key == pygame.K_DOWN:
                vel = pygame.Vector2(0, 1)
            if event.key == pygame.K_LEFT:
                vel = pygame.Vector2(-1, 0)
            if event.key == pygame.K_UP:
                vel = pygame.Vector2(0, -1)

    # fill buffer with white
    screen.fill("white")

    dot += vel
    square = pygame.Rect(dot * S, (S, S))
    screen.fill("black", square)

    # copy buffer to screen
    pygame.display.flip()

    # limits FPS
    clock.tick(20)

pygame.quit()

OK, that's fine, but we could really tighten it up by introducing a dictionary to hold the key/velocity associations. Like this:

key_vel = {
    pygame.K_RIGHT: pygame.Vector2(1, 0),
    pygame.K_DOWN: pygame.Vector2(0, 1),
    pygame.K_LEFT: pygame.Vector2(-1, 0),
    pygame.K_UP: pygame.Vector2(0, -1)
}

Add that just above the while loop. Now the whole KEYDOWN if statement can be replaced with this:

        if event.type == pygame.KEYDOWN and event.key in key_vel:
            vel = key_vel[event.key]

When in is used with a dictionary like this, it checks the dictionary's keys to see if they contain the given value.

Full code now:

import pygame

W = 30
H = 30
S = 20

# pygame setup
pygame.init()
screen = pygame.display.set_mode((W * S, H * S))
clock = pygame.time.Clock()
running = True

dot = pygame.Vector2(W / 2, H / 2)
vel = pygame.Vector2(1, 0)

key_vel = {
    pygame.K_RIGHT: pygame.Vector2(1, 0),
    pygame.K_DOWN: pygame.Vector2(0, 1),
    pygame.K_LEFT: pygame.Vector2(-1, 0),
    pygame.K_UP: pygame.Vector2(0, -1)
}

while running:
    # poll for events
    for event in pygame.event.get():
        # pygame.QUIT = user closed window
        if event.type == pygame.QUIT:
            running = False
        if event.type == pygame.KEYDOWN and event.key in key_vel:
            vel = key_vel[event.key]

    # fill buffer with white
    screen.fill("white")

    dot += vel
    square = pygame.Rect(dot * S, (S, S))
    screen.fill("black", square)

    # copy buffer to screen
    pygame.display.flip()

    # limits FPS
    clock.tick(20)

pygame.quit()

Pygame Snake, Pt. 2

David Newberry — Tue, 21 Apr 2026 22:23:58 +0000

In part 1, we just got pygame going. We have a 1000x1000 pixel canvas, with a 50x50 square moving more-or-less continuously. Snake is played on a fixed grid.

Instead of 50x50 pixels, we'll shrink the snake parts/tiles to 20x20, and we'll shrink the canvas to 600x600 pixels, treated as a 30x30 grid. (30 tiles x 20 pixels = 600 pixels.)

We'll add three variables to represent these values:

W = 30 # width
H = 30 # height
S = 20 # tile size

Now we should replace all other numbers in the code with references to these values.

screen = pygame.display.set_mode((W * S, H * S))

dot = pygame.Vector2(W / 2, H / 2) # approx. the center

    square = pygame.Rect(dot * S, (S, S))

These values are referenced in several places in the code. By using variables, you can update them in one place and it will be reflected everywhere. You can easily change the width, height, or tile size.

Add this just above square = ... (the line shown above):

    if dot.x > W: dot.x = 0

Full code at this point:

import pygame

W = 30
H = 30
S = 20

# pygame setup
pygame.init()
screen = pygame.display.set_mode((W * S, H * S))
clock = pygame.time.Clock()
running = True

dot = pygame.Vector2(W / 2, H / 2)

while running:
    # poll for events
    for event in pygame.event.get():
        # pygame.QUIT = user closed window
        if event.type == pygame.QUIT:
            running = False

    # fill buffer with white
    screen.fill("white")

    dot.x += 1
    if dot.x > W: dot.x = 0
    square = pygame.Rect(dot * S, (S, S))
    screen.fill("black", square)

    # copy buffer to screen
    pygame.display.flip()

    # limits FPS
    clock.tick(20)

pygame.quit()

Now we are treating the board itself as a 30x30 grid of 1x1 tiles. It is then being scaled up when being drawn. The code dot * S takes the Vector2 variable with a value like (15, 15). Assuming S = 20, dot * S would then produce the point (300, 300), i.e. (15 * 20, 15 * 20).

Pygame Snake, Pt. 1

David Newberry — Tue, 21 Apr 2026 20:50:00 +0000

Pygame is a module designed to allow us to make 2D games using Python. I am just learning pygame myself. I've always loved the game Snake, and I find that it makes a good learning project, too.

When using pygame, we will draw each frame of our animation to an offscreen buffer. That is basically like an invisible canvas that exists in memory. This allows us to draw the scene piece by piece, and then when its done copy the whole buffer to the onscreen canvas.

Pygame will also handle the event loop. We can specify how many frames per second we want our game to run at, and pygame will make sure the event loop runs no faster than that. Each time through the loop, we will draw one frame of animation.

We will also respond to events that pygame gives us. In particular, pygame will send us a KEYDOWN event whenever a key is pressed. Also, if the user closes the pygame window, it will send us a QUIT event, and the game should stop.

Here is a minimal pygame setup. Modified code from https://www.pygame.org/docs/

import pygame

# initialize pygame
pygame.init()

# create 1000x1000 pixel pygame window
screen = pygame.display.set_mode((1000, 1000))

# clock used to set FPS
clock = pygame.time.Clock()

# game runs as long as this is True
running = True

while running:
    # poll for events
    for event in pygame.event.get():
        # pygame.QUIT = user closed window
        if event.type == pygame.QUIT:
            running = False

    # fill buffer with white
    screen.fill("white")

    # copy buffer to screen
    pygame.display.flip()

    # limits FPS
    clock.tick(20)

pygame.quit()

The only thing about this code I don't like is that it requires the running variable. I wanted to use an infinite loop with break, but I can't do that since we have to iterate through potentially multiple events using an inner (for) loop, and Python has no way to break out of nested loops.

OK, so that's not a bad start, except that it is not exactly the most interesting "animation". Let's add a moving square.

To have an animated square, we'll need a new variable to keep track of its location. If we weren't using pygame, I might create a simple little class for this (to store x and y coordinates). But pygame already has such a class built-in: Vector2.

So let's add a new variable called dot and create a Vector2 value for it to hold:

dot = pygame.Vector2(500, 500)

Add that code just above the while loop.

Now inside the loop increment dot's x value, and then draw it at its new position. To draw a square, we have to create a Rect object. To do this, we will pass two pairs of values: the (x, y) coordinates of its upper-left corner, and its (width, height) values. The dot variable itself will function as the (x, y) coordinates of the square's upper-left corner.

    dot.x += 10
    square = pygame.Rect(dot, (50, 50))
    screen.fill("black", square)

Place this code right under screen.fill("white"). That command paints the entire buffer white, in anticipation of drawing a new frame. The new three lines of code draw a 50x50 black square at the location indicated by the dot variable.

Full code now:

import pygame

# pygame setup
pygame.init()
screen = pygame.display.set_mode((1000, 1000))
clock = pygame.time.Clock()
running = True

dot = pygame.Vector2(500, 500)

while running:
    # poll for events
    for event in pygame.event.get():
        # pygame.QUIT = user closed window
        if event.type == pygame.QUIT:
            running = False

    # fill buffer with white
    screen.fill("white")

    dot.x += 10
    square = pygame.Rect(dot, (50, 50))
    screen.fill("black", square)

    # copy buffer to screen
    pygame.display.flip()

    # limits FPS
    clock.tick(20)

pygame.quit()

If you have a little experience with Python, try this challenge: Add some code to check if dot.x > 1000, and if so, set it to 0. This should cause the dot to reappear on the left once it goes off the right edge.

Installing pygame for IDLE on Mac

David Newberry — Tue, 21 Apr 2026 19:13:08 +0000

IDLE is the easy-to-use IDE that comes bundled with Python by default. It's not the greatest IDE in the world, but it's been steadily improving with each release.

If you work a lot with Python, you might also have conda installed, or something similar, to create virtual environments with different Python versions and packaged installed. Working in IDLE is going to sidestep all that.

If you have ever installed Python from https://www.python.org/, you will have IDLE installed already. If not, go download and install it now. Note: If you have an older Mac and run into problems, you may need to use Python 3.10 instead of the newest version. (This worked for me on my Mac running OS X 12.7.6 "Monterey".)

The first step requires finding the actual binary associated with IDLE.

Right-click on IDLE and select Show Package Contents.
Inside the package, open Contents, then MacOS.
Right-click on Python and choose Show Original.

Now you've located the Python binary for your version of IDLE. Highlight the file and choose Edit -> Copy. The path to the file will probably be something like this:
/Library/Frameworks/Python.framework/Versions/3.14/Resources/Python.app/Contents/MacOS/Python

The next step is done using the Terminal. Find the Terminal app in /Applications/Utilities/ and launch it.

Once the Terminal has started up fully and you see the command prompt, start by pasting the file path to the python binary on your clipboard.

After that, you will type or paste the following (note that it starts with a space):

 -m pip install -U pygame --user

-U will cause pygame to be upgraded if it's installed already
--user will cause pygame to be installed for the current user, rather than for all users

So the whole command will look something like this:

/Library/Frameworks/Python.framework/Versions/3.14/Resources/Python.app/Contents/MacOS/Python -m pip install -U pygame --user

That's it! If everything goes well, pygame will install and be available inside IDLE. If you run into a problem, leave a comment.

React, Git[Hub], and VS Code for Beginners - Video 3

David Newberry — Thu, 23 Oct 2025 03:26:47 +0000

Video 2 (about 18 min):
https://youtu.be/e_2t2MNYDFQ

In this video, I start by installing VS Code.
I start VS Code, and then open up the React project directory created in the first video.
I spend a minute looking at the file hierarchy of the project, and then open up the source file for a component called Welcome, and use it to talk a little bit about how components work.

I look a little more at the files. I don't see a files called App.js or App.tsx, which is sometimes the root of the React app. In this case, I overlooked root.tsx, so I'll explore that in the next video.
I start the React development server by running:

npm run dev

In the browser, I then load the site by visiting the URL printed in the terminal; as I recall, localhost:5173.

Finally for this video, I go back to the source in VS Code and demonstrate how the React dev server takes care of essentially refreshing and "Hot [Re]loading" the content that gets changed, if you edit the source code.

In the next video installment, I plan to:

Create a new branch in git
Create a new React component
Discuss how state is handled in React
Merge the working branch into main*

(*) As it happens, I didn't have time in that stream! But next time, for sure.

React, Git[Hub], and VS Code for Beginners - Video 3

David Newberry — Thu, 23 Oct 2025 03:26:47 +0000

Previous video blog: Part 2

Video 3 (about 18 min):
(Stream 2)
https://youtu.be/e_2t2MNYDFQ

In this video, I start by installing VS Code.
I start VS Code, and then open up the React project directory created in the first video.
I spend a minute looking at the file hierarchy of the project, and then open up the source file for a component called Welcome, and use it to talk a little bit about how components work.

I look a little more at the files. I don't see a files called App.js or App.tsx, which is sometimes the root of the React app. In this case, I overlooked root.tsx, so I'll explore that in the next video.
I start the React development server by running:

npm run dev

In the browser, I then load the site by visiting the URL printed in the terminal; as I recall, localhost:5173.

Finally for this video, I go back to the source in VS Code and demonstrate how the React dev server takes care of essentially refreshing and "Hot [Re]loading" the content that gets changed, if you edit the source code.

In the next video installment, I plan to:

Create a new branch in git
Create a new React component
Discuss how state is handled in React
Merge the working branch into main*

(*) As it happens, I didn't have time in that stream! But next time, for sure.

React, Git[Hub], and VS Code for Beginners - Video 2

David Newberry — Thu, 23 Oct 2025 03:13:29 +0000

Previous video blog: Part 1

Video 2:
(Stream 1, Part 2)
https://youtu.be/Vwr2P7OWAWI

So now, with luck, you have a React Router app. Before diving into it, create a repository on GitHub.

Signing up for GitHub is pretty straightforward.
Once you have an account, you can create a new repository.

Most of the default settings are fine. For simplicity I named it the same thing as my local repository. I did not include anything in the initial repo, because all the files I want to upload will come from the local repo.
Next I tried to follow the instructs to push connect the local repository to the remote repo and push the changes, but I ran into a snag. I needed to authenticate git to GitHub.

The solution I eventually found was to install the GitHub CLI tool, which is a separate binary from git, called gh. There were a few options for installing gh. I chose to download the binary and then run it by cd-ing into the directory and running:
```
./gh auth login
```
The ./gh indicates that I want to run a binary in my current working directory.

Another good option would be to first install Homebrew, and then use it to install gh. On my primary development computer I have Homebrew installed. I don't have to avail myself of it often, but I probably end up using it a few times a year to install some binary that would be hard to install otherwise.

Once I'm authenticated with GitHub, I'm able to push my the project.

I turn next to VS Code, starting the download.

That's it for this video. Next time, I actually open VS Code!

Next video blog: Part 3

React, Git[Hub], and VS Code for Beginners - Video 1

David Newberry — Thu, 23 Oct 2025 03:10:45 +0000

React, GitHub, VS Code Video Blog

This series of blogs is an accompaniment to a weekly coding live-stream that I am doing. This series is focused on the three topics listed above, and is aimed at people who are just getting started. For that reason, I am working on a computer that has never been used for development before, and installing all the software I need along the way.

I also decided not to rehearse these videos. My hope is that there is value in seeing the messiness of the process, and what it looks like to work through questions and issues in real time. The downside is that sometimes I'll get stuck, or change my mind and backtrack. When I post videos to YouTube, I will edit them a bit for time by speeding up stretches where I'm not talking.

You can join the weekly live stream Wednesday afternoons (around 1 PM, Pacific) on Twitch, and find past weeks' videos on my YouTube:
https://www.twitch.tv/paxfeline
https://www.youtube.com/@GrokProgramming

In these blogs, I will list out the important steps and call out any mistakes I make during the stream.

Video 1:
(Stream 1, Part 1)
https://youtu.be/K13tgiff5mo

Preliminary: Get git. On Mac, one way to do this is install Xcode.

If you have no intention of doing app development, or any non-web development, you might want to skip Xcode and get git directly from http://git-scm.com/. One way to do this on Mac is to use Homebrew, which is another thing to install, but makes getting some other binaries easier.
In order to run the commands to create a new React app and start the React dev server, I install Node.js, which comes with command line tools npm and npx. To install Node.js, I first installed nvm -- "node version manager" -- by running:
```
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash
```
(Per https://github.com/nvm-sh/nvm?tab=readme-ov-file)

When nvm finished installing, it tried to modify my shell .profile script, which is run whenever a new shell session starts. In particular nvm wants to add some code that will export an environmental variable called NVM_DIR, and run a script to load nvm.
I used the command line program pico to create .profile and paste in the code nvm told me to, then opened up a new terminal window. This started a new shell session, which executed .profile (thus exporting the variable and loading nvm).
Next, I installed Node.js by running nvm install node.
At that point I could run the command that creates a new React Router app.

First, in the terminal, navigate to the directory you want to create your new app directory in.

Beware my mistake! I told the script to create the project at ~/first-react-app. (To be fair, I'm pretty sure the script asked "where" I wanted it.) Generally, a path like that is interpreted as "in the home directory, an entry called first-react-app." However, in this case, in my current working directory (which was my home directory -- so the tilde-slash was redundant), it created a directory named "~" and placed "first-react-app" inside that.

Don't make my mistake. Navigate to where you want your project, and then just enter the name of the project you're creating.

The command to create a new React Router app is:
```
npx create-react-router@latest my-react-router-app
```
(Per https://reactrouter.com/start/framework/installation)

During the creation process, you will be asked to give your project a name, if you would like to initialize a git repository (you probably do), and a few other questions.
To start the React development server, and see your new app, run the following commands to move into the project directory, install necessary packages, and then start the server.
```
cd my-react-router-app
npm i
npm run dev
```

End of Video 1

Next video blog: Part 2

Tagged String Literals

David Newberry — Thu, 13 Feb 2025 18:42:45 +0000

I love seeing a line of code and thinking, I can not parse that. And then learning to parse it.

This is valid JavaScript:

This is an instance of using a tagged template string.

This works too:

These examples are from MDN. Learn more there. I have no use for this knowledge right now, but one day I may!

Hearsay, part 2: Props

David Newberry — Wed, 05 Feb 2025 04:25:20 +0000

My previous post on hearsay discusses how this problem came up for me.

In hearsay, I want to be able to pass data into components using the props attribute, and specifying a JavaScript object literal. Something like this:

<hear-say src="Comp.html" props="{ n: Math.random() }"></hear-say>

Let's start with the fundamental task: the attribute should be interpreted as JavaScript code. So I created a custom getter:

get props()
{    
    const prop_att = this.getAttribute("props")?.trim() || "{}";
    const prop_func = Function("self", `return ${prop_att};`);
    return prop_func(this);
}

This creates a new function with a single argument self, and the body of the function simply returns whatever the expression evaluates to. So every time this component accesses self.props.n, it will call Math.random() and return a new value.

This is good, but a straightforward setter turned out to be a problem.

set props(val)
{
    this.setAttribute("props", JSON.stringify(val));
}

When I tried updating the value of props, I ran into a problem. A line of code like self.props = { ...self.props } (never mind accomplishing anything useful) would result in the value changing from { n: Math.random() } to, e.g., { n: 0.42 }. The code would be overwritten with a value.

It occurred to me that I could create another property, call it propsData, that could hold data written to props. This way you can write whatever you want to props, and it won't overwrite the code there.

I changed the setter to store data in propsData. And it places the data into a props-data attribute to make it easy to see when inspecting the DOM.

set props(val)
{
    this.propsData = val;
    this.setAttribute("props-data", JSON.stringify(val));
}

It's a little weird, the behavior isn't immediately intuitive I'm afraid, but on balance I think it's a good approach. Now I just had to give myself more work.

I thought it would be nice if the user could update props (or propsData, really) by mutating it. The solution that came to mind was to create a special Proxy for props that also incorporates propsData.

What I want it to do is sort of synthesize props and propsData. When code references props.foo, I want it to check both props and propsData for foo. If it exists in both places, it will prefer the propsData version. If foo doesn't exist in propsData, it will return whatever props.foo holds (if anything).

So far so good. But what if props holds nested objects? Code that sets, e.g., props.foo.bar.baz = 42 will run into an issue. As it goes down the props chain (getting first foo and then bar) the corresponding chain under propsData is empty (there is no propsData.foo, much less propsData.foo.bar). To compensate, if the getter finds an object property in the props chain and no corresponding propsData property, it will fill it in with an empty object.

This is basically how the current version of the code works, with one more provision. When evaluating an expression like props.foo.bar = 42, foo will return an object wrapped in a Proxy, which basically holds two things: the value for foo inside props, and inside propsData. This is exactly what it does for the props chain, but the propsData chain needs to be a little different.

At first I had it construct an object with one field for the props chain and one for the propsData chain. The Proxy target for foo in props.foo, for example, would look like { props: {bar: 42}, propsData: {} }. But that leaves the propsData chain disconnected from propsData. Instead, the Proxy target changed: a prop property was added. Now instead of the propsData property holding the value of, e.g., foo.bar, it would hold the value of foo, and prop would be bar.

This setup allows a backwards chaining effect. The Proxy for bar would look like { props: 42, propsData: propsData.foo, prop: "bar" }. The Proxy's setter runs target.propsData[target.prop] = val. target.propsData will be propsData.foo, so the effect is the same as setting propsData.foo.bar = val.

I hope you found this post somewhat useful. I fear this explanation is not as clear as I had hoped. If you have any questions, please feel free to comment.

Hearsay: "Talkative Web Components"

David Newberry — Tue, 04 Feb 2025 02:25:39 +0000

A few years ago, I learned enough React to cobble together a project I had in mind (RiffTube). Some parts of it I liked, and others I didn't. I've always leaned more toward vanilla JavaScript, and been wary of frameworks.

Recently I began reworking RiffTube, and updating Node packages caused everything to break. I eventually got things working again, but the process reminded me of what I don't love about React.

The idea for a framework started forming in my mind. There were two core pieces:

Components, similar to React
Communication via broadcasting

It seems to me that React's strategy for handling state is both redundant and at adds with the way JavaScript does everything else. The state of a web app can be specified by listing out all the values of all the variables (such as, and including, all the objects and all their properties).

React adds another layer of variables, etc., on top of that, which is what seems redundant. And then, when it's "official" state gets updated, it re-renders the component, which seems to me at odds with the JavaScript ethos.

So I started working on hearsay.

In hearsay, there is no "special" state. So if you want a component to do something, like update a property/UI, you can broadcast a message. You can specify a recipient, so that the appropriate recipient can respond -- but every component can "hear" the message, and decide if they need to respond or not.

Components are implemented with a fairly lightweight custom element (MDN article). The custom element, <hear-say>, provides a few capabilities and behaviors.

src attribute: Specify the path to an HTML file.
- The contents of this file will be attached as the shadow DOM for the custom element.
- A <script> element can be used to call a special setup() function allowing you to:
  - Attach custom element callbacks.
  - Specify a special init() method.
  - Specify a special react() method to respond to messages that are broadcast.
slot() method:

A helper method for filling slots in the shadow DOM.
props attribute:

Inspired by React, I wanted to be able to pass data into a component. Using this attribute, you can specify an object literal with regular JavaScript code.

The props attribute ended up being more complicated to implement than I realized at first. In particular, if props started out with a value like { n: Math.random() }, then doing this.props = { ...this.props, foo: 42 } would overwrite the code expression with a value, like { n: 0.123, foo: 42 }.

So I wanted to come up with a way to make using props easy and intuitive. The solution I came up with involves (what I think is) a pretty neat use of Proxy. The next post will be all about that.

Here is an example of a very simple hearsay component. It includes code that calls broadcast(), and also a react() method that responds to the message.

Clicker.html:

<script>
    setup({
        init: self =>
        {
            self.num = 0;
            self.slot("slot", self.num);
        },
        react: (self, data, recipient) =>
        {
            if (data == "inc")
            {
                self.num++;
                self.slot("slot", self.num);
            }
        },
    });
</script>
<h1>Clicker</h1>
<div>
    You have clicked
    <slot></slot>
    times!
</div>

<button onclick="broadcast('inc')">
    Click
</button>

The HTML file that wants to use this component would include the hearsay.js script file, and use a <hear-say src="Clicker.html"></hear-say> element to display it.

<html>
    <head>
        <script src="hearsay.js"></script>
    </head>
    <body>
        <hear-say struct="Clicker.html"></hear-say>
    </body>
</html>

More information and examples:

Hearsay on GitHub

Also see my hearsay "build journal"

Talking to a Gmail POP3 server with Python

David Newberry — Sat, 07 Sep 2024 18:37:33 +0000

There is a short (under 6 minutes) video to go along with this blog.

POP is a relatively old protocol; used to retrieve email from an email server. The first version was specified in 1984. The version still in use today, POP3, was specified in 1996. To try it out, I went about connecting to a Gmail POP3 server.

The first step was looking up the POP3 settings -- what server to connect to, on what port. Google led me here, where I found the following information.

pop.gmail.com

Requires SSL: Yes

Port: 995

It mentions that SSL is required. This wasn't something I dealt with 25 years ago, when I was last messing around with POP. I was afraid it would be a headache, but it turned out to be no challenge whatsoever; with a little help from the Python docs, I arrived at this code.

import socket
import ssl

hostname = 'pop.gmail.com'
context = ssl.create_default_context()

with socket.create_connection((hostname, 995)) as sock:
    with context.wrap_socket(sock, server_hostname=hostname) as s:
        print(s.version())

It connects, and tells me what version of SSL is in use... or something. Great success! Time to start a conversation with the server.

Borrowing from the official RFC for POP3, here's an example POP3 conversation between a client and server/

C: <open connection>
S:    +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
C:    USER mrose
S:    +OK mrose is a real hoopy frood
C:    PASS secret
S:    +OK mrose's maildrop has 2 messages (320 octets)
C:    STAT
S:    +OK 2 320
C:    LIST
S:    +OK 2 messages (320 octets)
S:    1 120
S:    2 200
S:    .
C:    RETR 1
S:    +OK 120 octets
S:    <the POP3 server sends message 1>
S:    .
C:    QUIT
S:    +OK dewey POP3 server signing off (maildrop empty)
C:  <close connection>

The first thing that happens is that the server sends a greeting to the client. Friendly. So I'll add code to receive a message from the server.

When you ask to receive data from a socket, you have to specify a buffer size. The docs recommend a power of 2, such as 4096. Many responses from the server will come through all at once. Some won't; sometimes a message from the server will be broken across sever reads, and the buffer may not be filled to its maximum even if there is more to come.

In the case of POP3, the way to tell if a message has come in completely depends on which message is coming in. Most of the time the server is sending a single line of text. (As we will see again later, these have a carriage return and line feed characters at the end of each line.) Certain messages that could have a much longer response use another way to show they're done: a period on a single line by itself.

import socket
import ssl

hostname = 'pop.gmail.com'
context = ssl.create_default_context()

with socket.create_connection((hostname, 995)) as sock:
    with context.wrap_socket(sock, server_hostname=hostname) as s:
        print(s.version())
        data = s.read(4096)
        print(data)

Run again and, we get a greeting. Another great success! Notice that the line ends with "\r\n" -- carriage return and line feed characters.

You have to pass a buffer size to the read method. It will then have a buffer that size available to read data from the server -- but there's no guarantee of how much data will come into the buffer at a time. This means that a protocol needs some way of specifying when a message is complete. There are numerous strategies possible. POP uses two: for all messages, lines are ended with \r\n. For a short (one line) message, this is all that's required. For multi-line responses, a period on a line by itself indicates that the message is complete.

TLSv1.3
b'+OK Gpop ready for requests from 2601:1c0:8301:b590:f408:d66a:3029:16ad dq2mb54750689ivb\r\n'

Now we need to start talking back to the server. Time to create an I/O (or O/I) loop; get some user input and send it to the server. Oops! I can't send a string directly; that gives me a TypeError. I need to convert the message to bytes. The string encode() method will do that (the default encoding of utf-8 works fine).

Only, when I run it -- oops again! Nothing happens when my message is sent to the server. Because I forgot that messages coming from the client also need to end with \r\n. Another tiny tweak gives us:

import socket
import ssl

hostname = 'pop.gmail.com'
context = ssl.create_default_context()

with socket.create_connection((hostname, 995)) as sock:
    with context.wrap_socket(sock, server_hostname=hostname) as s:
        print(s.version())
        while True:
            data = s.read(4096)
            print(data)
            msg = input() + "\r\n"
            s.send(msg.encode())

Great, now I can actually try to log in!

TLSv1.3
b'+OK Gpop ready for requests from 2601:1c0:8301:b590:f408:d66a:3029:16ad g4mb5147337iow\r\n'
USER grokprogramming
b'+OK send PASS\r\n'
PASS trustno1
b'-ERR [AUTH] Application-specific password required: https://support.google.com/accounts/answer/185833\r\n'

OK, so following that link takes me to a page where I can set up an application specific password. One potential stumbling block I came across: your account has to have two-factor authentication turned on in order for you to be able to make an application specific password, as far as I can tell. Why would I not have two-factor authentication turned on in the year of our Lorde 2024? I can't say. I do now.

Armed with an application specific password (mind you take out the spaces), I can log in! Then I'll issue the STAT command which will tell me how many messages I have, and their combined size. After that, I'll issue the LIST command, which will return a list of messages with an ID and a size for each one.

TLSv1.3
b'+OK Gpop ready for requests from 2601:1c0:8301:b590:f408:d66a:3029:16ad e18mb76868856iow\r\n'
USER grokprogramming
b'+OK send PASS\r\n'
PASS baygdsgkmihkckrb
b'+OK Welcome.\r\n'
STAT
b'+OK 263 14191565\r\n'
LIST
b'+OK 263 messages (14191565 bytes)\r\n1 2778\r\n2 2947\r\n3 6558\r\n4 9864\r\n5 35997\r\n6 45462\r\n7 45462\r\n8 63894\r\n9 11487\r\n10 74936\r\n11 74925\r\n12 11632\r\n13 32392\r\n14 74997\r\n15 51961\r\n16 15375\r\n17 46513\r\n18 21519\r\n19 15966\r\n20 27258\r\n21 28503\r\n22 35615\r\n23 86353\r\n24 280'

I've hit a bug in the code. The response for LIST spans multiple lines and, in this case, will require multiple buffer reads. The whole message will end with a period on a line by itself. Here I've received one buffer worth of the message, and now I'll have to hit return and send a blank message to the server in order for the code to advance to the next iteration of the loop and read from the buffer again.

I'll tweak the code so that the user always has the option of reading from the buffer again, or not. I'll also finally decode the incoming bytes from the server, so that the text renders nicer.

import socket
import ssl

hostname = 'pop.gmail.com'
context = ssl.create_default_context()

with socket.create_connection((hostname, 995)) as sock:
    with context.wrap_socket(sock, server_hostname=hostname) as s:
        print(s.version())
        while True:
            data = s.read(4096)
            print(data.decode())
            while input("more? y/[n]: ") == "y":
                data = s.read(4096)
                print(data.decode())
            msg = input("> ") + "\r\n"
            s.send(msg.encode())

And here's a full session including retrieving an email and sending the disconnect message.

> USER grokprogramming
+OK send PASS

more? y/[n]: 
> PASS trustno1
+OK Welcome.

more? y/[n]: 
> STAT
+OK 263 14191565

more? y/[n]: 
> LIST
+OK 263 messages (14191565 bytes)
1 2778
2 2947
3 6558
<...>
260 41300
261 114059
262 174321
263 39206
.

more? y/[n]: 
> RETR 1
+OK message follows
MIME-Version: 1.0
Received: by 10.76.81.230; Thu, 28 Jun 2012 20:21:50 -0700 (PDT)
Date: Thu, 28 Jun 2012 20:21:50 -0700
Message-ID: <CADBp03TWFOKcTOaK_0P7VV2GB+TZsoSd_W4G5nZKKs7pdk6cWQ@mail.gmail.com>
Subject: Customize Gmail with colors and themes
From: Gmail Team <mail-noreply@google.com>
To: Grok Programming <grokprogramming@gmail.com>
Content-Type: multipart/alternative; boundary=e0cb4e385592f8025004c393f2b4

--e0cb4e385592f8025004c393f2b4
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

To spice up your inbox with colors and themes, check out the Themes tab
under Settings.
       Customize Gmail =BB <https://mail.google.com/mail/#settings/themes>

Enjoy!

- The Gmail Team
[image: Themes thumbnails]

Please note that Themes are not available if you're using Internet Explorer
6.0. To take advantage of the latest Gmail features, please upgrade to a
fully supported
browser<http://support.google.com/mail/bin/answer.py?answer=3D6557&hl=3Den&=
utm_source=3Dwel-eml&utm_medium=3Deml&utm_campaign=3Den>
..

--e0cb4e385592f8025004c393f2b4
Content-Type: text/html; charset=ISO-8859-1

more? y/[n]: y

<html>
<font face="Arial, Helvetica, sans-serif">
<p>To spice up your inbox with colors and themes, check out the Themes tab
under Settings.</p>

<table cellpadding="0" cellspacing="0">
  <col style="width: 1px;"/>
  <col/>
  <col style="width: 1px;"/>
  <tr>
    <td></td>
    <td height="1px" style="background-color: #ddd"></td>
    <td></td>
  </tr>
  <tr>
    <td style="background-color: #ddd"></td>
    <td background="https://mail.google.com/mail/images/welcome-button-background.png"
        style="background-color: #ddd; background-repeat: repeat-x;
            padding: 10px; font-size: larger">
          <a href="https://mail.google.com/mail/#settings/themes"
            style="font-weight: bold; color: #000; text-decoration: none;
            display: block;">
      Customize Gmail &#187;</a>
    </td>
    <td style="ba
more? y/[n]: y
ckground-color: #ddd"></td>
  </tr>
 <tr>
    <td></td>
    <td height="1px" style="background-color: #ddd"></td>
    <td></td>
  </tr>
</table>

<p>Enjoy!</p>

<p>- The Gmail Team</p>

<img width="398" height="256" src="https://mail.google.com/mail/images/gmail_themes_2.png" alt="Themes thumbnails" />

<p><font size="-2" color="#999">Please note that Themes are not available if
you're using Internet Explorer 6.0. To take advantage of the latest Gmail
features, please
<a href="http://support.google.com/mail/bin/answer.py?answer=6557&hl=en&utm_source=wel-eml&utm_medium=eml&utm_campaign=en"><font color="#999">
upgrade to a fully supported browser</font></a>.</font></p>

</font>
</html>

--e0cb4e385592f8025004c393f2b4--
.

more? y/[n]: 
> QUIT
+OK Farewell.

more? y/[n]: 
>

Yet another great success! I was able to log in to the POP3 server and retrieve a message. The script in its current state is pretty flexible, but it requires a lot of work from the user. I'll make a few final tweaks to make interacting with the POP3 server a little easier: if the user starts a message to the server with a "!" it will be stripped out, but the script will read in data from the server until it gets to a period on a line by itself -- in other words, for commands with long responses. No "!" and the script will read in a single line, looking for the \r\n characters.

import socket
import ssl

hostname = 'pop.gmail.com'
context = ssl.create_default_context()

def read_until(s, eom):
    # read into the buffer at least once
    data = s.read(4096)
    # continue reading until end of message
    while data[-len(eom):] != eom:
        data += s.read(4096)
    # return incoming bytes decoded to a string
    return data.decode()

def read_single_line(s):
    return read_until(s, b"\r\n")

def read_muli_line(s):
    return read_until(s, b"\r\n.\r\n")

with socket.create_connection((hostname, 995)) as sock:
    with context.wrap_socket(sock, server_hostname=hostname) as s:
        print(s.version())
        print(read_single_line(s))
        msg = input("> ")
        # empty msg will close connection
        while msg != "":
            if msg[0] == "!":
                msg = msg[1:]
                long = True
            else:
                long = False
            msg += "\r\n"
            s.send(msg.encode())
            if long:
                print(read_muli_line(s))
            else:
                print(read_single_line(s))
            msg = input("> ")
        s.close()