Forem: Tony Colston

hot or not in Memphis in my life (python,sqlite)

Tony Colston — Mon, 27 Jun 2022 13:02:04 +0000

I saw this post

https://creon.substack.com/p/hot-or-not

And I decided to replicate it for myself using python and sqlite. I was born in the 60's so my personal statement is I am certain it is hotter now than when I was a kid.

So I went here to get data https://mesonet.agron.iastate.edu/request/download.phtml?network=TN_ASOS

And downloaded the data for TN near me. Mainly looking at air temperature.

From that I created a sqlite database and loaded the CSV.

CREATE TABLE data (
    station varchar,
    dt datetime,
    tmpf real,
    dewf real,
    relh real
)
.mode csv
.import asos.csv data

To make my life easier I made some new columns for year and month (to split out the date).

I then took the MAX temp for each day, the statement I am looking to validate is I think it is hotter now so MAX temp seems like a fine thing to use. Another reason for taking the MAX temp for each day is the data I am using has multiple measurements per day, I just need the MAX one for the day.

I made a 2nd table mainly to make my life easier.

create table mem_daily_max as select y,m,d,max(tmpf),max(dewf),max(relh) from data where station='MEM'  group by y,m,d

Then I averaged the temp for each month since 1990 and plotted it. Different months show different trends like the summer months show less changes. Memphis in the summer has always been and continues to be so.

If you look at a winter month though the trend is clear.

The raw data for December from 1960 till now

It is fairly hard to see much in that. The python code is below

import sqlite3

sql ="""
SELECT y,m,d,tmpf FROM 
mem_daily_max WHERE m=12 AND y>1960 
ORDER BY y,m,d ASC
"""

db = None
cur = None
try:
    db = sqlite3.connect("data.db")
    cur = db.cursor()
    cur.execute(sql)
    data = cur.fetchall()
finally:
    if cur is not None:
        cur.close()
    if db is not None:
        db.close()

pd = []
for (y,m,d,tmpf) in data:
    print(y,m,d,tmpf)
    pd.append(tmpf)

import matplotlib.pyplot as plt
plt.plot(pd)
plt.xlabel("max temp")
plt.savefig("foo.png")

So I took the average for each month to smooth it and the chart below only looks at December.

The average for December with a linear regression line. Which it is fairly clear that the air temp is going up in December at least over my lifetime.

import sqlite3

sql ="""
SELECT y,m,avg(tmpf) as tmp FROM mem_daily_max 
WHERE m=12 GROUP BY y,m ORDER BY y,m asc;
"""

# just used some code I found online for this function
def estimate_coef(x, y):
    # number of observations/points
    n = np.size(x)

    # mean of x and y vector
    m_x = np.mean(x)
    m_y = np.mean(y)

    # calculating cross-deviation and deviation about x
    SS_xy = np.sum(y*x) - n*m_y*m_x
    SS_xx = np.sum(x*x) - n*m_x*m_x

    # calculating regression coefficients
    b_1 = SS_xy / SS_xx
    b_0 = m_y - b_1*m_x

    return (b_0, b_1)

# just used some code I found online for this function
def plot_regression_line(x, y, b):
    # plotting the actual points as scatter plot
    plt.scatter(x, y, color = "m",
               marker = "o", s = 30)

    # predicted response vector
    y_pred = b[0] + b[1]*x

    # plotting the regression line
    plt.plot(x, y_pred, color = "g")

    # putting labels
    plt.xlabel('x')
    plt.ylabel('y')

    # function to show plot
    plt.savefig("goo.png")

db = None
cur = None
try:
    db = sqlite3.connect("data.db")
    cur = db.cursor()
    cur.execute(sql)
    data = cur.fetchall()
finally:
    if cur is not None:
        cur.close()
    if db is not None:
        db.close()

pd = []
for (y,m,tmp,) in data:
    pd.append(tmp)

# just plot the data
import matplotlib.pyplot as plt
plt.plot(pd)
plt.xlabel("max temp")
plt.savefig("foo.png")

# now run the regression
import numpy as np
x = np.array([i for i in range(len(pd))])
y = pd

b = estimate_coef(x,y)
print("Estimated coefficients:\nb_0 = {}  \
          \nb_1 = {}".format(b[0], b[1]))
plot_regression_line(x,y,b)

great list of "cool" sites from hnews

Tony Colston — Sun, 12 Jun 2022 14:23:06 +0000

This thread is great https://news.ycombinator.com/item?id=31708366

Here is a list of some of the cool sites there are others:

https://geizhals.de/ - this is a german site but the UI is nice and you can find a lot of stuff.
https://www.gsmarena.com/search.php3? - a phone search site. When I was at https://crossbrowsertesting.com we used this site a lot
https://www.howacarworks.com/ - how a car works
https://www.mcmaster.com/ - the UI here is so nice. those illustrations
https://ciechanow.ski/mechanical-watch/ - how does a mechanical watch work?
https://ciechanow.ski/exposing-floating-point/ - floating point. all programmers should read this.
http://worrydream.com/
https://apolloinrealtime.org/ - this is amazing if you are a space nerd.

There is a lot more stuff on the thread

note about conda envs

Tony Colston — Sun, 12 Jun 2022 13:59:42 +0000

I can never remember this for some reason ...

The command to list environments you have on disk

conda env list

The command to activate an environment

activate <envname>

I always set the PATH variable myself so those will only work if the anaconda directory and the anconda\Scripts directory on are on your PATH.

is there a defense of crypto? (bitcoin specifically)

Tony Colston — Fri, 10 Jun 2022 13:24:12 +0000

I saw this on hnews today: https://blog.cryptographyengineering.com/2022/06/09/in-defense-of-cryptocurrency/

Such raw naked hunger and greed on display here.
Literally burning enough energy to power an entire country to run hashing algorithms. And the days pass with hollow promises of better tech.
Fix that first, fix that now and show the world your community actually gives AF.
But you will not.
Nor will your community. Crypto is fine and happy to help make the world burn hotter.
If you hold bitcoin or mine bitcoin you have blood on your hands.
It is that simple.
Until bitcoin is stopped none of the rest of the article matters.
Regulate it? Attack it? You pick but either way it must be stopped.
The crypto community has no will to do it themselves.
The tech is interesting to me but a central database solves most of the problems and does not burn the world in the process.
Hopefully the newer blockchain tech will take over but I doubt it highly.

pretzels

Tony Colston — Sun, 17 Apr 2022 19:46:35 +0000

I made pretzels this weekend

I used this recipe
https://www.allrecipes.com/recipe/6737/mall-pretzels/

I actually followed the recipe instead of randomly changing things as I normally do.

I did not split the dough into small enough portions though. I think next time I will split into 16 parts. I think it will yield something that looks more like a pretzel instead of what I ended up with.

This recipe also does not require the dough to be boiled and uses a dip into water with baking soda instead of a boil. So much easier setup.

They still tasted good. :)

mandelbrot with turtles and Python

Tony Colston — Sat, 20 Nov 2021 22:34:59 +0000

I used Python turtles to draw an image of the madelbrot set.

Here is the code I found on the internet that does the math part.
https://www.codingame.com/playgrounds/2358/how-to-plot-the-mandelbrot-set/mandelbrot-set

But I wanted to draw this with a turtle.

All of my code is here in this gist
https://gist.github.com/tonetheman/806a4fe0924af085d614557c0ccb85e3

It is fairly slow to run and the UI locks up... but it will make a nice picture.

Crazy that you do this in turtle (though not too quickly).

And here is the image that the code will output

quick textures without extra files in Phaser3

Tony Colston — Tue, 09 Nov 2021 00:51:48 +0000

A really quick simple trick to generate textures to use in Phaser3.

You can use simple arrays of strings. The values are hexadecimal. You call another method that will generate the texture from the strings.

export let  bar = [
  "11111111111111111111",
  "1.FEDCBA9876543210.1",
  "11111111111111111111",
];
this.textures.generate("bar", {
    data : bar,
    pixelWidth : 16
});
this.textures.generate("bar8", {
    data : bar,
    pixelWidth: 8
});

Once you have the textures generate with the keys. Then you can use these as if they were image files all along.

let img = this.add.image(0,0,"bar");
img.setOrigin(0,0);

let img2 = this.add.image(0,64,"bar8");
img2.setOrigin(0,0);

The palette values look like they come from here
https://androidarts.com/palette/16pal.htm

The Phaser3 documentation calls this Arne16. See here: https://photonstorm.github.io/phaser3-docs/Phaser.Textures.TextureManager.html

data uri textures with Phaser3

Tony Colston — Sun, 07 Nov 2021 16:11:20 +0000

Loading textures from Phaser3 using a data uri instead of an image file.

Be clear that this does not save space. Loading from a data uri requires that the data (the image) is encoded in a way that can fit nicely into ASCII text. In this case it is encoded in Base 64 which is larger than the original file.

Loading Image elements in Javascript is asynchronous. So you must wait for the Image to be ready.

I used Scenes to handle the loading versus the usage of the texture.

You will have a LoadingScene that will load the data-uri into the TextureManager then once that is done we will move to the next GameScene that will use the textures in Sprite objects.

The tricky bit is that I am listening to the events on the TextureManager for the scene for "textureloaded" events. I keep up with the number of events I get and use that to move on to the next Scene.

How did I know that? I looked in the source for Phaser3! This is the call for addBase64
https://github.com/photonstorm/phaser/blob/a2956176dd5d64ebb6c00a4ad12660f1e6c824eb/src/textures/TextureManager.js#L265

And you can see in that source once the Image is loaded two events are fired. I picked the "addtexture" event.

The source is down below for the main portion of the code


import { mystring } from "./ds.js";
import { mystring as mystring2 } from "./ds2.js";

class LoadingTexturesOnly extends Phaser.Scene {
    constructor() {
        super("loading");
    }
    preload() {
        let count = 0;

        // had to read phaser source to find this
        // when you get a addtexture event the texture
        // is done loading
        this.textures.on("addtexture", () => {
            count++;

            // note I am checking for 2
            // this is the number of textures that I
            // expect to be loaded and it MUST MATCH
            // the number of times you call addBase64
            if (count==2) {
                // move on to the next scene
                this.scene.start("game");
            }
        });

        // ask phaser3 to load the data uri
        // I am loading 2 textures this way
        this.textures.addBase64("1",mystring);
        this.textures.addBase64("2",mystring2);
    }
}

class GameScene extends Phaser.Scene {
    constructor() {
        super("game");
    }

    create() {
        // the keys "1" and "2" were loaded
        // in the LoadingTexturesOnly scene above
        this.add.sprite(100,100,"1");
        this.add.sprite(100,200,"2");
    }

}

function main() {
    let gameOpts = {
        width : 600,
        height : 800,
        scene : [LoadingTexturesOnly,GameScene]
    };
    let game = new Phaser.Game(gameOpts);
}

window.onload = main;

References:
data url : https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs
data uri scheme: https://en.wikipedia.org/wiki/Data_URI_scheme
base64 encoding: https://developer.mozilla.org/en-US/docs/Glossary/Base64
import statement: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import
export statement: https://developer.mozilla.org/en-US/docs/web/javascript/reference/statements/export

bash relative pathing for writing files

Tony Colston — Sat, 10 Apr 2021 17:10:46 +0000

Say you have a process that needs to run and writes it logs relative to where it runs.

As a tiny example here is a python program that writes a file using a relative path (the filename is /tmp/cool.py)

outf = open("./dbg.txt","w")
outf.write("hey\n")
outf.close()

If you want to invoke that process from BASH then you have to be careful.

The file dbg.txt will be written relative to your working directory.

So in the example below the dbg.txt file is written in your home directory.

agcc@LAPTOP-BT02BD1T:~$ pwd
/home/agcc
agcc@LAPTOP-BT02BD1T:~$ python /tmp/cool.py
agcc@LAPTOP-BT02BD1T:~$ ls dbg.txt
dbg.txt

That is probably not what you wanted to do. More than likely you needed to change your working directory and then run the python script.

A really great tool to do that is subshell. In BASH that is parens around the commands.

Below I run the set of commands in a subshell. In the new subshell I change to a different directory and execute the python script then verify the file is present. I also printed my working directory.

Finally I executed an ls in my top shell (no longer in a subshell) and there is no file written in that directory.

agcc@LAPTOP-BT02BD1T:~$ (cd /tmp && python cool.py && ls dbg.txt && pwd)
dbg.txt
/tmp
agcc@LAPTOP-BT02BD1T:~$ pwd
/home/agcc
agcc@LAPTOP-BT02BD1T:~$ ls dbg.txt
ls: cannot access 'dbg.txt': No such file or directory

learning nim lang

Tony Colston — Mon, 14 Dec 2020 04:33:00 +0000

I have been working through advent of code trying to learn nim.

One thing that was not super clear to me was the overloading and definition of procs with objects/tuples.

I made a gist: https://gist.github.com/tonetheman/35c62e72d3d9cf7f333de40147bc5991

Nim looks like a ugly Python/Pascal. But how you call methods/fields on objects reminds me a lot of lua.

how to run https and localhost with caddy2

Tony Colston — Tue, 06 Oct 2020 22:54:30 +0000

Caddy alludes to this but due to reasons there is no example. And it took me a long while to figure it out.

If you run this command (on Windows at least)

caddy file-server --domain localhost

The first time Caddy will add a cert to your machine and then you can run https://localhost in your browser. YAY!

Again it was not obvious from the documentation in any way shape or form that this is the right way to do it. I am not convinced this is what they intended but it works.

your language PATH ...

Tony Colston — Mon, 24 Aug 2020 13:01:05 +0000

Great twitter thread
https://twitter.com/Grady_Booch/status/1297683779931144192

I had to think about my own path.

BASIC -> assembly -> PASCAL -> C -> C++ -> Java -> Visual Basic -> COM (not a language) -> SQL -> Python -> Javascript -> Perl -> bash -> PL/SQL -> C#.dot net -> VB.dot net -> JS.dot net -> more C# -> lua -> Clojure -> Racket

I think I missed some and COM is not a language but it was a moment where I really thought about function pointers and how binaries were put together at runtime. So I included it. Ha.