Forem: Jamie Read

Introducing Japr - The Project Linter

Jamie Read — Sun, 21 Jan 2024 20:59:55 +0000

Japr is a cross-language tool for rating and enforcing the overall quality of projects by looking at tool & language setup

Over summarized: It's a linter that makes sure you install linters (and some other stuff)

GitHub | PyPi | Docker

Japr

Japr is a linter to help you make sure your project is setup as best as it can be. It checks things like:

Do you have a readme? Does it have a getting started section or some installation docs?
Have you setup a linter for your language? Are you following the best practices for setting up those languages?
Is git setup correctly? Using main instead of master and excluding the right files?

Obviously project setup depends highly on the project's purpose so Japr has four configurations available:

Personal - A lightweight ruleset designed to tidy up personal projects
Team - A balanced ruleset designed for projects that belong to a single team
Inner Source - A comprehensive ruleset for projects that are accessible across an organisation and may be used by other teams (often referred to as 'inner source' projects
Open Source - A comprehensive ruleset for open source projects that are for anyone and everyone that might stumble upon the code

Japr is a very strict linter, it's opinionated and hard to satisfy so don't be upset if your project doesn't pass straight away! I don't have a single project that meets all of Japr's demands (even, ironically, Japr itself). Make use of the auto fixes and don't be afraid to suppress things if you disagree!

Rationale

Think about your favourite open source project. Chances are it's very easy to get started with because you just looked at the readme and followed the steps. Perhaps you've raised an issue and they'd setup issue templates to help you help them. Perhaps you committed some code, ran their linters and their CI/CD pipelines.

Now think about a time when the codebase hurt you, maybe that proof-of-concept code you inherited from a coworker full of .DS_Store files or that GitHub project that might have been perfect for you if you could actually build it.

A lot of the differences between those two projects probably seem like common sense. Create a readme, setup CI/CD, create a .gitignore but there's plenty more things we take for granted in the best projects.

We lint basic things like our code formatting which could also be fixed with common sense, why not lint our project setup?

Getting Started

You can install Japr using pip and run as you expect:

pip install japr
japr <path> -t <project-type>

Or just run it in Docker:

docker run --rm -v $(pwd):/app jamoyjamie/japr:v1.0.1 -t <project-type>

Project type should be one of personal, team, inner-source or open-source depending on what best suits your project!

There's also experimental auto fixing for some issues which you can try by adding --fix!

Future

I'll be using Japr on new projects and retrofitting old projects as I work on them again. I intend to add support for the other languages and tools I use as I go (e.g. terraform, helm) but I'll gladly accept advice, discussion and pull requests for other languages or general debate into the rulesets so that they represent a consensus.

Please give it a try, feedback and let me know what your highest project scores are!

Pyash (and why I can't have nice things anymore)

Jamie Read — Mon, 06 Jan 2020 23:47:10 +0000

A few months back an innocent thought came into my mind:

Can you make Python run this bash statement...?

cat "kittens.txt" | grep "fluff" > fluffy_kittens.txt

Yes. Well nearly. But, mostly yes.

An Adventure Begins

The Saturday after my innocent thought was spent trying to get as close as possible to making Python run the above line.

Most users probably see cat "kittens.txt" and think it isn't in the slightest syntactically correct and would be completely impossible to get to compile. You would be right actually. I never worked this one out.

So I lowered my goal a little:

cat("kittens.txt") | grep("fluff") > fluffy_kittens.txt

This is a little more reasonable! Now we have simple, Pythonic function calls surrounded by errors!

Let's start.

The first thing to do is to make cat and grep functions that we can call here, since we're obviously making function calls. I really wanted the ability to run any executable on my PC, just like you can in bash, because I really like that aspect of shell scripting, so I wrote something along the lines of this:

import os, glob

for path in os.environ["PATH"].split(";"):
    for executable_path in glob.glob(path + "\\*.exe"):
        name = os.path.splitext(os.path.basename(executable_path))[0]

        globals()[name] = lambda *args, e=executable_path: os.system("\"" + e + "\" " + " ".join(args))

(I work on a Windows machine with Git Bash, hence the search for .exe files)

It's janky, but this chunk of Python essentially searches through each path in the PATH variable for executables and creates a lambda function that runs the executable when called. It then adds that to the dictionary of globals so that you can now do things like:

mv("kittens.txt", "bed.txt")

Or even:

python("world-domination.py")

Great!

And it was at this point I realised that the print function had been overwritten to invoke print.exe instead...

Sigh 😒

Frankenstein's Python

This is a great start. At this point I actually moved that code into a library of its own so that you can import the functions individually (it helped with the whole print issue!)

from shell import cat, grep, kubectl # not you print

Let's look at a harder example:

ps() > "processes.txt"

In bash, this would take the output of the ps command and put it in the specified text file.

You'll notice that I've enclosed the name of the file in quotes, it's still valid bash so it doesn't count as cheating... 👀

This isn't so hard. In normal Python, this just gets evaluated as: is the result of ps() greater than the string "processes.txt"? True or False? If we return a custom object from ps() we can then overload the greater than operator to be able to compare itself with strings. (Overloading operators is essentially just a fancy term for redefining them with a custom implementation, so Python will use our implemention for performing greater than comparisons rather than its normal one.)

Here's a simple class we can return from ps() that does it:

class EvaluationResult:
    def __init__(self, output):
        self._output = output

    def __gt__(self, target):
        with open(target, 'w') as f:
            f.write(self._output)

In this case we've overridden the greater than operator to write to a file of the name taken from the string it's comparing against.

I won't show my full code, because at this point I switched to using subprocess.Popen instead of os.system to run the executables, and that's a whole other ropic I don't want to explain here.

Let's throw in a different example:

print(ps())

Here we want to output the result of ps to the console but with the above code you'll just get the memory address of our EvaluationResult class:

<__main__.EvaluationResult object at 0x03783030>

An easy fix, we just need to overload __str__:

class EvaluationResult:
    def __init__(self, output):
        self._output = output

    def __gt__(self, target):
        with open(target, 'w') as f:
            f.write(self._output)

    def __str__(self, target):
        with open(target, 'w') as f:
            f.write(self._output)

(__str__ is called when Python wants a string representation of an object, you can override this to provide better visibility into your classes while debuging if you like!)

So how about our original example then?

cat("kittens.txt") | grep("fluff") > fluffy_kittens.txt

The pipe operator (|, normally or) is also available to overload in Python, so all we need to do is overload it to pass the output of our first executable to the input of the second.

Unfortunately, Python, like most languages, executes cat("kittens.txt") first, then executes grep("fluff"), and then finally executes the or. By the time we get to our overloaded or operator we'll have already executed the second command and wont be able to pass it the output from the first (in actual fact, grep will have hung waiting for input, so the or overload we write won't ever be called here anyway, doh).

The only solution I came up with for this was to lazily run the executables underneath the functions, that is to say that when you run cat("kittens.txt") it doesn't actually run the cat executable until the or gets evaluated later, at which point we know where it's output goes.

class LazyInvocation:
    def __init__(self, output):
        self._output = output

    def __gt__(self, target):
        self.run()

        with open(target, 'w') as f:
            f.write(self._output)

    def __or__(self, target):
        self.run()
        target._input = self._output

    def run(self):
        # actually run the command, passing self._input in and assigning self._output

It makes everything a little wierd to run now though, because:

ps()

Won't actually run ps anymore...

Pyash

By the end of my Saturday, I had a nice set of bash syntax implemented!

from pyash import cat, grep, rm

cat("animals.txt") | grep("cat") | grep("-v", "caterpillar") > "nice_things.txt"
find(".", "-name", "*sunset*") >> "nice_things.txt"

print(nc("localhost", "22") < "nice_things.txt")

rm("nice_things.txt")

So I present to you my first PyPI repository, Pyash!

JamJar00 / pyash

Shell scripting... but in glorious Python!

Pyash

Hate shell scripting? Love Python! Meet the horrible Frankenstein of the two!

Pyash takes the best part of shell scripting, the ability to stream data through great chains of shell programs, and adds that ability to Python. In the process it retains the same familiar syntax but in a more Pythonic form!

from pyash import cat, grep
cat(".gitignore") | grep("env") > "out.txt"

Importing Shell Programs

Importing shell programs is as easy as

from pyash import grep, find, bash

Pyash automatically searches your PATH variable and loads any executable programs it finds, making them available through import.

Try to avoid importing * as you'll find basic builtin functions like print() suddenly don't work!

Running Shell Programs

Once you've imported the programs you want to import you can use them as you would any normal Python function splitting arguments or keeping them…

View on GitHub

('pyash' being 'python-bash', if you didn't get that... 😬)

I don't recommend you use it; I can't imagine your coworkers would be best pleased with you using it either, and frankly it fails a number of PEP-8 style guidelines. However, it does go to show how much you can abuse Python and what you can do with a little creativity!

It's not nice, nor is it a good idea and I feel like I've made a very beautiful language into a monster. This is why I can't have nice things.

I hope you've enjoyed this story, if you have any of your own thoughts of things that could be added to Pyash or ways in which it can abuse Python further please do comment below and/or give them a go yourself!

GIVEN my article WHEN a reader reads it THEN the reader writes better tests

Jamie Read — Fri, 22 Mar 2019 23:21:07 +0000

GIVEN a developer who hates writing tests

I'm not one for testing, that's evidenced by my most used side project having around 5% test coverage. I just can't bring myself to slog through and write every test case of the code I'm writing particularly when I don't have much time to work on side projects anymore.

WHEN placed in a software house that likes testing

However, in my current employment I'm expected to have 80% code coverage as part of the minimum to calling something 'stable' - hence I actually spend most of my working time writing unit, integration, and system tests.

AND he must write lots of tests in his job

Testing is something my company are very good at. We have hundreds of thousands of lines of code spread across hundreds of different repositories across tens of different projects and on any given day I could be working in any part of it.

The fact of the matter is that the tests save us countless headaches and make the insane amount of context switching significantly easier for us. Because of the high test coverage I know that I can hack and slash at code, and even perform massive refactors if I do so desire, and at the end of my rampage I can run the tests and know within minutes if there has been any changes to the code's behaviour.

I just don't need to worry about breaking code while I edit it.

THEN he learns some neat tricks in testing

This is the point where I tell you about the wonderous world of behaviour driven tests.

We have a convention to write every test with the GIVEN-WHEN-THEN structure. Here's an example from my side project:

[TestMethod]
public void TestReadStringDeserializesTheDataCorrectly()
{
    // GIVEN a buffer of serialized data
    mockMessageBuffer.Setup(m => m.Buffer).Returns(new byte[] { 0, 0, 0, 6, 65, 0, 66, 0, 67, 0 });
    mockMessageBuffer.Setup(m => m.Offset).Returns(0);
    mockMessageBuffer.Setup(m => m.Count).Returns(10);

    // WHEN I read a string from the reader
    string result = reader.ReadString();

    // THEN the value is as expected
    Assert.AreEqual("ABC", result);
}

I see this test and immediately know where to look for everything I need:

In the GIVEN section we setup any mocks we need, load in any test data from disk if we require it, and do any preparation we have to.
In the WHEN section we run the code being tested (and as best as we possibly can, only the code being tested) and obtain a result.
In the THEN section we assert that the behaviour of the code was as we expect.

Having this simple structure makes understanding what a test does trivial. I can look through hundreds of tests in development and code review and instantly grasp what a test is doing in a matter of seconds because every test in our company has this same, behaviour driven format.

We also find this style creeps out into other parts of our tests as well:

[TestInitialize]
public void Initialize()
{
    // GIVEN the object cache is disabled
    ObjectCache.Initialize(ObjectCacheSettings.DontUseCache);

    // AND a DarkRiftReader under test
    reader = DarkRiftReader.Create(mockMessageBuffer.Object);
}

You can start to see how we build more complex tests as well here. It's encouraged that if you do multiple things in any section that you document them with an AND section:

[TestMethod]
public void TestExtraLargeMemoryBlocksArePooledCorrectly()
{
    // GIVEN a memory pool with no previously pooled memory

    // WHEN I return an extra large memory block
    byte[] oldBlock = new byte[5000];
    memoryPool.ReturnInstance(oldBlock);

    // AND I request a new extra large memory block
    byte[] newBlock = memoryPool.GetInstance(4000);

    // THEN my memory block is the same
    Assert.AreSame(oldBlock, newBlock);
}

(I take a shortcut here, I should probably assign oldBlock in a GIVEN, but then this is really just as understandable)

AND can apply them anywhere

It also helps when writing tests!

Thinking about behaviour is a lot easier in English - that's why things like pseudo code and rubber duck debugging exist - so why not write tests in your native language and then use that as a template?

[TestMethod]
public void TestGetInstanceUsesPoolWhenHasInstances()
{
    // GIVEN a pool with a single element

    // WHEN I get an instance from the pool

    // THEN a new instance is not generated

    // AND the returned instance is the pooled object
}

[TestMethod]
public void TestGetInstanceUsesPoolWhenHasInstances()
{
    // GIVEN a pool with a single element
    object pooledObject = new object();
    objectPool.ReturnInstance(pooledObject);

    // WHEN I get an instance from the pool
    object result = objectPool.GetInstance();

    // THEN a new instance is not generated
    Assert.IsNull(newInstanceCaptor);

    // AND the returned instance is the pooled object
    Assert.IsNotNull(result);
    Assert.AreSame(pooledObject, result);
}

The exact order of the GIVEN-WHEN-THEN also doesn't matter, we often find that in the most complex of tests we need to assert behaviour in multiple places (THEN), or perform some step on the class under test (WHEN) before we can continue the setup (GIVEN):

@free
Scenario: I can disconnect from a server
    Given I have a running server
    And 1 client connected
    Then all clients should be connected
    And the server should have 1 client
    When I disconnect client 0
    Then 0 clients should be connected
    And 1 client should be disconnected
    And the server should have 0 clients
    And there should be no recycling warnings

Conclusion

You have no idea what my side project does, I never mentioned it, but you were hopefully still able to understand what these tests did, despite them covering some of the most complex and wacky optimisations I've had to add to it!

Adopting this behaviour, for me, makes testing significantly less cumbersome. For the developers I work with, it makes understanding my tests faster and easier. And in the merge review we can review the test behaviour using the comments and then check the code against the comments to make reviews of large quantities of tests a little more manageable.

I hope this article has made you think, and maybe you'll write a quick GIVEN-WHEN-THEN before you write your next test 🙂

I'm a C# dev that needs to build a web app. Sell me a tech stack.

Jamie Read — Wed, 22 Aug 2018 23:24:00 +0000

I'm interested in building myself a little web app. That's probably a simple task for most people here, but unfortunately when I left the world of web development jQuery was still considered 'pretty neat'.

So sell me your dream tech stack and tell me why I should build my web app with it.

For some background into the project:

A Platform-as-a-service for another of my side projects
Will need to interface into Digital Ocean's API to start and stop servers on the fly
Will open up to a moderately simple dashboard page showing current resources and deployments
Will need to have various step-by-step wizards for creating deployments
Will need to be able to authenticate users, possibly with 2FA in the future
Will need to interface with some form of payment provider (bonus points for recommendations!)

And background on myself:

Primarily a C# developer
Secondarily Python and Java
Some experience with devops
Some HTML, JS & CSS experience
Lots of relational database experience
Some NoSQL experience (specifically Firebase)

I look forward to reading any suggestions and recommendations people have! Feel free to ask for any more details etc. if you want!

Hazel Networking

Jamie Read — Mon, 21 Aug 2017 19:06:35 +0000

It's been a while since I've done much on it but I feel if there's anything I should showcase here it should be this!

Hazel

Hazel is networking library for C# that aims to simplify networking and take away some of the unneeded complexities and poor interfaces of sockets.

While I've only really advertised it on the Unity game engine forums it's fully compatible with .NET 3.5 and above (though not .NET core yet) so anyone who needs networking in .NET should be able to use it straight off.

It supports communication over TCP, UDP and its own RUDP and provides 3 major guarantees/abstractions for all of them:

Connection oriented
Message based
Thread safe

By providing a standard interface it becomes really trivial to switch protocol and I've aimed to make it as simple and lightweight to use as possible.

Code and Packages

You can get the source on GitHub or as a NuGet package here (MIT license, yay!)

Future (and a little past)

Hazel was a great little project for me and I had a great few weeks trying to invent and implement the RUDP algorithms and make them as efficient as possible. I really recommend it as a challenge to anyone who wants to learn more about networking or get a little experience in algorithmic thinking.

I originally intended Hazel to be the core of one of my more commercial side projects but sadly I've since changed that to a TCP/UDP socket pair in order to simplify the code base a bit. I still plan on maintaining Hazel as it's pretty much complete, so if anyone wants to use it or submit pull requests/issues etc. then I'll make sure to put some extra time into it with you!

One of the other visions I had for Hazel was to implement it in as many languages as possible in order to create a truly cross-platform networking library, hence the lightweightness of it, but that's perhaps something for the far future.

Any discussion, improvements or thoughts are more than welcome, I'd love for more people to give it a try!