Forem: Ross

You don't understand Agile!

Ross — Sat, 16 Jan 2021 12:23:20 +0000

You don't understand agile!

I don't think you really understand agile. Why do I think that? Because 90% of people in the industry seem to not know that they don't know what agile is and as a result, think they know what agile is. This results in pervasive misinformation. Just as a pure numbers exercise, it's a good bet -on my part- that you don't, simple.

"...as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don't know we don't know. And if one looks throughout the history of our country and other free countries, it is the latter category that tends to be the difficult ones." - Donald Rumsfeld, United States Secretary of Defense

How do you (think you) understand agile?

I propose that most agile wells are poisoned. People learn, self-led on the job and are riddled with bias and obstacles which taints their understanding & implementation. These people make and take part in faulty implementations, find the faults they injected, rage then write articles about them.

Ad-Hoc Agile

The major mistake we make when adopting agile is not consulting with experts. Taking in consultants or sending people on courses. Agile is different, it requires a change in attitude more than (certainly before) process and as a result:

we assume agile is largely a process change and not a culture/mentality and extract only the process which makes no sense in most cases without the culture
we dismiss parts as common sense that then don't get attention in implementation.
we don't strive for management buy-in an end up making massive compromises.
We try to phase agile in to avoid disruption and miss the core concept. That agile is about embracing disruption.

The second thing we do is assume that people talking about agile, know about agile...

Learning from "experts"/articles online...

Everywhere I see articles like this one which base their premise on massively flawed articles such as:

"Agile is the new waterfall…The reality of Agile is that you still have immutable decisions made by business people with no real understanding of technology. Those decisions are then forced on to developers. The end result is the same as Waterfall, only the names have changed.” — ayasin, Architect & Developer

Anyone who knows agile knows this is anything but. "immutable decisions" completely contradicts agile principle 2...so, not agile!

Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

Then in the same paragraph

Those decisions are then forced on to developers

doesn't seem to match with agile principle 4

Business people and developers must work together daily throughout the project.

And we are given the conclusion that "Agile is the new waterfall". Someone who has managed to be a developer so long they've become an architect seems like someone we can trust, right?

All Agile is

It's very specifically a change in thinking & culture. We strive to embrace uncertainty and cut out wastage so that we can deliver tiny amounts of high quality value quickly and repeatedly based on user feedback. We then turn that attitude inward and cut wastage, make tiny amounts of high quality process changes quickly and repeatedly based on performance in the previous iteration.

Anything that doesn't meet this, isn't agile. All process added and actions performed which decrease quality, time to increment and/or isn't user value focused will corrode all agile implementations.

Every process that slows development in order to forecast.
Every time there's a hand-off
Every user feedback you shun
Every failure to realise and focus the specific value - added
Every time you fail to trust each other
Every single time you say "that's not my job"

Now the question is, am I right? Get a certified consultant! Be ready to commit! Don't assume you know all you need to.

Should comments in code be considered failures in coding?

Ross — Mon, 26 Aug 2019 10:35:39 +0000

Uncle bob says:

A comment is a failure to express yourself in code. If you fail, then write a comment; but try not to fail.
— Uncle Bob Martin (@unclebobmartin) June 1, 2017

Should code be written in a way that's expressive enough on it's own?
If you disagree, what are your thoughts on keeping comments up to date with changes?
Is there a grey area? Examples?

Planning and Inaugural Meeting of the Edinburgh Kotlin User Group

Ross — Tue, 30 Jul 2019 15:28:08 +0000

NOTE: Adapted from my blog post at Symphonic Software

Why did we start a Kotlin User Group?

My company have recently started moving from Java code to Kotlin. We have never been shy in technology adaption. We started with a full .NET stack, moved through an .NET/Java implementation then to Angular/Java. Along the way through experimentation with Elm, Ember, JavaScript and REACT we picked up some Groovy and some Python. We even had Neil Gall -one of our Senior Developers- present our method of testing with Python at Europython 2018. It's important to us to move with the times and use the best tools to create the best product.

Although the team we have assembled at Symphonic has adapted to the changing technologies with skill and enthusiasm, we've found that conventions do not take hold as quickly. For at least a period with new languages we find ourselves creating debt. We've written non-Pythonic Python and bent Groovy to do things we didn't know how to express in it at the time.
This is because it takes time and exposure to veterans to get comfortable with the intricacies of a language. We can increase this exposure and perhaps decrease the time by talking to and interacting with people who use the language, so we decided to run a user group.

How did we do it?

Put together a budget on what we would like to spend to make it successful.
Put together some posters that we could paste up in tech sites around the city
Advertised on Reddit, Edinburgh Tech Calendar, Twitter and EventBrite
Purchased refreshments (beer, juice, pizza)
Decided on venue (our office) and put together a speaker ("Kotlin for Java Developers" by our Head of Engineering, Ewan Dawson)
Registered with Kotlin Community and notified them of the upcoming talk, they provided us with free T-shirts and stickers
Registered with Developer Community Support who provided us with a free 1 year JetBrains license to raffle off.
Wrote a small Kotlin program, containing an interesting quirk of the language which selected one of the present people to win the prize.

import kotlin.random.Random.Default.nextInt

/**
 * Number of people playing
 *
 * Q: Anyone know why val requires to be declared
 *    const yet 'lateinit val T = 9' indicates
 *    that val is immutable?
 */
const val PLAYERS = 10

/**
 * Original: Pick a winning number from 1 to players
 */
fun pickWinningNumber(players: Int = 1) : Int {
    return nextInt(1, players)
}

How did it go?

The 11 attendees enjoyed themselves if the SurveyMonkey survey I sent out afterwards is anything to go by. People signed in and I used that list as both a health and safety list for fire regulations and as the start of a mailing list to send out my survey.

The small program intended to pick a winner for the JetBrains license turned into a bit of a talking point. Firstly because of the interesting IDE complaint it generated, secondly on a discussion of how Kotlin can make it even more concise.

We had people put themselves and others forward for future talks and suggest ways of increasing membership, one of which was to create a Meetup page.

We are now planning our second meetup, with a backlog of three speakers ready for more. Currently bi-monthly until the membership and speaker list increases a little but going well.

P.S. We're looking for speakers on anything Kotlin related if anyone is interested.

Functional Enums in Java

Ross — Sun, 23 Jun 2019 20:14:29 +0000

A rehash of my GitHub.io post._

Introduction

The largest personal project I am currently undertaking, is that of an emulator (EmuRox). The main goal was to come up with a larger product that I am completely in control of and can experiment with tools and methods to become a better programmer. Sub goals include learning something about the machines I grew up with and maybe, perhaps getting to the point where I’ve written my own retro NES emulator. Long term, pie in the sky goal was to create a well designed, highly tested, accessible, open-source, pluggable, multi-emulator.

First and foremost though, I wanted to strongly focus on what I feel is important in a good Java program:

Highly Tested
Readable
Efficient
Maintainable / Extensible

Satisfying all of the above, IMHO I would say it was generally well designed.

So, I started with the most documented part of the NES. The MOS6502 processor. What I want to talk about today, is how I emulated op-codes and how my approach gave me a nice upgrade path when I realised my implementation wasn’t as great as it could be. As a side note, I'll be discussing why TDD is -if you understand and use it correctly- a very useful method in Java development.

Approach Number 1: The Result of Test Driven Development (TDD)

One of the approaches I’ve been playing with is TDD. Where I write a failing test, then write the code to make it work and iteratively build a fully tested, modular product. This led me to writing the minimal amount of code to make tests work. The result was great process and…this:

    public void step() {
        log.debug("STEP >>>");

        final OpCode opCode = OpCode.from(nextProgramByte().getRawValue());

        //Execute the opcode
        log.debug("Instruction: {}...", opCode.getOpCodeName());
        switch (opCode){
            case ASL_A:
                withRegister(Register.ACCUMULATOR, this::performASL);
            break;

            case ASL_Z:
                withByteAt(RoxWord.from(nextProgramByte()), this::performASL);
            break;

            case ASL_Z_IX:
                withByteXIndexedAt(RoxWord.from(nextProgramByte()), this::performASL);
            break;

            case ASL_ABS_IX:
                withByteXIndexedAt(nextProgramWord(), this::performASL);
            break;

            case ASL_ABS:
                withByteAt(nextProgramWord(), this::performASL);
            break;

            case LSR_A:
                withRegister(Register.ACCUMULATOR, this::performLSR);
            break;

            case LSR_Z:
                withByteAt(RoxWord.from(nextProgramByte()), this::performLSR);
            break;

            case LSR_Z_IX:
                withByteXIndexedAt(RoxWord.from(nextProgramByte()), this::performLSR);
            break;

            case LSR_ABS:
                withByteAt(nextProgramWord(), this::performLSR);
            break;

            case LSR_ABS_IX:
                withByteXIndexedAt(nextProgramWord(), this::performLSR);
            break;

            ...

This is of course truncated as there are many, many more opcodes on the 6502. Does this fit my criteria for a well designed Java program?

Highly Tested: Floats around 98-99% line coverage. Static analysis. Mutation tested to 98%

Readable: Partially. Each opcode has a path. The instructions to execute it are in English. Little annoying that you need to scroll so much and complicated instructions end up being messy to nest method calls so that they read fluidly.

Efficient: For large numbers of cases, switch statements are pretty efficient.

Maintainable: No. There are loads of methods that are there to reduce duplication which end up being duplication themselves. Parts of the logic are not separable like addressing and operations.

Extensible: Lets see…

So I wanted to see if I could do better.

The legacy & and the beauty of TDD

It led to some pretty hard to maintain code, but correct; Thanks to our tests, quickly and accurately verifiably correct. Which means we can mess with other implementations and the huge test space will theoretically catch any mistakes in a very short development cycle.

Approach Number 2: Functional Java Code

I want to minimise duplication, writing code once for each operation, and once for each addressing mode. As we already reference opcodes by Enum, I wondered if it was possible to parse an opcode value (0x0A) into it’s Enum (ASL_A) then simply call an execute() method on that Enum instance which fires off an attached lambda providing an environment (memory, registers and alu). As each Op-Code Enum knows it’s own Addressing Mode, it could call an address() on that Addressing Mode (which also calls an attached lambda) providing us with all we need to make composable instructions.

Overview

So my enum becomes a definition containing a byte value, it’s addressing mode and the operation performed. The OpCode is then the intersection of Addressing Mode and Operation

ASL_A(0x0A, AddressingMode.ACCUMULATOR, Operation.ASL);

The operation argument is an operation on the environment (a: accumulator, r: registers, m: memory), given an addressed value (v):

    public enum Operation implements AddressedValueInstruction {
        /** Shift all bits in byte left by one place, setting flags based on the result */
        ASL((a,r,m,v) -> {
            final RoxByte newValue = a.asl(v);
            r.setFlagsBasedOn(newValue);
            return newValue;
        }),

        ...

        @Override
        public RoxByte perform(Mos6502Alu alu, Registers registers, Memory memory, RoxByte value) {
            return instruction.perform(alu, registers, memory, value);
        }

The addressing mode uses an environment (r: registers, m: memory, a: accumulator) to address a value, runs a given operation (i) on it and places the return value back at the addressed location:

public enum AddressingMode implements Addressable {
    /** Expects no argument, operation will be performed using the Accumulator Register*/
    ACCUMULATOR("Accumulator", 1, (r, m, a, i) -> {
        final RoxByte value = r.getRegister(Registers.Register.ACCUMULATOR);
        r.setRegister(Registers.Register.ACCUMULATOR, i.perform(a, r, m, value));
    }),

    ...

    @Override
    public void address(Registers r, Memory m, Mos6502Alu alu, AddressedValueInstruction instruction) {
        address.address(r, m, alu, instruction);
    }

Then the op-code brings it all together by combining the addressing mode and operation:

public enum OpCode implements Instruction {

    ...

    @Override
    public void perform(Mos6502Alu alu, Registers registers, Memory memory) {
        addressingMode.address(registers, memory, alu, operation::perform);
    }

This means the huge switch statement becomes one line of code

opCode.perform(alu, registers, memory);

Problems

There are some instructions which don’t fit this pattern nicely:

AddressingMode.IMPLIED: The addressing mode is implied by the last two bits of the instruction byte
The IMPLIED instructions do slightly different addressing based on the Operation. I’ll need to deal with these as individual cases, where Operations do their own addressing.

Operation.JMP: It -unlike all other instructions- passes a word (not a byte) from the addressing mode that could be AddressingMode.INDIRECT
These do two things that make them hard to deal with. Firstly, their addressing is ABSOLUTE; the two byte (word) address that the Operation is supposed to use to load into the Program Counter. We could deal with this in the same way as the IMPLIED (in that the Operation then does some of it’s own addressing) if not for the case where JMP uses INDIRECT-ABSOLUTE addressing, which will take a two byte argument then from the address specified by that word, load a two byte address into the Program Counter. In this case, it cannot be done in the Operation because it has no idea what it’s Addressing Mode is.

Solution

So in the end we have a bit of a hybrid:

        final Mos6502OpCode opCode = Mos6502OpCode.from(nextProgramByte().getRawValue());

        //Execute the opcode
        log.debug("Instruction: {}...", opCode.getOpCodeName());
        switch (opCode){
            case JMP_ABS:
                registers.setPC(nextProgramWord());
            break;

            case JMP_IND:
                registers.setPC(getWordOfMemoryAt(nextProgramWord()));
            break;

            default:
                opCode.perform(alu, registers, memory);
                break;
        }

The edge cases make sure this is not the prettiest but it’s much nicer than it was and when I think of the next iteration to simplify this, I know that my tests are specific yet generic enough to support refactoring and redesign.

Thoughts on Automated Plant Growth Measurement

Ross — Fri, 14 Jun 2019 14:47:19 +0000

Introduction

I have a greenhouse. I grow chillies in said greenhouse. I would like to use technology to improve my yield.
Using a Raspberry Pi, some cheap electronics and some Python I've started to write the telemetry software that will allow me to make better choices. I've been thinking about how far I can take this.

One such area in the future that will provide the input to get closer to the entire system being closed is plant growth monitoring.

Plant Growth

So we need something to measure and as I see it, plants have two phases which can be measured

Phase 1: From Seed

Measuring germination would be a case of detecting something appearing from nothing, the measurement being time from planting to germination. Time should be pretty short (1-4 weeks) and any more is a failure.
Another useful metric might be the number of seeds which germinate from a given number planted.

Phase 2: From Germination

This is where it get interesting as this feedback can be fed into the cycle and adjustments made to improve. Useful metrics might be:

Height of plant
Physiological age: number of leaves
Size of leaves
Number of flowers/fruits
Changes in color
Weight of plant (with some compensation for watering)

Measuring Phases

So we can technically measure the first phase but unless we are doing several rounds of germination or using the data to feed back into the next year so I'll be concentrating on the second phase as it fits the brief of actions based on feedback to improve yield.

Techniques

So we know what we want to measure, how do we measure it?

Height

The simplest method of gauging plant height has been used for a very long time and is called an auxanometer. This is essentially a string is tied to the top of the plant, it goes over a pulley to a weight and as the plant grows the weight lowers. This is linear motion, which is easy to measure.

The other option is image processing, which we'll discuss as part of...

Leaf count, Leaf Size, Flower/Fruit Count, Color Changes

Without manual process, these are much harder to read. Ideas could be a camera system which does image processing^[1][2], LIDAR which creates 3D models, infrared or some combination such as XBox Kinect systems^[3][4][5].

Weight

An interesting option is plant weight. Plants will obviously increase in weight as they increase in size. There are a couple gotchas in that plants need watered and various states of being watered will have different weights and that offset will need to be accounted for. The other is that weight will take into account things on the plant; moisture, pests, dirt and there would be no distinction between leaf, stem, flower or fruit. The latter being a lesser concern.

Other

Things like structure of the plant and it's leaves and growing direction can be handy for detecting disease, lack of water, excess heat. Then being able to detect and possibly identify pests could lead to suggested or automated treatments.

A Way Forward

There are quick wins, the auxanometer and weighing platform but ultimately if we don't want multiple machines around each plant the efficient solution is image/lidar/infrared processing. It should do it all in a very compact way. This of course is not a quick win solution and will require quite advanced software.

So the plan, if I go forward with it will be to use a camera (perhaps an XBox Kinect) and write software to process height, leaf count & size, fruit & flower production and color.

Steps (quick wins first)

Get the camera setup attached to my power grid and adjust as necessary
Add software to tell plants apart even when they are moved, perhaps an identifying mark on the plantpot such as a number or even barcode, allowing us to encode things like plant type right there on the plant.
Expand software to detect changes in plant color
Expand software to detect height
Expand software and hardware to detect leaf/flower size/count

The last one may require hardware changes as a static camera and a static plant could mean that leafs/flowers could be hidden from the camera by the plant itself. This hardware expansion may also by useful for rotating plants away from the sun to help them build good stems.

What to do with all this data?

Very quickly from a little data we can start making the lives of our plants better, autonomously and suggesting manual remedies for common problems:

Under-watered or overheated plants tend to lose structural integrity in their leaves. We can water them, or open vents to cool the greenhouse.
Pests or disease will discolour the plant, some pests can be removed with a spray of water, others with natural pesticides (like a rhubarb/chilli tea) and others could suggest a change in gardening practices like adding more plants that encourage ladybirds to eat pest like aphids.
Lack of growth, of flowers could suggest that we are over watering or under feeding.
Flowers which don't reach maturity could suggest improper feeding or undesirable humidity levels.
Growth direction could indicate that plants need to be turned more regularly so that they have to reach towards the sun, strengthening their stems.
Struggling plants might suggest greenhouse changes such as a better base, insulation or the introduction of a thermal mass to limit temperature spikes.

And the more time we can stop thinking about common problems, the more we can start thinking about how to increase yield volume and quality.

References

^1: Lin, Ta-Te & Lai, Tsung-Cheng & Liu, Ting-Yu & Yeh, Yu-Hui & Liu, Chang-Chih & Chung, Wei-Chang. (2019). An Automatic Vision-Based Plant Growth Measurement System for Leafy Vegetables.

^2: Deep learning for image-based prediction of plant growth in city farms Iljazi, J. (Author). 25 Sep 2017

^3: Hu, Yang & Wang, Le & Xiang, Lirong & Wu, Qian & Jiang, Huanyu. (2018). Automatic Non-Destructive Growth Measurement of Leafy Vegetables Based on Kinect. Sensors. 18. 806. 10.3390/s18030806.

^4: Chunlei Xia, Longtan Wang, Bu-Keun Chung, and Jang-Myung Lee, et al. (2015). In Situ 3D Segmentation of Individual Plant Leaves Using
a RGB-D Camera for Agricultural Automation. Open Access Sensors ISSN 1424-8220

^5: Jorge Martinez-Guanter, Ángela Ribeiro,et al. (2019). Low-Cost Three-Dimensional Modeling of Crop Plants. Sensors 2019, 19, 2883; doi:10.3390/s19132883

Hardware Choices in Monitoring a Greenhouse

Ross — Sun, 09 Jun 2019 00:06:10 +0000

What do we need?

A power system for an off-grid, greenhouse telemetry Raspberry Pi setup. Turns out it's actually pretty easy, even for a power layman like myself.

We need power

First, make the Pi efficient. I'm controlling my Pi via VNC remote desktop but I'll move it to headless soon. I wont be using the HDMI so disabling that circuitry makes sense, let's save the 25mA that it uses:

/user/bin/tvservice -o

Then I don't need flashing lights, so another 10mA (5mA each) saved:

echo 0 | sudo tee /sys/class/leds/led0/brightness
echo 0 | sudo tee /sys/class/leds/led1/brightness

I've seen Pi power estimates (HDMI and lights are turned off) as high as 2 Watts per hour. I want extra power though to allow later expansions and because I don't trust any data based on Scottish weather. So let's just see how much bang we can get for our buck running it 24 hours.

Can we power off grid?

Using European Commission Data we can see that in summer (April-September) using a 50W solar panel, we can hope to generate between 130-200Wh per day with a yearly low of 40Wh (Ed in the table below). My position is a little awkward (east facing with a shadow cut off point) so I'll be somewhere below that figure.

So I could almost run this system throughout winter. If solar generation was spread out over 24 hours, which we know it isn't. A bonus with my chosen panel is that it comes with a controller to save me messing with electronics and making a mess of the power circuits.

So we need power at night

We need 48Wh to power the Pi (2Wh over 24h) and we're getting 130+ in a short time frame then nothing for the rest. We need to spread that out. So a battery. Something that will:

Give us room to grow
Will support plenty of discharging/recharging

So, lets say 80AH, that looks about as expensive as I'm willing to pay. This equates to 960wH (80Ah x 12V) or 20 Raspberry Pi days straight.
It's also a deep cycle (meaning it'll run pretty low, safely) so has quite a large buffer area for power issues and loads of extra power for expansion.

And that's it

It works, it's been running for 3 days straight in my home, 1 in my greenhouse. I get way more active solar charging time than I expected being east facing so I have more power than I know what to do with.

Now to think of something else to do with it all.

I'm also looking at products or techniques to allow me to monitor and log incoming solar charge and consumed battery.

Technology Choices in Monitoring a Greenhouse

Ross — Tue, 04 Jun 2019 08:36:47 +0000

Introduction

I've started to put together a telemetry setup for my greenhouse. I've been thinking about it for a while but ultimately I've left this a bit late. It's summer, although the constant Scottish rain begs to differ.

I've only recently installed a proper greenhouse and inspired by m'colleague Neil Galls Pirragor Series, I've gotten started. I'm not doing some fancy Rust cross cross compiling, resulting in installing a new OS like Neil. I've opted to move my complications somewhere else, by growing chillies instead of tomatoes in various varieties. Needing many more individual plants and finer grained plant care. This means monitoring of temperature and humidity is the most important concern with some further thought needed in automating the upkeep of the individual plants of which there are over 20.

Straight off the bat, off the shelf Rasberry Pi is a good choice for rapid development. Python is already available so I got started.

Need Input!

I need humidity and temperature monitored, hopefully at different points in the greenhouse and compared to outside. Internally I'll be using AM2302 humidity/temperature sensor and the Adafruit Python Drivers for it which makes it as easy as

humidity, temp = Adafruit_DHT.read_retry(sensor, pin)

And rather than weather-proofing sensors and thinking about wiring it up, I'll just use a weather REST API in the form of OpenWeatherMap and I get things like cloud cover and barometric pressure for free.

Problems?

I've noticed that randomly the AM2302 outputs a humidity of 3303.6 and a temperature of 70~ and the values had to be sanitised but otherwise, all good.

Those who cannot remember the past are condemned to repeat it!

A grandiose way of saying we need to store the data for comparison. Originally I stuck it all in a CSV file and tried -using Pygal- to write my own charts, which was time consuming and ultimately inflexible.

I need better control over the data which means a database. All of my data is going to be tied to a specific time so a time-series database is what I'm looking for. I installed InfluxDB on my Rasberry Pi, no hassle and with the help of the influxdb Python library, started to write my data there:

    db_client = InfluxDBClient('localhost', 8086, 'root', 'root', 'TelemetryHistory')
    record_entry = [
        {
            "measurement": "am2302",
            "tags": {
                "update": "whole",
                "device": "am2302",
                "location": "gh1"
            },
            "fields": {
                "temp": float(temp_reading),
                "humidity": float(humidity_reading)
            }
        }
    ]
    write_success = db_client.write_points(record_entry)

InfluxDB, bullet point review:

quick to install
very quick to get started being schemaless
very quick to learn
perfect for analytic data, with good visualisation with Grafana
it has other benefits I wont use (efficient eviction, high throughput)

Then, using Grafana on my laptop, I could connect to the database on my Rasperry Pi and view whatever selected, filtered, calculated data I wanted with the ability to draw very flexible, zoomable, pretty graphs. Granted of my room at this point but pretty nonetheless.

Next up!

I need to come up with a battery/solar panel configuration that will properly power all of this.