Forem: James MacIvor

For The Fear Of Fouling Up

James MacIvor — Fri, 22 Jan 2021 18:35:45 +0000

For The Fear Of Fouling Up

Forewarned

This article starts out talking about .NET culture and patterns and swiftly transforms into a discussion on my personal experience contributing to OSS. But if you want to engage with me on the former topic please reach out to me on Twitter or LinkedIn. I'd love to hear from you.

The Problem With Obligatory Patterns

I was reading Twitter and saw an interesting series of conversations unfold with David Fowler. David Fowler is a domain famous developer and is an architect at Microsoft. He authored the SignalR project which is the most popular method of enabling web sockets in a .NET application. He is also well known for his involvement in the community where he frequently shares what he's learning about and offers resources to fellow developers are also curious and learning. He will occasionally post a strong opinion and is not averse to a little bit of controversy for the benefit of the community at large.

This particular twitter thread was about how you don't need the full enterprise implementation for every little project. He wrote that not every tiny project benefits from dependency injection and repository patterns and so on. This tweet squarely fell into the "controversy" category.

He encountered a lot of resistance from .NET developers, but I deeply agreed with what he wrote. Those patterns help on larger projects, but sometimes they're useless bloat and can be discarded for smaller projects. Fowler argued that we should be thoughtful about where and why we use those patterns. While some felt that they would always end up using the patterns (so why not just implement them right away) I recalled instances where I didn't want to deal with the overhead. I've certainly worked on projects with patterns that did not make sense or were implemented poorly. Some patterns are better off not used than implemented poorly.

You can get a taste of the conversation from the following tweet from February 2020.

David Fowler 🇧🇧

@davidfowl

Your .net code does not need 15 projects with a class per command and query. You don’t need an interface for everything either. It’s amazing to read extremely simple samples using .net vs node vs go. It seems our ecosystem is attracted to complexity... #dotnet

04:13 AM - 08 Feb 2020

Fowler's initial 2019 tweet was about his backend Todo app implementations where developers criticized the repo for lacking some of the common patterns found in many .NET projects.

The exchange generated an entertaining and informative back and forth. As I read Fowler's tweets and started to feel like I understood his point of view I felt an affinity for his approach. I even agreed to his stance on testing in that many unit tests do not add value and once again we should exercise good judgement. A blanket requirement for test coverage makes sense for some development teams but sometimes it tramples the question that should be asked: "Do these tests make sense?"

Reading his tweets made me think about all the times I didn't reach for .NET Core for a personal project or proof of concept. Why not? Because it seemed like a lot of overhead. I didn't want my internal code reviewer berating me for not using the patterns necessary for a normal project. I didn't want to show it to another .NET developer and immediately feel like I have to apologize for not having those things - even though in the context of the project it does not make sense! So I'll just write it in Node.js or something because the expectation for fully thought out patterns are not as prevalent. I can focus on the functionality I want to explore and not deal with the overhead of a repository pattern or whatever.

Hearing David Fowler sound out a few of those same thoughts made me realize that all of this expectation in .NET isn't necessary. I felt I had permission to also dissent against the use of over-architected patterns.

So, when I saw in late December that Fowler had created a new project called FeatherHttp I was stoked. It appeared to me that this project was a manifestation of his thoughts around .NET bloat. I jumped on it and quickly started buzzing around the threads. I wanted to help write the documentation. I wanted to be a part of this effort. Also, here was a .NET open source project in the initial stages. I felt I could contribute. I was also very excited to work with David Fowler.

Getting Involved

Have you ever worked with someone who you admired so much that it actively caused you anxiety? I have.

I mean, not at first. At first I was 100% excitement. "Here's a starter template for the docs! Why are we doing this? Let me write the docs with an angle in mind for people who are not .NET. Also, who are we targeting? Should expect .NET familiarity?" So many questions. Buzz buzz buzz.

I create a basic statically generated site to host the docs. I am brimming with enthusiasm as I start trying to convert the examples into docs and show my worth. Things move quickly. I initially think to use Netlify but somebody says "lets use GitHub pages instead" so I reconfigure everything for that. This is actually kind of hard. GitHub pages requires a branch dedicated to the static pages. The static site generator needs to create the files and then GitHub Actions has to pick up that output and commit it to the github-pages branch. It's a solved problem, but just the idea of a dedicated branch only for the static pages seemed pretty different and novel to me.

Anyway, there are plenty of tutorials and guides to explain how to do that. I was pretty happy once I had a build system figured out and had submitted the PR to David.

I remember feeling so proud. "This is going great."

Fear

After that I saw there was an opportunity to help out with the GitHub actions to package up the dotnet template into a NuGet package. I had done none of this before and I was excited for the opportunity to learn. I had a fairly bare bones description in the issue but it cited another project that did something similar and I could crib off of that.

However, there were challenges. The FeatherHttp project used Nerdbank Git Versioning, but the example project did not. Also, the example project used a PowerShell script to set a couple of things up, but along the way David let me know that he's prefer not to use a PowerShell script unless absolutely necessary.

I iterated a lot. I made faulty assumptions and was misled by GitHub issues on other projects. I worked on these GitHub actions every night for a couple of weeks. I submitted several versions of it as a PR and found I was frequently missing the mark.

In fact, it's kind of fun to look back at my commit history for that and just... marvel at the struggle.

But with each breakthrough in understanding something else was happening. I allowed my insecurity to eat away at me.

"Why can't I just do this?"

David seemed to have the exact idea of what he wanted, and I felt like I was discovering that in jibs and jabs along the way.

I remember how I felt proud when David finally approved the PR and my code merged. I followed it closely to make sure it built properly and the GitHub Actions behaved exactly as I had tested. It worked!

But then, David found an issue and had to fix a few things. I felt like I had failed and missed the mark.

After that, when my confidence should have been soaring I felt disappointed instead. What I wanted was to talk to David and hear him actually talk about the project. Communicating over GitHub issues and PR comments felt like a poor substitute.

I puttered around with the docs site for a little while longer, but my anxiety continued its patter.

Why So Serious?

David Fowler, among the many software architects and coders that I'm aware of, is a strong leader endowed with uncommon skill, like an Achilles of code or something. Working with him, I believed, would be a step along the way to becoming like him. Where, now I am a strong coder for creating business solutions, I could become like David who is a strong programmer and architect creating tools for other developers to use.

So, first error was allowing David to become larger than life in my own head. The project became overshadowed by my inability to communicate well with David. Also, I felt the eyes of his other followers on me. By becoming part of this journey I opened myself up to scrutiny and criticism. Even though none was forthcoming from other developers, my inner voice provided plenty! I was unprepared for such feelings. I imagined a group of fellow software developers who were unfeeling and unkind, like a host of StackOverflow curmudgeons prepared to descend upon me.

All of this while constructing some docs pages and a few GitHub actions. How can the voices of my insecurity be so loud?

If you have ever suffered from anxiety you know there is a threshold that, when crossed, the anxiety cannot be calmed. No one can say anything to assuage it. There is nothing that can be done except to either walk away if you can, or to soldier on and shove the unpleasant emotions to the side.

For me, time and emotional distance will eventually lessen the anxiety. In fact, I've come to recognize that when I heap too much expectation on an experience or outcome I will almost surely self-sabotage. I find that keeping calm and resisting excitement is the key to preventing my fears and insecurities from overwhelming me in the face of a big opportunity.

Ultimately, this time, it was important to walk away.

Remembering Strength

I could think of no way to rid myself of this weakness and so I decided to do the coder equivalent of "hitting the gym." I doubled down on my projects at work and trained hard to be a better technical lead and developer.

This is not the first time my confidence has betrayed me. I know the cause of it and I know how to recover. The fact is that I'm pretty good at what I do. I occupy a great spot in my local community of developers. I can provide a leg up to less experienced developers, and I can be a steady pillar on a project. These fears rarely surface when I'm in somewhere "doing the work."

And therein is the answer for me, to practice my craft within a team, either leading or following. I can shine so long as I'm connected to a community of people. I am an extrovert in a field favoring introverts. It's no wonder that OSS is difficult for me as it's lonely hours spent in code away from the life giving connections. Others in OSS seek it out for that same quality. However, fellowship within work is the top coinage that can be offered to me, and I am eternally greedy for it.

Remembering this, I could reflect accurately on my experiences. What I wanted was to speak to David to hear him talk directly about his thoughts and goals and what he believed was right. That is the true work. The code is a reflection of those ideas. Lacking a supply of that, I'm left to create my own meaning based on what I can see written already. And herein I come to my second error.

OSS is a conversation occurring over a long time. Patience is key.

I know this because months later I contributed to a different project, rewriting a section of their docs. I'm tremendously proud of that contribution but it was made with completely different motivations. There was a task I wanted to do that was not accurately covered in the docs. This annoyed me and I decided no one else needed to suffer after me. I wrote up the change and put it in a PR. I felt completely confident in my motivation and my understanding of the project goals. I imagined a shared bond with a bunch of other similarly irritated developers. It all sailed through without a single problem and I felt rewarded. It was the equivalent of saying a single sentence in a very long conversation involving many other developers. This is as it should be. The developers with the vision for the project drive not just the technology itself, but the entire conversation. This conversation happens at a pace with which they are comfortable.

Now I can see that the fire of enthusiasm I brought into the FeatherHttp project would have better served as a slow burn, ready to be stoked to flames when the time was right.

Further I can see that big personalities don't matter. The code matters. The problem we seek to solve matters. Yes, even your local yokel tech lead can be awed by the fame and technical acumen of others. But the developer versions of the heroes from the Iliad do not walk among us. They don't have Twitter accounts. They are not real. All are normal people, some knowing more than others but still people just the same.

Onward

Why did I write this? What did I hope to gain by revealing a weakness, no matter if I did learn from it? One thing I hope happens is that others will find their own motivation to contribute to open source. Where I did it because I believed in the project and sought a connection with a person I admired, others will do it because they want to create a better experience for the next developer. Or maybe because they want to learn something new. There are no wrong reasons to contribute.

Another reason I wrote this is to better understand myself. Sometimes the act of putting a lesson to words defines it and brings it into clarity. This is, after all, a story of me versus myself. These internal struggles happen to all of us and it's good to reveal that. I want you to see that I struggle sometimes. I don't want to forget the lessons learned either.

Also this internal conflict represents a huge reason why people don't want to contribute to OSS. Why wrestle with OSS when I can write code within my team and receive the validation of having done a good job, or the gentle criticism from people who will look me in the eye when they say it.

I jumped at shadows and imaginary criticisms and missed the point. You needn't suffer this. Learn from my errors and engage with a vibrant community, but learn to be patient as well - with yourself most of all.

The Road To Developer Jobs

James MacIvor — Mon, 28 Dec 2020 21:59:25 +0000

The Road To Developer Jobs

It's hard out there.

I think nothing has shown me that more than following and reading the heart breaking updates from recent boot camp grads. They don't know it, but I've seen them. I've read their portfolios. I've read their posts on LinkedIn, which are alternating in bravery and despair. I feel for them.

I'm a lucky one. I taught myself to code while working an office job. I never used a boot camp, but I thought about it. I was personally invited to join one by the founder of The Odin Project. He had just started his Viking School and was looking at promising students in his free online course. I was tempted! He seemed like he could help me achieve my dream of being a developer.

Ultimately, the finances didn't make sense to me so I passed. Wow, how different it would be had I not. At that time boot camps were popping up everywhere but the demand for new developers was nowhere near sated. I would probably be happily writing on Ruby or JavaScript projects somewhere. Instead, I went down the path of .NET and got an enterprise job. It was pretty good for a while.

The point is that it was harder to acquire the base skills, but easier to get a job.

Now, it's different. Coding has become a symbol of upward mobility and people have flocked to the craft. This has naturally made the junior level jobs a little more scarce.

So what to do? I've been thinking about that.

The Easiest Way

Lets start with the easiest possible way to get a job. If you come from an office environment then the best thing is to network back into that company. Get in touch with people you know and try to find a way to reach the ears of a hiring manager in an IT department. Even if you're not in an office environment, say you come from a warehouse or a factory - there are still office staff there and you should definitely network in that direction.

You are now a triple threat to secure any open coding position. Not only do you have that base knowledge of coding (hopefully in the stack your company uses), but you also have knowledge about how the business operates and you're already accepted into the culture of the company. From a hiring manager's perspective this could be a slam dunk. It's way easier to bring in someone who kind of already knows their way around than someone who's completely fresh. You immediately put yourself on par with an experienced junior developer or even a mid-level.

If you were wise in your previous position and made friends and had a decent network then that makes you even more valuable. With a little time you can be someone who knows where to get that requirement clarified, or steers the a coworker ( or even the team! ) around a critical detail that only someone experienced in business operations would know. This is particularly true if you work in a field with complicated processes, like insurance or finance. You are made of gold if you've already got insight into existing systems and processes.

Of course that's not for everyone. Also, many people turn to code to flee their old job, not to return and conquer it. In fact, even though I label this as "easy" few people lay the ground work necessary to make this a reality. It does require a fair amount of networking. It does require you not hate your old employer. It is easier for people with office backgrounds. These differences add up to a big advantage. However, if this is you then you should definitely try this route.

Use every advantage at your disposal.

The Traditional Way

It's harder, but the boot camps are geared around this. This is the path of job fairs and competitive interviewing. If you are a super personable confident outgoing person who excelled in boot camp then you'll tie this up. No problem!

However, if you have self-doubt and uncertainty and occasionally self-sabotage or if you didn't do so great in boot camp (just well enough). Then you'll struggle here. But you can still pull it through by following the advice of your instructors and going all in studying and working on boosting your confidence.

The profiles of companies that target boot camp graduates vary. Some are Startups seeking inexpensive but promising junior developers. Some are enterprise companies trying to modernize their legacy applications and compete with more nimble companies. Some are companies afflicted with huge amounts of attrition and are trying to find more developers to maintain and build their applications while not increasing pay or improving working conditions. (You'll know who these companies are by the uneasy rumors that precede them. Also checkout Glassdoor!) And lastly, some seek talent from a wide array of places and will gladly take a boot camp graduate or a CS graduate based on who they think will do the job well and, of equal importance, who they personally want to work alongside.

Second career or military veteran boot camp students are appealing. Many of them have already worked in professional environments. They know how to dress, how to speak, and how to deal with adverse conditions. This something your average college grad will spend the next 2-3 years learning. Not only that, but boot camp students are humble. They've been on the sweaty side of employment negotiations almost every time. Part of the fun of hiring them is watching them grow as engineers and begin to understand their value to companies.

People can find themselves hovering along the traditional route for a long time. Especially candidates who seek 100% remote work or are in a locale already saturated with top talent. It's harder for people who don't easily fit in with others or who feel different. Folks who have struggled with the third rail of software development ("soft" skills) will find this much more difficult. They have to work three times as hard because they need to acquire and master the base coding skill and also acquire some of the important people skills.

So what do you do if the easy way isn't so easy and the traditional way has you circling the runway for an indefinite amount of time?

Another Way

If a month or two goes by and you can't find a position then it's reasonable to start thinking of alternatives.

I like people who are self-starters who will find a different game with rules that favor them. I value this skill and I'm not the only one.

Consider that keeping a portfolio of pretty but otherwise not useful projects can only get you so far. Why not try for real experience on your own terms?

I suggest looking around on Upwork or similar sites for a contract you think you can do for a price that suits you. Yes, I am suggesting going for real work. I know you're not experienced. But what if you tried anyway? What could you learn?

Also, what the hell, if nothing else is working what will you lose if you fail?

Upwork might be too intimidating at first, but what about churches? What about professional groups or charities? If you're already doing portfolio work then why not do work that's meaningful and for a good cause. Yes it's free or low paid work, but what's your portfolio paying? Also, you might ask if you could include the work you do as part of your portfolio. If you do good work you might ask them for references too.

But listen, and this is important, do this work because you want to do it, because you believe in their cause or you like them or whatever. Never allow anyone to brow beat you into doing free work. If you get an offer for free work and your emotional reaction is "meh". Pass. It's not worth it.

You know what, if put in the time to do something cool for a cause or people you like - you might as well form your own LLC. (Congrats you're hired!)

Also, those people you just did good work for have a friend at BigCo who needs a new developer on their team. Can they introduce you?

This is how it works. It makes it super easy to get hired by a bigger company if you already have successful projects and you have experience talking to business people. People who do this successfully are leaders in our industry. They are sought after. It's not easy and it's not for everyone. From what I've seen, people who try this and fail are still miles ahead of people who go into interviews with only a portfolio and hope.

Patience, Practice, and People

Ultimately though, patience, skill, and your network will be your friends in this initial stage as well as throughout your career.

Yes, get into those interviews and put your best face forward.
Yes, practice at the craft and build up to showcase your skill.
Yes, build up your network on LinkedIn and elsewhere.

That third point is the most important. And what do I really mean? I mean you should be trying to schedule chats with people who interest you. The goal from these chats is not to get an interview, but to learn. Ask about their business. Ask about what they like in their job. Ask about the culture. Listen to their responses and build follow up questions. Be respectful of their time and try not to keep them longer than 15-30 minutes.

You're not going to be good at this right away. You'll make mistakes and feel foolish from time to time. You'll make gaffes. Embrace that awkwardness! I try to do this three times a week even being fully employed. If you are not otherwise employed then you might try to do this three times a day. Don't exhaust yourself. If it feels like you're going through the motions then take the day off. Being genuine and authentic is important. If you're tired it makes it a lot harder!

I think it's reasonable to ask, what's the point of all that? The point is to build relationships with people. You'll want to start conversations and contribute thoughts toward any problems they might have. People who get good at this are super valuable and can get jobs anywhere.

If starting conversations up with strangers is not your thing then that's completely okay. This worked for me, but I know that we're all different. In a more general sense my advice is to find a way to sharpen your people skills. If you're not naturally a people person it will be hard, but it's incredibly important.

The Right Job Matters

Indeed, when you're first getting started it can seem like any job sounds great. I believe this. Getting that first year or so of experience is critical for setting up the rest of your career. But if you have options, I recommend consulting.

Working for a small homegrown consulting firm like HMB transformed me. I've been exposed to amazing and talented people. I've worked on interesting and engaging projects. I've been exposed to a wide variety of company cultures and discovered what works best for me. I've been challenged to become a better communicator, a better engineer, and a better person. I've even been challenged to be a leader. All of this and without the chilling bureaucracy of a larger company.

It means a lot when what you do matters to the success of the company in a tangible way.

Moreover, working with smaller companies and startups you are encouraged to steer your own success. Your success is their success and vice versa. It helps when your mission is more than providing value to faceless shareholders. Instead the mission of the business is something you shape with your own hands. It's something you can believe in. That matters to me.

But there are downsides as well. The benefits might not be as good as a big company. Large enterprise companies are stable. Changes are incremental at those places. Shakeups of an entire company do not occur over the course of one or two weeks, for example. There is value in that. Even though big enterprise employment is not my favorite (although I do enjoy working on a contract basis for such companies), I do recognize that such companies provide safety and stability for their employees. There is nothing wrong with seeking safety and stability!

There is another downside to big national and multinational companies to watch out for though. I mention it here briefly as I intend to expand on this in a future post. Big companies will disappoint you. It's slow, over the course of years at times. The rewards will never seem to be quite right. There will always be a little more red tape, an extra caveat, a "gotcha" that always works against you. Be cautious.

No matter where you go one thing will be true. It is up to you to seize control of your future. Do not let time and fate dictate your path. Listen to advice and learn from your mistakes but also invest time into finding the right advice and learning the right lessons. Many people will seek to influence you. Some will have agendas that include your best interest and some will not. And if this is your second career and you already know all of this then please forgive this gentle reminder.

Thoughts

We are a community of engineers and we want to encourage and promote each other. I want you to feel encouraged and hopeful. I want you to succeed.

Most of all, I am glad to welcome you into the community of developers.

Someday, when you've become established, I want you to remember this time and help a newbie out. Do something, an introduction, encouragement, mentorship - something.

All of those senior engineers, the people who don't quite seem to hear you. As they are so shall you be.

Remember where you came from and who helped you. Remember to help the people who come after you. This is the way.

ML.NET and Python Polynomial Linear Regression

James MacIvor — Mon, 28 Dec 2020 21:57:17 +0000

ML.NET and Python Polynomial Linear Regression

This is the fourth in a series talking about differences implementing machine learning tasks in ML.NET and Python

Here are the previous articles if you'd like to peruse them

What is Polynomial Linear Regression?

Polynomial linear regression is a bit of a strange customer. Where before our regressions could be consistently described as lines, a polynomial regression is a curve. A polynomial regression is appropriate when there is exponential growth in the dependent variable.

So something like this simple example:

Independent	Dependent
1	4
2	8
3	16
4	32
5	64

Our data looks the same as simple linear regression at first, but upon noticing exponential growth in the dependent variable we need a different equation. A linear regression line will not be accurate. In fact, observing a regression line from a typical simple regression may be how we can identify the need for a polynomial regression.

On data such as this we would need to determine the correct exponent for growth and then apply it to the independent variable.

In order to perform the regression we need to expand out the independent variable by expanding this equation out for each possible result of 0 - {possible exponent value}. In practice such a transformation would change the above example into something like this.

Independent1	Independent2	Independent3	Dependent
1	1	1	4
1	2	4	8
1	3	9	16
1	4	16	32
1	5	25	64

The reason we still call this linear regression despite the fact the growth curve is exponential is because the independent variables are still expressed linearly. Basically when we see 1, 2, 3, 4 ... and can say

"Ok, this variable is growing in a linear way."

Then, even though our dependent variable is growing exponentially we still consider this a linear regression.

Polynomial regression is commonly used for tasks such as predicting disease spread in a population.

The Data

Thank you Super Data Science Team

This dataset and the Python information comes from Kirill Eremenko and Hadelin de Ponteves along with the Super Data Science Team.. They provide an excellent quality data science course on Udemy. The data is reused here with permission.

Here we have a list of 10 positions within a company and their corresponding salary. You can see also that each position has been assigned a level which grows linearly relative to the salaries which grow exponentially. It really pays to move up!

Polynomial Linear Regression in Python

Try it yourself on Binder

What I like about the Python implementation is how consistent the steps are. There is very little variation between regression tasks and yet we are still able to get great results. This task has some small variations in terms of the transformation we'll be doing and how we load in the data.

First as usual, we import the libraries we'll need. We need pandas to load in the data from our file and numpy as it's a dependency of pandas. Finally we need maplotlib.pyplot to chart and visualize our data and results.

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

Now unlike previously we are not loading all of the columns, instead we're using Python array syntax to leave out the text column from the data. We can do that because having the Level column already does all of the work the text column would do. In fact, if the Level column did not already exist we would transform Position into something similar. If you recall discussing transformations from multiple linear regression article then you might remember OneHotEncoder which is a function to transform text columns into non-weighted vectors for the trainer to use. In this case though, transforming Position into keys makes sense since there is an order inherent to the data.

dataset = pd.read_csv('Position_Salaries.csv')
X = dataset.iloc[:, 1:-1].values #leaving out the Position column by reading in the data starting at index 1 as opposed to 0 
y = dataset.iloc[:, -1].values #we read in only the last column, Salary for our dependent variable

Next we'll want to transform the Level column so we can model can have a proper curve representing exponential growth. Once we do that we can use good old LinearRegression trainer to take care of the rest.

Python has a class called PolynomialFeatures that takes care of this transformation for us. Python makes this exceptionally easy as we will soon appreciate. Notice that the PolynomialFeatures class takes a constructor argument called degree. Degree will be the exponent we expect the dependent variable to grow by. In this case we choose four, but you can experiment for yourself with different exponents to see how that changes the curve.

from sklearn.preprocessing import PolynomialFeatures
poly_reg = PolynomialFeatures(degree = 4)
X_poly = poly_reg.fit_transform(X)
lin_reg_2 = LinearRegression()
lin_reg_2.fit(X_poly, y)

With that transform out of the way we can go ahead and model the data and do a prediction.

plt.scatter(X, y, color = 'red')
plt.plot(X, lin_reg_2.predict(poly_reg.fit_transform(X)), color = 'blue')
plt.title('Polynomial Regression')
plt.xlabel('Position level')
plt.ylabel('Salary')
plt.show()

It's all so easy in Python. The tools are very consistent and I find it easy to find examples with a simple search.

Polynomial Linear Regression in ML.NET

Try it yourself on Binder

This was a difficult road. Polynomial Linear Regression does not have a built in transform in ML.NET so I needed to devise a custom one. This led to many hours of reading through documentation and simple trial and error to find the correct way to do this.

I did eventually figure it out and I'm glad to share my findings with you. There are gaps in the ML.NET documentation that made this harder than it really needed to be. Once I found the solution it became suddenly simple to do this.

Let us first throw out the mlnet command line tool. For this task it could not really provide us with anything useful. Unlike with multiple linear regression it could not even get us close.

The generated code also required significant alterations in order to reach our goal, which is a similar result to the Python process.

This is not to say others have not struggled with this and perhaps found their own way.

I'm reminded of this GitHub issue.

Or this StackOverflow question.

So lets get started. We do start from the usual place of setting up our using statements and reading in the data.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Composition;
using Microsoft.ML;
using Microsoft.ML.Data;
using Microsoft.ML.Trainers;
using Microsoft.ML.Transforms;
using XPlot.Plotly;

I am once again using XPlot to visualize our data.

Here are our proposed input and output classes as well.

public class ModelInput
{

    [ColumnName("Position"), LoadColumn(0)]
    public string Position { get; set; }

    [ColumnName("Level"), LoadColumn(1)]
    public float Level { get; set; }

    [ColumnName("Salary"), LoadColumn(2)]
    public float Salary { get; set; }
}

public class ModelOutput
{
    public float Score { get; set; }
}

And lets read in that data!

var mlContext = new MLContext();
IDataView trainingDataView = mlContext.Data.LoadFromTextFile<ModelInput>(
    path: "./Position_Salaries.csv",
    hasHeader: true,
    separatorChar: ',',
    allowQuoting: true,
    allowSparse: false);

The next steps require a lot more thought and alteration from what we've done previously.

The first thing to think about is our transformation. We need to define a custom transformation, but how can we do that?

There are two main ways, one requires a contract be defined on the transformation so that a pipeline can be saved to a file. I don't intend to save my pipeline to a file in any case, but nevertheless I chose to pursue that approach as I believed it would provide anyone attempting to learn from this blog greater flexibility in what they are attempting to do.

We will need to define an output class for our transform. We will also need to pay more attention to the attributes assigned to the class. In particular, as we will be expanding a single vector into a multitude of vectors we will need to define an array of floats in order to capture this change.

If your intent with a transformation is to change a single property into multiple properties then an array is the simplest way to do this. However, if you have specific and consistent transformations in mind then you may also define new properties on your TransformOutput class. We also can now make some guesses at how the OneHotEncoding transform works under the hood. (If you remember that discussion from the previous entry in this series)

The caveat is that we must know the length of the array so we can put the proper value in the attribute. Per the documentation, leaving the argument off of the Vector attribute should allow us to put in an array of unknown length. In practice this does not work and an error will be thrown upon attempting to fit the data.

For polynomial transforms the array size will be the exponent value plus one. So, if I have an exponent of four, than the array size will be five.

public class TransformOutput
{
    [VectorType(5)]
    public float[] Features {get;set;} 

    public float Salary { get; set; }
}

Now we can create our custom transform. I'd like it to receive the same inputs and produce the same outputs as the python transform.

[CustomMappingFactoryAttribute("Features")]
public class PolynomialFeatures : CustomMappingFactory<ModelInput, TransformOutput>
{

    private readonly int _degree;
    public PolynomialFeatures(int degree)
    {
        _degree = degree;
    }

    public void Transform(ModelInput input, TransformOutput output)
    {
        output.Features = Enumerable.Range(0, _degree + 1).Select(i => (float)(Math.Pow(input.Level, i))).ToArray();
        output.Salary = input.Salary;
    }

    public override Action<ModelInput, TransformOutput> GetMapping()
    {
        return Transform;
    } 

}

Note that this implementation is good enough for our trivial example, but if you start dealing with extremely large numbers you'll need to likely use a better implementation. The requirements are that the transformation must be thread safe and the transformation must not have side effects. So no altering of anything aside from the output parameter.

I tested my custom transform just to make sure it was good to go. Note that the VectorType array length requirement does not matter until we reach the Fit method of the pipeline onto the data.

var polyFeaturesTest = new PolynomialFeatures(2); //array length not required to match VectorType... yet
var testInputs = Enumerable.Range(1, 10).Select(i => new ModelInput() {Level = i});
var testOutputs = testInputs.Select(ti => 
{
    var testOutput = new TransformOutput();
    polyFeaturesTest.Transform(ti, testOutput);
    return testOutput;
});
display(testOutputs)

Now we can create the pipeline. Since we transformed the Level column and Level is a tokenized version of Position I chose to drop both Level and Position. I only need the transformed feature to proceed with the regression training.

For trainer, neither SDCA or FastTree do a great job here. In fact, both produce some hilarious results. Here is SDCA for example. (If you do use SDCA don't forget to set maximumNumberOfIterations as a parameter for the trainer, otherwise you'll be waiting around for fine minutes.)

So I keep coming back to the Ols trainer. It works super well with very little configuration! The only problem with it is that it requires a dependency not naturally available on Linux. It's really a shame because it's the closest we get to the easy functionality of the python sklearn LinearRegression trainer.

Lucky for you, I do show how to install that dependency on Linux while discussing how to get up and running using binder.

Anyway, here we are training with Ols.

var polyFeatures = new PolynomialFeatures(4);
var dataProcessPipeline = mlContext.Transforms.CustomMapping<ModelInput, TransformOutput>(polyFeatures.GetMapping(), contractName: "PolynomialFeatures")
                            .Append(mlContext.Transforms.DropColumns("Position", "Level"))
                            .Append(mlContext.Transforms.Concatenate("Features", new[] {"Features"}));

 var trainer = mlContext.Regression.Trainers.Ols(featureColumnName: "Features", labelColumnName: "Salary");
 var trainingPipeline = dataProcessPipeline.Append(trainer);

And then training the data is once again easy in terms of lines of code, but if there is an issue in your pipeline this is where it will manifest.

var transformedDataView = trainingPipeline.Fit(trainingDataView).Transform(trainingDataView);
var transformedData = mlContext.Data.CreateEnumerable<TransformOutput>(transformedDataView, reuseRowObject: false);
display(transformedData);
var model = trainingPipeline.Fit(trainingDataView);

For fun, I'm also now showing the result of our transformation.

And at last we have the results of all of this - it's nearly identical to the python result.

Thoughts

Whew. What a ride. I mean, ML.NET really had me running to figure stuff out!

But there is value in that. I understand regression in ML.NET a lot better than I did. I read through a lot of their transformer implementation while researching the custom transformer.

I have questions for the ML.NET maintainers. For example, why not re-implement the sklearn transformers and trainers? They took a way different path that requires a lot more work to use this library. I'll be seeking out an ML.NET engineer to interview to answer these and other questions. We'll see if any of them feel inclined to talk to me. I respect the work they've done here. It does feel like they did not write this to compete with python, or to even do the same things as Python. I can't find any information to corroborate this, but it seems like they had different goals in mind. I'd like perspective on that.

I am not discouraged though! If anything, this series has inspired me to keep learning deeply about these different implementations. ML.NET is young enough that it has a lot of catching up to do. Don't forget that the scikit-learn (sklearn) repository has been in active development since 2007 while ML.NET was started in 2018.

I've invited a guest to co-write the next article with me. He's a Java developer and so for the first time we'll be attempting to compare implementations between .NET, Python and Java. We'll be discussing which machine learning task will be the most interesting to compare. If you have an opinion on that then you should definitely let me know via the contact me form.

Edit: We're going to do it on NLP!

Either way, eventually I'll be talking about Support Vector Regression! So look for that as well.

ML.NET and Python Multiple Linear Regression

James MacIvor — Mon, 07 Dec 2020 15:11:07 +0000

ML.NET and Python Multiple Linear Regression

This is the third in a series talking about the differences in implementation between ML.NET and Python for machine learning tasks. Don't worry though, you should be able to jump right in to read this post.

However, here are the previous posts if you'd like to take a look.

If you want to install ML.NET and setup a .NET notebook environment then I explain how to do that in these posts.

What is Multiple Linear Regression

It is the same thing as simple linear regression but with many more variables.

Also, data for a successful linear regression must adhere to some core assumptions.

I list these assumptions out in another article. However, understanding these will not be necessary for the ML.NET vs Python implementation comparison. Our data can be assumed to meet the requirements this time.

The Data

Thank you Super Data Science Team

This data represents a bunch of different startups. Our goal is to predict the profit of a startup.

We can see as independent variables we have R&D Spend, Administration, Marketing Spend, and the State in which the startup operates. The dependent variable is the Profit.

Multiple Linear Regression in Python

Try it for yourself on Binder

As usual, we start by importing our libraries and reading in the data.

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

dataset = pd.read_csv('50_Startups.csv')
X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, -1].values

Now, however, we need to do a data transformation. The State column cannot be represented as a string. Instead we're going to use methods imported from the sklearn library to transform it into three columns, a column for each State. The method that does the majority of the work here is OneHotEncoder. Now we can represent this data numerically.

You might wonder why a single column with values of 1, 2, and 3 are not used. The answer is that the ML algorithm will then seek to order those values even though there is no real ordering of States in that way. Making each State it's own column eliminates that bias.

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
ct = ColumnTransformer(transformers=[('encoder', OneHotEncoder(), [3])], remainder='passthrough')
X = np.array(ct.fit_transform(X))

Aside from that small change the steps are now almost identical to simple linear regression.

We now split into training and test sets.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

We train the model using the exact same function as we used for simple linear regression. The LinearRegression function is capable of training models for simple and multiple regression.

from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train, y_train)

And we can predict the results as normal.

y_pred = regressor.predict(X_test)

Lets see what the results were. Rather than chart the data we will transform it into two columns. The first column will be the predicted profits and the second will be actual profits.

np.set_printoptions(precision=2)
print(np.concatenate((y_pred.reshape(len(y_pred),1), y_test.reshape(len(y_test),1)),1))

Some of the predictions are pretty good while others are just sort of okay. Nevertheless, given the small sample this is a pretty good result for a multiple linear regression model. It's nice that there are few extra steps to train this model in spite of the fact that the data is much more complex relative to simple linear regression.

In fact, lets calculate the RSquared value as this will be relevant when we look at ML.NET. The RSquared value is a metric we can use to calculate the accuracy of a trainer algorithm.

from sklearn.metrics import r2_score
print(r2_score(y_test, y_pred))

The score for this regression is 0.9347068473282515, which is quite good given that we have a small data set. The goal for an RSquared score should be to get as close to 1 as possible. Scores at .5 or below means the model is more or less guessing for the prediction.

Multiple Linear Regression in ML.NET

Try it for yourself on Binder

Right away we start to see some major differences. First off, the algorithm detection experiment from the command-line tool shows some less than encouraging results for the proposed model.

We can get this output by running the command-line tool against our dataset with this command.

mlnet regression --dataset .\50_Startups.csv --label-col Profit

You can read more about the command-line tool here.

The columns are not labeled at this stage of the output, but they are AlgorithmName, RSquared, Absolute-loss, Squared-loss, RMS-loss, Duration, Iteration. You can read about how to interpret these results from Microsoft's guide.

But here we are getting negative values for our RSquared values and our other metrics are equally bad. So already things are not equivalent to Python. However, the command-line experiment still did narrow down to the top two training algorithms used for regression.

SdcaRegression
FastTreeRegression

Also, recall that in Python we used OneHotEncoder to transform the State column such that it could be more easily used by the training algorithm. At this stage, it's unclear if ML.NET has done the same transformation - although judging by the analysis output it would appear it has not.

Lets run a quick and dirty test to see if that column is the cause of our bad data by deleting the state column and running the command-line test again.

We see an improvement in that the negative numbers are smaller - but still not the kind of output we would like.

This indicates that the dataset is either too small for the mlnet command-line tool to create or that there is some other issue with the data that causes the mlnet command-line experiment to be no better than a guess as it regards making predictions based on this data.

But lets forgive the command-line tool and move on to see if the actual implementation does better. After all, we took our RSquared score for Python after we had completed a full implementation. I'd like to see parity with they Python sklearn LinearRegression trainer function.

Some of these steps are exactly the same as simple linear regression covered in the previous blog post.

First, we download some NuGet packages and set up our using statements. You'll notice I'm pulling in the NuGet package for the FastTree regression as well - that's because I want to experiment between regression trainers once we establish our test and training datasets.

// ML.NET Nuget packages installation
#r "nuget:Microsoft.ML,1.5.0"
#r "nuget:Microsoft.ML.Mkl.Components,1.5.0"
#r "nuget:Microsoft.ML.FastTree,1.5.0"
//Install XPlot package
#r "nuget:XPlot.Plotly,2.0.0"    
using Microsoft.ML;    
using Microsoft.ML.Data;
using Microsoft.ML.Trainers.FastTree;
using XPlot.Plotly;
using System;
using System.Linq;

Second, we set up our classes for input and output. The Score property for the output is the predicted profit.

public class ModelInput
{
    [ColumnName("R&D Spend"), LoadColumn(0)]
    public float R_D_Spend { get; set; }

    [ColumnName("Administration"), LoadColumn(1)]
    public float Administration { get; set; }

    [ColumnName("Marketing Spend"), LoadColumn(2)]
    public float Marketing_Spend { get; set; }

    [ColumnName("State"), LoadColumn(3)]
    public string State { get; set; }

    [ColumnName("Profit"), LoadColumn(4)]
    public float Profit { get; set; }

}

public class ModelOutput
{
    public float Score { get; set; }
}

Third, we create the MLContext instance and we read in the .csv file to create our test and training sets.

var mlContext = new MLContext(seed: 1);

IDataView trainingDataView = mlContext.Data.LoadFromTextFile<ModelInput>(
    path: @"./50_Startups.csv",
    hasHeader: true,
    separatorChar: ',',
    allowQuoting: true,
    allowSparse: false);

var split = mlContext.Data.TrainTestSplit(trainingDataView, testFraction: 0.2);

These first three steps don't change much aside from the properties in the models. But now we start to see some interesting differences.

var dataProcessPipeline = mlContext.Transforms.Categorical.OneHotEncoding(new[] { new InputOutputColumnPair("State", "State") })
                                      .Append(mlContext.Transforms.Concatenate("Features", new[] { "State", "R&D Spend", "Administration", "Marketing Spend" }));

Ah! Just as Python, we're using a tool called OneHotEncoding. Also, just as Python it transform the encoded columns into multiple columns. However, unlike Python it's more difficult to inspect the data to see what it did.

A Small Discussion on OneHotEncoding in ML.NET

Microsoft does provide a way inspect the transformed data which they go over in their discussion of the OneHotEncoding function.

The short of it is you must transform your model back into a DataView using a transform method. Then you can use the MLContext to convert the IDataView into an IEnumerable and inspect it as you would any list.

Here is the useful example from the Microsoft documentation.

var mlContext = new MLContext();

// Create a small dataset as an IEnumerable.
var samples = new[]
{
    new DataPoint {Education = "0-5yrs", ZipCode = "98005"},
    new DataPoint {Education = "0-5yrs", ZipCode = "98052"},
    new DataPoint {Education = "6-11yrs", ZipCode = "98005"},
    new DataPoint {Education = "6-11yrs", ZipCode = "98052"},
    new DataPoint {Education = "11-15yrs", ZipCode = "98005"}
};

// Convert training data to IDataView.
IDataView data = mlContext.Data.LoadFromEnumerable(samples);

// Multi column example: A pipeline for one hot encoding two columns
// 'Education' and 'ZipCode'.
var multiColumnKeyPipeline =
    mlContext.Transforms.Categorical.OneHotEncoding(
        new[]
        {
            new InputOutputColumnPair("Education"),
            new InputOutputColumnPair("ZipCode")
        });

// Fit and Transform data.
IDataView transformedData =
    multiColumnKeyPipeline.Fit(data).Transform(data);

var convertedData =
    mlContext.Data.CreateEnumerable<TransformedData>(transformedData,
        true);

Console.WriteLine(
    "One Hot Encoding of two columns 'Education' and 'ZipCode'.");

// One Hot Encoding of two columns 'Education' and 'ZipCode'.

foreach (TransformedData item in convertedData)
    Console.WriteLine("{0}\t\t\t{1}", string.Join(" ", item.Education),
        string.Join(" ", item.ZipCode));

// 1 0 0 1 0
// 1 0 0 0 1
// 0 1 0 1 0
// 0 1 0 0 1
// 0 0 1 1 0

It's a lot more work to view this data in C#, but there are also few reasons you'd want to inspect this data in a production scenario.

It's also good they include an example like ZipCode. Thinking about how this OneHotEncoding works and ZipCodes - you might imagine a column with hundreds of unique ZipCodes. Would this mean that OneHotEncoding generates hundreds of columns, one for each ZipCode?

Yes. It would. However, you can configure the transform to generate a key encoded column instead, which would encode the data into a single column (and therefore take on the drawbacks of that transform, a potentially less accurate model).

In the case of ZipCode, when there are hundreds of unique entries, why would you wish to include this data? Might you be better off without it? Or could you transform it into something less granular such as State, Province, County, Shire or what have you.

ML.NET and Python both provide wonderful tooling for handling data such as this, but ultimately it's still up to us to make reasonable decisions about each data point.

Let's get to training our model!

I ran the mlnet command-line tool multiple times and ultimately the FastTree regression trainer consistently had the least worst RSquared Score. It reached an RSquared Score in the -800s. Abysmal, but still better than before. In the generated project, we see this is how the regression trainer is added.

var trainer = mlContext.Regression.Trainers.FastTree(new FastTreeRegressionTrainer.Options() { NumberOfLeaves = 2, MinimumExampleCountPerLeaf = 10, NumberOfTrees = 100, LearningRate = 0.06297909f, Shrinkage = 0.0986204f, LabelColumnName = "Profit", FeatureColumnName = "Features" });

var trainingPipeline = dataProcessPipeline.Append(trainer);

We can then train our model.

var trainingPipeline = dataProcessPipeline.Append(trainer);

And now we can perform an evaluation of the trainer. What's the RSquared score now? Keep in mind, RSquared is not the only metric we can evaluate - but it's still a really good one for determining accuracy.

So lets take a look. Here's the code to generate an analysis.

Console.WriteLine("=============== Cross-validating to get model's accuracy metrics ===============");
var crossValidationResults = mlContext.Regression.CrossValidate(split.TrainSet, trainingPipeline, numberOfFolds: 5, labelColumnName: "Profit");
var L1 = crossValidationResults.Select(r => r.Metrics.MeanAbsoluteError);
var L2 = crossValidationResults.Select(r => r.Metrics.MeanSquaredError);
var RMS = crossValidationResults.Select(r => r.Metrics.RootMeanSquaredError);
var lossFunction = crossValidationResults.Select(r => r.Metrics.LossFunction);
var R2 = crossValidationResults.Select(r => r.Metrics.RSquared);

Console.WriteLine($" *************************************************************************************************************");
Console.WriteLine($"* Metrics for Regression model ");
Console.WriteLine($"*------------------------------------------------------------------------------------------------------------");
Console.WriteLine($"* Average L1 Loss: {L1.Average():0.###} ");
Console.WriteLine($"* Average L2 Loss: {L2.Average():0.###} ");
Console.WriteLine($"* Average RMS: {RMS.Average():0.###} ");
Console.WriteLine($"* Average Loss Function: {lossFunction.Average():0.###} ");
Console.WriteLine($"* Average R-squared: {R2.Average():0.###} ");
Console.WriteLine($" *************************************************************************************************************");

And here's the result.

These metrics are still truly awful. Should you actually make predictions on the test set you would see it really is no better than a wild guess at this point. It's not even close to the actual profit.

However, we see something encouraging. The actual implementation does improve the score. RSquared is now a mere -2.853.

It seems at a certain point we cannot trust our mlnet command-line experiments.

After some manual experimentation with some of the regression trainers I was able to generate this:

var dataProcessPipeline = mlContext.Transforms.Categorical.OneHotEncoding(new[] { new InputOutputColumnPair("State", "State") })
                                      .Append(mlContext.Transforms.Concatenate("Features", new[] { "State", "R&D Spend", "Administration", "Marketing Spend" }));

var trainer = mlContext.Regression.Trainers.Ols(labelColumnName: "Profit", featureColumnName: "Features");
var trainingPipeline = dataProcessPipeline.Append(trainer);

That's right, good old Ols regression trainer is back again. It did great for the simple linear regression and it continues to perform well enough for the multiple linear regression example.

It evaluated to this:

We have an RSquared score of 0.909! This is more or less okay I guess! It's still not as good as the RSquared of the LinearRegression algorithm from sklearn, but we're much closer.

Now if you compare the predicted profit to the actual profit like so -

var predEngine = mlContext.Model.CreatePredictionEngine<ModelInput, ModelOutput>(model);
var testSet = mlContext.Data.CreateEnumerable<ModelInput>(split.TestSet, reuseRowObject:false);
foreach (var ts in testSet) 
{
    var testInput = new ModelInput() 
    {
        State = ts.State,
        Marketing_Spend = ts.Marketing_Spend,
        R_D_Spend = ts.R_D_Spend,
        Administration = ts.Administration,
        Profit = 0.0F
    };
    var prediction = predEngine.Predict(testInput).Score;
    var actual = ts.Profit;
    Console.WriteLine($"Prediction: {prediction}, Actual: {actual}");
}

We see this -

Not bad! Pretty close in some instances! This is a pretty fair prediction of potential profits.

Thoughts

It would be easy, and perhaps accurate, to believe that Python is the better tool for this task.

This is misleading though because the Python steps come from a Udemy course. The data and process were tailored to work well with that ecosystem.

What are we really seeing here then?

ML.NET could still use this data and create predictions that were comparable to the Python predictions. Moreover, the tooling with ML.NET set me on a path where I could figure out for myself which training algorithm to use. The command-line tool did not hand me the answers right away, as it did with simple linear regression - but it set me on a path to figure it out.

Still, the sheer simplicity and ease of use from Python cannot be refuted. In Python's sklearn, we are not hunting for a linear regression training algorithm, there is only one: LinearRegression(). The parameters for that function are documented and intuitive.

With ML.NET, I'm reminded of the game Dwarf Fortress. Dwarf Fortress is well known for it's steep learning curve, but many players find the process of learning fun and rewarding. The value is there, hidden under trial and error. Success is found after a series of hilarious trials and failures.

Also with ML.NET, we also come away with a deeper understanding of what's going on under the hood. As we continue to use it we see opportunities to tweak and change the process to produce better results.

The Python libraries hand us success first and allow us to learn deeply later.

From a business perspective, Python is the clear favorite, at least in the short term. The business value will be successful predictions from a process created as cheaply as possible. However, in the long term a business would benefit from type safety and the resulting easier maintenance.

We'll see how ML.NET performs next to Python in my next post which will be looking at the Polynomial regression.

VSCode .NET Interactive Notebooks

James MacIvor — Mon, 30 Nov 2020 14:18:06 +0000

VSCode .NET Interactive Notebooks

Where Jupyter Notebooks Falls Short

Do you love intellisense? I do.

Which is why working in Jupyter Notebooks can be so frustrating. Part of the benefits of a strongly typed language is to enable tooling that can make us more efficient developers.

VS Code Extension

Well, lucky for us the folks who develop .NET Interactive love and support us. They have created an extension for VSCode called .NET Interactive Notebooks

Now before you go and download this extension be aware that it is in preview and only will work with Visual Studio Code Insiders. This is just a fancy way of saying that it's in active development and stuff will be broken.

That said, lets take a look at how well it works at the moment.

The Setup Experience

In order to get Visual Studio Code to set everything up properly you'll need to attempt to open a .ipynb file.

You can use the one from my gist if you don't have another one handy:

You'll be prompted to install the Python extension. Also, if you had the .ipynb file open when Python installed then you may need to close and reopen it. VSCode will also handle installing .NET Interactive in the background. In fact, VSCode is doing so much for me automatically it was a pretty nice experience to get setup.

Once everything is setup you should see the notebook render properly:

So, that's a fairly friendlier experience than Jupyter Notebooks - but do I get my intellisense?!

The answer is... I do!

So I know how I'll be writing my .NET notebooks from now on!

But if you still prefer Jupyter Notebooks with .NET then here is the guide to get set up with that too.

Share Your Dotnet Notebooks on Binder

James MacIvor — Mon, 23 Nov 2020 17:32:24 +0000

Share Your .NET Notebooks on Binder

What is Binder?

Binder is a free online service that allows you to run your jupyter notebooks with other people. It's similar to Google Colab but does not require a Google login. In fact, all it requires is a public repository with a .ipynb file.

It's a wonderful service and supports many languages right out of the box. To spin up a Python notebook you only need a simple requirements.txt file. If you intend to use R or Julia then an environment.yml file will do. You can follow Binder's guides to get this up and running.

Setup can be a little more difficult in .NET.

So, I'm happy to provide you with this base docker image: jmacivor/dotnet-binder:0.1.1 which will provide you with the .NET SDK as well as everything necessary to work on Binder. You can then copy in your notebooks and related files and run them easily.

If you are new to Docker you can set up a file similar to what I've created below. Again, you just need something like this in the root of the repository containing your *.ipynb.

Binder allows you to create shared notebooks from all kinds of repositories, notably from gists and github. You can setup a new github or gist. As long as you have a Dockerfile where you copy in a notebook then you're good to go!

FROM jmacivor/dotnet-binder:0.1.1

ARG NB_USER=jovyan
ARG NB_UID=1000

USER $NB_USER

ENV HOME=/home/$NB_USER

WORKDIR $HOME

COPY ml_net_simple_regression.ipynb $HOME/ml_net_simple_regression.ipynb
COPY Salary_Data.csv $HOME/Salary_Data.csv

Checkout the gist.

If you would like to design and test your notebook locally then follow my guide for getting setup with Jupyter Notebooks.

It took a bit of effort to create the base image used in the above Dockerfile. If you're curious about what all went into it then check it out below. There is definitely room for improvement.

This base image would benefit from the following changes:

Switch to dotnet alpine SDK in order to reduce the size of the overall image.
Find a way to download and cache the nuget packages for ML.NET so that users are not forced to download them from the notebook.
Upgrade the .NET base image to .NET 5 as soon as the dotnet-interactive tool is ready for .NET 5.

Here is the gist.

FROM mcr.microsoft.com/dotnet/core/sdk:3.1-focal

USER root
RUN cd /tmp
# now get the key:
RUN wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
# now install that key
RUN apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB
# now remove the public key file exit the root shell
RUN rm GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB

# we have to get this because the linux mlnet nuget expects a dependency that only ships with windows.
# so we add the public key as shown above and apt-get install intel-mkl-64bit-2020.0.088 
RUN sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list'

RUN apt-get update \
    && apt-get -y upgrade \
    && apt-get -y install python3 python3-pip python3-dev ipython3 intel-mkl-64bit-2020.0-088

RUN find /opt -name "libiomp5.so"
RUN ldconfig /opt/intel/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/

RUN pip3 install --no-cache notebook

ARG NB_USER=jovyan
ARG NB_UID=1000
RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER

USER $NB_USER

ENV HOME=/home/$NB_USER

WORKDIR $HOME

ENV PATH="${PATH}:$HOME/.dotnet/tools/"

RUN dotnet tool install -g --add-source "https://dotnet.myget.org/F/dotnet-try/api/v3/index.json" dotnet-interactive

RUN dotnet-interactive jupyter install

By the way, dotnet-interactive has their own Dockerfile for Binder. They go a completely different direction where they start from a base image made for Binder and then install the .NET dependencies and .NET from there. I did try this approach, but found a bunch of problems along the way. Still, you might enjoy trying it this way. The time to start up the Binder server appears to be about the same regardless of which approach you use.

ML.NET and Python Simple Regression

James MacIvor — Mon, 23 Nov 2020 16:27:49 +0000

ML.NET and Python Simple Regression

Why even bother with ML.NET

ML.NET has been around for a few years now, and yet the community excitement has been muted. Don't get me wrong, it definitely exists - but on the whole the sentiment is that this has been solved in other languages so why bother with ML.NET.

This is valid, but it ignores the many companies where the outcomes that can be achieved from machine learning are a goal, but they don't necessarily want to start creating and maintaining a bunch of Python projects. Companies that have invested in the .NET stack should look to ML.NET for as an alternative - but how close is it?

I'll be examining that in a series of blogs.

First things first, get set up with Jupyter Notebooks and the .NET runtime. Setup Jupyter Notebooks and .NET

Once you're all set up, lets take a look at the data.

The Data

Thank you

First this dataset and the Python information comes from is from Kirill Eremenko and Hadelin de Ponteves along with the Super Data Science Team.. They provide an excellent quality data science course on Udemy.

Here is the data:

This data has two columns, years of experience and salary. The data suggests that there is a relationship between these two things - so we should be able to do a simple regression to predict a salary based on the years of experience.

Simple Regression in Python

First - how do we do this in Python?

Try it for yourself on Binder

First we need to import the necessary files.

import matplotlib.pyplot as plt
import pandas as pd

Pandas will read in the data from the csv file while matplotlib will do the work of creating some graphs for us.

dataset = pd.read_csv('Salary_Data.csv')
X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, -1].values

This is wonderful simplicity. The dataset is split into x and y axes using simple array syntax.

Next we import a function from sklearn in order to split the dataset into train and test sets.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 1/3, random_state = 0)

Next we train the model. Again we use sklearn and import the simply named LinearRegression class in order to train the regressor model.

from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train, y_train)

Now we can predict the test set results and store them into a variable.

y_pred = regressor.predict(X_test)

And we can use the matplotlib library to visualize the results from our training set.

plt.scatter(X_train, y_train, color = 'red')
plt.plot(X_train, regressor.predict(X_train), color = 'blue')
plt.title('Salary vs Experience (Training set)')
plt.xlabel('Years of Experience')
plt.ylabel('Salary')
plt.show()

And finally, as we have the test prediction results we can now put those into a visualization and compare them to the actual data.

plt.scatter(X_test, y_test, color = 'red')
plt.plot(X_train, regressor.predict(X_train), color = 'blue')
plt.title('Salary vs Experience (Test set)')
plt.xlabel('Years of Experience')
plt.ylabel('Salary')
plt.show()

Simple Regression in ML.NET

And now how do we do the same simple regression in ML.NET?

You will notice some major differences here. ML.NET throws us a little further into the deep end. There are tools and support, but it's still a steeper learning curve than Python.

Try it for yourself on Binder

First things first, you have to install the nuget packages and set up the using statements. From a practical standpoint this is almost identical to Python. Jupyter Notebook has a special way to get a nuget package to download in a code block. All that has to be done is setting up a #r followed by nuget: and then the full name of the nuget package followed by a comma and the version desired. See below to see an example.

// ML.NET Nuget packages installation
#r "nuget:Microsoft.ML,1.5.0"
#r "nuget:Microsoft.ML.Mkl.Components,1.5.0"
//Install XPlot package
#r "nuget:XPlot.Plotly,2.0.0"    
using Microsoft.ML;    
using Microsoft.ML.Data;
using XPlot.Plotly;
using System;
using System.Linq;

This works just fine when running from a local Jupyter Notebook, but on Binder the nuget download has to be attempted twice and always works on the second time.

Now, unlike Python where we could immediately pull in the data we have to setup some classes. As C# is a strongly typed language we will have classes to model the data we'll be working with. This is a bit more effort for setup, but placed me squarely in my comfort zone.

public class ModelInput
{
    [ColumnName("YearsExperience"), LoadColumn(0)]
    public float YearsExperience { get; set; }

    [ColumnName("Salary"), LoadColumn(1)]
    public float Salary { get; set; }
}

public class ModelOutput
{    
    public float Score { get; set; }
}

Now that we've defined our classes we can move forward with loading in the data. ML.NET uses a class called MLContext in order to load in data and perform operations on it. This class is a sort of combination of pandas and sklearn from Python. MLContext has a property called Data that can load the data from the csv file. It can also split the data up for us into training and test sets.

var mlContext = new MLContext(seed: 1);

IDataView trainingDataView = mlContext.Data.LoadFromTextFile<ModelInput>(
                                            path: "./Salary_Data.csv",
                                            hasHeader: true,
                                            separatorChar: ',',
                                            allowQuoting: true,
                                            allowSparse: false);

var split = mlContext.Data.TrainTestSplit(trainingDataView, testFraction: 0.2);
//split.TrainSet
//split.TestSet

Now Python has the matplotlib library which is a wonderful and versatile charting library. .NET has several libraries that can do similar things but require different contexts and setup. Luckily, FsLab has created something that works very well on Jupyter Notebooks called XPlot.

Here we see how to model the training data and create a scatter plot. Matplotlib can also create a regression line with only a few arguments but XPlot requires a bit more setup as we will see later.

var years = split.TrainSet.GetColumn<float>("YearsExperience").ToArray();
var salary = split.TrainSet.GetColumn<float>("Salary").ToArray();

var yearsChart = Chart.Plot(new Graph.Scatter
{   
   x = years,
   y = salary,
   mode = "markers"
});

yearsChart.WithTitle("Years Vs Salary");
display(yearsChart);

Now at last we can create the pipeline which will allow us to train the model. A big difference here is the algorithm used. Where Python had the aptly named LinearRegression() class to handle this, we have a different experience with ML.NET. We are confronted with a bunch of confusingly named choices. For this, I chose to use the mlnet command line tool to analyze the dataset and and select the appropriate algorithm for me. This time it chose the Ols() algorithm. The command line tool does print comparitive metrics for each algorithm tested so you can view for yourself why it selected whichever algorithm it did.

I go over the command line tool a bit more in this post.

Also, unlike Python, you still have to define features even for this simple case in ML.NET.

var dataProcessPipeline = mlContext.Transforms.Concatenate("Features", new[] { "YearsExperience" })
                          .Append(mlContext.Transforms.NormalizeMinMax("Features", "Features"));
// Set the training algorithm 
var trainer = mlContext.Regression.Trainers.Ols(labelColumnName: "Salary", featureColumnName: "Features");

// Build training pipeline
IEstimator<ITransformer> trainingPipeline = dataProcessPipeline.Append(trainer);

Now we can train our model. Python and ML.NET make this step almost comically easy.

ITransformer mlModel = trainingPipeline.Fit(split.TrainSet);

We can also now evaluate our pipeline. ML.NET has a method that will iterate over the validations a set amount of times (numberOfFolds) and then display the averaged results.

var crossValidationResults = mlContext.Regression.CrossValidate(trainingDataView, trainingPipeline, numberOfFolds: 5, labelColumnName: "Salary");
var L1 = crossValidationResults.Select(r => r.Metrics.MeanAbsoluteError);
var L2 = crossValidationResults.Select(r => r.Metrics.MeanSquaredError);
var RMS = crossValidationResults.Select(r => r.Metrics.RootMeanSquaredError);
var lossFunction = crossValidationResults.Select(r => r.Metrics.LossFunction);
var R2 = crossValidationResults.Select(r => r.Metrics.RSquared);

Console.WriteLine($" *************************************************************************************************************");
Console.WriteLine($"* Metrics for Regression model ");
Console.WriteLine($"*------------------------------------------------------------------------------------------------------------");
Console.WriteLine($"* Average L1 Loss: {L1.Average():0.###} ");
Console.WriteLine($"* Average L2 Loss: {L2.Average():0.###} ");
Console.WriteLine($"* Average RMS: {RMS.Average():0.###} ");
Console.WriteLine($"* Average Loss Function: {lossFunction.Average():0.###} ");
Console.WriteLine($"* Average R-squared: {R2.Average():0.###} ");
Console.WriteLine($" *************************************************************************************************************");

And now we can test our model and try some predictions. I believe there are more efficient ways to perform predictions over a bulk dataset than what I have done - but I'm familiar with Linq so I chose to use that to perform my predictions.

var predEngine = mlContext.Model.CreatePredictionEngine<ModelInput, ModelOutput>(mlModel);
var enumerableTestSet = mlContext.Data.CreateEnumerable<ModelInput>(split.TestSet, reuseRowObject: false)
        .Select(ts => new ModelInput() { YearsExperience = ts.YearsExperience });         
var preductionResults = enumerableTestSet.Select(ts => predEngine.Predict(ts));

foreach (var predictionResult in preductionResults)
{
    Console.WriteLine($"\n\nPredicted Salary: {predictionResult.Score}\n\n");
}

Lets put all of this into a chart next to the actual data. (At last, a regression line!)

var testYears = enumerableTestSet.Select(ts => ts.YearsExperience).ToArray();
var actualSalaries = mlContext.Data.CreateEnumerable<ModelInput>(split.TestSet, reuseRowObject: false).Select(ts => ts.Salary).ToArray();
var predictedSalaries = preductionResults.Select(r => r.Score).ToArray();

var actual = new Graph.Scatter()
{
    x = testYears,
    y = actualSalaries,
    mode = "markers",
    name = "Actual"
};

var predicted = new Graph.Scatter()
{
    x = testYears,
    y = predictedSalaries,
    mode = "line",
    name = "Predicted"
};

var chart = Chart.Plot(new[] {actual, predicted});
var layout = new Layout.Layout(){barmode = "group", title=""};
chart.WithLayout(layout);
chart.WithXTitle("Years");
chart.WithYTitle("Salary");
chart.WithLegend(true);
chart.Width = 700;
chart.Height = 400;

display(chart);

Thoughts

This was an interesting exercise to see the two languages / libraries side by side and evaluate each approach. However, this exercise was too simple to really draw any solid conclusions. I'll be looking at a multiple linear regression next to see if a more complex example can reveal more about these different implementations.

C# is my comfort zone. I felt more secure working with the language. However, in the context of Jupyter Notebooks, C# is pretty rough, and oddly enough, based on the discussion and blog posts I read while researching, it rides in on the coattails on F#. I did a little extra to help out anyone who comes after me who wants to use Binder. Here's my How To Guide for .NET with Binder

That said there are features of both implementations that I like. Python requires a lot less code to accomplish the same tasks. However the mlnet command line tool is amazing and makes it all so easy to spin up a project. We'll see where a more complicated task takes us in my next post.

Kendo UI - Quick Guide to Grids

James MacIvor — Fri, 30 Oct 2020 13:26:46 +0000

Kendo UI - Quick Guide to Grids

You may not have heard of Kendo UI and that's okay. They're yet another one of Telerik's products, right up there with Sitefinity. The main thing you need to know is that Kendo UI works great as long as you stay strictly within Telerik's suggested use cases. It becomes a nightmare as soon as you attempt something outside of that. I'm speaking directly about the ASP.NET Core HtmlHelpers. The jQuery, Vue and other versions may work differently.

Perhaps one of the most useful Kendo features is the Grid. The grid is a table with sorting, paging and filtering all built in. It can also hook into your MVC endpoints to read, update and delete that data. Kendo in .NET Core uses jQuery Ajax methods to talk to those endpoints. (Telerik loves jQuery dependencies.)

Grids are strongly typed from your Model as well. They can easily create columns based on your model properties and it's also easy to create additional columns. In fact, Grids are pretty damn awesome as long as you adhere to the strict method of creating them.

Here is some code to get an example grid up and running (This would go inside of a div element on a cshtml page.):

@(Html.Kendo().Grid<MyModel>()
.Name("MyGrid")
.Columns(columns =>
{
    columns.Command(c => c.Destroy()).Width(50).Title("Command");
    columns.Bound(i => i.Id).Width(50).MinResizableWidth(50).Title("Id");
    columns.Bound(i => i.Name).Width(100).MinResizableWidth(100).Title("Name");
    columns.Bound(i => i.Status).Width(100).MinResizableWidth(100).Title("Status");                
})
.Editable(editable => editable.Mode(GridEditMode.InLine).DisplayDeleteConfirmation(false))
.HtmlAttributes(new { style = "height: 550px;" })
.Scrollable()
.Groupable()
.Sortable()
.Pageable()
.Filterable()
.Resizable(resize => resize.Columns(true))
.DataSource(dataSource => dataSource
    .Custom()
    .Schema(s => s
        .Parse(@<text> 
        function(data) {                        
            if (!data || typeof(data) === 'string' ) 
                data = [{ Id: "N/A", Name: "NA", Status: "N/A" }]
            return data;
        }
        </text>)
        .Model(m =>
        {
            m.Id(p => p.Id);
        }))                    
    .Transport(t =>
    {
        t.Read(r => r.Action("GetMyModels", "ControllerName"));
        t.Destroy(d => d.Action("DeleteAModel", "ControllerName"));
    })
    .PageSize(20))
)

Lets talk through all of this. First notice that you have to give the grid a model. This is not optional. There is no dynamic option. You must define a class and provide it to the Grid<> method. You must also give your grid a name. This will become the html id of the grid later. It will allow you to select the grid using jQuery and do ... stuff.

Next we have columns. As stated, columns can be quickly defined using Model properties, but notice that you can also define their width and resize attributes. You can provide them a title as well. You can also, in place of a property, define a Template column.Template("<div>My Cool Custom Thing</div>") from here you can also still access a property by using kendo expression syntax: #=MyModelProperty# Sometimes you can also use JavaScript expressions between the #= and the #, but that's not always the case. It requires experimentation.

Notice also we can define a Command column. You can have Destroy, Edit, Create and a few others for your Command. You can also define a custom command which will use your own JavaScript. Nice.

You will always need to .Editable(editable => editable.Mode(GridEditMode.InLine).DisplayDeleteConfirmation(false)) in the case of the delete command otherwise it will popup the basic browser alert statement to ask for confirmation.

Next we have a whole host of things that can be enabled on the grid, Groupable, Sortable, Filterable, Pageable, etc. all of these enable features on the grid so a user can mess with the data displayed to get it to their liking.

Finally we have DataSource. This one is the biggest pain, but it is absolutely necessary. In the Telerik documentation you will see reference to code like this

DataSource(datasource => datasource
    .Ajax()
    .Read(r => ...))

You don't have to bother with that crap. Kendo requires that you set up your endpoints a certain way with a specific wrapper around your data. It's probably fine, I guess - but I prefer to return plain old JSON from my endpoints that has the data I need. There might be additional Kendo magic you can unlock by doing it the other way though. However, all of the paging and sorting etc. works just fine without it.

Instead, the preferable way is to use Custom() as shown in the first code example. Doing it this way also makes the schema function available (it's not available for Ajax()) where you can run a parse method and transform data from your endpoint if you wish. You can also define a default row (as shown above) if you don't receive expected results. The Model method allows you to define custom fields and use an alternate Id from your model. This is important if you have MVC methods that need to accept different values than the default Id. For example, if you have a delete endpoint that uses the Name property then you could said Model(m => m.Id(p => p.Name)) making it so that value is what is passed to the MVC destroy method by default.

Transport wraps all of the MVC actions and you can use API endpoints from different servers as well as your own MVC endpoints. It provides the most flexibility in retrieving and manipulating your data. It is only available under the Custom() method, which is perhaps another reason to never bother with Ajax().

Finally you can define the PageSize for items to be displayed on the page.

Now, what if you have some detail data that you want displayed in a sub grid? Well Kendo makes that easy as long as you strictly adhere to their process.

@(Html.Kendo().Grid<MyModel>()
.Name("MyGrid")
.Columns(columns =>
{
    columns.Command(c => c.Destroy()).Width(50).Title("Command");
    columns.Bound(i => i.Id).Width(50).MinResizableWidth(50).Title("Id");
    columns.Bound(i => i.Name).Width(100).MinResizableWidth(100).Title("Name");
    columns.Bound(i => i.Status).Width(100).MinResizableWidth(100).Title("Status");                
})
.ClientDetailTemplateId("template")
.Editable(editable => editable.Mode(GridEditMode.InLine).DisplayDeleteConfirmation(false))
.HtmlAttributes(new { style = "height: 550px;" })
.Scrollable()
.Groupable()
.Sortable()
.Pageable()
.Filterable()
.Resizable(resize => resize.Columns(true))
.DataSource(dataSource => dataSource
    .Custom()
    .Schema(s => s
        .Parse(@<text> 
        function(data) {                        
            if (!data || typeof(data) === 'string' ) 
                data = [{ Id: "N/A", Name: "NA", Status: "N/A" }]
            return data;
        }
        </text>)
        .Model(m =>
        {
            m.Id(p => p.Id);
        }))                    
    .Transport(t =>
    {
        t.Read(r => r.Action("GetMyModels", "ControllerName"));
        t.Destroy(d => d.Action("DeleteAModel", "ControllerName"));
    })
    .PageSize(20))
)


<script id="template" type="text/kendo-tmpl">
    @(Html.Kendo().Grid<DetailModel>()
        .Name("template_#=Id#")
        .Columns(columns =>
        {
            columns.Bound(c => c.Address).Width(100).MinResizableWidth(50).Title("Address");
            columns.Bound(c => c.State).Width(100).MinResizableWidth(50).Title("State");
            columns.Bound(c => c.Country).Width(100).MinResizableWidth(50).Title("Country");
        })
        .Scrollable()
        .Groupable()
        .Sortable()
        .Pageable()
        .Filterable()
        .DataSource(dataSource => dataSource
        .Custom()
        .Schema(s => s
            .Parse(@<text>
            function(data) {
                if (!data || typeof(data) === 'string' )
                    data = [{ EmailType: "N/A", NoticeType: "N/A" }]
                return data;
            }
            </text>))
        .Transport(t =>
        {
            t.Read(r => r.Action("GetBounceTrackingDetails", "EDisclosures", new { Id = "#=Id#" }).Data("getParams"));
        })
        .PageSize(20))
        .ToClientTemplate()
    )
</script>

Now we have another template wrapped in a script tag. The script tag has an id which must match text in this method: ClientDetailTemplateId("template") in your parent grid.

The detail grid is pretty much the same, but keep in mind you'll have to escape the # in any #=kendo-expression# in the Columns area. They still work okay everywhere else, probably. You might have JSON with a sub collection and you might wish to pass the sub collection to the Kendo client template.

Sorry this is impossible. You cannot pass a collection from the parent to the child and expect it to be used in the grid columns. Instead, you have to define yet another endpoint and Kendo will perform an ajax call to retrieve detail information every time detail information is selected by a user. This is suboptimal but that's what Telerik wants and who are we to disagree?

Finally, and maybe most importantly, you must call ToClientTemplate() at the end of the child grid definition.

Oh, just in case you're like me and you like to use constructors in your models, you must define a parameter-less constructor or the Kendo UI grid will fail silently. Not only will it fail silently, but any HTML defined in your csthml file below the grid will fail to be added to the page. The lack of error messages will have you laughing for hours, so sorry to steal the joy of this problem. Define parameter-less constructors and you'll be good to go.

So, Kendo UI is nice when it works and you stay within the defined edge cases. It's a nightmare when you dare to set foot without, so just don't.

Good luck oh hapless developer. I pray for you.

Tale of Terror - A Scary Web Dev Story

James MacIvor — Fri, 30 Oct 2020 13:25:12 +0000

Tale of Terror - A Scary Web Dev Story

Mark Menzies

Intro

Wes Bos asked for scary stories on Twitter recently. I emailed him but was inspired to write about it myself.Update! My story was read on SyntaxFM ep. 294 starting on 40:19! Thank you Wes Bos and Scott Tolinski!

A Warm October

My story takes place many years ago. I was hired by a consulting company to work on an EPiServer (CMS) implementation for a client. My skill level at the time was probably just barely mid-level, but mainly in C# and .NET tech, not so much with JavaScript. I was a pretty fair hand with the internals of the CMS though so I thought I could figure it out, also I reasoned that my tech lead had a decade of experience so I could lean on him a little bit.

Arriving on site, the first thing the client told us was "No frameworks, plain JavaScript / jQuery only".

Okay, I thought, but then won't it take longer for us to have a reactive site? But there was more.

"We'll be ajaxing in the widgets from the CMS to other projects so the JavaScript can't conflict with any of those projects." When I asked what version of jQuery the other projects were using he said "Oh, various. Some are on version 2, others on version 1.9. That's not a problem, right?"

This sounded bad to me - wouldn't that be rough to know whether our code would work if we couldn't be sure what dependencies were available?

My tech lead was confident though - "No problem! We can do it!"

Okay, whew, well at least my tech lead had the goods. Over the next week or so while we familiarized with the project, the client would make some more restrictions around the JavaScript and CSS to use. Like no build steps for JavaScript, no jQuery plugins, etc etc.

The tech lead would sagely nod his head. "No problem!" He'd say.

A Cold November

One day I was working on some frontend stuff and had a problem. I was inexperienced with JavaScript and CSS and some elements just would not position themselves correctly. I asked my tech lead for help. As I'm explaining the problem he gestures at my screen and says "What is this tool you're using here?"

I had a dual monitor set up. One screen had the page and the other had Chrome Dev Tools. I didn't have anything fancy on the screen so I was puzzled.

"What do you mean? The console? Or something else?"

"No, what is this tool? It seems pretty useful."

"You mean... Chrome Dev Tools?"

"Oh! Is that what that is? That's built into Chrome then?"

My heart was in my stomach. Did he not know Chrome Dev Tools? What else might he not know? Am I the one who's wrong? Maybe dev tools aren't standard tooling for all web dev.

From a place of pure self doubt I withheld judgement and explained the tools. Then I asked "So what do you use?"

He showed me how he worked through JavaScript using an elaborate system of alert statements to debug his code. I kept my face straight, but internally I knew. I was in danger.

I soon learned that he also didn't know MVC. He also didn't know JavaScript too well. He asked me if I had ever used Moo Tools. This was the first time I had ever heard of Moo Tools. (It was a competitor for jQuery in like 2006 FYI.)

"I'm so glad you're on this project and can help me out with this stuff." He said to me one day.

Cold sweat trickled down my spine. I smiled.

"Sure. No problem."

Over the weeks he struggles with user stories and I pick up more. Soon, the client was bypassing him and asking me directly to plan and implement the features.

The Real Horror Begins

I, barely a midlevel developer, start to plan the architecture of our little piece of the project. The .NET piece seems fine just following the established patterns I already knew. The JavaScript left a lot to be desired.

I read a lot. I made friends with developers on other projects and started asking them lots of questions. I was determined to nail this. I believed I could do it.

I decided that since we couldn't use frameworks but needed framework like behavior then I would build my own custom framework. A mini-framework. Just enough to get things working. I started writing JavaScript - a lot of JavaScript!

I had read about something called MutationObserver which is something in JavaScript that allowed me to observe changes to elements and then react to them. I decided that would be the core of my project. I built a central engine around this. I made JavaScript functions representing pieces of functionality I needed and overloaded the prototype to get the functionality I needed.

I created a custom dependency injection setup for my functions. I believed I could create something similar to Angular.js, just a touch simpler. I worked late every night for weeks. I was marking features as completed but I would frequently revisit them.

Really, nothing was ever finished because I kept changing it.

I worked into a delirium. I worked until I thought I was crazy.

I believed, I really believed I was making a fairy tale palace of JavaScript. You read this and you think you know how bad it was.

It was worse.

The Dungeon

Things were getting close to completed, or so I thought. But everything always seemed to be broken.

Things happened on the site but I had no idea why. Race conditions abounded. The off-by-one errors had off-by-one errors.

Slowly, painfully, I began to realize that this was my fault. My design was flawed. My assumptions, wrong. All the work, the long hours - wasted. I became bitter.

I was behind schedule but I was not alone. Every project the client had seemed to be struggling. The depth of the issues in the application I was working were lost as other pieces of the project seemed to fly off the rails. One project team eventually admitted they had completely forgot to implement security. Other teams scrambled to offer personnel for a "security task force" to "shore up" the security. One senior level project manager joked "Well it's like we built the car and forgot to put locks on the doors 🤷‍♂️ Oopsies!"

I kept sliding more and more fixes in, trying to fix my app - to make it good even. A good result seemed so close, yet so far away. I started to feel trapped within a prison of my own making. No one could really help me either. My tech lead was still there, but seemed lost and confused. He had been marginalized and now I owned this mess. The consultancy had long ago let me know that their hands were tied. I had made them aware of the situation with the tech lead and they had confirmed the problem, but could not act without disrupting the project.

I had turned this opportunity into an exercise in hubris.

The client, realizing that there might be some issues maintaining the application after I left, assigned one of their own developers to the team. I went over the project and we laughed at it with typical gallows humor. We were now both imprisoned.

However, things started getting better. Now I had someone to bounce ideas off of. We began to conspire a way out of this mess. We were going to introduce Vue.js.

Perhaps I would not be hoisted by my own petard!

Vue.js was designed around being able to incrementally refactor. It was small and fast and we didn't need anything except the core functionality. It worked just fine alongside jQuery and we found a way to make it work with Angular.js.

We were saved!

Slowly, we started to see bugs disappear. Things were working faster and more reliably. The code started to look less like a mess and more like something we could maintain.

But the project as a whole was not getting better. Project management wanted more late nights. They wanted more sacrifices. Code for the Code God!

Escape From New Work

At last, I had enough. I'd worked on the project for about 8 months and I didn't see anything that made me want to stay.

The consultancy I worked for had promised a $500 bonus if I stayed another 7 months, but I wasn't interested in peanuts. The client had pulled me into a room with three project managers and let me know it was time to start working 12 hour days. When I refused ( I wasn't getting paid for that time after all ) the handler from the consultancy emailed me from his tropical vacation letting me know that "You will comply with the overtime request."

It was time to get out.

I did have one person who I owed an explanation though, and that was my co-conspirator developer who helped introduce Vue.js. Thing was, he completely understood - in fact he was already looking too. We were going to abandon this project and let this app fend for itself. The Vue.js refactoring had a long way to go, but neither of us could stomach the other dysfunctional pieces of the project.

I found another job that prioritized work/life balance and left. After a couple of weeks I referred my new friend and he left as well.

We ran out like the building was on fire.

Beware

That project is still there. Years later I saw wanted ads from the company. "EPiServer developer needed! JavaScript experience a must! Vue.js experience preferred!"

The refactor is still in progress. The old broken JavaScript still scraping by.

Fellow developers, the dungeon I created still exists, seeking new inmates. That JavaScript "framework", a monster that few can slay. When you're out there, scrolling through ads thinking about your next gig, remember this.

Be diligent lest you also be imprisoned!

But Hey I'm Fine

Is this me? Am I the monster?

Take A Look At ML.NET

James MacIvor — Thu, 17 Sep 2020 19:47:30 +0000

Take A Look At ML.NET

What is ML.NET?

ML.NET is a service from Microsoft that allows you to add machine learning to your existing applications. It requires very little setup and can use existing data in your database to start making predictions. If you've been curious about machine learning, but come from a .NET background then this might be a good starting point for you.

Machine learning has different types of predictions it can make based on the data you give it. Here is a list of what ML.NET can do out of the box along with a short description and an example or two.

Prediction	Description	Example
Classification/Categorization	Divides records into categories or classifications.	Can determine if a message is mean or nice.
Regression/Predict continuous values	Predicts a value associated with some data.	Could count objects in an image. Could predict house prices based on size and location.
Anomaly Detection	Detects a piece of data that does not fit with the set.	Could determine if a collection of similar images has one that is very different. Could find a transaction that differs in a major way from the rest of a set.
Recommendations	Predicts a desired addition to a set based on the set contents.	Suggests additional products a shopper might want to buy.
Time series/sequential data	Makes predictions based on the order of previous data.	Weather forecasts, stock market predictions.
Image classification	Divides images into classifications	Can determine if an image is a hotdog 😂🌭

All of these seem to be tremendously useful and ML.NET makes it easy to start using them in your existing application using your existing data.

How Do I Get Started?

You've got to have the .NET Core 3.1 SDK installed on your computer (windows, linux or wsl or macOS)

It's super easy to start using ML.NET. You can set it up locally by running this command.

dotnet tool install -g mlnet

This worked really well in wsl bash, but I had some problems running it in Powershell. According to the docs, this will work in MacOS and Linux just fine though.

So for windows you may need to use the visual studio installer, where you select the .NET Core cross-platform development option and elect for ML.NET like so:

ML.NET Is Installed, Now What?

Now we can start thinking about how we want to use it.

Here is a snippet of code from the Microsoft Explanation.

static void Main(string[] args)
{
    MLContext mlContext = new MLContext();

    // 1. Import or create training data
    HouseData[] houseData = {
        new HouseData() { Size = 1.1F, Price = 1.2F },
        new HouseData() { Size = 1.9F, Price = 2.3F },
        new HouseData() { Size = 2.8F, Price = 3.0F },
        new HouseData() { Size = 3.4F, Price = 3.7F } };
    IDataView trainingData = mlContext.Data.LoadFromEnumerable(houseData);

    // 2. Specify data preparation and model training pipeline
    var pipeline = mlContext.Transforms.Concatenate("Features", new[] { "Size" })
        .Append(mlContext.Regression.Trainers.Sdca(labelColumnName: "Price", maximumNumberOfIterations: 100));

    // 3. Train model
    var model = pipeline.Fit(trainingData);

    // 4. Make a prediction
    var size = new HouseData() { Size = 2.5F };
    var price = mlContext.Model.CreatePredictionEngine<HouseData, Prediction>(model).Predict(size);

    Console.WriteLine($"Predicted price for size: {size.Size*1000} sq ft= {price.Price*100:C}k");

    // Predicted price for size: 2500 sq ft= $261.98k
}

What Is This Code Doing?

Here we cook up some training data about house sizes, providing a size and a price.
We create a machine learning pipeline and tell it what to expect from the training data and how many times to go over it (This time it's a regression).
We take our training data and throw it into the pipeline using the Fit command.
We make up a new house size and feed it to the machine learning model and it spits out a price.

How Can We Use This?

Now, like most starting out examples, this is pretty trivial. In fact, as you read this you might be thinking "Why do I need machine learning for this? This is a very simple algorithm."

Ha, yeah, it is. However, we can continue to add additional properties to correlate, such as location. Also, real life data may not be so linear.

For me, when I saw this example I immediately started thinking about data I have worked with in various databases and how I could use that to start making predictions that would be useful for the business.

For example, I worked with a company that processed OARRS data in order to track prescription compliance for a patient. OARRS data is kept for patients who are prescribed controlled substances. Using ML.NET we could classify a patient into high or low risk categories based on prescription history. We could aggregate the prescription history for patients like so:

List of current prescriptions	Span of time for continuous prescriptions in days	Number of unique prescription fill locations

From there, we would need to pull from existing data whether or not a patient was compliant with their prescription instructions. This is something that the medical staff tracks as part of their counseling tasks. We can make a value judgement based on that data. A result of non-compliant would mean that "High Risk" and compliant would be "Low Risk". Using this, we can create a set of training data that more or less looks like this:

Current prescriptions	Continuous prescriptions in days	Unique fill locations	Risk
methadone	30	1	Low
oxycontin	100	5	High
adderall	1000	10	Low
percocet	200	2	Low

Lets say we save this aggregated anonymous data into a csv or tsv file.

Now, we can use the ML.NET command line tool to generate the code for us so we can adapt it into our application.

mlnet classification --dataset "anonymous-aggregate-prescription-data.tsv" --label-col "Risk" --train-time 60

The ML.NET tool will take the data saved in our file and attempt to predict the Risk column. We've specified that it should train on the data for 60 seconds, but really that time should be modified based on how big the dataset is. The bigger the data, the more time it should take.

Using the command line tool will also show the best machine learning algorithm to use, which is interesting and potentially helpful.

The tool will generate two projects, one will be a library that has a MLModel.zip file attached - the SampleClassification.Model project, and the other will be a console app where the model is consumed. The SampleClassification.ConsoleApp project.

Lets take a look at some code that would be spit out in the Model project. The tool will have generated two classes here, ModelInput class and a ModelOutput class. The ModelInput will be a representation of the above information sans the field to be predicted. The ModelOutput class has the field to be predicted and a float type value showing the percent certainty the algorthm made the prediction.

public class ConsumeModel
{
    private static Lazy<PredictionEngine<ModelInput, ModelOutput>> PredictionEngine = new Lazy<PredictionEngine<ModelInput, ModelOutput>>(CreatePredictionEngine);

    public static string MLNetModelPath = Directory.GetFiles(Directory.GetCurrentDirectory(), "MLModel.zip", SearchOption.AllDirectories).FirstOrDefault();

    // For more info on consuming ML.NET models, visit https://aka.ms/mlnet-consume
    // Method for consuming model in your app
    public static ModelOutput Predict(ModelInput input)
    {
        ModelOutput result = PredictionEngine.Value.Predict(input);
        return result;
    }

    public static PredictionEngine<ModelInput, ModelOutput> CreatePredictionEngine()
    {
        // Create new MLContext
        MLContext mlContext = new MLContext();

        // Load model & create prediction engine
        ITransformer mlModel = mlContext.Model.Load(MLNetModelPath, out var modelInputSchema);
        var predEngine = mlContext.Model.CreatePredictionEngine<ModelInput, ModelOutput>(mlModel);

        return predEngine;
    }
}

This is then consumed by the example console application like so:

class Program
{
    static void Main(string[] args)
    {
        // Create single instance of sample data from first line of dataset for model input
        ModelInput sampleData = new ModelInput()
        {
            Prescriptions: "methadone",
            Days: 300,
            Locations: 3
        };

        // Make a single prediction on the sample data and print results
        var predictionResult = ConsumeModel.Predict(sampleData);

        //predictionResult.Risk == "Low"
    }
}

Moreover, as we continue to explore the SampleClassification.ConsoleApp we will also see it generate a ModelBuilder.cs using the algorithm that it had selected. We can adopt this code into our application to continuously refresh the model.

public static class ModelBuilder
{
    private static string TRAIN_DATA_FILEPATH = @"/path-origina-training-data.tsv";
    private static string MODEL_FILE = ConsumeModel.MLNetModelPath;

    // Create MLContext to be shared across the model creation workflow objects 
    // Set a random seed for repeatable/deterministic results across multiple trainings.
    private static MLContext mlContext = new MLContext(seed: 1);

    public static void CreateModel()
    {
        // Load Data
        IDataView trainingDataView = mlContext.Data.LoadFromTextFile<ModelInput>(
                                        path: TRAIN_DATA_FILEPATH,
                                        hasHeader: true,
                                        separatorChar: '\t',
                                        allowQuoting: true,
                                        allowSparse: false);

        // Build training pipeline
        IEstimator<ITransformer> trainingPipeline = BuildTrainingPipeline(mlContext);

        // Train Model
        ITransformer mlModel = TrainModel(mlContext, trainingDataView, trainingPipeline);

        // Evaluate quality of Model
        Evaluate(mlContext, trainingDataView, trainingPipeline);

        // Save model
        SaveModel(mlContext, mlModel, MODEL_FILE, trainingDataView.Schema);
    }

    public static IEstimator<ITransformer> BuildTrainingPipeline(MLContext mlContext)
    {
        // Data process configuration with pipeline data transformations 
        var dataProcessPipeline = mlContext.Transforms.Conversion.MapValueToKey("Risk", "Risk")
                                    .Append(mlContext.Transforms.Text.FeaturizeText("Prescriptions", "Prescriptions") }))
                                    .Append(mlContext.Transforms.Text.FeaturizeText("Days", "Days"))
                                    .Append(mlContext.Transforms.Text.FeaturizeText("Locations", "Locations"))
                                    .Append(mlContext.Transforms.Concatenate("Features", new[] { "Prescriptions", "Days", "Locations" }))
                                    .Append(mlContext.Transforms.NormalizeMinMax("Features", "Features"))
                                    .AppendCacheCheckpoint(mlContext);
        // Set the training algorithm 
        var trainer = mlContext.MulticlassClassification.Trainers.OneVersusAll(mlContext.BinaryClassification.Trainers.AveragedPerceptron(labelColumnName: "Sentiment", numberOfIterations: 10, featureColumnName: "Features"), labelColumnName: "Risk")
                                    .Append(mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabel", "PredictedLabel"));

        var trainingPipeline = dataProcessPipeline.Append(trainer);

        return trainingPipeline;
    }

We can see where we could start pulling data from a database at the Load Data step, and start thinking about how frequently we might want to refresh this data.

And now, using this anonymous training data, the machine learning model can start making predictions about risk category for a patient based on their existing prescription history. All of this could be incorporated into an existing application by creating a machine learning service within it without any need to share potentially sensitive information with a cloud service.

For this example, I did not move forward with generating this service. While I think that making risk determinations from OARRS data could be interesting, and it's certainly possible, there are ethical concerns that would need to be considered before proceeding, such as: Do I need consent from patients to aggregate this data, even though it's anonymous? So this is all purely hypothetical based on applications I have worked on in the past.

Always be careful to consider the ethical ramifications of a machine learning project before diving into it.

Cool! How Do I Start With This?

I recommend doing the ML.NET tutorial from Microsoft. It will hit on some of the stuff covered here and will get you spun up with your own training data from wikipedia. The code examples in this blog are adapted from that tutorial.

Good luck and have fun!

Nuxt, Offline-First PWA Tutorial

James MacIvor — Mon, 14 Sep 2020 00:04:39 +0000

Nuxt: Offline-First PWA Tutorial

I Love Vue

I love Vue.js. It's a fantastic JavaScript framework - easily outclassing in my mind the other offerings - which is saying something because I think they are cool too. And yeah, they do some things better than Vue. Some of them have done things before Vue and Vue adopted it later. But even so, there are solid reasons for preferring Vue.

Just to run through them quickly:

Vue has a great learning curve.
It encourages incremental adoption into legacy projects.
Development is driven by community needs.
Vue maintainers have friendly relationships with other framework maintainers, allowing for a free exchange of ideas and concepts.

Maybe Some Things Could be Better

There are more points to make here but, let me move on to a criticism of Vue.js - depending on the type of application you intend to build, or the size of the application, scaffolding a Vue.js app, even with the CLI can be a chore.

Also, there are details around creating the Vue app that must be observed - it's not difficult, but if you intend to use Vuex or Vue router then a small amount of configuration is required before you can spin up a store or expect your routes to work.

There can be quite a bit of variation in where files are kept as well, whether there is a components folder, or how to do the layouts. Certainly the Vue CLI helps with this when it scaffolds a new project, however for Vue router for example, you'll need to continue to modify the router index.js in order to keep an accurate record of routes.

It's small changes, but something that could easily be forgotten as an application continues to grow and change. While it's true that any application, regardless of language or framework will eventually have a checklist of items that need maintained, we should try as best we can to keep the list small and automate when it makes sense.

Enter Nuxt.js.

If I'm creating a new Vue project, unless it's very small, it's rare when I'll use the Vue CLI instead of using Nuxt. Nuxt makes everything easy. Routing is already preconfigured out of the box and dynamically determined based on the directory structure. It has easily understood options for determining layout, middleware and components.

In fact, you can view the structure of this blog as an example. Here is the GitHub.

Nuxt can fill many roles. In the case of this blog, it's a git based CMS and static site generator. But I've also used it to create a single page application, providing a rich experience for the browser. It can be a universal application, providing prerendered or even server side rendered pages very easily. In fact, to create a server side rendered application instead of a single page application here is the line of configuration from nuxt.config.js:

ssr: true, // will be a spa if set to false

It's difficult to describe how Nuxt can be used without providing an example - so lets create a Nuxt application together.

Example App Requirements

First, lets get some requirements.

Let's make this a survey app. It's for fire hydrant inspectors who are required to periodically inspect fire hydrants (pretty sure this is a fake job). So it has these requirements:

Requires login of a user.
Collects data about a fire hydrant as a form. (Hydrant serial number and condition)
Must work offline (fire hydrants aren't always in proximity to cell towers or WIFI).
Must transmit fire hydrant data when connected (or reconnected).
Made to work on mobile or tablet.

Okay great. Lets make some decisions about the app's architecture.

It needs a login, so we'll use Auth0. Auth0 will allow us to integrate with other authentication systems so we can have a support for a variety of fire hydrant inspector companies.

It needs to have a form, so we'll use bootstrap to cover that requirement. There are newer, better(?), CSS frameworks available but bootstrap will give us everything we need (and a lot we don't) with very little work.

Hmm, requirements 3, 4, and 5 really point to PWA (Progressive Web Application). So we'll make this app a PWA as well.

OK. Now what? Well all of this can be done via Nuxt.

Do It In Nuxt

For authentication we could use @nuxtjs/auth. This is perfect because it has a built in integration with Auth0. But if I didn't want to use Auth0, it has built-in support for a handful of other authentication providers, or we can extend it to use any authentication provider we need.

Nuxtjs/auth has a dependency of @nuxtjs/axios an ajax library - which is perfect since we'll need that to transmit our form data anyway.

We'll also need to create this form. We selected bootstrap, so we'll use bootstrap-vue which has a handy Nuxt.js module to make all of this easy. Also, bootstrap-vue has a way to specify which bootstrap features we're using so we can use webpack (which is built into Nuxt) to treeshake out the rest. Great! It's not a perfect fix for bootstrap's drawbacks, but it's something.

Finally, we have this PWA requirement. There is a module for that too. @nuxtjs/pwa looks to have everything we need. It will handle all of the icon and manifest stuff, and allow us to easily register a service worker and determine what routes and behavior should be used when a user is offline.

Now, notice all of the things I'm not needing to specify. I don't need to call up Vue router because that's already in the mix. Vuex is also in there and it's a dependency of nuxt/auth, but we may not need it ourselves.

We create the project by running this command:

npx create-nuxt-app <project-name>

While this runs it will allow us some options to pull in some of our dependencies. PWA is an option, but an additional npm installation is still required for that. We'll also be using Axios so that could make more sense to use for now. We can also opt for bootstrap, or a bunch of other CSS frameworks.

Lets go ahead and pull in the auth module:

npm install @nuxtjs/auth

And, if you haven't already, make sure to install Axios:

npm install @nuxtjs/axios

and then we modify the nuxt.config.js to add to the modules property:

modules: [
  '@nuxtjs/axios',
  '@nuxtjs/auth'
],

auth: {
  // Options
}

The scaffolding tool also helped us to select a testing tool. I chose Jest.

Thanks for the reminder Nuxt! Testing is important. But I will not be talking about testing for this tutorial, sorry.

Finally, @nuxtjs/auth reminds us that we need to initialize the Vuex store by adding an index.js file to the store directory. Nuxt will automatically import Vuex and configure it for use when the index.js file is added. (This prevents Vuex being added to projects that don't need it.)

Next lets pull in the PWA module:

npm install @nuxtjs/pwa

And lets not forget to set up our manifest!

manifest: {
    name: 'Fire hydrant surveyor',
    short_name: 'Hydrant Surveyor',
    lang: 'en',
    display: 'standalone',
},

Nuxt PWA is actually 5 different modules in one, only one of which will need some custom code from us.

The icon module - which will configure and make available the icon for the PWA icon.
The meta module - which will set up some some common options for the mobile application.
The manifest module - which creates the manifest file with the configured values.
The workbox module - which allows setting service workers and cacheable resources - this is where we'll be doing the bulk of the work for offline behavior.
The one signal module - which injects an API which allows for creating push notifications on the mobile device.

Lets take look at what all of this looks like:

Here it is!

Just like that we have a functioning application with PWA capabilities, a CSS framework and authentication built in.

At this point we should consider our remaining tasks:

Create an Auth0 account and add the appropriate info to the nuxt config.
Build the appropriate pages with the auth0 log in and the fire hydrant survey form.
Add cacheable assets to workbox (included from PWA module).
Configure and customize the service worker to handle offline workflow.

Lets go for it. Starting with #1.

Creating an Auth0 account is super easy. You can log in using GitHub. Auth0 will automatically set up an application for you. You can then grab all of the information you need for the nuxtjs/auth module. You will need to set up a few things in Auth0 Application Settings, such as allowed origins, allowed callback URIs and similar. You can refer to Auth0 documentation on how to do that.

In nuxt.config.js you'll need to define the redirect and strategies objects. Note that the callback and login cannot be the same value. The module needs to route to a different page in order to finish processing the user data returned from the login event.

In nuxt.config.js:

auth: {
    redirect: {
      login: '/',
      callback: options.redirectUri
    },
    strategies: {
      local: false,
      auth0: {
        domain: options.domain,
        client_id: options.client_id,
      }
    }
  },

The options object is defined in a separate file: auth_config.js. I did this for my project for convenience, but for a real project I would use a .env file so I could inject the correct values for each environment via the CI/CD pipeline.

export const options = {
        domain: '...',
        client_id: '...',
        redirectUri: '/signed-in' //or whatever you configure in Auth0 Application Settings
}

While the values contained therein are not secret it is nonetheless recommended that the file not be checked into source control. You can later modify this file to have an audience property and to change the values based on build environment.

Next we'll modify the main route to contain a login button.

<div class="links">
  <button
    class="button--green"
    @click="login"
  >
    Login
  </button>
</div>

and we'll define a login method in the Vue instance.

export default {
  methods: {
    login() {
      this.$auth.loginWith('auth0')
    }
  }
}

Note: As of this writing there is an issue that requires the installation of an additional npm package.

npm install nanoid@2.1.11

Now when you test this out you should be redirected to an Auth0 login page. Upon successfully signing up or logging in you will be redirected back to the redirect_uri, which in this example project I set as http://localhost:3000.

Now, lets further modify the component template so as to display something different once we're logged in.

<div class="links">
  <b-button
    v-if="!$auth.loggedIn"
    variant="primary"
    size="lg"
    @click="login"
  >
    Login
  </b-button>
  <b-button
    v-else
    variant="warning"
    @click="logout"
    size="lg"
  >
    Logout
  </b-button>
</div>

Notice we're starting to switch to using bootstrap-vue components for the buttons. The b-button component accepts a variant and a size prop among other things.

Now lets make sure we get the script piece of the page right:

import { mapGetters } from 'vuex'
export default {
  methods: {
    login() {
      this.$auth.loginWith('auth0')
    },
    logout() {
      this.$auth.logout();
    }
  },
  computed: mapGetters(['isAuthenticated']),
}

Great! Now with these simple changes we have an application with authentication. So we've got #1 and part of #2 done now.

Here's the result in GitHub.

The remainder of #2 is to build the survey form. Lets do that real quick as a component.

Bootstrap-vue makes it all pretty easy. It encompasses the bootstrap classes as components

<template>
  <b-container fluid>
    <b-form-row>
      <b-col sm="3">
        <label for="serial-serialNumber">Hydrant Serial serialNumber</label>
      </b-col>
      <b-col sm="9">
        <b-form-input
          type="text"
          v-model="serialNumber"
          id="serial-serialNumber"
          placeholder="Enter the hydrant serial serialNumber"
        ></b-form-input>
      </b-col>
    </b-form-row>
    <b-form-row>
      <b-col sm="3">
        <label for="condition">Hydrant Condition</label>
      </b-col>
      <b-col sm="9">
        <b-form-select v-model="condition" :options="options" id="condition"></b-form-select>
      </b-col>
    </b-form-row>
    <b-form-row align-h="end">
        <b-col cols="*">
            <b-button @click="submit">Submit</b-button>
        </b-col>
    </b-form-row>
  </b-container>
</template>

Bootstrap-vue took the route of making the grid system into components. I sometimes quibble with that decision, but it is easy to use. In this template, the container (b-container) has a collection of rows (b-row or b-form-row). Each row can have up to 12 columns. Additional columns wrap to the next line. The b-col component can represent any number of columns between 1 and 12. You can then decide how many columns it should take up for each viewport size.

For example, you may want bootstrap b-col to take up 12 columns (full width) on mobile so you'd specify cols="12" as an attribute on the b-col component. But then you might decide it should take up 6 columns (half width) on tablet so you'd specify sm="6", which is also an attribute. This allows you to declare in the html how a elements should be displayed, in terms of width, on each viewport. Pretty handy!

Aside from the fairly decent grid system that bootstrap-vue abstracts for us, there are a number of utility components we can use. In this form, I only need an input, a select, and a button - so I use the bootstrap-vue version of these. Bootstrap has out of the box support for good accessibility on forms and the options on components can remind you of things - like setting a placeholder for example.

This is a very simple form - we only need a couple of things. We'll be sending the completed result back to the parent by emitting the result. Here is the script portion of the component:

export default {
  data() {
    return {
      serialNumber: "",
      condition: null,
      options: [
        { value: null, text: "Please choose a hydrant condition." },
        { value: "poor", text: "Poor" },
        { value: "fair", text: "Fair" },
        { value: "good", text: "Good" },
        { value: "excellent", text: "Excellent" },
      ],
    };
  },
  methods: {
    submit() {
      this.$emit("submit-form", { serialNumber, condition });
    },
  },
};

Now the parent component can handle the result as necessary. Lets take a look at the parent actually. You know what? Lets go ahead and refactor that to use bootstrap-vue as well.

<template>
  <b-container class="pt-5">
    <b-row align-h="center" class="mt-5">
      <b-col cols="*">
        <h1 class="title">Fire Hydrant Surveyor</h1>
      </b-col>
    </b-row>
    <b-row align-h="center" v-if="$auth.loggedIn">
      <b-col sm="9" class="my-4">
        <survey-form @submitForm="handleFormResult"></survey-form>
      </b-col>
    </b-row>
    <b-row align-h="center" class="mt-3">
      <b-col cols="*">
        <b-button v-if="!$auth.loggedIn" variant="primary" size="lg" @click="login">Login</b-button>
        <b-button v-else variant="warning" @click="logout" size="lg">Logout</b-button>
      </b-col>
    </b-row>
  </b-container>
</template>

In the parent we'll also need to import the component and define the handler:

import surveyForm from '../components/survey-form'

export default {
  components: [
    surveyForm
  ],
...
  methods: {
    ...
    async handleFormResult(formObj) {
      //do stuff
    }
  }

Here is the GitHub for this leg of our adventure

Now we need to figure out what to do with this form data. We'll be sending the data to our server, but how do we want to do that? Also we'll need to create the service worker to handle the offline behavior.

I found a nice test API someone created so I can send the form data without having to deploy a whole other project, I just needed to massage the data a little bit for it to work.

async handleFormResult(formObj) {
      //https://jsonplaceholder.typicode.com/posts is a test API I'm borrowing 
      //I'm making the data fit because I'm too lazy to make my own test API
      const post = {
        title: formObj.serialNumber,
        body: formObj.condition,
        userId: 1
      }
      try {
        const result = await this.$axios.$post('https://jsonplaceholder.typicode.com/posts', post);
        console.log(result);
      } catch(e) {
        console.log(e);
      }
    }

Okay, so now when I send the request I'm getting a result posted to the console. Everything looks good as long as I'm online.

But what about when I'm offline?

Turns out, all we need is the nuxt/pwa module where workbox will take care of it all for us.

In order to enable the behavior we want - which is resending failed requests that occur when the app is offline - we need to create a special plugin for workbox.

In the plugins folder I created a file called workbox-sync.js and added this code:

const bgSyncPlugin = new workbox.backgroundSync.BackgroundSyncPlugin('formQueue', {
    maxRetentionTime: 24 * 60 // Retry for max of 24 Hours (specified in minutes)
});

workbox.routing.registerRoute(
    /https:\/\/jsonplaceholder\.typicode\.com\/posts/,
    new workbox.strategies.NetworkOnly({
      plugins: [bgSyncPlugin]
    }),
    'POST'
  );

We are creating a new background sync plugin and then we're registering that plugin on a route. The workbox register route method takes 3 arguments, a regex for the route (so you can potentially use a regex to define the same behavior for a range of similar routes), a strategy, and the http verb.

Next you need to add this plugin to the workbox via these lines of configuration in nuxt.config.js:

  pwa: {
    ...
    workbox: {
      cachingExtensions: '@/plugins/workbox-sync.js',
      enabled: true //should be off actually per workbox docs due to complications when used in prod
    }
  }

Note that you cannot do imports in the workbox-sync.js file. The reason is that the plugin is injected into the middle of the sw.js script that the workbox module creates for us. Imports cannot be performed in the middle of a script.

Also, you'll have noticed that I have enabled: true but per the @nuxtjs/pwa workbox documentation you should not typically do this since it can cause issues when you are switching between dev and production environments. I do it here because it's so convenient. Otherwise, the service worker is not enabled while running in dev.

Now when I run npm run dev workbox will create the service worker. If I toggle the application to offline via the browser devtools then the posts to the server will fail, but as soon as I switch back to online then the service worker re-sends the requests.

Lets see this in action.

Here we are sending a successful request.

But lets change our status to offline from the dev tools and watch a request fail.

Now the service worker takes care of re-sending the request. It will use the same information as what was originally sent, so beware if you are using any sort of expiring authentication data. But if we're reasonable sure the authentication token will remain valid until our users can get back online then this solution will work great.

If you cannot count on that, then you may need to go with an alternate solution where you keep the data upon a failed request in an indexdb using localforage. Then you would need to create a custom service worker that would need to determine if you are back online and resend the data using the most recent authentication credentials available.

For the final result of our code lets take a look here.

We met all of our requirements with very little custom code.

Now for deployment, Auth0 needs configured to accept my production domain name.

Also, we need to go ahead and refactor out auth.config.js and replace it with .env.

I went ahead and did that, first installing cross-env

npm install cross-env

Then I created .env file and populated it like so:

DOMAIN='...auth0.com'
CLIENTID='aHashFromAuth0'
REDIRECTURI='/signed-in'

I then removed the import of auth.config from nuxt.config and replaced the options with the following:

auth: {
    redirect: {
      login: '/',
      callback: process.env.REDIRECTURI
    },
    strategies: {
      local: false,
      auth0: {
        domain: process.env.DOMAIN,
        client_id: process.env.CLIENTID,
      }
    }
  },

Now I can inject the variables via the CI/CD pipeline.

And here is the final result.

Nuxt has a huge variety of libraries and plugins that can help you achieve what you're seeking to do. It's great to get something off the ground quickly so you can hone in on the business needs.

I hope this is helpful for anyone doing an offline first application!

The Cure For Impostor Syndrome

James MacIvor — Tue, 01 Sep 2020 14:34:39 +0000

We Suffer From Our Comparisons

The Four Sights, a depiction of the suffering witnessed by Buddha

Every single person I know has had it.
It can keep you awake at night.
It makes you timid when you should be bold.
It's the voice that convinces you to take the easy path.
- Don't take risks.
- Don't challenge the senior developer.
- Don't go for that promotion.

Of course, I'm talking about Impostor Syndrome.

What Even Is Impostor Syndrome?

Impostor Syndrome runs rampant in our industry, preying on our fragile mental health. It's likely most frequently seen with junior developers, but I've seen it with mid levels and senior developers as well.

And, in a good team, it can be somewhat mitigated. There is a support structure of people who will reassure you that you're doing a good job. Or will make themselves a little bit vulnerable to show you that you're not alone. This is good healthy communication.

But not all teams are like that. In fact, that might be a bit of an exception. Too often, there are teams where people feel no obligation to reassure you, or worse they compensate for their own feelings of inadequacy by exacerbating yours.

So, the working definition for Impostor Syndrome is this (from Wikipedia):

Impostor Syndrome is a psychological pattern in which an individual doubts their accomplishments and has a persistent internalized fear of being exposed as a "fraud".

And here is the rub - the root of Impostor Syndrome is disbelief in your own success. How does that happen? You're accomplishments are right there in the ol' resume. You can track through a long list of completed user stories. You can see old PRs of your contributions - and yet no matter what evidence is presented this feeling persists.

This is a defining symptom of Impostor Syndrome. the psychological journal verywellmind has an article that has the following bulleted list of characteristics:

Self-doubt
An inability to realistically assess your competence and skills
Attributing your success to external factors
Berating your performance
Fear that you won't live up to expectations
Overachieving
Sabotaging your own success
Setting very challenging goals and feeling disappointed when you fall short

Do any of these sound familiar? The fact is that we all experience some of this throughout our careers, but composed together we start to feel as though we are frauds.

What Causes It?

It comes from comparing yourself to others, not how they are but how you imagine them to be. There are so many tremendously smart people in tech, it's tempting to see them as giants. Like they, through 100% of their own merit, were given opportunities to work on new technologies and present themselves to the world. Even though many of them would be the first to deny this alt-version of themselves! People like Scott Hanselman are quick to remind others that he once stood where you are. Moreover, Dan Abramov directly addresses reality vs. hype for developer expertise in his blog post.

More common, but less obvious, is when you compare yourself to people whom you think you should have parity and are shocked when they have deep knowledge of something unfamiliar to you. This happens to all of us. How quick we are to scold ourselves for not making the same decisions, studying the same things, making the same sacrifices. We're not fair to ourselves.

And yet, it's useless to talk ourselves out of these feelings. It's a rare day when I can negotiate with my feelings regardless! I just hide the negative ones, same as you I expect.

In the Time article "How to Deal With Impostor Syndrome" Psychologist Audrey Ervine says

There’s no single answer. Some experts believe it has to do with personality traits—like anxiety or neuroticism—while others focus on family or behavioral causes, Sometimes childhood memories, such as feeling that your grades were never good enough for your parents or that your siblings outshone you in certain areas, can leave a lasting impact.

The verywellmind article further explains the causes of Impostor Syndrome.

We also know that entering a new role can trigger impostor syndrome. For example, starting college or university might leave you feeling as though you don't belong and are not capable.

Moreover, we learn that Social Anxiety Disorder and Impostor Syndrome overlap, which makes a lot of sense when you consider that Impostor Syndrome results in a great deal of anxiety regarding how we're perceived by others.

Wait, You Promised A Cure?

Yes, I did.

I'm sorry to say, but the "cure" takes a while. It comes in the form of constant affirmation. And, if you believe that you're experiencing this in conjunction with Social Anxiety Disorder, then additional steps might be needed.

The first thing you can do that really helps is talk to someone. This is why being on a team that offers that kind of support is important. If you feel safe talking to someone on your team, then much of this can be dispelled. Don't try to solo it! It's important to be honest with yourself and others.

Another thing you can do is to focus on other people who might be going through the same thing. I achieved this by becoming much more serious about mentoring. Teaching people how to code and mentoring junior developers helps me see how far I've come. At the same time, I can see where they might feel insecure and it's a good opportunity to listen and reassure them.

I strongly recommend if you're a junior developer - try teaching a curious friend the basics of coding. If your a mid level or senior then commit yourself to mentoring a junior developer. It can be a great reminder of the learning and leaps you've made since you began this journey when someone else is looking up to you. This is a good way to constructively assess your abilities.

I also recommend seeking out those stronger developers as mentors for yourself. Put yourself in a position to listen to them rather than compete with them. It's a joy to work with super competent people who care about their work and their team.

We are constantly comparing ourselves to others. It can't be helped! It's how our brains are wired, but if that's all you do then you'll fail to see your own progress. Even better though, as you continue to teach and mentor you'll see the people you boosted begin to surpass you. This is a good thing! This means you were an EXCELLENT mentor. Be proud. It's also great to have a mentor rooting for you, invested in your success - and ready to dispel any notion that you're a fraud.

It's natural and right to use mentoring and teaching to improve ourselves and the people around us. It breaks down those social walls that hold us apart and creates a community.

Like with anything of this nature, there are no magic pills to make those feelings of inadequacy go away. We've got to do the work of finding our place and accepting ourselves as we are.