Forem: Bernd Wechner

Hero to Zero: How Not To Manage Staff Redundancies

Bernd Wechner — Sat, 21 Dec 2024 07:57:11 +0000

Are you suffering a downturn in business? Need to downsize? Well, lend me an ear, and I'll tell you a tale, a tale of what not to do ... with a hint on what you can do.

Redundancies are a fact of life. It's a pretty word used as a euphemism if you will, for sacking or firing, which is, by intent, impersonal. This has nothing to do with the staff member concerned, it's not related to their performance in any way shape or form, but money is short and hard decisions need to be made. Some jobs are declared "redundant". Which may, on occasion, be true, and yet it will also be used, as often as not, to simply describe the fact of necessary cost reduction – for which unaffordable is a better word than redundant (and arguably more flattering to the staff thus affected).

I've witnessed quite a few redundancies and been made redundant no less than thrice. The biggest farce I recall was a wave of what we called voluntary redundancies in the 1980s. It swept the steel industry at the time only to find that, well, the people who opted voluntarily to move on (a voluntary redundancy) were those that were confident of finding one jobs, the most employable people. The ones who stayed were those least confident that could find another job, the least employable. It was unilaterally considered a bad move after the fact and after the first wave, it became voluntary redundancy on approval!

But what I'm here for is to recount the tale of a redundancy that epitomises to my mind what not to do, to your colleagues and fellow human being, an experience so unnecessarily dehumanising and hostile it defies comprehension.

Whence it came

To be clear, I recount the tail of my own redundancy. But to understand it, we need to understand its context, and from whence it stems.

A small to medium business, that in the 22 years I worked there in its various names and guises, wavered between about 10 and 30 staff members at a guess. I was hired, on the back of a few years working in the field of training course design and delivery and management system installations across a global customer base, to design and deliver a training course for this business in a field of science new to me, that had been sold for delivery in 12 weeks but didn't actually exist yet. Always up for a challenge, I took home a text book from which I read cover to cover (the bible of this science) and for my first day, met my new boss at the airport, for a trip to Malaysia where he was presenting the product and felt this would be as good an intro as any.

Back at home base later, I took on front line customer support for their software product, and writing all the version 3 help file updates as part of the upskilling I had to do, while working on a three-day training course in the use of this software and the basic principles of the science that apply.

The course was a hit. Delivered in Seattle and Anchorage with warm and positive feedback. And I went on to deliver it in various other places before ... a child was born.

At that point, I wanted to reduce my travel (I'd been on the road for most of a decade by then and for two years prior to taking this job, in aeroplanes every weekend to deliver a training course, and install systems across the world and had enough and wanted to be there for my daughter. So I did a job swap in house, with a guy we'd hired into Quality Assurance, and so began my sojourn there.

In time, I dropped to a 4-day week wanting to be a present father in the life of my daughter, and we grew our business to purchase another, and then start a new blue sky division that would produce a new product into a new market to diversify our interests and protect the business against the vagaries on one product in one market. Along the way, I progressed to managing the Customer Support, Quality Assurance and Documentation needs across three business teams with teams of about a dozen individual support staff, testers and writers.

To fund this blue-sky product, we courted venture capitalists, and so had a glorious time of growth. Alas, internal conflicts and mismanagement also ensured we were running out of money and time and nowhere near an MVP (Minimum Viable Product) and I moved into project management tasked with delivery of a v1 MVP before we ran dry.

I worked on estimation and scheduling methods and after a false start, we got a v1 MVP out on the committed date with the committed feature set. Followed that with a v1.1 service release 3 months later on time, on budget.

But sales were slow, too slow, and we ran out of money. The team was folded down to minimal size, a programmer, a tester, a salesman, and I was snapped up into the sales team on our cash-cow product. My first redundancy. No paperwork, nothing formal, all verbal agreements and org chart alterations.

I started by working no necessary upgrades to an in-house license management system (C#, ASP .NET, and MS-SQL work) and direct sales in the Asia Pacific region. One surprise after another, not long after I was managing global sales and marketing for the product and a team of sales staff and resellers.

At which point more surprises, we lost our CEO and held internal interviews for someone to fill those shoes. I felt fully capable of the task, but nevertheless compelled to recommend a colleague who I thought would do a better job of it (besides, I wasn't really sure what a CEO does all day, and didn't much savour the idea). So my colleague took the job and I managed Sales under him.

I pushed for significant reforms to sales and marketing (pricing reforms, reseller network growth, marketing and growth of passive sales - we had grown to this point on direct sales, rebranding with new brochures and conference and trade show bling and uniforms for our staff). I secured only some of that before my colleague decided he didn't really know what a CEO did all day either or, better said, and more honestly, decided that it wasn't for him, resigned and became a school teacher (which I dare say did not include a pay rise - which to my mind is a touching measure of what it means to be doing something you love - he's loving teaching to this day, and was a pretty good CEO too I should add).

A new CEO on deck (again with my recommendation and support) I did ultimately execute all those reforms and then turned to focus on the business relations as the global partner manager, growing our internation reseller network and empowering them to sell more rapidly, and courting the hardware suppliers we sold on the back of for OEM style deals.

In time though I pushed that as far as it would go readily, and I itched once more for creative work on the development cycle and negotiated with my CEO for an opportunity back in the development team. Out of the blue one-day I was called in to a meeting, and served a redundancy notice, with an offer to take a job in Quality Assurance on the development team.

My second redundancy with an offer in the development team as I'd sought. No-brainer really. Not my first and top preference, but as it happens Quality Assurance was one area in the team that felt could use some attention, some work on automation and rigour in particular and I had a lot to offer there. I had learned Python (and Matlab) on my tenure in Sales (as our customer base was using Matlab, Python and R and I hadn't got around to R yet) and could see great opportunities to apply that in this area. I fed back all the same to my CEO that I did not like this style one jot.

I would much rather be called into a meeting to discuss the plan of moving to quality assurance, and have the business needs and priorities on the table before the deal is finalised before it is a fait accompli and a formal redundancy with what can only be seen as a rather impersonal and concerning - decision made behind my back. At this point I was reporting directly to the CEO - a position I was almost offered at one point - and had been working closely with him for 5 or more years and had what I felt was a personable and friendly relationship. Such an unexpected, surprise delivery of dry formal decision made behind my back was most unwelcome in that context.

I did however, appreciate action on the front of my efforts to negotiate a return to the development team and creative expression and the offer of a place in that team and also credited them with solid efforts to keep the remunerative impacts modest and minimal. All the same I had two young sons now and found that on a 30 hour week that I was rushing everywhere all the time so I asked if as part of this transition I could drop to 25 hours a week, 5 x 5 hour workdays. That was granted.

On a personal front, 5 hour days are the thing! With two young sons in early primary school I was able to walk them to school and a good few afternoons collect them and take them to various sporting activities. I could be, what my father never was for me, present, there, in their lives and they in mine. And there is no greater gift.

I worked five odd years on the quality assurance team, cutting a niche there for the heavy mathematical work (wideband signal analyses among other things - I am still a telecoms engineer deep down I guess) and data science work (file formats and such) and I made huge strides in rigour and automation for the team.

Not without hiccups. On account of laudable diversity in our team and my re-entry as a peer after a decade of absence and some new staff and some old staff, my efforts to modernise were not immediately met with open arms. We'd adopted Teams over COVID and on a channel for sharing innovations I'd announced an early one I was excited about a massive time saver and rigour-boost in regression test analyses, and was publicly dissed by a colleague. That escalated quickly ... but a disciplinary meeting aside (on the subject of dissing colleagues in public, where a private communication would serve to resolve any misunderstanding or disagreements before going public) my tail between my legs and head tucked in, in-house public announcements were avoided and progress continued admirably in consultation with the QA lead and other development staff as needed.

I had just got this to the point of good enough, and a desire to present new tools and methods first to a select group of receptive folk and then with some luck to the whole team. Tools for increasing the rigour and efficiency of our QA efforts and first moves into full automation of our UI testing (big job to do well and would need full team buy-in).

December was approaching, and our half yearly reviews and I was going to table the possibility of a roll out and presentation, to sound that out, I felt the time was right, work sufficiently mature and the team culturally ready.

My goal was to present this and seek an exit from the business at some point. I had done my run on quality assurance here now and was keen for a change, a new adventure. I had been looking on and off for other opportunities for years and my poor wife had always to remind me that I had a job most people dream of. I had 5-hour work days on a good rate, could come and go mostly as I pleased and work fairly autonomously. What's not to love?

Well for me, there the challenge. I had more to do on this job but it was definitely exiting the innovation, and proof of concept phase, well into evolution and now needed commitment and hard yards and well it's not that I wouldn't be up for that, but it'd been 5 years on this and well, that's been more or less my limit to date.

My CV is littered with me getting antsy after 3-5 years and looking for something new to do, new stuff to learn and master. New problems to solve. Things fix, build create. And I didn't want another 5 years on this really, it was enough. But she (my wife) had a point, I had two young sons a mortgage, and well, financial security held great appeal as did this wonderful flexibility to be part of my boy's lives. So I never acted on it, just flirted with the idea and dropped the odd EOI (expression of interest - a term I use here for something far more modest than a job application).

Redundant again

(sung to the tune of Sick Again)

Unbeknownst to me, the perfect storm was brewing ...

It was late spring, the days were long and on Wednesday afternoon I was landscaping the front garden at home and had pulled a back muscle in the process, doh! Woke Thursday stiff and sore. Been here before, and not a fan, and know to be careful (I remmeber the time it took two morphine shots before they could pick me upoff the floor one time I tripped this back of mine, not good, not keen on a repeat).

Still, I also know that gentle movement is better than not, so I did ride my bike into the office (late, because I was working the afternoon shift anyhow on account an egagement in town that night) - I commute on a bicycle BTW. I put in a days work (at my standing desk). Leaving the office late-ish, and just putting my helmet on by the front door, still doubled over a little with stiff sore back and wonderig how wise it was to ride in, my CEO pases me on the way to his car and thrusts a letter into my hand (I bet you know where this going ...).

I started opening and he said "no need to open it now" and continued to the car. I did open it now. A redundancy notice. Hmmmm.

Really didn't take my feedback to heart last time it seems. But this time was going to be a new level of twisted ...

It announced a meeting on Monday Nov 11. A meeting, obligated by law, in which I was to have an opportunity to suggest re-deployment options to consider. The letter was dry and impersonal, pointed out clearly that this was a business decision to do with my role not performance related, or personal in any way.

A redacted copy of the letter I received
Redacted so as to protect all third parties from any implied criticism or otherwise. A mere matter of record so you can see what a formulaic redundancy letter reads like, bearing in mind this comes out of the blue with no forewarning and is thrust into your hand by a colleague (the CEO you've worked closely with for years).

Friday morning my back was worse so I on the floor with phone in hand. Went to write my team on teams that I'd be taking the day off with abad back and found I was locked out of Teams, and Outlook. Texted my CEO:

I notice that on my phone Teams and Outlook logged out and I can't log on. Related? Or poorly timed coincidence?

he replied:

Yes. Related. It's normal procedure to stop access in the period between initial notice and final determination. You are given the time off. We can discuss further on Monday.

I lost a lot of sleep that weekend. Here I was 22 years in the business a decade working with this CEO, in a busines of about 20 people, who all new each other well, mostly as friends, and socilised together. I had worn almost every hat one can wear in the business. And I'm treated suddenly like a threat. A danger. A risk.

Normal procedure? For ASIO? the CIA? a Military base? An Embassy? But us? Seriously? Or is this a case of tin-pot depotism gone wild, the little master of little kingdom? THis is not "normal" anywhere that I have worked ever and I've worked in some big places. I've worked in a national telcom, in a national steel producer, and nowhere have I ever been where my boss would not have bent the normal bureacratic procedure (which is far more rigid in national enterprises than a 20 strong family of IT workers) as far as they could to make the experience friendly and human. Not a single one, ever.

Heck, I kid ye not when I say, I would afford my enemies that luxury if I were in that position (in fact I have let folk go and I have been in that position and I know how to do it - oh and I do have enemies - in the words of Victor Hugo "You have enemies? Why, it is the story of every man who has done a great deed or created a new idea."). To wit, the sleep I lost that weekend was not to sorrow, nor pity, nor even anger, but to WTF (is there an emotion for that? ChatGTP thinks dumbfounded astonishment, baffled exasperation or stunned confusion, so no, no simple emotion and I think WTF has brevity benefits).

I get an invite to Monday's meeting on Teams. Ironic, given I have no access to Teams. Described with "This meeting scheduled as per my letter. Please let me know if you’d prefer in-person.". I replied "I'd rather come in thanks." and did. Whatever goes down here can be done looking me in the eye, not over a camera or worse a voice link. Just how many steps can we take to dehumanise this I wondered?

As per the letter this meeting was for them to provide the full context of the redundancy and to afford me the opportunity to propose a redeployment.

The reason was amusing. And no surprise. I might even have predicted it. But still the tight adherence to minimal legal obligations here and the language twists that demands amuse me, I admit.

To put that into context there is no way that a QA role is redundant in this business. There are 4 other staff employed on QA. As it happens though because that blue-sky product failed and our venture capitalists basically lost their money, they wanted to grab the money and run - time to sell the business while the cash cow was earning.

We started this process at the time that I moved into Sales and in fact I was on the team driving it and gave the project its name, Bacchus (a pun on "back us" and the Roman god of revelry and wine). But in all the years since, we had come close once or twice but never clinched a sale.

Since I managed sales we'd been on a rising sales trajectory but it was levelling out (also predicted) but because expenditures tracked the very questionable year on year 20% growth targets in what is a very niche market sector, 2023 was apparently the first year we ran at a loss, cash negative. We could afford it, had cash in the bank, but nevertheless we ran at a loss. We did have a record quarter following that too, but then these are the vagaries of a small-niche global market (we're quite sensitive to exhange rates too trading globally often in local currencies).

Anyhow, a sale was looking good when we were growing and after last year we need a) to cut costs and b) to look at how we can be more "saleable". And in the back room, the decision was made that a part time QA role did not align with that need. Another part-timer was made renundant too (in the admin area). So the irony is, I'm redundant because of project Bacchus that I was part of from the outset (albeit no more, since I moved out of upper management into QA I was no longer involved).

The opportunity to propose redeployment made me laugh. And I shared as much in the meeting. I mean they know, and I know, that I know how the business runs. I know the nature of the business. And I know as well as they know that I'm not genuinely redundant, I'm just not afforable (they can probably hire a graduate full time into a full time QA role for what they pay me).

And they know too, that without the books on the table, recent years financial reports, and the coming years credible projections (with a view on the sales pipeline - our cash cow has a 6-18 month sales pipeline is a budgeted-for product not a discretionary expense) and above all, minutes from recent board meetings.

All of which I'd want, nay need, to make a credible and serious propsal for a redeployment. I reckon I'd find one, no guarantee they'd take it but to pitch one I'd need to do the business analysis which after 5 years in QA I was a little blind to (mushrooms as they say, kept in the dark and fed ...). So no, I had no capability to pitch a credible redeployment opportunity, thanks for invite though.

The redundancy includes a mandated 5 weeks of notice. But they are happy to pay that out, I don't have to work it if I don't want to. But if I want to have some time for a handover it's available. Again, I look at these guys (two of them in the meeting I waived my right and invite for a support person) with a cocked eyebrow. You're saying you'll pay me regardless and I can come in and work here or spend my time looking for another job? Um is that a real question? I told them I could use a day maybe two to document all I'd done and have some hand over meetings and/or presentations and I would like to do that.

I told them again, that it would have been trivial to have a meeting before this letter, as colleagues and friends. We could have come to an agreement we could have timed it better, I'm fine with moving on, I have no need for redeployment, but here we are on my birthday discussing my dismissal, a mere week after a positive RSVP to the staff Christmas party. Merry Christmas Scrooge.

They recoiled a little "It's you're Birthday!?". "Nah, it's tomorrow". They relax visible. I retort "That's all right then I guess". I let them know, here I am 22 years in, much of it in senior management, a sharehlder of the business, and you shove a letter into my hand and lock me out of all the systems, like I'm a spy, thief or vandal or emotionally unstable. "It's not personal, just the 'process'".

It's not personal? Catch-22 my friend. The fact that in a workplace of under two dozen colleagues, with a guy you've worked with for a decade, who recommended you to the CEO position, who's been here 22 years and calls this place family, your precious "process" can't flex a little to make it personal. That is personal. That you couldn't be bothered making this process more personal for me.In refusing to make it personaly, you have made it personal.

The CEO made it clear that he takes his legal obligations very seriously. I thanked him for that. It is appreciated. All the same it seems minimal observance and not one jot more is what he's keen on here.

A redacted copy of the letter I received to summarise the followup meeting
Redacted so as to protect all third parties from any implied criticism or otherwise. A mere matter of record so you can see how the story closed from the business side.

I had to take a week off for this bad back anyhow, but then come in and clear my desk. I asked if I could come in on the Monday after and my CEO wrote:

Under the circumstances, and given your views about the redundancy process and our workplace (dehumanising, hostile, treated like a statistic), I do not believe that it’s appropriate for you to attend the premises during normal work hours.

Agan, WTF? He read the feedback on his "process" right, but it has actually nothing to do with the workplace or anyone in it, including him. Let's be clear, this is a lovely man, and a great colleague, a wonderful salesman and CEO. We simply differ on the relative priorties of "process" and "humanity" in a small business where everyone knows one another, we're all family. But shrug. Evidence in hand, dehumanising and hostile. So I had to come in out of hours.

I also asked about coming in for a day to document my assets and present some materials as a handover - given I'm technically employed for 5 weeks (the notice period). His reply:

To re-confirm, your last day of employment here was Monday 11 November and you have been fully paid in lieu of notice period. You are not “technically employed” ...

How to execute a redundancy

As per my CEO's express intent, abide by the law. But the law, as a rule is there to protect employees from abuse (in Australia it is and we're governed under the Professional Employees Award 2020 for example), and don't use it as a guide to limit what you do to make this process human. The law tables the minimal requirements it is not a guidebook.

Remember you are dealing with people. And you are not a monster. Don't let anyone convince you of it and don't let them tell you the "process" must be followed.

Rule 1: Meet first. Never deliver a letter first. Talk first. Let them know what's up in person. Not like some legal notice. We're not in court. We work together.

Rule 2: Because you followed Rule 1 and met first, never treat them like a threat, or a danger, like someone who's going to steal from you or vandalise you, unless you assess a genuine risk of just that. Which is one more reason to follow Rule 1, to assess the risk of and emotionally unstable repsonse. Never suppose someone is a risk, never mind someone you know very well isn't, someone who is a member of the family, a colleague, friend, shareholder, invested in the business.

Really it's not that hard, two basic rules to keep the process human.

Yes, if you're kicking someone out of the CIA maybe make it clean and fast. But if you're a private business no matter how large, try, for the love of all you respect, to keep it human, two simple rules (consult them first, and don't treat them like a threat unless they are) to keep it human. In a huge business you may need to stand for that. As the CEO you have zero excuse and only a lack of caring and desire or, of course, that saving dodge: "this hurts me more than it hurts you, but unfortunately my hands are tied, this is the 'process' and I wish it were better" ... doesn't (dodge the responsibility that is). You take the wrap.

Disclaimer

I have opted to use a very personal experience to illustrate two simple rules to follow that can make redundancy a human experience and not a hostile one. In so doing it needs to be stated there are no ill feelings and now judgements surrounding the matter. In sharing a personal story there is the risk of inadvertently embarrassing someone and to avoid that I have carefully removed all identifying materials and references as best I can. All the people in this story are wonderful people, doing the best vthey can with the skills, knowledge, tools and pressures they have and face. The story attempts to capture the sense of surprise and wow, surrounding the dehumanisation of a close colleague and attempts only to pinpoint the key difference in style or opinion that exists between key characters in that story. Both are presented:

The view that "process" exists to be followed, and that laws should be adhered to minimally.
The view that "people" matter more than process and that laws do no limit how nice and human you can be, they only set minimal standards.

Availability

Well, given the nature of the story told, I am of course as at December 2024 available for exciting and interesting work opportunities again. Though I am enjoying a holiday too ;-).

Attribution

This a people story, and not a particularly short one, so I wanted some nice people pictures to break up the prose a bit. I found those on Freepik which has a good selection of free images that can be used with attribution (a limit of 3 downloads a day anonymously, and 10 per day for an account).

Dumping Data with Python's CSV DictWriter

Bernd Wechner — Tue, 30 May 2023 08:34:33 +0000

I really love Python's csv module. But I do wish it was a little better documented.

The DictWriter lets you write CSV files very neatly and semantically by defining each row as a Python dict.

Here's the example right out of the docs:

import csv

with open('names.csv', 'w', newline='') as csvfile:
    fieldnames = ['first_name', 'last_name']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)

    writer.writeheader()
    writer.writerow({'first_name': 'Baked', 'last_name': 'Beans'})
    writer.writerow({'first_name': 'Lovely', 'last_name': 'Spam'})
    writer.writerow({'first_name': 'Wonderful', 'last_name': 'Spam'})

Alas, it overlooks what I would consider a (if not the) standard use case, the one I come across all the time, in which I don't write the rows out one by one literally, but in a loop, more like:

import csv

data = [('Baked', 'Beans'),
        ('Lovely', 'Spam'),
        ('Wonderful', 'Spam'),
       ]

with open('names.csv', 'w', newline='') as csvfile:
    fieldnames = ['first_name', 'last_name']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)

    writer.writeheader()
    for datum in data:
        writer.writerow({'first_name': datum[0], 
                         'last_name': datum[1]})

Of course:

The source won't generally be a list of tuples, rather an iterator of objects more generally, and I'm dumping some of their properties.
I will be working with a much longer list of fieldnames.

which, raises the spectre of repeating the same list of field names, most definitely not DRY, and highly undesirable and difficult to maintain as I tweak the list of fields I want to dump to CSV.

And yet I find no documentation that gets around that. So, having just nutted one out and tested it, it's worth putting down in a document (right here and now).

The Problem

The problem in a nutshell is that csv.DictWriter demands to know the fieldnames (and writer.writeheader() needs to have them known), but they are specified in the dictionary build inside the loop. fieldnames is not even an optional argument to csv.DictWriter and the writer is poorly documented.

The Solution

The solution rests in two empirically determined (for lack of documentation) facts:

csv.DictWriter accepts fieldnames=None
the writer if returns has a fieldnames attribute that can be set post creation.

To wit, this works beautifully:

import csv

data = [('Baked', 'Beans'),
        ('Lovely', 'Spam'),
        ('Wonderful', 'Spam'),
       ]

with open('names.csv', 'w', newline='') as csvfile:
    writer = csv.DictWriter(csvfile, fieldnames=None)

    for i, datum in enumerate(data):
        row = {'first_name': datum[0], 
               'last_name': datum[1]}

        if i == 0:
             writer.fieldnames = row.keys()
             writer.writeheader()

        writer.writerow(row)

And now the row keys are not doubly specified. And the CSV file receives its header row.

A simple paradigm once discovered, I am using for rapidly dumping CSV files describing objects, mostly for testing and study purposes. It means I can play with the row definition in situ, adding fields and changing fields etc, without having to change them in two places and I get the benefit of a DictWriter and its simple syntax for writing CSV files.

Selecting Things in All of the Nominated Categories

Bernd Wechner — Thu, 11 May 2023 08:44:00 +0000

I came across a problem this week, that I was sure, had a canonical solution (i.e. standard and documented). But no, not that I could find. So yet again a process of reading, searching, testing, trying, talking (on Stack Overflow, with ChatGPT) and resulting notes and even a sandbox (demo).

Problem Statement

The context: A database (nominally behind a website) containing Things, and those Things are in Categories.

The environment: Django as a web framework and Python as a backend, but raw SQL will do if needed.

In the Django context the simplest scenario is:

from django.db import models

class Thing(models.Model):
    name = models.CharField('Name of the Thing', max_length=100)

class Category(models.Model):
    name = models.CharField('Name of the Category', max_length=100)
    things = models.ManyToManyField(Thing,
                                    verbose_name='Things',
                                    related_name='categories')

Which of course is backed by three tables in a (nominally, Postgresql, though I did trials in SQLite) database (in pro forma - Django will actually use slightly different names depending on context for such tables):

thing which has one thing per row
category - which has one category per row
thing_category - which has a (thing id, category id) row for each relationship between things and categories.

You might prefer to think of things and categories in some other way more familiar to you (anything at all, with a many-to-many relationship suffices), perhaps students and courses, papers and authors, people and interests, food and its properties ... whatever really, I have several applications so asked this question in the general terms of things and categories.

The task: Given a list of categories, say [1, 4, 7, 23], write one query which returns all the things that are in all of these categories (but not limited to them, i.e. can be in other categories as well).

In turns out that the partner problem, finding all the things that are in any of those categories is easy. In Django:

things = Thing.objects.filter(categories__in=[1, 4, 7, 23])

although...

this returns duplicate things, a wonderfully bizarre feature of Django, and to prevent that one needs to add .distinct()

This, in effect, is an ANY/OR result, returning all things that are in any of the listed categories, alternately, all things that are in category 1 or category 4 or category 7 or category 23.

What I need is an ALL/AND result, returning all things that in all of the listed categories, alternately, all things that are category 1 and category 4 and category 7 and category 23.

Solutions

TLDR: solution

cats = [1, 4, 7, 23]
things = Thing.objects.annotate(
            cat_count=Count('id', filter=Q(categories__in=cats))
         ).filter(cat_count=len(cats))

Field Lookups

categories__in is what Django calls a field lookup. They are very handy and the first port-of-call was to try and find a field lookup that does this. Alas none was to be found. The __in lookup is used to perform an ANY/OR match. There is no equivalent for ALL/AND lookups like say __in_all - alas. It's possible to write custom lookups and I have done that before, and was tempted here, but the SQL solution is non-trivial to implement in a custom lookup (can be done, but the method is poorly documented and would need more research).

Turns out the SQL should I (or someone else) choose to build a lookup like __in_all can work on the joining table alone, so with the tables and example above:

SELECT thing_id
FROM thing_category
WHERE category_id in (1, 4, 7, 23)
GROUP BY thing_id
HAVING Count(*) = 4;

This works well because the WHERE clause excludes all other categories, so only those in the list will be counted by the Count(*) function. To wit, this fetches all the relations that are (at least) in the listed categories (they may be in more). Precisely what I am targeting. Of course, this assumes there are no duplicate rows in the table (a safe assumption with Django joining tables).

Interestingly, the __in lookup which returns the ANY/OR set of things can be written in similar SQL as:

SELECT thing_id
FROM thing_category
WHERE category_id in (1, 4, 7, 23)
GROUP BY thing_id
HAVING Count(*) >= 1;

In lieu of a custom lookup or raw SQL query though, a better Django solution is preferred.

Filtered Counts

Django filters are often a challenge to write. It is both a wonderful abstraction of the underlying database and, at the same time, a very different approach and way of thinking. There's always this small challenge with Django, if you have SQL you know works, translating that to Django filter.

It turns out the Django Count aggregate function has a (poorly documented) filter argument.

This:

things = Thing.objects.annotate(cat_count=Count('categories'))

is fairly standard Django (that many will recognise), which adds a count of the categories as an attribute named cat_count to each of returned things.

But we don't want a count of the categories every thing is in, we want a count of the categories from our list that each thing is in. It turns out Django supports that through the rather poorly documented filter addition to the Count aggregate.

So we can limit the count to that list of desire categories using a Count filter, and that does use a field lookup! As in:

cats = [1, 4, 7, 23]
things = Thing.objects.annotate(cat_count=Count('categories', filter=Q(categories__in=cats)))

And now we have all things annotated with a count of categories from 0 to 4 (the length of cats) as the Count was only on a filtered set of them.

To get all things that are in ALL categories we can filter for those that have 4 reported, so:

cats = [1, 4, 7, 23]
things = Thing.objects.annotate(cat_count=Count('categories', filter=Q(categories__in=cats)))
things = things.filter(cat_count=len(cats))

Revisiting ANY/OR (a digression)

Ironically, we of course replicate what the __in lookup does using this annotation too, so that this:

cats = [1, 4, 7, 23]
things = Thing.objects.filter(categories__in=cats)

is almost the same (as in, produces the same results as):

cats = [1, 4, 7, 23]
things = Thing.objects.annotate(cat_count=Count('categories', filter=Q(categories__in=cats)))
things = things.filter(cat_count__gte=1)

Which uses the __gte field lookup. Alas, the first cut, duplicates things, Django does a lazy join and then filter on the join table, so things appear once per matching category. Doh! Like that would be of use to anyone, ever, in any context? So We have to help Django do the sensible thing and explicitly request distinct things:

cats = [1, 4, 7, 23]
things = Thing.objects.filter(categories__in=cats).distinct()

And now the two queries return near identical results (the ordering can be different unless we specify it).

Drilling down to SQL

We can of course ask Django for the SQL it generates (turns out that's not trivial but that is a digression for another article). And it's interesting to compare the SQL that the __in lookup generates compared with our annotating approach.

Here's what an __in lookup generates (in pro forma):

SELECT DISTINCT thing.id, thing.name
FROM thing
INNER JOIN category_things ON (thing.id = category_things.thing_id)
WHERE category_things.category_id IN (1, 4, 7, 23)

and here's what our annotation approach produces (in pro forma):

SELECT 
  thing.id, 
  thing.name, 
  COUNT(category_things.category_id) 
    FILTER (WHERE category_things.category_id IN (1, 4, 7, 23)) 
      AS "cat_count"
FROM thing
LEFT OUTER JOIN 
  category_things ON (thing.id = category_things.thing_id)
GROUP BY 
  thing.id, thing.name
HAVING 
  COUNT(category_things.category_id) 
  FILTER (WHERE (category_things.category_id IN (1, 4, 7, 23))) 
  >= 1

Django's SQL clearly does not adhere to DRY, given it reproduces COUNT(category_things.category_id) FILTER (WHERE category_things.category_id IN (1, 4, 7, 23)) in two places while this is also valid (and tested, it works):

SELECT 
  thing.id, 
  thing.name, 
  COUNT(category_things.category_id) 
    FILTER (WHERE category_things.category_id IN (1, 4, 7, 23)) 
      AS "cat_count"
FROM thing
LEFT OUTER JOIN 
  category_things ON (thing.id = category_things.thing_id)
GROUP BY 
  thing.id, thing.name
HAVING 
  cat_count >= 1

caveat this may be an artefact of the SQL drill down, or generation, which as I said is not trivial in Django, but the subject for another article, point here being is we are not necessarily seeing here the SQL Django actually sends to the database engine - though in all honesty it pretty likely is

For completeness of course the SQL for the ALL/AND request that I was pursuing at outset becomes:

SELECT 
  thing.id, 
  thing.name, 
  COUNT(category_things.category_id) 
    FILTER (WHERE category_things.category_id IN (1, 4, 7, 23)) 
      AS "cat_count"
FROM thing
LEFT OUTER JOIN 
  category_things ON (thing.id = category_things.thing_id)
GROUP BY 
  thing.id, thing.name
HAVING 
  cat_count = 4

The Research Effort

For the record, the trail of research that went into the notes, that morphed into this article, on realisation that esoteric snippets of acquired knowledge are worth recording and sharing are:

Updating a Bokeh Graph in a Django Website

Bernd Wechner — Wed, 16 Mar 2022 03:30:53 +0000

With our first simple graph on an existing Django page, the challenge is to update it when the data on the page updates.

To recap: I have a Django page, on it is a table of data, and I have added a Bokeh histogram that summarises the frequency distribution of one of that table's columns. To get there we:

Added some Bokeh JavaScript and css links to our Django template
Installed Bokeh on the webserver, added the BokehApp to Django's INSTALLED_APPS.
In the Django view (which loads and renders the template) created the graph (first a figure, then vbars, then the div and script components to add to template,
Added the div and script components to our Django template.

And we have a simple histogram on the page. Nice.

But this page, allows users to specify a number of filters, and click a refresh button, which requests new data (with these filters applied, using an AJAX request) and updates the table.

It now needs to update the graph as well!

Updating the data in a Bokeh graph (the options)

Turns out, there is no easy, or obvious way to do that. Bokeh is full of features, and can update all right, any number of ways, but not this way, not easily, not in any way I could find in the documentation. What options does it provide? At least these:

Run a Bokeh server. This is a server you can run which can run python code of your wanting. It's easy to run: bokeh serve --show myapp.py (though on a live webserver would need to be installed as a service and whatever port(s) - port 5006 by default - it uses to communicate with Bokeh in the client must be opened on the firewall, the gateway and any reverse proxy etc.). In short, this is a very nice prospect and feature as it communicates over a websocket which has a number of advantages over AJAX. But, all the same, it's overkill for this trial. And parked. What I want is a way for my client side JavaScript, when the AJAX call completes and a response is received, to update the Bokeh Graph. I don't want a whole new piece of infrastructure (just yet, not on the first date).
Using BokehJS. Looks very promising. Basically a JavaScript interface to Bokeh (✔️). But the documentation is discouraging for this trial. The documentation boldly warns: The BokehJS APIs is still in development and may undergo changes in future releases. and all the examples it provides basically see the three key Python steps we used (creating a figure, vbars and components) moving entirely client side into JavaScript. That has some appeal too. Alas it totally does not explain if and how one might, using JavaScript, provide new data to a Bokeh graph created and provided by server side Bokeh. So this too we park for now, and bookmark the idea for later. For this trial, I am seeking a way to update the graph we created in part 1.
Configuring the graph to use an AjaxDataSource. Rather than providing two lists of numbers as a data source for the graph, we can provide it with a URL and a polling interval. And it will pool back checking for new data to that URL. But no, oh so close, but not what I want. I don't want polled AJAX calls, I already have a user-requested AJAX call (filter criteria changed, new data requested). So this too gets parked. Not a bad feature for live updating of data (though the Bokeh Server is probably better) but not good for this trial.
Using a CustomJS callback. This is another brilliant feature. Server side we can write some custom JS and attach it to a client side element to call back to, on given events, like for example data changing! Yes. And no .... Alas It can be attached to a whole range of Bokeh events, but not to our event (the click-handler for my button), the one that triggers an AJAX call to get new data and update the table. That button! I can't attach custom JS to that button and its click event, only to Bokeh widgets that have the js_on_change method (with which the callback is registered with that widget). Again, ever so close, but no prize. That said, we will come to this one later in this trial, it does have a use, even in this trial (clue: we can use it to register a JavaScript callback that adjusts the axes, ticks and labels, which is called when the data on the vbars is changed - but we still have to find a way to change that data, client-side).

And with that (four exhaustively research options parked), we throw our hands up in the air, file a Stack Overflow plea for help, and hope for the best. But, no answers arriving in any hurry and wanting to soldier on, we are left with no options but to reverse engineer something (which is technical jargon for working out how to do something in lieu of documentation or mentorship)

Reverse engineering Bokeh (client-side)

Scratching my head, I was wondering "if, client-side, I have new data from AJAX call to my server, and I have a Bokeh histogram, how can I tell that histogram that I have new data and give it that data?" That was the crux of the problem that, in all my reading, and research, to that point, I had not found an answer to.

The first, baby-step, in reverse engineering, of course, is to take a look at the code. Fortunately, in nay modern browser, I can now press F12 and up propose the source code for the page and a wonderful suite of debugging tools. My, how easy things have become since the 1980s when I first started reverse engineering games by disassembling them to strip the copy protection from them (looks furtively around the room for any disapproving glares, and hopes the statute of limitations is in full swing).

Now if you recall, a web page with a Bokeh graph on it has a div and script provided by bokeh.embed.components and injected into the page using our Django template.

Step 1: That div

Here's what the div looks like:



<div class="bk-root" id="591aa428-0ac3-4085-938e-481caf339309" data-root-id="1233">

wherein, we see our first two clues. Clearly an id that the Bokeh JavaScript library will use to find the div (and I'm betting at this point the script contains a reference to it) and a data-root-id that sounds a lot like a lead in our hunt for a way to furbish this div with new data!

Searching that page for that id surprise, surprise it crops in the script, under:



const render_items = [{"docid":"e71600f9-95b3-4b84-9210-eff6b3036e17","root_ids":["1233"],"roots":{"1233":"591aa428-0ac3-4085-938e-481caf339309"}}];

Where we see the data-root-id and id bound once again.

Step 2: That script

The script itself yields none more clue. It starts with:



<script type="text/javascript">
    (function() {
          const fn = function() {
            Bokeh.safely(function() {

which tells us, there's a JavaScript object called Bokeh, that we might interface with. And what better way to explore that object than to use the browsers debugger.

Step 3: That Bokeh object

Perhaps the most straightforward way to catch JavaScript in action, after the whole page has loaded and everything has rendered and, it's in a clear run state is to set an Event Listener Breakpoint on the mouse-click event.

Doing that, and then clicking on the Bokeh graph (I elected to click on the Refresh Tool button that the graph provides) breaks in the debugger, and there's a wonderful list of the variables, and under Global we see the Bokeh object.

By expanding it and looking at all its many many properties with patience a small clue might be found. But we might find faster clues by looking at and following the code.

We broke in an event handler:



function documentClick(e) {

to be exact. F9 is the debugger's Step key, so pressing F9 a few times and watching where we go may yield fruit. In fact, 11 F9 presses and we find outselves here:



this.throttled_paint = (0, throttle_1.throttle)(() => this.repaint(), 1000 / 60);

And that, is why I thought clicking the refresh Tool button might be a fine idea. There it is. this.repaint. But what is this?

Easy, we just type this into the console and see that it's:



PlotView {removed: Signal0, _ready: Promise, _slots: WeakMap, _idle_notified: true, model: Plot, …}

Bingo!

Step 4: That PlotView

Now PlotView sounds good. In the console in fact I type this. and after teh . the debugger intellisense predictor throws up a list of all the attributes of that object. I flick down them, and there's one that catches my eye model.

So this.model. and voila it has an id, this.model.id and it is ... 1233! yes, that is the data-root-id of our div.

Getting warm now.

Looking at the Bokeh object I play hte same game. Bokeh. throws up a long list of attributes again, and I page down, and one catches my eye again. It is documents. Bokeh.documents seems to be an Array with one entry so Bokeh.documents[0]. lets us walk through the attributes of a document and voila, there is Bokeh.documents[0]._all_models and it has an interesting value:



Map(32) {"1233" => Plot, "1256" => Toolbar, "1250" => PanTool, "1251" => WheelZoomTool, "1252" => BoxZoomTool, …}

32 entries and the first one is our plot. So let's try:

Bokeh.documents[0].get_model_by_id(1233)

no luck. How about:

Bokeh.documents[0].get_model_by_id("1233")

and that looks good, it's a Plot object so lets try:



Bokeh.documents[0].get_model_by_id(1233) == this.model

which is true! So we have a way of getting models in Javascript!

Step 5: Finding the data source

Alas, the Plot object has no eaisly found data source. Poking around inside of it I do find it has renderers again a list with one entry, and so exploring this.renderers[0] I see it has a data_source property, which in turn has a data property and, you get the idea ... this.model.data_source.data is on the money:



this.model.renderers[0].data_source.data
{top: Array(12), x: Array(12)}

And top and x are the very thinhgs I provided to figure.vbar at the server end and each of these 12 value Arrays are the very lists I provided. Found!

Step 6: That data-root-id, again

Recall, that in my Django view I built the graph as follows:



from bokeh.plotting import figure
from bokeh.embed import components

def view_Events(request):
    # Collect the categories and values
    (players, frequency) = Event.frequency("players", events)

    # Create the figure
    plot = figure(height=350,
                  x_axis_label="Count of Players",
                  y_axis_label="Number of Events",
                  background_fill_alpha=0,
                  border_fill_alpha=0,
                  tools="pan,wheel_zoom,box_zoom,save,reset")

    # And the bars
    bars = plot.vbar(x=players, top=frequency, width=0.9)

    # Build the context variables
    graph_script, graph_div = components(plot)

    # Add them to context
    context = {"graph_script": graph_script,
               "graph_div": graph_div}

    # Render the view
    return render(request, 'events.html', context=context)

I checked plot and sure enough it has an id. And plot.id here is exaclty the same value that appears in the data-root-id of the div and is the id of the Plot object that was repainting itself when we walked through the click handler. The threads come together.

Changing the data source

Sure enough if I pass plot.id into context of the template and use client side it can find the Plot using that id.

And so on the client side I have:



const plot = Bokeh.documents[0].get_model_by_id(plotid)
const source = plot.renderers[0].data_source
source.data.x = players;
source.data.top = frequency;

where plotid came in during page load via the Django template context, and players and frequency are delivered by the AJAX call.

And that sort of works. The values are set.

But the graph does not change. Hmmmm

Step 7: What's wrong?

I can see the data source is changed. Why hasn't the graph changed? I'm missing something.

Scratching my head a little, I think, the missing link must be that repaint command. That is, we have given the Plot new data, but we need to tell it that it has new data, tell it that it needs to redraw itself somehow.

So I break in the code above and inspect the source variable for clues. And the very first attribute on its list is change, which rings a loud bell. It has the value:



Signal0 {sender: ColumnDataSource, name: "change"}

And so is a singal of some sort, that we might send to the Plot. Inspecting the attributes of the signal we find it has (4th on its list), an emit method.

So I add:



source.change.emit();

to code above to get:



const plot = Bokeh.documents[0].get_model_by_id(plotid)
const source = plot.renderers[0].data_source
source.data.x = players;
source.data.top = frequency; 
source.change.emit();

And voila, now the graph updates. We hit paydirt!

Step 8: Do we really need renderers?

Not real comfortable with the use of renderers and at this stage seeing that there are a lot of models ...:



Bokeh.documents[0]._all_models
Map(32) {"1233" => Plot, "1256" => Toolbar, "1250" => PanTool, "1251" => WheelZoomTool, "1252" => BoxZoomTool, …}

32 models and among them when I expand the list are Vbars!

So I try passing not plot.id into the context, but bars.id and the AJAX response code becomes:



const bars = Bokeh.documents[0].get_model_by_id(barsid)
const source = bars.data_source
source.data.x = players;
source.data.top = frequency;
source.change.emit();

And it works beautifully. No renderers needed. The connection is clear now too. At the python side I pass the x/y data to vbar() and at thye JavaScript side I get the vbar and change it's data source and emit a change signal ...all works wonderfully.

The only thing I remain mildly uncomfortable about is the documents[0]. I'm stil not clear what documents are, when there might be more than one and so on.

Car vectors graciously created and made available by macrovector - www.freepik.com

Adding Graphs to a Django Website

Bernd Wechner — Thu, 10 Mar 2022 03:21:35 +0000

Another journey, captured in my journal, that I think is worth sharing. To wit, a story (of learning and how that proceeds), and a summary (of how to add graphs to a Django website).

The story

I have a Django website on which I am presenting a table of data and some statistics on that table (maxima, minima, medians etc.). I wanted to add a histogram (a bar chart) that would summarise the frequency distribution of a value in one column of the table.

If you struggle with abstractions, I can help a little with specifics too. In my case, it is a table of events, and one of the columns records how many people attended the event. The histogram then would have the number of attendees across the bottom and each bar represent how often that number of attendees was recorded.

This is, in fact, just such a histogram:

taken from a table that list events and has a count of attendees (players, as they are gaming events).

Now the existing page (no graph, just a table of events and some summary statistics) has a set of filters on it, so that you can look at a subset of all events - based on weekdays, or venue, or club, or host, or league, or duration or attendee counts etc ... A set of filters can be chosen, and a refresh button clicked and voilà, the data is updated in the blink of an eye (with an AJAX call back to the server that fetches data and updates the relevant page elements).

And that, then, is the context, the desire to plug a graph into such an existing Django page. And as usual, it proved to be, an adventure, with notes taken, and one worth sharing.

Choosing a tool

Such a journey starts with an idea and no rush. At least on this project it does, quite in contrast to many projects, notably those that need to monetise rapidly or beat the competition to market. But this project is a backburner on which notes are taken, and jobs tackled as time permits.

And it starts with the idea, and over time the idea collected three candidates:

Long story short, this list is not exhaustive, there will be more out there, and each of these tools takes a little reading and comparing to arrive at a choice.

Flot loses ground early, as it's jQuery dependence in an era of jQuery decline (in popularity) didn't strike me as an investment into the future.

Dash and Bokeh go head-to-head in a number of online comparisons, the best of which I found was Paul Iacomi's:

https://pauliacomi.com/2020/06/07/plotly-v-bokeh.html

I opted for Bokeh in the end based on those comparisons. You can spend a lot of time comparing a lot of alternatives, of course, and always end up taking a gamble and then investing in one. If you have more free time or are paid to, you can go one further and try several of them, compare them, and write up your experiences. I don't, and so I did some basic reading and said: Bokeh, you're the one.

The first graph

Bokeh themselves, provide a lovely little sample for a histogram:

https://docs.bokeh.org/en/latest/docs/gallery/bar_basic.html

And there are a number of online pages running through the basic steps of a Bokeh graph in Django ranging from the, IMHO, bad¹, to the OK².

But, assuming you have a Django site already (my situation and if you're reading here, possibly yours too - cf. the title) it is worth summarising the needs.

Getting the basics into place

You'll need to install the Python libraries - in whatever context your Django (backend) site is running in, of course. It's as simple as pip install bokeh
You'll need to register Bokeh with Django as an app. Add BokehApp to (the end of) your INSTALLED_APPS setting.
Your view will need to provide a few things in context to the Django template for rendering. Specifically, and at least, to things you will produce in that view using Bokeh (more on that later):
- A div string: Just a string that contains an HTML div element that will house the graph itself. You can give it any name you like in the context, but if you called it bokeh_graph_div you'd simply stick it in your template as {{ bokeh_graph_div | safe }} where you want the graph to be.
- A script string: Just a string that contains an HTML script element which you put anywhere in your template (though it's customary to put these at the start or end of the template I guess). This is the script that will actually draw the graph onto a canvas it places in the div. Again, if you called this bokeh_graph_script in your context you'd include it in the template with {{ bokeh_graph_script | safe }} in your template. Note though this is just a tiny hook script, the bulk of the code that does that drawing and more, is in JavaScript libraries that you will need to include in your template as well.

Your template will need to provide a few things too:

It will need to include the div and script strings described above.

It will need to include style sheets for the Bokeh elements. Basic, recommendation is to include a basic Bokeh style sheet, and another for its widgets and to source them from a CDN:

<link href="http://cdn.pydata.org/bokeh/release/bokeh-2.4.2.min.css" rel="stylesheet" type="text/css">
<link href="http://cdn.pydata.org/bokeh/release/bokeh-widgets-2.4.2.min.css" rel="stylesheet" type="text/css">

It will need to include the JavaScript that actually draws the graph and provides all of its features. Four includes seem commonly recommended:

<script src="https://cdn.bokeh.org/bokeh/release/bokeh-2.4.2.min.js"></script>
<script src="https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.4.2.min.js"></script>
<script src="https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.4.2.min.js"></script>
<script src="https://cdn.bokeh.org/bokeh/release/bokeh-api-2.4.2.min.js"></script>

With all that in place, you actually have a graph! But for one thing, the details of generating its content, the div and script mentioned earlier.

Defining the graph's content

For a histogram, you'll need to prepare your data in two parallel lists one for the categories (X axis) and one for the values (Y axis).

In pro forma:

categories = [ ... ]
values = [ ... ]

where these two lists have the same length.

Then there are two objects of interest to create (the div and script), left.

Create a figure which will hold the graph. This can be as simple as:
```
from bokeh.plotting import figure
my_figure = figure()
```
bokeh.plotting.figure has a pile of options you can play with of course, for sizing, colouring, gridding, labelling and more. What it doesn't deal with is data. It's like the backdrop into which we can now drop our data somehow.
Plonk your data onto the figure. Here I'm interested in a histogram and the glyph of choice is the vbar and dropping it into the figure can be as simple as:
```
bars = my_figure.vbar(x=cateories, top=values)
```
Build the Django context variables. There's a single method (bokeh.embed.components) that will create them for you from your figure:
```
from bokeh.embed import components
bokeh_graph_script, bokeh_graph_div = components(my_figure)
```
and then in your view, just add them to the context delivered to the template. A good tutorial will help, but I'm assuming you have a Django site you're dropping this into and so it might simply resemble:
```
context.update({"bokeh_graph_script": bokeh_graph_script,
            "bokeh_graph_div": bokeh_graph_div})
```

Now you will see a lovely histogram of your values vs. categories in the page where you put the div. And you can look at the options for bokeh.plotting.figure and bokeh.plotting.figure.vbar to make it look nice and the way you like. Oh, and heck if you want something more than a histogram there are plenty of other glyphs to choose from.

A specific example that worked for me

With all that in mind an example I had running and liked:

from bokeh.plotting import figure
from bokeh.embed import components

def view_Events(request):
    # Collect the categories and values
    (players, frequency) = Event.frequency("players", events)

    # Create the figure
    plot = figure(height=350,
                  x_axis_label="Count of Players",
                  y_axis_label="Number of Events",
                  background_fill_alpha=0,
                  border_fill_alpha=0,
                  tools="pan,wheel_zoom,box_zoom,save,reset")

    # And the bars
    bars = plot.vbar(x=players, top=frequency, width=0.9)

    # Build the context variables
    graph_script, graph_div = components(plot)

    # Add them to context
    context = {"graph_script": graph_script,
               "graph_div": graph_div}

    # Render the view
    return render(request, 'events.html', context=context)

That is in fact the code I used, with distracting details removed (as in, this view also contains a table of events and a table of stats and a pile of filtering controls and such which have been removed from this sample). Similarly, if I reduce the template events.html by removing all that distracting detail it looks like:

{% extends "base.html" %}

{% block styles %}
     <link href=”http://cdn.pydata.org/bokeh/release/bokeh-2.4.2.min.css" rel=”stylesheet” type=”text/css”>
     <link href=”http://cdn.pydata.org/bokeh/release/bokeh-widgets-2.4.2.min.css" rel=”stylesheet” type=”text/css”>   
{% endblock %}

{% block content %}
    <h1>Graph of Event Attendance</h1>
    {{ graph_div | safe }}
{% endblock %}

{% block scripts %}
    <script src="https://cdn.bokeh.org/bokeh/release/bokeh-2.4.2.min.js"></script>
    <script src="https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.4.2.min.js"></script>
    <script src="https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.4.2.min.js"></script>
    <script src="https://cdn.bokeh.org/bokeh/release/bokeh-api-2.4.2.min.js"></script>
    {{ graph_script | safe }}
{% endblock %}

When base.html is a Django template with placeholders for styles, content and scripts.

Making it interactive

Here, the adventure truly begins. Bokeh is a rich library with high interactivity touted as one of its assets. The central feature of that is being able to update data in the graph in response to user requests.

Alas, it proved not to be so simple at all, very poorly documented, and requiring some reverse engineering. Which is precisely where a journal comes in handy and why it becomes useful to document.

And this story is long enough so the next chapter will come ... next.

Credits and Footnotes

Pointing dude photo created by drobotdean - www.freepik.com
Coming soon photo created by rawpixel.com - www.freepik.com

In case you don't like random judgements passed on websites, or, you're the unfortunate author of that one, here's why it's bad: The code blocks are all inconsistent in width and style and font size, the whole thing is as a consequence a jarring bother to try and read, and for my needs it repeats unnecessarily all the Django hoo-ha. A better page would just say "Set up a Django project" linking to another page that explains how - written by same author or someone else, they abound, and then get into the meaty stuff that the title promises, namely "Integrating Bokeh visualisations into Django Projects". Surprisingly enough if I'm drawn by that title (and I was), I know what a Django project is and I have one (or a few) and I'm looking to integrate Bokeh visualisations not have my hand held with clumsily formatted code examples through the basics of getting to where I already am, with a Django project. ↩
This one is OK because the code blocks and images are at least in tune with the article. It's short of Great because it falls behind an annoying login demand (which you can lose simply by disabling Javascript with any old browser addon and reloading the page) and because it repeats the annoying handhold through setting up a Django site. Which is not annoying if that's what you need, but it's not what you need when you click on an article titled "Integrating Bokeh Visualisations in Django Application" - because if you click on that it's because you have a Django application and want to integrate Bokeh visualisation into it. ↩

I failed categorically to find a page I'd class as Good alas - and admit I find the web, driven by this endless need for publishing and novelty and enormous population of wannabe writers clamouring for attention just does reruns of the same mediocre stuff over and over!

As an astute reader you will, of course, have noted that I am writing on-line about my disappointment with the bulk of on-line writing and the deep irony of that. I do to.

Of course, I am not clamouring for attention, not doing reruns of the same mediocre stuff, over and over, nor assuming with every story that you are feree of context and a total noob, while at he same time trying to ensure I link all jargon and enw ideas to explanatory pages. Nor am I begging you for your likes, subscribes or whatever - I am just happy to share stories and notes now and then as much to myself and for you, and because I believe that learning (and particularly retention) is driven by story, and there are enough dry facts out there, but my story is often one of finding dry facts the hard way (because I could not find any documentation on it) and involve exploration and often reverse engineering. These are the stories I most like having clearly recorded.

Bringing it all together: Copy With Style

Bernd Wechner — Thu, 06 Jan 2022 02:51:20 +0000

And now the tour is done, all that is left is to present the Copy With Style interface and how it is used.

We expose a single class Copy_With_Style that can be instantiated as follows in client side Javascript. For example:

const clipboard = new Copy_With_Style({ button: document.getElementById("button_to_copy"),
                                        element: document.getElementById("element_to_copy"),
                                        stylesheets: ["default.css"],
                                      });

To clarify, this is yet more delectably clear and intuitive JavaScript notation (not)! It is the de facto standard all the same, for passing a list of optional, named arguments to a function (in this case, a class constructor).

It works using a JavaScript object which is an arbitrary container for attributes and is described well enough by others.

Important to note is only that in reality there is only one argument, which is an object and can be written like so in Javascript:

const myobj = {}

Looks a little like a Python dictionary to me but hey in JavaScript it's an object but looks and works the same way more or less as a dictionary. That is, it can contain properties and be initialized as follows:

const myobj = {prop1: val1, prop2: val2, prop3: val3}

JavaScript is a little more flexible here and you could write also:

const myobj = new Object();
myobj.prop1 = val1;
myobj.prop2 = val2;
myobj.prop3 = val3;

What you need to know is that the single object argument that Copy_With_Style takes can have these attributes (default values are shown and if missing are mandatory arguments, or attributes or properties or whatever you like to call them):

Argument	Default value	Description
button	None	an HTML element that can be clicked. Ideally a button element. If it has a progress element as a child or sibling this can be used for monitoring progress on style in-lining. Unnecessary unless you are copying very large HTML elements with many 10s or 100s of thousands of children.
element	None	an HTML element that will be copied ()with all its children) to the clipboard when button is clicked.
stylesheets	`[]`	An array of strings that represent CSS style sheets. For example `["default.css", "extras.css"]`. If this is a non empty array then only styles from these sheets will be included in the copy. if you know your element only draws styles from specific sheets, then specifying them can speed up the style inlining and/or shrink the size of the copy. Otherwise all the stylesheets the document includes will be used.
mode	`"attribute"`	A string. Either "attribute" or "tag". If "tag" then a `<style>` tag is added to the copy and the element's style attributes are left untouched. This is fast and can conserve pseudo elements like `:hover`. Most email clients can't cope with this, but it will produce a rich copy in HTML contexts that do. If "attribute" then the style attributes of element and all its children will be updated with style information taken from the stylesheets and the browsers computed styles, to produce as true a copy as possible. This produces a larger copy generally than "tag" but is respected by most email clients today. It's also a lot slower to produce if your element is large enough canhave significant processing costs.
defer	`[50000,0]`	Meaningful only if mode == "attribute" and determines if and how often the style in-liner will defer to to UI to keep the UI responsive. Possible values are: `true`: defer to the UI after ever element is processed. Not recommended, slows down processing immensely. `false`: never defer to the UI while in-lining. Will lock the UI until finished. No problem for small elements, can be bothersome for very large elements. `[threshold, frequency]`: The UI is deferred to only if more than `threshold` elements are being copied, and only once ever `frequency` elements are processed. If `frequency` is `0` and a progress bar is specified, it is is optimised to be number of elements per progress bar pixel
triggers	`["button"]`	Meaningful only if mode == "attribute" and determines how and when style in-lining is triggered. This is an array of triggers and can contain: `"button"`: to request that style in-lining happen when the copy button is clicked. `"schedule"`: to schedule a style in-lining once the DOM is fully rendered. `"observe"`: to request that element be observed, and if it's seen to change, then a style in-lining will be triggered. This is useful if element is responsive to user interactions. If defer is set to maintain a responsive UI any change to element will trigger a request for any existing in-lining to bail and start one anew. Sensible combinations are: `["button"]` for small and moderate elements. `["schedule", "observe"]` for extremely large elements.
progress	`false`	Meaningful only if mode == "attribute" and requests that a progress bar be displayed to convey the progress of style in-lining. Accepts the following values: `false`: no progress bar is used. `true`: a progress bar is used if an HTML progress element is found as a sibling or child of element. an HTML progress element: specify an element if you prefer and it will be used. If a progress bar is being used then defer is also set, to `[0, 0]` if it is not set (`false`) or the threshold is set to 0 if it is an Array of 2 elements. This is necessary because without a deferral to UI the progress bar will not update (render).
copy_wrapper	`true`	element is wrapped in a simple `<div>` with id `copy_me_with_style` before styling (either by tag or attribute as specified by mode). If `true` the wrapper will be place on the clipboard, if `false` only its contents will be (i.e element and any style information added).
class_button	`"copy_with_style"`	The CSS class assigned to the provided button. This is the buttons rest state though it conserves this class across all states. When clicked in this state the button will trigger a copy preparation if necessary and a copy of prepared data to the clipboard.
class_preparing	`"preparing_for_copy"`	The CSS class assigned to the provided button when copy preparation is in progress. This may be very very quick (near instantaneous) or take some while, depending on choice of mode and size of the element. Most things are very fast, but "attribute" mode with very large elements can be slow. When in this state the button will either be disabled or trigger a restart of the preparation depending on configuration.
class_ready	`"ready_to_copy"`	The CSS class assigned to the provided button when copy preparation is in complete and a copy is ready to place on the clipboard. When in this state the button will simply copy the prepared texts and HTML to the clipboard.
deep_exclusions	`null`	Meaningful only if mode == "attribute" and provides a function to call, which accepts an HTML element as its only argument, and returns true if that element and all its children should be excluded from the copy. The default implementation excludes all hidden (not visible) elements. If provided, this function replaces the default implementation.
shallow_exclusions	`null`	Meaningful only if mode == "attribute" and provides a function to call, which accepts an HTML element as its only argument, and returns true if that element and and only that element should be excluded from the copy (its children are grafted onto the parent). The default implementation excludes all all A tags that link internal to the site (href begins with /) and DIV tags that have the class "tooltip". If provided, this function replaces the default implementation.
extra_deep_exclusions	`null`	Identical to deep_exclusions, except that it augments rather than replaces the default implementation.
extra_shallow_exclusions	`null`	Identical to shallow_exclusions, except that it augments rather than replaces the default implementation.
debug	`false`	If `true`, debugging information will be written to the console. Useful for checking the scheduling and observation and copy event triggers and such. Was used in developing and tuning this little class and remains in place for future use.
log_performance	`false`	If `true` will log style in-lining performance to the console. This was used to arrive at the performance statistics discussed above.
log_HTML_to_console	`false`	If `true` will log the styled HTML to the console, where it can be inspected. Useful for debugging if pasting brings no joy.
log_text_to_console	`false`	If `true` will log the styled text to the console, where it can be inspected. Useful for debugging if pasting brings no joy.
check_clone_integrity	`false`	When adding styles element is cloned and it is this clone that is styled and added to the clipboard. `true` request that after cloning its integrity is checked. It's never failed, and there's no reason it should, and this is unlikely to be of any great use.
classes_to_debug	`[]`	An array of CSS class names. If specified will break in the browser debugger during style in-lining when an element with one of the named classes is being processed. A nice way to drill down to specific classes to inspect the JavaScript variables if for any reason in-lining is not producing joy for a given class.
styles_to_debug	`[]`	An array of style names. If specified will break in the browser debugger during style in-lining when an element with one of the named styles being applied is being processed. A nice way to drill down to specific styles to inspect the JavaScript variables if for any reason in-lining is not producing joy for a given class. if classes_to_debug are defined will break if both a class and style match are found. This can of course easily be tuned in code as needed.

Conclusion

And that brings this 12 part series to a conclusion, having introduced a small JavaScript class/library, Copy With Style:

https://github.com/bernd-wechner/Copy-with-Style/blob/master/README.md

It is currently 857 lines, albeit unminimised runs at 35kB thanks to what is hopefully clean documented code. It does drop to 16kB when minimised and comments all stripped:

https://www.minifier.org/
https://html-cleaner.com/js/

though few minimisers I tried cope with the class definition (bizarrre).

I hope it finds some use. After the survey of existing options failed to provide a sensible one for a client side Copy button, I was stuck writing one and it was a journey.

Monitoring HTML Elements For Change

Bernd Wechner — Wed, 05 Jan 2022 11:32:38 +0000

While trying to build a simple clipboard copy button, I found the need to in-line styles if they are to survive emailing, and that in-lining could, for very large elements be rather costly (in time), of which was born a need to let UI events happen while doing the job. But because my website is interactive, the element being copied might change while the styles are being in-lined!

In the previous post I introduced the code base to which I begin to refer in these notes. That digression was needed to make sense of the ongoing notes as they refer in greater detail to actual implementation details.

What if an element changes while we're in-lining?

A couple of things are worth noting here:

If I click the copy button, I want a copy of what we see on the browser now, after the change we made. But if the style in-liner is running when I click copy button, some most surprising things can happen, because Copy With Style is nice enough to await defer_to_UI() at intervals I've managed to change the element while it's running. Turns out that if I time this right I can get a copy that is half how the element was, and half how it is. A kind of chimera element.

To help visualise this, in a standard use-case of mine I might be presenting a whole pile of tables (they'd be leaderboards in games in case it's important, lists of players and scores), and I might want to highlight certain players, or add or remove columns from these boards and the site has plenty of options to do just that. If I make such a change within 40 odd seconds of the page being presented (during which the initial render of a very large number of such leaderboards is having their styles in-lined ready to arm a copy button) and then click the copy button, I get this chimera of a copy. The first half before the change and the last half after depending on exactly when I change options, or how many times even.

So we need to watch the element for changes, and if it changes, we have to start style in-lining process a new.
If we're making changes though and every change we make kick fires a new in-lining of styles, we could see an explosion of in-lining jobs on the event queue. And so we need not just to start style in-lining again we also need some way of telling any running style in-liner to finish, quit, stop, bail.

Watching an element for changes

A MutationObserver is handy for this. We can create one with a handler, that is run every time an element changes. That handler can check to see if a style in-liner is running, and if so ask it to stop, and wait until it has, before starting a new style in-liner.

In the implementation of Copy_With_Style, #observe_element instantiates this MutationObserver and associates it with the element (this.element). The observer is stored as #observer and #mutation_handler is the handler called when #observer notices that element changed.

A little synchronisation is needed here. If scheduling the preparation for copy (i.e. in-lining of element styles) then we don't want to start watching for changes until the style in-liner has started (there will be a lot of them, changes, that is, to the element, while the page is being rendered).

To wit, even if we're not scheduling a style in-lining when the DOM is ready, we do not want to start observing the element until the DOM is ready (to avoid a flood change observations). So two schedulers are implemented #schedule_preparation which schedules the style in-liner to start when the DOM is ready and #schedule_observation which schedules an observer to start when the DOM is ready. These are started by the constructor based on the configured triggers in Copy With Style.

Communicating and executing a bail and restart

All the communications required (for one event-triggered function to request a waiting function to bail, and then waiting for it to do so, before starting it anew) takes place with properties of the Copy_With_Style instance.

Specifically it uses:

#is_being_prepared which is set to true when the in-liner (prepare_copy) starts.
#is_prepared which is set to true when it's completed successfully and this.HTML and this.text are set and ready to copied to the clipboard.
#bail which is set by the observer when this.element changes and the style in-liner is running (#is_being_prepared is true), to ask it to drop what it's doing and bail.
#bailed is set by the style in-liner to let the observer know it's done that (which it was requested to do).

Remember that JavaScript is (essentially) single threaded and that this requires a to and fro dance with await defer_to_UI() which takes on a broader meaning now, and could rightly be thought of, or called, defer_to_others() as we use it for one function to communicate with another function.

The observer when running must set #bail then await defer_to_UI() to re-queue itself and give the style in-liner a chance to run and and it in turn checks #bail whenever it continues after an await defer_to_UI(). If it sees #bail set, it bails, setting #bailed first then returns (and thus comes off the queue and is no longer awaiting, has stopped processing elements). Eventually the observer's turn comes up again and it can check if #bailed is set. If not of course it can await defer_to_UI() again as often as needed until it is. In practice I have never seen it not honoured immediately - when the observer defers to UI it goes to the end of the queue and by definition a callback to the awaiting style in-liner is already on the queue ahead of it.

The only way that the observer might need a second check of #bailed is if the in-liner fails to check #bail and act on it when it is called back after it's deferral to UI.

Currently that only issues a warning to the console if it's encountered, but I've not seen it happen yet and if it did happen it suggests a bug.

Copy With Style: JavaScript Curios

Bernd Wechner — Mon, 27 Dec 2021 11:53:47 +0000

As I progress with the notes I took driving The Thing Is ... and I approach completion, slowly I reach the point were we need to look at more specific code examples, and it is time therefore for the great unveiling ... The code rests on github:

https://github.com/bernd-wechner/Copy-with-Style

and we will digress quickly from the exploration of detecting changes in the DOM, to introduce the basic implementation and cover some of the JavaScript curios uncovered in implementing it. These are notes I took while implementing it, meaning they constituted learning for me, and struck me as not very intuitive warranting a specific mention and memory jog).

A JavaScript Class

It's convenient to capture feature-provision such as this (copying an HTML buffer to the system clipboard with styles intact) in a JavaScript Class. Convenient because classes encapsulate code and isolate it from the broader JavaScript environment providing a clean interface.

JavaScript classes are fairly customary and familiar in modern languages, and Javascript offers them in a fairly familiar manner. There's a great introduction on mozilla.org.

The main surprise we touched on in this little library is that members are not declared in the customary manner. That is:

Properties need no var, const or let are just assigned a value. In a truly Pythonesque manner.
Methods also need no function in fact won't accept them. They are declared otherwise, just a functions are, but without the function keyword. Go figure.

I have to admit, this is a most puzzling feature of JavaScript syntax if ever I have seen one.

Private Members

Private members are a relative (and welcome) newcomer to JavaScript, and are declared simply by using # as the first character in the name of the property or method. Alas being relatively new this causes havoc with some of the linters, beautifiers and editors out there. Let's hope that goes away soon (confident it will). For the record I use Eclipse with Wild Web Developer, the Atom editor as beautifier and online JavaScript linters.

Still privacy is a thing well worth requesting and respecting and one of features Python classes lack. It is a welcome additions to JavaScript.

`this` is a Thing

this is a keyword with different meanings in different contexts. As far as we were concerned, inside a class it refers to the instance of the class. And so this.property is how to refer to a property of this instance of the class.

Unfortunately in event handlers, this is an event instance, and if the handler is a class method, well ... when it refers to this.property no joy is had, as this is an event and not an instance of the class. Python handles "this" much more elegantly in my opinion making the instance references explicit arguments to methods, but this is not a piece on Python.

The solution in JavaScript is the bind() method, which functions have, and which provides the value for this.

So method passed as an event handler will have this holding an event, not the class instance, but method.bind(object) will have object as this when it runs. Of course, if we want it have the class instance as this it's simply method.bind(this) which is a kind of JavaScript custom it turns out.

This (double entendre inescapable) is seen clearly in the #copy_to_clipboard() method in Copy With Style but also in a couple of schedulers and an observer.

This then is the heart of Copy With Style in a sense:

#copy_to_clipboard() {
    function handler(event) {
    if (this.debug) console.log(`copy event handler triggered...`)
        event.clipboardData.setData('text/html', this.HTML);
        event.clipboardData.setData('text/plain', this.text);
        event.preventDefault();
        document.removeEventListener('copy', handler, true);
    }

    document.addEventListener('copy', handler.bind(this), true);
    document.execCommand('copy');
}

As you can see, it wants this.HTML and this.text which are Copy With Style instance properties set by this.prepare_copy() and so we make the class instance available in this by passing not handler but handler.bind(this) to the copy event listener.

That is another JavaScript curio that takes some getting used to. The binding of this to this ...

`async` is a Thing Too

The async keyword is rather misleading alas. As JavaScript is essentially single threaded and it alludes to, well, multithreading or multiprocessing but it is not to be. As discussed above, all async does is flag a function as a Promise. That is, a function runs and returns, but an async function just returns having put the function on the end of an event loop queue. It will run then, as soon as possible after everything else that was queued runs. What this does is free the event loop up to run the functions already ahead of this newly queued async function.

But if we await the function when we call it, then it does not return immediately. It queues the async function as before, but not before telling the promise to call back right here when it's done. That is await provides a means to continue running at this line as if we'd never left (state preserved), while permitting the queued promise to run (by saving its state and returning).

To summarise, if we have two functions:

function normal() { console.log("I'm normal"); return "I'm done"; }
async function oddball() {console.log("I'm an oddball"); return "I'm done"; }

then calls to these functions operate as follows:

result = normal() runs normal() and returns when it's done. result contains "I'm done".

result = await oddball() puts oddball() at the end of the event loop queue, and waits for its turn to come round, and for it to run and then returns with result containing "I'm done" (the mechanism of that wait is though, to save state, and register with the promise a call-back to this line with state intact when it's done - creating the local illusion of a blocking call while actually returning - and it's because await returns, and only returns a promise that it can only be used in async functions, which are functions that return promises.

result = oddball() puts oddball() at the end of the event loop queue, and returns immediately, with result now a Promise object. Not much use if we want its returned value. We can't get hold of it's returned value this way.

We can though, get hold of the returned value of the function oddball() ... which is "I'm done" by attaching a function to it via .then(function_to_call_with_return_value_of_oddball_as_an argument) sort of like this:

oddball().then(return_value => { result = return_value })

In case it's not clear this return_value => { result = return_value } is just an anonymous function, we could as well write the above line:

function noname(return_value) { result = return_value; }
oddball().then(noname);

This will set the value of result when oddball() returns.

There's one problem. If we don't give oddball() a chance to run first it won't complete. For example if we do this:

let result = "nothing"
function noname(return_value) { result = return_value; }
oddball().then(noname);
console.log(result)

Then on the console we will see:

nothing
I'm an oddball

That is, result, has the value "nothing" not the value that oddball() returned ("I'm done") because we didn't give oddball() a chance to run before we logged the result to the console. We need to pause our run, put ourselves on the event queue, let oddball() run then if we check result it will contain "I'm done". And that looks like this:

result = "nothing"
function noname(return_value) { result = return_value; }
result_is_ready = oddball().then(noname);
await result_is_ready;
console.log(result)

Where await is the "pause" that puts our continuation onto the end of the event queue, and returns, and (calling code willing) lets oddball() run. To wit, this will produce on the console:

I'm an oddball
I'm done

I wrote "Calling code willing" because await returns, all it can do is cede control to whoever called the function it is in. it is not until this bubbled to the top of the JavaScript processing stack if you will that JavaScript is free to run tasks on the event queue.

That is consider this:

result = "nothing"
async function outer() {
    async function inner() {
        function noname(return_value) { result = return_value; }
        result_is_ready = oddball().then(noname);
        await result_is_ready;
        console.log(result)
    }
    inner();
    while (true) {}; // An infinite - neverending - loop
}

outer();
// JavaScript is done now.

oddball() never runs, and it never calls back to the state-preserved await because outer() never returns and the JavaScript engine remains occupied evermore with while (true) {};. And while that's an extreme and artificial example, the point is that any code in place of that infinite loop or analogously placed after the promise was queued but running before JavaScript finished (like any code after outer() and before // JavaScript is done now.) will run before oddball() ever does. Because JavaScript has to fall idle before it will take the next task off the queue. Running code always has the option and runs the risk, of hogging all of the JavaScript engine's attention!

An voila! This may appear convoluted, and it certainly is a little, but the central idea is that a Promise schedules the function to run by placing it at the end of an event queue, and when it runs, we can ask it to run a function of our choosing that receives its return value as an argument. But unless we take a break and pop ourselves onto the same event queue behind the promised function we will never see it fulfilled, this is the single-threaded nature of JavaScript on display.

In summary, async means a function no longer returns its return value, but instead returns a Promise, and the return value can be accessed either by using await or by offering the promise a function to call with the return value as an argument (using .then(function)).

`async` chains 🔗

Because an async function does not return the function's return value but a Promise object, to get the result we must await it. Therein lies a small domino effect. await can only be used in an async function. That's nice little JavaScript rule there, a tautology even, as await returns a promise and async is the keywords that flags a function as a promise returning function (and not a value returning function).

Which all means that if we want a result from an async function we can only get it in an async function (i.e. we can only await a result in an async function). And so one async begets another and they chain ... and next thing you know it all your functions are async ;-).

In Copy_With_Style, this chain of async bubbles all the way up to the button click handlers copy() and to_clipboard(). At which point we breathe a sigh of relief because the click handler can accept an async function, in no small part because it really doesn't care about a return value.

I see Jim provided a little insight into that on Stack Overflow. Namely that this area is not well documented and standardised to begin with (i.e. what the return value of event handlers does if anything). Either way we can take for granted and be thankful that the click handler is happy to have an async function.

Still, what does that mean? Consider this, click the button and your handler is called and runs. That's the normal scenario.

Now with an async function, click the button and the async function runs. But all that means is it puts the actual function on to the end of a event queue and it will run when its turn comes around. In short it doesn't run immediately and the return value isn't accessible in the same way.

In practice this means very little because, as it happens, JavaScript has a dedicated queue for promises, and resolves all the code generated promises before it looks at UI events, which have their own queue. These are generally called the micro and macro task queues, with the macro queue only consulted when the micro queue is empty and the micro queue only consulted when the JavaScript engine falls idle (has nothing else to do), and promises generally occupying the micro queue and UI events the macro queue.

Which is why, as we discussed earlier and see in the implementation of:

#defer_to_UI(how_long = 0) {
    return new Promise(resolve => setTimeout(resolve, how_long));
}

A simple promise is not sufficient to see UI events processed, because promises are queued on the micro queue and executed before any UI events on the macro queue get to be handled. Which is where the browser supplied setTimeout() function comes in which queues a function on the macro queue! In fact it should be names PutOnMacroQueue(function) ...

The Object as an Argument

Discussed under Bringing it All Together essentially the way JavaScript supports (or better said does not support) named function arguments with default values and optional provision by the caller (all Python does natively) it requires you to replace all arguments with a single object that has properties. And there's a syntax that is tolerable but fluffy.

The Width of `<progress>`

I'm pleased to implement a <progress> element here, as the lack of one had been a long standing gripe with HTML with umpteen 3rd party solutions. Finally standardised and stylable. It's great to see HTML and JavaScript evolving.

To use it sensibly though we wanted to restrict updates to then and when it would cause a visible change, and so we needed a measure of elements to process (cycles in our function) per pixel of progress bar width. The former we know, the latter is tricky.

Turns out it has no property to tells us that. The nearest thing available is .clientWidth which strangely includes padding, and so we have to write:

#bar_width(progress_bar) {
    const style = window.getComputedStyle(progress_bar);
    return progress_bar.clientWidth - parseFloat(style.paddingLeft) - parseFloat(style.paddingRight)
}

removing the padding. But to make sure I got it right I checked the width this returned and took screenshots of the progress bar in Chromium and Firefox and lo and behold both returned the self same width (133 pixels) and I measured the bar on Firefox as 133 pixels (tick ✔) but Chromium renders it 146 pixels wide (bzzzt ✘). Go figure. Little can done about that, but it seems browser support in this space might be a little variable if, in the ideal, close to the mark. And for my purposes the smaller value (the consistently returned 133) is fine as not only is the difference small, this is the conservative value yielding more cycles per pixel and fewer updates.

Monitoring Performance

Bernd Wechner — Mon, 13 Dec 2021 07:56:26 +0000

(a.k.a. How often shall we defer to the UI?)

In my effort to build a simple clipboard copy button, I found the need to in-line styles if they are to survive emailing, and that in-lining could, for very large elements be rather costly (in time), of which was born a need to let UI events happen while doing the job.

With all that under my belt, knowing how long it was likely to take would also be nice, a bit of UI feedback on progress of the style in-lining operation. Basically, a progress bar ...

Progress bars were once a problem in JavaScript, with lots of tips, tricks and 3rd party solutions. But no more, we now have the progress element and it's a breeze. We can just add:

<progress value="0"/>

to a button, or elsewhere, and style it as we wish, display it as needed and update it's max and value to suit. The default width, it seems, when we put I on my page was around the 150 pixels mark, but it can be styled as desired.

And that answers our "How often shall we defer to the UI?" question implicitly. While not a fault-free, universally-suitable answer, it's not a bad start to update no more often than it takes to advance that progress bar by one pixel (i.e to make a visible change to it).

For 100,000 elements say and a 150 pixel bar that's an ominous 666 elements per pixel.

If we drop the frequency of deferrals to the UI to once every 666 elements they are in-lined in 42 seconds (an auspicious figure). Much improved performance.

To recap:

30 seconds with a locked UI,
11 minutes if we defer to the UI after ever element (~100,000 times - i.e. 30 seconds of style processing an 10+minutes of promise management), and
42 seconds if we temper the UI deferrals to 150 times over the duration of in-lining (for a 150pixel progress bar).

Empirical tests confirm that, during those 42 seconds, the UI is admirably responsive, no noticeable lag and so we're content!

But still, in 42 seconds the element being copied might change! That is, because we are deferring to the UI and maintaining responsive UI while in-lining styles, user events processed can change the element that we're copying! So we need to both know something has changed and start again if it has, but that is another chapter of The Thing Is ...

Deferring to the User Interface

Bernd Wechner — Thu, 25 Nov 2021 10:13:37 +0000

When executing long-running JavaScript code the web browser's User Interface (UI) remains unresponsive thanks to the infamous single-threaded nature of JavaScript. Consequently it's useful in time-consuming JavaScript code, to defer back to the UI from time to time, to let user events like scrolling, clicking and typing all take their natural course.

That proves to be quite a tricky thing. Or not, depending on how you look at it.

Mario Figueiredo provides a solution on Stack Overflow and I recast it here, essentially as follows:

function defer_to_UI(how_long = 0) {
     return new Promise(resolve => setTimeout(resolve, how_long));
}

You have to love JavaScript no? Such abundant clarity not? (and I've taken liberties to clarify Mario's code a little too).

So what is actually going on here?

setTimeout, as it happens, schedules something to happen in the future by a specified number of milliseconds (how_long) and if that is 0ms it schedules it to happen ASAP in the future. What does that mean?

Recalling that JavaScript is essentially single-threaded, what this does is put the call to resolve() on a queue. The same queue as it happens that UI events are sitting in, waiting to be handled.

In the context of the long running style in-lining code I'm running, they are not being handled as the styles are being in-lined. Because the in-lining function is running, they are waiting patiently in the queue until it's done. Which as you may recall could be 30 seconds.

setTimeout() is a function that asks the browser to run a timer (for how_long microseconds), and when the timer is up, to place a call to the function (resolve() in this case) on the event loop queue. But the function resolve is a mystery here. It is provided as the argument to a function that is wrapped in a promise. Wowsers, what?

Promises, promises

A promise is just a JavaScript object that maintains a state of either "pending", "fulfilled" or "rejected", with a few convenient callback hooks to set that state.

When a Promise is instantiated it's state is initially "pending" and its constructor takes one argument, which must be a function. That function is immediately executed, and given two arguments, also functions, the first of which must be called to set the state of the promise to "fulfilled" and the second of which must be called to set the state of the promise to "rejected". In a sense the argument to the constructor of a promise is the promised function - it is run, and to let the world know it succeeded it is asked to call the function it received as first argument, and to let the world know it failed it is asked to call the function provided as its second argument, and if it calls neither well ... we'll get to that (but no prize if you guess: the promise simply remains "pending").

To illustrate, a few examples are helpful.

A promise that is immediately fulfilled (which isn't wildly useful) is created with:

new Promise(resolve => resolve());

That is the function just calls the first argument to flag the promise as "fulfilled". Note that this also uses JavaScript's arrow notation for functions. It can also be written any number of different ways using other notations (JavaScript is oh so flexible in that space it seems). Here are a few:

new Promise(function(x) {x()}); - note the name of the first argument is irrelevant. It's the promised function's argument, and can be called whatever you like. All that is important is that the promised function knows this argument is itself a function that it must call to tell the world that it has delivered on the promise - that it is done.

function deliver_me_my_promise(set_state_to_fulfilled) {
    set_state_to_fulfilled();
}
new Promise(deliver_me_my_promised);

Which is (literally) more colourful, as now we give the promised function a name, it's no longer anonymous, we've called it, oh so descriptively deliver_me_my_promise() and its first argument has a revealing name too: set_state_to_fulfilled. All it does is call set_state_to_fulfilled() and that indeed is all that happens, the Promise object now has a state of "fulfilled"

Of course, an immediately fulfilling promise isn't so useful. Nothing much happens.

But, wait ...

Enter await - which is going to make it useful as we will shortly see.

await is an operator, that waits on a Promise to fulfill (or reject, but let's ignore that for now). And so:

await new Promise(resolve => resolve());

will do nothing, just return. The promise thus created, we just noticed, is fulfilled when it's created and await checks to see if it's fulfilled and if so, returns. Which it does. This is in fact one of JavaScript's more interesting null statements, the equivalent of a Python pass.

But what if the promise is pending? That is, it has not yet called either its first or second argument.

In that case await, true to its name, does not return. It waits patiently (we'll get to how later).

To illustrate though. we could write a promise that is perpetually pending easily enough, by not calling the provided resolver. Here's another useless promise, one that is never fulfilled and always pending:

new Promise(resolve = {}) - as it happens, {} is the more common form of a JavaScript "do nothing" statement. Hang on, just for a laugh, you probably realised we could write this cute, always pending, promise using the fancier "do nothing" statement we just saw:

new Promise(resolve = await new Promise(resolve = resolve()))

Pointless, and for good humour alone of course to define one promise in terms of another that does nothing.

We could again write this in different ways like:

function deliver_me_my_promise(set_state_to_fulfilled) { }
new Promise(deliver_me_my_promise);

Essentially the promise executor (deliver_me_my_promise() in this case), the function it runs when it's created, never calls the provided resolver, never tells the Promise that it is "fulfilled" and so it sits there in a perpetual "pending" state.

Now if we await that:

await new Promise(resolve => {});

await never returns. Just waits forever. This never-fulfilled promise is of course as useless as the immediately-fulfilled promise was. But helps to illustrate what await does.

As an aside, the means by which it waits is another topic for another day perhaps, but is often described as syntactic sugar around a yield/next structure, which means, in a nutshell only, that:

the function that calls await has its state saved (probably on the heap - where most stuff is stored),
registers this fact with the promise (where it's remembered), returns (to the function calling it) and
when it is next called it will continue at the line after the await.

It is next called as it happens, when the promised function calls either of the functions it received as arguments, the first to fulfill and the second to reject.

These functions are provided by the promise and what they do when called, is set the state of the promise accordingly and call the function that is awaiting (remember that it registered its wait with the promise, and its state was saved so that on next call it continues on the line after the await). This is the callback hook we referred to earlier, that is when the promised function calls either of its first two arguments, that sets the state of the promise to fulfilled or rejected respectively and calls back any functions that registered themselves with the promise as "waiting".

More useful promises

Your browser (through what's called its Web API) of course provides more useful promises (than our immediately and never fulfilled promises). fetch is such a useful built-in promise. It fetches a URL, which can take some time, so returns a promise. If you're happy to wait for the URL to return data then await fetch(URL) is fine and await will pause until it is no longer pending but fulfilled (or rejected). Of course if you don't want to wait for URL you can attach a callback to the promise using it's .then() method as in:

fetch(URL).then(call_this_function).

That simply registers call_this_function with the promise, to be called when the promised function calls its first argument. Very much like await registers the function it's in that way.

Which means that call_this_function won't be called until JavaScript is idle (aka the stack is empty or all your JavaScript functions have finished). Only then does the event loop look at this queue and call the next thing in it (pulling it off the queue).

The most useful promise (to us here)

We have come full loop, back to setTimeout(). This is a native JavaScript function places a(nother) function call on the end of the event loop's queue and so this:

await new Promise(first_argument => setTimeout(first_argument, 0));

creates a Promise that runs setTimeout(first_argument, 0) which places a call to first_argument() on the end of the event loop's queue.

first_argument() sets the state of the promise to "fulfilled" and calls any functions that registered themselves with the promise earlier.

await does just that, registers itself with the Promise requesting a call back to the same line and it registers that interest, perforce, before first_argument() is called, because the call to first_argument() is at the end of the event queue which JavaScript only starts processing now that we have given up control with await.

So while JavaScript is running, await registered it's desire to be called back when first_argument() is called, and first_argument() is called after all JavaScript has finished running, and all events queued ahead of the first_argument() call on the event loop have been called (and completed).

On the event loop, it's first-in best-dressed, and any UI events that were queued while JavaScript was running are dealt with (browser updates the rendered DOM) and then when they are done, the call to first_argument() (that setTimeout() put there) eventually runs and we carry on where we left off and the UI events have been dealt with.

Of course we tend to name that first argument resolve to produce: await new Promise(resolve => setTimeout(resolve, 0));. It doesn't matter what it's called, the promise just provides a call back function as the first argument and the name is a conventional mnemonic to remind us this is the function to call, to let the promise know it is fulfilled and to call any registered callbacks.

To recap:

When the Promise is instantiated it runs the function provided as its only argument - the nameless function function noname(resolve) { setTimeout(resolve, 0) } immediately.
setTimeout then puts a call to resolve() on the end of a event loop queue, behind any waiting UI events.
await pauses until resolve is called, which does not happen until after all the UI events that were queued before it are called (and handled)

More on pausing

But what does "pausing" mean here? This is a curio worth understanding too, because the event loop queue is not processed until JavaScript is done. So how does it come to be done if it's paused?

The trick is that await returns, it doesn't pause at all, that is a misnomer. It does save the state of the function it's in first and registers a callback to that state with a promise, but after that it returns. That is the await statement is a fancy variant of the return statement. When you execute await you are practically executing a dressed variant of return.

The JavaScript continues executing in the function that called the one that the await was in.

To make clear what is happening here, a firm rule exists in JavaScript, that a function that uses await must be marked async. In a nutshell this is illegal:

function myfunc() {
    await new Promise(resolve => setTimeout(resolve, 0));
    return "I'm done";
}

we are obliged to write it so:

async function myfunc() {
    await new Promise(resolve => setTimeout(resolve, 0));
    return "I'm done";
}

In no small part, this is intended to remind anyone who calls this function that it may not be finished when it returns ...

Forsooth, guess what? function myfunc() returns a string (or it would if it were legal and we took the await out), but async function myfunc() returns a promise for the string and that promise is still "pending" if await is called, only when myfunc() returns "I'm done" is the promise marked "fulfilled" (and and registered call backs are called).

And so if you're content with a promise you can call myfunc() and it will return a promise. But if you need a string, you can call await myfunc() and you can see in this manner that await begets await and async is a bit catchy, like a cold ... once a function uses await and hence must be async, slowly most functions that rely on it become async as well, if they need results, otherwise a chain of promises is returned which is is fine too, if all you need is a promise. He who awaits gets the results of the promise!

But we were wondering how await pauses. And it should be clear that it doesn't nor does it have to, JavaScript will still run to completion even though it is seemingly paused.

It is not paused, it has saved its state, registered a callback with a Promise (just an object on the heap), which the browser has access to as well. But in so doing it returns to its calling function. That function can either:

also await this function, in which case the same applies (recursively up all the calling functions until eventually JavaScript runs to completion, the awaits all having returned).
not await, be a normal function, which just runs to completion.

Then again ...

Rather than await it is often useful to register explicit callbacks with .then().

In the above example we could await myfunc() or myfunc().then(call_this_function). Both register a call back with the promise. The await registers a call back to the same function (state preserved). The .then() registers a call back to call_this_function (any function we name).

Either way, both return and JavaScript runs to completion. it is when a browser event calls resolve() (the first argument supplied to the promised function) that the promise's state is updated to "fulfilled" and the promise honours the call back requests registered with it (either back to the await statement or the function registered with .then())

And setTimeout() is the means by which we place a call to resolve() on the browser's to-do list!

Wrapping up

But back to our cause which is deferring to the UI. We have a solution and we know how it works now.

function defer_to_UI(how_long = 0) {
     return new Promise(resolve => setTimeout(resolve, how_long));
}

And in our time-consuming (UI locking) style in-lining function, from time to time we can await defer_to_UI().

The function of the await being only to save the state of our function, register a desire to be called back to this same line, and return. So when we are called back we continue on the next line like nothing happened ... except of course that we released control (returned) and offered JavaScript the chance to process tasks on the event queue (the UI responds).

From time to time, but ...

How often?

If we look at my most extreme example of about 100,000 elements with styles to in-line taking about 30 seconds to do, if we defer to the UI after each element is processed it takes about 11 minutes to complete! Wow! And that is with no UI interactions. That, apparently, is the mere overhead of re-queuing ourselves 100,000 times.

So clearly we don't want to do that. Instead we'd defer_to_UI at some lower frequency, but what frequency? Either way, to get a feel for things, we should add a progress bar to the style in-lining function.

Which we will look at in the next article ...

HTML Style In-lining Performance: And what to do about it

Bernd Wechner — Thu, 18 Nov 2021 07:47:46 +0000

As part of the initially simply project of a copy button on a web page, I discovered a need to in-line styles if they are to be conserved when pasted into an email, and that there are no satisfactory out-of-the-box client-side JavaScript style in-lining libraries to be found. So I opt instead, for a native JavaScript solution.

But it turns out that it's not quite so zippy in-lining styles like that. I'm not sure any other method is faster (except for Chrome's native method which on a Select and Copy in-lines them as good as instantly).

A minor mystery, as we should be able to get performance as good as the browsers rendering (after all, the browser is interpreting the CSS on each element and working out where to place it and how to draw it, and our effort to simply work out a subset of that and inline it should by rights be as quick). But we don't seem to get native browser performance (one might suppose the native C or C++ browser implementation has an advantage of the interpreted Javascript implementation I guess).

Profiling it reveals no major culprits, it's all just milliseconds here, milliseconds there and it's not bad performance. On a fairly ordinary element that I'd like to copy (a table of tables) I have about 3,500 elements, and these have styles in-lined in about 1 second As a delay after pressing the copy button and having the copy available on clipboard that's not intolerable, it's quite acceptable, if not impressive.

But what if we want to copy a lot? Well on one of my largest datasets I'd ever likely copy, I have about 100,000 elements and these have styles in-lined in about 30 seconds. That is getting a little clumsy to say the least. It means that if you click the copy button and then paste somewhere, chances are it's not available yet it's still being in-lined and not on the clipboard yet!

What better reason to look into a few Javascript tricks to streamline this and hone our skills, to make that experience a bit slicker?

Key strategies for improving the UX on such large copies (that can take 30 seconds to prepare) follow:

Separation of Tasks

My first thought is to separate the tasks, of in-lining styles and of copying to the clipboard. If we in-line the styles when the page loads, then a copy is available instantly more or less to place on the clipboard when the copy button is clicked.

Alas if we do that in the javascript of the page we run into a couple of small problems:

Javascript is infamously single-threaded and this freezes the UI for the 30 seconds it takes to inline those styles. Longer even if you're more ambitious than I am with your copying ... (i.e. copying an even larger HTML element)
if the element in question is itself generated by Javascript (as mine are) or worse fetched via Javascript (AJAX), then the styles in-lined are not even true and complete because the element hasn't been fully rendered while the in-lining is running.

So ideally we want to do the in-lining after the page is fully rendered and complete - that is, schedule it for later.

Scheduling of Tasks

The most generic way to do that, to ensure it runs not only after the DOM is fully loaded but also after all the dynamically Javascript-rendered elements are all settled, we can schedule the in-lining to happen when the page's ready state is "complete". That's easily done by watching for changes in ready state using a custom handler attached to the Document: readystatechange event.

But ... even when it's running at the end of rendering, it can lock the UI up for those 30 seconds, quite a problem on an interactive page that just won't respond and appear locked while that's going on. And so ideally we want to free up the UI somehow while in-lining these styles.

Deferring to the UI

To free up Javascript to handle UI interactions while our script is running, is essential for good UX. That is our script must defer to the UI ...

There is in fact a fairly widely used and cited little JavaScript snippet that does just that:

function defer() {return new Promise(r => setTimeout(r, 0));}

await defer();

But I have nowhere found any lucid explanation of what is going on there how and why this works, and I'll explore that in the next post.

In-lining Styles Client Side

Bernd Wechner — Fri, 12 Nov 2021 07:05:29 +0000

Wanting to in-line styles, the first port of call, not least given Chrome-based browsers seem to in-line styles when using Select and Copy, is to find an existing solution. Alas that search yielded no fruit.

Here are some options that a search on the web uncovers.

But none of them work well here. Let's look at why ... (or if you don't care why not, just jump straight to the Native Javascript solution at the end).

Juice 👎

Looks sort of nice but oh so complicated. So many options. On the up side they do clarify that it can and does run client-side, and provide a test site:

https://automattic.github.io/juice/

And it in-lines <style> tagged HTML wonderfully. Works well.

Still, it's overkill, brings with it dependencies cheerio, mensch and slick and a look into the code behind the on-line demo makes rapidly clear that Juice is parsing the style sheets and applying them in what looks to be, must be, a product of its age. It has over 1500 lines of Javascript and the repo was created in 2011 - the state of Javascript was woeful then compared with today (though not as woeful as in 2001 😏).

So we have to give Juice full credit and admiration for its age and endurance, and that it's still being maintained and is useful. But I'll respectfully pass it by as I'm looking for a minimalist, native Javascript solution as far as possible, using the state of Javascript in 2021 a whole decade later (which in IT terms is an eon).

inline-css 👎

Immediately disconcerting is the npm documentation for this package. It says simply "Inspired by juice" and fails to tell us why the authors felt that embarking on a new project was worth their while. It hints at the fact that it's better than Juice in some way or other (why else start a new project inspired by it?), but lays no claim as to if or how that might be the case. A dark horse.

The github repo dates to 2015 and so, it's not as long-in-the-tooth as Juice, but still, it likely leans on a more meager JavaScript standard than we have available today and quite possibly, like Juice resorts to parsing CSS code and interpreting it. It has about 1700 lines of Javascript and so is even larger than Juice! The last code contributions on github are early 2020 so at least it's being maintained which is good.

Still, it also depends on cheerio, and so falls a little short of a native solution but worse, it does not provide us with any suggestion as to run-context (server side or client side).

It's on npm, which is Node.js which suggests server-side JavaScript. It doesn't rule out that it works client-side, but there is zero indication provided that it does nor a test site, the way Juice provided. And like it or not, I associate npm primarily with Node.js and server side applications (rightly or wrongly).

So for my needs, it goes respectfully into the too-hard basket. Failed, in a sense, by the poor quality of its own introductory presentation. An irony, because there is some nice documentation there, just not a nice introduction that spells out whether it runs client-side or not, how to if so, and how it's different from Juice and why we'd use it. All the obvious questions I have surfing in on it.

css-inliner 👎

Long story short, the introductory documentation fails this package even more than that for inline-css did. Again, it's on npm, and so looks targeted at server-side use and makes no effort to mention the other two (Juice or inline-css), or differentiate itself from them. The github repo dates back to 2015 as well and so it looks at some level to be an example of modern parallel development, this and inline-css emerging at the same time. And unsurprisingly, it again has around 1600 lines of JavaScript. The last code contribution was 2020 so again, at least it's being maintained which is good I guess.

Still, no hint as to the run-context it supports (server-side and/or client-side) but hints, via its npm references and templating languages that it's a server-side tool not a client-side tool. Which doesn't rule out client-side application but again no claim that it works in that context, and no tips on how (if it does) and no demo site, so it's a respectful pass. Still seeking a lean native Javascript solution in preference to some pet project lacking clarity and exhibiting bloat.

Native Javascript 👍

The failure to find an existing solution, leads me to (seeking) a native JavaScript implementation and the journey of discovery that lies behind that.

document.styleSheets provides us with a collection of CSSStyleSheets and CSSStyleSheet.rules in turn provides all of the CSSStyleRules in use and Element.matches() tells us if CSSStyleRule.selectorText applies to a given element.

window.getComputedStyle() provides rapid access to the complete set of computed styles. That is, crucially it has taken all those styles that position things relative to the browser window and calculated where, in real coordinates it landed. Among other things. Crucially, it also has resolved all of CSS variables. It describes what an element actually looks like on the screen, rather than in the abstract CSS.

We can use the style sheets and the computed styles to update the style attribute of each element and then, when all is done the innerText and the innerHTML or outerHTML are available for the the text/plain and text/html MIME parts respectively.

Given we're in-lining styles, and we don't want to actually change the element in the DOM (we're just aiming to copy it to the clipboard in one form or another), we have to take a copy of the DOM element and in-line the styles on that. We have to use the computed style from the source element however as it is rendered on screen and the copy is not (hence has not got computed styles). To wit, we will have a source and target element one that is in the DOM and provides computed styles, and one that is not in the DOM and receives in-lined style attributes.

Importantly an implementation like this is maybe 200, or 300 lines of JavaScript and not 1500.

That more or less covers us and we have a terse native JavaScript in-lining solution!

But ... performance, performance ... turns out that it's not slow, but it's not fast either, let's take a look at performance next, before we dive into an actual code implementation of the above schema - next week's article.