Forem: Eugene Cheah

How I Became a 100x Developer - As a Junior

Eugene Cheah — Thu, 05 Jan 2023 03:30:05 +0000

The idea of a 10x developer, or a rockstar developer, being better than every other developer at everything, is a myth.

Instead, you should strive to be a 100x developer in very specific situations, a 10x developer in some, and a 1x developer in as many as possible. It's important to be aware of the areas you excel at and the areas that you may need to work on.

Unless you work for a large tech company, it's likely you'll be wearing multiple hats. On a daily basis, you'll need to be able to switch between front-end, back-end, and language-specific development, such as Java or TypeScript. This is due to the ever-changing nature of our industry and the sheer number of ways to achieve the same tasks - like printing “Hello World” in over 400 different programming languages.

It's impossible to be good at everything, but it is possible to be great at something.

To elaborate, I'd like to share a story from over a decade ago which was a life-changing moment that had a huge impact on my career.

How I Became a 100x Developer - as a Junior

I worked with a large MNC and was part of a team that was struggling to keep up with demand.

They had built a custom Java web application that thousands of employees were using daily, but as the number of records going in and out of the SQL servers scaled up. Things were slowing down. They had already upgraded the SQL server to its limits and needed new solutions. An obvious one was to use memcache to cache some of the results.

Unfortunately, memcache had been banned due to some security incidents in the past and the lack of HA. The managers had been trying to get approval for memcache for over a year but failed. Meanwhile, the devs fought an uphill battle as every 10% improvement was negated by double that in user growth.

This is where I came in, and joined.

I had no idea about their past 1 year of struggle and had been brought on board as a junior developer to help make some minor improvements to the UI, while the seniors focused on performance.

However because, I was very annoyed by the slow performance of the dev server. And I had been playing with Hazelcast - a brand new Java-based server clustering and caching library.

I used it as an alternative to memcache. And was able to get it running on the application, which had no special restrictions or special approval requirements, that killed all previous solutions.

And had a working prototype with a demo ready within the same week. For one page in the platform, the API calls went from more than 10 seconds to under a second. It was a game-changer.

The whole team jumped on board, and within a month, we had a "clustering solution which is definitely not memcache" in production.

As my team manager and his boss treated everyone to a banquet, my manager said to me, that when he brought me on board as a smart kid, he had never expected me to be a 10x or 100x programmer who could solve their problems overnight.

And that statement struck me hard.

Because I was an imposter – nothing but a lucky imposter.

The Lucky Imposter – Who Was a 0.1x Developer

Technically, I was not a 1x or even a 10x developer. I knew less than my teammates, and I was learning on the job.

My peers could do the front end better than me and the team knew how to optimize SQL (and taught me, thank you!). My biggest credit was that I was able to do the equivalent of “npm install miracle” and read its manual on how to configure it.

In hindsight, there were many issues I had dodged by luck.

Politics (which blocked memcache in the first place)
Use case (we had no write contention safety, meaning if 2 edits happen in the same ms, the cache will be badly updated. Thankfully, this rarely ever happens)
Stability (v1 of hazelcast crashes like crazy, thankfully the API server was designed to be deployed in HA mode, so this was an annoyance to the infra team at worst)
Security (Having an open port to all data on the API server in a cluster, was a bad idea on my part, and a very junior mistake until another senior thankfully read the docs on how to secure it)
Language stack (if the server was in Python, .NET, C# or Ruby, I would have no idea how to even include a local cluster cache)

I was lucky to have a handful of one-trick ponies, which made me 100x in the right situations. Which were only possible because of the foundational work put in place by the rest of the team.

I was also lucky that my managers and bosses allowed me to choose the problems I wanted to fix, as they fast-tracked me, and threw me across multiple teams to "fix their problems". While allowing me to maximise my luck-to-impact ratio and repeat the “miracles”.

With experience, I also realised the opposite – I am slow at front-end development. This was what I was originally supposed to be doing before I was fast-tracked as a 10x engineer and it made me reflect a lot over the years. As it could have been the path I would have been stuck on.

It also struck me that in hindsight, having "mastered" hazelcast, and various other technologies. How much it would have failed for other teams, even in the same organisation, were it not for the unique situation, and safeguards, which should have been credited to the rest of the team.

I struck lucky, as a junior, if anything else.

// Detect dark theme var iframe = document.getElementById('tweet-1566060367825883136-88'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1566060367825883136&theme=dark" }

This is why Senior Developers Tend to be 10x Junior Developers

Once we look beyond the generalisations of engineering, it becomes clear that everyone has their skills, which can range from 100x to 0.1x.

Over the years, as I moved across many teams and projects, I eventually became a “senior” developer. During this time, I realised that seniors generally have more skills, and know what they are good and bad at. They can proactively inform their managers, and prioritise tasks accordingly. This doesn’t mean they know how to do everything, but it means they know what not to do.

Juniors, on the other hand, struggle with this as they lack the experience to know what they are good at or bad at. There isn’t an easy way to find this out other than to just “try”.

Once this is understood, it becomes apparent that it’s all about maximising the number of things you know you can do reliably and well. It’s also about maximising your luck factor when it comes to being assigned tasks, and ensuring you’re not stuck with things you’re bad at.

In a way for juniors, their progress will depend heavily on the alignment of their luck and skillset. Seniors, had more (but not full) control, over how to steer their progress.

In this case: Luck isn’t binary. You can increase the number of situations that align with you.

How to Maximise Your Situation Luck at 100x and 10x

For everyone, but especially for juniors and those just starting out, the best thing to do is to keep trying new things in technology and exploring. This will help you figure out what you’re good at and what you’re bad at.

For those who have a better idea of what they are good at, the next step is to expand the number of situations where they can perform their best. This involves some practice and research into adjacent technologies to the ones they already know. Once you become great at something, make sure to own it so your managers and bosses know that this is what they should give you. This will increase the chances of hitting those 10x or 100x situations more frequently.

Find at least one thing you’re good at, even if it’s a very small and narrow thing. Build up from there. (Some notable examples include regex and configuring webpack.)

For seniors, if you haven’t done so already, start looking beyond the technical aspects of development. This doesn’t mean you have to move into management. But with your unique technical knowledge, you can play a more active role in understanding the user or business use case. This will allow you to optimize with your managers to ensure 10x or 100x success rates for you and your team.

For those who are job hunting, if you know you’re good at something, even if it’s 10x, do some research and dig deeper into your job hunt. Try to find a well-funded startup or MNC that is facing difficulty in the exact thing you’re good at. While this can be difficult to pull off and does involve some networking, if you place yourself in the right place and time, it will maximize the impact for both you and the company involved.

These are all things, you can do slowly over time, to improve the luck factor. And deliver that 10x experience more consistently.

Special shout out to read @swyx article on how to create luck: https://www.swyx.io/create-luck/

As a CTO and Team Leader Today, This Still Holds - 10x Is a Myth - Everyone Is Both 100x and 0.1x

Some may think that the CTO of a UI testing tool such as Uilicious.com would be good with front-end technologies. However, this is far from the truth. Our average fresh hire or junior in the company is typically faster than me the CTO when it comes to writing front-end components with modern front-end frameworks such as Vue.js or React.

This has changed my perspective as a team leader/manager. It showed me how toxic and bad the 10x engineer myth is. It’s an oversimplification of labels and generalization of very unique situations. It’s a toxic misconception that needs to be fixed.

Almost every startup 10x engineer myth, when looked into deeper, turns out to have an individual with unique knowledge in the right place and time. And was able to avoid the tasks they were 0.1x at.

In my case, it was focusing on infrastructure over front-end development. Where instead, I have other team members who are 1000x times better at doing frontend development.

This is an important realization as a manager because, for every individual who is 100x at one task, there’s a task that makes them 0.1x as they get stuck in a time sink. It’s about understanding and recognizing what each individual is 100x at, and what they’re 0.1x at.

While this sounds very straightforward, this is significantly harder in practice.

There are many things a team leader or developer will not know till they have tried it before. Or are given the time and opportunity, to hone and master the skillsets involved in those specific situations. There are also many things, especially new technologies, that no one is good at it, and is all about taking risks and a chances.

It’s the team leader’s job to organize tasks accordingly to best suit everyone’s needs. And if that is not possible, it's also about understanding that even if someone can’t be 100x in your situation now, they can potentially be elsewhere in other teams.

And is on every individual developer, to figure out what they are good and bad at, and to communicate that to their team accordingly.

So to death with the 10x Engineer Myth, Long live the 100x Engineer in specific situations

PS: If you have not made any new year's resolutions. Perhaps increasing your luck surface area could be it!

~ 🖖 live long and prosper

Eugene Cheah: CTO of uilicious.com

Originally posted on: https://substack.tech-talk-cto.com/p/dev-to-cto-how-do-i-become-a-10x

Image credits

Austin Neill - rockstar image: https://unsplash.com/photos/sxxnM-tRlVg
XKCD - comic on standards: https://xkcd.com/927/
CenturiiC - Comic on experienced engineers: https://twitter.com/CenturiiC/status/1566060367825883136
Lucas Santos - Dice picture: https://unsplash.com/photos/XIIsv6AshJY
Vlad Hilitanu - Lego figs: https://unsplash.com/photos/1FI2QAYPa-Y

VC & Founders Note : ChatGPT / LLM, is a major change moving forward - heres why

Eugene Cheah — Fri, 16 Dec 2022 03:41:05 +0000

TLDR

Let's ignore the public hype that is chatGPT for a moment, its ethics or its implications to society (for now), and let's focus on the implications of building an AI based on its research and the tech, for your startup or your portfolio.

TLDR 1: Dataset is no longer a hard requirement

The idea that you need tons of data to make text-based AI useful is now partially outdated

Even an extremely small dataset can produce valuable results (1 sentence ~ 1GB)

Something which nearly all startups are either sitting on, or is able to produce by hand

TLDR 2: It has very broad use cases

You can think of GPT 3.5 as a remote & paid “college intern on demand”

You can never trust interns to be 100% right, but they can provide useful value with proper supervision, guidance and planning

Value that can now grow and scale seamlessly (unlike real human interns)

TLDR 3: Cost and Limitations

Classic unit economics model for SaaS and online services will be a challenge due to the extremely high running cost of most LLM

OpenAI also has a (temporary) effective monopoly, for the infrastructure in this space, and can be both a partner and potential unintentional future competitor at the same time.

If you were on holiday, and have no idea what are ChatGPT or LLM (large language models) - you can find the briefing about the technology here:

Introducing AI / ChatGPT, and how it dreams in 2022

Eugene Cheah ・ Dec 6 '22

#openai #beginners #tooling #ai

So what do I mean in longer form?

Grab a cup of coffee and settle in for a bit…

Part 1: Data is no longer a Hard Requirement

Crash Course On Traditional Text-Based AI Data Requirements

Note: This article is discussing only text-based models and does not apply to image models in many cases (e.g., computer vision).

In the past, building an AI model typically involved gathering a large enough dataset for a specific use case. Different AI model training techniques and tools existed to optimize the training process or its computational costs. However, the size and quality of the dataset was still one of the predominant factors in training an AI.

This would result in a graph something like the following:

All numbers are large approximations and will change drastically depending on the use case, AI model, and data quality. The scales on the axes are intentionally vague and inaccurate because they are subjective. For technical numbers, read the papers in the citations for specific use cases.

The general idea remains, however, where small-scale datasets produce effectively random data before slowly becoming better and finally being somewhat useful at approximately the ~500GB part.

This led to the dataset arms race, occurring in nearly every field of specialization related to AI (especially anything computer vision-related). Which played out across multiple startups from 2009 onwards until today, with a few notable startups being acquired in part for their valuable datasets (and usually talent).

Because the arms race was constantly escalating, it became increasingly difficult for new startups to compete with their AI models (small datasets) against incumbents (larger datasets).

For example, at uilicious.com (low-code UI test automation), we used our limited dataset to train an AI model (called TAMI v0.1). We found that it was spewing garbage half the time, leading us to set the model aside and grow the company without AI while building up our dataset.

In a very overgeneralised and non-technical fashion, AI trained in this way will be referred to as "Specialized Models" from here onwards.

The Start of LLMs or Large Language Models

In the quest for a truly universal or general-purpose AI, especially in the field of human speech interactions (as humans make for the most random of all variables), efforts arose to train a new type of text-based AI for extremely large datasets of public data (think Wikipedia, Quora, StackOverflow, and half of the internet text).

Because these new models pushed the boundaries of dataset sizes and model size (think brain size), the methods used to build them diverged from specialized models (which tend to have more emphasis on accuracy and efficiency).

Text AI models trained in this way are now collectively called "Large Language Models" (LLMs).

The downside of this approach was huge, preventing its early application in many use cases:

Extremely large datasets need to be built and maintained (GPT-3 used 45 TB)
Training costs millions of dollars (GPT-3 cost over $4M to train, GPT-3.5/chatGPT is presumably higher)
Limited use for specialized use cases due to the lack of private training data
Lower accuracy than specialized models in specialized use cases
Expensive to run (more on this in limitations later)

LLM was dominated by Google (who already had the data and had the motive to do so for their AI assistant) and OpenAI initially. Later, Microsoft, Amazon, Apple, Salesforce, Meta, and a few others joined the race.

However, due to the sheer size and cost to train such a model, it's generally reserved for major tech companies with deep pockets.

Though the first few generations of LLMs may have produced disappointing results, as they lost out to nearly every specialized model in every task, things changed as years passed and they scaled up in both dataset size and model size.

Their benefits became more visible:

Facts and accuracy aside, they became really good at talking to humans
In many (but not all) cases: they are really good at learning new specialized knowledge when given the datasets in an appropriate format (you can't just dump it in)

This made drastic changes to the curves:

It also sparked a new wave of AI-backed business models in 2019. Startups could now train new AI models on top of existing LLMs with datasets within their reach, from chatbots to presentation slide generators, code copilots, copywriting and even D&D gamemasters.

No longer was these AI models in the exclusive domain of large tech companies. For a small fee for training and running the AI on their infrastructure, OpenAI and Google started opening up their models for others to build upon as a means to profit from the LLMs.

This was beneficial to startups, too, as they no longer needed millions of dollars in R&D investment required to build these large models internally, allowing them to reach the market faster with their proof-of-concepts.

Data was still, in a way, king. It may have moved the curve to be more accessible, but it still required a team to build up large datasets and tune the model. As a result, while many of the 2019 startups found it significantly easier to build their prototypes, many found it difficult to cross the "useful" threshold, as they needed to scale up their dataset with diminishing returns.

This is consistent with early internal trials of uilicious.com TAMI AI models - basically it was between interesting and "meh".

But that's when things start to really change in 2022....

How Large Language Models are Few-Shot Learners

The release of GPT3.5 (or chatGPT as it's known on the internet) last month (December 2022) had huge implications for the way AI models are trained and fine-tuned.

It has unlocked the possibility of creating usable AI with extremely small datasets - something that most startups have access to or can create by hand. This is a fundamental shift in how we think about AI training.

At uilicious.com, we were stunned when we discovered that a small sample dataset smaller than 1GB, of our larger ~100GB dataset. When converted and optimised with the new training techniques, had exceeded the "useful threshold" - the point at which the AI could be utilised by our users and outperform everything we had previously.

Subsequent experiments with larger datasets showed diminishing returns. The key takeaway, was how little data was needed to "make a useful product". It took us less time to build a proof-of-concept AI, fine-tuned for our proprietary use case, than it took to write this article.

Using GPT3.5 as a basic building block, it is now possible to build useful AI applications for various use cases without needing a specialised team or individual. Depending on the use case, the dataset can be as small as a single sentence or paragraph, or up to 100MB or 1GB - a size that is achievable for many startups.

Alternatively, if you can get chatGPT to act in a way that you find useful and of value to your startup. You can build it as a dedicated AI service.

The drastic reduction in required dataset made it possible for us to build a "useful" AI model with only a small fraction of our full dataset - something that had previously been "useless" or "impossible" at our scale.

In many ways data is no longer king, integration and finding useful application is the true kingmakers with this new AI.

The screenshot is a demo of our AI in action, generating full UI testing scripts, from a prompt, as an example integration.

Something which is now doable in a week, with the right team, and the right data.

Do reach out to us if you are interested in the future of AI and UI testing.

Part 2: The Hunt for Use Cases (and AI Interns) Has Begun

The intern framework for finding use cases

GPT-3.5 and all other large language models have limits on their accuracy and reliability. This is partly due to their nature of dreaming out answers.

While in theory (yet to be proven), given a large enough specialized dataset, its accuracy can be drastically improved for specialized use cases. However, if you have such a large dataset, chances are you could already build "specialized models".

Ultimately still, this limit on accuracy and reliability is only a critical issue in sensitive industries (e.g. medicine). For most cases, it is merely a distraction in finding use cases. Once the AI crosses the “good enough” threshold.

A healthier and more realistic approach to the problem is to imagine GPT-3.5 as a remote and paid “smart college intern on demand”.

Because realistically, the AI has all the same limitations in such use cases, in addition to being remote and online:

Never trust interns to be 100% right all the time (let alone diagnose medicine)
Create a workflow that allows the intern to safely provide useful value, with the appropriate supervision, guidance, and planning
For one-day or one-month interns, you would create a formal training plan to quickly get them productive in one hour or one week of training, respectively
You should be paying your interns; they are not free (both for humans and AI)

The only real benefit the AI model has over real-life interns is:

They are not affected by school seasons
You can scale both up and down at any second
The training is something you can do once per batch and scale up to as many instances

The downside the AI has over the humans is that they cannot get you coffee in person.

Once you frame it in these terms, it becomes significantly easier to figure out how to integrate the AI into existing business processes or products.

Need to generate article ideas? Get a one-day intern. Need to convert some proprietary Java to JavaScript code? Get a one-month intern with some basic training on how to do the process. And so on and so forth - at no point in the process should the human be removed from the loop in supervising and iterating with the intern.

On a slightly more technical level:

A one-day intern can be quickly built using prompt engineering; anything you manage to get chatGPT to do fits into this category. The downside is there is a practical limit (about 2000 words) of what you can fit into the training data. The plus side is you can experiment and set this up in seconds, and it's very easy to setup or iterate with chatGPT.

A one-month intern (or anything in between) is where you start having a formal training dataset, which they can memorize and learn. They will be able to handle more situations according to the training materials. The downside is you actually need to prepare that training material; however, combined with both techniques, it can get very far.

Part 3: Cost and Business Limitations

Cost per Prompt Will Make or Break the Business Model

This is the biggest weakness for this newer approach in AI, which is based on bigger and better models. Unfortunately, there is no free lunch. While it is cheaper and easier to train for specific tasks in terms of dataset size, it is significantly more expensive to run compared to more traditional AI models.

It is not cheap; the cost per prompt and answer range anywhere between one cent and fifty cents, depending on how much data was needed to train or use in the process. This is drastically higher than a typical API server, which can handle a million requests per dollar.

To put it in simpler terms, it costs more in server hardware to handle one AI query for one user within a given second than it does to serve a million user requests for a typical medium-sized Shopify website. This is not due to OpenAI or Azure soley trying to make a profit; it is down to the pure server hardware infrastructure required to run such large models.

As a result, as powerful as it is, incorporating such Large Language Model AI has a hefty price tag and may not be applicable for all use cases due to this limitation alone.

The end result is that, while many use cases can benefit from the use of such an AI, not all use cases can afford it; and this should be an important consideration for any implementation.

Example A: Support Service Chat

For a support service desk, a typical support staff can handle ten customers per hour, with each customer having an average of fifteen back-and-forth prompts. If it is five cents per prompt - this adds up to $7.50 s an hour if an AI was used to mimic a single support staff.

This is not only cheaper than the typical US call center staff median salary of $15 per hour, it is also much more flexible (no staffing overhead, can scale up and down instantly).

Likewise, it is also possible to take the same "intern" approach, where these support AI serve only as L1 support, allowing the humans to handle the more complicated cases. In this scenario, it makes sense when done and scaled appropriately on a per-prompt (or per-hour) basis.

Example B: A SaaS Service to Draft Emails

The average office worker replies to approximately forty emails per work day, or approximately 880 emails per month. Even at five cents per email, that would be $44 per month on average per user just to handle the email replies.

What makes it worse is that it's reasonable to expect that, with such a service, an office worker would be able to reply to more emails on average. It would not be unreasonable for the average to double to two thousand emails, or one hundred dollars a month, in pure AI costs alone. In this case, if the SaaS startup were to do a simple, let's say ten-dollar-per-month pricing, they can get into heavy potential losses over time.

This pricing cost and business model runs counter to the typical, per-user pricing model that is common in SaaS. Which is why it is common for such integrations to have a "credits" system as a means to cap the usage, and a means of billing, of such an AI.

It is expected that overtime, with better fine-tuning, competition, and cost optimization, the price per prompt can go down. Another notable method is to use the original more expensive AI first at launch, while collecting more data, which is then used to train a more specialized and cheaper model. However, all of these methods go into deep technical details, which may be unique for each use case, and generally require significant time and effort.

And even then, while it can be a drastic ten times in savings, it is fundamentally more expensive than traditional SaaS API services.

OpenAI Has an Effective Monopoly (For Now)

Prices May Be Stuck Until Competition Arrives

While there are existing open-source Large Language Models, in very frank terms, they are either comparable to GPT2 or somewhere in between that and GPT3.5.

For some simple cases, once they start building up a reasonable dataset, these smaller (and cheaper) models may be useful to migrate into as a means of cost cutting.

However, for other complex cases, such a move may not be possible due to the complexity of their AI, giving OpenAI an effective monopoly with no incentive for lower pricing.

It is believed, however, that over the course of the next one or two years, the open-source community will catch up and, in doing so, perhaps allow prices to improve due to better alternative infrastructure providers. However, because this is an uncertain future, it is worth highlighting.

OpenAI Is a Potential Unwilling Competitor for Some Models

While not intentionally, it is critical that startups within this space build feature sets that are defensible beyond their prompt-to-text bots.

For example, there were several smaller startups who built GPT3-based bots or SaaS around specific prompt-to-text use cases, like name generators or idea generators with a simple interface.

Literally overnight, with the launch of chatGPT, these small single-use-case text-to-text SaaS were made redundant, as everyday folks can now get the same functionality via chatGPT for free. While it may not have been OpenAI's intention to compete with the very partners that build on them, it may be its inevitable nature to do so, as they keep improving their model and chatGPT. As such, to ensure this does not repeat itself, it is critical for any business model around this technology to figure out what additional value they provide beyond just prompt-to-text, perhaps better user experience or integrations with existing toolings, etc.

Lastly: It Ain't Perfect

A reminder of the intern model - do not expect to use this for curing cancer tomorrow. So please do not go stuffing AI into every product and startup on Earth when it makes no benefit to the end user to do so.

~ Until next time 🖖 live long and prosper

Eugene Cheah : CTO of uilicious.com

This article was originally posted on the authors substack

Notes that didn't fit in

In the AI / ML industry, the ability for a LLM to quickly learn new concepts and apply them is quantified and measured by tests called "Zero-Shot", "One-Shot" and "Few-Shot" learnings.

Generally the better the AI does in these test, the less data you need to train it for your use case.

In my opinion: This makes sense in hindsight - who would have thought that a Neural Network modeled after humans, would act like humans. And benefit from the T-Shape education model. Where a large breadth of generalised knowledge, help improve ones ability to learn specialised knowledge in one are of expertise. (this statement is not backed by any data)

swyx also does an excellent job in trying to consolidate various information, in this rapidly chaotic and growing space (recommended readings)

The Day The AGI Was Born: https://lspace.swyx.io/p/everything-we-know-about-chatgpt
How Open Source is eating AI: https://lspace.swyx.io/p/open-source-ai
His live AI notes: https://github.com/sw-yx/ai-notes/blob/main/TEXT.md#chatgpt

Currently BLOOM is the main opensource contender for GPT3 (not 3.5): https://towardsdatascience.com/run-bloom-the-largest-open-access-ai-model-on-your-desktop-computer-f48e1e2a9a32

Other sources and citations

GPT-3 45TB training set: https://www.springboard.com/blog/data-science/machine-learning-gpt-3-open-ai/
GPT-3 estimated $4 million++ plus training cost: https://lambdalabs.com/blog/demystifying-gpt-3
Bloom AI, an opensource competitor $1.6 million training cost: https://techcrunch.com/2022/07/12/a-year-in-the-making-bigsciences-ai-language-model-is-finally-available/
T-Shaped model of education and work
- https://wordspy.com/index.php?word=t-shaped
- https://collegeinfogeek.com/become-t-shaped-person/
Large language models papers, for zero and few shot learnings
- https://arxiv.org/abs/2205.11916 (zero shot)
- https://arxiv.org/abs/2005.14165 (few shot)
Call center salary rates in US : https://www.comparably.com/salaries/salaries-for-call-center-agent
Average email sent by an office worker a day: https://www.templafy.com/blog/how-many-emails-are-sent-every-day-top-email-statistics-your-business-needs-to-know/
OpenAI announcement of chatGPT / GPT3.5 : https://openai.com/blog/chatgpt/
Unsplash photo credits

Introducing AI / ChatGPT, and how it dreams in 2022

Eugene Cheah — Tue, 06 Dec 2022 10:02:28 +0000

with large language models (like chatGPT), and AI art generation - everything we know about tech in the next few years, maybe changing drastically

This article focuses more of the current state of AI, on 5th Dec 2022. As a means of briefing others outside this industry, what has been going on recently, and what are its limitations. With a focus on recent developments in chatGPT, over long term speculations.

Recommended music for the article - GITS Stand Alone Complex: Be Human - @youtube-music, @spotify

In 2022 artificial intelligence (AI) has made some major strides in its capabilities. Largely in the two following area

Prompt to text generation
Prompt to image generation

Collectively, these "prompt-to-output" models are designed to take a prompt (or a question) and generate a finished product in either text (like a clickbait headline, or programming code) or image form (an illustrated art image, or stock images) based on that prompt.

This type of AI is incredibly powerful due to how simple it is to use, and has the potential to revolutionise many different fields, and can literally feel like magic.

An example would be how a single prompt like

Hey, I'm trying to come up with some interesting, fantastical way of decorating a living room for a design project

Can be used to generate the following image

Or write business plans for a website

Or even be used to help generate entire novels

With many more new unknown use cases to come.

Because its full scope of its capabilities really requires you to try it, to better understand the potential for yourself. I would highly recommend giving these services a try, while its still free at https://chat.openai.com/chat (it may turn to a paid service in future)

Holy Crap, can it generate anything?

Yes, but that includes dreaming up lies.

In an extremely convincing fashion that even experts can find it hard to tell the difference.

Dare Obasanjo 🐀

@carnage4life

ChatGPT is an AI that has mastered a unique human skill, bullshitting. It knows what the shape of a good answer looks like but often not the details.

But we’re now learning that even technical people are happy with an answer that looks right instead of is right.

10:41 AM - 05 Dec 2022

It can also be very stubborn in how it present itself, so as hard as i tried to get it to help me write this article, unfortunately it just couldn't be "me".

PicoCreator

@picocreator

So can #ChatGPT do all my article writing and documentation?

Yes, it can help bounce ideas, and drafts. And basically be a writing buddy

No, it seems to be very very stubborn in writing in a certain set of styles. And not one that I am used to, or is me

But is it useful? YES

04:08 AM - 06 Dec 2022

In a weird way, you should view these AI as unique assistant who ...

is a naive smart kid
who is slightly high or dreaming in the clouds
have very unique and flawed views
and is very willing to bluff their way, in providing a response, even if they dun have the answers

If you ever dealt with such an individual, you know that there are random nuggets of pure gold wisdom that they will say from time to time, and bulls#!t you will need to filter, as it dreams up the answer.

The later act of filtering, is potentially the biggest danger of this new technology.

As it can be very very difficult, because of how convincing the AI structure its answer, if you are not an expert in the respective field.

For example, as an experiment, I asked it questions in domains i had zero (or even negative) expertise, like cooking or chemistry. And I realised I have nearly no capability, in validating its bluffs.

This is despite my best attempts to google and fact check it at times, where it may give sometimes answers that is slightly different, but I have honestly no idea if its right or wrong. Unless I try it myself.

Which can potentially be dangerous, seeing as how it can get things wrong in my area of expertise.

How is that possible, wasn't it trained on nearly a terabyte of data?

Yes, but it is not thinking, it is dreaming.

To elaborate, let’s start with an oversimplified explanation on how these AI models fundamentally work.

First the AI is given a large dataset of articles or images from various sources, it uses this data to provide the foundation of its knowledge, loosely connected together. This is then trained together with possible prompts and responses given by the user.

Once this is completed, we have a working neural network or trained model with all this loosely connected data. Frozen in time.

When a prompt or question is given, the AI is not thinking of an answer or deliberating from the trained model. In a way, it is hallucinating the answer using the snapshot of knowledge it has.

Now, lets use a human analogy instead

The human analogy, would be your brain right now, with all your past knowledge, or experiences in school or in life. Snapshot and recorded, in your head. Minus your emotions, and your feelings, or anything hormonal related.

When a prompt is given, imagine it as being told to you, before going to sleep into a dream. For example "go buy milk tomorrow, when you wake up".

For most folks, sleep is when they as an individual have little to no control with how the story goes (ignoring the lucky few lucid dreamers).

In this state, your mind in its dream state, starts to take that last phrase that is stuck in your head, and starts dreaming a story around that phrase.

Where you are an individual inside a dream, and your brain the dream world doing its best to be as convincing as possible that its real and not a dream world, continuously writing the story back and forth with your interactions and actions.

All while using the snapshot of all your past knowledge and experience, in your brain. To make your dreamworld as realistic as possible

Flipping it back around

You, with your prompts is a participant in a dream.
With the AI being the dream world, trying its best to be, as convincingly as possible, a good dream. Where it answer your questions in a very believable manner.

And just like your dream world, it can be at times entirely realistic, and factually accurate, or as imaginary (and perhaps exciting) as it wants to be

As such, this dreamlike state, allows it both to make basic math mistakes with high confidence

Ioannis Papapanagiotou

@ipapapa

Before we all get excited about ChatGPT, I asked a simple elementary school math question. It gave me a wrong answer, x=2.25. The correct answer is x=2. The error is in the arrow. How come AI be wrong?

04:18 AM - 02 Dec 2022

To hilariously answer questions as a cat when instructed

Jeff Yang

@originalspin

ChatGPT really sold it with that final Meow!

07:41 AM - 02 Dec 2022

To very impressively, even emulating interactions with a virtual machine (rather accurately!)

What does it mean for the future of work?

It's difficult to predict exactly how the advancements in AI will impact the future of work, but there are some potential scenarios to consider.

One thing that seems clear is that the current wave of "prompt-to-output" models is having its "iPhone revolution moment". Just as the iPhone changed how everyday people interact with software and ushered in the era of smartphones, prompt-to-output models have the potential to change how we work and interact with tools.

It has already started fundamentally changing industries, today. Especially in SEO copywriting, and stock illustration / photo industries.

The next wave could manifest in waves of AI assistant work tools and practical enhancements to existing tools, ultimately making automation more accessible and lowering the barrier of entry for performing tasks.

Regardless of the forms of changes, this is a significant development and every startup founder (including me) and product owner should be taking note of how to integrate AI into their business. Just as the smartphone/iPhone era made entire industries and tools obsolete and created new domains of app-based tools, the rise of AI has the potential to do the same.

However, it's important to note that not all tools will be impacted by AI. Some industries, may not see much change beyond a new skin or UI. Only time will tell which side your business falls on.

PS: if you are feeling FOMO, and is in the tech industry, chill - the iPhone and app revolution happened in months and years, not days. Do go on with your life, and your holiday plans.

As long as you are not ignoring the potential changes, and making steps towards it - you are good.

If you are outside the tech industry, keep a look out, be ready to learn new tools, across the next few years - that may change your industry. And be prepared to learn and use them to your advantage.

~ Until next time 🖖 live long and prosper

SAINE is Mathematically One of the Best WORDLE Starting Words - Here's Why

Eugene Cheah — Fri, 15 Apr 2022 22:52:57 +0000

TLDR: I wrote a Wordle solver bot with Javascript and UIlicious. You can run this snippet any day to get your daily Wordle solution. Try and see if you can get a better score than the bot! Feel free to edit it and optimise the solver algorithm!

The wordler solver is covered in 3 parts, which covers the

UI interaction code (link here)

Wordle Statistical model, and the math behind it (this article)

Unit testing and benchmarking of the wordle solver (@ TODO)

All the statistical examples can be found at the following link: https://uilicio.us/wordle-statistics-sample
And is generated via code used here: https://github.com/uilicious/wordle-solver-and-tester

Cat planning our Wordle strategy

Our WORDLE strategy

Disclaimer, I do not claim for this to be the best WORDLE strategy (yet), But it is a rather good one =)

Before going into the statistics, let's first cover our WORLDE strategy.

One thing that humans do really well that classic computers are bad at is to "intuitively" figure things out. Unless I plan to train a neural network, the computer program I am developing will need to use a classic dictionary word list to make its guesses.

The one thing that computers do well however, is memorising giant lists of words or data. And performing math on it. So let's use this to our advantage by doing the following sequence.

Given the 2 word list, one full of possible answers ( ~2.3k words), and another with the full word list (13k words)...

Filter out words in the possible answer list against the current game state from past guesses.
Count the number of times each character appears in the answer word list in their respective word positions.
From the full word list, choose the word which is most likely to find a correct character guess. Independently score it preferring words that give more information in the first 4 rounds with either an exact or partial match.
Choose the highest scoring word and try it.
Repeat from the top if needed.

Also to clarify: We do not memorise the previous Wordle solutions (I felt that was cheating as the system could end up just memorising the day-to-day list in sequence).

While the exact details of the scoring change slightly depending on the round - to optimise things. It does not change the overall concept on a high level.

So how does this work in practice? For our current iteration of the Wordle strategy, we will view this in practice step by step (without code).

Counting statistics is "easy"

Our starting word: SAINE - and why you should use it

An alias to sain, meaning: to make the sign of the cross over (oneself) so as to bless or protect from evil or sin

There has been lots of advice on the starting 2 words. And it makes sense that this word follows with:

3 very common vowels
All unique characters

But let's figure out why the solver chooses this word by looking at the numbers.

Statistics of the characters in Wordle.

Based on the statistics. "SAINE" has the highest chance of finding an exact green character match as an opening word while having 3 of the top vowels.

Reading the raw distributing table can be understandably hard to digest. So let me recontextualize those numbers here. SAINE has a …

91.06% chance of partial matching at least 1 character (yellow/green);
55.94% chance of partial matching at least 2 characters (yellow/green);
50.24% chance of exactly matching at least 1 character (green).

There is a really high chance of getting at least one or 2 major clues. Inversely, because there are few words without A, I, and E, failure to match is a "huge clue."

Not bad for an opening huh?

What about other popular opening words such as "CRANE" and "ADEPT"?

Statistics of first word guess for SAINE, CRANE and ADEPT.

The only key advantage, for "CRANE / ADEPT" is they both have a 0.04% chance of successfully doing a 1-word guess. I suspect the flaw of the previous public analysis was how they were limiting the opening word to the known answer list. I believe however, we should be using the full word list instead, to maximize the probabilities of clues in favor of a very narrow chance of doing a 1-word guess.

More importantly, SAINE has a significantly higher chance (by ~7%) of guessing an exact match (green) on the first try. Which is incredibly helpful as a clue.

With the starting word debate out of the way, we will see how the system reacts with the different results!

Wordle results before and after migrating to the NYTimes.

Understanding how the 2nd guess PLUTO, leads to PAUSE

So let us look at how the 2nd word is chosen for the answer "PAUSE" (left-hand side).

We are given the following information:

Letter I & N is not in the word.
A is the 2nd character, E is the 5th character.
S is either the 3rd or 4th character (it is not the 1st character).
Only 12 possible answers are in the word list

Pretty standard Wordle stuff. But let's look at how does this impacts the stats on the remaining possible answers. And how “PLUTO” was chosen.

Statistics of the 2nd word characters when the answer is PAUSE.

With only 12 words left, let’s find the most efficient way to eliminate our options.

Because we already have information on the letters S, E, A, our scoring system will avoid them with a penalty (marked in red).
The next biggest probabilities of characters in positions we do not know about are P, U, and T, in positions 1, 3, and 4, respectively (marked in orange).
Finally, because we have positions 2 and 5 solved, we can make use of this to get additional information. So let us try our best to use the characters L & C.

From the following constraints, the only valid word to try was PLUTO. Although we know the letter O is not in the final answer, there is no word for “PLUTC”. Also, while the word PLUTO was not in the answer list, it was in the full word list, making it a valid guess.

After the submission, the system now knows the following:

L, T, O is not in the word (O is a confirmation of an assumption).
P is the 1st character, U is the 3rd character.
S is the 4th character (is it can no longer be the 3rd character).

This means I no longer need stats because there is only one truth: PAUSE.

bender dancing to a successful answer

But what is with the crazy guessing pattern? With MOHUR, BLYPE, and finally ULCER

The Mohur is a gold coin that was formerly minted by several governments, including British India and some of the princely states which existed alongside it.

The statistics here is being evaluated live and the changes are dependent on the result of each round. So the result differed where:

Letter S, A, I, N is not in the word.
Letter E may be in positions 1, 2, 3, 4 but not 5.

Filtering out the possible answer list leads to the following statistics:

Statistics of the 2nd word characters. When the answer is ULCER.

This may not make sense at first because the word does not start with C or B. With 187 words left, this is where the details of the scoring start to matter.

Once again, we ignore the letter E, which is the information we already know (in red).
The letter R at the end has the highest weightage (of 62 words) by a large margin. Finding out the answer for this character either halves (no match) the possibility list, or one-thirds it (it matches).
From here, the next step is finding the word with the most matches against other letters in the respective positions such as O at position 2 (score of 37), followed by any remaining characters. Matching either their positional list or unique letters list.

This may not be the most optimal choice given the known answer list. However, this is intentional because, we want to focus on the information again until we have a very high confidence in a word. And in this process, we would penalize duplicate letters to focus on reducing the number of options.(Arguably there is room for improvements here)

The results from the guess were interesting.

Statistics of the 3rd-word characters when the answer is ULCER.

As weird and confusing as the word MOHUR was, the result reduced the possibility to 12 words. Once again, it tries to prioritise’s trying new characters and giving it the obscure word of BLYPE.

BLYPE: a piece or shred of skin

This word reduces the list of possibilities down to one word - ULCER, which is the final answer.

Side Note: If you notice, it is willing to try characters it knows that is not in the official answer list. This is intentional, and helps take into account Wordle clones because if the actual answer chosen is not within the original answer list, the system will automatically fall back to using the full word list instead. Making this more resilient against Wordle varients.

Minion issuing a warning

⚠️ Code warning ahead: If you were only here for the math and stats then skip to the end. The remaining content in this article consists of JS code.

Show me the code

The full solving class can be found here.

This article will be focusing on the core functionality that is required to make this process work and not all parts of the code. If you have not read Part 1, read it here.

The code here has been simplified to skip the "boilerplate" stuff.

The solver needs to do the following:

Given the current game state, filter the possible word list.
Given the filtered word list, calculate the statistics.
Given the statistics, and game state, score the word following the strategy.
Suggest a word.

Let us break it down, piece by piece.

Normalize The game state object (generated in part 1)

For every round, the game state would be generated with:

.history [ ] : An array of past wordle guesses.
.pos [ ]: An array of objects containing the following info.
- .hintSet : set of characters that are valid "hints".
- .foundChar: characters that are confirmed for the given position.

This is generated by using the information on the screen in part 1.

However, for our use case, we would need to normalize some of the common data set we would need. Through the _normalizeStateObj function, we get the following.

badCharSet: Characters we know that is not in the word;
goodCharSet: Characters we have confirmed are in the word.

This is easily generated by iterating .history, and the .pos data for building the good character list first. Then using that to build the bad character list inversely against the historical word list.

/**
 * Given the state object, normalize various values, using the minimum "required" value.
 * This does not provide as much data as `WordleSolvingAlgo` focusing on the minimum required
 * to make the current system work
 * 
 * @param {Object} state 
 * 
 * @return {Object} state object normalized
 */
function _normalizeStateObj( state ) {

    // Setup the initial charset
    state.badCharSet = new Set();
    state.goodCharSet = new Set();

    // Lets build the good charset
    for(let i=0; i<state.wordLength; ++i) {
        if( state.pos[i].foundChar ) {
            state.goodCharSet.add(state.pos[i].foundChar);
        }
        for(let char of state.pos[i].hintSet) {
            state.goodCharSet.add(char);
        }
    }

    // Lets iterate history and build badCharSet
    for(let i=0; i<state.history.length; ++i) {
        const word = state.history[i];

        for( let w=0; w<word.length; ++w ) {
            // check the individual char
            let char = word.charAt(w);

            // If char is not in good set
            if( !state.goodCharSet.has(char) ) {
                // its in the bad set
                state.badCharSet.add(char);
            }
        }
    }

    // Return the normalize state object
    return state;
}

Filtering the possible word list

Now that we have the current game state, let us look into filtering the word list:

/**
 * Given the wordList, filter only for possible answers, using the state object.
 * And returns the filtered list. This function just returns the wordList, if state == null
 * @param {Array<String>} wordList 
 * @param {Object} state 
 */
function filterWordList( wordList, state ) {
    // Skip if its not setup
    if( state == null || wordList.length <= 0 ) {
        return wordList;
    }

    // Get the word length
    const wordLength = wordList[0].length;

    // Filter and return
    return wordList.filter(function(s) {
        // Filtering logic
        // ....

        // all checks pass, return true
        return true;
    });
}

For the filtering logic, we first remove the words within the badCharSet.

// filter out invalid words (aka hard mode)
for(const bad of state.badCharSet) {
    // PS : this does nothing if the set is empty
    if(s.includes(bad)) {
        return false;
    }
}

Followed by filtering out the words with wrong hint locations:

// filter out words with wrong hint locations, for each character position
for(let i=0; i<wordLength; ++i) {
    // Get the word character
    let sChar = s.charAt(i);

    // Check if the chracter, conflicts with an existing found char (green)
    if(state.pos[i].foundChar && sChar != state.pos[i].foundChar) {
        return false;
    }

    // Check if the character is already a known mismatch (yellow, partial match)
    // for each position
    for(const bad of state.pos[i].hintSet) {
        if(sChar == bad) {
            return false;
        }
    }
}

For the subsequent words without all the known found (exact, and partial) matches:

// filter out words WITHOUT the hinted chars
// PS : this does nothing if the set is empty
for(const good of state.goodCharSet) {
    if(!s.includes(good)) {
        return false;
    }
}

In addition, we have a variant to filter out unique words for filterForUniqueWordList. This has no character duplicates and is used in the first few rounds:

let wordCharSet = new Set();
// iterate the characters
for(const char of s) {
    // Update the word charset
    wordCharSet.add(char);
}

// There is duplicate characters
if( wordCharSet.size != s.length ) {
    return false;
}

Generating the word statistics

After filtering for all possible answers remaining, statistics are generated via charsetStatistics( dictArray )

This is done by building an object for the type of statistics. Iterate the word list and incrementing the numbers:

/**
 * Analyze the given dictionary array, to get character statistics
 * This will return the required statistics model, to be used in guessing a word.
 * 
 * Which is provided in 3 major parts, using an object, which uses the character as a key, followed by its frequency as a number
 * 
 * - overall : Frequency of apperance of each character
 * - unique  : Frequency of apperance of each character per word (meaning, duplicates in 1 word is ignored)
 * - positional : An array of object, which provides the frequency of apperance unique to that word position
 * 
 * Note that because it is possible for the dataset to not have characters in the list / positional location,
 * you should assume any result without a key, means a freqency of 0
 * 
 * @param {Array<String>} dictArray - containg various words, of equal length
 * 
 * @return Object with the respective, overall / unique / positional stats
 **/
charsetStatistics( dictArray ) {
    // Safety check
    if( dictArray == null || dictArray.length <= 0 ) {
        throw `Unexpected empty dictionary list, unable to perform charsetStatistics / guesses`;
    }

    // The overall stats, for each character
    let overallStats = {};

    // The overall stats, for each unique charcter 
    // (ignore duplicates in word)
    let overallUniqueStats = {};

    // The stats, for each character slot
    let positionalStats = [];

    // Lets initialize the positionalStats
    let wordLen = dictArray[0].length;
    for(let i=0; i<wordLen; ++i) {
        positionalStats[i] = {};
    }

    // Lets iterate the full dictionary
    for( const word of dictArray ) {

        // Character set for the word
        const charSet = new Set();

        // For each character, populate the overall stats
        for( let i=0; i<wordLen; ++i ) {
            // Get the character
            const char = word.charAt(i);

            // Increment the overall stat
            this._incrementObjectProperty( overallStats, char );

            // Populate the charset, for overall unique stats
            charSet.add( char );

            // Increment each positional stat
            this._incrementObjectProperty( positionalStats[i], char );
        }

        // Populate the unique stats
        for( const char of charSet ) {
            // Increment the overall unique stat
            this._incrementObjectProperty( overallUniqueStats, char );
        }
    }

    // Lets return the stats obj
    return {
        overall: overallStats,
        unique: overallUniqueStats,
        positional: positionalStats
    }
}

This is fairly straightforward for loopings across every word and every character increment within the respective statistical count.

The only gotcha is that we are unable to do a ++ increment on an object property when it's not initialized. This would result in the following error:

// This will give an exception for 
// TypeError: Cannot read properties of undefined (reading 'a')
let obj; obj["a"]++;

So we would need to use a simple helper function to increment our needed use case properly:

/**
* Increment an object key, used at various stages of the counting process
* @param {Object} obj 
* @param {String} key 
**/
_incrementObjectProperty( obj, key ) {
    if( obj[key] > 0 ) {
        obj[key]++;
    } else {
        obj[key] = 1;
    }
}

Judges giving their score

Scoring each word

At the heart of the solver is the scoring logic. Which is ranked on every possible word input with the given statistics and state.

Disclaimer: I do not claim that this is the most optimal word scoring function there is for Wordle. It definitely can be improved, but it’s quite good from my testing so far. =)

/**
* The heart of the wordle solving system.
* 
* @param {Object} charStats, output from charsetStats 
* @param {String} word to score 
* @param {Object} state object (to refine score)
* 
* @return {Number} representing the word score (may have decimal places)
**/
function scoreWord( charStats, word, state = null ) {
    // Character set for the word, used to check for uniqueness
    const charSet = new Set();

    // the final score to return
    let score = 0;

    // Wordle Strategy note:
    //
    // - Penalize duplicate characters, as they limit the amount of information we get
    // - Priotize characters with high positional score, this helps increase the chances of "exact green matches" early
    //   reducing the effort required to deduce "partial yello matches"
    // - If there is a tie, in positional score, tie break it with "unique" score and overall score
    //   this tends to be relevent in the last <100 matches
    //
    // - We used to favour positional score, over unique score in the last few rounds only
    //   but after several trial and errors run, we found it was better to just use positonal score all the way

    // Lets do scoring math
    // ...

    // Return the score
    return score;
}

This goes through various stages: First, we add a safety net to prevent the system from suggesting a word again (huge negative score).

// Skip attempted words - like WHY ???
if( state && state.history ) {
    if( state.history.indexOf(word) >= 0 ) {
        return -1000*1000;
    }
}

We then iterate each character of the word, and score them respectively:

// For each character, populate the overall stats
for( let i=0; i<word.length; ++i ) {
    // Get the character
    const char = word.charAt(i);

    // Does scoring for each character
    // ...
}

Penalizing words with repeated characters or known characters

// skip scoring of known character matches
// or the attempted character hints
if( state ) {
    // Skip known chars (good/found)
    if( state.pos && state.pos[i].foundChar == char ) {
        score += -50;
        charSet.add( char );
        continue;
    }

    // Skip scoring of duplicate char
    if( charSet.has( char ) ) {
        score += -25;
        continue;
    }

    // Skip known chars (good/found)
    if( state.goodCharSet && state.goodCharSet.has(char) ) {
        score += -10;
        charSet.add( char );
        continue;
    }
} else {
    // Skip scoring of duplicate char 
    if( charSet.has( char ) ) {
        score += -25;
        continue;
    }
}

// Populate the charset, we check this to favour words of unique chars
charSet.add( char );

Finally, we calculate the score for each positional statistic with the unique character scoring being used as tie-breakers:

// Dev Note:
//
// In general - we should always do a check if the "character" exists in the list.
// This helps handle some NaN situations, where the character has no score
// this is possible because the valid list will include words, that can be inputted
// but is not part of the filtered list - see `charsetStatistics`

if( charStats.positional[i][char] ) {
    score += charStats.positional[i][char]*10000;
} 
if (charStats.unique[char]) {
    score += charStats.unique[char]
}

// -- Loops to the next char -- //

Now that we have the scoring function we can start putting together all of the parts for the "suggestWord" function.

The simpsons : you've just gotta choose one

Suggesting a word (putting it together)

We have statistics that can then be used to score words. Now, let us put it together to suggest the best scoring word.

We start with being given the game state:

normalize game state;
generate filtered, unique, and full word lists

/**
* Given the minimum state object, suggest the next word to attempt a guess.
* 
* ---
* # "state" object definition
* 
* The solver, requires to know the existing wordle state information so this would consist of (at minimum)
* 
* .history[]    : an array of past wordle guesses
* .pos[]        : an array of objects containing the following info
*    .hintSet   : set of characters that are valid "hints"
*    .foundChar : characters that are confirmed for the given position
* 
* The above is compliant with the WordleAlgoTester state object format
* Additional values will be added to the state object, using the above given information
* --- 
* 
* @param {Object} state 
* 
* @return {String} word guess to perform
*/
suggestWord( state ) {

    // Normalize the state object
    state = this._normalizeStateObj(state);

    // Let'sLets get the respective wordlist
    let fullWordList = this.fullWordList;
    let filteredWordList = this.filterWordList( this.filteredWordList, state );
    let uniqueWordList = this.filterForUniqueWords( this.uniqueWordList, state );

    // As an object
    let wordList = {
        full: fullWordList,
        unique: uniqueWordList,
        filtered: filteredWordList
    };

    // Lets do work on the various wordlist, and state
    // this is refactored as `suggestWord_fromStateAndWordList`
    // in the code base
    // ....
}

Once we have the various game states and word lists, We can decide on the "statistics word list", which we use to generate the statistics model.

// Let's decide on which word list we use for the statistics
// which should be the filtered word list **unless** there is 
// no possible answers on that list, which is possible when
// the system is being used against a WORDLE variant
//
// In such a case, lets fall back to the filtered version of the "full
// word list", instead of the filtered version of the "answer list".
let statsList = wordList.filtered;
if( wordList.filtered == null || wordList.filtered.length <= 0 ) {
    console.warn("[WARNING]: Unexpected empty 'filtered' wordlist, with no possible answers : falling back to full word list");
    statsList = this.filterWordList( wordList.full, state );
}
if( wordList.filtered == null || wordList.filtered.length <= 0 ) {
    console.warn("[WARNING]: Unexpected empty 'filtered' wordlist, with no possible answers : despite processing from full list, using it raw");
    statsList = wordList.full;
}

Once we decide on the stats word list, we receive the stats:

// Get the charset stats
const charStats = this.charsetStatistics(statsList);

Now we decide the word list to use in deciding a word. We refer to this list as the "scoredList".

In the first few rounds, we aim to use unique words as much as possible. Which would not include characters that we have tried previously. This may include words we know are not in the possible answer list.

This is intentional as we are optimizing for information gain but instead over a small random chance of early success.

However when this is emptied, or the game is in the final few rounds, we will fall back to the full list. During the final round, we will always guess using the filtered list when possible: (just give it our best answer).

// sort the scored list, use unique words in first few rounds
let scoredList = wordList.unique;

// Use valid list from round 5 onwards
// or when the unique list is drained 
if( scoredList.length == 0 || state.round >= 5 ) {
    scoredList = wordList.full;
}

// Use filtered list in last 2 round, or when its a gurantee "win"
if( wordList.filtered.length > 0 && //
    (wordList.filtered.length < state.roundLeft || state.roundLeft <= 1) //
) {
    scoredList = wordList.filtered;
}

Once we decided on the scoring list to apply the stats, Let us score and sort it:

// Self reference
const self = this;

// Score word sorting
scoredList = scoredList.slice(0).sort(function(a,b) {
    // Get the score
    let bScore = self.scoreWord( charStats, b, state, finalStretch );
    let aScore = self.scoreWord( charStats, a, state, finalStretch );

    // And arrange them accordingly
    if( bScore > aScore ) {
        return 1;
    } else if( bScore == aScore ) {
        // Tie breakers - rare
        // as we already have score breakers in the algori
        if( b > a ) {
            return 1;
        } else if( a > b ) {
            return -1;
        }
        // Equality tie ???
        return 0;
    } else {
        return -1;
    }
});

And return the highest scoring item:

// Return the highest scoring word guess
return scoredList[0];

4 pugs, dressed up as the avengers

Putting it all together

With the UI interaction code done in part 1. Hit the "Run" button to see how our Wordle bot does!

Screen recording of the wordle being solved

Hey, not bad, my bot solved today's Wordle!

Human and robots as friends

Take away - for the humans?

Because the bots will use a rather "inhuman" technique of calculating probabilities with giant dictionaries.

Most humans will find that this is borderline of doing really weird and crazy guesses. Believe the math because it works.

While playing for the human team, your takeaway from this article is that you should just start with the word "SAINE", or whatever words you would like.

It is up to you since this is your game after all! =) Have fun.

Happy Wordling! 🖖🏼🚀

First published here

I Created a Bot to Solve Wordle so I Never Have to Ever Again

Eugene Cheah — Sat, 12 Feb 2022 05:05:17 +0000

The wordler solver is covered in 3 parts, which covers the

UI interaction code (this article)

Wordle Statistical model, and the math behind it (link here)

Unit testing and benchmarking of the wordle solver (@ TODO)

You can edit or rerun the script, by clicking on the GIF

If you haven't already heard of Wordle, it's a fun puzzle game that challenges you to guess a valid five-letter word in six tries or less.

My wife is really addicted to Wordle, and she's been bugging me to try it every day. I suppose for her, there's a thrill in getting the right word in the least amount of tries. And because we're both gamers and math-nerds, naturally we talk about strategies to solve Wordle in the least number of guesses. But when it comes to actually putting those theories to use and trying my hand at Wordle, I came to realise...

I suck at Wordle!

English isn't my strong suite per say...

I speak Javascript better.

I speak many languages, including Python ;-)

So what better way to solve Wordle, then to write a program to solve it for me. HA!

Let's science this @*#$!

After a weekend of hacking I finshed coding up my Wordle solver bot using JS and UIlicious, and I would say it works pretty well, and has managed to make the correct guess mostly within 3-4 tries so far.

Tada, solved wordle with a bot, on UIlicious

It has mostly stirred up my wife's competitive spirit and annoyed her, since the bot does better than her on most days. But at least she's not bugging me everyday to play Wordle anymore.

Is it cheating to write a bot? No, I'd say it's totally fair game!

Curious to find out how the Wordle solver work? Let me explain!

Basic setup

We're going to use UIlicious to interact with the Wordle website, fill in our guesses, and evaluate the hints. While UIlicious is primarily a cross-browser UI testing tool, nobody said that's all you can use it for, so why not solve some Wordles with it.

Let's setup our solver bot on UIlicious Snippets - think of it as Codepen, but for UI testing, and it's entirely free to use for executing tests on Chrome. UIlicious lets you run Javascript, so I'm going to be writing the solver in JS.

The first thing we'll have to do get the bot to navigate to the Wordle site. We use the I.goTo command to go to the official Worldle site (https://www.powerlanguage.co.uk/wordle/).

I.goTo("https://www.powerlanguage.co.uk/wordle/")

Edit: Since the acquisition of Wordle by NY Times, there's now a cookie banner, which we need to either accept or dismiss, using the I.click command.

I.click("Reject") // to reject cookies and close the cookie banner

Let's just make sure the app has loaded, with quick check for the phrase "Guess the Wordle" using the I.see command:

I.see("Guess the Wordle")

Finally, let's dismiss the tutorial popup by using I.click with a pixel offset to click outside the popup:

// clicks 40px down and 80px right from the top-left corner of the webpage
I.click('/html/body', 40, 80)

Now, we'll just hit that "Run" button to run the browser automation script, and make sure this brings us to an empty Wordle board to play with.

Now we're ready to Wordle!

Next steps

Ok, now let's dive into writing more complex parts of the solver.

The bot will need to be able to:

Enter letters into Wordle
Read the hints from Wordle
Find the solution based on the hints (I'll explain this in a follow up post)

This will be a piece of cake!

1. Entering letters into Wordle

We've only got up to six tries to guess the Wordle, so we'll start with a simple for loop to make 6 attempts to guess the word:

for(let r=0 ; r<6 ; ++r) {
  // solver code will go in here
}

Now we're going to generate a couple of words and enter our guesses into the game. I'm not going to dive into how solver.suggestWord(gameState) works at the moment, but what it basically does is suggest a word based on the previous guesses and the revealed hints. I'll explain how the solver works in a follow-up post, stay tuned!

let guessWord = null
for(let r=0 ; r<6 ; ++r) {

  // guess the next word
  guessWord = solver.suggestWord(gameState)

}

After computing the guessWord, I need the bot to enter the word into the game board, which is pretty straightforward. I use the command I.type to enter the guess word, and I.pressEnter to submit the guess. I've also added a small wait for the game board to finish its animation and allow input for the next guess, because the solver can be too fast sometimes and might attempt its next guess before the game board is ready.

let guessWord = null
for(let r=0 ; r<6 ; ++r) {
  // ... 
  // Does some stuff to get gameState

  // guess the next word
  guessWord = solver.suggestWord(gameState)

  // enter and submit the guess word into the Wordle game board
  I.type(guessWord);
  I.pressEnter();

  // add a small wait for the animation to finish before entering the next guess
  I.wait(2)

}

And that's it!

2. Reading the hints from Wordle

The next important step is after each guess, we need to evaluate the hints to make our next guess. This is a bit trickier, as we'll need to do a bit of scraping to get our hints from the HTML.

First, lets grab each row of the game board. The Wordle app is entirely written using Web Components (impressive!), so we'll need to make use of document.querySelector and .shadowRoot to dive into the <game-app> element and grab each of the <game-row> element which stores important information about the revealed hints.

let rowList = document.querySelector("game-app").shadowRoot. //
        querySelector("game-theme-manager"). //
        querySelector("#board").querySelectorAll("game-row");

Here's what we'll get when we run this query in the browser console:

Oh look what we have here!

Each of these game-row element has two very useful data attributes that we want:

_letters: tells you guess that you've made for this row, e.g. "hello"
_evaluation: an array of hints for each letter, e.g. ["absent", "present", "correct", "absent", "absent"]. "correct" if the letter is in the correct position. "present" if the letter is in the word, but is in the wrong position. And "absent" if the letter isn't in the word.

Now we're going to build a state object to store all of these information and pass it to our solver. The code below is nothing too fancy. We'll wrap this in a function because we want to run it after each guess to get the updated hints.

function() {

    // We prepare the state object which we would want to return
    let state = { history: [], pos:[
        { hintSet:[] },{ hintSet:[] },{ hintSet:[] },{ hintSet:[] },{ hintSet:[] }
    ] };

    // Lets get the list of result rows from the UI DOM
    // unfortunately this is more complicated then needed due to shadowDoms
    let rowList = document.querySelector("game-app").shadowRoot. //
    querySelector("game-theme-manager"). //
    querySelector("#board").querySelectorAll("game-row"); //

    // Lets build the state by reading each row
    for(let r=0; r<rowList.length; ++r) {
        let row = rowList[r];
        // a row can be blank, if there was no guess made
        if( row._letters && row._letters.length > 0 ) {
            let word = row._letters;
            // Add it to history list
            state.history.push( word );

            // Iterate each char in the word
            let allCorrect = true;
            for(let i=0; i<5; ++i) {
                let c = word[i];
                if( row._evaluation[i] == "absent" ) {
                    // does nothing
                    allCorrect = false;
                } else if( row._evaluation[i] == "present" ) {
                    state.pos[i].hintSet.push( c );
                    allCorrect = false;
                } else if( row._evaluation[i] == "correct" ) {
                    state.pos[i].foundChar = c;
                } else {
                    throw "Unexpected row evaluation : "+row._evaluation[i];
                }
            }

            // Configure the final "allCorrect" state
            state.allCorrect = allCorrect;
        }
    }

    // And return the state obj
    return state;
}

This piece of code above needs to be executed the browser, so I need to tell the Wordle bot (which runs on a sandboxed NodeJS instance) to run the funciton in the browser using UI.execute.

function getWordleStateFromUI(){
  return UI.execute(function(){
    // put the above code snippet in here to get the game state
  })
}

Okay, now we want to use this function to get the game state after each guess, so let's go back to our for loop and update it:


let guessWord = null;
for(let r=0; r<6; ++r) {

    // get the game state
    let gameState = getWordleStateFromUI();

    // if we got all the letters correct, we've won, hurray!
    // exit the loop in that case
    if( gameState.allCorrect ) {
        break;
    }

    // guess the next word
    guessWord = solver.suggestWord(gameState);

    // enter and submit the guess word into the Wordle game board
    I.type(guessWord);
    I.pressEnter();

    // add a small wait for the animation to finish before entering the next guess
    I.wait(2);
}

Whew! Now that the bot can enter its guesses into the board and get the revealed hints, we can work on writing the algorithm to solve the Wordle in six attempts or less!

3. Finding the solution based on the hints... in a follow up post!

Writing the solver algoritm is the biggest challenge, and a whole lot of fun. I spent quite a bit of time reading up on optimal strategies (there's even papers written about this), and tweaking the algoritm. But this deserves a post on it's own, so I'm going to write a follow up post after, stay tune!

Putting everything together

Finally, let's put together the wordle solver:

Before the start of the for loop, we'll initialise the solver with:

const solver = new WordleSolvingAlgo(fullWordList, filteredWordList);

And after we exit the for the loop, we want to query the game state again. At this point, we've either guessed the correct word in less than 6 attempts, or have made 6 guesses but don't know what's the final outcome. We'll get the final game state one more time, and report the outcome using TEST.log.pass if the bot won, or TEST.log.fail if the bot lost game.


// this comes after exiting the for loop...

// get the final game state
let gameState = getWordleStateFromUI();

// is it all correct? 
if( gameState.allCorrect ) {
    // CONGRATS!!!
    I.see("NEXT WORDLE");
    TEST.log.pass("==== END OF GAME ====")
    TEST.log.pass("## The wordle is : "+guessWord.toUpperCase())
} else {
    // DARN! Try better next time!
    TEST.log.fail("==== END OF GAME ====")
    TEST.log.fail("## The last guess was : "+guessWord.toUpperCase())
}

Here's what everything looks altogther (except for the solver algo part!)


// Open Wordle!
I.goTo("https://www.powerlanguage.co.uk/wordle/")

// to reject cookies and close the cookie banner
I.click("Reject") 

// Is Wordle loaded?
I.see("Guess the Wordle")

// Solve Wordle!
runTheWordleSolver()

//----------------------------------
// This function runs the wordle solver,
// which will make up to 6 attempts to guess the word
function runTheWordleSolver() {

    // initialise the solver
    // stay tune for more about the solving algorithm!
    const solver = new WordleSolvingAlgo(fullWordList, filteredWordList)

    let guessWord = null;

    // make up to 6 guesses
    for(let r=0; r<6; ++r) {

        // get the game state (includes the revealed hints)
        let gameState = getWordleStateFromUI();

        // if we got all the letters correct, we've won, hurray!
        // exit the loop in that case
        if( gameState.allCorrect ) {
            break;
        }

        // guess the next word
        guessWord = solver.suggestWord(gameState);

        // enter and submit the guess word into the Wordle game board
        I.type(guessWord);
        I.pressEnter();

        // add a small wait for the animation to finish before entering the next guess
        I.wait(2);
    }

    // at this point, we either guessed the correct word in less than 6 attempts, or have made 6 guesses and don't know what's the outcome yet.

    // get the final game state
    let gameState = getWordleStateFromUI();

    // is it all correct? 
    if( gameState.allCorrect ) {
        // CONGRATS!!!
        I.see("NEXT WORDLE");
        TEST.log.pass("==== END OF GAME ====")
        TEST.log.pass("## The wordle is : "+guessWord.toUpperCase())
    } else {
        // DARN! Try better next time!
        TEST.log.fail("==== END OF GAME ====")
        TEST.log.fail("## The last guess was : "+guessWord.toUpperCase())
    }

}

Drumroll... The moment of Truth!

Let's smash the "Run" button and see how our Wordle bot does!

Hey, not bad, my bot solved today's Wordle in 4 attempts!

Mmmmm, sweet sweet glory!

Stay tuned for part 2. I'll explain the solver engine works, and the math behind it. Or if you want to dive into the full code yourself, you can check it out over here, and maybe even replace WordleSolvingAlgo with your own algorithm to solve Wordle:

https://snippet.uilicious.com/test/public/N5qZKraAaBsAgFuSN8wxCL

At the mean time ….
Happy Wordling! 🖖🏼🚀

live migrating VM's - cool tech for the future, that you should avoid using today (Review & Benchmark of GCP E2 Instances)

Eugene Cheah — Mon, 16 Dec 2019 15:44:43 +0000

Should I buy summary: NO

Unless you are using GCP E2 specifically for, non-CPU dependent burst workloads (eg. background tasks), temporary autoscale, or are planning to use committed use discount - stay with N1/N2.

For nearly all other workloads, it makes no economic sense to use E2, due to its approximately 50% reduction in CPU performance (+/-).

If you want the raw numbers, skip the prelude section and go straight to the benchmark section below. or alternatively our github

However after benchmarking it, and reviewing the tech behind it. I realized there is a lot more then meets the eye - how E2 could however potentially be game-changing for cloud providers in the future ... Assuming they have the "courage" to use it ...

(read on to learn more)

Prelude: Understanding the VM workload

Before understanding E2, you will need to understand a typical VM workload.

Where most server VM workloads, are underutilized "most of the time", with the occasional bump in resource consumption, during certain hours of the day, or when they appear on Reddit.

This can be easily be observed by looking at most VM resource usage charts.

The chart above is a browser test automation server, attached to one of several clients within ulicious.com. Notice how this server spikes twice with a small gap in between.

This is extremely typical for work applications representing the start of the workday, followed by lunch, and the rest of the workday (after offsetting time of day, for different timezone)

Also note, on how the system is idle under 40% most of the time (running some background tasks).

What this means is that, for most data centers, thousands of servers are paid for in full, but are underutilized, and burning through GigaWatts of electricity.

Prelude: The auto-scaling cloud era

For infrastructure providers to lower overall cost, and pass these savings to their customers, as a competitive advantage. New ideas came along on how to safely maximize the usage of idle resources.

First came auto-scaling, where VM's can be added only when needed, allowing cloud consumers to allocate only the bare minimum server resources required to handle the idle workload, and buffer surges to request as auto-scaling kicks in (there is a time lag).

Significantly cutting down on idle resource wastage.

This change, along with changes in the billing model. Started the "cloud revolution".

And it was only the beginning...

Prelude: Shuffling short-lived servers

New ideas were then subsequently introduced for a new type of server workload, which can be easily shuffled around, to maximize CPU utilization and performance for all users.

The essence of it is simple, slowly put as many applications as possible into a physical server, until it approaches near its limits (maybe. 80%), once it crosses the limit, pop out an application and redistribute to another server - either by immediately terminating it, or waiting for its execution to be completed and redirecting its workload.

This is achieved either through extremely short-lived tasks (eg. Function as a service, serverless). Or VMs which are designed to be terminated when needed (eg. Pre-emptible, spot instances).

The downside of the two approaches given above, however, is that the application software may need to be re-designed to support such workload patterns.

How E2 - changes the rules of cloud shuffling

To quote the E2 announcement page ...

After VMs are placed on a host, we continuously monitor VM performance and wait times so that if the resource demands of the VMs increase, we can use live migration to transparently shift E2 load to other hosts in the data center.

Without getting too deep into how much ~~pretty crazy and awesome~~ engineering goes into writing a new hypervisor.

In a nutshell, what E2 does, is enable google to abstract a live running VM from its CPU hardware. Performing migration in between instances with near-zero ms downtime (assumption).

Your VM servers can be running on physical server A in the first hour, and be running in physical server B in the next hour as its resources are being used up.

So instead of shuffling around a specially designed VM workload. It can shuffle around any generic VM workload instead. Allowing much better utilization of resources, with lower cost, without reducing the user experience ... Or at least it should in theory ...

Onto the benchmarks!

Side note, while live migration's of VM's are not new tech. This marks the first time it is integrated as part of a cloud offering, to lower the cost of the product. Also, for those who have used it before, the list of issues is endless - which google presumingly has resolved for their custom E2 instances.

Benchmarking: Show me the numbers!

Full details on the benchmark raw numbers the steps involved can be found at GitHub link

The following benchmarks were performed in us-central1-f region. Using N1/N2/E2-standard-4 instances. With N1-standard-4 serving the baseline for comparison.

Covering the following

Sysbench CPU
Sysbench Memory
Sysbench Mutex
Nginx + Wrk
Redis-Benchmark

Sysbench CPU

While its no surprise that a new hypervisor designed to share CPU's across multiple workloads would be "slower". A 69% reduction might be too much for most people to stomach.

However, that is not the only detail to keep track of

E2 also sees the largest variance in request statistics, across min/avg/95 percentile. This is in contrast to NX series benchmark (below) which these 3 numbers would be mostly the same.

Sysbench Memory / Mutex

On the plus side, it seems like E2 instances, with the newer generation of memory hardware and clock speeds, blow pretty much N1/N2 instance workload out of the water. By very surprisingly large margins.

So it's +1/-1 for now.

Workload benchmark: Nginx + Wrk, Redis-Benchmark

Unfortunately, despite much better memory performance. The penalty in CPU performance results in an approximate ~50% reduction in workload performance for even memory-based workloads.

Lies, damned lies and benchmarks

These numbers are just indicators for comparison between equivalent GCP instance types. As your application workload may be a unique snowflake, there will probably be differences which you may want to benchmark on your own.

Also this is meant for GCP to GCP comparison, and not GCP to other cloud provider comparison.

Note, that as I am unable to induce a live migration event, and benchmark its performance under such load. Until we can find a way to get data on this, let's just presume its in milliseconds? maybe? (not that it changes my review)

Pricing Review: is it worth it ??

In case that was not clear: NO

If it was a poorer performer at a lower price E2 would make for a compelling offer. This, however, is what's the confusing thing about the E2 launch.

While its marketing materials say "up to 30% savings". The reality is much more complicated.

Or would I even dare say misleading?

You see for N1/N2 instances, they receive sustained usage discount. That scales between 0-to-30% when used continuously for a month. With E2 instances not having any sustained use discount (as it's built-in).

So in a sustained 24/7 usage not only is the cost marginally higher, it has a much worse performance profile.

And unfortunately, it is this pricing structure which makes E2, a really cool tech with little to no use case. Especially considering that the VM overall capacity is expected to suffer from an approximately 50% penalty in performance.

If it's not clear, I made a table to elaborate on.

So unless you read through this whole review, and tested your application performance for it. Stick to the NX series of instances.

Overall Conclusion: Wasted opportunity

Internally for Uilicious "pokemon collection of cloud providers" we currently see no use for E2. And will be sticking with our N1 instances. As our main GCP server.

Despite all that, I do really look forward to the next iteration of E2, because as improvements are made to the hypervisor, and Moores law hold true. It would be about 2 more years where it outright replaces the N1 series, as the "better choice".

More importantly, what this technology opens up is a new possibility. Of a future instance type (E3 ?), where one could be paying for raw CPU / ram usage directly instead. For any VM workload. Making the previous optimization (preemptible, serverless) potentially obsolete.

Giving even legacy application developers a "have your cake and eat it too moment", where they can take any existing workload, and with no additional application change, with the benefit of "preemptible" instances.

If Google Cloud has not realized it yet, if done correctly they can make huge wins in enterprise sales, which they desperately need (aka the people still running 20+ year old software).

Till then, I will wait for GCP or another cloud provider to make such a change.

~ Happy Testing 🖖🚀

About Uilicious

Uilicious is a simple and robust solution for automating UI testing for web applications. Writing test scripts to validate your web platforms, can be as easy as the script below.

Which will run tests like these ...

Catfeeding: Uilicious testing inboxkittens XD

👊 Test your own web app, with a free trial on us today

(personal opinion rant) On GCP lacking the courage to leverage on E2 tech 😡

The whole purpose of E2 is to create a new dynamic migratable workload

So why is there even a preemptible option which makes no sense in almost any scenario when compared to other preemptible options?

Also isn't the very point of E2 series designed to help for long low-CPU usage workload, why does your pricing structure not favor it?

These are just the tip of the iceberg on Google confusing launched messaging.

If GCP removed preemptible discount option and made this new lower-performing line of VM's 30% cheaper, where it sits nicely between preemptible N1 workload, and higher performing sustained N2 workload.

It would have become a serious consideration and contender. However, without doing so, it's just cool tech, desperately trying to find a use case.

Sadly, and frankly, the only reason I would see why GCP is reluctant to do price cut for a new instance type is that they either

legitimately fear a massive price war with Amazon (which is famously known to be willing to out-bleed their competitors),
fear the new lower-priced product will eat into their revenue from existing customers.
Or worse, they just didn't think of it.

Considering that this is the same company that made serverless tech in 2008, way ahead of any of their competitors, and not capitalize on it. There is a good chance it's the last option (deja vu?)

All of this is disappointing for me to see, considering the massive amount of R&D and engineering resources went into making this happen. Which is a very classic google problem, really strong tech, that is disconnected from their business goals and users.

Finally GCP please stop abusing the phrase "TCO", or "total cost of ownership" - we sysadmin and infrastructure personal have the tendency to think in terms of months or years (you know the server's entire potential lifespan and total cost of ownership) ... Some of us actually find it ~~insulting~~ confusing when the term was used to imply savings when comparing to the existing long-running workload. When you actually meant to compare extremely short-lived workload instead.

We actually calculate server expenses, and such misleading marketing just leads us into wasted time and effort evaluating our options and making an article on it in the process.

~ Peace

So meetup.com is going to charge attendees in future - what's next for event organizers?

Eugene Cheah — Tue, 15 Oct 2019 18:48:16 +0000

Context.

Meetup.com was testing a new pricing model, where the attendee pays a $2 fee. The backlash was predictable, especially those organizing free events. And they have since reverted on it and started damage control.

For the original announcement see:

Discussion?

Should the tech community start looking elsewhere? If so, is there really a good alternative now?
Is dev.to considering stepping in?

Note that this has to be viewed from different angles.

For example, for a small new meetup (<10 attendees), not organizing on meetup.com could kill it.

For established ones, alternatives are viable. But limits their network effects. So it's now a pros/cons based on cost.

What are your thoughts on it?

How do you manage deployment configs? (Especially large scale cloud agnostic ones)

Eugene Cheah — Wed, 26 Jun 2019 09:40:42 +0000

Context

For the past few weeks, I have been using a crazy mix of the following. All within a git repository (infrastructure as code), as we migrate most of our infrastructure and workload from one provider to another.

terraform
nodejs
bashscripts
kubernetes yaml

With lots and lots of json, and yaml configuration files. Mostly generated from one system, to be piped to another (about 10k lines worth).

terraform which is commonly sold as a single solution for everything (it isnt), rapidly broke apart for us once we started doing deployments outside the usual AWS / GCP, and its poor support for kubernetes (which we use heavily)

So we started patching up missing orchestration with nodejs + bashscripts.

And now we have a giant soup of scripts updating other scripts configuration and applying them. Not exactly the most "elegant" solution.

The question : How do you do it?

So wondering out loud, for those who do really large scale deployments, especially with a small team.

Is it normal to always throw in the towel at the end, and code up a custom configuration management script to handle all this chaos? If so do you all normally do it in bash? or a custom application (like java)? or some other CLI scripting language.

Alternatively, is it normal to just grow a really large sysadmin team, each managing a subset of the system?

It feels like I am reinventing the wheel on these things, yet I somehow feel like there would have been a solution out there for this.

Sidetrack: a large part of me just feels like redoing terraform in nodejs out of frustration, to support my use cases.

Clarification on scale

I do believe there are multiple cloud specific offerings out there, the reason we do not use any of them is currently we run on the following list of providers.

gcp
aws
digital ocean
linode
alicloud
hetzner
bare metal on-premise stuff

Covering the following regions

singapore
unitedkingdom
germany-frankfurt
india-bangalore
canada-toronto
usa-newyork
usa-sanfranc
netherlands-amsterdam
indonesia-jarkata
hongkong
taiwan
+3 other data centers

The complex web of providers comes in part from the need to support regions where another provider either does not exist, or does poorly performance wise.

All to run UI tests at https://uilicious.com !??!

Are we "developers" gatekeeping "knowledge" from our juniors and peers? 🤦

Eugene Cheah — Wed, 05 Jun 2019 15:14:25 +0000

Background Context

My startup co-founder recently wrote an article on what I previously considered "common knowledge", an article which went "viral".

Unfortunately, as per the rules of the internet. Anything "viral" will draw a fair bit of criticism and even personal attacks, especially outside dev.to; Some which I saw hurt my co-founder personally, as we look through them.

However, this is not about sexism, javascript, nor the internet rule (all which has valid problems worth highlighting separately).

This is about "developers", and our role in gatekeeping "knowledge" like the following from others

Javascript Array.push is 945x faster than Array.concat 🤯🤔

Shi Ling ・ May 2 '19 ・ 8 min read

#showdev #javascript #webperf

Personal attacks aside, what caught my attention, that drove me to write this article - was the following comment:

Isn't it obvious that ________________ are you a _____,
Never ever write such [ ] articles again.

Side note: As its not my intention to shame/attribute the original author. The quote is paraphrased to make it un-googleable and to remove expletives. To be clear, the exact wording is different

It struck me not only because of how cruel it was at the end - but how the statement, was something I was guilty of saying as well.

SL: Really?! push made such a big difference
EU: Well concat needs to create a new array, and copy everything
SL: Doesn't the javascript jvm, automatically optimize such execution? Seeing that I do not use the old array
EU: Well it doesn't, and unfortunately due to the spec ...
SL: I should write an article on this
EU: Isn't it very obvious, I doubt it's worth writing
SL: It wasn't obvious to me
EU: (in my head - I am surprised you didn't know)
EU: Ok, it's worth a try then SL stands for Shi Ling, my co-founder. While EU represents me, Eugene

While it's much lighter in tone, the line of reasoning to not write the article is the same.

In the position of seniority, even lightly brushing off of such content being written can have chilling effects on other junior developers and peers. Especially in a much larger organization. So while such statements may not have deterred Shi Ling (she wrote it in the end), it could however for the rest of my peers around me silence them.

And was I wrong in my criticism!

Looking at the number of positive retweets and comments on how "I need to rewrite my code", or "I never knew", occurred. It wasn't obvious, and it is an article worth writing.

However, what hit me really hard - was when out of "frustration" or immaturity. I regrettably looked into the commenter's profile.

I realized it wasn't just a simple careless statement by a young teenager.

It was from a well respectable senior engineer, in charge of a large team of developers, with a similar background to me. To me, it was terrifying.

I was looking into the mirror, of what could be me in the future

And on further digging around the various comments, he was not the only one...

Which brings me to my next point...

PSA: What is common to you, may not be to another developer. So think twice before suppressing such content.

As an industry, programming is relatively immature, of being less then 200 years old (starting from Ada Lovelace). The web industry is even younger at being under 30 years old.

This pales drastically to

scientific medicine: ~2500 years old
architecture and construction: >5000 years old

As a result, the technology we learn seems to be constantly changing, with little in the way of a stable (think 12+ years) long term standard for anything.

By this definition: any assumptions you made on what is learned for your generation as "common knowledge" that isn't worth spreading, is a false assumption made to your junior and peers.

We should also kick out the notation that, if one does not "understand this", that individual is not deserving to be called a developer.

To clarify, my co-founder is by no means a "junior" developer. We both have very very different specializations.

While I may have really deep dive knowledge on the ES5 spec, and how V8 works internally, which distorted my view on what's considered "common knowledge" for javascript.

Inversely, she has really deep encyclopedia-like knowledge on the various quirks of CSS, and how modern front end frameworks like vue.js works. Something which I admittedly still make very fundamental novice mistakes till today.

So if my assumptions are wrong to a fellow "senior" developer with different specialization, wouldn't it be worse for a new "junior" developer. Especially with the rise of coding boot camps.

So until we can figure out a common stable universal engineering standard for programming, which we can assume as "common knowledge", like how architects or doctors do (which we probably will not for another 50 years). I am heavily ringing this bell now, as we are like no other industry out there.

Never make that assumption.

Heck, to the unconvinced old seniors. Many of us (including me) used to consider C style memory pointer manipulation as essential common knowledge, which in absence, we used as grounds for a "no hire" for juniors.

How wrong are we now in modern web development and the programming language they use?

Bro-grammer? Couldn't we be kinder?

Even with valid criticism...

Ask yourself, would you have possibly made the same mistake?
Would you like to have such things said to you, or your loved ones?
Is it constructive criticism?

Instead of gatekeeping what should be written or not written on "superior grounds". We could instead encourage improvement with our criticism.

For example, one repeated criticism. Is regarding benchmarking methodology. However instead of statements like.

Worst benchmarking ever

A kinder, more constructive comment would be

We could improve the benchmark by doing ____ instead of ____

And to say it upfront, for most of the constructive criticism on benchmarking, you were right. While it probably will not change the end result, due to the large difference in between. It is a valid area of improvement, and both my co-founder and I do thank you for that.

Or as XKCD lovingly put it

Saying 'what kind of an idiot doesn't know about the Yellowstone supervolcano' is so much more boring than telling someone about the Yellowstone supervolcano for the first time.

So which would you rather be?

Bro-grammer?

When I said this is not about sexism, I mean it.

The suppression of knowledge probably happens regardless of gender. It definitely happens from senior to junior as well.

However, the phrase "Bro-grammer" is used intentionally.

I am sure there are exceptions (there always is), but this is more on us the majority, the male gender of the development community, then the other way.

And while I will not have any statistical numbers to back it up.

The minority groups within our community know very well how it feels like to have their voice heavily suppressed.

The minority groups are the ones most mindful on not gatekeeping "knowledge" from others.

For almost all the personal attacks / mean criticism I seen to my co-founder was from our side of the male development community.

And it's on us to improve ourselves on this, to improve our community. Consciously, in speaking out against it. Or sub-consciously in our words and actions.

I am not asking each one of us to achieve an unreasonable perfectionist standard (as there will be slips of tounges), nor am I criticizing of who you are now.

I am saying this, as someone who has made mistakes as well. To make small incremental personal, mindful changes in us individuals that will allow us to push the community forward.

So ask yourself the man in the mirror? Will you make that change?

To improve the community around you. In encouraging the sharing of knowledge (as trivial as it may seem to you, or even me). Or to share such knowledge yourself.

I have seen similar things happen to other articles/tweets, anything I can do to help?

Instead of fighting head on, and "feeding the trolls".

In cases of unconstructive criticism (eg. worse benchmark), in place of the author, asked politely

What do you mean by _____, how could the content be improved on?

This gives the individual, a chance to clarify as they may not always mean harm intentionally, sometimes it could simply be words being lost in translation.

Just keep in mind to remain respectful, constructive and clear to defuse any tension

For really harsh comments or personal attacks, it might be better to reply to the comment, or author, while ignoring the bully. To let them know that there are others who appreciated their work. While lightly pointing at the criticism.

Personally I found the content useful, regardless of what anyone else says. Thank you

The above is universal regardless of content type (it can be applied outside of the development community).

And finally, for the developer who engages in such criticism, and partly the reason I wrote this article - is so I can point to it in the future for others. As it happens to them.

Personally I think what you are doing is great, regardless of what anyone else says. Thank you. And for the developers who say otherwise, we should be encouraging instead of suppressing our peers.

Are we "developers" gatekeeping "knowledge" from our juniors and peers? 🤦

Eugene Cheah ・ Jun 5 '19 ・ 8 min read

#discuss #beginners #motivation

Keep in mind while doing any of the above, as you run the risk of diverting the critical comment from the author and onto yourself. If it gets personal on you, do not feed into it. Be strong and ignore it.

Know you have done a part in letting the content creator, know that the criticism is not the only voice there. And as an observer to the receiving end, even if they end up in silence, I will say it helps go a long way in knowing there was someone there.

For all the unsung heroes, who did so. Thank you.

Keep doing your best!

To my co-founder, even if you face a double standard in your work previously, or the triple standards in investors as you fundraise presently, or the cruelty of the internet as you write content.

To all the other working moms, single moms, LGBTQ, and other minority groups.

keep on coding
keep on improving
keep on writing
keep on doing, the various great things you are working on
do not let the criticism suppress you
push that glass ceiling
and be great at what you do

Peace 🖖

Additional advice from comments

Also, consider reading up on recurse.com/social-rules.

Tim Downey • Jun 6 '19

Thanks for writing up this post @picocreator !

Along these lines, I am a big fan of the Recurse Center's social rules -- particularly in this case the one about "no feigning surprise." Their explanation of this:

Feigned surprise is when you act surprised when someone doesn’t know something. Responding with surprise in this situation makes people feel bad for not knowing things and less likely to ask questions in the future, which makes it harder for them to learn.

No feigning surprise isn’t a great name. When someone acts surprised when you don’t know something, it doesn’t matter whether they’re pretending to be surprised or actually surprised. The effect is the same: the next time you have a question, you’re more likely to keep your mouth shut. An accurate name for this rule would be no acting surprised when someone doesn’t know something, but it’s a mouthful, and at this point, the current name has stuck.

Feigning (or actually being) surprised is definitely not as mean spirited as some of the examples you referenced, but it can still discourage folks that are new to an area from speaking up or reaching out for help. Just wanted to note this as well since I think it's pretty easy for (otherwise well-intentioned) people to do this without meaning to.

And understand the difference between passive/active gate-keeping

K • Jun 5 '19

I think the problem is two-fold.

One part is the active gate-keeping. People running around telling everyone they or the tools they use aren't good enough.

How can you use PHP when there is Java? How Java when there is C#? Why switch to JavaScript when PHP finally became a real programming language?

Even my programming techniques professor said JavaScript was a toy and nothing compared to Java.

People shitting on each others' work for all kind of senseless reasons instead of helping each other out. That needs to stop and I have the feeling that more women and minorities in the industry will just lead to that. CoCs are spreading like a wildfire and people become generally kinder. The few old-school uber-nerds I saw here on dev.to didn't stay for long.

On the other hand, we have passive gate-keeping or simply ignorance. I guess that is what you're talking about. This can have many reasons and I can only talk about the ones I encountered.

I often had the impression I'm a bad-to-mediocre developer, so I didn't value most of the stuff I knew and this lead me to think I was one of the last in the industry to learn stuff. Took me a year to understand functions, took me a few months to understand closures, took me another year to understand monads, etc. I was always a slow learner, swimming with 9, biking with 14, so I guess that's where the impression came from that everyone else already knew the stuff I just learned.

I also had to work with many of such old-school uber-nerds who choose tech for projects and generally acted like they had it all figured out. Took me a few years of freelancing and blogging to find out most of them are just talking big and there are a whole bunch of people out there who don't know half as much as I do and would love to work with or learn from me.

History sidetrack

For those interested in the accuracy of the yearly figures used above.

1991 : HTTP protocol 0.9 was formalised
1975 : The rise of portable computer such as SCAMP / IBM 5100
1958 : Fortran/LISP, the predecessor to most modern programming languages.
1842 : Ada Lovelace, published the first known computer program
300 BC : Herophilos, the first greek anatomist to push scientific based medicine
3600 BC : Megalithic Temples of Malta, the oldest known free-standing structure on earth, ie not carved into a mountain

I have been sitting on this piece for some time, it is intentionally written in a way to not blame individuals for their past action, but to encourage them to reflect back on how to do better.

As writing is hardly my biggest strength, and I am sure my approach to this topic may be flawed as well. Any discussion, for similar experiences on this issue, either to encourage others, or to suggest other possible ideas, or even amendments that should be made to this article. Will be gladly appreciated.

⚙️ Explain Selenium & Webdrivers automation (Like I'm Five)

Eugene Cheah — Wed, 15 May 2019 07:42:31 +0000

This was originally drafted as an answer to the question ...

Explain Selenium & Webdrivers Like I'm Five

Edwin Mak ・ May 12 '19

#explainlikeimfive #selenium #e2e

That grew into an entire post on its own 😅 (I talk too much)

As I'm the co-founder of Uilicious.com - a full end-to-end testing platform built on webdriver servers. This post is meant to be a series of key facts I wish I knew at the start of the "webdriver" learning journey.

Oneliner to explaining Selenium / Webdriver

Automation standard, to control all your web browsers via an API.

Webdriver Specification vs Selenium Implementation

Think along the lines of HTML5 vs Chrome
Webdriver refers to the specification
Selenium refers to an opensource implementation of the specification.

Selenium is currently the most prominent implementation of webdriver, with several additional commands in selenium that are not part of the webdriver specification. Causing confusion on the distinction in between.

This is particularly important, to debug discrepancies in both documentation, for every browser.

However as they are used extremely interchangeably, it can get incredibly confusing in guides and tutorials. This is partially due to their mixed history together.

Understanding this difference will probably save you hours of documentation pain down the line.

Every child is a miracle

Webdriver Specification is a mini-miracle on its own

For those who have done work across multiple browsers, especially in the earlier years (10+ years ago). You will know how much of an impossibility it was to get a group of browser vendors, who are competitors to each other, agree on something together.

The problem for webdriver, is that it's made worse by being a relatively "low priority" item, with resource constraints. Being a feature that will never be experienced directly by the end users.

With every browser only having a single to a handful of developers working on their webdriver implementation.

The fact that we have a somewhat working protocol alone, in w3c, is a small miracle from the many years of hard work from the Selenium development team.

It may be at times incomplete and inconsistent among browsers, however, it is a spec that we can at least somewhat agree on, and be proud of since it started formalizing in 2011.

So a small round of applause for them 👏

I may be deeply critical, I do however greatly respect what they have done so far.

Choose any flavour

Webdriver is a HTTP API, with many client-side library implementations

You can choose your language/poison of choice here

HTTP API: W3C Specification
Javascript: webdriver.io (this is confusing as they use the same name)
PHP: php-webdriver
Ruby: Selenium::Remote::Driver

One side advice, like the early days of CSS/JS on multiple browsers, learn to distrust the specifications compatibility between different browsers/clients. Documentation is inconsistent and all over the place.

While there is MDN mozilla.org that will happily tell you what works (or not) on which browsers, for CSS/JS.

Webdriver as of now do not have an equivalent online resource. Internally this is something we may plan to spin out on our own in Uilicious, considering we are starting to pile up a mountain of random post-it notes on how the protocol differs for each browser.

Webdriver is a "Remote Control Interface"

The phrase "Remote Control Interface" is the exact quote taken from the specification.

Webdriver is built as an automation protocol, making it less than ideal for test automation as they have very different objectives. For example, webdriver has very limited assertion commands.

To use webdriver for testing, most users will use a testing solution, which extends the commands for end-to-end testing.

For example, to check if a text exists on screen in webdriver, you will need to run the following commands.

find the element where you expect the text to be
get the element and its inner HTML text string
check if text exists in the result text string

For uilicious (and a few other test solutions) we simplify it as a single command

I.see("Text You Expect to see on screen")

Need to run an army of safari tests? You may need one of these

Webdriver Server

Example: Selenium-standalone

Typically a webdriver server refers to a single VM / Physical instance of the desired OS of choice, with the "webdriver server" program installed.

Alternatively, this could include mobile devices attached to the webdriver serve, typically by a USB cable.

This will then run typically up to 1 session at a time, on a single browser. Where the automation will take place from.

Webdriver Grid/Router

Example: Selenium-grid

A dedicated server program, which connects multiple webdriver servers together for use on a single API endpoint. This is used to get around the limitations that each server will have, and run multiple sessions on a larger scale.

Along with user authentication, and limitations for access to the grid (to prevent any one user from running too many tests at a time)

Cloud providers for webdriver, Saucelab and browserstack provides such endpoints with devices as old as blackberries. Alongside the latest nightly build of chrome.

Test Networking

One common analogy we use in the office, is to think of uilicious commands as instructions we give over the phone. To the operator on the other side being the webdriver server. Executing commands such as I.goTo, I.see, and I.click

So from a networking endpoint, to test any website - the webdriver server on the other side must be able to load the given testing URL.

Likewise, just as how someone on the other side of the phone is unable to connect to a website hosted on your laptop on localhost directly. Neither can the webdriver server.

However in a similar situation, you are able to provide access through the usage of other tools such as ngrok, or port forwarding.

Hopefully that's enough key pointers for anyone new to webdriver who wish to dive deeply into it 🙂

So about uilicious, which is built on webdriver?

What I work on in uilicious, is to run test scripts like these

// Lets go to dev.to
I.goTo("https://dev.to")

// Fill up search
I.fill("Search", "uilicious")
I.pressEnter()

// I should see myself or my co-founder
I.see("Shi Ling")
I.see("Eugene Cheah")

And churn out sharable test result

Like the above. All through a fully managed cloud platform. https://uilicious.com
(Go give it a try 😉)

Happy Shipping 🖖🏼🚀

😼 KittenRouter: Opensource 🔍 Elasticsearch Monitoring, Logging & 🚦 Failover of webtraffic (with ☁️ Cloudflare Workers)

Eugene Cheah — Tue, 14 May 2019 14:11:24 +0000

Special thanks to Nai Jie for the logo (he also did InboxKitten logo)

Introducing KittenRouter - Serverless cousins to InboxKitten

Since we were using Cloudflare serverless kittens for our API. Why not expand this for the UI assets as well?

Log every web request on cloudflare (This is an enterprise only feature), helping us to track server failures.
Automated failover on server errors, to maintain overall site availability (High Availability!).
All as a reusable class module

Keeping the project small along the "do one thing and do it well" concept, the pseudo code could be summarized as below.

when receiving a request {
   for( each backend configuration ) {
      if (invalid backend configuration - wrong domain, etc) {
         continue - to next backend configuration iteration
      }
      perform cachable request to the backend
      log request to ElasticSearch backend - or alternative log provider
      if( 
         request result is valid || 
         is last configuration || 
         default backend configuration
      ) {
         return result - Ends request processing
      }
      continue - to next backend configuration iteration
   }      
}

# PS: The above is conceptually the synchronous flow. In the actual
# implementation, logging should occur asynchronously without blocking

This allows us to configure redundancies within the backend, with commons.host servers coming in first, followed by firebase as "backup".

Background context (why we made this?)

Why we migrated opensource 😼inboxkitten (77 million serverless request) from 🔥Firebase to ☁️Cloudflare workers & 🐑CommonsHost

Eugene Cheah for Uilicious ・ Mar 27 '19

#showdev #whywe #serverless #javascript

When we migrated our UI frontend from Firebase to Commons.Host (an opensource and public CDN/Static hosting providers), one of the issues we faced was random 404's in random remote locations. Like Russia - possibly due to the lack of servers there.

Commons.host, server locations at point of writing

And because we are supporters of the growing 🇸🇬 Commons Host project, we decided to stand by them together, instead of ditching them for alternatives.

We wanted a better way to debug such issues while improving the overall reliability of our service.

Side note to any big-brother authorities, we only track static assets loading with IP address masking on inboxkitten.com. No logging is done on the API. So yea, do not bother asking me for such data.

Elasticsearch Monitoring: Commons Host

What's awesome, is using the logging done by KittenRouter, of our commons host deployment, we can start having hard data on its status.

78 errors out of 12,069 requests ~ 0.65% failure rate

Is that currently in such a setup, we have 99.35% of our traffic routed through the free public servers of commons.host.

And for the remaining 0.65% of request which fails, They automatically fall back onto firebase infrastructure, under its free tier.

While it's not yet 9 nines, considering that these commons.host servers and cloudflare servers are scattered around the world. It's well within the 98% of real world user metrics of most CDNs

Furthermore, with the header information provided, we can keep track of the error rate on a per server basis. And with the KittenRouter setup on Cloudflare, you can configure such monitoring for any existing site without application code change.

We have since provided the commons.host team access to the above elasticsearch cluster for extracting useful data on their infrastructure under real user load.

Clarification notes, for privacy reasons, commons.host do not perform such level of logging on their systems for sites hosted on their platform.

The intention here for us at Uilicious, is to provide them with useful real-world usage metrics of their servers, and effectively exempting our website, inboxkitten.com, from such privacy protection, to help them improve their service.

Elasticsearch Monitoring: InboxKitten

Finally, we are able to easily use cloudflare country codes to help us keep track of our Kitten world domination map - live!. Meow-hahaha!

Greenland, and Madagascar - I got my ~~plague inc~~ InboxKitten eyes set on you!

Cool! Where can I get the code

For deployment and configuration details...

uilicious / KittenRouter

What is KittenRouter

KittenRouter is a routing script for Cloudflare Workers that attempts to connect to a list of specified servers and redirect the request to whichever server that is currently 'alive' at that point of time. It is extremely useful when you have servers that may go down or are unavailable to process the request and KittenRouter can automatically attempt to redirect the request to the next configured URL for processing.

At the same time, it can be configured to log down information to your ElasticSearch server for analytical purposes. Some of the information logged are the status of the servers, country of the request and etc. For the full details, see the index.js file.

How to use KittenRouter

Ultimately, KittenRouter is used together with Cloudflare workers. There are two ways in which you can use KittenRouter on your Cloudflare worker script,

Using NPM modules
Adding KittenRouter manually

1)

…

View on GitHub

or alternatively NPM

That's cool, but Why do I even need Inboxkitten disposable email for again?

One of the key use cases currently, and why we built this project, is to perform email validations as part of our automated test scripts. Such as the following...


// Lets goto inbox kitten
I.goTo("https://inboxkitten.com");
I.see("Open-Source Disposable Email");

// Go to a random inbox inbox 
I.fill("email", SAMPLE.id(22));
I.click("Get Mail Nyow!");

// Check that its empty
I.see("There for no messages for this kitten :(");

// Testing for regular email
// (sent using a jenkins perodic build)
I.goTo("https://inboxkitten.com");
I.see("Open-Source Disposable Email");
I.fill("email", "ik-reciever-f7s1g28");
I.click("Get Mail Nyow!");

// See an email we expect, nyow
I.see("Testing inboxkitten subject");

With sharable test results such as

Plus it's simple, cool, and fun to tinker around with.

What's next for KittenRouter?

Very rough roadmap of improvements needed for KittenRouter #3

PicoCreator posted on May 10, 2019

What's next then? For KittenRouter

More configurable backend options
- Request domain, path, region
- Request timeout
- Error / Passing HTTP code control
- Custom headers control
- Perform request as specified domain on specific IP
More logging options
- parameters logging whitelist/blacklist for
  - Headers
  - Cookies
More logging providers
- ???

View on GitHub

Let's see where this InboxKitten journey goes next... Till then we will be taking a cat nap!

Happy Shipping 🖖🏼🚀

🤔 UI framework wars & testing survey (of Startups & SME's in 🇸🇬 Singapore)

Eugene Cheah — Tue, 23 Apr 2019 08:30:34 +0000

Hey there,

For the past month, the team of Uilicious has been going around one north (our country startup hub), and a few other business hubs, knocking over 200 doors to survey the folks and find out what's the most popular front end frameworks used in production and the state of testing in Singapore 🇸🇬

Something that Uilicious as a company did on a much smaller scale 2 years ago, to get the general sense of things. And hopefully, plan to repeat for the next few years.

🤔 Sounds good? let's get to the results

UI Framework Wars - in production?

the endgame answer everyone wants to know on the ongoing UI "standard" war

Note that the total is more than 100%, as users are allowed to select multiple options

React was hands down the winner 🔥 when it comes to applications developed in production.

The emphasis on production is intentional. After all, with the recent changes and hype of newer frameworks it is important to remember that it is slow and tedious for production workload under daily usage and development to constantly stay on the hype train and takes a long time to migrate from one framework to another. With JQuery being a close 2nd place on the list or higher for easily the past 10 years 😴

Another surprising contender, Ruby on Rails. Originally not an option among the list, it had such a significant amount of responders in the "Others" free text option, that it was added onto the list.

Sad contenders for me personally, is Vue.js is not as popular as I hoped (disclosure, we are Vue.js supporters and help organize their meetups locally).

Another perhaps more surprising result: 0 response for polymer web components, which was an option in the survey. Probably because it's a little too bleeding edge for production 🚅

Development Team Size?

Of the teams surveyed, a good half has a development team size between 5-10. This is a good indicator that most companies have outgrown their initial startup days.

Who does testing?

Interestingly, despite the labor crunch of programmers, most companies heavily rely (48.5%) on their developers for testing.

This is worrying in two ways. One is that developers are typically not a good testers of their own code due to tunnel vision. Two, they are also the most costly option, with the average developer salary being easily double a dedicated manual QA salary 😶

However what was even more shocking 😱 was that a significant 15% of respondent indicated that they do not test, and this included large dev teams that are >10.

Followed by another 15%, who use their customer themselves, or their business stakeholders for testing.

Although some indicated that this is not a problem, especially those who serve as a solution vendor, this will lead to an unhealthy adversarial relationship between the development team and the business team.
Tension worsen when previously fixed bugs reappear through regression testing and the business team who in most cases, faces the wrath of the clients and the impact of any client business lost.

Something that ideally a good QA team or even a single dedicated QA tester can help manage, which thankfully 20% of the respondents had. So a small win here 👍

Side note: The results below, excludes teams without a testing process.

Unkown represent respondents who are not directly involved in the full testing process, and hence could not give an accurate answer

How much time does your team spend on testing?

While not shown in the chart, having a dedicated QA helps bring the overall development team time spent on testing downwards, with most in the 0-10%, and 10-25% category 😃

Insufficient test coverage tends to be the main issue for teams who spend more than 25% of their time testing. Along with the practical time-consuming nature of manually performing testing on various devices and screen sizes ⌛

This is further compounded if the developers are the testers, as it is not an effective use of their time.

How often do you test your application?

A large 40% of teams seem to be running on a weekly or bi-weekly release cycle of development and testing.

Followed by as and when deployments are done on an (unknown schedule), which is typically once a month or less.

Finally a round of applause for the rare 3% who either automate their test to run every single workday or on every commit 😎

What tools do you use to test?

Despite the whole growth of high tech web apps using the next buzz from AI to the blockchain. Testing is happening in rather low-tech ways, manually for all these apps 🤐

One thing heartening to see is the rise of test automation at 27%, this is significantly higher then the ~5% result we saw two years ago. Perhaps indicative of a slowly maturing industry.

How often do you get a bug report from customers/business team?

A particularly painful question for most. From what it seems, daily and weekly bug reports seem to be the daily grind faced by 75% of the teams.

Something which almost every dev dread to get from a customer 😭

What is the biggest challenge for testing?

Finally, there is the open-ended question on what is the biggest challenge teams faced for testing.

Unsurprisingly, communication is still the number one challenge at 38%. A great shout out to The Mythical Man-Month for those who read it.

Miscommunication of requirements ranges anywhere between any of the following: Customers, Business Team, Developers, Designers, Managers and Testers.

This is followed by the lack of manpower in the current talent crunch (27%) and the scale of web testing various screen sizes, pages, and browsers (18%). All of which is something Uilicious as a company will be glad to help with.

And finally the startup classic, no budget. At 12%, this never changes 😫

The following are personal observations and opinions regarding the survey. AKA. it's not backed by hard data.

Personal observations on testing

What we found interesting was that most companies surveyed either have their entire development team or the majority of the team outsourced overseas, an indicator of the ongoing programmer manpower crunch in Singapore 🤔

A worrying observation, from the various interview, was that most teams who test manually, only tend to do so for new features. Without any regression check. Or more scarily, those that do no testing.

This is understandable for extremely early-stage startups, who just started. But, several were not in this stage, including larger "Series-A/B" level startups.

This is worrying personally as a consumer of the modern web, as this would explain many times what has happened when extremely obvious bugs appear when using an application (such as login and checkout), where simple regression testing is missing.

It also raises questions in me, if a company is not willing to invest any resources into testing functionality, what resources are in place to ensure the security of their system? Especially considering my own personal data on some of these applications.

Something I hope will be improved on in time 😧

Personal observations on UI framework

On the UI framework war side of things, many responders during the interview process mentioned plans to experiment and move away from React onto other frameworks for new projects or as part of their migration plan. So perhaps next year React will slowly topple down?

bias mode engaged

Go Vue, go 🏎️

So how can Uilicious help make this better?

Uilicious.com testing platform

First, we are a test automation platform aiming to remove all the pain points in website testing (hence the survey). And we are constantly iterating based on feedback.

To get users up and running quickly, our platform allows testers to easily write test scripts like these ...

// Lets go to dev.to
I.goTo("https://dev.to")

// Fill up search
I.fill("Search", "uilicious")
I.pressEnter()

// I should see myself or my co-founder
I.see("Shi Ling")
I.see("Eugene Cheah")

And churn out sharable test result like these ...

Uilicious QA-as-a-service

Second, we have since then soft launched Uilicious Services.

As made obvious in the survey result. Most teams lack the resources required to form their own dedicated Quality Assurance team. And that is something we can help with, from gathering testing requirements to the writing and maintaining of tests.

If the above testing challenges in the survey apply to you, let us know and we will be glad to help with our Uilicious testing service, to get your full QA process up and running with our experienced QA engineers.

Also once the team grows large enough to have their own dedicated QA team, they can take over the role of writing and maintaining test scripts on the platform.

We hope that through the testing service, we can help teams to convince their project managers on to improve the overall quality of their website and product 😉 and get dev teams up and running faster, with their projects, alongside our test platform.

bias mode dis-engaged

What should we improve on for next year survey?

One notable thing: is that due to the current survey size (~200), and the nature of how the data is collected (almost entirely at one-north). There is a heavy bias towards smaller startups. So take the result as vague indicators at best. Something we hope to improve on in the future 😵

Another thing to note - due to the nature of how this survey is conducted, the result greatly underrepresents the larger development teams.

Many of the much larger startups (who probably have a larger development team) would have an office manager at the entrance, turning away any survey requests 😥

When we repeat the survey next year, we will ask for both the local and overseas team size separately, which would give some insights on the rate of outsourcing in Singapore for developers.

Similarly, there are clear obvious differences in testing requirements for solutions vendor, or a web company. Something we would try to add next year. Along with the question of "how old the company is".

So if you have any ideas on how we should improve the survey, do let me know.

Till then, as a startup founder myself. I gotta get back to investors and fundraising to grow Uilicious, so gotta go now...