Forem: hiroyone

The Hidden Mindset and Skills of Japanese 10x Developers

hiroyone — Sun, 07 May 2023 14:29:17 +0000

In this post, I reveal the undisclosed information regarding Japanese 10x developers that I had the opportunity to work with during my five-year stint in Tokyo. Furthermore, I share the valuable insights they imparted to me, which helped me improve my productivity to a similar level as theirs.

Introduction

The software company I used to work for in Tokyo was distinct in two ways.

Firstly, their engineers were full-stack developers who also specialized in specific domains such as backend, frontend, machine learning, security, network, cloud architecture, sound, game, and so on. They worked on specific domains for their projects, but also possessed general knowledge and interest in other software fields. The company believed that limiting engineers by job titles would impede their growth, and having full-stack engineers would be more beneficial for projects.

Secondly, the company had a purely flat organizational structure, unlike other software companies that have hierarchical distinctions among junior, middle, senior, and staff roles. The only difference was the CTO, who was a co-founder. Although the company had 400 full-time employees and a few hundreds contract engineers, which caused some disorder and chaos, it generally functioned well.

How 10x developers are cultivated and discovered

How can a completely flat and somewhat chaotic organization cultivate 10x developers, and how are these exceptional developers identified without a traditional corporate hierarchy?

The full-stack software engineers at this company must survive the development of production-grade, consumer-facing web services across multiple domains, including frontend, backend, and infrastructure. The blurred lines between these domains encourage engineers to understand and seamlessly integrate the entire system flow. This results in most engineers becoming battle-tested and highly skilled after five years of work, and 10x developers typically remain at the company for over ten years.

At the same time, engineers identify 10x developers solely through word-of-mouth and reputation, as many engineers work on multiple projects at once. Respect is earned not through seniority or title but through exceptional results achieved in a shorter time than their peers. In other words, 10x developers make a significant impact on projects in terms of software architecture design, coding quality and quantity, and debugging.

Traits of Japanese 10x developers

The attributes listed below are commonly associated with Japanese software developers who demonstrate exceptional productivity and proficiency in their work. These traits are not limited to engineers within my previous company, as they are often mentioned in Japanese online forums.

Passionate about coding: They write 500 to 1000 lines of code per day because they genuinely enjoy coding.
Genuine geeks (Otaku): They graduate from technical high schools or colleges and are heavily influenced by the Otaku community. They are introverted and enjoy games, anime, and comics, but dislike traveling long distances.
Proficient in multiple programming languages: They are skilled in writing code in all kinds of programming languages, including interpreted, compiled, frontend, and backend languages such as Python, JS, Java or Kotlin, Golang, Rust, etc.
Excellent problem solvers: They excel not only in writing code but also in debugging problems. They are a combination of all kinds of analysts.
Anonymous: They rarely appear in public and prefer to remain anonymous. Their real names cannot be found on their presentations or blogs, but they are active on Twitter using nicknames.

Lessons I learned from Japanese 10x developers

I have discovered two basic concepts that describe the effectiveness of 10x developers:

Concentration
Knowledge and experience

Routines for Concentration Enhancement

Time Limit: Instead of working all day, it's better to set specific working hours, which is 8 hours for most people. This is because when there is a time limit for a task, concentration is improved. This can be compared to body training, where trainees focus on an exercise for a specific duration, such as an hour, to achieve maximum concentration. Example: One Japanese 10x developer arrives at the office on time and leaves on time or early if all scheduled tasks are complete. He also cleaned up his computer state and sorted out his backlogs before leaving the office so that he can straightly start working on his tasks the next day. Tip: It's also possible to extend the working time by 10 to 15 minutes to complete one more task, similar to doing a few extra reps during a body workout to achieve maximum results.
Something small, easy, and simple: Begin with small, easy, and simple tasks rather than heavy and complex ones. This is similar to how one would start a bonfire by burning leaves and papers before adding wood. Example: Japanese 10x developers usually begin their day by either addressing minor bugs or checking their Slack app. Tip: It's recommended to divide a large task into smaller portions ahead of time.
Quantity measurement: Quantify the achievement like a game. It is good to set key quantitative metrics to measure achievements. Example: Japanese 10x developers kept track of the number of pull requests, closed issues, cost-saving amounts, etc.
Least Best Practices: Aim for results that are not necessarily considered "bad" but may be less effective or efficient. They know avoiding anti-patterns is more crucial than achieving the best solution. For instance, the best code is subjective, contextual, and hard to achieve most of the time, which slows the way of thinking and working; meanwhile, the poor code is self-evident and easy to spot. Example: Japanese 10x developers usually receive a few coding suggestions for their pull requests. These suggestions are nice and good to accept, but none of them is a critical problem in terms of algorithm and security.
No distractions: Have no distractive tasks for getting straight to work for a day. Have as little stuff to do in the morning as possible such as cooking breakfast and commuting. Example: One of the Japanese 10x developers told me he prefers to wake up early and prepare early as fewer meetings and inquiries occur in the morning.

Knowledge and Experience gains

All lessons below are unsurprising, but many engineers tend to forget them after a while.

Continuous learning: Keep learning in addition to working hours. Technology is evolving rapidly every day, and learning is inevitable to enhance productivity in the long run. Example: When my previous company decided to add a new cloud service provider, a Japanese 10x developer learned the related material and got the provider's cloud certificate within a month. However, he was already an expert in another cloud service and had plenty of projects using that cloud service.
Just write and debug it: Most software engineers start their careers by writing, testing, and debugging code which requires a few years to get handy with. As their work is approved and respected, they are more involved with high-level technical designs and discussions. But 10x developers never forgot to do the most foundational things: write code and debug it. That is because they innately like pure programming, remember their first motivation when they were young, and hate to get their skills rusty. Example: Japanese 10x developers write 500~1000 lines of code almost every day. I was stunned by the number of new features and bug fixes they made through pull requests.
Opportunity seeker: The fast-paced technological advancements call for project and product managers to plan many new and innovative projects. These projects can be tough for developers, but they offer an opportunity to learn new skills and gain experience. Example: Most Japanese 10x developers are nerds and introverted, so they generally avoid talking and discussing with others. But when it comes to new technical challenges and adoptions, they start raising voices and expanding their opinions, especially for cool technologies.

Conclusions

The 10x developers' traits and skills explained above come from my five-year working experience in Japan. But most essay contents are paraphrased by Chat-GPT, which is probably 1000x more productive than human engineers.

Cheers to 10x developers and the cutting-edge AI technology (^_^*)bbb

Priority vs Outputs: why 10x developers are valuable in software development projects

hiroyone — Fri, 07 Apr 2023 15:25:36 +0000

Unexpected Project Failures

Until recently, I was working at a large software vendor in Japan for five years, in which we offered web/tech solutions to many non-tech clients.

There, I observed several big projects that endlessly continued without seeing a single product release and the candidate release date kept postponed for a few times even though the team consists of experienced project managers, product owners, and average developers. Weird!

I was lucky to witness another type of project which managed to release new products and features even though
the team only had an inexperienced manager who was also a former developer but had one 10x developers as well as other middle-level developers. Their project management had some mess and the overall quality of the product experience had room for sophistication, but they kept making progress.

Observing these two different types of project's outcomes left me with an important lesson to learn, whose value and practices are still worthy of attentions in most projects nowadays.

I compared these two types of projects attentively and gave thoughts on what characteristics had a significant impact on better progress. In the end, I found two keywords which are surprisingly related.

The opposite of prioritization

To begin with, can you answer the following question with confidence?

Quiz: What is the opposite of prioritization in the context of a software development project?

I guess most people would never thought about it.

Short Answer: Outputs

Outputs here mean the amount of technical design, implementation, and testing you have attained in a certain period of time: namely, one day, one month, or one quarter.

It would be surprising for you to know these two keywords are correlated in the context of software development and actually mean the opposite. But I guarantee that achieving high outputs will lead projects to finish.

Prioritization, Prioritization, Prioritization, BUT...

I bet you have encountered the situation where a project manager repeatedly emphasised the importance of prioritizing tasks - sorting out the tasks in the urgent order.

And for the sake of better prioritizations, more communications were encouraged among project managers, engineers, and stakeholder. Hence, teams increased the number of meetings to once or twice every day.

Prioritization has pros but also cons! It is not a silver bullet.

Pros:

Prioritization makes a team focus on finding critical problems for users, identifying why these problems occurred, and quickly fixing them with a few lines of code in an urgent manner. This type of management style is suitable for already released and maintenance products.

Cons:

The problem with this prioritization-centered approach appears vividly when a lot of new features are to be developed, and more foundational work is needed with a lot of effort. But both prioritized task executions and solid foundational work are hard to achieve simultaneously for the following reasons.

1. Prioritized tasks vs development order

Tasks sorted by prioritization are often less efficient for development productivity because prioritized tasks pay more attentions to something that are visible to users/stake holders, while it would be actually easier for developers to start with the foundational part.

I prepared an intuitive illustration of pyramid construction below, a visible analogy to software construction.

As you can imagine, constructing from visible parts requires more efforts for each block and is not robust. It may even sounds ridiculous but this is often a reality.

On the other hand, constructing pyramid from the foundation can make more progress with the same effort, but would be less satisfactory to users.

As a result, for the same amount of effort, construction from the foundation can be 12 blocks ahead of construction from what are important to users.

2. Context Switching Cost

Frequent prioritizations must come with the context switching of tasks for developers, but it comes with a high cost, as many experts have already suggested (e.g. this article). Developers outputs will decline sharply if their priority tasks are changing even a few times a month.

3. Development Mindset

Priority-focused teams tend to forget the value of foundational work and only do minor fixes on the surface to solve urgent problems. But the system internals becomes messier with the accumulated temporary fixes, which deteriorate the long-run productivity and extensibility.

Project Manager A: Documentation is of course important but we don't have time to fix this document now. It is not prioritized.

Engineer Lead B: Such a feature is desired for the safe delivery of this service but we are short of hands and cannot spare time for it now.

When conversations like the above are heard in your project, that is an indicator that the importance of the foundational work is ignored and the product's development productivity is impaired in the long run.

Why outputs of work is important in a project?

So how can we handle the other type of development?; there is no strict deadline for each task, but the entire roadmap is heavy and long because a lot of new features are to be integrated into the product.

As you may know, the more components are involved in one product, the more fine integrating works are needed. Interface adjustments get severer, integrated testings get to be more important, more integrated documentations are desired as well.

That is why MAX OUTPUTS Strategy comes into play. Contrary to the frequent prioritizations, the project focuses on the amount of outputs team members produce each day. e.g., pull requests, issue tickets, and design documents.

Since there is no strict deadline, members can focus on laying the groundwork for the product - building the components from the bottom layer one by one, which makes the product robust and stable in the long run.

While the main motivation for prioritized work is urgency, the essential motivation for this type of development is development tempo, which sets the basic amount of outputs produced everyday. Because tasks have somewhat similar traits in terms of volumes, qualities, and flows, developers aim to attain the same quantity of outputs everyday.

For the sake of MAX OUTPUTS of team members, more quiet working time is introduced. Each member focuses on their assigned development tasks and they are encouraged to produce more outputs. Hence, teams tend to show their work by commit logs, pull requests, or testing outputs.

Why 10x developers are valuable in software development project

The above comparison between prioritization and output makes it easier to understand why an inclusion of 10x developer matters to the overall progress of a project.

A lot of software development are actually expected to develop new features rather than just maintenance the service. And thus output of the project is relatively more important than fine prioritizations.

As I said in the beginning, I saw some large projects fail, which put emphasis on prioritizing tasks and fixing some hot issues appearing on the surface but ignored the importance of building the vast volume of (foundational) works for the project. I believe that this happened because these project managers did not have large software development experience in the past.

At the same time, I also witnessed the participation of 10x developers in a project often solved the project stagnation. They are skilled and mostly quiet in team meetings. They solely focus on adding features and fixing bugs to reduce the issue tickets. They are also good at scaffolding the foundation on which other developers can build components safely, smoothly, and neatly. Yet, their values are often underrated because they prefer hiding behind others and keeping their contribution as secret as possible.

The below picture is another pyramid analogy with 10x developer to project.

What can we do as a project?

It is impossible to suddenly hire 10x developers tomorrow, but it is possible to kick off the following strategies to achieve similar results 10x developers bring.

Recognise among team members that output is as important as prioritization in a software development.
Set no meeting day once or twice a week.
Avoid bringing urgent tasks to developers.
Encourage working early or late for developers so that they can secure more quiet time to focus on development.
Encourage developers to keep their development tempo. Split tasks into small pieces and set the goal for weekly achievement. E.g., the number of solved issue tickets and pull requests.
Have morning stand-ups among developers to share their outputs yesterday.
Visualise the outputs of members. Simple bar graphs or line graphs are good enough to show members' progress.

These team-wide MAX OUTPUT practices will enhance the team's overall progress and software stability in the long run.

Importance-Urgency Matrix for Engineers

Planning the allocation of outputs based on the importance and urgency of tasks matters in real projects. The diagram below illustrates how much percentage of outputs should be allocated to each task type.

As you can imagine, enhancing the outputs of Not Urgent && Important tasks plays a vital role in the future landscape of the project; The more systemic automation brought from these important works, the less urgent tasks team members will face and thus can spend more time on more development and testing of important features. And the positive spiral will continue.

Thank you for reading my article.

Happy Coding!

Why 10x developers really matter in software development project

hiroyone — Mon, 06 Mar 2023 07:08:33 +0000

Unexpected Project Failures

Until recently, I was working at a large software vendor in Japan for five years, in which we offered web/tech solutions for many non-tech clients.

There, I saw several projects that endlessly continued without seeing a single product release and the candidate release date kept postponed for a few times even though the team consists of experienced project managers, product owners, and average developers.

I was lucky to observe another type of project which managed to release new products and features even though
the team only had an inexperienced manager who was also a former developer but had one 10x developers as well as other middle-level developers. Their project management had some mess and the overall quality of the product experience had room for sophistication, but they kept making progress.

Observing these two different types of project's outcomes left me with an important lesson, whose value and practices are still worthy of attentions in most projects nowadays.

I compared two projects carefully and gave thoughts on what characteristic had significant impact on the progress. In the end, I found two keywords, which are surprisingly related.

The opposite of prioritization

To begin with, can you answer the following question with confidence?

Quiz: What is the opposite of prioritization in the context of a software development project?

I guess most people would never thought about it.

Answer: Outputs

Outputs here mean the amount of technical design, implementation, and testing you have attained in a certain period of time: namely, one day, one month, or one quarter.

Prioritization, Prioritization, Prioritization, BUT...

I bet you have encountered the situation where a project manager repeatedly emphasised the importance of prioritizing members' tasks - sorting out the tasks in the urgent order.

And for the sake of better prioritizations, more communications were encouraged among project managers, engineers, and stakeholders, and hence teams increased the number of meetings, to once or twice every day.

Prioritization has pros but also cons!

Pros:

These teams are good at finding a problem from users' perspectives, identifying why the problem occurred, and quickly fixing it with a few lines of code in an urgent manner. Thus this type of development style is suitable for patching already released and maintenance products.

Cons:

The problem with this approach appears when a lot of new features are to be developed, more foundational work is needed with a lot of efforts. But both prioritized task executions and solid foundational work are hard to achieve simultaneously for the following reasons.

Prioritized tasks vs development order

Tasks sorted by prioritization are often less efficient for productive developments because prioritized tasks focus more on something that are visible to users/stake holders while it would be easier for developers to start with the foundational part.

I prepared an intuitive illustration of pyramid construction below.

On the other hand, constructing pyramid from the foundation can make more progress with the same effort, but would be less satisfactory to users.

Context Switching

Frequent prioritizations must come with the context switching of tasks for engineers, as many experts already suggested it like in this article. Engineers outputs will decline sharply if their priority tasks are changing even a few times a month.

Development Mindset

Priority-first people tend to ignore hard work and end up only doing minor fixes on the surface to solve the urgent problem. But the system internals become messier with the accumulated temporary fixes which deteriorate the long-run productivity and extensibility.

Project Manager A: "Documentation is of course important but we don't have time to fix this document now. It is not prioritized."

Engineer Lead B: "Such a feature is desired for the safe delivery of this service but we are short of hands and cannot spare time for it now."

Why outputs of work is important in a project?

On the other hand, the other type of development has the opposite characteristics; there is no strict deadline for each task, but the entire roadmap is rather heavy and long because a lot of new features are to be integrated to the product.

While the main motivation for prioritized work is urgency, the essential motivation for this type of development is the development rhythm, which sets the basic amount of outputs produced everyday. Because tasks have somewhat similar traits in terms of volumes, qualities, and flows, developers aim to attain the same quantity of outputs everyday.

Why 10x developers really matter in software development project

The above comparison between prioritization and output makes it easier to understand why an inclusion of 10x developer matters to the overall progress of a project.

What can we do as a project?

It is impossible to suddenly hire 10x developers tomorrow, but it is possible to kick off the following strategies to achieve similar results 10x developers bring.

Recognise among team members that output is as important as prioritization in a software development.
Set no meeting day once or twice a week.
Avoid bringing urgent tasks to developers.
Encourage working early or late for developers so that they can secure more quiet time to focus on development.
Encourage developers to get the development rhythm. Split tasks into small pieces and set the goal for weekly achievement. E.g., the number of solved issue tickets and pull requests.
Have morning stand-ups among developers to share their outputs yesterday.
Visualise the outputs of members. Simple bar graphs or line graphs are good enough to show members' progress.

These team-wide MAX OUTPUT practices will enhance the team's overall progress and software stability in the long run.

Importance-Urgency Matrix for Engineers

Planning the allocation of outputs based on the importance and urgency of tasks matters in real projects. The below diagram illustrates how much percentage of outputs should be allocated to the each task type.

Thank you for reading my article.

Happy Coding!

Only using the iPad as my main machine, here's a review of software development experience

hiroyone — Sun, 03 Jul 2022 03:14:02 +0000

Motivation

I’ve been using the iPad exclusively as my main computing machine since June 2022. (This post is, of course, written on my iPad.)

Before then, I relied heavily on a Macbook Pro 13-inch at my previous office and didn’t have my home computer for the last two years after I gave my one to my friend’s son who aspired to study online.

I do serious software development with Python, Typescript, Kotlin, Golang, and Rust. And thus, I was wondering if I could do serious web development with the iPad.

After a quick online research, I found that even programming seems possible on the iPad with its keyboard as well as well-known touch-screen creative activities. So, I bought and tried using the iPad for light development and learning experience.

Spec Details

11-inch iPad Pro Wi-Fi 128G
Magic Keyboard (Not Smart keyboard)

The 13-inch screen was a bit too heavy to carry, and Smart Keyboard was too thin for a heavy typing workload.

What happened in the first month?

1st week: programming seems impossible!! Just watch Youtube!
2nd week: a code fix is possible, but serious devevelopment is not.
3rd week: entire software development works with cloud IDEs and cloud services! AWS, GCP, Gitpot!

Pros

Stateless! No dependency and data are left in the local computer!
It is so lightweight that I can be anywhere with it! I flew to a couple of vacation places and enjoyed coding at hotels and cafes.
The touch screen is terrific when I want subtle and interactive maneuvers. For instance, I showed my journey map to my friends.

Cons

Need to know alternative tricks to local machine development.

No terminal (use virtual machine such as ec2 and gce)
No local Git (use cloud machine or Working Copy)
Limited local IDEs (Gitpod, Stackblitz, Code Server)

Overall

Coding is definitely possible
Serious software development is limited due to the heavy reliance on cloud IDEs and cloud services. You need to bear with its inconvenience in exchange for lightweight device benefits.
This setup is not recommended for beginners. It is better to learn computer basics with standard computers.

Cloud IDE articles

A guide to writing code on an iPad
We tried 10 online code editors

Don’t prioritize speed for e2e testing but consider smarter test schedules

hiroyone — Mon, 27 Jun 2022 01:33:43 +0000

End-to-end(e2e) testing such as by Cypress and Playwright is considered slow compared to unit/integrated testings such as by Jest.

In this post, however, I would like to argue that a few devOps practices can dodge this slow performance issue in the devOps cycle and the developers can take fully advantage of e2e based quality assurance.

Why is the slow performance considered an issue, by the way?

Testing performance is usually considered significant because developers want to include it in the CI/CD cycle, which should get completed as fast as possible.

It usually takes more than 30 secs to execute one e2e testing for one web page. And thus, it would take more than 5 mins when e2e testing covers more than 10 pages.

Common CI/CD pipeline

Make a pull request on a feature branch
Build/compile the application
Run tests
Merge the new branch to the main branch
Deploy the new application to servers

Due to the popularity of the above CI/CD pattern, front-end developers struggle to include e2e testing in Step 2. However, it means that the testing tightly binds with its development & deployment flow.

Problems

Every new push to the remote repository runs entire new tests. Too much waste of computing resources.
New development/refactoring/configuration easily breaks some of the tests. And we want to fix broken tests at a different time.

Run entire tests only in specific situations

Include static test (eslint/prettier) and unit test (Jest) in Step 2, but not entire e2e tests.

Of course, it is necessary to run prepared e2e tests before releasing the application to the production environment. But Git tags usually come into play for controlling a production release these days. In fact, the Github action supports tag based actions. Thus, we can separate the CI/CD phase and the production release phase.

Run e2e tests for only affected parts

We can search modified files by git commands on a feature branch and run e2e tests to these files only on push to the remote repository.

So, for example, if a developer modified pageA.js, pageB.js and pageC.js, the CI/CD pipeline runs pageA.jstest.js, pageB.jstest.js and pageC.jstest.js

git ls-files | -I % /bin/bash -c '<test-command> %test.js;'

Or you can manually run tests regarding the affected pages and upload the test results to the pull request description as a part of the internal regulations.

Run all tests in parallel

Most e2e automation tools such as Cypress and Playwright support running tests in parallel.

Each test should not be too lengthly because parallel processing mechanism is file-based. If the test file length is more than 200 lines, it would be wise to split the test file into two.

Split your e2e testing into interactive testing and scenario testing

hiroyone — Sun, 01 May 2022 08:59:21 +0000

I have been confronted with end-to-end(e2e) front-end testing in my enterprise project for the last couple of months and found a few practical tips worth developers' attention before starting their front-end testing. Let me share one of them in this post.

Split your e2e testings into interactive testing and scenario testing

Interactive testing (or integrated testing) means any behavioral testing within a specific page, such as opening/closing a modal window and binding input data.

On the other hand, scenario testing ensures successful page transitions and data consistency between pages. For example, a purchase scenario testing starts from the Top page a user enters to the Thanks page where the same user exits.

1st Reason: Avoid Flaky Results

There are two main motivations for splitting these two tests. We can avoid flaky results and separate SSR and CSR pages.

First, it would be too challenging to avoid flaky (unstable) behaviors if you mix both interactive and scenario testing into one extensive testing.

In general, scenario testing tends to be flakier than interactive testing because it involves page transitions and data transfers, while interactive testing involves only one or a few components.

Moreover, it is usually not until the finished phase that the scenario testing starts to pass stably. In contrast, interactive testing will pass as soon as developers finish implementing a few components, and they are less susceptible to external data and web services.

2nd Reason: Mock data v.s. Real data

Lastly, interactive testing goes better with mock data because we want to test a variety of component behaviors including both normal and edge cases. Most e2e testing framework supports such API response interceptions.
On the other hand, scenario testing fits better with real API data as data consistency is the key part.

Interactive Testing → Mock Data
Scenario Testing → Actual API Data

3rd Reason: Separate SSR and CSR pages

Secondly, interactive testing within a specific page can test SSR behaviors, while scenario testing across pages can test CSR behaviors. Developers cannot be ignorant of these differences, or runtime behaviors would fail in untested situations.

Decoupling into two types of testing enables us to run these tests independently, helps us develop e2e testing in a predictive manner, and analyze the cause of testing failures.

Testing Directory Structure
It is more maintainable to have the same folder path and file names in the in the testing folder as the routes/pages folder.

For instance, cypress/integrations/top.spec.js tests all interactive cases in the Top page.

And interactive testings are stored under the integrations folder (instead of interactive folder) because Cypress has this folder by default.

Here is a sample folder directory if you choose Cypress.

cypress
 ├ integrations
 | ├ top.spec.js
 | ├ search.spec.js
 | ├ product.spec.js
 | ├ cart.spec.js
 | └ order.spec.js
 |
 ├ scenarios
 | ├ purchase
 | ├ inquiry
 | ├ cancellation 
 | └ content-reader
 |
 ├ plugins
 ├ fixtures
 └ others (downloads, screenshots, etc.)

That is it. I hope this tip will help you plan your e2e testing.

Thanks!

Don't develop your web apps with full SSR but use hybrid SSR/CSR approach!

hiroyone — Sat, 16 Apr 2022 12:14:18 +0000

Nowadays, many frontend web apps are using Nodejs based frameworks such as Nextjs or Nuxtjs. But some of their large-scale web apps are suffering performance issues. Here, I am going to explain one of the system design topics frontend engineers would face.

Lesson: Don't develop your web apps with full SSR but use hybrid SSR/CSR approach

First of all, You can learn the concepts of SSR, CSR, SSG, and ISR in the following article.

Visual Explanation and Comparison of CSR, SSR, SSG and ISR

Furthermore, full Server-side rendering (Full SSR) is to generate web content on the server at every request time. On the other hand, most frameworks also support a hybrid SSR/CSR approach, which produces web contents on the server at the first request time but generates other pages on user browsers via internal page navigations.

Some people would consider that the full SSR is better than hybrid SSR/CSR for security, performance, and SEO reasons as long as they can bear the increasing server cost. But this is almost surely false because Nodejs, the base of Nextjs and Nuxtjs runtime, is a single-threaded application.

It would be best to read this official article to learn how Nodejs takes advantage of single-threading.

Don't Block the Event Loop (or the Worker Pool)

In short, a single-threaded application is precisely like an excellent director. A Nodejs director can receive tons of tasks from clients (browsers) one after another, give instructions to his backend workers, and receive the task outcomes. Unfortunately, he is not a good worker, and it isn't nice to assume he can do both direction tasks and backend work. Thus, any computation-intensive jobs such as full SSR or large JSON parsing would lead to the tremendous latency of the entire application response.

You can also read a well-written story from Maaspteh about his struggle with making his full-SSR web app performant.

NodeJS and React SSR, the need for foul play

It is safe to assume that hybrid SSR/CSR applications with moderately optimized code can produce 100 to 200 requests per second at the stress testing, but the full SSR site can bring out only 16 to 50 requests per second. Alas!

Instead, you should fully take advantage of the nature of single-threaded applications in the following way.

Use SSG/ISG rendering for mostly static but potentially frequently updated pages. Nextjs strongly supports this idea.
Make use of lazy loading and conditional rendering for above-the-fold contents. Especially, be aware of components with a large DOM size like accordion menus and dynamic recommendation contents.

As a final note, you could try using full SSR for specific pages whose visuals change almost entirely from the previous page, as the browser would be too slow to parse, render, and re-style the new DOM trees. In such a case, a developer just need to change internal links such as next/links to the standard tag.

Happy Coding!

Restructuring Frontend Testing Pyramid: alternative to Unit/Integration/E2E approach

hiroyone — Sun, 20 Mar 2022 16:03:57 +0000

Many web developers prefer to use such traditional terminology as Unit / Integration / End-to-End testings in their projects as these terms are still so familiar in the old-school software engineering fields.

But shall we keep using these distinctions when component-oriented frontend frameworks are so predominant nowadays?

Let me compare the old-school testing pyramid with the new pyramid and prioritize testings based on cost performance.

The old-school testing pyramid

Unit Tests: The meaning of "Unit" is so ambiguous in the context of frontend development. Does it mean testing a UI component, or a bigger component like a todo table, or even an individual page? Many people readily imagine different things, especially for those who are from backend and infrastructure teams.
Integration Tests: Again, The meaning of "Integration" is also unclear for the frontend development. Does it mean interactions among components or data consistency across pages?
End-to-End Testing: It is hard to tell from what end to what end people are talking about.

The restructured frontend testing pyramid

Let me reconstruct the frontend testing pyramid based on the test types and their priorities for modern framework development.

I also prepared the table version with classic testing names.

Code Quality Checks

The naming of "Static Testing" makes sense by itself, but what it actually does is analyze code in the repository and identify poorly written code.

There is no doubt that this test produces the most significant value for the cost. All developers need to do is just add syntax rules (ESLint) and format rules (Prettier) at the beginning of the project. Javascript and even Typescript are innately susceptible to runtime errors, but enforcing these coding rules in the build process amazingly captures most coding errors in the compile errors like Java or Golang.

Interaction & Scenario Testing

What developers call "Integration Testing" in the frontend context consists of two types of testing - interactions and scenario.

Interaction testing is to ensure the quality of interactive features in a specific page, such as data binding in radio buttons, click events, and input validations.

Scenario testing is to secure data consistency and transitions across pages. For example, an online shopping experience comprises views from the search page, product page, cart page, checkout page, and a thanks page. Aside from testing each interactive feature, it is valuable to guarantee that users can accomplish a typical shopping experience.

I highly prioritize Interaction and Scenario Testings to Logic (Unit) Testing because focusing on actual user experience solves the most problems users would face and most errors found in Logic Testing can be covered by Interaction and Scenario Testings👍

There are a few automation tools for these purposes, but my recommendation at this moment is Cypress, which is battle-tested and easy to start.

Performance Testing

This name is already self-explanatory, and developers just need to run Lighthouse or its CI to kickstart. Typical performance tools check speed, SEO, accessibility and other best practices for web.

These optimizations are usually easier to handle in the middle or the end of the project. Thus, I set the performance testing to a lower priority.

Logic Testing

Some components or functions have complex rules in them, thus, testing its logic (by tools like Jest) makes sense. This is part of the Unit Testing, but the name sounds too ambiguous, as explained before.

Rather than intensively testing logic, it is wiser not to bring and develop complex rules to the frontend development in the first place. Only acceptable are presentation logics, and backends should handle all complicated logic for the applications.

Visual Testing

Many developers call "UI Testing" but Visual Testing sounds more evident for checking component and page stylings.

I put this testing to the lowest priority but it depends on project goals. In my case, stylings are more robust under the scoped-component development with React/Vue and the Inverted Triangle CSS architecture.

In large projects, developers are encouraged to collaboratively work with UI designers and exchange opinions about component and page stylings with tools like Storybook.

Conclusion

I emphasized the following points in this article:

Explicit testing names compared to the old-school testing names
More priority on interaction and scenario testing based on cost performance
Automation Tools for each testing

I hope this article was helpful to other developers!

How to troubleshoot CORS Preflight / Options errors in 2022

hiroyone — Sat, 05 Feb 2022 05:51:25 +0000

TL;DR

Reference this original flow chart I created for this post.

Introduction: What is CORS? What is preflight? What is the Options method?

The CORS stands for the Cross-Origin Resource Sharing. It matters when a currently viewed website is trying to fetch a resource like JSON from another web server (cross-origin) because browsers restrict cross-origin HTTP requests initiated from scripts.

As a safety precaution, before sending the original request, browsers send a cross-origin server a so-called preflight request whose headers include both the HTTP method and headers that will be used in the original request. The preflight request uses the HTTP OPTIONS method.

For more precise definitions and related discussions, The official document is the best.

The rest of this post discusses common situational questions and answers to solve them.

Question 1: Options request is sent to the backend dev-server, but the response status code is 405 (Method Not Allowed). What should the frontend do to resolve the issue?

Cross-Origin Request Blocked: The Same Origin Policy disallows
reading the remote resource at https://cors-2022.com. (Reason: XXX).

Answer 1

You can adopt one of the two choices below.

Ask the backend to handle the option method. A preflight request is inevitable and appropriate for security reasons in some situations.
Consider the possibility of switching from a preflight request to a simple request. Simple requests don't trigger a CORS preflight.

To make your request simple, you need to satisfy all of the conditions written here.

Question 2: The backend-dev team prepared a server stub auto-generated by Swagger Codegen for our frontend-dev team. But I see some CORS errors related to the options method.

Answer 2

You can adopt one of the last three choices below.

(Don't✋) Modify the server stub implementation manually. It would be a poor choice because a server stub can no longer be auto-generated.
Suppose that your app is made from Vue or React. Then, use framework built-in proxy features such as the React Dev Server Proxy (See this post) or Vue Proxy
Use a general proxy tool like CORS Anywhere. This proxy manipulates the request header to temporarily circumvent the CORS protection.
Use a browser extension such as Allow CORS

How to Review Pull Requests 3x faster

hiroyone — Sun, 30 Jan 2022 14:23:17 +0000

How much time do you spend on average for reviewing a pull request - 3min, 10mins, or 30mins?

Even experienced programmers spend between 10 mins to 30 mins on average to review a pull request from their colleagues on some project.

Throughout various enterprise projects (Typescript, Python, Golang, Java), I reviewed more than 100 pull requests and learned to review a pull request 3x faster and more accurately, and then get even faster with automated tool's help.

No.1: Use a pull request template

Have you ever thought the following in your heart?

What changes are made and what are not?

Where are the relevant documentation for functional requirements, specifications, and designs?

The solution to the above challenges is easy, but many newbie projects miss it.

Use a pull request template!!

It ensures that every pull request has the necessary information in sorted order.

At least, it's good to include the following points.

1. Description
2. A list of Changes
3. Future todos
4. Screenshots or movies
5. Test Cases and results
6. How to review your code

You can check my repository for a complete pull request template.

No.2: Don't review business logic but prepare a test for it

Have you seen the following comments?

I tried your new feature on my local PC, but it did not work.

I checked your code, and the logic in line XX should be the opposite of the specification.

That would be generous but also too much consuming to guarantee business logic by manually checking the coding counterpart.

For example, look at the following control flow.

if (condition1) {
  //  10 lines of code here
} else if (condition2) {
  //  20 lines of code here
} else if (condition3) {
  //  20 lines of code here
} else {
  //  20 lines of code here
}

To review the above code, you need to go back and forth between the pull request and documentation 3 or 4 times.

And the real problem of this code is not about the logic but many if/else statements.

Anyway, it is more efficient to use (automated) testings to assure functional qualities of your code.

The procedure is so simple.

1. Write test cases to a test sheet
2. Review if test cases are sufficient
3. Write corresponding test code
4. Run the test program in every pull request

There are many automated testing frameworks in every programming language and CI/CD tools in code hosting platforms.

No.3: Configure a linter and a formatter properly

Have you seen the review comments like the below?

This variable should be a camelCase.

This JSON object should be indented.

Any project with comments like the above fails to introduce a syntax checker and a code formatter properly.

For instance, ESLint and Prettier are the de fact standard for javascript/typescript development. So it is important to configure eslintrc and prettierrc files properly.

Furthermore, it is also crucial not to allow exceptions without rationals. Think carefully about why a violated lint rule improves/secures your code, and your code could still be an exception.

The above No.1, No.2, No.3 practices make pull request reviews 3x faster than not.

It should take between 3 mins to 10 mins for a meaningful review.

You should only focus on:

Readability
Performance
Abstraction

A useful rule of thumb is to keep every file less than 200 lines in the repository.

But how can you spend even less time for pull request review?

There is an automated code quality review tool.

No.4: Use Code Quality Review tool

A good Code Quality Review tool makes pull request reviews another 3x faster, so it is 10x from the beginning.

I prefer to use Codebeat because it gives a sufficient static code analysis, rich CI/CD supports, a managed service. Moreover, it allows us to focus only on poor but easily fixable code on their Quick Wins panel.

On the other hand, SonarQube is more configurable in-depth and would be helpful in the advanced project requirement.

I recommend two articles for detailed tool comparisons.

Thank you for reading, and good luck with your review!🙌

How I cracked the Google Cloud Professional Machine Learning Engineer Certification Exam

hiroyone — Sat, 29 Jan 2022 07:57:48 +0000

TL;DR

Google Cloud launched the Professional Machine Learning(ML) Engineer Certification Exam on Oct 15, 2020🎉 (Not beta but GA version)
I passed the exam as one of the first test-takers on Oct. 17 in Tokyo, Japan.💪
Many exam questions were centered around practical use cases, not just about GCP service understanding.
Also, many test questions were beyond the scope of GCP services; Pure ML problems were asked.
To cover the exam, it is good to have a robust & comprehensive understanding of ML problems and solutions: ML foundations, major ML models, feature engineering, Tensorflow, model understanding tools, GCP services, Deep Learning technology.
Knowledge gained from the Professional Data Engineer Certification is partially but definitely helpful to this ML exam.
Compared with the Professional Cloud Architect and the Professional Data Engineer Certifications I passed previously, I am certain that this exam is way more difficult, practical, and advanced, requiring both GCP service familiarity and ML domain expertise.

A list of recommended articles concerning this exam

Motivation

Google Cloud Professional Machine Learning Engineer Certification Exam was launched on Oct 15, 2020🎉 (Not beta but GA version)

I passed the exam as one of the first test-takers on Oct. 17 in Tokyo, Japan. I wanted to be the first Japanese who passed this unprecedentedly hard exam, requiring all GCP, ML, and programming skills (let alone English proficiency).

About me

I already passed two other GCP exams in the past: Professional Cloud Architect and Professional Data Engineer Certifications, whose expertise was definitely helpful to this ML exam.

Also, I've already had some preliminary knowledge and experience in machine learning and programming.

Worked as an ML engineer in the Japanese IT industry for the past few years.
Won second place in a mid-size public ML competition in 2018.
Completed Python fundamentals, Pandas, Numpy, NLP, Matplotlib, and others on-line in six months before starting my career.

Roadmap

Preliminary Step: Data Science and ML fundamentals in Python (100hrs)🤖
Tensorflow (20hrs)
Input Data Pipeline, Models Trainings, Evaluation Metrics, Model Analysis Tool📊, CPU/GPU/TPU
GCP Solutions (100hrs)
ML Models: pre-trained AI solutions → AutoML → Custom Model → Kubeflow Pipeline
ML Preps: AI Notebooks, Data Labeling Service, Data Loss Prevention
Big Data: BigQuery, Cloud Storage
ETL: Dataflow, Composer, Datafusion, Pubsub and so many others😊 Feature Engineering (20hrs)
- Categorization/Discretization
- Feature crosses
- Sparse data
- Embedding
- Memorization & Generalization
- Data leakage (Deep learning) training issues and best practices (100hrs)
DNN/CNN/RNN, Wide and Deep Model, Autoencoder
Overfitting/Underfitting: L1&L2 Regularizations, Dropouts
Fast Computation: Stochastic Gradient Descent, Mini-batch
Fast Gradient Descent: Adam (Momentum & RMS Props & learning rate decay)
Vanishing Gradient: ReLu
Exploding Gradient: Lower learning rate
Internal Covariate Shift: batch normalization
Dead Relu: Leaky Relu

Conclusion

As described above, there are a wide variety of ML subjects to cover in addition to GCP ML services, and I found that this exam was still hard to solve. That is why I felt that this exam is way harder than other GCP certification exams.
According to the Google Cloud Certified directory, there are 156 registered people who passed the ML exam in the world, three months after its launch.

I am happy and proud of myself for passing this tough exam, but I may have just lucked out.😂

Good luck to you too!👼

Best Practices for reviewing Pull Requests in projects

hiroyone — Sat, 22 Jan 2022 12:13:11 +0000

Are you really confident in reviewing your colleague's code when they submit a pull request to you?

Can you proudly answer the crucial qualities you should look into in the code?

To be honest, I had been not confident at all for a long time. I was so afraid and overwhelmed to see pull requests with massive, unstructured lines of code changes submitted by a diverse level of other software engineers, even though I had already gained confidence in my programming skills.

However, I started to review my colleague's pull request in a million-dollar enterprise project which was so large and challenging. I have done more than 100 code reviews intensively in a couple of months and discovered solid insights into essential perspectives every reviewer should have in a practical and sustainable way.

I am happy to share them below.

No.1: Traceable and Revertible Pull Requests

Pull requests should be traceable and revertible after your review so that we can analyze, debug, or even revert code commits long days after we find a bug in a program.

Don't try to discover all kinds of bugs

Catching all kinds of bugs with human eyes is too much time-consuming and almost impossible. Business logic functionalities should be guaranteed by (automatic) testings rather than manual code reviews.

Do prepare a crystal-clear pull request template

## Description

Please write a summary of the changes in this pull request.

Fixes # (issue)

## Changes

Please write a list of this PR's changes here.

## Todos

Please write a list of future changes here.

## How can we see the results or reproduce them?

Please attach screenshots or videos so that reviewers can easily confirm what features are implemented.

## How Has This Been Tested?

Please describe the tests that you ran to verify your changes.

- [ ] Unit test
- [ ] Integration test
- [ ] End-to-end test
- [ ] System test

## How do you want others to review your code?

- [ ] Commit by commit
- [ ] Entire diff
- [ ] Others

This is from my repository.

What about the pull request size? Shouldn't it be small?

Yes, but there is no way to enforce this systemically. It is already too late when an inexperienced programmer posts a large pull request.

No.2: Address bad code only!

Don't pursue the best implementations on every bit of code.

Do comment only on poorly implemented code that impairs the robustness of the repository's structure.

Why?
The best code is subjective, contextual, hard to prove most of the time. The poor code is self-evident and easy to spot.

Examples of Bad Code

Duplicated code
More than two nested something (functions, objects)
Misleading variable names
More than ten function arguments
Hard-coding: mixes of logic and data

Examples of So-So code

Verbose function name
Less popular or outdated programming syntax
Non-conventional variable name but it still makes sense (e.g. addMore instead of readMore )

It is good to avoid an unnecessary dispute with your colleagues on tiny code performance or styles. A few disagreements over minor things with colleagues would ruin the relationship and the atmosphere in a long-run project.

Nonetheless, experienced programmers tend to pursue the best implementations and occasionally enforce them because that is how they improved their programming skills in the past.