Forem: Okuto Oyama

Thank you Webpacker, Goodbye Webpacker

Okuto Oyama — Thu, 15 Dec 2022 03:00:00 +0000

Original Post (Japanese) : ありがとう Webpacker さようなら Webpacker - クラウドワークスエンジニアブログ

Hello. I'm Okuto Oyama, an engineer at CrowdWorks. I'm on a team that leads in eliminating technical debt.

This article is replace Webpacker with Simpacker and webpack.

As an assumption, our products have been running on monolithic Ruby on Rails applications for over a decade. Currently, we are trying to break the tight coupling between front-end and back-end as part of our technical debt resolution.

I hope this is helpful for anyone who wants to separate front-end responsibilities from Webpacker!

What is Webpacker?

Webpacker is the official library for webpack front-end development in Ruby on Rails applications. The advantage of this library is that it provides a generic configuration for those who are not familiar with the webpack build system, making it easy to quickly develop modern front-ends.

For more information on CrowdWorks' implementation of Webpacker, please see our previous blog posts.

CrowdWorks流！Webpacker活用術 - クラウドワークスエンジニアブログ

Why quitting Webpacker?

I have decided to quit Webpacker, and I will give the reasons why I have decided to quit.

Because Webpacker was declared retired in January of this year.

There is no need to continue using Webpacker as the next RC version will not be officially released in the future.
- There is an option to use Shakapacker
- However, since we plan to move away from Rails-dependent front-end development at CrowdWorks, this is not the right choice for us.

Because the modules that depend on Webpacker are hidden.

The internal Webpacker dependencies are hidden, making it hard to see what is being used.
- Babel settings were not included in CrowdWorks package.json, causing some build failures.
- Debugging itself is not easy.
  - I tried the v14 series of stylelint postcss must be included, but even if the lint passed, the webpack build failed.
    - I wanted to find the cause, but at first glance I couldn't figure out what was wrong with the configuration.
- By migrating to bare webpack, we can think about the configuration ourselves.

Because it is hard to do performance tuning

Not only webpack, but also Rails assets:precompile builds are done by CI.
- We are extending resource classes because CI and deployments are failing due to the long time it takes.
- Not enough fine-tuning of webpack.
- It may be possible to reduce this problem by updating Webpacker itself.
  - The webpack included in the stable version of Webpacker used to be v4.

In light of the above, we decided to eliminate the use of Webpacker itself and allow the front-end responsibilities to be handled by the front-end.

Incidentally, CrowdWorks would like to migrate to a front-end framework in the future, and if everything can run on that framework, the removal of Webpacker itself will be unnecessary. However, since the transition to a framework is not yet in sight, we have decided that using webpack is the right way to maintain the status quo.

Organizing what to do

First of all, when I quit Webpacker, I organized what I was going to do. There have been several articles about quitting Webpacker in the past, so I used them as a reference.

I mainly referred to pixiv's article and iCARE's article. I would like to take this opportunity to thank them.

webpack configuration
- Migrate Webpacker DSL (config/webpack) to webpack.config.js
- Install necessary plugins and modules.
- Configure webpack-dev-server
Remove Webpacker
- Remove from gem
- Make config/webpacker.yml obsolete
Implementing webpack view helpers in Rails
- Create helper code
- Create test code
Docker, CI configuration changes
- Incorporate webpack build mechanism into CI
- Adding webpack processing to docker-compose configuration
- Staging CI checks
- Checking docker-compose behavior
Checking the behavior of docker-compose in local staging
- Local screen checks (webpack build-related screens only)
- Screen check at staging (only screens related to webpack builds)

Examine the Webpacker implementation

The first thing I did was investigate the Webpacker implementation.

First, I tried to output the Webpacker configuration to get a rough idea of how the whole thing works.

The way to do this is quite simple. I put the logs in console.dir(webpackConfig, { depth: null }); at the end of module.exports in each environment's DSL file placed in config/webpack. I have tried this. I can't go into the details of the function-based code, but this gave me a rough idea of the general shape of the code.

The following is the webpack configuration included in Webpacker.

summary

entry
- Configuration of files to be loaded by Webpacker.
output
- Configuration of the file to be built and exported.
resolve
- Configuration of how modules can be resolved.
resolveLoader
- Configure only name resolution for webpack loader package.
node
- Configures whether certain Node.js globals and modules can be polyfilled or mocked
  - Allows code written for the Node.js environment to run in other environments, such as browsers.
loaders
- webpack only understands JS and JSON. Therefore, the part that converts other files so that they can be deciphered.
  - Things like xxxx-loader are the equivalent.
plugins
- webpack plugin settings
mode
- Development environment settings
- You can specify production, development, etc.
devtool
- Source map generation and how it is generated.
stats
- Settings to control bundle information displayed.
bail
- Settings to fail when an error occurs.
optimization
- Settings related to optimization execution

The next step is to investigate the detailed Webpacker implementation based on the results.

https://github.com/rails/webpacker/tree/5-x-stable

The version of Webpacker used at CrowdWorks was version 5, so we checked the 5-x-stable branch. The part I was interested in was webpack.config.js, so I just looked at its source.

webpacker/package/environments/ ... Environment settings. It contains basic webpack settings.
webpacker/package/rules/ ... List of rules for loaders. Only refer to the ones you use.

You should also check the required package.json, babel.config.js, postcss.config.js.

Compare build results between Webpacker and webpack alone

I experimented to see if Webpacker can be built by itself while referring to the internal implementation.

In CrowdWorks, the front-end tools (such as Storybook) are in a separate hierarchy of directories, so I installed a new one there and set up webpack to reference the front-end code in the application.

After confirming that the build went through, I compared the differences between the results built in the staging environment and the results built with webpack alone.

Since CrowdWorks uses CircleCI for our application build pipeline, we saved the built files in artifacts.

- run:
  name: Compress Artifacts
  command: tar -cvzf assets-precompile.tar public/packs
- store_artifacts:
  name: Uploading artifacts - store results of assets:precompile
  path: assets-precompile.tar

We reviewed the missing configuration, the number of files, and the differences in the contents of the files, and finally, we were able to get the files to almost match ¹.

Incorporate webpack build results into your Rails application

After confirming that the build results are OK, the next step is to incorporate them into the Rails application.

Webpacker uses webpack-assets-manifest to create manifest.json, which is then delivered to View helper in Rails.

I needed to create an alternative View helper to replace Webpacker, but decided to use the Simpacker gem rather than implement the helper on my own.

Unlike Webpacker, Simpacker does not manage any configuration on the webpack side and is only involved with the path to the output manifest.json. The responsibilities are nicely divided, and the configuration itself is very simple.

I have been using Split Chunks since I was using Webpacker, so I decided to replace it by introducing the following helper.

def javascript_packs_with_chunks_tag(*names, **options)
  paths = names.flat_map{ |name| simpacker_context.manifest.lookup!("entrypoints", name, "js") }.uniq
  javascript_include_tag(*paths, **options)
end

def stylesheet_packs_with_chunks_tag(*names, **options)
  paths = names.flat_map{ |name| simpacker_context.manifest.lookup!("entrypoints", name, "css") }.uniq
  stylesheet_link_tag(*paths, **options)
rescue Simpacker::Manifest::MissingEntryError
  # If you don't extract css, you get an error, but don't return anything
end

Make webpack-dev-server work

Webpacker also includes webpack-dev-server, so we made it work as well. Since I had Simpacker installed, I only had to change the devServer part in webpack.config.js to make it work with npm scripts.

The problem with getting webpack-dev-server to work was that the included version was a 3 series that didn't work well with the same settings as Webpacker.

Rather than investigating the cause in detail, I thought that if it was for development only and didn't affect production, I should just update it to the latest version, so I updated to the latest 4 series and migrated the part about devServer settings.

It works with this as a result. If you're looking to move on from the Webpacker 5, here's what you need to know.

Check the behavior on CI

As mentioned above, deployment of CrowdWorks is managed by CircleCI. Because it's a monolithic application, the front-end build mechanism is built into it, and we'll review the settings when we switch to webpack to make sure it doesn't fail.

Webpacker used to build webpack at assets:precompile time, but since it no longer does so, we need to add our webpack build commands.

While reviewing the configuration, I also included a workflow uploading source map files to Rollbar for the front-end error monitoring system. However, for some reason, I caught an event that prevented the uploading of the files.

After investigating the cause, I found that it was due to the --ignore-optional option that I had added when doing the yarn install.

This was because the source map files generated by webpack depended on optional dependencies, and thus were not generated properly without the required modules. Therefore, we decided to remove this option and let it install.

Ask engineers to check if it works in their local environment

Since CrowdWorks uses Docker in its development, we asked our engineers to check if it works with Simpacker + webpack.

Docker startup and build results were fine, but we found that Hot Module Reloads in webpack-dev-server were not working properly. After that, we fixed it while checking the operation with my team.

Conduct a mob review session

Since the changes made to the PR were very significant this time, we conducted a mob review session to reduce the prerequisite knowledge and cognitive load among my team.

Since I had been working on this project for a long time, I was able to look back on what I had done, and the causes of some parts that were unclear as to their operation were clarified. Some of the omissions found in this review have also been corrected to reflect.

Formulate Architecture Decision Records

We have been keeping Architecture Decision Records (ADR) since September of this year to keep the reasons for architecture selection.

This initiative was not in place when we discontinued Webpacker, but since we were going to adopt Simpacker in this transition, we decided to develop an ADR for it and have it reviewed by our engineers.

Incidentally, this is the first document after the ADR initiative was introduced.

Release

It's been a long process, but now that we have all the necessary responses, we're finally releasing it. Even in the staging environment, I had verified it many times and thought there would be no problem with the release, but I had shared it with all the engineers beforehand just in case.

I was very nervous about the release because it was a very big change, but after the release was completed, we determined that there were no problems with operation and no alerts and that users would not be affected, so we reported the completion and shared the operating procedures again.

I was in a daze for a while, maybe glad that I released it successfully...

Reflection

We look back on our efforts to quit Webpacker this time.

Good

I was able to complete the migration to Simpacker + webpack itself.
We were able to split webpack management and build responsibilities from Rails.
We were able to resolve the EOL situation.

Problem

adding their own npm packages that were included with Webpacker
- Try ⇒ We will update and prune it
As the npm packages were being updated at the same time, there were many conflicts in package.json, making it difficult to rebase
- We should have verified the module with the module itself added and released it first
- Try ⇒ Make a mechanism to verify build diffs (to build psychological safety for future updates)
Migration work was being handled by @okuto_oyama alone
- The work was being done by one person, @okutoyama, who had been doing the research singlehandedly, and was also fixing the problem
  - @bugfire also helped me with some of the stuck areas
- The work could have been divided among the team since it was related to the development of the entire engineering team
- Try ⇒ After investigating, you don't start as is, but set tasks one by one after you get a general perspective

Conclusion

We started our research in April of this year, and although there was a period during which we had to stop due to other work within the team, we were finally able to migrate and release it on November 24. We are very happy that we were able to finish it before the end of the year.

I've had a few struggles with Webpacker, but it's also the fact that it's been a mainstay of CrowdWorks' modern front-end development for a long time. Thank you for fighting with us so far.

Webpacker脱却記事も多いので、フロントエンド人材が育ったら引き剥がすことがそこまで難しくなさそうなのもポイントです。

Webpackerをフロントエンド環境の補助輪と考えて、将来存在が足かせになったタイミングでWebpackerのレイヤは剥がしてもらえばよいと考えています。

メンテ不能になったフロントエンド環境を立て直す話 - クラウドワークスエンジニアブログ

After rereading articles about rebuilding the front-end environment that had existed in the past, I personally felt a lingering sense of joy that we had reached a point where it was OK to eliminate this layer.

Although I was able to quit Webpacker, I hope to continue working on improving the build flow without compromising the developer experience, including upgrading webpack.

References

Where there were differences in some of the build results, we decided to allow them because we were able to confirm and determine that there were no problems with the display and operation. ↩

I want everyone to know about the design system.

Okuto Oyama — Fri, 03 Jan 2020 06:19:15 +0000

This article is an English translation of みんなに知ってもらいたいデザインシステムのこと written in December 2019.

Hello everyone. My name is @yamanoku and I have been working at CrowdWorks, Inc since October 2019.

I usually go to study meetings and conferences, wander around the Internet, drink alcohol, and take care of my father. He is currently a front end engineer joining the design system team.

In this article, I would like to introduce you to the design system, where you need it, what engineers and designers need to know, and what you need to know.

About the Design System

design system. If you work on the web as well as the front end, you may have heard this word recently. Here's what it's all about.

The design system is defined in the following article.

A design system offers a library of visual style, components, and other concerns documented and released by an individual, team or community as code and design tools so that adopting products can be more efficient and cohesive.

Defining Design Systems

When you say "Documentation" or "Library" you might think of it as a style guide that contains the rules for UI color and shape, or a pattern library that allows components to be deployed in different ways.

However, the difference in the design system is "The ability to extend the product" that style guides and pattern libraries cannot.

Style guides and pattern libraries were nothing more than list-oriented guides, and they were not designed to be complete or to keep up with product growth.

The design system encompasses these and defines design principles, information design, infrastructure, and accessibility. They also embody the identity of the product, not just the guides and libraries, but what they can do and what they can solve.

The principles and principles will also be documented, making it easier to understand "On what basis should I do it?" when creating new functions or modifying products.

In addition to the UI components that define a product's identity, Adobe's Spectrum design system, which was announced earlier this year, also has a system for broader design, with words like "Voice and Tone" and "Grammar and Mechanics" and style guidelines for writing.

In 2017, the United States announced its national design system , and efforts are under way to create a consistent website for each country.

https://designsystem.digital.gov/

Also in Japan, various companies are starting to give examples of their use. The following is an example. (Only Japanese text)

In March of this year, a voluntary community called designsystems.tokyo, which is working on design systems at an operating company, started to discuss and discuss how to introduce the system. CrowdWorks is also a member of this community and participated in each study group.

designsystems.tokyo - connpass

When is a design system needed?

Here's when you'll need it.

It's a waste of time to say this, but if you don't feel the need to build it (and communication and so on.), you don't have to force it. It's not just a design system, there's nothing more scary than something that just starts.

結局デザインシステムは何なのか : could (Only Japanese text)

However, if you are asked to define advanced requirements and need to release them quickly, it seems difficult to tackle them without a design system. This is because we can make the collaboration between designers and engineers more smooth, and we can develop products with consistent quality even if the service expands.

I don't know if a new UI can be properly evaluated when starting a new business. However, the design system has developed components that provide a consistent experience.
By combining them properly, even a small team can efficiently develop products while maintaining quality.

Furthermore, by preventing the mixing of components and ensuring uniformity, it is possible to reduce the wasteful process called wheel redevelopment.

If you have 25 teams in your company and each team makes a button, it costs $1 million to make a good button.

ー Nathan Curtis

Previously a $1 million conference was held in our company to develop a functional and visual relationship between these components.

デザインシステムの基本の基を考えていくお仕事をしています。 - CrowdWorks Designer Blog | クラウドワークスデザイナーブログ (Only Japanese text)

With that in mind, here are some things that need to be considered from the perspective of engineers and designers.

What we want our engineers to know

What we need to communicate with designers is representation of the state in words.

Consider a component called a button.
There are various states, whether it is default, in focus, active, or disabled, even with just one button.

And if the button itself is a link (a tag), you have to define it differently (Disabled not allowed etc.).

These may be easy to imagine for implementors, but if you don't specify what the component is in, you can leak it when designing, so you need to make it clear.

There is also a design system called Design Tokens . This allows you to replace these values in bulk using the defined tokens.
The design system can guarantee a single source of truth (Only reliable source of information).
You can think of it as something like CSS Custom Properties or Sass variables.

$brand-primary: rgb(21, 137, 238);
$brand-primary-active: rgb(0, 122, 221);
$brand-primary-transparent: rgba(21, 137, 238, 0.1);

When defining tokens, you should also consider the term itself What kind of state does it show?, How much responsibility do you have? when creating your design system.
Confusion in the definition of words can lead to design breakdowns in the future.

Design Tokensを考える前に俺たちは変数名について考え直すことがあるのではないか - yamaScrapbox (Only Japanese text)

What I want designers to know

Designers are often responsible for visual representation, but they need to make sure that the "Why does it express that?" is understood by others.

An example of Shopify's design system "Polaris" is to capture cultural differences in internationalization.

For example, in Japan, we have added the input items of the prefecture after the country, and we have not set up the name input form in North American style. These examples will help you better understand why this design works.

https://polaris.shopify.com/foundations/internationalization#section-plan-for-cultural-differences

The design also requires that the tools used be carefully selected.
Sketch, Figma, Invision, Storybook, and other tools are available to help designers collaborate with other industries. However, there is no clear answer, such as "You can do it with this.". I think it's better to use them based on the idea of what kind of world they are trying to realize.

Figma's ability to build a design system isn't just about cross-linking, it's also about making it more practical, with analytics visualizations and limited onboarding and other design data.

Design, manage, and deploy your design systems with Figma

As for collaboration, Sketch also announced simultaneous editing. It's a good idea to keep an eye on tool updates.

What I want everyone to know

In the end, I want to make it more comprehensive and make it known to everyone involved in a non-executive design system.

As you build a design system from a single product, is there something called a completion? That's the question. The answer is "NO".

As I mentioned earlier, the design system will be expanded, and it is assumed that everyone can design with the maintained quality.
Therefore, it is necessary to operate to be able to expand on the assumption that there is no clear answer such as 0 or 1.

The design system is a living thing. Let's raise them little by little.

If you're building a design system, you want to make it more perfect for every pattern. But I realized that the design system is not perfect.
As products and organizations grow and change, so does the design system. The design system is a living thing.

デザインシステムのプロジェクトの実践で学んだ3つの大切なノウハウ - CrowdWorks Designer Blog | クラウドワークスデザイナーブログ (Only Japanese text)

Because the design system is a product, it can fail.
In the article explaining the failures, there are examples such as lack of organizational support and communication, lack of initial investment, and lack of use.
Even though the article was written in 2017, it is still well understood.

Why Design Systems Fail ◆ 24 ways

Sustainability and development are essential, and failure to do so can lead to a vicious cycle of design. It is desirable to have a solid acceptance system.

If the design system goes in the wrong direction, it can result in bad things. Depending on the design, you can hurt people or remove certain layers, so what is the purpose of your product? What kind of future are you trying to paint? We should not forget to have such a viewpoint.

Conclusion

That's all I wanted you to know about the design system.

Using a design system allows you to be creative flexibly while maintaining a certain level of quality. However, if you proceed without a clear sense of purpose or understanding what you should not do, you may spread incorrect designs.

The frontend team, which started last year, is working on a design system for CrowdWorks by trial and error.
It is still closed inside, but I hope to show it to everyone in the future.

Forem: Okuto Oyama

Thank you Webpacker, Goodbye Webpacker

What is Webpacker?

Why quitting Webpacker?

Because Webpacker was declared retired in January of this year.

Because the modules that depend on Webpacker are hidden.

Because it is hard to do performance tuning

Organizing what to do

Examine the Webpacker implementation

Compare build results between Webpacker and webpack alone

Incorporate webpack build results into your Rails application

Make webpack-dev-server work

Check the behavior on CI

Ask engineers to check if it works in their local environment

Conduct a mob review session

Formulate Architecture Decision Records

Release

Reflection

Good

Problem

Conclusion

References

I want everyone to know about the design system.

About the Design System

When is a design system needed?

What we want our engineers to know

What I want designers to know

What I want everyone to know

Conclusion

References （Only Japanese text）