Forem: Billy Kong

What is libv8 and Why its Installation Fails

Billy Kong — Sun, 14 Jun 2020 05:38:09 +0000

Background

When I cloned a project and tried to run bundle install, I receive the error

ERROR: Failed to build gem native extension.
...
fatal error: 'climits' file not found #include <climits>

The error is raised when bundler tries to require libv8, which is in turn required by therubyracer.

If you are reading this article, I believe I can assume that you are having a similar problem. I am going to share my findings here.

In this blog, we will try to answer the following questions:

Why do we need libv8?
- a. How is it related to therubyracer?
Why does installation through bundle install fails?
How do we fix it?

Why do we need `libv8`?

Before we talk about libv8, let's first take a look what is V8

V8 is Google’s open source high-performance JavaScript and WebAssembly engine, written in C++. It is used in Chrome and in Node.js, among others.

So, basically, V8 is the Javascript engine in Chrome. And libv8 is the ruby interface for the V8 engine used by therubyracer.

How is it related to `therubyracer`?

The therubyracer gem embeds the V8 Javascript Interpreter into ruby. With it, we can call Javascript functions or use Javascript objects directly in ruby runtime.

Why does installation through `bundle install` fails?

The installation failed as therubyracer gem calls the libv8 gem, and it fails to build a native extension, i.e. v8. I am able to repeat the same bug with:

$ gem install therubyracer
Building native extensions. This could take a while...
ERROR:  Error installing therubyracer:
    ERROR: Failed to build gem native extension.

    current directory: /Users/billykong/.rvm/gems/ruby-2.6.5/gems/libv8-3.16.14.19/ext/libv8
...
Compiling v8 for x64
Using python 2.7.16
Using compiler: c++ (clang version 11.0.0)
Unable to find a compiler officially supported by v8.
It is recommended to use GCC v4.4 or higher
Beginning compilation. This will take some time.
...

The error in your case may be different but still there is one interesting finding here. The therubyracer gem installation script is trying to compile v8 from source! I am not saying it is impossible, but I got a feeling that the compilation is the cause of issues for many of us. Just the first few posts I found when I google "libv8 error":

Compiling something requires all the dependencies to be present. Since v8 is written in C++ and probably many of us don't have a C++ development environment setup ready for use. Perhaps it is easier just to download a suitable binary of v8 instead of building it from source.

How do we fix it?

# 1. Install v8 ourselves
$ brew install v8-315
# 2. Install libv8 using the v8 binary we just installed
$ gem install libv8 -v '3.16.14.19' -- --with-system-v8
# 3. Install therubyracer using the v8 binary we just installed
$ gem install therubyracer -- --with-v8-dir=/usr/local/opt/v8@315
# 4. Install the remaining dependencies
$ bundle install

Note: The -- before the --with-system-v8 and --with-v8-dir=/usr/local/opt/v8@315 is necessary when we supply library paths for native extensions, i.e. using specified libraries/binaries instead of building them.

Conclusion

therubyracer or some other gems may requires native extensions to function properly. While some tries to build the extension from source code, it often fails because the dependencies required by the native extensions are not readily installed. We may solve the issues by supplying our own binaries.

Hope that you have find the information useful. If you find any error or mistake in the above content, please do feel free to leave a comment below.

This blog is also published in billykong.github.io.

Reference

Remote Team Git-based Communication

Billy Kong — Sun, 14 Jun 2020 05:33:25 +0000

Motivation

The COVID-19 pandemic has brought many changes to our lives, one of them being many people are working from home now. It is not necessarily a hinterance of communication, but we do need to adapt. Many things that used to be communicated verbally, needs to be communicated textually.

We have switched to remote working since January 2020. During the time, we have natually developed a git-based communication workflow. If you are working in the software industry, you will see that it is not unlike the pull request workflow most of us used for source code. The precise and asynchronous communication model for source code works just as well for software design, schedule planning, and team administration discussion.

So far the efficiency is even better than before COVID-19 in addition of better transparency. I would like to share our approach here with some details. For anyone who may be looking for a way to build and run a remote team, I hope this is useful.

Overview

We call our way document-based communication(or git-based communication). It is inspired by the handbook-based communication shared by GitLab. Since our team is much smaller, 4 developers only, our handbook is much simpler.

The handbook is a git repository of markdown files organised into directories. Really, not much difference with your local drive or share drive. But using git helps to iterate ideas much faster. Here is the overall directory strcuture:

.
|--engineering-team
|  |--README.md
|  |--merge-request-flow.md
|  `--recruitment-process.md
|--features
|  |--README.md
|  |--feature-1.md
|  `--feature-2.md
|--technical-documentation
|  |--README.md
|  |--investigations
|  |  |--2020-04-01-dummy-issue-1-investigation-report.md
|  |  `--2020-05-20-dummy-issue-2-investigation-report.md
|  |--architecture-diagram.md
|  `--analytics-etl.md
`--README.md

What kind of document is needed for the day-to-day operations?

Team SOP guidelines
- This directory holds the day-to-day workflow descriptions.
- E.g. merge-request-flow.md
  - Simple description of how to do merge request/pull request since not all teammates may be familiar with open source software workflow
  - Define who should be assigned reviewers and criteria for a review to pass
  - Define who is responsible for merging and what to do it there is code conflict
- Once written, contents in this directory rarely changes.
Features
- This directory holds all the new features or features updates requirements
- If your team has the wonderful practice of writing design doc, here is where it should be kept.
- For our case, we put the followings guidelines in /features/README.md:
  - What should be included in a feature doc?
    - textual description of user flow with links to graphic assets
    - technical design description
    - contentious points
    - estimation
- If your features is complicated, feel free to open subfolders inside features/
Technical notes
- This directory is for long-term documentations and technical knowledge sharing, a bit like a wiki or knowledge base.
- When our teammates consider certain new tech to be added to our services, we require each other to write down what we have: read, opinions, and experience on trying out things here, such that we can learn from each other more easily, and we can monitor the quality of "research" tasks so management can be more comfortable assigning such tasks.
- The output is very similar to blogs such as this one.
- We have an investigation subfolder here as well
  - When there are tricky bugs or issues, we write down what happened and how we solve them.

Why use `markdown`?

Easy to write
- It takes 30sec to learn for generic document like this one, seriously.
Easy for source code control
- It makes the merge/pull request flow possible as elaborated below.
This blog is written in markdown, you can find the source here.

How to communicate?

We use the merge request flow. We didn't invent this workflow, it is the common practice used in open source softwares. If you are using GitHub it is the pull request flow. The similar flow is called merge request flow in GitLab.

Bascially, you do the followings:

Create new branch
Make changes
git push
Create merge request in gitlab.com
Assign reviewer(i.e. Assignee)
Done! The reviewer will be notified and be able to comment on your request.

What is gained?

Efficiency: for our team of 4, we have about 10 threads of communication per day, each is precisely scoped to a line in a document.
Constructive disagreement: as teammates has more time to contemplate each comment before dashing it out. Here emoji helps.
Sunshine factor: as all comments are recorded and accessible to all teammates. It really encourage the best of behaviour from our experience.
Async communication: teammates can check their notification periodically and continue discussion without losing context. And the developers are not interrupted neither! Big win here!

What needs to be changed?

Write it down instead of speak it out
- Not everyone is comfortable with it on Day 1.
- It takes us about a week to get used to writing doc instead of Zoom calling or Slack chatting.
Sending links of doc, along with some copy and paste
- Other teams may still prefer more traditional email communication.
- No hard feeling. Just send them the link along with some content of the doc copy and pasted into the email.
Higher throughput, but more latency
- The async nature of merge request flow means we won't get response as fast as tapping one's shoulder in office.
- But the upside is higher quality communication and easier to discuss mutliple things without getting the context mixed up. Communication threads in merge requests are precise to a single line.

Conclusion

It is an efficient, impartial, system.
It requires some technical capability, but really not much.
You may encounter resistence, but even if you can only successfully promote it within your team. The benefit is still worths it.

By the way, the GitLab Handbook is really a valuable reference to how to run different departments of a software company there. I am so happy that they are able to condense their valuable experience into a single document and open source it.

This blog is also published in billykong.github.io.

References

Deploying a Classic 3-tiers Application using aws-cdk

Billy Kong — Sun, 14 Jun 2020 05:23:49 +0000

Background

I have been looking for ways to adapt infrastructure-as-code with my team. But the initial complexity is a big deterant. Remember we not only have to output the initial configuration, but to maintain it as well.

aws-cdk is released on 2019-07-11.
It is simpler than writing a CloudFromation template from scratch.
Perhaps it is a good entry point for teams that want to adapt infrastructure-as-code.

A Classic 3-Tiers Application

With load-balancer tier, stateless application logic tier, and database tier.

Tier	Componenet	AWS Service	Subnet
1	Load-balancer	AWS ELB	Public
2	Application Logic	AWS ECS Fargate	Private
3	Database	AWS RDS Aurora	Isolated

Following the security practice of separating subnets for different tiers, the application will be deployed into:

a public subnet(with two-way Internet access),
a private subnet(with out-going Internet access only), and
a isolated subnet(no Internet access either way).

We are also using environment variable to pass database credentials as it is easier to reuse existing docker image.

Here is the aws-cdk stack that I managed to get working:

Deploy

If you want to deploy it and poke around, you can checkout the GitHub repository here. The deployment instruction is written in README.md.

Note that we should install the same version of aws-cdk and other @aws-cdk/* dependencies. It seems even minor version difference may be incompatible. I used v1.38.0.

Some Rooms for Improvements

Use separate route tables for each subnet.
Database security group should allow traffic from the private subnet only.
Calling AWS Secret Manager API from application code for database credential is probably more secure, but it will require some custom code. If you expect to reuse the same Docker image in, say, Kubernetes, it may cause problems.
I couldn't quite get the DatabaseCluster construct to work. So I used the CloudFormation verions CfnDBCluster. If you managed to use DatabaseCluster, please feel free to leave a comment.

This blog is also published in billykong.github.io

Forem: Billy Kong

What is libv8 and Why its Installation Fails

Background

In this blog, we will try to answer the following questions:

Why do we need libv8?

How is it related to therubyracer?

Why does installation through bundle install fails?

How do we fix it?

Conclusion

Reference

Remote Team Git-based Communication

Motivation

Overview

What kind of document is needed for the day-to-day operations?

Why use markdown?

How to communicate?

What is gained?

What needs to be changed?

Conclusion

References

Deploying a Classic 3-tiers Application using aws-cdk

Background

A Classic 3-Tiers Application

Deploy

Some Rooms for Improvements

References

Why do we need `libv8`?

How is it related to `therubyracer`?

Why does installation through `bundle install` fails?

Why use `markdown`?