Forem: Lyn Chen 🦄

Storing Billions of VM Metrics with TimescaleDB

Lyn Chen 🦄 — Tue, 25 Oct 2022 19:07:02 +0000

Originally published here.

With the recent launch of our Layerfile analytics project, we encountered a cool engineering problem - how to store billions of virtual machine (VM) metrics in PostgreSQL.

The problem

With the recent launch of our Layerfile analytics project, we encountered a cool engineering problem - how to store billions of virtual machine (VM) metrics in PostgreSQL. The metrics we wanted to track were the VM's CPU, disk and memory usage at a one second interval throughout the duration of its life. Our data can be considered time-series data as it is timestamped and append only.

While PostgreSQL will be sufficient for our first few million rows, we will start to see its performance decline as the number of writes increases and our table grows. The main pain points we faced included:

Reads and writes being too slow at scale
Threat of growing table size causing disk pressure
With these points in mind, we chose TimescaleDB.

What is TimescaleDB?

TimescaleDB is a time-series database built on top of PostgreSQL. It is an extension to PostgreSQL that implements tools to aide in the handling and processing of time-series data. The Timescale extension runs within a PostgreSQL server processing queries first to determine the planning and execution of queries involving time-series data. This enables much higher data ingestion rates and greatly improves query performance.

How we used Timescale to solve our problem

The first step taken was to convert the table storing our metrics into a Hypertable. Hypertables look and feel like regular PostgreSQL tables but are partitioned by Timescale into time based chunks. Creating a Hypertable allows us to take advantage of Timescale's full suite of tooling in addition to the ingestion and query performance.

The next step was to create a compression policy on the Hypertable. Timescale will compress chunks of data older than an interval you specify in a compression policy, in our case, 1 week. This data can still be accessed but not updated. Enabling compression on a Hypertable has between a 91-96% storage savings.

Finally, we used continuous aggregates. They are similar to PostgreSQL's materialized views but can be refreshed using a policy. We created a continuous aggregate that averages each stat over a 15 second interval refreshing every minute. This allows us to select already processed data from a smaller table, ready to be consumed by the front end.

Crypto miners are killing free CI

Lyn Chen 🦄 — Sun, 25 Apr 2021 22:06:45 +0000

Originally published here: https://layerci.com/blog/crypto-miners-are-killing-free-ci/

CI providers like GitLab, TravisCI, and Shippable are all worsening or shutting down their free tiers due to cryptocurrency mining attacks.

On September 1st, 2020, GitLab announced that their free CI offering was being restricted in response to "usage." Two months later, TravisCI announced that a similar restriction in response to "significant abuse."

Concurrently with these pricing changes, the market capitalization of mineable cryptocurrencies has exploded.

These events are related: As the market capitalization of cryptocurrency surged from $190 billion in January of 2020 to $2 trillion in April of 2021, it's become profitable for bad actors to make a full time job of attacking the free tiers of platform-as-a-service providers.

717 GitHub commits in one month
"testronan" is an avid Flask user. Every hour they make a commit to their only GitHub repository: "testronan/MyFirstRepository-Flask"

The prolific programmer is certainly making sure that their contributions are well tested. Their repository contains configurations for five different CI providers: TravisCI, CircleCI, GitHub Actions, Wercker, and LayerCI.

Seemingly quite proficient at shell scripting, their CI tasks run "listen.sh": A shell script that combines a complicated NodeJS script with some seemingly random numbers:

MyFirstRepository-Flask has nothing to do with Flask or webservers. It hosts cryptocurrency mining scripts that send WebDollars to an anonymous address. The numbers correspond to installation options for the NodeJS implementation of WebDollar

The repository is not attacking GitHub directly, instead it abuses GitHub actions' "cron" feature to create a new commit every hour and mine WebDollars on four other CI providers.

At WebDollar's April peak price of $.0005, the repository was making $77USD per month - a considerable sum in many countries, especially given that the only tools required are a laptop and an internet connection.

The two wallet addresses that receive these coins are:

https://www.webdscan.io/address/WEBD%24gBJhmuwat3kvP2@%232E4K2zXX967grh9L43%24
https://www.webdscan.io/address/WEBD%24gCszFRxzuMDbyNXnCXszoB2aIMSuV9kgbb%24
Headless browser cryptocurrency mining
"vippro99" is less subtle about their intentions. Out of dozens of repositories, most are related to cryptocurrency or browser automation.

The nodejs-monney repository contains various scripts to start instances of chrome with the Google's popular puppeteer project.

The idea is simple: Mining cryptocurrency directly in CI is somewhat easily detectable (with executable content analysis, for example) but browser automation is a common workload within CI.

The referenced GitHub pages website contains a simple browser-based Monero miner, reminiscent of Coinhive.

As of writing, the account is currently attacking JFrog's Shippable CI, which (perhaps relatedly) announced the end of its free tier earlier this year.

"vippro99"'s comments indicate that they are in Vietnam. At the current price of Monero, each instance of their cryptocurrency miner on Shippable is giving $2.5USD per month, so maintaining a mere 60 concurrent instances would be equivalent to a full time job in that country.

A solution for crypto
Ethereum, the second most popular cryptocurrency, recently announced plans to fully disable computation-based mining as a way to earn new Ethereum, switching entirely to a proof-of-stake (POS) validation model.

Beyond the environmental impact of traditional "proof of work" mining, there are externalities in many other fields like worldwide GPU shortages and attacks on free tiers of compute platforms like CI.

Providers can do their best to enforce terms of service, but as long as it's profitable and untraceable to make such attacks, they will continue to become more sophisticated and circumvent measures. The only long-term way that we will continue to be able to enjoy free tiers on Heroku, Netlify, and GitHub are to switch away from proof-of-work.

Introducing Wrap.sh - an internal hackathon project

Lyn Chen 🦄 — Mon, 01 Feb 2021 22:00:40 +0000

Hey DEV Community!Some of the coolest people I've met were through university hackathons. It was always a great way to take a break from school/work, get to know others, and come up with interesting ideas to hard problems. I remember discovering my love for Awake Chocolate, learning new tools and pitching nervously the next day. What I loved most about these events is that no matter who won, everyone felt that they contributed and made something worth sharing.

Unfortunately, I hadn't worked at a workplace before LayerCI where experimentation was so engrained into the company DNA. Why did we ever stop making things together and instead only go to engineering conferences? Smack in the middle of the pandemic, our team decided to do an internal hackathon together and the results were awesome.I remember discovering my love for Awake Chocolate, learning new tools and pitching nervously the next day. What I loved most about these events is that no matter who won, everyone felt that they contributed and made something worth sharing.

Our very first open source project was born!

I'm excited to share the winning, internal hackathon project idea my teammate Dmitri on the engineering team led - it's a bash script that can be installed with anyone's CI to create preview environments without having to spend the engineering hours building it yourself.

Try for free here: https://wrap.sh/

Below is a high level overview on the experience. Feel free to try it out and let us know what you think at hello@wrap.sh