Forem: Yoshi Yamaguchi

What're inside OpenTelemetry Collector and ADOT?

Yoshi Yamaguchi — Mon, 03 Mar 2025 09:31:18 +0000

AWS has some observability services including Amazon CloudWatch, Amazon X-Ray, Amazon OpenSearch Service, and Amazon Managed Service for Prometheus. AWS Distro for OpenTelemetry (ADOT) is an option for sending telemetry data from applications you implement yourself to these services.

In addition to a general description of the OpenTelemetry Collector, this article describes ADOT in detail.

TL;DR

ADOT is a distribution of OpenTelemetry Collector for AWS that comes with the components you need to use the AWS and its partner solutions. If you want to use OpenTelemetry Collector in a more elaborate way, custom build is recommended.

What is OpenTelemetry?

Though I know that many AWS users are already using OpenTelemetry, just in case let me briefly explain OpenTelemetry, a project under CNCF. OpenTelemetry is a project under CNCF that standardizes the instrumentation, collection and export of telemetry signals from the system.

What is OpenTelemetry Collector?

The OpenTelemetry Collector (hereafter "Collector") is an agent for instrumenting, collecting, and exporting telemetry that is being specified and a reference implementation is being developed by the OpenTelemetry project. The reference implementation runs as a single binary and is written in Go.

It consists of three main components: receiver, processor, and exporter, and the required components are specified and made into a single binary before distribution.

Receiver: Responsible for receiving telemetry
Processor: Responsible for processing and buffering of telemetry
Exporter: Responsible for transmitting telemetry to the backend

The components included in the binary can be freely linked in the configuration file, but only in the direction of receiver, processor, and exporter ¹. Receivers and exporters can only be linked one at a time within a pipeline, while processors can be linked in multiple times.

For example, the following configuration file shows receiving telemetry at OTLP, adjusting the flow rate of telemetry to be sent using the batch and memory_limiter processors, and then sending telemetry again at OTLP.

receivers:
  otlp:

processors:
  batch:
  memory_limiter:
    limit_mib: 1536
    spike_limit_mib: 512
    check_interval: 5s

exporters:
  otlp:

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]

The Collector's source code is available in GitHub repositories open-telemetry/opentelemetry-collector and open-telemetry/opentelemetry-collector-contrib in which the source code of each component are available. In addition, they distribute several pre-built binaries of the collectors as the official OpenTelemetry project versions, as well as the OCI container that ships with them.

As of February 2025, four different builds are available for the officially distributed collectors, depending on the type of components included ².

otelcol-otlp: Binary that can only receive and send OTLP. The file size is very small but has almost no functionality.
otelcol: Binaries that support OSS that should be supported by OpenTelemetry project.
otelcol-k8s: Binary with additional components for Kubernetes in addition to otelcol.
otelcol-contrib: This is a binary that includes all components managed under the OpenTelemetry project. The file size is very large, although all components, including those for Observability SaaS, can be used.

What is ADOT (AWS Distro for OpenTelemetry)?

If you want to use AWS components, the only way to achieve this with officially distributed collectors is to use otelcol-contrib. However, this would be a waste of a large binary because it includes a large number of unnecessary components. Therefore, AWS provides a Collector distribution called ADOT.

ADOT pre-builds and distributes the components used in the official otelcol distribution ³, plus additional components that are useful for using AWS services and partner SaaS solutions. The binaries distributed here can be invoked by Amazon ECS, AWS Lambda, etc.

For basic use of OpenTelemetry on AWS, using ADOT would be the first option.

Collector custom build

On AWS, it is better to use ADOT Collector, but there will be components in ADOT that you will not use, and if you want to use OpenTelemetry in a more advanced way, you may not want to use these unnecessary components, or conversely, you may want to use components not included in ADOT. For example, the Span Metrics Connector component, which converts distributed trace spans into metrics, is not included in ADOT. For such advanced use cases, the OpenTelemetry Collector builder (ocb) can be used.

ocb is a convenient tool that allows you to build a Collector binary from the configuration in YAML file that lists the components you need in the form of Go package names. For example, to build a collector with only the following components, you need to create the following YAML file otelcol-foo.yaml

dist:
  name: otelcol-foo
  description: Custom OpenTelemery Collector for Project foo AWS account
  output_path: ./otelcol-foo

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.119.0

processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.119.0
  - gomod: go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.119.0

exporters:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter v0.119.0

Using this YAML file, a custom-built Collector is built by executing the ocb command as follows

./ocb --config otelcol-foo.yaml

Conclusion

I hope this article has deepened your understanding of the contents of ADOT, which has evolved along with the development of OpenTelemetry. In fact, the contents of ADOT have changed a little with each version. However, ADOT is only a custom build, so there is nothing to fear if you know what it is. We hope you will find it useful in selecting the best collector for your project.

In the advanced use cases, the components called "Connector" is used, which loop backs the telemetry from the exporter to the receive. Because this artile aims at explaining basic concept of Collector, I omit the explanation of that part. ↩
Manifest files for each binary build can be found in the open-telemetry/opentelemetry-collector-releases repository. (otelcol-otlp, otelcol, otelcol-k8s, otelcol-contrib) ↩
The list of components included in ADOT can be found in the repository README. The actual components can be found in the Go source code. ↩

Revisit base container image for AWS services

Yoshi Yamaguchi — Mon, 03 Feb 2025 07:51:58 +0000

Hi, I'm Yoshi (@ymotongpoo). Are you enjoying building your container images for serverless and container runtime solutions in AWS? Many container runtime services exist in AWS, including Amazon ECS, Amazon EKS and AWS Lambda. The first step in using such container runtimes is to create a container image. AWS provides amazonlinux images, and they are about 50 MB by default. Some people may be trying to reduce the size of the image in order to launch the container image as quickly as possible.

There are already many practices to reduce container size, but in this article, I would like to focus on the base image and discuss the characteristics of each. This article assumes that images are built based on Dockerfile. Unless otherwise noted below, image sizes are for the latest tag for arm64 as of January 9, 2025.

TL;DR

I introduced scratch, busybox, alpine, Debian slim, UBI, Distroless, and Chainguard Image as base images for minimizing container size.

scratch

The most primitive way to create a minimal container image is to use an official Docker image called scratch.

This scratch image is used in the bottom layer of many container images. For example, the hello-world image that they usualy use to check if Docker is working properly is created with the following Dockerfile (where hello is the executable binary included in the repository where the image is created.)

FROM scratch
COPY hello /
CMD ["/hello"]

scratch, to put it bluntly, only contains nothing and is only available to talk to the kernel of the host OS, so you need to provide all the other necessary files by yourself. In exchange for the freedom to add the files you only need, you have to copy files at a very primitive level (there is no package manager, of course). If you want to create an application environment, you will need to copy a lot of files, including dependent libraries, and it will be difficult to put everything in a Dockerfile. In many cases, you will need to use a container image that ships with some common dependency libraries.

busybox

The busybox container is, as the name suggests, a base image that contains the minimum environment for BusyBox to work. It is also one of the official Docker images. The actual Dockerfile for creating this image is as follows:

FROM scratch
ADD busybox.tar.gz /
CMD ["sh"]

You can see that busybox.tar.gz¹ is simply extracted on top of the scratch image we introduced earlier.

Although busybox uses ash, it can use shells, is small (1.76MB), and has a choice of standard C libraries (libc): glibc and musl ². This image is an option if you just want to run a simple shell script or a binary that runs by itself.

However, there is no package manager, so if you want to install libraries and so on, you have to copy them in the same way as scratch. You may need to do a lot of work to write a Dockerfile-based description.

alpine

alpine is a base image using a very lightweight Linux distribution called Alpine Linux, which includes BusyBox and musl environments, as well as a package management tool called APK (Alpine Package Manager). The image size is 3.81MB, which is very lightweight. Many of you are probably using alpine as a base image to create container images.

Although alpine is lightweight and the APK makes it easy to create an environment, its use of musl in the libc makes it inconvenient in many situations. For example, there are the following inconveniences:

Does not support extensions provided by glibc
There are thread-ussafe cases when running on musl
Difficulty in handling shared libraries (musl can't do lazy binding and can't unload)
Small thread stack size, easy to encounter segmentation violations

I have had experienced cases where Python and other programs didn't work well with alpine.

Because of these troubles, it is my understanding that the preferences of the base image is now diverse as follows, assuming to use glibc for libc:

a slightly larger image that includes a shell and package manager and can be easily described in a Dockerfile
a very small image with only the necessary configuration for each individual use

Debian slim

debian:xxx-slim is part of Debian's official Docker images and, as the name suggests, is a Debian-based image with a small image size. The shell is bash by default, and you can freely install stable packages released by Debian via apt command.

With an image size of 26.76 MB (stable-slim), it is much larger than other minimum images, but considering that the runtimes of various languages themselves are tens to hundreds of MB, the image size would be affordable as an application image.

Debian slim images are used as the slim base image for official images of many languages due to its stability. (Footnote: Example on Python 3.13.1)

Red Hat Universal Base Image

While most of the official images released by the OSS project are Debian based, there is also demand for images from RHEL-based distributions. For this purpose, Red Hat offers an image called Universal Base Image (UBI).

redhat/ubi9-micro: Minimal environment with almost nothing but bash and glibc, 6.33MB, similar to busybox.
redhat/ubi9-minimal: Same purpose as Debian slim, but with microdnf. 35.84MB.

There are two other image types, Standard and Init, but I don't detail them here because they are out of the scope of the main purpose of the minimum image.

Distroless

Debian slim and UBI are easy to use in the direction of creating a very small image while taking into account the use of a package manager, but on the other hand, they are too large to satisfy the demand to create a minimum image for a specific language environment. If you want to satisfy such a demand, Distroless image may be the right choice for you.

As of January 2025, the latest version of Distroless is based on Debian 12.4. This does not mean that it is based on Debian slim, but rather that it only extracts the deb files from the Debian repositories for the packages needed for scratch.

The image will vary depending on the package it is in, but these are the two basic ones.

gcr.io/distroless/static-debian12 (less than 800KB): contains ca-certificate files, base-files, tzdata and other minimal packages
gcr.io/distroless/base-debian12 (8.5MB): contains glibc and libssl in addition to static-debian12

For example, you can create a very small image on top of static-debian12 with a Go executable binary that can run as a single binary without depending on libc. There are also images based on these two images for commonly used languages, such as the followings for Java and Python.

gcr.io/distroless/java17-debian12 (88MB): contains base-debian12 plus OpenJDK 17 and its dependencies distributed by Debian
gcr.io/distroless/java21-debian12 (68MB): contains Temurin OpenJDK 21 and its dependencies in addition to base-debian12
gcr.io/distroless/python3-debian12 (21MB): contains python3.11-minimal and its dependencies in addition to base-debian12

Since these images do not include the shell, if you really want to use the shell during development, for example, there is a debug image for each of these images. This can be done by specifying the tag debug (e.g. gcr.io/distroless/base-debian12:debug) for the above images, and you will get an image with busybox installed in addition to the environment.

Chainguard Images

As the name suggests, these are publicly available container images from Chainguard. Because Chainguard is a company specialising in container security and supply chain security, these container images have security measures in place in many ways, and the images they distribute advocate "low-to-no CVEs".

Chainguard Images is greatly influenced by Distroless, which is introduced above, and while following the essence of Distroless, it has developed its own unique style. For example:

OSS distribution for container image creation called Wolfi OS
apko, an new OCI container image builder
Developed our own apk package builder called melenge

By taking full advantage of these, they are creating a minimum size container with only packages that are always provided by SBOM and instantly patched for security. The publicly available images they provide, for example, if you want something like busybox or Distroless' base image, are as follows

cgr.dev/chainguard/glibc-dynamic (4.13MB)
cgr.dev/chainguard/busybox (3.59MB)

If you want a small base for additional packages like alpine, or a minimum image for each language like Distroless, you can use the following images:

cgr.dev/chainguard/wolfi-base (5.40MB)
cgr.dev/chainguard/python (20.9MB)
cgr.dev/chainguard/jdk (104.1MB)

Chainguard Images are awesome, but one point to note is that only images classified as Developer Images are freely available. For example, the images listed above are only freely available under the latest tag, and if you want to stick to a specific version, you need to use Chainguard's paid-for service.

Conclusion

In this article, we introduced various base images for container image minimization. There is a wide range of configurations required to create container images when running applications, as many different conditions need to be taken into account. In addition to Docker, there are also other container creation tools such as Buildah, Buildpacks, Bazel, ko, kaniko, etc., that can be used for a variety of purposes. I hope this article will be helpful in determining the best base image for your development process and usage.

The exact tarball is a modified version for each CPU platform. ↩
uClibc is also available, but it is not used by many. ↩

Run remote install scripts without using `curl` command in Ansible

Yoshi Yamaguchi — Thu, 04 Mar 2021 00:54:12 +0000

TL;DR

Use ansible.builtin.uri to fetch the script content, and pass it through stdin parameter of ansible.builtin.shell.

Execution environment

Ansible: 2.10.5
Python: 3.8.7

Sample case

There are a lot of cases where the install instructions of tool X introduces the remote script to run. For instance, rustup gives the script as the primary method to install it.

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

In the case you install rustup via Ansible, you may want to write lines like the following:

- name: Run rustup
  ansible.builtin.shell:
    cmd: curl -sSf https://sh.rustup.rs | sh -s -- -y

(I added -s option to install the script without prompt that blocks the Ansible installation process.)

Ansible warns when it runs the task like this:

TASK [common : Run rustup] **********************************************************************************
[WARNING]: Consider using the get_url or uri module rather than running 'curl'.  If you need to use command
because get_url or uri is insufficient you can add 'warn: false' to this command task or set
'command_warnings=False' in ansible.cfg to get rid of this message.
changed: [dev]

Solution

The easiest way to avoid this is to use get_url to download the shell script and run it.

- name: Download rustup
  ansible.builtin.get_url:
    url: https://sh.rustup.rs
    dest: /tmp/rustup.sh

- name: Run rustup
  ansible.builtin.shell:
    cmd: cat /tmp/rustup.sh | sh -s -- -y

Otherwise, you can directly pass the script contents into stdin of shell with stdin parameter.

- name: Fetch rustup
  ansible.builtin.uri:
    url: https://sh.rustup.rs
    return_content: yes
  register: rustup_installer

- name: Run rustup installer
  ansible.builtin.shell:
    cmd: sh -s -- -y
    stdin: "{{ rustup_installer.content }}"

Automatically installing Cloud Operations agents with Policy Agent

Yoshi Yamaguchi — Wed, 12 Aug 2020 06:25:53 +0000

TL;DR

Agent Policy frees you from installing Google Cloud Logging and Google Cloud Monitoring agents to Google Compute Engine instances.

Preface

In Google Cloud Platform (GCP), Google Cloud Logging and Google Cloud Monitoring (hereinafter called Logging and Monitoring) are integrated with peer GCP services, and the users can observe system/audit logs and system metrics without any configurations. Also, application and container runtimes such as Google App Engine and Google Kubernetes Engine sends logs and metrics automatically to Logging and Monitoring.

However, when you run applications directly on top of Google Compute Engine, you had to install and setup the agents for Google Cloud Logging and Google Cloud Monitoring respectively all by yourselves.

How to setup Agent Policy

The required steps are explained in the official document and this post provides the additional demo to explain how it works.

Install `gcloud alpha` components

Agent Policy is still in alpha, and you need to install the alpha components to try it.

$ gcloud components install alpha

Set up proper privileges

GCP (and any other cloud platforms) always about IAMs. Setting up proper privileges and roles are important, or even mandatory to run specific features. Agent Policy is not the exception and you need to set up multiple IAM roles to multiple users and service accounts. However, the good news is the official supplemental shell script is provided.

set-permissions.sh

The details of what it does are well described in the document. In short, it assigns roles/osconfig to appropriate users and service accounts.

Create Agent Policy

After setting up IAMs, now the main part. Create an Agent Policy with gcloud command.

For example, this example create an Agent Policy named ops-agents-debian: it installs Logging and Monitoring agents to new GCE instances generated from Debian 10 images. (See the list of public images)

$ gcloud alpha compute instances ops-agents policies create ops-agents-debian \
  --agent-rules="type=logging,version=current-major,package-state=installed,enable-autoupgrade=true;type=metrics,version=current-major,package-state=installed,enable-autoupgrade=true" \
  --os-types=short-name=debian,version=10

And all set! This example applies to all Debian 10 images. In the real use cases, you may want to set up more complex condition to apply Agent Policy, and using --group-labels should be better idea. Please find the details of the options of gcloud alpha compute instances ops-agents policies create.

Experiments

Case 1: Create a new instance with Debian 10

Let's see if the policy installs agents to Debian 10 instances. Run the following command.

$ gcloud compute instances create test0 \
  --image-project debian-cloud \
  --image-family=debian-10 \
  --zone=us-central1-a \
  --preemptible \
  --boot-disk-auto-delete
Created [https://www.googleapis.com/compute/v1/projects/agents-install-test/zones/us-central1-a/instances/test0].
NAME   ZONE           MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
test0  us-central1-a  n1-standard-1  true         XX.XX.XX.XX   XX.XX.XX.XX  RUNNING

And confirm if Logging and Monitoring agents are installed and running.

$ gcloud compute ssh test0 --zone=us-central1-a
Writing 3 keys to /home/ymotongpoo/.ssh/google_compute_known_hosts
Enter passphrase for key '/home/ymotongpoo/.ssh/google_compute_engine':
Linux test0 4.19.0-10-cloud-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
ymotongpoo@test0:~$ sudo service google-fluentd status
● google-fluentd.service - LSB: data collector for Treasure Data
   Loaded: loaded (/etc/init.d/google-fluentd; generated)
   Active: active (running) since Tue 2020-08-11 08:50:36 UTC; 1min 17s ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 110 (limit: 4373)
   Memory: 66.9M
   CGroup: /system.slice/google-fluentd.service
           └─2128 /opt/google-fluentd/embedded/bin/ruby /usr/sbin/google-fluentd --log /var/log/google-fluentd/goo

Aug 11 08:50:36 test0 systemd[1]: Starting LSB: data collector for Treasure Data...
Aug 11 08:50:36 test0 google-fluentd[2106]: Starting google-fluentd 1.7.1: google-fluentd.
Aug 11 08:50:36 test0 systemd[1]: Started LSB: data collector for Treasure Data.
ymotongpoo@test0:~$ sudo service stackdriver-agent status
● stackdriver-agent.service - LSB: start and stop Stackdriver Agent
   Loaded: loaded (/etc/init.d/stackdriver-agent; generated)
   Active: active (running) since Tue 2020-08-11 08:50:42 UTC; 1min 32s ago
     Docs: man:systemd-sysv-generator(8)
    Tasks: 13 (limit: 4373)
   Memory: 6.2M
   CGroup: /system.slice/stackdriver-agent.service
           └─2470 /opt/stackdriver/collectd/sbin/stackdriver-collectd -C /etc/stackdriver/collectd.conf -P /var/ru

Aug 11 08:50:42 test0 collectd[2469]: plugin_load: plugin "write_gcm" successfully loaded.
Aug 11 08:50:42 test0 collectd[2469]: plugin_load: plugin "match_regex" successfully loaded.
Aug 11 08:50:42 test0 collectd[2469]: plugin_load: plugin "match_throttle_metadata_keys" successfully loaded.
Aug 11 08:50:42 test0 collectd[2469]: plugin_load: plugin "stackdriver_agent" successfully loaded.
Aug 11 08:50:42 test0 collectd[2469]: plugin_load: plugin "exec" successfully loaded.
Aug 11 08:50:42 test0 collectd[2469]: plugin_load: plugin "aggregation" successfully loaded.
Aug 11 08:50:42 test0 stackdriver-agent[2449]: .
Aug 11 08:50:42 test0 systemd[1]: Started LSB: start and stop Stackdriver Agent.
Aug 11 08:50:42 test0 collectd[2470]: Initialization complete, entering read-loop.
Aug 11 08:50:42 test0 collectd[2470]: tcpconns plugin: Reading from netlink succeeded. Will use the netlink method

Yes, they are running!

Case 2: Create a new instance with CentOS 8

This time, I create an instance with CentOS 8, which doesn't apply to the ops-agent-debian policy I made.

$ gcloud compute instances create test1 \
  --image-project=centos-cloud \
  --image-family=centos-8 \
  --zone=us-central1-a \
  --preemptible \
  --boot-disk-auto-delete
Created [https://www.googleapis.com/compute/v1/projects/agents-install-test/zones/us-central1-a/instances/test0].
NAME   ZONE           MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
test1  us-central1-a  n1-standard-1  true         XX.XX.XX.XX   XX.XX.XX.XX  RUNNING

$ gcloud compute ssh test1 --zone=us-central1-a                                                                  Writing 3 keys to /home/ymotongpoo/.ssh/google_compute_known_hosts
Enter passphrase for key '/home/ymotongpoo/.ssh/google_compute_engine':
[ymotongpoo@test0 ~]$ sudo service google-fluentd status
Redirecting to /bin/systemctl status google-fluentd.service
Unit google-fluentd.service could not be found.

[ymotongpoo@test0 ~]$ sudo service stackdriver-agent status
Redirecting to /bin/systemctl status stackdriver-agent.service
Unit stackdriver-agent.service could not be found.

This time, the agents are not installed.

Notes

As of Aug 12th, 2020, this feature is in alpha and only supported for the direct use of public images on GCE. (The supported short names are centos, debian, rhel, sles, sles-sap and ubuntu) This doesn't support containers running on GCE.

Also, this only works on gcloud command and is not available on Cloud Console and public APIs. In order to confirm existing policies and their details, you need to run the following commands respectively.

gcloud alpha compute instances ops-agents list
gcloud alpha compute instances ops-agents describe POLICY_ID

Because this is still in alpha, we appreciate your feedback. Please drop your experiences to ops-agent-policy-feedback@google.com or @ymotongpoo.

Easter eggs in Go source code

Yoshi Yamaguchi — Tue, 21 Jul 2020 01:14:40 +0000

Preface

A week ago, I had a small chat with my friend over some Slack workspace, something like:

friend: "Does anyone know why time.minWall defaults to 1885?"
me: "No idea, maybe it's from the year in Back To The Future 3?"

At that time, I was almost joking but I had no idea why it was set to 1885 neither. Though the fact behind it does not have anything to do with my daily coding in Go, I couldn't help asking the behind-the-scene. I asked my peer Gophers in the team chat, but it seemed no one there couldn't reach the clues for the decision.

Finally, I sent an email to Russ Cox (@_rsc) directly to know the background for the decision.

`time.minWall`

A const value time.minWall is set to 1885.

src/time/time.go - go - Git at Google

const (
    hasMonotonic = 1 << 63
    maxWall      = wallToInternal + (1<<33 - 1) // year 2157
    minWall      = wallToInternal               // year 1885
    nsecMask     = 1<<30 - 1
    nsecShift    = 30
)

This is the constant value that defines the epoch of values in time package. We often use UNIX epoch (i.e. 00:00:00 UTC on 1st January 1970), but it is just one epoch out of a couple of standards or implementations to express datetime values, and of course Go is a multi platform language, so the epoch in Go need to cover all those platforms.

Russ Cox had given comments on Go's epoch publicly in the following places. The first one is on the GitHub issue thread:

OK, this is checked in for Go 1.9. I moved the internal epoch to 1885 (max year 2157) to avoid any possible problem with NTP-epoch-derived times. I also adjusted the design doc to reflect this change and some minor encoding changes.

But this comment only mentions why and when they moved the internal epoch to 1885, but not why they didn't to any other years, such as 1900.

The second one is the proposal doc for the move of the internal epoch.

Proposal: Monotonic Elapsed Time Measurement in Go

Unix-based systems often use 1970, and Windows-based systems often use 1980. We are unaware of any systems using earlier default wall times, but since the NTP protocol epoch uses 1900, it seemed more future-proof to choose a year before 1900.

Okay, still the rationale supports any years before 1900, not specifically 1885. I was almost sure that the year is derived from Back To The Future 3, but I wanted 100% sure for it and I reached out to the person who only knows the decision behind it; I sent an email to Russ to ask the reason. He gave a response within a day (it's super quick, considering timezone difference between EDT and JST) saying:

Yes, people talked me into moving it back before 1900, and 1885 was the obvious choice due to its historical significance to Hill Valley, California. :-)

I knew it!! Also, it was more than happy that I could get the true answer from who made the decision.

`http.aLongTimeAgo`

While I appreciated the answer from Russ, it was not the end of the story. There was the another line in Russ' reply.

See also http.aLongTimeAgo, which is now sadly set to time.Unix(1, 0) but used to be time.Unix(233431200, 0).

In Go 1.14.5, as he says, it's set to time.Unix(1, 0).

Go 1.14.5: src/net/http/http.go - go - Git at Google

// aLongTimeAgo is a non-zero time, far in the past, used for
// immediate cancellation of network operations.
var aLongTimeAgo = time.Unix(1, 0)

So let's confirm its original value when it was introduced to the source code. You can find it in Go 1.8.

Go 1.8: src/net/http/http.go - go - Git at Google

// aLongTimeAgo is a non-zero time, far in the past, used for
// immediate cancelation of network operations.
var aLongTimeAgo = time.Unix(233431200, 0)

Let's convert the UNIX time to human readable format. Of course, Go gives super easy way to do it.

Sample code to read UNIX time in human readable format

package main

import (
    "fmt"
    "time"
)

func main() {
    pdt, _ := time.LoadLocation("America/Los_Angeles")
    t := time.Unix(233431200, 0).In(pdt)
    fmt.Println(t)
}

The result is:

1977-05-25 11:00:00 -0700 PDT

We got 2 hints: "A Long Time Ago" and "May 25th, 1977". What these 2 lead to is...

Of course, nothing but Star Wars: Episode IV. It was released on May 25th, 1977 (Wikipedia)

How this value is used in Go package is to force cancelling existing connections immediately by giving deadlines in past time, such as:

cr.conn.rwc.SetReadDeadline(aLongTimeAgo)

Some people reacted on the changes when the value was changed to time.Unix(1, 0). I love to see these kinds of comments; those fun chats occasionally happen in changelists.

net, net/http: adjust time-in-past constant even earlier

Acknowledgement

Russ Cox: Thank you for replying to my question and the additional information. Also, thank you for the approval to share this publicly. Quoting his comment:

Feel free. No secrets here.

Chris Broadfoot and Valentin Deleplace: Thank you for finding out the clues together on the group chat.
mattn: Thank you for letting me know the GitHub thread where people reacted on the value.

Note

This entry was originally posted to my Japanese blog on July 17th.

https://ymotongpoo.hatenablog.com/entry/2020/07/17/093000

Search Go packages from navigation bar

Yoshi Yamaguchi — Fri, 31 Jan 2020 19:45:49 +0000

Go team recently announced the new package search system, pkg.go.dev, will take over the search on godoc.org.

It is often the case to use browsers for searching Go package on Google and godoc.org, but it seems it's time to create the shortcut for pkg.go.dev as the engineer who uses Go for daily basis. However, typing "pkg.go.dev" in the browser navigation takes some time and we'd like to reduce the typing effort if we can. So, let's do it!

In Google Chrome, there's a setting for search engine, and there's a section titled "Manage search engines", where you can set up arbitrary URLs for querying something.

You may already have a bunch of search engines registered, and here you can add another one for pkg.go.dev as the following:

In this dialog, you need to put 3 settings, and especially important ones are "Keyword" and "Query URL".

"Query URL" is straightforward. This is the URL template when you query anything with the search engine, and you can specify %s as the placeholder for the query parameters or path name. In the case of pkg.go.dev, the value is https://pkg.go.dev/search?q=%s.

"Keyword" is what makes your productivity awesome. This is the shortcut name you want to type in the navigation bar of Google Chrome. With the combination of "keyword" and tab key, you can trigger the search engine. For instance, I used pkg for the trigger keyword of pkg.go.dev, so if you type pkg in the navigation bar and press tab key, then you'll see the following notation on the left side of the navigation bar:

In that status, now you can type in Go package name you are looking for, such as go.opentelemetry.io, then hit enter key. You can just type in a part of package names as well such as error and so on.

Here you go! Now it got much easier to find package from the browser.

Photo credit is by https://www.thewritingreader.com/blog/2017/11/22/prompt-4079-so-many-packages/

Ignore ESLint rules on specific lines

Yoshi Yamaguchi — Wed, 01 Jan 2020 23:28:11 +0000

On writing codes for Google Apps Script with clasp, it is often the case you get warning from ESLint like "@typescript-eslint/no-unused-vars" on the functions because they are called from triggers in most cases.

For that, you can use inline comments to disable rule of ESLint as follows:

function mp4Organize(): void { // eslint-disable-line @typescript-eslint/no-unused-vars
  ...
}

Reference

ESLint: Configuring ESLint - Disabling Rules with Inline Comments

Easiest and fastest way to run Docker image on GCE

Yoshi Yamaguchi — Sun, 10 Nov 2019 10:37:23 +0000

Because I'm in Developer Relations team, it is often the case where I want to run some Docker image exposed to public for tentative demo purpose, in such as codelab, workshop or hackathon events.

In those cases, I recommend to use Google Compute Engine because it has the feature to run Docker images on booting GCE instances.
Using the feature, you can run and make some service public in 2 commands. One is for firewall setting, the other is GCE instance creation.

For example, when you'd like to run Jaeger all-in-one image and make it public Jaeger endpoint, you can run the following commands:

$ gcloud compute firewall-rules create jaeger-rules \
    --allow udp:5775,udp:6831,udp:6832,tcp:5778,tcp:16686,tcp:14268,tcp:9411 \
    --direction ingress \
    --priority 1000 \
    --target-tags jaeger

$ gcloud compute instances create-with-container jaeger-vm \
    --container-image jaegertracing/all-in-one:1.14 \
    --boot-disk-size 30GB \
    --tags jaeger

Photo credit by https://www.marineinsight.com/guidelines/speed-of-a-ship-at-sea/

Forem: Yoshi Yamaguchi

What're inside OpenTelemetry Collector and ADOT?

TL;DR

What is OpenTelemetry?

What is OpenTelemetry Collector?

What is ADOT (AWS Distro for OpenTelemetry)?

Collector custom build

Conclusion

Revisit base container image for AWS services

TL;DR

scratch

busybox

alpine

Debian slim

Red Hat Universal Base Image

Distroless

Chainguard Images

Conclusion

Run remote install scripts without using `curl` command in Ansible

TL;DR

Execution environment

Sample case

Solution

Automatically installing Cloud Operations agents with Policy Agent

TL;DR

Preface

How to setup Agent Policy

Install gcloud alpha components

Set up proper privileges

Create Agent Policy

Experiments

Case 1: Create a new instance with Debian 10

Case 2: Create a new instance with CentOS 8

Notes

Easter eggs in Go source code

Preface

time.minWall

http.aLongTimeAgo

Acknowledgement

Note

Search Go packages from navigation bar

Ignore ESLint rules on specific lines

Reference

Easiest and fastest way to run Docker image on GCE

Install `gcloud alpha` components

`time.minWall`

`http.aLongTimeAgo`