<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>Forem: Niko Kiirala</title>
    <description>The latest articles on Forem by Niko Kiirala (@nikokiirala).</description>
    <link>https://forem.com/nikokiirala</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F2374087%2F493fb2db-1078-45cf-a5d7-21e2150b2ecd.jpg</url>
      <title>Forem: Niko Kiirala</title>
      <link>https://forem.com/nikokiirala</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://forem.com/feed/nikokiirala"/>
    <language>en</language>
    <item>
      <title>NATing on the cheap on AWS</title>
      <dc:creator>Niko Kiirala</dc:creator>
      <pubDate>Thu, 12 Dec 2024 12:15:27 +0000</pubDate>
      <link>https://forem.com/nikokiirala/nating-on-the-cheap-on-aws-2ln3</link>
      <guid>https://forem.com/nikokiirala/nating-on-the-cheap-on-aws-2ln3</guid>
      <description>&lt;p&gt;Let's consider this case: you have something running in AWS and that something needs occasional access to public internet. You can either give that something public IP addresses or there needs to be a NAT between your private network and the public internet. Let's consider this second case, setting up a NAT. &lt;/p&gt;

&lt;p&gt;In my use case, specifically, I'm running an EC2 instance that has a small web server on it. I also have a CodePipeline + CodeBuild setup that builds and deploys a new version of the server whenever the production branch in the Git repo is pushed to. This CodePipeline + CodeBuild setup is excellently suited for running in a private network as there's no point in having it visible from the public internet. It does, however, need to fetch stuff from the public internet - get dependencies from package repositories and such. Thus I need to have a NAT gateway, but I have no need for all the bells and whistles of AWS-provided NAT gateways. Not to mention, I have just a small budget as well.&lt;/p&gt;

&lt;p&gt;Let me be clear on one thing from the get-go: AWS-provided &lt;a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html" rel="noopener noreferrer"&gt;NAT gateways&lt;/a&gt; work great. Fast, stable, simple to use - all sorts of things you'd want for production infrastructure. Also I'd like to give a shout-out to &lt;a href="https://github.com/chime/terraform-aws-alternat" rel="noopener noreferrer"&gt;alterNAT&lt;/a&gt; that is a production-grade replacement for NAT gateways, and from whom I've copied bits and pieces for my version.&lt;/p&gt;

&lt;p&gt;In case you have some testing environment, low-usage servers or such, a NAT gateway can get costly. The gateway has a base fee, a per-gigabyte fee for traffic and you need to pay for a public IP, too. These can easily end up costing more than the rest of your usage - especially so if you qualify for and use free tier services.&lt;/p&gt;

&lt;p&gt;If you're already running an EC2 instance with a public IP, it's possible to set up a NAT for no cost. Running a low-spec EC2 instance just for this purpose is fairly cheap, too. The solution is a &lt;a href="https://docs.aws.amazon.com/vpc/latest/userguide/work-with-nat-instances.html" rel="noopener noreferrer"&gt;NAT instance&lt;/a&gt;. &lt;/p&gt;

&lt;h2&gt;
  
  
  Set up the EC2 instance
&lt;/h2&gt;

&lt;p&gt;You can use an EC2 instance you already have or set up a new one just for the purpose. This instance needs three important settings:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A public IP address: the NAT gateway needs to be in a public subnet, with an Internet Gateway attached, and it needs to have a public IP address. All traffic from your internal subnets will appear to originate from this IP.&lt;/li&gt;
&lt;li&gt;Source/destination check disabled: by default, EC2 instances can only see their own traffic. Incoming traffic where the target IP is not one of the instance's own IPs is filtered out, as is outgoing traffic where source IP doesn't belong to the instance. However, when the instance is doing routing, we need it to do both, so we need to disable the check.&lt;/li&gt;
&lt;li&gt;User data blob: this contains the necessary instructions to set up the NAT functionality at every boot. The user data blob gets processed by &lt;a href="https://cloudinit.readthedocs.io/en/latest/index.html" rel="noopener noreferrer"&gt;cloud-init&lt;/a&gt; that's active by default in the Amazon Linux 2023 AMI that I'm using.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;I'll be using Terraform to set up my instance.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight terraform"&gt;&lt;code&gt;&lt;span class="k"&gt;data&lt;/span&gt; &lt;span class="s2"&gt;"aws_ami"&lt;/span&gt; &lt;span class="s2"&gt;"arm64-ecs"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;owners&lt;/span&gt;      &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"amazon"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="nx"&gt;name_regex&lt;/span&gt;  &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"^al2023-ami-ecs"&lt;/span&gt;
  &lt;span class="nx"&gt;most_recent&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="nx"&gt;filter&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;name&lt;/span&gt;   &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"architecture"&lt;/span&gt;
    &lt;span class="nx"&gt;values&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"arm64"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_instance"&lt;/span&gt; &lt;span class="s2"&gt;"arm64_machine"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;ami&lt;/span&gt;           &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;aws_ami&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;arm64-ecs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;image_id&lt;/span&gt;
  &lt;span class="nx"&gt;instance_type&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"t4g.small"&lt;/span&gt;

  &lt;span class="nx"&gt;availability_zone&lt;/span&gt;           &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;aws_availability_zones&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;main&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;names&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="nx"&gt;iam_instance_profile&lt;/span&gt;        &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;aws_iam_instance_profile&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;machine_profile&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;name&lt;/span&gt;
  &lt;span class="nx"&gt;key_name&lt;/span&gt;                    &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;aws_key_pair&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;machine_access&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;key_name&lt;/span&gt;
  &lt;span class="nx"&gt;subnet_id&lt;/span&gt;                   &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;aws_subnet&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;main&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;
  &lt;span class="nx"&gt;vpc_security_group_ids&lt;/span&gt;      &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;aws_security_group&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ecs_on_ec2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="nx"&gt;associate_public_ip_address&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="nx"&gt;source_dest_check&lt;/span&gt;           &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;

  &lt;span class="nx"&gt;user_data&lt;/span&gt;                   &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kd"&gt;local&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;instance_cloudconfig&lt;/span&gt;
  &lt;span class="nx"&gt;user_data_replace_on_change&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;

  &lt;span class="nx"&gt;tags&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;merge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;local&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;common_tags&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;Name&lt;/span&gt;             &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="kd"&gt;local&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;service_name&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;-arm64"&lt;/span&gt;
  &lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The three important bits described above are already visible in this code snippet:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;code&gt;associate_public_ip_address = true&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;source_dest_check           = false&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;user_data                   = local.instance_cloudconfig&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  User data blob
&lt;/h2&gt;

&lt;p&gt;I could build a new AMI that has the necessary bits and pieces to set up the instance to perform NAT at every boot. That's a fairly heavy process, though, so instead of that I'm using &lt;em&gt;user data&lt;/em&gt; to drop a small script and a few configuration files to the instance at boot.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight terraform"&gt;&lt;code&gt;&lt;span class="nx"&gt;locals&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;instance_cloudconfig_data&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;write_files&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;path&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"/var/lib/cloud/scripts/per-boot/tinynat.sh"&lt;/span&gt;
        &lt;span class="nx"&gt;permissions&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"0744"&lt;/span&gt;
        &lt;span class="nx"&gt;owner&lt;/span&gt;       &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"root:root"&lt;/span&gt;
        &lt;span class="nx"&gt;content&lt;/span&gt;     &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;file&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"initscript/tinynat.sh"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;path&lt;/span&gt;        &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"/etc/tinynat.conf"&lt;/span&gt;
        &lt;span class="nx"&gt;permissions&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"0644"&lt;/span&gt;
        &lt;span class="nx"&gt;owner&lt;/span&gt;       &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"root:root"&lt;/span&gt;
        &lt;span class="nx"&gt;content&lt;/span&gt;     &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;file&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"initscript/tinynat.conf"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;path&lt;/span&gt;        &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"/etc/tinynat-route-table-ids.csv"&lt;/span&gt;
        &lt;span class="nx"&gt;permissions&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"0644"&lt;/span&gt;
        &lt;span class="nx"&gt;owner&lt;/span&gt;       &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"root:root"&lt;/span&gt;
        &lt;span class="nx"&gt;content&lt;/span&gt;     &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;aws_route_table&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;private&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;path&lt;/span&gt;        &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"/etc/tinynat-private-nets.csv"&lt;/span&gt;
        &lt;span class="nx"&gt;permissions&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"0644"&lt;/span&gt;
        &lt;span class="nx"&gt;owner&lt;/span&gt;       &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"root:root"&lt;/span&gt;
        &lt;span class="nx"&gt;content&lt;/span&gt;     &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;","&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;aws_subnet&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;private&lt;/span&gt;&lt;span class="p"&gt;[*].&lt;/span&gt;&lt;span class="nx"&gt;cidr_block&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="nx"&gt;instance_cloudconfig&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;-&lt;/span&gt;&lt;span class="no"&gt;END&lt;/span&gt;&lt;span class="sh"&gt;
    #cloud-config
    ${jsonencode(local.instance_cloudconfig_data)}
&lt;/span&gt;&lt;span class="no"&gt;  END
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;instance_cloudconfig_data&lt;/code&gt; is a Terraform object, containing &lt;a href="https://cloudinit.readthedocs.io/en/latest/index.html" rel="noopener noreferrer"&gt;cloud-init&lt;/a&gt; instructions. Here, we drop four files in the file system: the setup script, a config file containing IDs of route tables belonging to private networks (likely just one ID), a config file containing IP address ranges of private networks that should get routed and finally a third config file that tells where to find the two other config files.&lt;/p&gt;

&lt;p&gt;The setup script is stored under &lt;code&gt;/var/lib/cloud/scripts/per-boot&lt;/code&gt; so that it runs every time the instance is booted up. It's making iptables changes that are not persisted across power cycles, so it needs to get run at every boot.&lt;/p&gt;

&lt;p&gt;This script, &lt;code&gt;initscript/tinynat.sh&lt;/code&gt; is as follows:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/bin/bash&lt;/span&gt;

&lt;span class="c"&gt;# Based on alterNAT&lt;/span&gt;
&lt;span class="c"&gt;# https://github.com/chime/terraform-aws-alternat/blob/main/scripts/alternat.sh&lt;/span&gt;

&lt;span class="c"&gt;# Send output to a file and to the console&lt;/span&gt;
&lt;span class="c"&gt;# Credit to the alestic blog for this one-liner&lt;/span&gt;
&lt;span class="c"&gt;# https://alestic.com/2010/12/ec2-user-data-output/&lt;/span&gt;
&lt;span class="nb"&gt;exec&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;(&lt;/span&gt;&lt;span class="nb"&gt;tee&lt;/span&gt; /var/log/alternat.log|logger &lt;span class="nt"&gt;-t&lt;/span&gt; user-data &lt;span class="nt"&gt;-s&lt;/span&gt; 2&amp;gt;/dev/console&lt;span class="o"&gt;)&lt;/span&gt; 2&amp;gt;&amp;amp;1

&lt;span class="nb"&gt;shopt&lt;/span&gt; &lt;span class="nt"&gt;-s&lt;/span&gt; expand_aliases

panic&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
  &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"tinyNAT setup failed"&lt;/span&gt;
  &lt;span class="nb"&gt;exit &lt;/span&gt;1
&lt;span class="o"&gt;}&lt;/span&gt;

load_config&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
   &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-f&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$CONFIG_FILE&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
      &lt;span class="nb"&gt;.&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$CONFIG_FILE&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
   &lt;span class="k"&gt;else
      &lt;/span&gt;panic &lt;span class="s2"&gt;"Config file &lt;/span&gt;&lt;span class="nv"&gt;$CONFIG_FILE&lt;/span&gt;&lt;span class="s2"&gt; not found"&lt;/span&gt;
   &lt;span class="k"&gt;fi
   &lt;/span&gt;validate_var &lt;span class="s2"&gt;"route_table_ids_csv"&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$route_table_ids_csv&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
   validate_var &lt;span class="s2"&gt;"private_net_cidr_ranges"&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$private_net_cidr_ranges&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

validate_var&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
   &lt;span class="nv"&gt;var_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
   &lt;span class="nv"&gt;var_val&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
   &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
      &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Config var &lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="nv"&gt;$var_name&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s2"&gt; is unset"&lt;/span&gt;
      &lt;span class="nb"&gt;exit &lt;/span&gt;1
   &lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="c"&gt;# configure_nat() sets up Linux to act as a NAT device.&lt;/span&gt;
&lt;span class="c"&gt;# See https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Instance.html#NATInstance&lt;/span&gt;
configure_nat&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
   &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Beginning NAT configuration"&lt;/span&gt;

   &lt;span class="nb"&gt;local &lt;/span&gt;&lt;span class="nv"&gt;adapter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;ens5

   &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Configuration before enabling NAT:"&lt;/span&gt;
   sysctl net.ipv4.ip_forward net.ipv4.conf.&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;adapter&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;.send_redirects net.ipv4.ip_local_port_range
   &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"iptables default"&lt;/span&gt;
   iptables &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="nt"&gt;-L&lt;/span&gt;
   &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"iptables NAT"&lt;/span&gt;
   iptables &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="nt"&gt;-t&lt;/span&gt; nat &lt;span class="nt"&gt;-L&lt;/span&gt;

   &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Enabling NAT..."&lt;/span&gt;

   &lt;span class="nv"&gt;IFS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;','&lt;/span&gt; &lt;span class="nb"&gt;read&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="nt"&gt;-a&lt;/span&gt; vpc_cidrs &amp;lt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;private_net_cidr_ranges&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
   &lt;span class="k"&gt;for &lt;/span&gt;cidr &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;vpc_cidrs&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
   &lt;span class="k"&gt;do
      &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Adding routing for private network &lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;cidr&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
      &lt;span class="o"&gt;(&lt;/span&gt;iptables &lt;span class="nt"&gt;-t&lt;/span&gt; nat &lt;span class="nt"&gt;-C&lt;/span&gt; POSTROUTING &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;adapter&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; &lt;span class="nt"&gt;-s&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$cidr&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-j&lt;/span&gt; MASQUERADE 2&amp;gt;/dev/null &lt;span class="o"&gt;||&lt;/span&gt;
      iptables &lt;span class="nt"&gt;-t&lt;/span&gt; nat &lt;span class="nt"&gt;-A&lt;/span&gt; POSTROUTING &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;adapter&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; &lt;span class="nt"&gt;-s&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$cidr&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-j&lt;/span&gt; MASQUERADE&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt;
      panic
   &lt;span class="k"&gt;done

   &lt;/span&gt;iptables &lt;span class="nt"&gt;-N&lt;/span&gt; DOCKER-USER
   iptables &lt;span class="nt"&gt;-C&lt;/span&gt; DOCKER-USER &lt;span class="nt"&gt;-i&lt;/span&gt; &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;adapter&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;adapter&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; &lt;span class="nt"&gt;-j&lt;/span&gt; ACCEPT 2&amp;gt;/dev/null &lt;span class="o"&gt;||&lt;/span&gt;
      iptables &lt;span class="nt"&gt;-I&lt;/span&gt; DOCKER-USER &lt;span class="nt"&gt;-i&lt;/span&gt; &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;adapter&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;adapter&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; &lt;span class="nt"&gt;-j&lt;/span&gt; ACCEPT &lt;span class="o"&gt;||&lt;/span&gt; panic

   iptables &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="nt"&gt;-L&lt;/span&gt; DOCKER-USER
   iptables &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="nt"&gt;-t&lt;/span&gt; nat &lt;span class="nt"&gt;-L&lt;/span&gt; POSTROUTING

   &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"NAT configuration complete"&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="c"&gt;# First try to replace an existing route&lt;/span&gt;
&lt;span class="c"&gt;# If no route exists already (e.g. first time set up) then create the route.&lt;/span&gt;
configure_route_table&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
   &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Configuring route tables"&lt;/span&gt;

   &lt;span class="nv"&gt;IFS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;','&lt;/span&gt; &lt;span class="nb"&gt;read&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="nt"&gt;-a&lt;/span&gt; route_table_ids &amp;lt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;route_table_ids_csv&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;

   &lt;span class="k"&gt;for &lt;/span&gt;route_table_id &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;route_table_ids&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
   &lt;span class="k"&gt;do
      &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Attempting to find route table &lt;/span&gt;&lt;span class="nv"&gt;$route_table_id&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
      &lt;span class="nb"&gt;local &lt;/span&gt;&lt;span class="nv"&gt;rtb_id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;aws ec2 describe-route-tables &lt;span class="nt"&gt;--filters&lt;/span&gt; &lt;span class="nv"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;route-table-id,Values&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;route_table_id&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt; &lt;span class="nt"&gt;--query&lt;/span&gt; &lt;span class="s1"&gt;'RouteTables[0].RouteTableId'&lt;/span&gt; | &lt;span class="nb"&gt;tr&lt;/span&gt; &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'"'&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
      &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-z&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$rtb_id&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
         &lt;/span&gt;panic &lt;span class="s2"&gt;"Unable to find route table &lt;/span&gt;&lt;span class="nv"&gt;$rtb_id&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
      &lt;span class="k"&gt;fi

      &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Found route table &lt;/span&gt;&lt;span class="nv"&gt;$rtb_id&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
      &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Replacing route to 0.0.0.0/0 for &lt;/span&gt;&lt;span class="nv"&gt;$rtb_id&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
      aws ec2 replace-route &lt;span class="nt"&gt;--route-table-id&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$rtb_id&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--instance-id&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$INSTANCE_ID&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--destination-cidr-block&lt;/span&gt; 0.0.0.0/0
      &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nv"&gt;$?&lt;/span&gt; &lt;span class="nt"&gt;-eq&lt;/span&gt; 0 &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
         &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Successfully replaced route to 0.0.0.0/0 via instance &lt;/span&gt;&lt;span class="nv"&gt;$INSTANCE_ID&lt;/span&gt;&lt;span class="s2"&gt; for route table &lt;/span&gt;&lt;span class="nv"&gt;$rtb_id&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
         &lt;span class="k"&gt;continue
      fi

      &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Unable to replace route. Attempting to create route"&lt;/span&gt;
      aws ec2 create-route &lt;span class="nt"&gt;--route-table-id&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$rtb_id&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--instance-id&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$INSTANCE_ID&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--destination-cidr-block&lt;/span&gt; 0.0.0.0/0
      &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nv"&gt;$?&lt;/span&gt; &lt;span class="nt"&gt;-eq&lt;/span&gt; 0 &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
         &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Successfully created route to 0.0.0.0/0 via instance &lt;/span&gt;&lt;span class="nv"&gt;$INSTANCE_ID&lt;/span&gt;&lt;span class="s2"&gt; for route table &lt;/span&gt;&lt;span class="nv"&gt;$rtb_id&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
      &lt;span class="k"&gt;else
         &lt;/span&gt;panic &lt;span class="s2"&gt;"Unable to replace or create the route!"&lt;/span&gt;
      &lt;span class="k"&gt;fi
   done&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="c"&gt;# tinynatconfig file containing inputs needed for initialization&lt;/span&gt;
&lt;span class="nv"&gt;CONFIG_FILE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"/etc/tinynat.conf"&lt;/span&gt;

load_config

&lt;span class="nv"&gt;curl_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"curl --silent --fail"&lt;/span&gt;

&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Requesting IMDSv2 token"&lt;/span&gt;
&lt;span class="nv"&gt;token&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nv"&gt;$curl_cmd&lt;/span&gt; &lt;span class="nt"&gt;-X&lt;/span&gt; PUT &lt;span class="s2"&gt;"http://169.254.169.254/latest/api/token"&lt;/span&gt; &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"X-aws-ec2-metadata-token-ttl-seconds: 900"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;alias &lt;/span&gt;&lt;span class="nv"&gt;CURL_WITH_TOKEN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$curl_cmd&lt;/span&gt;&lt;span class="s2"&gt; -H &lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s2"&gt;X-aws-ec2-metadata-token: &lt;/span&gt;&lt;span class="nv"&gt;$token&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;

&lt;span class="c"&gt;# Set CLI Output to text&lt;/span&gt;
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;AWS_DEFAULT_OUTPUT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;

&lt;span class="c"&gt;# Disable pager output&lt;/span&gt;
&lt;span class="c"&gt;# https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-pagination.html#cli-usage-pagination-clientside&lt;/span&gt;
&lt;span class="c"&gt;# This is not needed in aws cli v1 which is installed on the current version of Amazon Linux 2.&lt;/span&gt;
&lt;span class="c"&gt;# However, it may be needed to prevent breakage if they update to cli v2 in the future.&lt;/span&gt;
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;AWS_PAGER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;

&lt;span class="c"&gt;# Set Instance Identity URI&lt;/span&gt;
&lt;span class="nv"&gt;II_URI&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"http://169.254.169.254/latest/dynamic/instance-identity/document"&lt;/span&gt;

&lt;span class="c"&gt;# Retrieve the instance ID&lt;/span&gt;
&lt;span class="nv"&gt;INSTANCE_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;CURL_WITH_TOKEN &lt;span class="nv"&gt;$II_URI&lt;/span&gt; | &lt;span class="nb"&gt;grep &lt;/span&gt;instanceId | &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="nt"&gt;-F&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt; &lt;span class="s1"&gt;'{print $4}'&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;

&lt;span class="c"&gt;# Set region of NAT instance&lt;/span&gt;
&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;AWS_DEFAULT_REGION&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;CURL_WITH_TOKEN &lt;span class="nv"&gt;$II_URI&lt;/span&gt; | &lt;span class="nb"&gt;grep &lt;/span&gt;region | &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="nt"&gt;-F&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt; &lt;span class="s1"&gt;'{print $4}'&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;

&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Beginning self-managed NAT configuration"&lt;/span&gt;
configure_nat
configure_route_table
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Configuration completed successfully!"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the configuration file &lt;code&gt;initscript/tinynat.conf&lt;/code&gt; is simple two-line affair.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;route_table_ids_csv&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/etc/tinynat-route-table-ids.csv
&lt;span class="nv"&gt;private_net_cidr_ranges&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/etc/tinynat-private-nets.csv
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Permissions setup
&lt;/h2&gt;

&lt;p&gt;The init script reads info about the route table of your internal network and modifies it so that the NAT instance is the default gateway of the internal network. In order for it to do that, the EC2 instance needs an IAM policy that allows those two actions.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight terraform"&gt;&lt;code&gt;&lt;span class="k"&gt;data&lt;/span&gt; &lt;span class="s2"&gt;"aws_iam_policy_document"&lt;/span&gt; &lt;span class="s2"&gt;"tinynat_ec2_policy"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;statement&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;sid&lt;/span&gt;    &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"tinynatDescribeRoutePermissions"&lt;/span&gt;
    &lt;span class="nx"&gt;effect&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"Allow"&lt;/span&gt;
    &lt;span class="nx"&gt;actions&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="s2"&gt;"ec2:DescribeRouteTables"&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="nx"&gt;resources&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"*"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="nx"&gt;statement&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;sid&lt;/span&gt;    &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"tinynatModifyRoutePermissions"&lt;/span&gt;
    &lt;span class="nx"&gt;effect&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"Allow"&lt;/span&gt;
    &lt;span class="nx"&gt;actions&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="s2"&gt;"ec2:CreateRoute"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s2"&gt;"ec2:ReplaceRoute"&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="nx"&gt;resources&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="nx"&gt;for&lt;/span&gt; &lt;span class="nx"&gt;route_table&lt;/span&gt; &lt;span class="nx"&gt;in&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;aws_route_table&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;private&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
      &lt;span class="err"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"arn:aws:ec2:&lt;/span&gt;&lt;span class="k"&gt;${data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;aws_region&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;current&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;name&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;:&lt;/span&gt;&lt;span class="k"&gt;${data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;aws_caller_identity&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;current&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;:route-table/&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;route_table&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;resource&lt;/span&gt; &lt;span class="s2"&gt;"aws_iam_role_policy"&lt;/span&gt; &lt;span class="s2"&gt;"tinynat_ec2"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;name&lt;/span&gt;   &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"tinynat-policy"&lt;/span&gt;
  &lt;span class="nx"&gt;policy&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;aws_iam_policy_document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;tinynat_ec2_policy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;json&lt;/span&gt;
  &lt;span class="nx"&gt;role&lt;/span&gt;   &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;aws_iam_role&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;machine_role&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;name&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You'll also need to verify that your security groups allow instances in your private network to contact the NAT instance, using any protocols and ports they may need for their internet-facing traffic. Also, the NAT instance needs to be allowed to receive such traffic and to send it to public internet. Quite likely, said traffic will be TCP to port 443, aka. HTTPS connections.&lt;/p&gt;

&lt;h2&gt;
  
  
  Wrap-up
&lt;/h2&gt;

&lt;p&gt;With all this set up, servers in your internal network should now be able to contact the public internet.&lt;/p&gt;

&lt;p&gt;When viewing the setup through AWS web UI, the resource map of your VPC should show that your private networks are connected to the routing table you specified in &lt;code&gt;local.instance_cloudconfig_data&lt;/code&gt;, but it won't show where that table is routing the traffic to. Viewing the route table, you should see a route with destination &lt;code&gt;0.0.0.0/0&lt;/code&gt; i.e. the default route with an ENI as its target. Finally, viewing that ENI, under Instance Details there should be a link to your NAT instance.&lt;/p&gt;

&lt;p&gt;This, I hope, makes running small-time applications and various under AWS more approachable. For me, at least, this approach provided me major cost savings for running a tiny application, since I could make my existing EC2 instance double as a NAT gateway instead of running one separately.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>network</category>
      <category>devops</category>
      <category>terraform</category>
    </item>
    <item>
      <title>Compile Protocol Buffers &amp; gRPC to Typescript with Yarn</title>
      <dc:creator>Niko Kiirala</dc:creator>
      <pubDate>Wed, 27 Nov 2024 09:11:35 +0000</pubDate>
      <link>https://forem.com/nikokiirala/compile-protocol-buffers-grpc-to-typescript-with-yarn-54j</link>
      <guid>https://forem.com/nikokiirala/compile-protocol-buffers-grpc-to-typescript-with-yarn-54j</guid>
      <description>&lt;p&gt;Earlier I explored on the high level why one might use Protocol Buffers and gRPC. Now I'll start to go into the details how this is actually implemented.&lt;/p&gt;

&lt;p&gt;There's a &lt;a href="https://github.com/kiirala/web-grpc" rel="noopener noreferrer"&gt;web-grpc repository on GitHub&lt;/a&gt; that shows a full example implementation of what I describe in this post.&lt;/p&gt;

&lt;p&gt;There are various libraries for protobuf and gRPC on web, and they have significant differences. I will use the &lt;a href="https://github.com/stephenh/ts-proto" rel="noopener noreferrer"&gt;ts-proto&lt;/a&gt; library - this library generates idiomatic Typescript classes at compile time from &lt;code&gt;.proto&lt;/code&gt; message definitions, giving a good development experience. You get working autocomplete in IDE, compile-time type checking and reading and writing message fields works like with any other TS object. It also supports both protobuf and JSON wire formats for data. It does not implement gRPC out of the box, but provides support for gRPC-Web proxy connections. For my use case, where I encode &lt;a href="https://cloud.google.com/endpoints/docs/grpc/transcoding" rel="noopener noreferrer"&gt;gRPC as HTTP+JSON&lt;/a&gt; requests, there's unfortunately no direct support currently.&lt;/p&gt;

&lt;p&gt;In order to integrate protobuf build into Yarn build pipeline, we'll need multiple parts, and unfortunately many of them are quite non-obvious to set up:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;protoc compiler: reads &lt;code&gt;.proto&lt;/code&gt; files, and uses plugins to write implementations for them in various languages&lt;/li&gt;
&lt;li&gt;protoc plugin for Typescript: included in the &lt;code&gt;ts-proto&lt;/code&gt; library, creates Typescript class definitions and encoding/decoding methods for them&lt;/li&gt;
&lt;li&gt;Protocol Buffers common types, distributed as a large collection of &lt;code&gt;.proto&lt;/code&gt; files&lt;/li&gt;
&lt;li&gt;Compilation script to call &lt;code&gt;protoc&lt;/code&gt; with the right parameters, paths, etc.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  protoc compiler
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;protoc&lt;/code&gt; compiler can be installed &lt;a href="https://grpc.io/docs/protoc-installation/" rel="noopener noreferrer"&gt;by various means&lt;/a&gt;. I am going to use the &lt;a href="https://www.npmjs.com/package/grpc-tools" rel="noopener noreferrer"&gt;grpc-tools&lt;/a&gt; NPM package - a method not included in the official documents - since that allows building the packages with Yarn without installing additional software. The downside of this is limiting the platforms where the build can be done: ARM processors are not supported, Linux builds must be done on glibc-based distros and likely other similar limitations.&lt;/p&gt;

&lt;h2&gt;
  
  
  Protocol Buffer common types
&lt;/h2&gt;

&lt;p&gt;Common types extend what can be described in protocol buffer messages. Some of the types, such as types for dates and times, are distributed with the compiler. Vast majority of the types are, however, distributed separately. Most importantly, we'll need the &lt;code&gt;google.api.http&lt;/code&gt; type that is used to describe the gRPC to HTTP+JSON mapping.&lt;/p&gt;

&lt;p&gt;There are again many ways to get the common types. They can be copied into your codebase next to your own &lt;code&gt;.proto&lt;/code&gt; files. They can be linked as a submodule to your Git repository. I prefer not to pull this kind of dependencies to my codebase, however, so I'm using the &lt;a href="https://www.npmjs.com/package/google-proto-files" rel="noopener noreferrer"&gt;google-proto-files&lt;/a&gt; NPM package.&lt;/p&gt;

&lt;h2&gt;
  
  
  Compilation script
&lt;/h2&gt;

&lt;p&gt;It's time to put the compilation script together. The script will need to locate the &lt;code&gt;protoc&lt;/code&gt; binary, the Typescript plugin, the common type files and your own &lt;code&gt;.proto&lt;/code&gt; files on disk and to call the &lt;code&gt;protoc&lt;/code&gt; binary with the correct parameters.&lt;/p&gt;

&lt;p&gt;When using Yarn with the &lt;a href="https://yarnpkg.com/features/pnp" rel="noopener noreferrer"&gt;PnP&lt;/a&gt; mode of installing packages, we need special considerations. Usually Yarn in PnP mode installs packages as &lt;code&gt;.zip&lt;/code&gt; files instead of populating a massive &lt;code&gt;node_modules&lt;/code&gt; directory. However, running a binary from inside a &lt;code&gt;.zip&lt;/code&gt; file is non-trivial and even if we'd manage that, the &lt;code&gt;protoc&lt;/code&gt; binary would not know how to load a plugin or &lt;code&gt;.proto&lt;/code&gt; files from inside a &lt;code&gt;.zip&lt;/code&gt; file. To support such cases, Yarn allows installing packages in unplugged mode, where the contents of single packages are extracted as ordinary files on disk.&lt;/p&gt;

&lt;p&gt;Some packages define unplugged mode by default, for example &lt;a href="https://www.npmjs.com/package/grpc-tools" rel="noopener noreferrer"&gt;grpc-tools&lt;/a&gt;, but all packages might not. Either they haven't been written with PnP in mind, or their main use case might not require unplugging. To unplug these, we can add a &lt;code&gt;dependenciesMeta&lt;/code&gt; section to our &lt;code&gt;package.json&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="nl"&gt;"dependenciesMeta"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"google-proto-files@4.2.0"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"unplugged"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"ts-proto@2.4.0"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"unplugged"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note that the packages in &lt;code&gt;dependenciesMeta&lt;/code&gt; are specifies with their exact version number. This number needs to match whichever version is actually installed, so I would highly recommend editing &lt;code&gt;devDependencies&lt;/code&gt; section that these packages are imported by their exact version number, too. For example, the dependency should be &lt;code&gt;"ts-proto": "2.4.0",&lt;/code&gt; instead of the default format &lt;code&gt;"ts-proto": "^2.4.0",&lt;/code&gt; - note the caret in the latter one.&lt;/p&gt;

&lt;p&gt;With the dependencies installed and unplugged, we can create a Node program to locate the various pieces and run the &lt;code&gt;protoc&lt;/code&gt; program with correct parameters. The full version of this script is &lt;a href="https://github.com/kiirala/web-grpc/blob/main/web-client/bin/build-proto.js" rel="noopener noreferrer"&gt;build-proto.js (GitHub)&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In this script we can import the &lt;code&gt;pnp&lt;/code&gt; module to locate any dependencies.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nx"&gt;pnp&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;../.pnp.cjs&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nx"&gt;pnp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setup&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;selfName&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;protocPath&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;pnp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;resolveRequest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;grpc-tools/bin/protoc&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;selfName&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;tsExtension&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;pnp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;resolveRequest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;ts-proto/protoc-gen-ts_proto&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;selfName&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;protoLibsPath&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;pnp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;resolveToUnqualified&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;google-proto-files&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;selfName&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;These functions are further described in the &lt;a href="https://yarnpkg.com/advanced/pnpapi" rel="noopener noreferrer"&gt;PnP API documentation&lt;/a&gt;. Importantly here, &lt;code&gt;resolveRequest&lt;/code&gt; locates a specific file inside a package, whereas &lt;code&gt;resolveToUnqualified&lt;/code&gt; can locate directories, like the root directory of a package here. We also could use &lt;code&gt;import pnp from 'pnpapi'&lt;/code&gt; as described in the documentation, but then this script could only be run through &lt;code&gt;yarn node bin/build-proto.js&lt;/code&gt;. As a flip side, when importing the &lt;code&gt;.pnp.cjs&lt;/code&gt; directly, we need to be explicit about where the build script is located compared to the package root directory.&lt;/p&gt;

&lt;p&gt;Before invoking the compiler, we still need to locate the &lt;code&gt;.proto&lt;/code&gt; files to compile. Here, the code is written with the assumption that the files are all located in a single directory and the directory name is passed as a command line argument to the script.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nx"&gt;path&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;node:path&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;readdir&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;node:fs/promises&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;protoDir&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;protoFiles&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;readdir&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;protoDir&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;endsWith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;.proto&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;protoDir&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;x&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;replaceAll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="se"&gt;\\&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;As an improvement, it would be useful to change this so that it recurses into any subdirectories and locates any &lt;code&gt;.proto&lt;/code&gt; files in those, too.&lt;/p&gt;

&lt;p&gt;Finally, the script invokes &lt;code&gt;protoc&lt;/code&gt; with parameters derived from these.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;outputDir&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;src/proto&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
  &lt;span class="s2"&gt;`--plugin=&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;tsExtension&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;`--ts_proto_out=&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;outputDir&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;--ts_proto_opt=esModuleInterop=true&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;`--proto_path=&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;protoDir&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;`--proto_path=&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;protoLibsPath&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;...&lt;/span&gt;&lt;span class="nx"&gt;protoFiles&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;];&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;protoc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;spawn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;protocPath&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;args&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;protoc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;stdout&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;on&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;data&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`stdout: &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="nx"&gt;protoc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;on&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;data&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`stderr: &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="nx"&gt;protoc&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;on&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;close&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;code&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`child process exited with code &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;code&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now we have the ability to run the compilation manually. In order to run it automatically whenever the package is installed, we can use the &lt;a href="https://dev.toyarn-plugin-after-install"&gt;https://github.com/mhassan1/yarn-plugin-after-install&lt;/a&gt; plugin for Yarn. This plugin can be installed by calling &lt;code&gt;yarn plugin import&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;yarn plugin import https://raw.githubusercontent.com/mhassan1/yarn-plugin-after-install/v0.6.0/bundles/@yarnpkg/plugin-after-install.js
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When installed, we can add a &lt;code&gt;afterInstall&lt;/code&gt; clause to &lt;code&gt;.yarnrc.yml&lt;/code&gt; configuration file.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;afterInstall&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;yarn node bin/build-proto.js ../proto&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now, whenever we run &lt;code&gt;yarn&lt;/code&gt; to install dependencies, the &lt;code&gt;.proto&lt;/code&gt; files will get built automatically, too.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ yarn
➤ YN0000: · Yarn 4.5.3
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0000: └ Completed in 0s 259ms
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0000: └ Completed
Running `afterInstall` hook...
Working directory: /workspaces/web-grpc/web-client
Script directory: /workspaces/web-grpc/web-client/bin
Script name /workspaces/web-grpc/web-client/bin/build-proto.js
/workspaces/web-grpc/web-client/.yarn/unplugged/google-proto-files-npm-4.2.0-28512554de/node_modules/google-proto-files/
Running protoc [
  '/workspaces/web-grpc/web-client/.yarn/unplugged/grpc-tools-npm-1.12.4-956df6794d/node_modules/grpc-tools/bin/protoc',
  '--plugin=/workspaces/web-grpc/web-client/.yarn/unplugged/ts-proto-npm-2.4.0-c5c2c1ec55/node_modules/ts-proto/protoc-gen-ts_proto',
  '--ts_proto_out=src/proto',
  '--ts_proto_opt=esModuleInterop=true',
  '--proto_path=../proto',
  '--proto_path=/workspaces/web-grpc/web-client/.yarn/unplugged/google-proto-files-npm-4.2.0-28512554de/node_modules/google-proto-files/',
  '../proto/notes.proto',
  '../proto/ping.proto'
]
child process exited with code 0
➤ YN0000: · Done with warnings in 1s 101ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This setup gets us a long way to an excellent build pipeline. Without installing extra tools on your dev system, or on a build machine, this setup gives us Typescript definitions for our protobuf messages. There are improvements to be had as well. Automated builds as the &lt;code&gt;.proto&lt;/code&gt; files are modified would be a big one. Expanding the range of supported platforms with Windows and ARM would be big, too.&lt;/p&gt;

&lt;p&gt;In next posts I'll look into building the same &lt;code&gt;.proto&lt;/code&gt; files for a .NET server and connecting the web client and the server together. Also, remember to check out &lt;a href="https://github.com/kiirala/web-grpc" rel="noopener noreferrer"&gt;web-grpc repository on GitHub&lt;/a&gt; for a full implementation of the techniques described in this post.&lt;/p&gt;

</description>
      <category>protobuf</category>
      <category>typescript</category>
      <category>webdev</category>
      <category>yarn</category>
    </item>
    <item>
      <title>Strongly typed web APIs with gRPC</title>
      <dc:creator>Niko Kiirala</dc:creator>
      <pubDate>Thu, 07 Nov 2024 14:46:11 +0000</pubDate>
      <link>https://forem.com/nikokiirala/strongly-typed-web-apis-with-grpc-2ne</link>
      <guid>https://forem.com/nikokiirala/strongly-typed-web-apis-with-grpc-2ne</guid>
      <description>&lt;p&gt;Your web application receives JSON data from the server and shows it in the UI. Simple, right? Indeed, but also filled with pitfalls. Mistyped a field name in UI code? UI crashes. Server leaves out a field you thought would always be there? UI crashes. You quickly become vigilant about checking that you actually received the data you expected to receive, but it never ceases being a burden.&lt;/p&gt;

&lt;p&gt;I’m going to describe using gRPC, a cross-platform framework for APIs, to make sure the client and the server agree on message contents. This gives us several great features:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Single source of truth. Each API is described in Protocol Buffers language, also known as a &lt;code&gt;.proto&lt;/code&gt; file, and in that language only. From this description you can generate API code for just about any commonly used programming language out there.&lt;/li&gt;
&lt;li&gt;Clear upgrade path. Protocol Buffer messages have clearly defined semantics for common changes like adding and removing fields from messages. Very often you can just update message description and old clients work seamlessly with new servers and old servers work with new clients as well. When the API changes are too large, it’s simple to do a versioned API.&lt;/li&gt;
&lt;li&gt;Cross-platform. A single server implementation can easily talk to a Typescript web app, an Android app, an iOS app and more.&lt;/li&gt;
&lt;li&gt;Compact on-wire binary format. OK, this is only partially true – a lot of what I’ll describe will use JSON as the wire format between the client and the server. But typically gRPC would use the Protocol Buffer binary format that is often significantly smaller than JSON.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What is gRPC?
&lt;/h2&gt;

&lt;p&gt;The gRPC is a cross-platform framework for communicating between different pieces of software over network. It was originally developed by Google and is extensively used within Google, but is not in any way tied to them. You might think the name is related to the origins, but it &lt;em&gt;obviously&lt;/em&gt; is just a recursive acronym for “gRPC Remote Procedure Calls”. It’s closely related to Protocol Buffers, a domain-specific language for describing messages and a binary format for storing these messages.&lt;/p&gt;

&lt;p&gt;In a typical application, a gRPC server receives and transmits messages in Protocol Buffers binary format over a HTTP/2 channel. This works nicely for server-to-server communications and for Android, iOS, Windows, Mac, Linux, etc. applications.&lt;/p&gt;

&lt;p&gt;For clients running inside a web browser, however, a different approach must be taken. While modern browsers can talk HTTP/2, a gRPC implementation would require a low-level access to the HTTP/2 connection, and no browser provides that and likely never will. Instead we’ll need to use some sort of a translation layer. This can either be a separate proxy server, or built-in to the application server. With this layer, the browsers can do HTTP requests as usual and the data can be transmitted either in the Protocol Buffers binary format or in the JSON format.&lt;/p&gt;

&lt;p&gt;In browser applications, using JSON as wire format gives certain advantages. Notably the in-browser developer tools automatically pretty-print JSON messages for you, whereas it’s practically impossible to read a Protocol Buffers message in the network traffic view. This is not an all-or-nothing choice, though, since HTTP content negotiation features make it possible to write servers and clients that can use either format. For example, debug version of your app could use JSON and production version Protocol Buffers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Typescript for write-time type checking
&lt;/h2&gt;

&lt;p&gt;Typescript, an extended version of Javascript developed by Microsoft, is the modern approach in web application development to giving types to variables and for doing compile-time type checking. Not the only one, mind – for example, Google separately developed their own way of attaching type info to Javascript that used type declarations in specially formatted code comments.&lt;/p&gt;

&lt;p&gt;Typescript helps somewhat with handling API message contents. If you manually create a type definition for the data you expect to receive, your IDE can auto-complete field names for you and immediately warn you for mistyped field names and for assuming a wrong type for a field. If your backend is also written in Typescript, you can even share the same type declaration between the server and the client. With that, you could already be fairly certain that the messages sent by the server match the ones the clients expect to receive.&lt;/p&gt;

&lt;p&gt;It is possible to generate Typescript API interfaces and message type declarations from protocol buffer files. With this approach, you only need to declare the message types once instead of writing them separately for each programming language you use. This is especially important when modifying the message contents, such as adding new fields. If you need to update several declarations separately they are likely to diverge, but a single declaration will not.&lt;/p&gt;

&lt;p&gt;The gRPC client libraries don’t just give compile-time type checks, but also runtime assistance. Say, what happens when you leave a field unset on the server and send the message as JSON? Depending on your JSON library and its settings, typically the field would either get left out of the message altogether or it would get set to null. The gRPC client libraries, however, ensure that the runtime object decoded from the JSON contains all the fields that were declared in the proto file at the compile time and that they are of the correct type.&lt;/p&gt;

&lt;h2&gt;
  
  
  Coming up
&lt;/h2&gt;

&lt;p&gt;In future posts I’m going to delve into the details on how exactly the above gets done. Especially I will look into building a C#/.NET server that talks with a Typescript/React application that uses Yarn as package manager. These topics will include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;How to automatically compile &lt;code&gt;.proto&lt;/code&gt; files

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/nikokiirala/compile-protocol-buffers-grpc-to-typescript-with-yarn-54j"&gt;Into Typescript code using Yarn&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Into a .NET package using &lt;code&gt;csproj&lt;/code&gt; definition and &lt;code&gt;dotnet&lt;/code&gt; tools&lt;/li&gt;
&lt;/ul&gt;


&lt;/li&gt;

&lt;li&gt;How to write a .NET single controller that talks to both native gRPC clients and to web clients&lt;/li&gt;

&lt;li&gt;How to call these gRPC services from a React app&lt;/li&gt;

&lt;/ul&gt;

</description>
      <category>webdev</category>
      <category>typescript</category>
      <category>fullstack</category>
      <category>grpc</category>
    </item>
  </channel>
</rss>
