Forem: wayofthepie

Decode a certificate

wayofthepie — Mon, 22 Jun 2020 20:40:55 +0000

Introduction

In this post I will leave a note at the end of some sections linking to the latest code up to that point. It will look like this:

🔗 wayofthepie/cert-decoder@a9d761e

That link points to the latest code from the last post.

Some missing things
Read a certificate
- x509-parser API
Validate the file is a certificate
- Error handling
- Refactor
- Parse the der encoded cert
- Print the certificate
Tiny refactor
Conclusion

Some missing things

After publishing the first post I realized I missed a positive test, a test which checks that everything went ok. Let's write that.

#[test]
fn should_succeed() {
    let args = vec!["a-file".to_owned()];
    let validator = FakeValidator { is_file: true };
    let result = execute(validator, args);
    assert!(result.is_ok());
}

It's a good idea to see it fail first. Let's change a behaviour it expects by making it return an error if the file does exist:

- if !validator.is_file(path) {
+ if validator.is_file(path) {
   return Err(
...

Now it will fail:

✦ ➜ cargo -q test should_succeed

running 1 test
F
failures:

---- test::should_succeed stdout ----
thread 'test::should_succeed' panicked at 'assertion failed: result.is_ok()', src/main.rs:98:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    test::should_succeed

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 2 filtered out

error: test failed, to rerun pass '--bin cert-decode'

And if we revert our change:

- if validator.is_file(path) {
+ if !validator.is_file(path) {
...

It will pass:

✦ ➜ cargo -q test should_succeed

running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out

Great! Now for some new things.

🔗 wayofthepie/cert-decoder@66da25a

Read a certificate

Time to read a certificate. The x509-parser crate will allow us to do this. Add as a dependency to Cargo.toml.

[package]
name = "cert-decode"
version = "0.1.0"
authors = ["Stephen OBrien <wayofthepie@users.noreply.github.com>"]
edition = "2018"

[dependencies]
x509-parser = "0.7.0"

📝 Note

I use cargo-edit to update Cargo.toml. You can install it by running:
✦ ➜ cargo install cargo-edit
Then to add a dependency you just need to run:
✦ ➜ cargo add x509-parser
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding x509-parser v0.7.0 to dependencies
See the README for more details. The code blocks in this note have
incorrect formatting, they are centered. I raised a bug about this, see thepracticaldev/dev.to#8767.

It's a good idea to rebuild when adding a new dependency.

✦ ➜ cargo build
    Updating crates.io index
   Compiling autocfg v1.0.0
   Compiling bitflags v1.2.1
   Compiling ryu v1.0.5
   Compiling lexical-core v0.7.4
   Compiling memchr v2.3.3
   Compiling version_check v0.9.2
   Compiling arrayvec v0.5.1
   Compiling static_assertions v1.1.0
   Compiling cfg-if v0.1.10
   Compiling libc v0.2.71
   Compiling base64 v0.11.0
   Compiling nom v5.1.2
   Compiling num-traits v0.2.12
   Compiling num-integer v0.1.43
   Compiling num-bigint v0.2.6
   Compiling time v01.3
   Compiling rusticata-macros v2.1.0
   Compiling der-parser v3.0.4
   Compiling x509-parser v0.7.0
   Compiling cert-decode v0.1.0 (/home/chaospie/repos/blog-cert-decode/cert-decode)
    Finished dev [unoptimize + debuginfo] target(s) in 8.70s

The x509-parser crate has pulled in a bunch of transitive dependencies. Not too many though. When working on a larger project you may want to view the overall dependency hierarchy. You can do this with cargo tree. For example:

✦ ➜ cargo tree
cert-decode v0.1.0 (/home/chaospie/repos/blog-cert-decode/cert-decode)
└── x509-parser v0.7.0
    ├── base64 v0.11.0
    ├── der-parser v3.0.4
    │   ├── nom v5.1.2
    │   │   ├── lexical-core v0.7.4
    │   │   │   ├── arrayvec v0.5.1
    │   │   │   ├── bitflags v1.2.1
    │   │   │   ├── cfg-if v0.1.10
    │   │   │   ├── ryu v1.0.5
    │   │   │   └── static_assertions v1.1.0
    │   │   └── memchr v2.3.3
    │   │   [build-dependencies]
    │   │   └── version_check v0.9.2
    │   ├── num-bigint v0.2.6
    │   │   ├── num-integer v0.1.43
    │   │   │   └── num-traits v0.2.12
    │   │   │       [build-dependencies]
    │   │   │       └── autocfg v1.0.0
    │   │   │   [build-dependencies]
    │   │   │   └── autocfg v1.0.0
    │   │   └── num-traits v0.2.12 (*)
    │   │   [build-dependencies]
    │   │   └── autocfg v1.0.0
    │   └── rusticata-macros v2.1.0
    │       └── nom v5.1.2 (*)
    ├── nom v5.1.2 (*)
    ├── num-bigint v0.2.6 (*)
    ├── rusticata-macros v2.1.0 (*)
    └── time v0.1.43
        └── libc v0.2.71

📝 Note

As of Rust 1.44.0 cargo tree is part of cargo if you are using a version before that you will need to install cargo-tree. You should update to the latest Rust if there is no reason to be on a version less than 1.44.0.

I've used the x509-parser crate in the past so I know a bit about it's API. But let's do some exploration anyway. First, let's set a goal and make a tiny test list. Our goal is simply to be able to print certificate details. Up to now, we have verified the argument we pass is a file, so I think what we should do next is:

Read and print certificate

☐ Validate file is a certificate
☐ Print the certificate

Let's dive into the docs for x509-parser.

x509-parser API

All crates published to crates.io should have docs on docs.rs. These docs will contain the public API of the crate and whatever further documentation the author added. We're using version 0.7.0 of the x509-parser crate, the docs for this are here. The very first example in these docs does almost what we need:

use x509_parser::parse_x509_der;

static IGCA_DER: &'static [u8] = include_bytes!("../assets/IGC_A.der");

let res = parse_x509_der(IGCA_DER);
match res {
    Ok((rem, cert)) => {
        assert!(rem.is_empty());
        //
        assert_eq!(cert.tbs_certificate.version, 2);
    },
    _ => panic!("x509 parsing failed: {:?}", res),
}

It seems we could use the parse_x509_der function to parse our certificate. Our certificate should be in PEM format however, that was a constraint we set in the initial post. Is there anything that can deal directly with PEM certificates in this API?

There is! The x509_parser::pem module has functionality for doing just this. The second example in that modules docs does just what we want, it uses the pem_to_der function to convert a PEM encoded certificate into DER (Distinguished Encoding Rules) and then calls parse_x509_der on that DER to build a X509Certificate. Here is the example:

use x509_parser::pem::pem_to_der;
use x509_parser::parse_x509_der;

static IGCA_PEM: &'static [u8] = include_bytes!("../assets/IGC_A.pem");

let res = pem_to_der(IGCA_PEM);
match res {
    Ok((rem, pem)) => {
        assert!(rem.is_empty());
        //
        assert_eq!(pem.label, String::from("CERTIFICATE"));
        //
        let res_x509 = parse_x509_der(&pem.contents);
        assert!(res_x509.is_ok());
    },
    _ => panic!("PEM parsing failed: {:?}", res),
}

Don't worry if you don't understand everything in this example, it's not too important for this post. Let's look closer at pem_to_der.

pub fn pem_to_der<'a>(i: &'a [u8]) -> IResult<&'a [u8], Pem, PEMError>

It takes a &'a [u8], a slice of bytes with a lifetime¹ of 'a, and returns IResult<&'a [u8], Pem, PEMError>. In short lifetimes tell the compiler how long a reference lives. In this specific case, the slice i which the function takes as an argument must live as long as the slice returned in the IResult return type, as both have a lifetime of 'a. This tells us the slice in the return type must either be i or a subslice of i. For more information see Generic Types, Traits, and Lifetimes.

Glossing over some other details which are outside the scope of this post, this IResult is effectively just a Result type which we saw in the first post. It can return Ok with some value or Err with an error. In this case, the type of the value in Err will be PEMError.

Validate the file is a certificate

Now we have enough knowledge to write a test in the case our cert is not PEM encoded. First, let's get a cert and save it so we can use that in our tests.

➜ openssl s_client -connect google.com:443 2>/dev/null < /dev/null \
>     | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p' > google.com.crt

➜ cat google.com.crt
-----BEGIN CERTIFICATE-----
MIIJTzCCCDegAwIBAgIQVvrczQ6+8BwIAAAAAENV5zANBgkqhkiG9w0BAQsFADBC
MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVR29vZ2xlIFRydXN0IFNlcnZpY2VzMRMw
...
d5JOd+lJOypPGs0/p5OrR8B84Y7wyKFD/EXaKYVMZ4RUXnoAi5DF5RLKNAmnt7R9
V6z8Kz2boaY5oZ0gvrA49R6T+u3yrstte931N49lwpaVsoA=
-----END CERTIFICATE-----

I've stored this cert in the repo here. The test is as follows:

#[test]
fn should_error_if_given_argument_is_not_a_pem_encoded_certificate() {
    let args = vec!["real-cert".to_owned()];
    let validator = FakeValidator { is_file: true };
    let result = execute(validator, args);
    assert!(result.is_err())
}

The update to the execute function will need a bit of refactoring, but first, let's implement it in the simplest way possible.

fn execute(validator: impl PathValidator, args: Vec<String>) -> Result<(), String> {
    if args.len() != 1 {
        let error = format!(
            "{}{}",
            "Error: did not receive a single argument, ",
            "please invoke cert-decoder as follows: ./cert-decoder /path/to/cert."
        );
        return Err(error);
    }
    let path = &args[0];
    if !validator.is_file(path) {
        return Err(
            "Error: path given is not a regular file, please update to point to a certificate."
                .to_owned(),
        );
    }
    // read file to string
    let cert = std::fs::read_to_string(path).unwrap();
    // pem to der
    let _ = pem_to_der(cert.as_bytes()).unwrap();
    Ok(())
}

We use std::fs::read_to_string to read the file path we pass as an argument directly to a string. This call returns a Result as it can fail if the path does not exist. But we know it does exist at this point, so we just unwrap the value, giving us our cert as a string. Then we pass that string as bytes, by calling the as_bytes function on it, to pem_to_der. This can fail and because here we just call unwrap this will panic if pem_to_der returns an Err value instead of and Ok value.

To see what I mean, update the test so it reads Cargo.toml.

#[test]
fn should_error_if_given_argument_is_not_a_pem_encoded_certificate() {
    let args = vec!["Cargo.toml".to_owned()];
    let validator = FakeValidator { is_file: true };
    let result = execute(validator, args);
    assert!(result.is_err())
}

It will fail as follows because Cargo.toml is not PEM encoded:

➜ cargo -q test pem

running 1 test
F
failures:

---- test::should_error_if_given_argument_is_not_a_pem_encoded_certificate stdout ----
thread 'test::should_error_if_given_argument_is_not_a_pem_encoded_certificate' panicked at 'called `Result::unwrap()` on an `Err` value: Error(MissingHeader)', src/main.rs:33:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    test::should_error_if_given_argument_is_not_a_pem_encoded_certificate

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 3 filtered out

error: test failed, to rerun pass '--bin cert-decode'

Even though it did error, it didn't do so in a way we could handle in our test. It would be better to not call unwrap on the return of pem_to_der. To do so, we need to change the return type of execute so it allows us to return both our existing String errors and the PEMError which pem_to_der returns.

Error handling

There are many different ways to handle errors in Rust. A lot is going on in this space currently in regards libaries and discussions in the language itself. How you handle errors in a library vs in an application can vary wildly too. I don't want to add any more dependencies here and I also want to keep this simple, so we'll use the most general type for handling errors, Box<dyn std::error::Error>².

Box is a simple way of allocating something on the heap in Rust. Box<dyn std::error::Error> is a trait object³, this allows us to return a value of any type that implements the std::error::Error trait.

Let's refactor. First, update execute as follows.

-fn execute(validator: impl PathValidator, args: Vec<String>) -> Result<(), String> {
+fn execute(
+    validator: impl PathValidator,
+    args: Vec<String>,
+) -> Result<(), Box<dyn std::error::Error>> {
     if args.len() != 1 {
         let error = format!(
             "{}{}",
             "Error: did not receive a single argument, ",
             "please invoke cert-decoder as follows: ./cert-decoder /path/to/cert."
         );
-        return Err(error);
+        return Err(error.into());
     }
     let path = &args[0];
     if !validator.is_file(path) {
         return Err(
             "Error: path given is not a regular file, please update to point to a certificate."
-                .to_owned(),
+                .into(),
         );
     }
     let cert = std::fs::read_to_string(path).unwrap();
-    let _ = pem_to_der(cert.as_bytes()).unwrap();
+    let _ = pem_to_der(cert.as_bytes())?;
     Ok(())
 }

We change the return type to of execute to Result<(), Box<dyn std::error::Error>>. We were previously returning String's from our custom errors, we can call into on our strings and this will convert them into Box<dyn std::error::Error>. There is an instance of From for converting a String to a Box<dyn std::error::Error>, because of this we get an Into instance for automatically.

Finally, we add a ?⁴ to immediately return the error if pem_to_der returns an error. Next update main.

-fn main() -> Result<(), String> {
+fn main() -> Result<(), Box<dyn std::error::Error>> {
     let args = std::env::args().skip(1).collect();
     let validator = CertValidator;
     execute(validator, args)
 }

We just change the return type here. Finally, update the tests.

 #[cfg(test)]
 mod test {
     ... 
     #[test]
     fn should_error_if_not_given_a_single_argument() {
         ...
         assert!(result.is_err());
         assert_eq!(
-            result.err().unwrap(),
+            format!("{}", result.err().unwrap()),
             format!(
                 "{}{}",
                 "Error: did not receive a single argument, ",
                 "please invoke cert-decoder as follows: ./cert-decoder /path/to/cert."
             )
         );
     }

     #[test]
     fn should_error_if_argument_is_not_a_regular_file() {
         ...
         assert!(result.is_err());
         assert_eq!(
-            result.err().unwrap(),
+            format!("{}", result.err().unwrap()),
             "Error: path given is not a regular file, please update to point to a certificate."
         );
     }

     #[test]
     fn should_error_if_given_argument_is_not_a_pem_encoded_certificate() {
        ...
     }

     #[test]
     fn should_succeed() {
-        let args = vec!["a-file".to_owned()];
+        let args = vec!["resources/google.com.crt".to_owned()];
         let validator = FakeValidator { is_file: true };
         let result = execute(validator, args);
         assert!(result.is_ok());
     }

We call format on the error message to turn the Box<dyn std::error::Error> in to a string. We also change the should_succeed test to read the real cert. This does IO, but that's ok for now. Re-run and the tests should be green.

✦ ➜ cargo -q test

running 4 tests
....
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

🔗 wayofthepie/cert-decoder@37cbc87

Refactor

Now we have refactored to allow returning different types of errors, read, and decoded the PEM certificate into DER format. Let's clean things up a little. We are doing IO again, so let's tackle that first. Right now we are passing an implementation of PathValidator to execute. It would make sense to expand what this trait does, but we should rename it. Let's call it FileProcessor. Implementations will have is_file and read_to_string so this makes sense. Let's also rename CertValidator to CertProcessor.

 use std::path::Path;
 use x509_parser::pem::pem_to_der;

-trait PathValidator {
+trait FileProcessor {
     fn is_file(&self, path: &str) -> bool;
 }

-struct CertValidator;
+struct CertProcessor;

-impl PathValidator for CertValidator {
+impl FileProcessor for CertProcessor {
     fn is_file(&self, path: &str) -> bool {
         Path::new(path).is_file()
     }
 }

 fn execute(
-    validator: impl PathValidator,
+    validator: impl FileProcessor,
     args: Vec<String>,
 ) -> Result<(), Box<dyn std::error::Error>> {
    ...
    Ok(())
 }

 fn main() -> Result<(), Box<dyn std::error::Error>> {
     let args = std::env::args().skip(1).collect();
-    let validator = CertValidator;
+    let validator = CertProcessor;
     execute(validator, args)
 }

 #[cfg(test)]
 mod test {

-    use crate::{execute, PathValidator};
+    use crate::{execute, FileProcessor};

-    struct FakeValidator {
+    struct FakeProcessor {
         is_file: bool,
     }

-    impl PathValidator for FakeValidator {
+    impl FileProcessor for FakeProcessor {
         fn is_file(&self, _: &str) -> bool {
             self.is_file
         }
     }

     #[test]
     fn should_error_if_not_given_a_single_argument() {
         // arrange
         let args = Vec::new();
-        let validator = FakeValidator { is_file: false };
+        let validator = FakeProcessor { is_file: false };

         ...
     }

     #[test]
     fn should_error_if_argument_is_not_a_regular_file() {
         // arrange
         let args = vec!["not-a-regular-file".to_owned()];
-        let validator = FakeValidator { is_file: false };
+        let validator = FakeProcessor { is_file: false };

         ...
     }

     #[test]
     fn should_error_if_given_argument_is_not_a_pem_encoded_certificate() {
         let args = vec!["Cargo.toml".to_owned()];
-        let validator = FakeValidator { is_file: true };
+        let validator = FakeProcessor { is_file: true };
         let result = execute(validator, args);
         assert!(result.is_err())
     }

     #[test]
     fn should_succeed() {
         let args = vec!["resources/google.com.crt".to_owned()];
-        let validator = FakeValidator { is_file: true };
+        let validator = FakeProcessor { is_file: true };
         let result = execute(validator, args);
         assert!(result.is_ok());
     }
 }

📝 Note
You may have noticed I forgot to update the name of the variables from validator to something more appropriate like processor!
This was indeed a mistake. I added a small refactor section near the end of the post which fixes this.

🔗 wayofthepie/cert-decoder@a30ae7c

Now we can add a read_to_string method to the FileProcessor trait and implement.

 use std::path::Path;
 use x509_parser::pem::pem_to_der;

 trait FileProcessor {
     fn is_file(&self, path: &str) -> bool;
+    fn read_to_string(&self, path: &str) -> Result<String, Box<dyn std::error::Error>>;
 }

 struct CertProcessor;

 impl FileProcessor for CertProcessor {
     fn is_file(&self, path: &str) -> bool {
         Path::new(path).is_file()
     }
+    fn read_to_string(&self, path: &str) -> Result<String, Box<dyn std::error::Error>> {
+        Ok(std::fs::read_to_string(path)?)
+    }
 }

 fn execute(
-    validator: impl FileProcessor,
+    processor: impl FileProcessor,
     args: Vec<String>,
 ) -> Result<(), Box<dyn std::error::Error>> {
     if args.len() != 1 {
         let error = format!(
             "{}{}",
             "Error: did not receive a single argument, ",
             "please invoke cert-decoder as follows: ./cert-decoder /path/to/cert."
         );
         return Err(error.into());
     }
     let path = &args[0];
-    if !validator.is_file(path) {
+    if !processor.is_file(path) {
         return Err(
             "Error: path given is not a regular file, please update to point to a certificate."
                 .into(),
         );
     }
-    let cert = std::fs::read_to_string(path).unwrap();
+    let cert = processor.read_to_string(path)?;
     let _ = pem_to_der(cert.as_bytes())?;
     Ok(())
 }

 fn main() -> Result<(), Box<dyn std::error::Error>> {
     let args = std::env::args().skip(1).collect();
     let validator = CertProcessor;
     execute(validator, args)
 }

 #[cfg(test)]
 mod test {

     use crate::{execute, FileProcessor};

     struct FakeProcessor {
         is_file: bool,
+        file_str: String,
     }

     impl FileProcessor for FakeProcessor {
         fn is_file(&self, _: &str) -> bool {
             self.is_file
         }
+        fn read_to_string(&self, _: &str) -> Result<String, Box<dyn std::error::Error>> {
+            Ok(self.file_str.clone())
+        }
     }

     #[test]
     fn should_error_if_not_given_a_single_argument() {
         // arrange
         let args = Vec::new();
-        let validator = FakeProcessor { is_file: false };
+        let validator = FakeProcessor {
+            is_file: false,
+            file_str: "".to_owned(),
+        };

         // act
         let result = execute(validator, args);

         // assert
         assert!(result.is_err());
         assert_eq!(
             format!("{}", result.err().unwrap()),
             format!(
                 "{}{}",
                 "Error: did not receive a single argument, ",
                 "please invoke cert-decoder as follows: ./cert-decoder /path/to/cert."
             )
         );
     }

     #[test]
     fn should_error_if_argument_is_not_a_regular_file() {
         // arrange
         let args = vec!["not-a-regular-file".to_owned()];
-        let validator = FakeProcessor { is_file: false };
+        let validator = FakeProcessor {
+            is_file: false,
+            file_str: "".to_owned(),
+        };

         // act
         let result = execute(validator, args);

         // assert
         assert!(result.is_err());
         assert_eq!(
             format!("{}", result.err().unwrap()),
             "Error: path given is not a regular file, please update to point to a certificate."
         );
     }

     #[test]
     fn should_error_if_given_argument_is_not_a_pem_encoded_certificate() {
         let args = vec!["Cargo.toml".to_owned()];
-        let validator = FakeProcessor { is_file: true };
+        let validator = FakeProcessor {
+            is_file: true,
+            file_str: "".to_owned(),
+        };
         let result = execute(validator, args);
         assert!(result.is_err())
     }

     #[test]
     fn should_succeed() {
-        let args = vec!["resources/google.com.crt".to_owned()];
-        let validator = FakeProcessor { is_file: true };
+        let cert = include_str!("../resources/google.com.crt");
+        let args = vec!["doesnt-really-matter".to_owned()];
+        let validator = FakeProcessor {
+            is_file: true,
+            file_str: cert.to_owned(),
+        };
         let result = execute(validator, args);
         assert!(result.is_ok());
     }

📝 Note

In the should_succeed test we use the include_str macro to read the real cert at compile time. This is cleaner than pasting the cert directly in the test.

🔗 wayofthepie/cert-decoder@436d85f

We can improve the tests by deriving⁵ Default for our FakeProcessor. This will give us a basic implementation of FakeProcessor, defaulting all the fields to the value of the Default implementation for their type. For example, the default for bool is false and for String is the empty string.

 #[cfg(test)]
 mod test {

     use crate::{execute, FileProcessor};

     ...
-
+    #[derive(Default)]
     struct FakeProcessor {
         is_file: bool,
         file_str: String,
     }

     #[test]
     fn should_error_if_not_given_a_single_argument() {
-        // arrange
         let args = Vec::new();
-        let validator = FakeProcessor {
-            is_file: false,
-            file_str: "".to_owned(),
-        };
-
-        // act
+        let validator = FakeProcessor::default();
         let result = execute(validator, args);
-
-        // assert
         assert!(result.is_err());
         assert_eq!(
             format!("{}", result.err().unwrap()),
             format!(
                 "{}{}",
                 "Error: did not receive a single argument, ",
                 "please invoke cert-decoder as follows: ./cert-decoder /path/to/cert."
             )
         );
     }

     #[test]
     fn should_error_if_argument_is_not_a_regular_file() {
-        // arrange
         let args = vec!["not-a-regular-file".to_owned()];
-        let validator = FakeProcessor {
-            is_file: false,
-            file_str: "".to_owned(),
-        };
-
-        // act
+        let validator = FakeProcessor::default();
         let result = execute(validator, args);
-
-        // assert
         assert!(result.is_err());
         assert_eq!(
             format!("{}", result.err().unwrap()),
             "Error: path given is not a regular file, please update to point to a certificate."
         );
     }

     #[test]
     fn should_error_if_given_argument_is_not_a_pem_encoded_certificate() {
         let args = vec!["Cargo.toml".to_owned()];
         let validator = FakeProcessor {
             is_file: true,
-            file_str: "".to_owned(),
+            ..FakeProcessor::default()
         };
         let result = execute(validator, args);
         assert!(result.is_err())
     }
     ... 
 }

📝 Note
In a test above we used struct update syntax, ..FakeProcessor::default(). This will "fill in" any fields we do not explicitly set. It will allow us to add more fields to FileProcessor if needed and not have to update all tests.

After each change, you should run the tests! If you run them now they should still be green.

✦ ➜ cargo -q test

running 4 tests
....
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

🔗 wayofthepie/cert-decoder@e984bd7f

Parse the der encoded cert

Let's parse the DER bytes into an X509Certificate. In the x509 parser API section we saw an example of this using the parse_x509_der function. It can fail, so first, a test.

#[test]
fn should_error_if_argument_is_not_a_valid_certificate() {
    let cert = include_str!("../resources/bad.crt");
    let args = vec!["doesnt-really-matter".to_owned()];
    let processor = FakeProcessor {
        is_file: true,
        file_str: cert.to_owned(),
    };
    let result = execute(processor, args);
    assert!(result.is_err());
}

I have added a file called bad.crt to the resources folder. This just contains a base64 encoded string, which is not a valid certificate. So it will succeed in the pem_to_der call but calling parse_x509_der should return an error. First, let's see this test fail.

✦ ➜ cargo -q test

running 5 tests
...F.
failures:

---- test::should_error_if_argument_is_not_a_valid_certificate stdout ----
thread 'test::should_error_if_argument_is_not_a_valid_certificate' panicked at 'assertion failed: result.is_err()', src/main.rs:118:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    test::should_error_if_argument_is_not_a_valid_certificate

test result: FAILED. 4 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--bin cert-decode'

Great! Now, let's parse the DER we get.

fn execute(
    processor: impl FileProcessor,
    args: Vec<String>,
) -> Result<(), Box<dyn std::error::Error>> {
    // stripped out irrelevant code 
    ...

    let cert = processor.read_to_string(path)?;
    let (_, pem) = pem_to_der(cert.as_bytes())?;
    let _ = parse_x509_der(&pem.contents)?;
    Ok(())
}

And re-run the tests.

✦ ➜ cargo -q test

running 5 tests
.....
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Great! We didn't need to update the should_succeed test, meaning it is reading our real certificate correctly. There are few things we can improve here, but first, let's mark off the first item in our test list.

Read and print certificate

✔️ Validate file is a certificate
☐ Print the certificate

🔗 wayofthepie/cert-decoder@11ff773

Print the certificate

It turns out we will have to do a bit more processing to get a human-readable output format, so I'm going to cheat here! On success, parse_x509_der returns a tuple with remaining bytes and an X509Certificate. The X509Certificate type implements Debug so we can print its debug format.


 fn execute(
     processor: impl FileProcessor,
     args: Vec<String>,
 ) -> Result<(), Box<dyn std::error::Error>> {
     ... 
     let cert = processor.read_to_string(path)?;
     let (_, pem) = pem_to_der(cert.as_bytes())?;
-    let _ = parse_x509_der(&pem.contents)?;
+    let (_, cert) = parse_x509_der(&pem.contents)?;
+    let output = format!("{:#?}", cert.tbs_certificate);
+    println!("{}", output);
     Ok(())
 }

 ...

We use the debug format specifier {:?} in the format macro. We also add a # to pretty print it, {:#?}. The only thing we print here is the tbs_certificate field as that contains all the details we will need. To test this, let's run the actual cli.

📝 Note
From the above change you might be thinking "Why create a pre-formatted string and pass that to println!? Couldn't you just use the debug format specifier directly in println!?". You can do this, try it and do cargo -q run -- resources/google.com.crt. Now do it again with a pipe - cargo -q run -- resources/google.com.crt | head -n20 - it will fail. I may do a short post on why this happens.

✦ ➜ cargo -q run -- resources/google.com.crt | head -n20
TbsCertificate {
    version: 2,
    serial: BigUint {
        data: [
            4412903,
            134217728,
            247394332,
            1459281101,
        ],
    },
    signature: AlgorithmIdentifier {
        algorithm: OID(1.2.840.113549.1.1.11),
        parameters: BerObject {
            class: 0,
            structured: 0,
            tag: EndOfContent,
            content: ContextSpecific(
                EndOfContent,
                Some(
                    BerObject {

Pretty unreadable! But it's a start, we're making some headway. In the next post, we'll clean this up.

Read and print certificate

✔️ Validate file is a certificate
✔️ Print the certificate

🔗 wayofthepie/cert-decoder@11058e3

Tiny refactor

I realized I misnamed a few things. I missed them when refactoring. In the tests, most of the FakeProcessor variables are still called validator. Similarly in main. Let's update those.

  ...

 fn main() -> Result<(), Box<dyn std::error::Error>> {
     let args = std::env::args().skip(1).collect();
-    let validator = CertProcessor;
-    execute(validator, args)
+    let processor = CertProcessor;
+    execute(processor, args)
 }

 #[cfg(test)]
 mod test {

     ...


     #[test]
     fn should_error_if_not_given_a_single_argument() {
         let args = Vec::new();
-        let validator = FakeProcessor::default();
-        let result = execute(validator, args);
+        let processor = FakeProcessor::default();
+        let result = execute(processor, args);
         assert!(result.is_err());
         assert_eq!(
             format!("{}", result.err().unwrap()),
             format!(
                 "{}{}",
                 "Error: did not receive a single argument, ",
                 "please invoke cert-decoder as follows: ./cert-decoder /path/to/cert."
             )
         );
     }

     #[test]
     fn should_error_if_argument_is_not_a_regular_file() {
         let args = vec!["not-a-regular-file".to_owned()];
-        let validator = FakeProcessor::default();
-        let result = execute(validator, args);
+        let processor = FakeProcessor::default();
+        let result = execute(processor, args);
         assert!(result.is_err());
         assert_eq!(
             format!("{}", result.err().unwrap()),
             "Error: path given is not a regular file, please update to point to a certificate."
         );
     }

     #[test]
     fn should_error_if_given_argument_is_not_a_pem_encoded_certificate() {
         let args = vec!["Cargo.toml".to_owned()];
-        let validator = FakeProcessor {
+        let processor = FakeProcessor {
             is_file: true,
             ..FakeProcessor::default()
         };
-        let result = execute(validator, args);
+        let result = execute(processor, args);
         assert!(result.is_err())
     }

     ...

     #[test]
     fn should_succeed() {
         let cert = include_str!("../resources/google.com.crt");
         let args = vec!["doesnt-really-matter".to_owned()];
-        let validator = FakeProcessor {
+        let processor = FakeProcessor {
             is_file: true,
             file_str: cert.to_owned(),
         };
-        let result = execute(validator, args);
+        let result = execute(processor, args);
         println!("{:#?}", result);
         assert!(result.is_ok());
     }
 }

🔗 wayofthepie/cert-decoder@371ea84

Conclusion

There were a few things I glossed over that appeared in this post. I will take note of them and make sure they appear in one of the next posts. For example lifetimes, a better explanation of Box, and that println issue I mentioned in a note in the last section.

There is also a small display issue when an error occurs. For example, if we pass no argument:

✦ ➜ cargo -q run --
Error: "Error: did not receive a single argument, please invoke cert-decoder as follows: ./cert-decoder /path/to/cert."

It repeats the word "Error" and also wraps our error string in quotes. We will fix this too in the next post!

See Generic Types, Traits, and Lifetimes in the Rust book. ↩
Error handling in Rust is a big topic, to get started see the Error Handling chapter in the Rust book. For more on boxed errors see Boxing errors.d ↩
See Using Trait Objects That Allow for Values of Different Types in the Rust book. ↩
For more on how the ? works see The ? operator for easier error handling. ↩
For more on deriving see Derive ↩

Three bytes to an integer

wayofthepie — Sun, 21 Jun 2020 14:30:31 +0000

This post is a quick overview of how I turned an initially poor solution to a problem into a better one, and the thought process behind it.

The problem

I'm working on pulling information out of the Certificate Transparency Logs. The logs are made up of a JSON list of log entries. A single entry is a JSON object with two fields, leaf_input and extra_data. The leaf_input field contains base64 encoded binary data. I want to pull information out of this binary data.

To outline the structure of the binary data I'll use the following notation, indicating byte numbers:

[n] - indicates byte n, so [0] is the first byte.
[n..=m] - indicates a range of bytes, inclusive, so [2..=4] means bytes 2, 3 and 4.
[n..] - indicates the remaining bytes in the binary data after and including index n.

[0] [1] [2..=9] [10..=11] [12..=14] [15..]
 |   |     |        |         |      |
 |   |     |        |         |      |- (rest) the rest of the bytes
 |   |     |        |         |      
 |   |     |        |         |- (length) the length of the certificate      
 |   |     |        |               
 |   |     |        | - the log entry type, not relevant in this post              
 |   |     |                       
 |   |     | - timestamp, not relevant in this post                       
 |   |                            
 |   | - signature type, not relevant in this post                       
 |                               
 | - version, not relevant in this post

The relevant parts here are the length (bytes [12..=14]) and the rest (indicated by [15..] above). rest contains an X.509 certificate we want, and the three bytes in the length define an integer which tells us how many bytes that certificate makes up in rest. Once we read length bytes from rest, the remainder of rest is data that we do not need.

A solution

So we need to turn the three bytes which make up length into an integer. An integer whose size is 3 bytes would be a 24-bit int. Rust does not support 24-bit ints. The closest is a u32, a 32-bit unsigned int. You can't just turn 3 bytes into a u32 however. The first solution I came up with was something like the following code.

// "bytes" is the full binary data inside the "leaf_input" field
fn get_cert_length(bytes: &[u8]) -> u32 {
    let length_bytes_slice = &bytes[12..=14];
    let mut length_bytes = b.iter().cloned().collect::<Vec<u8>>();
    length_bytes.reverse();
    length_bytes.push(0);
    return u32::from_le_bytes(length_bytes.as_slice().try_into().unwrap());
}

This is not great! We allocate a new Vec and we have to pad it with a zero to make it 4 bytes in length. Because we cannot push a value onto the start of a Vec we first reverse it. Then we convert it into an array of type [u8; 4] - this is what length_bytes.as_slice().try_into().unwrap() does. Finally, we read the bytes in little endian format as a u32. Way too much happening here for what on the surface seems like a simple conversion!

I thought about it some more and came up with a better solution.

A better solution

First, here is an outline of my thought process. Let's say bytes 12 to 14 look as follows, in hex notation:

[00, 05, 4c]

Then the value for the length is 00054c in hex, which is 1356 in decimal. 00054c is 000000 + 000500 + 00004c. If we take the bytes 00, 05 and 4c individually, how do we get 00054c?

Well, each time we move a number in the hex string one space to the left, we multiply the number by a power of 16. For example, moving 000005, which is 5 in decimal, to 000050, which is 80 in decimal, we had to multiply 5 by 16. If we move it two places, we would need to multiply it by 256, which is 16 squared, so 000005 to 000500 (1280 in decimal) is 5 * 256. Concretely, to convert 000500 hex to decimal:

\begin{aligned} 000500 &= (0 * 16 ^5) + (0 * 16 ^4) + (0 * 16 ^3) + (5 * 16 ^2) + (0 * 16 ^1) + (0 * 16 ^0) \\ &= 5 * 16^2 \\ &= 1280 \end{aligned}

With this knowledge we can update get_cert_length as follows:

fn get_cert_length(bytes: &[u8]) -> u32 {
    (bytes[12] as u32 * 65536) + (bytes[13] as u32 * 256) + bytes[14] as u32
}

Much cleaner than the original version! The values we multiply each byte with are always a power of 2 so we can also use bit-shifts instead of the multiplications:

fn get_cert_length(bytes: &[u8]) -> u32 {
    ((bytes[12] as u32) << 16) + ((bytes[13] as u32) << 8) + bytes[14] as u32
}

I ended up using the bit-shift version in the end.

Morning silence and broken drivers

wayofthepie — Sun, 14 Jun 2020 11:36:25 +0000

Silence

I woke up this morning and booted my desktop. No sound! Some digging around and I noticed dmesg spitting out an error. I don't have that message saved but here are the details from the kernel logs.

✦ ➜ journalctl --system --since yesterday | grep "Focusrite Scarlett" -C10
...
Jun 14 06:45:59 sky kernel: usb 1-1.2.3: USB disconnect, device number 8
Jun 14 06:46:00 sky kernel: usb 1-1.2.3: new high-speed USB device number 9 using ehci-pci
Jun 14 06:46:00 sky kernel: usb 1-1.2.3: New USB device found, idVendor=1235, idProduct=8203, bcdDevice= 6.2f
Jun 14 06:46:00 sky kernel: usb 1-1.2.3: New USB device strings: Mfr=1, Product=3, SerialNumber=2
Jun 14 06:46:00 sky kernel: usb 1-1.2.3: Product: Scarlett 6i6 USB
Jun 14 06:46:00 sky kernel: usb 1-1.2.3: Manufacturer: Focusrite
Jun 14 06:46:00 sky kernel: usb 1-1.2.3: SerialNumber: 00009810
Jun 14 06:46:00 sky kernel: usb 1-1.2.3: Focusrite Scarlett Gen 2 Mixer Driver disabled; use options snd_usb_audio device_setup=1 to enable and report any issues to g@b4.v
...

It looks like a driver update in the latest kernel broke my USB audio device, a pretty old Focusrite Scarlett. A quick google of the error brings you to these release notes. This is a fork of the kernel where the Focusrite Scarlett driver development happens. They mention adding one of the following lines to /etc/modprobe.d/scarlett.conf.

options snd_usb_audio device_setup=1,1,1,1
options snd_usb_audio vid=0x1235 pid=0x8212 device_setup=1

Great! I'm not sure what vid and pid are. However, if you look back up at the kernel logs there is a line that looks like this:

New USB device found, idVendor=1235, idProduct=8203, bcdDevice= 6.2f

vid is probably the Vendor ID and pid product ID. I had a quick look at the commit for that release and that clarified it. There are a number of constants defined in sound/usb/mixer_quirks.c.

    case USB_ID(0x1235, 0x8203): /* Focusrite Scarlett 6i6 2nd Gen */
    case USB_ID(0x1235, 0x8204): /* Focusrite Scarlett 18i8 2nd Gen */
    case USB_ID(0x1235, 0x8201): /* Focusrite Scarlett 18i20 2nd Gen */
    case USB_ID(0x1235, 0x8212): /* Focusrite Scarlett 4i4 3rd Gen */
    case USB_ID(0x1235, 0x8213): /* Focusrite Scarlett 8i6 3rd Gen */

The kernel logs have idProduct=8203, this matches the constant with the comment for the Focusrite Scarlett 6i6 2nd Gen, which is the device I have. Great, so now we know what vid and pid should be. Add this to /etc/modprobe.d/scarlett.conf.

options snd_usb_audio vid=0x1235 pid=0x8203 device_setup=1

Run sudo systemctl restart systemd-modules-load.service to reload the kernel modules. Check they are loaded:

✦ ➜ modprobe -c | grep snd_usb_audio | grep options
options snd_usb_audio vid=0x1235 pid=0x8203 device_setup=1

Still no sound...

Still silence

Now what? Some more googling around brought me to this forum thread. There is some discussion about the issue there, luckily the author of the driver is also posting! In one of the comments they mention:

Please upgrade to 5.4.1 first; there are known issues with the 6i6 support in 5.4.0 and it won't work very well even after you enable it.

What version of the kernel am I on?

➜ uname -r
5.4.0-37-generic

Ah! A likely suspect. Let's compile a custom kernel, say the latest 5.4.x kernel. It's been a while since I've done this.

➜ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.4.46.tar.xz

➜ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.4.46.tar.sign

➜ unxz linux-5.4.46.tar.xz

# I haven't stored the public keys for linus/greg on this machine yet
➜ gpg --locate-keys torvalds@kernel.org gregkh@kernel.org
gpg: WARNING: unacceptable HTTP redirect from server was cleaned up
gpg: key 38DBBDC86092693E: public key "Greg Kroah-Hartman <gregkh@kernel.org>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: WARNING: unacceptable HTTP redirect from server was cleaned up
gpg: key 79BE3E4300411886: public key "Linus Torvalds <torvalds@kernel.org>" imported
gpg: Total number processed: 1
gpg:               imported: 1
pub   rsa4096 2011-09-23 [SC]
      647F28654894E3BD457199BE38DBBDC86092693E
uid           [ unknown] Greg Kroah-Hartman <gregkh@kernel.org>
sub   rsa4096 2011-09-23 [E]

pub   rsa2048 2011-09-20 [SC]
      ABAF11C65A2970B130ABE3C479BE3E4300411886
uid           [ unknown] Linus Torvalds <torvalds@kernel.org>
sub   rsa2048 2011-09-20 [E]

# Verify the signature, can ignore the warning I just haven't trusted
➜ gpg --verify linux-5.4.46.tar.sign
gpg: assuming signed data in 'linux-5.4.46.tar'
gpg: Signature made Wed 10 Jun 2020 19:25:28 IST
gpg:                using RSA key 647F28654894E3BD457199BE38DBBDC86092693E
gpg: Good signature from "Greg Kroah-Hartman <gregkh@kernel.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 647F 2865 4894 E3BD 4571  99BE 38DB BDC8 6092 693E

# I haven't trusted the key so I got a warning, to do so
➜ gpg --tofu-policy good 647F28654894E3BD457199BE38DBBDC86092693E
gpg: Setting TOFU trust policy for new binding <key: 647F28654894E3BD457199BE38DBBDC86092693E, user id: Greg Kroah-Hartman <gregkh@kernel.org>> to good.

➜ gpg --trust-model tofu --verify linux-5.4.46.tar.sign
gpg: assuming signed data in 'linux-5.4.46.tar'
gpg: Signature made Wed 10 Jun 2020 19:25:28 IST
gpg:                using RSA key 647F28654894E3BD457199BE38DBBDC86092693E
gpg: Good signature from "Greg Kroah-Hartman <gregkh@kernel.org>" [full]
gpg: gregkh@kernel.org: Verified 1 signatures in the past 0 seconds.  Encrypted
     0 messages.

➜ tar xf linux-5.4.46.tar
➜ cd linux-5.4.46/

# Copy the current kernels config
➜ cp /boot/config-$(uname -r) .config

# Once menuconfig loads just save and exit, no need to change anything
➜ make menuconfig

# This can take a while...
➜ time make -j12
  HOSTCC  scripts/basic/fixdep
  DESCEND  objtool
...

# Install the modules
➜ sudo make modules_install
...

# Install the kernel
➜ sudo make install
...

make install should setup the boot config needed in grub, I just need to reboot now. Reboot... and nothing!

Silence and broken graphics

This time dmesg is giving a different error.

[  159.097340] usb 1-1.1: Product: Scarlett 6i6 USB
[  159.097343] usb 1-1.1: Manufacturer: Focusrite
[  159.097344] usb 1-1.1: SerialNumber: 00009810
[  164.318711] usb 1-1.1: Scarlett Gen 2 USB response result cmd 0 was -110

On top of that, I broke the nvidia drivers 😄 these need to be compiled for the kernel you are on. This is my lovely 1440p display:

Purge them and flip back to noveau for now:

➜ sudo apt-get remove --purge nvidia-*
➜ reboot
➜ lshw -c video
  *-display
       description: VGA compatible controller
       product: GP102 [GeForce GTX 1080 Ti]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:03:00.0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: vga_controller bus_master cap_list rom
       configuration: driver=nouveau latency=0 # running noveau drivers now
       resources: irq:73 memory:fa000000-faffffff memory:c0000000-cfffffff memory:d0000000-d1ffffff ioport:e000(size=128) memory:c0000-dffff

Much better:

Back to the dmesg error - Scarlett Gen 2 USB response result cmd 0 was -110. Looking at alsamixer this shows it is not recognizing the device properly.

I wonder if it is the USB port I'm using... It is! Using a different port:

I get the expected "Front left", "Front center" sound output when running speaker-test -t wav -c 6. This is weird, as other devices work fine on that port and it was previously working there too. Something else to investigate another time I guess.

Conclusion

In the end there were a number of issues. This is the first time a driver issue has caused me trouble on linux in years. But my hardware is getting old, it's probably time to upgrade a few things. Also:

Don't try to debug and fix an issue when you have just woken up and have had no caffeine injected into your system yet.
All computers are broken.

Rust and GitHub Actions

wayofthepie — Sun, 07 Jun 2020 17:18:42 +0000

📝 Note

Setting up GitHub Actions is entirely optional for this series. I do not link back to anything in this post in any future post, so you don't need to set this up if you don't want to.

Introduction

Before we continue implementing our CLI, let's take time to set up some GitHub Actions to build and test our commits. We'll use the actions defined in the actions-rs GitHub Organization.

Build and test
Clippy lint
- Block PR's on failing checks
  - Clippy deny lints
  - Protected branch rule
  - Clean up and merge
Security audit
Conclusion

Build and test

The workflow I normally follow when coding is to create a short-lived branch on which all commits are built and tested then create a Pull Request (PR) into the master branch. Let's set this up. Check out a new branch, let's call it actions. Then, create a file called build.yml under .github/workflows/.

➜ mkdir -p .github/workflows
➜ touch .github/workflows/build.yml

We'll create a simple action that builds and tests our CLI. Add the following to build.yml.

on: [push]
name: build
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: stable
      - name: build
        uses: actions-rs/cargo@v1
        with:
          command: build
      - name: test
        uses: actions-rs/cargo@v1
        with:
          command: test

on defines the event(s)¹ which should trigger this workflow. We set it to run on a push to any branch. The workflow has a single job, called build. This job runs on the ubuntu-latest Virtual Environment. It checks out our code, installs the stable Rust toolchain and runs a cargo build then runs a cargo test.

Commit this change and push to your repo. GitHub Actions will then run this workflow. Once complete, when you click the Actions tab on your repo, you should see something like this:

I had two builds, as I rebased a commit, you will likely only see one. Click into the build, then click the build job on the left with the checkmark and it will show you the steps in the workflow.

This is a good start. All commits will be built and tested. We can add a check to PR's so that they will block merges if this workflow fails. But first, let's add some improvements. The code up to this point can be seen here.

Clippy lint

One thing we did not do in the last post was to run a linter. Rust has an awesome linter called Clippy which can highlight quite a lot of issues. Let's run it in our project now.

➜ cargo clippy
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s

No issues, great! Like any test, let's see it fail. Or in this case, make Clippy give out to us. Let's add something Clippy does not like to main.rs.

fn main() -> Result<(), String> {
    assert!(false); // clippy won't like this 
    let args = std::env::args().skip(1).collect();
    let validator = CertValidator;
    execute(validator, args)
}

Assertions on a constant are optimized out by the compiler², so Clippy will warn about this.

➜ cargo clippy
warning: `assert!(false)` should probably be replaced
  --> src/main.rs:35:5
   |
35 |     assert!(false);
   |     ^^^^^^^^^^^^^^^
   |
   = note: `#[warn(clippy::assertions_on_constants)]` on by default
   = help: use `panic!()` or `unreachable!()`
 = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#assertions_on_constants
   = note: this warning originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

warning: 1 warning emitted

    Finished dev [unoptimized + debuginfo] target(s) in 0.01s

📝 Note

Sometimes on a subsequent run, Clippy will just say it's successful and not run correctly. See this issue for more information. If you think this is happening you can force a full re-run by modifying a file, e.g. touch src/main.rs; cargo clippy. It seems this should be fixed, but might only be fixed in nightly Rust, not stable.

Great! Just what we want. Let's leave this assertion here and update our existing workflow to add a lint step which runs Clippy. The clippy-check action will allow us to easily do this.

on: [push]
name: build
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions-rs/toolchain@v1
        with:
          toolchain: table
      - name: build
        uses: actions-rs/cargo@v1
        with:
          command: build
      - name: test
        uses: actions-rs/cargo@v1
        with:
          command: test
      - name: lint
        uses: actions-rs/clippy-check@v1
        with:
          token: ${{ secrets.GITHUB_TOKEN }}

Commit both the assertion and the lint addition to the workflow and push. Once the workflow run finishes, click into the Clippy step and you should see something like this:

The warning is now attached to a commit and links to a specific file. Click the text "Check warning on line 35 in src/main.rs" in the annotation and it should bring you to exactly where this issue occurs.

This is nice! Create a PR and you will see this annotation exactly as it is above in the Files Changed tab. I find this useful, even if I am the sole developer. Sometimes I forget to run Clippy and I always do a quick review of the code in a PR. Create a PR into the master branch for this and you will also see the checks have all passed. Don't merge it yet!

Let's make one last change. The code up this point can be found here.

Block PR's on failing checks

If we somehow manage to check something in which does not compile, has failing tests, or has Clippy lint issues which it is set to deny, we should block the PR, i.e. not allow it to be merged until these issues are fixed. This may seem like overkill if you are the only dev on a repo, but I do this all the time because I make these mistakes! No matter how careful I try to be.

Before we configure our repo to block PRs, a quick note on Clippy lints.

Clippy deny lints

The lint we have in our PR above is just a warning. Clippy also has several lints that will cause it to fail if detected. These are lints at the Deny level in the list of Clippy lints. For example, the lint approx_constant checks for constants which are defined under std::f32::consts or std::f64::consts. If it finds matching constants in your code it will error out.

To see this in action, update main with an unused variable which approximates pi. Before you do that, make sure you have committed your changes up to here, so we can easily reset. I've commented out the assert!(false) we added earlier so we don't also get a warning for that from Clippy.

fn main() -> Result<(), String> {
    // assert!(false);
    let _ = 3.14;
    let args = std::env::args().skip(1).collect();
    let validator = CertValidator;
    execute(validator, args)
}

Run Clippy and it will give an error, and return a non-zero exit code.

➜ cargo clippy
    Checking cert-decode v0.1.0 (/home/chaospie/repos/blog-cert-decode/cert-decode)
error: approximate value of `f{32, 64}::consts::PI` found. Consider using it directly
  --> src/main.rs:36:13
   |
36 |     let _ = 3.14;
   |             ^^^^
   |
   = note: `#[deny(clippy::approx_constant)]` on by default
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#approx_constant

error: aborting due to previous error

error: could not compile `cert-decode`.

To learn more, run the command again with --verbose.

➜ echo $?
101

echo $? will print the return code of the last command executed in the shell. Don't forget to reset these changes, either manually or by running git checkout src/main.rs.

Protected branch rule

The configuration for blocking PRs is defined with a protected branch³ rule. Go to Settings -> Branches -> Branch protection rules -> Add rule in you repo. Set the Branch name pattern to master and select the status checks to verify in PRs as follows.

📝 Note

I have an extra status check here called build_and_test because I had initially defined the job in our workflow YAML build_and_test. It is now called build.

Now if we raise a PR with any failure in our build job - the job in our workflow YAML - or the Clippy step specifically, the PR will be blocked from merging. This is what it would look like:

And the details of the lint error:

Note that because I am the only dev, I am an admin on this repo so I can still merge. But it is glaringly obvious something is wrong. This has saved me a few times.

Clean up and merge

The PR you raised, before our Clippy deny lints digression, has a warning because of the assert!(false) call in main. Clean that up, commit, push, and merge it. main should look as follows again:

fn main() -> Result<(), String> {
    let args = std::env::args().skip(1).collect();
    let validator = CertValidator;
    execute(validator, args)
}

The only change in your PR should be the build.yml file. Once merged the workflow will run again on master and be successful. There is one more workflow I usually add to Rust repositories. So let's add that also. See here for the code up to this point.

Security audit

GitHub tracks vulnerabilities from several supported language package managers. It will alert you if your repository contains a vulnerable dependency. It doesn't support cargo yet, I'm not sure if they have plans to support it either. However, cargo has the cargo-audit plugin and we can use this in a workflow. actions-rs has an action just for this, audit-check.

Let's create a new branch called audit-workflow and add a new workflow at .github/workflows/security-audit.yml.

name: Security audit
on:
  schedule:
    - cron: "0 8 * * *"
  push:
    paths:
      - "**/Cargo.*"
  pull_request:
    branches:
      - master
jobs:
  security_audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - uses: actions-rs/audit-check@v1
        with:
          token: ${{ secrets.GITHUB_TOKEN }}

This will run the audit check at 8 am UTC every day, whenever we commit and push a change to any branch which updates Cargo.toml or Cargo.lock and whenever we create a PR into master.

Why all these cases? Well, we do it daily at 8 am as a vulnerability can be found in dependencies at any time, so we should check daily that nothing new was found. We do it on changes to Cargo.* files so we can immediately detect if a change in dependencies has introduced and vulnerability. Finally, we do it on a PR into master so we can be sure, at least at the time of the PR, that we are not pushing known vulnerabilities to our master branch.

Commit and push this workflow change to our branch. Let's test that it works. To do so add a dependency to Cargo.toml.

[package]
name = "cert-decode"
version = "0.1.0"
authors = ["Stephen OBrien <wayofthepie@users.noreply.github.com>"]
edition = "2018"

[dependencies]
x509-parser = "0.7.0"

Run a build with cargo build so Cargo.lock gets updated. Because this is just a test, we can edit the last commit and then later remove the dependency and edit again. Don't ever do this on a shared branch! This is how to do it:

$ git add Cargo.* 
$ git commit --amend --no-edit
$ git push origin +HEAD

The +HEAD in the git push command says to force push to the branch HEAD⁴ points to. A force push rewrites history, this is why you should not do it on a shared branch. HEAD is the current branch. As we have changed the Cargo.* files the audit workflow should run. It will hopefully be successful.

To go back to the previous state you can remove the dependency from Cargo.toml:

[package]
name = "cert-decode"
version = "0.1.0"
authors = ["Stephen OBrien <wayofthepie@users.noreply.github.com>"]
edition = "2018"

[dependencies]

Build with cargo build, and edit the last commit again.

$ git add Cargo.* 
$ git commit --amend --no-edit
$ git push origin +HEAD

Before creating another PR for this, you should update the branch protection settings and add security_audit as a status check.

Now you can create a PR into master and this should run again. If it's green, merge it. The code up to the end of this post can be found here.

📝 Note

This audit workflow can take some time to run as it first needs to install cargo-audit. This takes about 5 minutes looking at the logs for a run. We can improve this with caching, but I'll leave that for another post.

Conclusion

We now have:

A build and test workflow for all our commits.
Security auditing on our dependencies.
Checks which block PR's if we have broken builds, lint issues, or vulnerabilities in dependencies

This is something I always set up at the start of a project. We'll go back to implementing the CLI in the next post.

See Events that trigger workflows. ↩
See the assertions_on_constants lint. ↩
See Configuring protected branches. ↩
For more on the HEAD reference see here. ↩

A Rust CLI for decoding certs

wayofthepie — Tue, 02 Jun 2020 17:40:03 +0000

Introduction

In this series of posts, we'll start to build a simple cli in Rust for decoding X.509 certificates. I'll try to keep it as beginner-friendly as possible, by explaining things as best I can when they may be unclear. Some basic Rust knowledge should hopefully be all you need. Feel free to ask any questions in the comments below!

📝 Note

If you are completely new to Rust and don't even have it installed, you can install it using rustup. You should hopefully still be able to follow along.

This first post will mainly focus on a Test Driven Development workflow. I wrote this as I was developing so mistakes or things which I overlooked and ended up factoring out afterward are all kept in.

What are we building?
Some constraints to simplify the cli
Test list
Validate args
- A note on this workflow
- Add an error message
- Check argument is a path that exists
- Factor out IO
- Check argument is a file
- Another small refactor
In the next post

What are we building?

I generally use the openssl cli for pulling information out of certificates. Specifically the x509 command, for example:

# get certificate
$ openssl s_client -connect google.com:443 2>/dev/null < /dev/null \
    | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p' > google.com.crt 

# extract info
$ openssl x509 -text -noout -in google.com.crt | head -n 5
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            1a:86:8b:0d:af:9b:c7:34:08:00:00:00:00:3e:bd:97

All this does is print out a certificate as a human-readable string. I've written about what information certificates contain here if interested. What we want to do in the next few posts is to see if we can build something similar to the openssl x509 -text output from above in a Rust cli.

Some constraints to simplify the cli

Let's think about the API for this cli, how a user will call it. To keep things simple, it should just take a file path to a single certificate. So we should end up with a call like so:

$ cert-decoder /path/to/some/cert.pem

Another constraint to keep this simple at the beginning is the encoding of the certificate. Let's say the certificate must be PEM ¹ (Privacy Enhanced Mail) encoded. This is the encoding you generally see certificates with. It's a string which begins with a line comprised of -----BEGIN CERTIFICATE-----, then some base64 encoded DER (Distinguished Encoding Rules) and ends with a line comprised of -----END CERTIFICATE-----. It looks as follows, with some of the base64 string stripped out and replaced with ...:

-----BEGIN CERTIFICATE-----
MIIJTzCCCDegAwIBAgIQGoaLDa+bxzQIAAAAAD69lzANBgkqhkiG9w0BAQsFADBC
...
DMCTA95gzVKezFCaUidRU9UyHOFzltfYDt7HRlp7MwWoPLM=
-----END CERTIFICATE-----

Test list

We have a basic API and some constraints now. Let's make a test list before we do anything else. This is simply a list of tests we should write to cover some piece of functionality. Let's just cover the most simple thing initially, validating the arguments.

Validate args

☐ Only allow a single argument
☐ Argument is a path that exists
☐ Argument should be a single file

Implementing this we can start to use it and see where we should go next.

Validate args

First, create a new project with cargo new cert-decoder. main.rs should look as follows initially.

fn main() {
    println!("Hello, world!")
}

Let's create a test for the first item on our test list.

fn main() {
    println!("Hello, world!")
}

#[cfg(test)]
mod test {

    #[test]
    fn should_error_if_not_given_a_single_argument() {
        // create fake args list, with no args
        // run function with args
        // check that it returns an error
    }
}

Above I've outlined what we need to do in this test in comments. It's not easy to test this through the main function as main does not take any arguments in its signature. In Rust to read program arguments you use the function std::env::args, more on that later.

Let's create a new function called execute. We know we want this function to return an error so we will make it return a Result.

fn execute(args: Vec<String>) -> Result<(), ()> {
    Ok(())
}

fn main() {
    println!("Hello, world!")
}

#[cfg(test)]
mod test {

    #[test]
    fn should_error_if_not_given_a_single_argument() {
        // create fake args list, with no args
        // run function with args
        // check that it returns an error
    }
}

For now execute just returns an Ok result with a value of (). This is called the unit type. Now let's write our test.

fn execute(args: Vec<String>) -> Result<(), ()> {
    Ok(())
}

fn main() {
    println!("Hello, world!")
}

#[cfg(test)]
mod test {

    use crate::execute;

    #[test]
    fn should_error_if_not_given_a_single_argument() {
        let args = Vec::new();
        let result = execute(args);
        assert!(result.is_err());
    }
}

If we run this with cargo -q test² it will fail. There will also be a warning about the args parameter in execute not being used, but we can ignore that for now.

➜ cargo -q test
...
running 1 test
F
failures:

---- test::should_error_if_not_given_a_single_argument stdout ----
thread 'test::should_error_if_not_given_a_single_argument' panicked at 'assertion failed: result.is_err()', src/ma
in.rs:18:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    test::should_error_if_not_given_a_single_argument

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--bin cert-decode'

Let's make it pass by making a tiny change.

fn execute(args: Vec<String>) -> Result<(), ()> {
    Err(())
}

fn main() {
    println!("Hello, world!")
}

#[cfg(test)]
mod test {

    use crate::execute;

    #[test]
    fn should_error_if_not_given_a_single_argument() {
        let args = Vec::new();
        let result = execute(args);
        assert!(result.is_err());
    }
}

Let's re-run and it should be green.

➜ cargo -q test
...
running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Great! Now we just need to make it return an error only in the case we mention in our first test list item. When we do not receive just a single argument.

fn execute(args: Vec<String>) -> Result<(), ()> {
    if args.len() != 1 {
        return Err(());
    }
    Ok(())
}

Re-run and it should still be green.

➜ cargo -q test

running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Awesome! We should change this to return an error message. But first a quick note about this workflow.

A note on this workflow

This is how I generally write code. What's the smallest useful change I can make to add functionality? In this case, to get started it was argument validation. With that in mind, I decomposed this further into items that look like individually testable parts of the behavior we want in argument validation. I made a list of them so I can tick them off as I go. This helps to keep the focus on a single small piece of functionality. It's especially useful when decomposing complex changes.

To implement, write a test that specifies the behavior of one of the items. Then, make it fail, make it pass with any code, change to implement the real behavior, and finally refactor where appropriate.

Add an error message

Let's improve the argument length check by returning a useful error message. One thing I always try to keep in mind is to make the error message actionable. For example, here we could just say "Error: did not receive a single argument.". However, it would be better to also add an action - "Error: did not receive a single argument. Please invoke cert-decoder as follows: './cert-decoder /path/to/cert'.". Not only are we telling a user what went wrong, but we are also telling them how to fix it.

Let's update our test.

#[test]
fn should_error_if_not_given_a_single_argument() {
    // arrange
    let args = Vec::new();

    // act
    let result = execute(args);

    // assert
    assert!(result.is_err());
    assert_eq!(
        result.err().unwrap(),
        format!(
            "{}{}",
            "Error: did not receive a single argument, ",
            "please invoke cert-decoder as follows: ./cert-decoder /path/to/cert."
        )
    );
}

📝 Note

result.err() will return an Option. The Option will be Some if the value of the Result is Err. This Some will contain the same value Err contained. The Option will be None if the Result is Ok.

In this case we know for certain the Result's value is Err as we asserted it before with is_err. So the Option returned from result.err() will be Some and we can safely call unwrap on that Some to get the value it contains.

This won't compile because we are trying to compare two different types now. result.err().unwrap() will return () and we are trying to compare this to a string. So we need to update the return type of execute. We could also add the correct return value, the string we are asserting execute returns. But I like to do things in tiny changes so let's just update the type first and return an empty string.

fn execute(args: Vec<String>) -> Result<(), String> {
    if args.len() != 1 {
        return Err("".to_owned());
    }
    Ok(())
}

Now if we run cargo -q test it will compile but the test will fail.

➜ cargo -q test

running 1 test
F
failures:

---- test::should_error_if_not_given_a_single_argument stdout ----
thread 'test::should_error_if_not_given_a_single_argument' panicked at 'assertion failed: `(left == right)`
  left: `""`,
 right: `"Error: did not receive a single argument, please invoke cert-decoder as follows: ./cert-decoder /path/to/cert"`', src/main.rs:27:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
...

So let's make some changes.

fn execute(args: Vec<String>) -> Result<(), String> {
    if args.len() != 1 {
        let error = format!(
            "{}{}",
            "Error: did not receive a single argument, ",
            "please invoke cert-decoder as follows: ./cert-decoder /path/to/cert."
        );
        return Err(error);
    }
    Ok(())
}

And re-run.

➜ cargo -q test

running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Great! Our test is green and we have an argument length check with a nice error message. It won't do anything if we try to run it though. To fix that, let's update main.

fn main() -> Result<(), String> {
    let args = std::env::args().skip(1).collect();
    execute(args)
}

Here we add a real call to get the args in main, std::env::args().skip(1).collect(). The name of the binary will be part of the args, that is why we skip one argument.

📝 Note

Let's breakdown the std::env::args().skip(1).collect(). If you understood it you can skip this note.

std::env::args() will return a value of type Args. Args implements the Iterator trait, which means it has skip and collect methods. We skip one item, which is the binary name and collect the rest. Because we are using this as an argument to execute, Rust can infer its type which is Vec<String>.

Now we can run with no argument using cargo run and it will give an error.

➜ cargo -q run
Error: "Error: did not receive a single argument, please invoke cert-decoder as follows: ./cert-decoder /path/to/c
ert."

And if we pass an argument it will print nothing.

➜ cargo -q run -- something

Anything after the -- will be passed to our program as arguments. We can check off the first item on our test list now.

Validate args

✔️ Only allow a single argument
☐ Argument is a path that exists
☐ Argument should be a single file

Check argument is a path that exists

First, a test.

#[test]
fn should_error_if_argument_is_not_a_path_which_exists() {
    // arrange
    let args = vec!["does-not-exist".to_owned()];

    // act
    let result = execute(args);

    // assert
    assert!(result.is_err());
    assert_eq!(
        result.err().unwrap(),
        "Error: path given as argument does not exist, it must be a path to a certificate!"
    );
}

As always, run the test to see it fail first. It will fail because the result will not be an error. Let's make it return an error.

use std::path::Path;

fn execute(args: Vec<String>) -> Result<(), String> {
    if args.len() != 1 {
        let error = format!(
            "{}{}",
            "Error: did not receive a single argument, ",
            "please invoke cert-decoder as follows: ./cert-decoder /path/to/cert."
        );
        return Err(error);
    }
    let path = Path::new(&args[0]);
    if !path.exists() {
        return Err(
            "Error: path given as argument does not exist, it must be a path to a certificate!"
                .to_owned(),
        );
    }
    Ok(())
}

Now the test will pass. Path contains many operations for interacting with filesystems. The call to exists above will check the real filesystem to see if the path we give as an argument exists. This is generally not something you want to do in your tests.

Before we factor out this IO, let's tick this item off on our test list as it is working.

Validate args

✔️ Only allow a single argument
✔️ Argument is a path that exists
☐ Argument should be a single file

The code up to this point can be seen here.

Factor out IO

Almost any code you write is going to do some kind of IO, e.g. network calls, file reads. Normally we don't want this to happen in a test. Beyond just keeping IO out of tests, making it clearer where IO does happen is very useful for readability and refactoring. Especially as a system grows.

Right now what we want to do is make our code testable under different scenarios without touching the real filesystem. There are a few ways to do this, but I generally use traits and have a real and fake implementation of the trait. Let's define a trait that has an exists method, just like Path.

trait PathValidator {
    fn exists(&self, path: &str) -> bool;
}

It took me a while to come up with a name I was somewhat happy with for this trait. Naming is hard. Now that we have a trait, let's implement it for the real case.

struct CertValidator;

impl PathValidator for CertValidator {
    fn exists(&self, path: &str) -> bool {
        Path::new(path).exists()
    }
}

Now our tests need a version of this also.

#[cfg(test)]
mod test {

    use crate::{execute, PathValidator};

    struct FakeValidator {
        is_path: bool,
    }

    impl PathValidator for FakeValidator {
        fn exists(&self, _: &str) -> bool {
            self.is_path
        }
    }   
    ...
}

Finally, we need to refactor the execute function to take something that implements PathValidator as a parameter and update our tests to reflect this. The change as a whole is as follows, with some notes in comments.

use std::path::Path;

trait PathValidator {
    fn exists(&self, path: &str) -> bool;
}

struct CertValidator;

impl PathValidator for CertValidator {
    fn exists(&self, path: &str) -> bool {
        Path::new(path).exists()
    }
}

// Here we change the signature of execute to also take a value of type `impl PathValidator`, 
// see the note after this code block for more information. 
fn execute(validator: impl PathValidator, args: Vec<String>) -> Result<(), String> {
    if args.len() != 1 {
        let error = format!(
            "{}{}",
            "Error: did not receive a single argument, ",
            "please invoke cert-decoder as follows: ./cert-decoder /path/to/cert."
        );
        return Err(error);
    }
    let path = &args[0];

    // Instead of calling Path's exists method, we call exists on our 
    // PathValidator implementation.
    if !validator.exists(path) {
        return Err(
            "Error: path given as argument does not exist, it must be a path to a certificate!"
                .to_owned(),
        );
    }
    Ok(())
}

fn main() -> Result<(), String> {
    let args = std::env::args().skip(1).collect();

    // Here we create our real PathValidator implementation,
    // CertValidator, which touches the filesystem.
    let validator = CertValidator;
    execute(validator, args)
}

#[cfg(test)]
mod test {

    use crate::{execute, PathValidator};

    struct FakeValidator {
        is_path: bool,
    }

    impl PathValidator for FakeValidator {
        fn exists(&self, _: &str) -> bool {
            self.is_path
        }
    }

    #[test]
    fn should_error_if_not_given_a_single_argument() {
        // arrange
        let args = Vec::new();

        // We construct a FakeValidator that says all paths exist.
        // It will never be called in this test, however.
        let validator = FakeValidator { is_path: true };

        // act
        let result = execute(validator, args);

        // assert
        assert!(result.is_err());
        assert_eq!(
            result.err().unwrap(),
            format!(
                "{}{}",
                "Error: did not receive a single argument, ",
                "please invoke cert-decoder as follows: ./cert-decoder /path/to/cert."
            )
        );
    }

    #[test]
    fn should_error_if_argument_is_not_a_path_which_exists() {
        // arrange
        let args = vec!["does-not-exist".to_owned()];

        // We construct a validator that says no path exists.
        // This will cause our test to fail and return the error we want,
        // mimicing a path which does not exist without touching the real
        // filesystem.
        let validator = FakeValidator { is_path: false };

        // act
        let result = execute(validator, args);

        // assert
        assert!(result.is_err());
        assert_eq!(
            result.err().unwrap(),
            "Error: path given as argument does not exist, it must be a path to a certificate!"
        );
    }
}

📝 Note

The type of validator in the execute function is impl PathValidator. This simply says validator can be a value of any type which implements PathValidator. You can read more about this in the Rust book in the section on Traits as Parameters.

Run the tests again and they should still be green! If you run with cargo run as we did earlier, it should still work as expected. The code up to this point can be seen here.

Check argument is a file

What's left on our test list?

Validate args

✔️ Only allow a single argument
✔️ Argument is a path that exists
☐ Argument should be a single file

Right now the argument we pass can be several file types other than a regular file, for example, a directory. As we only validate if the given path exists, not what that path points to. So let's validate it is also a file. First, a test.

#[cfg(test)]
mod test {

    use crate::{execute, PathValidator};

    struct FakeValidator {
        is_path: bool,
        is_file: bool,
    }

    impl PathValidator for FakeValidator {
        fn exists(&self, _: &str) -> bool {
            self.is_path
        }

        fn is_file(&self, _: &str) -> bool {
            self.is_file
        }
    }

    ... 

    #[test]
    fn should_error_if_argument_is_not_a_regular_file() {
        // arrange
        let args = vec!["not-a-regular-file".to_owned()];
        let validator = FakeValidator {
            is_path: true,
            is_file: false,
        };

        // act
        let result = execute(validator, args);

        // assert
        assert!(result.is_err());
        assert_eq!(
            result.err().unwrap(),
            "Error: path given is not a regular file, please update to point to a certificate."
        );
    }
}

This won't compile as there is no is_file method on our PathValidator trait. Here I've also updated the FakeValidator in the other test, setting is_file to false. Let's update that and implement the real version next.

trait PathValidator {
    fn exists(&self, path: &str) -> bool;
    fn is_file(&self, path: &str) -> bool;
}

struct CertValidator;

impl PathValidator for CertValidator {
    fn exists(&self, path: &str) -> bool {
        Path::new(path).exists()
    }

    fn is_file(&self, path: &str) -> bool {
        Path::new(path).is_file()
    }
}

Now we have a new is_file method. If we re-run the test it should fail as we are not returning an error. You can run a single test with cargo test by passing its name, or a string that is contained in its name.

➜ cargo -q test should_error_if_argument_is_not_a_regular_file

running 1 test
F
failures:

---- test::should_error_if_argument_is_not_a_regular_file stdout ----
thread 'test::should_error_if_argument_is_not_a_regular_file' panicked at 'assertion failed: result.is_err()', src
/main.rs:122:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    test::should_error_if_argument_is_not_a_regular_file

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 2 filtered out

error: test failed, to rerun pass '--bin cert-decode'

Let's make it return an error if the path given is not a regular file.

fn execute(validator: impl PathValidator, args: Vec<String>) -> Result<(), String> {
    if args.len() != 1 {
        let error = format!(
            "{}{}",
            "Error: did not receive a single argument, ",
            "please invoke cert-decoder as follows: ./cert-decoder /path/to/cert."
        );
        return Err(error);
    }
    let path = &args[0];
    if !validator.exists(path) {
        return Err(
            "Error: path given as argument does not exist, it must be a path to a certificate!"
                .to_owned(),
        );
    }
    if !validator.is_file(path) {
        return Err(
            "Error: path given is not a regular file, please update to point to a certificate."
                .to_owned(),
        );
    }
    Ok(())
}

Re-run and all tests should be green.

➜ cargo -q test

running 3 tests
...
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Great! The code up to this point can be seen here. We can now tick this off on the test list.

Validate args

✔️ Only allow a single argument
✔️ Argument is a path that exists
✔️ Argument should be a single file

Another small refactor

We can refactor this a bit. If we have a regular file then it must be a path that exists. This means we can get rid of the exists check and its test entirely. Here is a diff of this change.

diff --git a/src/main.rs b/src/main.rs
index 171545b..2ec2d92 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -1,17 +1,12 @@
 use std::path::Path;

 trait PathValidator {
-    fn exists(&self, path: &str) -> bool;
     fn is_file(&self, path: &str) -> bool;
 }

 struct CertValidator;

 impl PathValidator for CertValidator {
-    fn exists(&self, path: &str) -> bool {
-        Path::new(path).exists()
-    }
-
     fn is_file(&self, path: &str) -> bool {
         Path::new(path).is_file()
     }
@@ -27,12 +22,6 @@ fn execute(validator: impl PathValidator, args: Vec<String>) -> Result<(), Strin
         return Err(error);
     }
     let path = &args[0];
-    if !validator.exists(path) {
-        return Err(
-            "Error: path given as argument does not exist, it must be a path to a certificate!"
-                .to_owned(),
-        );
-    }
     if !validator.is_file(path) {
         return Err(
             "Error: path given is not a regular file, please update to point to a certificate."
@@ -54,15 +43,10 @@ mod test {
     use crate::{execute, PathValidator};

     struct FakeValidator {
-        is_path: bool,
         is_file: bool,
     }

     impl PathValidator for FakeValidator {
-        fn exists(&self, _: &str) -> bool {
-            self.is_path
-        }
-
         fn is_file(&self, _: &str) -> bool {
             self.is_file
         }
@@ -72,10 +56,7 @@ mod test {
     fn should_error_if_not_given_a_single_argument() {
         // arrange
         let args = Vec::new();
-        let validator = FakeValidator {
-            is_path: true,
-            is_file: false,
-        };
+        let validator = FakeValidator { is_file: false };

         // act
         let result = execute(validator, args);
@@ -92,34 +73,11 @@ mod test {
         );
     }

-    #[test]
-    fn should_error_if_argument_is_not_a_path_which_exists() {
-        // arrange
-        let args = vec!["does-not-exist".to_owned()];
-        let validator = FakeValidator {
-            is_path: false,
-            is_file: false,
-        };
-
-        // act
-        let result = execute(validator, args);
-
-        // assert
-        assert!(result.is_err());
-        assert_eq!(
-            result.err().unwrap(),
-            "Error: path given as argument does not exist, it must be a path to a certificate!"
-        );
-    }
-
     #[test]
     fn should_error_if_argument_is_not_a_regular_file() {
         // arrange
         let args = vec!["not-a-regular-file".to_owned()];
-        let validator = FakeValidator {
-            is_path: true,
-            is_file: false,
-        };
+        let validator = FakeValidator { is_file: false };

         // act
         let result = execute(validator, args);

The code up to this point can be seen here.

In the next post

In the next few posts, we'll look at reading information out of the certificate using the x509_parser crate. We'll also switch to using structopt for argument parsing. I didn't use structopt here as I wanted to keep things simple and mainly focus on the workflow, show how we can evolve functionality in small testable steps. I'm hoping the value of this workflow will be more apparent as we add more functionality to this cli, and things get more complex.

There are also a couple of things implemented here which you would not see in general Rust code. For example, returning a String as the Err value of a Result. This is ok, we will refactor them as we go. It's better to start with something small, get it working, then refactor than to try to make it perfect right away.

PEM encoding - see Section 2 of RFC 7468. ↩
The -q in cargo -q test is short for --quiet. ↩

Structure of an SSL (X.509) certificate

wayofthepie — Fri, 22 May 2020 10:45:35 +0000

Introduction

I've been working on some tooling for pulling certificate information out of the certificate transparency logs on and off for a while. I started looking at this again after a few weeks away from it and I've forgotten quite a lot! I've even forgotten some of the basics of what makes a certificate. In this post I want to dive into the structure of a certificate, what it is made of at a high level. I won't talk much about how certificates are used in protocols (e.g. Transport Layer Security (TLS)).

This post started as a reference for myself but other folks may find it interesting or useful. It has a lot of external references to RFC's which are stored in footnotes. As of now there is a small bug with how footnotes work where the footnote itself is hidden behind the top bar, see dev.to #7760 for more info. I might try fix it myself this weekend.

Quick note, SSL certificates are X.509 certificates. The term SSL certificate is deeply ingrained on the web, and even though the SSL protocol should no longer be used this term is still used everywhere.

Information in a certificate
- Side note on the openssl command
A breakdown of the main fields
Certificate
- Signature and Signature Algorithm
TBSCertificate
- Version
- Serial Number
- Signature Algorithm
  - This field is duplicated, why?
- Issuer
- Validity
  - Certificate validity and the Brazilian government
- Subject and Subject Public Key Info
Conclusion

Information in a certificate

We'll use the openssl cli to retrieve a certificate, then we can start looking into its structure. If the openssl cli is not installed you should be able to install it through your operating system's package manager.

$ openssl s_client -connect google.com:443 2>/dev/null < /dev/null \
    | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p' > google.com.crt

$ cat google.com.crt
-----BEGIN CERTIFICATE-----
MIIJRDCCCCygAwIBAgIRAJvxi9ebRliZAgAAAABjmHEwDQYJKoZIhvcNAQELBQAw
QjELMAkGA1UEBhMCVVMxHjAcBgNVBAoTFUdvb2dsZSBUcnVzdCBTZXJ2aWNlczET
MBEGA1UEAxMKR1RTIENBIDFPMTAeFw0yMDA0MTUyMDE2NDdaFw0yMDA3MDgyMDE2
...
V+hT9mqgeN10ryOWyN74CvBaw73K3hobSkDAyQS1HkbAqJP9VTuvjZl4PE0ndaIN
yiz/84k5xbSwxO++BuJgMUwj+WaLcvDW
-----END CERTIFICATE-----

$ openssl x509 -text -noout -in google.com.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            9b:f1:8b:d7:9b:46:58:99:02:00:00:00:00:63:98:71
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Google Trust Services, CN = GTS CA 1O1
        Validity
            Not Before: Apr 15 20:16:47 2020 GMT
            Not After : Jul  8 20:16:47 2020 GMT
        Subject: C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:8e:a4:03:0d:0c:a7:1d:52:28:80:ba:89:51:b9:
                    45:7a:7a:60:33:a5:ab:25:a4:05:c8:32:d9:b6:5c:
                    ...
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                D0:7D:02:36:9B:CD:47:0B:C5:9C:51:0F:27:A7:70:65:5A:C5:50:E9
            X509v3 Authority Key Identifier:
                keyid:98:D1:F8:6E:10:EB:CF:9B:EC:60:9F:18:90:1B:A0:EB:7D:09:FD:2B

            Authority Information Access:
                OCSP - URI:http://ocsp.pki.goog/gts1o1
                CA Issuers - URI:http://pki.goog/gsr2/GTS1O1.crt

            X509v3 Subject Alternative Name:
                DNS:*.google.com, DNS:*.android.com, DNS:*.appengine.google.com, ... 
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.2
                Policy: 1.3.6.1.4.1.11129.2.5.3

            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://crl.pki.goog/GTS1O1.crl

            CT Precertificate SCTs:
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : B2:1E:05:CC:8B:A2:CD:8A:20:4E:87:66:F9:2B:B9:8A:
                                25:20:67:6B:DA:FA:70:E7:B2:49:53:2D:EF:8B:90:5E
                    Timestamp : Apr 15 21:16:49.089 2020 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:20:26:77:E7:A4:C6:F9:D3:C0:0E:95:15:3C:
                                A2:08:F0:DB:77:9F:1F:7A:EC:7A:26:9B:E8:82:95:33:
                                ...
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 5E:A7:73:F9:DF:56:C0:E7:B5:36:48:7D:D0:49:E0:32:
                                7A:91:9A:0C:84:A1:12:12:84:18:75:96:81:71:45:58
                    Timestamp : Apr 15 21:16:49.137 2020 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:FC:5A:10:9B:63:81:BB:16:81:8B:D5:
                                88:AF:09:A1:D8:83:FD:C3:86:CB:B1:CD:55:71:FF:76:
                                ...
    Signature Algorithm: sha256WithRSAEncryption
         20:69:ba:0b:e5:b4:7a:36:f7:4f:d2:b2:0f:0d:c1:10:b0:12:
         7e:13:f9:f1:ca:6c:a0:c2:46:21:fb:8a:fd:a8:66:a9:96:43:
         ...

There's quite a lot of information in the certificate. Before we break this down, a quick side note on the openssl command we used in the above code block. Feel free to skip this next section if you understood it.

Side note on the openssl command

The command we used was:

$ openssl s_client -connect google.com:443 2>/dev/null < /dev/null \
    | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p' > google.com.crt

openssl s_client is used for connecting to hosts over TLS (and originally SSL, but no server should be using this anymore...). With -connect we tell it to connect to google.com on port 443. 2>/dev/null says to redirect anything that goes to stderr in the output of the openssl s_client command into /dev/null. Essentially this says ignore stderr. < /dev/null says read /dev/null into the stdin of the process, in thi s case stdin of openssl. Doing this always returns an end of file.
+openssl s_client is used for connecting to hosts over TLS (and originally SSL, but no site should be using this anymore...).

If you don't do this stdin redirect the openssl s_client command will hang waiting for input on stdin, doing this redirect means that once it tries to read from stdin it will read an end of file and the read will finish. The last part of the command, | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p' > google.com.crt, pipes the output from openssl s_client into sed which pulls the certificate out of the output and redirects it to a file called google.com.crt.

So why does it try to wait for input? There are few commands you can send it. Normally I'd just send Q to quit, however you can also tell it to send a http request. Try it! Run openssl s_client -connect google.com:443 then when it hangs type in:

GET / HTTP/1.1

And press enter twice. It should return the JS/html/css that makes up google.com. That was a bit of a digression, back to certs now.

A breakdown of the main fields

Now we have google.com's cert, what do all the fields mean? Let's break it down one by one. As we do this I will also mention the type and structure of the given field in the ASN.1 (Abstract Syntax Notation¹) specification for certificates and try to explain things which may not be obvious. Wherever I outline a part of the definition I will start it with a comment # ASN.1. I will also strip out pieces of information which are not relevant to a particular section and replace them with .... The full ASN.1 definition can be found in Appendix A.1 of RFC 5280 - X.509 Public Key Infrastructure.

Note that the ASN.1 spec for certificates describes a high level representation of certificate information. As bytes, this information is further encoded as DER (Distinguished Encoding Rules). I won't go into the details of ASN.1 -> DER encoding², but this post should lay some groundwork to make this encoding clearer in a future post.

Above we used openssl to pull out the information from an existing cert. The fields mentioned there can have many possible values. The main structure is defined as follows in ASN.1 - don't worry if you don't understand it, we will cover each field in depth.

# ASN.1
Certificate  ::=  SEQUENCE  {
     tbsCertificate       TBSCertificate,
     signatureAlgorithm   AlgorithmIdentifier,
     signature            BIT STRING  }

TBSCertificate  ::=  SEQUENCE  {
     version         [0]  Version DEFAULT v1,
     serialNumber         CertificateSerialNumber,
     signature            AlgorithmIdentifier,
     issuer               Name,
     validity             Validity,
     subject              Name,
     subjectPublicKeyInfo SubjectPublicKeyInfo,
     issuerUniqueID  [1]  IMPLICIT UniqueIdentifier OPTIONAL,
                          -- If present, version MUST be v2 or v3
     subjectUniqueID [2]  IMPLICIT UniqueIdentifier OPTIONAL,
                          -- If present, version MUST be v2 or v3
     extensions      [3]  Extensions OPTIONAL
                          -- If present, version MUST be v3 --  }

Certificate

Certificate is an ASN.1 SEQUENCE. A SEQUENCE is an ordered list of values. In this case a list with tbsCertificate, signatureAlgorithm and signature. First let's look at signature and signatureAlgorithm.

Signature and Signature Algorithm

In the openssl output for google.com's cert both Signature³ and Signature Algorithm⁴ can be seen right at the bottom:

Signature Algorithm: sha256WithRSAEncryption
     20:69:ba:0b:e5:b4:7a:36:f7:4f:d2:b2:0f:0d:c1:10:b0:12:
     7e:13:f9:f1:ca:6c:a0:c2:46:21:fb:8a:fd:a8:66:a9:96:43:
     ...

I've stripped out part of the Signature and replaced it with ... as it's not relevant. The Signature Algorithm field indicates the algorithm used by the issuing Certificate Authority (CA) to sign this certificate. Here its value is sha256WithRSAEncryption. For more information on this algorithm see Section 5 of RFC 4055 - Additional Algorithms and Identifiers for RSA ... and RFC 2313 - PKCS #1: RSA Encryption. Signature and SignatureAlgorithm are defined as follows in the ASN.1 spec:

# ASN.1
Certificate  ::=  SEQUENCE  {
     ...
     signatureAlgorithm   AlgorithmIdentifier,
     signature            BIT STRING  }

signature has type BIT STRING, this is simply a string of bits. In this case those bits contain a digital signature computed from the tbsCertificate field of this cert, using the algorithm defined in signatureAlgorithm. signatureAlgorithm is of type AlgorithmIdentifier. openssl encodes this as : separated hexadecimal, e.g. 20:69:ba:0b:e5:b4.... AlgorithmIdentifier itself is a SEQUENCE, an ordered list with the given items. In this case a list with two values, algorithm and parameters.

# ASN.1
AlgorithmIdentifier  ::=  SEQUENCE  {
     algorithm               OBJECT IDENTIFIER,
     parameters              ANY DEFINED BY algorithm OPTIONAL  }
                                -- contains a value of the type
                                -- registered for use with the
                                -- algorithm object identifier value

Here algorithm has type OBJECT IDENTIFIER. An OBJECT IDENTIFIER, or OID, is a standard way of identifying objects defined by the International Telecommunications Union (ITU). It is defined in RFC 3061 - A URN Namespace of Object Identifiers and is definitely not something I'll go into more detail on in this post, it is a big topic! For the curious, the OID for sha256WithRSAEncryption is 1.2.840.113549.1.1.11⁵. It is defined in RFC 4055 Section 5, to fully understand that definition you may have to go down a rabbit hole of RFC's...

parameters defines parameters to the specified algorithm. We will see an example of this in the Subject and Subject Public Key Info section.

TBSCertificate

TBSCertificate contains the information on the subject of the certificate and the issuing CA. We will cover all fields except extensions, issuerUniqueID and subjectUniqueID. There is quite a lot in extensions, so I will leave it for another post. issuerUniqueID and subjectUniqueID are optional and do not appear in google.com's cert. It is also recommended that these not be set⁶.

Version

In the cert we decoded, the Version⁷ is as follows:

Version: 3 (0x2)

Version is defined as:

# ASN.1
TBSCertificate  ::=  SEQUENCE  {
     version         [0]  Version DEFAULT v1,
     ...
}
...
Version  ::=  INTEGER  {  v1(0), v2(1), v3(2)  }

This says the version field of a certificate can be one of three values - 0 which means the version is v1, 1 which means the version is v2 and 2 which means the version is v3. It defaults to v1, which is 0. This explains why there is a 3 and a 0x2 in the output from openssl, it's showing the version number (3) and the real value of the field (2).

So what does version mean? There are three versions of X.509 certificates out in the wild, it just indicates which version of the X.509 spec a given cert is using. I will gloss over the differences in the versions in this post as its not too important for an overview. You will mainly see X.509 v3 certificates in the wild.

Serial Number

The Serial Number⁸ is a unique integer given to the certificate. It is unique for all certificates issued from the same CA, e.g. Digicert or Lets Encrypt, not globally unique. Meaning two certs from different CA's could potentially have the same Serial Number. Serial Number looks as follows in the cert we decoded:

Serial Number:
    9b:f1:8b:d7:9b:46:58:99:02:00:00:00:00:63:98:71

And its ASN.1 definition:

# ASN.1
TBSCertificate  ::=  SEQUENCE  {
     ...
     serialNumber         CertificateSerialNumber,
     ...
}

CertificateSerialNumber  ::=  INTEGER

It is of type CertificateSerialNumber, which is just an alias for an integer.

Signature Algorithm

This field ⁹ must contain the same algorithm as the signatureAlgorithm field outside the tbsCertficate which we discussed in the Signature and Signature Algorithm section. And in this case it does:

Signature Algorithm: sha256WithRSAEncryption

This is the signature field in the TBSCertificate:

# ASN.1 
TBSCertificate  ::=  SEQUENCE  {
     ...
     signature            AlgorithmIdentifier,
     ...
}

We looked at AlgorithmIdentifier in Signature and Signature Algorithm.

This field is duplicated, why?

This was interesting as the reason for the duplication was not clear. We saw the Signature Algorithm appear in the top level Certificate and it also appears inside the TBSCertificate. Section 4.1.2.3 of RFC 5280, which talks about the signature field within TBSCertificate, states:

This field MUST contain the same algorithm identifier as the signatureAlgorithm field in the sequence Certificate (Section 4.1.1.2)

But nowhere in that RFC does it give a reason. Section 1 of RFC 6211 - Cryptographic Message Syntax (CMS) gives some clue as to a possible reason - to prevent algorithm substitution attacks:

The Cryptographic Message Syntax [CMS], unlike X.509/PKIX certificates [RFC5280], is vulnerable to algorithm substitution attacks. In an algorithm substitution attack, the attacker changes either the algorithm being used or the parameters of the algorithm in order to change the result of a signature verification process. In X.509 certificates, the signature algorithm is protected because it is duplicated in the TBSCertificate.signature field with the proviso that the validator is to compare both fields as part of the signature validation process.

I don't know enough about this to comment further. But I am currently investigating. Once I understand more I will write about it.

Issuer

The Issuer¹⁰ field is a unique identifier for the CA issuing this certificate.

Issuer: C = US, O = Google Trust Services, CN = GTS CA 1O1

The cert we decoded was issued by Google Trust Services. Google have a number of CA's under Google Trust Services see https://pki.goog/ for more details. The Issuer field along with the Serial Number will uniquely identify a certificate, as long as the Issuer is a globally trusted CA.

Issuer is defined as a Name in the spec:

# ASN.1
TBSCertificate  ::=  SEQUENCE  {
     ...
     issuer               Name,
     ...
}

Name itself is a bit weird:

# ASN.1 
Name ::= CHOICE { -- only one possibility for now --
      rdnSequence  RDNSequence }

RDNSequence ::= SEQUENCE OF RelativeDistinguishedName 

...

RelativeDistinguishedName ::= SET SIZE (1..MAX) OF AttributeTypeAndValue

# AttributeTypeAndValue is defined as follows
AttributeTypeAndValue ::= SEQUENCE {
     type     AttributeType,
     value    AttributeValue }

AttributeType ::= OBJECT IDENTIFIER

AttributeValue ::= ANY -- DEFINED BY AttributeType

CHOICE defines a list of options to pick from. So, Name is a choice with a single option, an RDNSequence, and it even says there is only one possibility for now. I don't really know why this is, it could be for forwards compatibility on changes to the spec.

Have a look at the definition of RDNSequence, it is defined as SEQUENCE OF RelativeDistinguishedName. This is different than SEQUENCE which we saw in the Signature Algorithm section. SEQUENCE OF defines a list of values which are all the same type, in this case they are all RelativeDistinguishedName.

We haven't seen SET yet either. This defines a set of objects. SET SIZE (1..MAX) OF AttributeTypeAndValue defines a set of AttributeTypeAndValue's of max size MAX, I'm not sure what value MAX is bound to here either, will edit and add this information once I realise what it is.

In short, Name is a list of OID/value pairs where the value is some object bound by that OID. Section 2.3 of RFC 2253 - LDAP v3 describes RelativeDistinguishedName and its encoding in more depth.

Validity

Validity¹¹ specifies the time window a certificate is valid between.

Validity
    Not Before: Apr 15 20:16:47 2020 GMT
    Not After : Jul  8 20:16:47 2020 GMT

The cert we decoded is valid from Apr 15 20:16:47 2020 GMT to Jul 8 20:16:47 2020 GMT. The certificate is invalid outside of this timeframe.

# ASN.1
TBSCertificate  ::=  SEQUENCE  {
     ...
     validity             Validity,
     ...
}
...
Validity ::= SEQUENCE {
     notBefore      Time,
     notAfter       Time  }

Time ::= CHOICE {
     utcTime        UTCTime,
     generalTime    GeneralizedTime }

Validity has two fields which are both of type Time. Time can be one of two types, UTCTime¹² or GeneralizedTime¹³. Dates before the year 2050 must be encoded as UTCTime and dates on or after the year 2050 must be encoded as GeneralizedTime. This is outlined in the definition of Validity in RFC 5280.

Certificate validity and the Brazilian government

I stumbled across this issue on github while on my certificate information seeking adventures. As mentioned above, dates before the year 2050 should be encoded as UTCTime, but the Brazilian government had their own specification which required the use of GeneralizedTime for all dates. This is a good example of what you see in an RFC/specification not being exactly what you see in a real system.

Subject and Subject Public Key Info

Subject¹⁴ identifies the owner of the public key in the Subject Public Key Info¹⁵ section , which defines the "thing" this certificate identifies. In this case it's identifying google domains.

Subject: C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.google.com

Its definition is:

# ASN.1
TBSCertificate  ::=  SEQUENCE  {
    ... 
    subject              Name,
    ...
}

Name was explained in the Issuer section, so it should be clear what this is. Subject Public Key Info has a few fields. Let's look at its ASN.1 definition first.

# ASN.1
TBSCertificate  ::=  SEQUENCE  {
    ...
    subjectPublicKeyInfo SubjectPublicKeyInfo,
    ...
}
...
SubjectPublicKeyInfo  ::=  SEQUENCE  {
        algorithm            AlgorithmIdentifier,
        subjectPublicKey     BIT STRING  }
...

Let's see the actual value of Subject Public Key Info in the certifcate again:

Subject Public Key Info:
  Public Key Algorithm: id-ecPublicKey
      Public-Key: (256 bit)
      pub:
          04:0f:45:4e:2f:0c:a7:88:9a:b9:24:ff:57:50:dc:
          f1:ab:6e:dd:3e:7f:82:26:30:a7:12:9f:81:8a:27:
          9d:7d:06:2e:d3:e2:50:3b:ce:6c:2d:2e:5b:32:ce:
          7d:eb:86:06:7c:8c:29:2b:47:61:de:f0:ca:f8:b7:
          98:00:21:6a:34
      ASN1 OID: prime256v1
      NIST CURVE: P-256

Public Key Algorithm defines the algorithm the key can be used with. This is the algorithm field in the SubjectPublicKeyInfo field defined above. The rest is the BIT STRING. The BIT STRING itself is constrained by the type of algorithm. The information in the fields the BIT STRING decodes to is beyond this post. If interested see Section 2.3.5 of RFC 3279 - Algorithms and Identifiers X.509 PKI ... for more on id-ecPublicKey.

Conclusion

If you made it to here, kudos! I hope this has given you a better understanding of what really makes up a certificate, and the sheer complexity around X.509 in general. There is a lot I glossed over in this post which I hope to drill deeper into in the future.

In the next post I'll cover the TBSCertificate extensions fields.

ASN.1 - see Introduction to ASN.1. ↩
ASN.1 to DER encoding - see ASN.1 encoding rules. ↩
Signature - see Section 4.1.1.3 of RFC 5280. ↩
Signature Algorithm in Certificate - see Section 4.1.2.3 of RFC 5280. ↩
OID for sha256WithRSAEncryption - see https://oidref.com/1.2.840.113549.1.1.11. ↩
issuerUniqueID and subjectUniqueID - see Section 4.1.2.8 of RFC 5280. ↩
Version - see Section 4.1.2.1 of RFC 5280. ↩
Serial Number - see Section 4.1.2.2 of RFC 5280. ↩
Signature Algorithm in TBSCertificate - see Section 4.1.1.2 of RFC 5280. ↩
Issuer - see Secion 4.2.1.4 of RFC 5280 ↩
Validity - see Secion 4.2.1.5 of RFC 5280 ↩
UTCTime - see Section 4.1.2.5.1 of RFC 5280. ↩
GeneralizedTime - see Section 4.1.2.5.2 of RFC 5280. ↩
Subject - see Secion 4.2.1.6 of RFC 5280 ↩
Subject Public Key - see Secion 4.2.1.6 of RFC 5280 ↩

Improving Our GitHub Actions Runner Orchestrator

wayofthepie — Sat, 08 Feb 2020 22:22:22 +0000

Github checks api
- Making use of this new information
Using kubernetes jobs to schedule actions runners
- Storing our token properly
Running our script as a kubernetes cronjob
- Assigning the correct permissions
Fixing a bug in our logic
Conclusion

In the last post we got a simple actions runner orchestrator running with bash and cron. We also noted a few issues with that version. In this post we will fix up the following issues:

Instead of launching a runner per commit, we will instead launch a runner per check request.
Instead of running local docker containers, we will run kubernetes jobs.
Instead of just running locally with cron, we will create a kubernetes CronJob.

Let's do it!

Github checks api

When a check run is requested, the github checks api will reflect this. For example:

$ curl -s \
    -H "accept: application/vnd.github.antiope-preview+json" \
    -H "authorization: token ${PAT}" \
    https://api.github.com/repos/wayofthepie/gh-app-test/commits/e13119b0/check-runs
{
  "total_count": 1,
  "check_runs": [
    {
      "id": 433544203,
      "node_id": "MDg6Q2hlY2tSdW40MzM1NDQyMDM=",
      "head_sha": "e13119b07d81e4c587882b2f7c9d7a730810f709",
      "external_id": "ca395085-040a-526b-2ce8-bdc85f692774",
      "url": "https://api.github.com/repos/wayofthepie/gh-app-test/check-runs/433544203",
      "html_url": "https://github.com/wayofthepie/gh-app-test/runs/433544203",
      "details_url": "https://help.github.com/en/actions",
      "status": "queued",
      "conclusion": null,
      "started_at": "2020-02-08T14:40:27Z",
      "completed_at": null,
...

This will return the status of all the check runs for the given commit (above in the url e13119b0 is the short ref for the given commit). As you can see above the status of the first check run is queued, in this case it means it's awaiting a runner to be executed on.

Using this information will also allow our script to be completely stateless. In the previous post it had to keep track of the last commit in a file, with the checks API we no longer need this.

Making use of this new information

Now we can change the logic in our orchestration script as follows:

Get the latest commit.
Get all requested check runs for that commit.
For each requested run launch an actions runner.

Here is the updated script:

#!/usr/bin/env bash

PAT=$1
OWNER=$2
REPO=$3

# make sure we have values for all our arguments
[ -z ${PAT} ] || [ -z ${OWNER} ] || [ -z $REPO ] && {
    echo "Incorrect usage, example: ./orc.sh personal-access-token owner some-repo"
    exit 1
}

# get the latest commit
latest_commit=$(curl -s -H "authorization: token ${PAT}"  \
    https://api.github.com/repos/${OWNER}/${REPO}/commits |\
    jq -r .[0].sha)

# for each check run requested for this commit, get the "status"
# field and assign to the "check_status" variable 
for check_status in $(curl -s \
    -H "accept: application/vnd.github.antiope-preview+json" \
    -H "authorization: token ${PAT}"\
    https://api.github.com/repos/${OWNER}/${REPO}/commits/${latest_commit}/check-runs \
    | jq -r '.check_runs[] | "\(.status)"'); do

    # if "check_status" is queued launch an action runner
    if [ "${check_status}" == "queued" ]; then
        echo "Launching actions runner ..."
        docker run -d --rm actions-image \
            ${OWNER} \
            ${REPO} \
            ${PAT} \
            $(uuidgen)
    fi
done

The code up to this point can be found here.

Add a new commit to the repository you have been running the actions against and run ./orc.sh ${PAT} ${OWNER} ${REPO}, it should start a container and run the build.

Using kubernetes jobs to schedule actions runners

If we move to using kubernetes instead of our local docker daemon we can scale out much easier. First let's update to launch the actions runners as kubernetes jobs instead of direct docker containers.

First let's create a cluster. There are many ways to do this, I use Google Cloud so I'm going to create a cluster on google cloud. This has a cost, see https://kubernetes.io/docs/setup/ for examples of local cluster setups.

To create a cluster on google cloud:

$ gcloud container clusters create actions-spawner --region europe-west2-c --num-nodes 1
WARNING: Currently VPC-native is not ...
WARNING: Newly created clusters ...
...
Creating cluster actions-spawner in europe-west2-c... Cluster is being health-checked (master is healthy)...done.
Created [https://container.googleapis.com/v1/projects/monthly-hacking/zones/europe-west2-c/clusters/actions-spawner].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/europe-west2-c/actions-spawner?project=monthly-
hacking
kubeconfig entry generated for actions-spawner.
NAME             LOCATION        MASTER_VERSION  MASTER_IP     MACHINE_TYPE   NODE_VERSION    NUM_NODES  STATUS
actions-spawner  europe-west2-c  1.13.11-gke.23  35.189.78.16  n1-standard-1  1.13.11-gke.23  1          RUNNING

This will create a cluster with one node, we don't need more than that to test. Make sure you have kubectl installed. Auth should be setup automatically for kubectl. To test let's try to list the nodes:

$ kubectl get nodes
NAME                                             STATUS   ROLES    AGE     VERSION
gke-actions-spawner-default-pool-f1380f72-3wm5   Ready    <none>   4m47s   v1.13.11-gke.23

Looks good! Now, let's update our orchestration script:

#!/usr/bin/env bash

PAT=$1
OWNER=$2
REPO=$3

# make sure we have values for all our arguments
[ -z ${PAT} ] || [ -z ${OWNER} ] || [ -z $REPO ] && {
    echo "Incorrect usage, example: ./orc.sh personal-access-token owner some-repo"
    exit 1
}

# get the latest commit
latest_commit=$(curl -s -H "authorization: token ${PAT}"  \
    https://api.github.com/repos/${OWNER}/${REPO}/commits |\
    jq -r .[0].sha)

# for each check run requested for this commit, get the "status"
# field and assign to the "check_status" variable
for check_status in $(curl -s \
    -H "accept: application/vnd.github.antiope-preview+json" \
    -H "authorization: token ${PAT}"\
    https://api.github.com/repos/${OWNER}/${REPO}/commits/${latest_commit}/check-runs \
    | jq -r '.check_runs[] | "\(.status)"'); do

    # if "check_status" is queued launch an action runner
    if [ "${check_status}" == "queued" ]; then
        echo "Found check run request with status ${check_status}, launching job ..."
        cat job.yaml \
            | sed -r "s/NAME/$(uuidgen)/; s/OWNER/${OWNER}/; s/REPO/${REPO}; s/TOKEN/${TOKEN}" \
            | kubectl apply -f -
    else
        echo "Found check run request with status '${check_status}', nothing to do ..."
    fi
done

And create the job.yaml, the specification for our kubernetes job:

apiVersion: batch/v1
kind: Job
metadata:
  name: {NAME}
spec:
  template:
    spec:
      containers:
      - name: {NAME}
        image: wayofthepie/actions-image
        args: ["{OWNER}", "{REPO}", "{TOKEN}"]
      restartPolicy: Never
  backoffLimit: 4

⚠️ WARNING ⚠️
The token here should be a stored as a kubernetes secret. Using the token as I have above is not good practice. I will fix this later in this post.

Let's test it:

$ ./orc.sh ${PAT} ${OWNER} ${REPO}
Found check run request with status 'completed', nothing to do ...

Great, it works when there are no runs requested. Commit to the repo you are testing against and run again:

$ ./orc.sh ${PAT} ${OWNER} ${REPO}
Found check run request with status queued, launching job ...
job.batch/990a0d3d-bb98-419e-abc5-ca4fa48ca328 created

Looks like it worked. Let's see what's running:

$ kubectl get jobs
NAME                                   COMPLETIONS   DURATION   AGE
990a0d3d-bb98-419e-abc5-ca4fa48ca328   0/1           5s         5s

$ kubectl get pods
NAME                                         READY   STATUS      RESTARTS   AGE
990a0d3d-bb98-419e-abc5-ca4fa48ca328-wxn5c   1/1     Running     0          8s

$ kubectl logs 990a0d3d-bb98-419e-abc5-ca4fa48ca328-wxn5c -f
Unrecognized command-line input arguments: 'name'. For usage refer to: .\config.cmd --help or ./config.sh --help

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration

Enter the name of runner: [press Enter for 990a0d3d-bb98-419e-abc5-ca4fa48ca328-wxn5c]
√ Runner successfully added
√ Runner connection is good
# Runner settings


√ Settings Saved.


√ Connected to GitHub

2020-02-08 16:35:49Z: Listening for Jobs
2020-02-08 16:35:53Z: Running job: build

Great! It kicked off a build. However, notice the warning at the start:

Unrecognized command-line input arguments: 'name'. For usage refer to: .\config.cmd --help or ./config.sh --help

Something is wrong... A quick look through orc.sh and job.yaml highlights the issue - we are missing the fourth argument for the wayofthepie/actions-runner image! This sets the name of the actions runner. Let's fix it up:

apiVersion: batch/v1
kind: Job
metadata:
  name: {NAME}
spec:
  template:
    spec:
      containers:
      - name: {NAME}
        image: wayofthepie/actions-image
        # here we add the name argument
        args: ["{OWNER}", "{REPO}", "{TOKEN}", "{NAME}"] 
      restartPolicy: Never
  backoffLimit: 4

Let's commit to the test repo and run again:

$ ./orc.sh ${PAT} ${TOKEN} ${REPO}
Found check run request with status queued, launching job ...
job.batch/7abcb7a1-b1bb-4641-88af-fc4562e29bb7 created

$ kubectl get jobs
NAME                                   COMPLETIONS   DURATION   AGE
7abcb7a1-b1bb-4641-88af-fc4562e29bb7   0/1           4s         4s
990a0d3d-bb98-419e-abc5-ca4fa48ca328   1/1           46s        12m

$ kubectl get pods
NAME                                         READY   STATUS      RESTARTS   AGE
7abcb7a1-b1bb-4641-88af-fc4562e29bb7-q5jdd   1/1     Running     0          13s
990a0d3d-bb98-419e-abc5-ca4fa48ca328-wxn5c   0/1     Completed   0          12m

$ kubectl logs 7abcb7a1-b1bb-4641-88af-fc4562e29bb7-q5jdd

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration


√ Runner successfully added
√ Runner connection is good

# Runner settings


√ Settings Saved.


√ Connected to GitHub

2020-02-08 16:48:30Z: Listening for Jobs
2020-02-08 16:48:34Z: Running job: build
2020-02-08 16:48:52Z: Job build completed with result: Succeeded

# Runner removal


√ Runner removed successfully
√ Removed .credentials
√ Removed .runner

Great! No warnings, all working as expected. The code up to this point can be seen here.

Storing our token properly

Right now our personal access token gets stored in the definition of our job! If we retrieve our job we can see it:

$ kubectl get jobs 990a0d3d-bb98-419e-abc5-ca4fa48ca328 -o json
{
    "apiVersion": "batch/v1",
    "kind": "Job",
    "metadata": {
        ...
    },
    "spec": {
        ...
        "template": {
           ...
            "spec": {
                "containers": [
                    {
                        "args": [
                            "wayofthepie",
                            "gh-app-test",
                            "THE TOKEN IS IN HERE!!!"
                        ],
                        "image": "wayofthepie/actions-image",
                        "imagePullPolicy": "Always",
                        "name": "990a0d3d-bb98-419e-abc5-ca4fa48ca328",
                        ...
                    }
                ],
    ...
}

This is not good! We should be storing this as a kubernetes secret. Let's do that. First create a secret.yaml defining our secret:

apiVersion: v1
kind: Secret
metadata:
  name: github-token
type: Opaque
stringData:
  token: {TOKEN}

To create the secret:

$ cat secret.yaml \
    | sed -r "s/\{TOKEN\}/${YOUR_PAT}/"\
    | kubectl apply -f -
secret/github-token created

$ kubectl get secrets
NAME                  TYPE                                  DATA   AGE
...
github-token          Opaque                                1      30s

With that created let's update our job spec in job.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: {NAME}
spec:
  template:
    spec:
      containers:
      - name: {NAME}
        image: wayofthepie/actions-image
        args: ["{OWNER}", "{REPO}", "$(GITHUB_TOKEN)", "{NAME}"]
        env:
        - name: GITHUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: github-token
              key: token
      restartPolicy: Never
  backoffLimit: 4

See these docs for more on reading secret data into env vars.

Also note the syntax to reference the GITHUB_TOKEN it uses $() and not ${}, see here.

Re-run and everything should work:

$ ./orc.sh ${PAT} ${OWNER} ${REPO}
Found check run request with status queued, launching job ...
job.batch/eb1e314d-594b-4253-ae8a-74c797a2cd76 created

$ kubectl get pods
NAME                                         READY   STATUS              RESTARTS   AGE
eb1e314d-594b-4253-ae8a-74c797a2cd76-x47lr   0/1     ContainerCreating   0          3s

$ kubectl logs -f eb1e314d-594b-4253-ae8a-74c797a2cd76-x47lr
...
2020-02-08 17:41:08Z: Listening for Jobs
2020-02-08 17:41:12Z: Running job: build
2020-02-08 17:41:30Z: Job build completed with result: Succeeded

# Runner removal


√ Runner removed successfully
√ Removed .credentials
√ Removed .runner

Great! We should also clean up orc.sh:

#!/usr/bin/env bash

PAT=$1
OWNER=$2
REPO=$3

# make sure we have values for all our arguments
[ -z ${PAT} ] || [ -z ${OWNER} ] || [ -z $REPO ] && {
    echo "Incorrect usage, example: ./orc.sh personal-access-token owner some-repo"
    exit 1
}

# get the latest commit
latest_commit=$(curl -s -H "authorization: token ${PAT}"  \
    https://api.github.com/repos/${OWNER}/${REPO}/commits |\
    jq -r .[0].sha)

# for each check run requested for this commit, get the "status"
# field and assign to the "check_status" variable
for check_status in $(curl -s \
    -H "accept: application/vnd.github.antiope-preview+json" \
    -H "authorization: token ${PAT}"\
    https://api.github.com/repos/${OWNER}/${REPO}/commits/${latest_commit}/check-runs \
    | jq -r '.check_runs[] | "\(.status)"'); do

    # if "check_status" is queued launch an action runner
    if [ "${check_status}" == "queued" ]; then
        echo "Found check run request with status ${check_status}, launching job ..."
        cat job.yaml \
            # we removed the {TOKEN} replacement here
            | sed -r "s/\{NAME\}/$(uuidgen)/g; s/\{OWNER\}/${OWNER}/; s/\{REPO\}/${REPO}/" \
            | kubectl apply -f -
    else
        echo "Found check run request with status '${check_status}', nothing to do ..."
    fi
done

The code up to this point can be found here.

Running our script as a kubernetes cronjob

To run our orchestrator script as a kubernetes cronjob we first need to create a docker image:

FROM ubuntu

RUN useradd -m actions \
    && apt-get update \
    && apt-get install -y \
    curl \
    jq \
    uuid-runtime

RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.17.0/bin/linux/amd64/kubectl \
    && mv kubectl /usr/local/bin \
    && chmod +x /usr/local/bin/kubectl

WORKDIR /home/actions

USER actions

COPY orc.sh .
ENTRYPOINT ["./orc.sh"]

I built this as wayofthepie/actions-orchestrator and pushed to the public docker registry. Next, let' create a CronJob spec:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: actions-orchestrator
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: actions-orchestrator
            image: wayofthepie/actions-orchestrator
            args: ["$(GITHUB_TOKEN)", "{OWNER}", "{REPO}"]
            env:
            - name: GITHUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: github-token
                  key: token
          restartPolicy: Never

This will run our orchestrator every minute. To create the job, replace with your own repo and owner:

$ cat cron.yaml \
    | sed -r "s/\{OWNER\}/wayofthepie/; s/\{REPO\}/gh-app-test/" \
    | kubectl apply -f -
cronjob.batch/actions-orchestrator created

$ kubectl get cronjob
NAME                   SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
actions-orchestrator   */1 * * * *   False     0        <none>          7s

$ kubectl get pods
NAME                                         READY   STATUS              RESTARTS   AGE
actions-orchestrator-1581193620-4j8jz        0/1     ContainerCreating   0          1s

$ kubectl logs actions-orchestrator-1581193620-4j8jz
Found check run request with status queued, launching job ...
Error from server (Forbidden): error when retrieving current configuration of:
...
from server for: "STDIN": jobs.batch "316b08ed-89e7-4321-a521-897c7a40fa50" is forbidden: User "system:serviceaccount:default:default" cannot get resource "
jobs" in API group "batch" in the namespace "default"

An error! Let's delete the cronjob so it doesn't keep running, kubectl delete cronjob actions-orchestrator, and investigate.

Assigning the correct permissions

It seems the default service account we get in the pod does not have access to the jobs resource. To fix this we need to create a ClusterRole and ClusterRoleBinding:

roleRef:
  kind: ClusterRole
  name: jobs-manager
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: default
  name: jobs-manager
rules:
- apiGroups: ["batch", "extensions"]
  resources: ["jobs"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

And create it:

$ kubectl apply -f cluster-role.yaml
clusterrole.rbac.authorization.k8s.io/default created

Re-create our cronjob:

$ cat cron.yaml | sed -r "s/\{OWNER\}/wayofthepie/; s/\{REPO\}/gh-app-test/" | kubectl apply -f -
cronjob.batch/actions-orchestrator created

# it should run every minute
$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
actions-orchestrator-1581196020-tmbll   1/1     Running   0          4s

Great! Now if we commit to the test repo it should create a new job for the requested check runs.

Fixing a bug in our logic

We still only check the last commit for check requests meaning we can still miss requests, leaving check runs for some commits idle. This is a bug.. The real fix for this would require either a lot of API calls or using webhooks. But for now we can look at the last 5 minutes of commits rather than just the last commit. If we run the script every minute there is a much smaller chance of a missing check runs.

The updates to orc.sh are:

#!/usr/bin/env bash

PAT=$1
OWNER=$2
REPO=$3

# make sure we have values for all our arguments
[ -z ${PAT} ] || [ -z ${OWNER} ] || [ -z $REPO ] && {
    echo "Incorrect usage, example: ./orc.sh personal-access-token owner some-repo"
    exit 1
}

# get the date format in the format the github api wants
function five_minutes_ago {
    echo $(date --iso-8601=seconds --date='5 minutes ago' | awk -F'+' '{print $1}')
}

echo "Getting commits from the last 5 minutes ..."
commits=$(curl -s -H "accept: application/vnd.github.antiope-preview+json" \
    -H "authorization: token ${PAT}" \
    https://api.github.com/repos/${OWNER}/${REPO}/commits?since="$(five_minutes_ago)Z" \
    | jq -r .[].sha)

for commit in ${commits[@]}; do
    echo "Checking ${commit} for check requests ..."

    # for each check run requested for this commit, get the "status"
    # field and assign to the "check_status" variable
    for check_status in $(curl -s \
        -H "accept: application/vnd.github.antiope-preview+json" \
        -H "authorization: token ${PAT}"\
        https://api.github.com/repos/${OWNER}/${REPO}/commits/${commit}/check-runs \
        | jq -r '.check_runs[] | "\(.status)"'); do

        # if "check_status" is queued launch an action runner
        if [ "${check_status}" == "queued" ]; then
            echo "Found check run request with status ${check_status}, launching job ..."
            cat job.yaml \
                | sed -r "s/\{NAME\}/$(uuidgen)/g; s/\{OWNER\}/${OWNER}/; s/\{REPO\}/${REPO}/" \
                | kubectl apply -f -
        else
            echo "Found check run request with status '${check_status}', nothing to do ..."
        fi
    done
done

Rebuild the actions orchestrator image, push and it should all work! The image up to this point is tagged as wayofthepie/actions-orchestrator:8-2-2020.

The code up to this point can be found here.

Conclusion

We now have away of orchestrating actions runners on kubernetes. There are still a few issues however:

There is no error recovery and the error messages are pretty bad. For example if for some reason the cronjob does not run for 5+ minutes we may miss commits and check runs.
It would be much better to use webhooks here.
We currently only support watching a single repository.
Things are getting complicated with bash!

I will tackle some of these in the next post.

Hacking Together A GitHub Actions Runner Orchestrator

wayofthepie — Sun, 02 Feb 2020 19:59:05 +0000

Run a single build and cleanup
A simple orchestrator
- Detecting new commits
- An issue with our process
- Changing tactics
- Adding cron
Conclusion

In the last post, we built a docker image that allows us to connect and disconnect actions runners when we want. But we need to do this manually when we need one.

In this post, I'm going to hack together an orchestrator that launches an actions runner per commit, with bash and cron!

Run a single build and cleanup

Right now when we run a container and connect an actions runner, it will stay alive until we manually kill it, building any new commits. It would be nicer if it ran a single build then cleaned itself up.

Back in the previous post we found the Runner.Common Constants.cs. I mentioned that there were some other interesting pieces of code in that class, outside of what we were looking for at the time. One of those is the CommandLine Flags class. This contains the help option, which looks like it maps to config.sh --help. But it also contains a once option.

Digging about a bit more and seeing where this is used uncovers a variable called RunOnce in Runner.Listener CommandSettings.

...

public bool RunOnce => TestFlag(Constants.Runner.CommandLine.Flags.Once);

...

This looks promising. It is most likely an option we can pass to ./run.sh. Let's find out by updating entrypoint.sh!

#!/usr/bin/env bash

OWNER=$1
REPO=$2
PAT=$3
NAME=$4

cleanup() {
    token=$(curl -s -XPOST -H "authorization: token ${PAT}" \
        https://api.github.com/repos/${OWNER}/${REPO}/actions/runners/registration-token | jq -r .token)
    ./config.sh remove --token $token
}

token=$(curl -s -XPOST \
    -H "authorization: token ${PAT}" \
    https://api.github.com/repos/${OWNER}/${REPO}/actions/runners/registration-token | jq -r .token)

./config.sh \
    --url https://github.com/${OWNER}/${REPO} \
    --token ${token} \
    --name ${NAME} \
    --work _work

# add the --once option here
./run.sh --once

cleanup

Rebuild and launch the runner:

$ docker run -ti --rm actions-image ${OWNER} ${REPO} ${PAT} ${NAME}

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration


√ Runner successfully added
√ Runner connection is good

# Runner settings


√ Settings Saved.


√ Connected to GitHub

2020-02-02 18:05:11Z: Listening for Jobs

For the next steps to work, you will need to have an action workflow set up, here is an example:

name: CI

on: [push]

jobs:
  build:

    runs-on: self-hosted

    steps:
    - uses: actions/checkout@v2
    - name: Run a one-line script
      run: echo Hello, world!
    - name: Run a multi-line script
      run: |
        echo Add other actions to build,
        echo test, and deploy your project.
        sleep 10

Add this to the repository at .github/workflows/ci.yml. It should appear under the Actions tab in the repo. Now you can add a new commit to the repo, and your connected action runner should start the workflow:

...
2020-02-02 18:06:42Z: Running job: build
2020-02-02 18:07:00Z: Job build completed with result: Succeeded

# Runner removal

√ Runner removed successfully
√ Removed .credentials
√ Removed .runner

Awesome! It ran the action workflow on the new commit, exited, and cleaned up. Exactly what we wanted. Now we can build a simple orchestrator.

A simple orchestrator

I'm going to keep this dead simple. Here is the workflow I am thinking of:

Every minute check for new commits
If a new commit is detected, launch a runner

That's it! This has a few problems, for example if there are multiple new commits this will just launch a single runner which will build one commit and the other commits may never be built. However, if we get this flow working we can improve as we go.

Detecting new commits

The simplest way I can think of detecting new commits is by using the /repos/:owner/:repo/commits API. This returns a lot of info, all we need is the SHA1 hash of the last commit. For example:

$ curl -s -H "authorization: token ${PAT}"  \
    https://api.github.com/repos/wayofthepie/gh-app-test/commits \
    | grep sha \
    | head -n1 
"sha": "e0654f66fcb8061a439e404f1bf1b2c8b7116537",

Simple! We can make this a bit cleaner with jq:

curl -s -H "authorization: token ${PAT}"  \
    https://api.github.com/repos/wayofthepie/gh-app-test/commits \
    | jq -r .[0].sha
e0654f66fcb8061a439e404f1bf1b2c8b7116537

jq -r .[0].sha says get me the first object in an array, and get the value of the sha field from that. Give to me as a raw string (not wrapped in quotes).

With this, we can write a shell script (orc.sh) that detects new commits and launches a runner:

#!/usr/bin/env bash
##
# Detects new commits in a given repo by checking every minute
# and storing a reference to the last seen commit in a file called
# prev.
##

PAT=$1
OWNER=$2
REPO=$3
prev=prev

# make sure we have values for all our arguments
[ -z ${PAT} ] || [ -z ${OWNER} ] || [ -z $REPO ] && {
    echo "Incorrect usage, example: ./orc.sh personal-access-token owner some-repo"
    exit 1
}

# make sure the prev file exists
touch ${prev}

# get the latest commit
latest_commit=$(curl -s -H "authorization: token ${PAT}"  \
    https://api.github.com/repos/${OWNER}/${REPO}/commits |\
    jq -r .[0].sha)

# read the last commit we saw
prev_commit=$(cat ${prev})

echo "Latest commit is ${latest_commit}"
echo "Previous commit is ${prev_commit}"

# compare the commits, running an action runner if they differ
if [ "${latest_commit}" != "${prev_commit}" ]; then
    echo "Detected new commit, starting runner"
    docker run -d --rm actions-image \
        ${OWNER} \
        ${REPO} \
        ${PAT} \
        $(uuidgen)
fi

# update the previous commit store with the latest commit
echo ${latest_commit} > ${prev}

To test it, create a new commit in the repo you are connection your actions runners to and run this script against that repo, e.g.:

$ ./orc.sh ${PAT} ${OWNER} ${REPO}
Latest commit is 4ceb832f5386c812c0a296968197a1d242f8e8fd
Previous commit is c7e9b463a90a4c5eeab16ba9ccf2afdc1b28153b
Detected new commit, starting runner
68101fbf5c801afaa378ebbb6dc988c5e264fa784bc2c1353025f125f56bbc22

Awesome! To see the container:

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
68101fbf5c80        actions-image       "./entrypoint.sh way…"   26 seconds ago      Up 25 seconds                           adoring_perlman

And the logs:

$ docker logs -f adoring_perlman

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration


√ Runner successfully added
√ Runner connection is good

# Runner settings


√ Settings Saved.


√ Connected to GitHub

2020-02-02 18:38:25Z: Listening for Jobs

An issue with our process

Hmm it doesn't seem to be starting any jobs, and if we look at the repo we see that the check has failed!

The reason for this:

This is a problem. It seems you always need at least one self-hosted runner running, or commits will initially fail. For now we can manually kick off the checks again from the UI with the Re-run checks button, and our runner should start the job.

$ docker logs -f adoring_perlman
...
2020-02-02 18:38:25Z: Listening for Jobs
2020-02-02 18:42:37Z: Running job: build
2020-02-02 18:43:00Z: Job build completed with result: Succeeded

# Runner removal


√ Runner removed successfully
√ Removed .credentials
√ Removed .runner

Given this issue, we need to change tactics!

Changing tactics

From a bit of testing it seems we can get around this issue by configuring a runner that we never actually run. Let's update the entrypoint script to allow registration only:

#!/usr/bin/env bash

OWNER=$1
REPO=$2
PAT=$3
NAME=$4

# if set this script will only run ./config.sh
# it will not run the actions runner
REGISTER_ONLY=$5

cleanup() {
    token=$(curl -s -XPOST -H "authorization: token ${PAT}" \
        https://api.github.com/repos/${OWNER}/${REPO}/actions/runners/registration-token | jq -r .token)
    ./config.sh remove --token $token
}

token=$(curl -s -XPOST \
    -H "authorization: token ${PAT}" \
    https://api.github.com/repos/wayofthepie/gh-app-test/actions/runners/registration-token | jq -r .token)

./config.sh \
    --url https://github.com/${OWNER}/${REPO} \
    --token ${token} \
    --name ${NAME} \
    --work _work

if [ -z ${REGISTER_ONLY} ]; then
    ./run.sh --once
    cleanup
fi

The code up to this point can be found here. Now let's build:

$ docker build -t actions-image .
Sending build context to Docker daemon    105kB
Step 1/9 : FROM ubuntu
 ---> 775349758637
...
Successfully built f82508c0ebb8
Successfully tagged actions-image:latest

And run, but this time run it so that it only configures and does not stay waiting for commits:

$ docker run -ti --rm actions-image ${OWNER} ${REPO} ${PAT} test true

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration


√ Runner successfully added
√ Runner connection is good

# Runner settings


√ Settings Saved.

We are setting the fifth parameter to true here, this will in turn set the ${REGISTER_ONLY} variable in our entrypoint.sh script, meaning ./run.sh will never be called. Now if we go to the UI and Settings -> Actions in our repo, we should see an offline action called test:

Great! Now let's commit to our repo again, and hopefully the check run will be queued and will not have failed:

Awesome! Finally let's spin up a runner to actually run the build:

$ ./orc.sh ${PAT} ${OWNER} ${REPO}
Latest commit is 1fb4c5fd212e8aba177a15b6016c68b7c005b74a
Previous commit is
Detected new commit, starting runner
4ecebf7f0a5fdd75c216d490ef67afa7a9d3e0c3b30848c150b148238eccbd7c

$ docker logs suspicious_ardinghelli -f

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration


√ Runner successfully added
√ Runner connection is good

# Runner settings


√ Settings Saved.


√ Connected to GitHub

2020-02-02 19:14:33Z: Listening for Jobs
2020-02-02 19:14:36Z: Running job: build
2020-02-02 19:14:54Z: Job build completed with result: Succeeded

# Runner removal


√ Runner removed successfully
√ Removed .credentials
√ Removed .runner

Woop! All working again. Now that's working again, let's add some improvements to orc.sh and add a scheduler.

Adding cron

Let's get orc.sh running every minute with cron. Just a small update first so that it writes the prev file in the directory itself is located in:

#!/usr/bin/env bash
##
# Detects new commits in a given repo by checking every minute
# and storing a reference to the last seen commit in a file called
# prev.
##

PAT=$1
OWNER=$2
REPO=$3

# the directory of this script 
cur_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
prev="${cur_dir}/prev"

# make sure we have values for all our arguments
[ -z ${PAT} ] || [ -z ${OWNER} ] || [ -z $REPO ] && {
    echo "Incorrect usage, example: ./orc.sh personal-access-token owner some-repo"
    exit 1
}

# make sure the prev file exists
touch ${prev}

# get the latest commit
latest_commit=$(curl -s -H "authorization: token ${PAT}"  \
    https://api.github.com/repos/${OWNER}/${REPO}/commits |\
    jq -r .[0].sha)

# read the last commit we saw
prev_commit=$(cat ${prev})

echo "Latest commit is ${latest_commit}"
echo "Previous commit is ${prev_commit}"

# compare the commits, running an action runner if they differ
if [ "${latest_commit}" != "${prev_commit}" ]; then
    echo "Detected new commit, starting runner"
    docker run -d --rm actions-image \
        ${OWNER} \
        ${REPO} \
        ${PAT} \
        $(uuidgen)
fi

# update the previous commit store with the latest commit
echo ${latest_commit} > ${prev}

I took the cur_dir var from this post: https://stackoverflow.com/questions/59895/how-to-get-the-source-directory-of-a-bash-script-from-within-the-script-itself.

Latest code up to this point is here.

To add orc.sh to cron, first move it to /var/tmp (we can pick a better location later) and then run cronab -e to edit the crontab for your user:

$ cp orc.sh /var/tmp
$ crontab -e 

# Edit this file to introduce tasks to be run by cron.
#
...
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command
* * * * * /var/tmp/orc.sh ${PAT} ${OWNER} ${REPO}

You can make sure it is running by checking the cron logs, it should run every minute:

$ journalctl -u cron -f
-- Logs begin at Sat 2019-08-10 18:05:10 IST. --
...
Feb 02 19:30:01 sky CRON[31363]: (chaospie) CMD (/var/tmp/orc.sh ${PAT} ${OWNER} ${REPO} )

Now if you commit, a container should be created and run your actions! Note that because orc.sh did not know about any previous commit it will always launch a container on its initial run.

Conclusion

We now have a super simple way of launching actions runners per commit. However the way we are doing this has a lot of of problems:

orc.sh will launch a container when it initially runs, whether there is something to build or not.
We launch a runner per commit, if there are multiple workflows defined in a repo we will only ever build one and the others will lay idle. We can fix this by using information from the checks API to figure out when to launch a runner and how many we may need.
The cron checks every minute for new commits, we can do better than this. For example we could use webhooks so the builds run in realtime.
In my testing I have seen some cases where a runner fails to be cleaned up and sometimes you cannot even remove them from the UI. When this has happened I've had to register a new runner with the same name and then remove that.
Instead of bash + cron it would be better to create a webservice for this, especially if we are going down the webhooks route.
This can currently only run on a single machine and isn't really scalable as is.

In the next post I'll tackle some of these issues. The code up to this point can be found here.

Improving our github actions runner image

wayofthepie — Sun, 02 Feb 2020 16:15:17 +0000

Table of Contents
Undocumented parameters for config.sh
- Diving into the source
Automated token retrieval
- Generating a personal access token
- Automatically generating a registration token
Automated runner removal
Conclusion

In the last post we built a docker image which can launch an actions runner. The image definition up to the end of that post is here.

There are a few issues with the setup in that image.

We have a few hacks in the entrypoint.sh script for configuring the runner - e.g. passing the name via an echo.
Anytime you want to start a new runner you need to get the latest token from the UI.
You need to manually unregister runners which you want to remove.

I'm going to tackle these three issues now. To help, we should dive into the actions runner source code.

Undocumented parameters for config.sh

I was pretty sure the help section for config.sh was missing some flags. All it prints is:

$ ./config.sh  --help

Commands:,
 ./config.sh          Configures the runner
 ./config.sh remove   Unconfigures the runner
 ./run.sh             Runs the runner interactively. Does not require any options.

Options:
 --version  Prints the runner version
 --commit   Prints the runner commit
 --help     Prints the help for each command

To configure a runner we need the following four things at least:

The url to the repo to register the runner against, this has a --url option.
The token this has a --token option.
The name of the runner, we cheat in our entrypoint.sh by echoing a name into the execution of config.sh. This sets the first prompt as the value we echo.
The work directory, here we also cheat with echo and because we echo a single value this defaults to _work.

./config.sh calls out to a .net binary called Runner.Listener, the source code can be found here. What can we find in this?

Diving into the source

Our aim is to find any flags which are not documented in ./config --help that may help us automate configuration. The source code is large enough that blindly searching through would be tedious, so what information do we have that can narrow down our search?

When we run config.sh it defaults the work folder to _work:

$ ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}

...
Enter name of work folder: [press Enter for _work]
...

So, their must be a string in the code with the value _work. If we search in the actions-runner repo, we find it in a few places. The most promising is Runner.Common Constants.cs.

This file contains quite a few interesting constants which may be of use later on! For our current goal, there is the CommandLine Args class.

public static class Args
{
    public static readonly string Auth = "auth";
    public static readonly string MonitorSocketAddress = "monitorsocketaddress";
    public static readonly string Name = "name";
    public static readonly string Pool = "pool";
    public static readonly string StartupType = "startuptype";
    public static readonly string Url = "url";
    public static readonly string UserName = "username";
    public static readonly string WindowsLogonAccount = "windowslogonaccount";
    public static readonly string Work = "work";

    // Secret args. Must be added to the "Secrets" getter as well.
    public static readonly string Token = "token";
    public static readonly string WindowsLogonPassword = "windowslogonpassword";
    public static string[] Secrets => new[]
    {
        Token,
        WindowsLogonPassword,
    };
}

This looks promising. It mentions two args we already use - --token and --url. Let's see if we can set name and the work dir similarly:

$ ./config.sh --url https://github.com/${OWNER}/${REPO} \
    --token ${TOKEN} \
    --name my-runner \
    --work _work

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration


√ Runner successfully added
√ Runner connection is good

# Runner settings


√ Settings Saved.

It works! Now there are no prompts. We can update our docker image's entrypoint.sh from the previous post to take this into account:

#!/usr/bin/env bash

OWNER=$1
REPO=$2
TOKEN=$3
NAME=$4

./config.sh \
    --url https://github.com/${OWNER}/${REPO} \
    --token ${TOKEN} \
    --name ${NAME} \
    --work _work

./run.sh

See here for the full code up to this point including the Dockerfile.

Automated token retrieval

Currently anytime we want to run a new action we need to login to the UI and retrieve a new token. Recently github release an API to generate this token, see the self hosted runners API docs. We can take advantage of this in our entrypoint.sh script.

Generating a personal access token

To generate an actions registration token we first need to generate a Personal Access Token that we can use to call the github API. I'll refer to these as PAT's from here on in so as not to confuse them with the registration token, as they are both tokens of a different type!

To generate a PAT, go to Settings in your account:

Then Developer Settings in the left panel:

Go to Personal Access Tokens:

Finally click the Generate a new token button:

Give your token a name that associates it with what it's doing. I've called it actions-runner-registration. I am not 100% sure the exact scopes that it needs, setting all repo scopes works however so we can do this for now.

Scroll to the bottom and click Generate Token.

Make sure to copy the token on the next page and store it in a safe place! This is the only chance you get to see it, and if you do not you will have to generate a new one.

Automatically generating a registration token

Now we have our PAT, we can use the API to generate an actions runner registration token. The docs for the different methods of auth with the github API can be found here. Let's keep his simple and use curl.

$ curl -XPOST \
    -H "authorization: token ${YOUR_PAT}" \ # ${YOUR_PAT} is the PAT we generated above
    https://api.github.com/repos/${OWNER}/${REPO}/actions/runners/registration-token
{
  "token": "*******",
  "expires_at": "2020-02-02T16:21:21.410+00:00"
}

It works! With this we can update our entrypoint.sh script again.

#!/usr/bin/env bash

OWNER=$1
REPO=$2
PAT=$3
NAME=$4

token=$(curl -s -XPOST \
    -H "authorization: token ${PAT}" \
    https://api.github.com/repos/wayofthepie/gh-app-test/actions/runners/registration-token |\
    jq -r .token)

./config.sh \
    --url https://github.com/${OWNER}/${REPO} \
    --token ${token} \
    --name ${NAME} \
    --work _work

./run.sh

If you have never used the jq command before, jq .token means take the token field out of the given json. The json returned from the curl call has two fields, token and expires_at. If we used jq without the -r flag it would return the token wrapped in quotes, we want it without quotes. So the full command is jq -r .token.

We need to use curl and jq now, neither of these are in our docker image so we need to also update our Dockerfile.

FROM ubuntu

ENV RUNNER_VERSION=2.164.0

RUN useradd -m actions \
    && apt-get update \
    && apt-get install -y \
    wget \
    # add curl and jq
    curl \
    jq

RUN cd /home/actions && mkdir actions-runner && cd actions-runner \
    && wget https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \
    && tar xzf ./actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
WORKDIR /home/actions/actions-runner

RUN chown -R actions ~actions && /home/actions/actions-runner/bin/installdependencies.sh

USER actions

COPY entrypoint.sh .
ENTRYPOINT ["./entrypoint.sh"]

The source for the image up to this point can be found here.

Let's build it and run, see if it works:

$ docker build -t actions-image .
Sending build context to Docker daemon  81.92kB
Step 1/9 : FROM ubuntu
 ---> 775349758637
...
Step 9/9 : ENTRYPOINT ["./entrypoint.sh"]
 ---> Using cache
 ---> 0dff3c9bac89
Successfully built 0dff3c9bac89
Successfully tagged actions-image:latest

$ docker run -ti --rm actions-image ${OWNER} ${REPO} ${PAT} my-runner

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration


A runner exists with the same name
Would you like to replace the existing runner? (Y/N) [press Enter for N]

Ah an issue! We registered a runner called my-runner previously in this post and never removed it. We can manually remove for now and try again:

$ docker run -ti --rm actions-image ${OWNER} ${REPO} ${PAT} my-runner

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration


√ Runner successfully added
√ Runner connection is good

# Runner settings


√ Settings Saved.


√ Connected to GitHub

2020-02-02 15:42:22Z: Listening for Jobs

It works! Now we don't have to generate a token manually each time we want to start a new runner. One major issue remains, the fact you have to manually remove runners.

Automated runner removal

When we kill the container running an actions runner it should automatically unregister itself. To manually unregister we use the ./config remove command. Let's add some small changes to our entrypoint so it will do this automatically.

#!/usr/bin/env bash

OWNER=$1
REPO=$2
PAT=$3
NAME=$4

cleanup() {
    token=$(curl -s -XPOST -H "authorization: token ${PAT}" \
        https://api.github.com/repos/${OWNER}/${REPO}/actions/runners/registration-token |\
        jq -r .token)
    ./config.sh remove --token $token
}

token=$(curl -s -XPOST \
    -H "authorization: token ${PAT}" \
    https://api.github.com/repos/wayofthepie/gh-app-test/actions/runners/registration-token |\
    jq -r .token)

./config.sh \
    --url https://github.com/${OWNER}/${REPO} \
    --token ${token} \
    --name ${NAME} \
    --work _work

./run.sh

cleanup

The cleanup function is run at the end of the script, it gets a new registration token and called ./config.sh remove --token ${TOKEN} to unregister the runner. The code to this point can be found here. Does it all work?

$ docker build -t actions-image .
Sending build context to Docker daemon  88.06kB
Step 1/9 : FROM ubuntu
 ---> 775349758637
...
Successfully built 1de80d7b74d6
Successfully tagged actions-image:latest

$ docker run -ti --rm actions-image ${OWNER} ${REPO} ${PAT} my-runner

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration


√ Runner successfully added
√ Runner connection is good

# Runner settings


√ Settings Saved.


√ Connected to GitHub

2020-02-02 16:05:31Z: Listening for Jobs
^CExiting...

# Runner removal


√ Runner removed successfully
√ Removed .credentials
√ Removed .runner

I used CTRL-C to cancel and it ran the cleanup!

Conclusion

We can now easily register multiple actions runners on the fly and have the automatically clean up when we are finished with them. However, we still need to launch them manually. What if we could launch a runner per commit, have them run a single build, and disappear? That's the dream. We'll star working towards that in the next post.

Ephemeral Self-Hosted Github Actions Runners

wayofthepie — Sat, 01 Feb 2020 22:13:21 +0000

Github actions
What makes an action runner?
What is currently automatable?
- Initial download
- Configuring the runner
- Running the runner
- Quick test in a docker container
Dockerizing a runner
Conclusion

Github actions

Out of the box self-hosted actions runners are static so must be run on their own VM or something like a StatefulSet in kubernetes. This makes them quite costly. Wouldn't it be nice if they spun up per commit, ran a single job, and cleaned up afterwards?

In the next few posts I'm going to attempt to build a system that does this. This first post will run step by step through how I initially dockerized a self-hosted action.

What makes an action runner?

Self-hosted action runners can be registered against a given repo by following the instructions given in the Settings -> Actions menu in a repo. In short, the instructions are:


$ mkdir actions-runner && cd actions-runner
$ curl -O -L https://github.com/actions/runner/releases/download/v2.164.0/actions-runner-linux-x64-2.164.0.tar.gz
$ tar xzf ./actions-runner-linux-x64-2.164.0.tar.gz
$ ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}
$ ./run.sh

√ Connected to GitHub

2020-02-01 21:01:15Z: Listening for Jobs

Note that ${TOKEN} here and in the rest of this post refers to the actions runner registration token which you receive from the UI.

This will start a self-hosted action runner which will listen for new commits and run actions configured for those commits. Both the config.sh and run.sh scripts call out to a .Net core binary which does the real work.

This is great, now you can run actions on your own infrastructure. If you work in a large corporation this is a huge deal, as generally build tools and other resources your builds may need are only accessible from internal networks. However, there are a few issues with how these self-hosted actions run:

If you have many repositories - let's say you have a private github organization - you now have at the very least a VM, kubernetes pod, or some resource which must be constantly running. If your organization has 100+ repos this can be quite costly resource-wise.
It's not straightforward to add new actions runners for new repos that are created.
It's also not straightforward to scale these actions if a repo gets a lot of commits daily.
The scripts which are run in the setup instructions require some manual intervention (at least by default, more on that later) so automating it may be tricky.

The question is, can we fix these issues? Let's start by abstracting the setup, and create a docker image to do so.

What is currently automatable?

Before we begin to create an image for the runner, let's go through the setup steps and see what currently needs manual intervention.

Initial download

The initial download is fully automatable, no manual intervention necessary:

$ cd /var/tmp
$ mkdir actions-runner && cd actions-runner
$ curl -s -O -L https://github.com/actions/runner/releases/download/v2.164.0/actions-runner-linux-x64-2.164.0.tar.gz
$ tar xzf ./actions-runner-linux-x64-2.164.0.tar.gz

We should have a directory with the following contents:

$ ls -la
total 73680
drwxr-xr-x  4 chaospie chaospie     4096 Dec 18 20:41 ./
drwxrwxrwt 12 root     root         4096 Feb  1 20:13 ../
-rw-rw-r--  1 chaospie chaospie 75400617 Feb  1 20:14 actions-runner-linux-x64-2.164.0.tar.gz
drwxr-xr-x  2 chaospie chaospie    16384 Dec 18 20:41 bin/
-rwxr-xr-x  1 chaospie chaospie     2671 Dec 18 20:40 config.sh*
-rwxr-xr-x  1 chaospie chaospie      623 Dec 18 20:40 env.sh*
drwxr-xr-x  4 chaospie chaospie     4096 Dec 18 20:41 externals/
-rwxr-xr-x  1 chaospie chaospie     1666 Dec 18 20:40 run.sh*

Configuring the runner

Let's configure our runner.

$ ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration

Enter the name of runner: [press Enter for sky]

It seems configuration wants manual input. Let's see if anything is mentioned in the help for config.sh:

 ./config.sh --help

Commands:,
 ./config.sh          Configures the runner
 ./config.sh remove   Unconfigures the runner
 ./run.sh             Runs the runner interactively. Does not require any options.

Options:
 --version  Prints the runner version
 --commit   Prints the runner commit
 --help     Prints the help for each command

Nothing. It doesn't look there is a parameter for setting the name of the action runner. We can cheat for now:

$ echo my-runner | ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration

Enter the name of runner: [press Enter for sky]
√ Runner successfully added
√ Runner connection is good

# Runner settings

Enter name of work folder: [press Enter for _work]
√ Settings Saved.

Great, a fully automated config setup! Note this also defaults the second manual input request, the work folder, to _work.

Running the runner

There are no params or manual input requests for run.sh, so this is clearly automatable.

Quick test in a docker container

Let's do a quick test in a docker container:

$ docker run -ti --rm ubuntu:18.04 bash

$ mkdir actions-runner && cd actions-runner

$ curl -O -L https://github.com/actions/runner/releases/download/v2.164.0/actions-runner-linux-x64-2.164.0.tar.gz
bash: curl: command not found

# Ah! There is no curl in the default ubuntu 18.04 image, let's just install it for now.

$ apt update && apt install -y curl
Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
...
done.

$ tar xvf actions-runner-linux-x64-2.164.0.tar.gz
./
./bin/
./bin/System.Security.Cryptography.OpenSsl.dll
./bin/System.Memory.dll
./bin/System.Runtime.Serialization.Json.dll
./bin/System.IO.Compression.FileSystem.dll
...

$ ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}
Must not run with sudo

It seems we can't run config.sh as root, or with sudo! This makes sense, generally you shouldn't run builds with an elevated user. It seems like there will be a bit of work to get this containerized.

Dockerizing a runner

Let's create a simple Dockerfile to fix this:

FROM ubuntu

ENV RUNNER_VERSION=2.164.0

RUN useradd -m actions \
    && apt-get update && apt-get install -y wget

RUN cd /home/actions && mkdir actions-runner && cd actions-runner \
    && wget https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \
    && tar xzf ./actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
WORKDIR /home/actions/actions-runner
USER actions

Build and run:

$ docker build -t actions-image .
Sending build context to Docker daemon  70.66kB
Step 1/9 : FROM ubuntu
 ---> 775349758637
...

$ docker run -ti --rm actions-image

$ ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}
Libicu's dependencies is missing for Dotnet Core 3.0
Execute ./bin/installdependencies.sh to install any missing Dotnet Core 3.0 dependencies.

A new error! Seems we are missing dependencies. Conveniently the actions runner setup gives us a script to set everything up. Let's update our Dockerfile:

FROM ubuntu

ENV RUNNER_VERSION=2.164.0

RUN useradd -m actions \
    && apt-get update && apt-get install -y wget

RUN cd /home/actions && mkdir actions-runner && cd actions-runner \
    && wget https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \
    && tar xzf ./actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz

WORKDIR /home/actions/actions-runner

# Here we install the dependencies needed by the runner
RUN /home/actions/actions-runner/bin/installdependencies.sh

USER actions

Build and run again:

$ docker build -t actions-image .
Sending build context to Docker daemon  70.66kB
Step 1/9 : FROM ubuntu
 ---> 775349758637
...

$ docker run -ti --rm actions-image

$ ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}
touch: cannot touch '.env': Permission denied
./env.sh: line 36: .path: Permission denied
Unhandled exception. System.UnauthorizedAccessException: Access to the path '/home/actions/actions-runner/_diag' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.FileSystem.CreateDirectory(String fullPath)
   at System.IO.Directory.CreateDirectory(String path)
   at GitHub.Runner.Common.HostTraceListener..ctor(String logFileDirectory, String logFilePrefix, Int32 pageSizeLimit, Int32 retentionDays)
   at GitHub.Runner.Common.HostContext..ctor(String hostType, String logFile)
   at GitHub.Runner.Listener.Program.Main(String[] args)
./config.sh: line 79:    44 Aborted                 (core dumped) ./bin/Runner.Listener configure "$@"

Ah, more errors! This time permissions issues. If we check the permissions on the files in that directory we can see the owner is wrong. The owner of all these files should be the actions user:

$ ls -la
total 73676
drwxr-xr-x 4    1001     115     4096 Dec 18 20:41 ./
drwxr-xr-x 1 actions actions     4096 Feb  1 20:43 ../
-rw-r--r-- 1 root    root    75400617 Dec 18 20:44 actions-runner-linux-x64-2.164.0.tar.gz
drwxr-xr-x 2    1001     115    16384 Dec 18 20:41 bin/
-rwxr-xr-x 1    1001     115     2671 Dec 18 20:40 config.sh*
-rwxr-xr-x 1    1001     115      623 Dec 18 20:40 env.sh*
drwxr-xr-x 4    1001     115     4096 Dec 18 20:41 externals/
-rwxr-xr-x 1    1001     115     1666 Dec 18 20:40 run.sh*

Another Dockerfile update:

FROM ubuntu

ENV RUNNER_VERSION=2.164.0

RUN useradd -m actions \
    && apt-get update && apt-get install -y wget

RUN cd /home/actions && mkdir actions-runner && cd actions-runner \
    && wget https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \
    && tar xzf ./actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
WORKDIR /home/actions/actions-runner

# Here we change owner to user actions on the actions user's home directory
RUN chown -R actions ~actions && /home/actions/actions-runner/bin/installdependencies.sh

USER actions

And finally run again:

$ docker build -t actions-image .
Sending build context to Docker daemon  70.66kB
Step 1/9 : FROM ubuntu
 ---> 775349758637
...

$ docker run -ti --rm actions-image

$ ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration

Enter the name of runner: [press Enter for 0ccb204b3990] my-runner

√ Runner successfully added
√ Runner connection is good

# Runner settings

Enter name of work folder: [press Enter for _work]

√ Settings Saved.

$ ./run.sh

√ Connected to GitHub

2020-02-01 21:22:24Z: Listening for Jobs

Great! Now let's make it generic for any repository. To do that, let's create an entrypoint.sh as follows:

#!/usr/bin/env bash

OWNER=$1
REPO=$2
TOKEN=$3
NAME=$4

echo ${NAME} | ./config.sh --url https://github.com/${OWNER}/${REPO} --token ${TOKEN}

./run.sh

And update our Dockerfile:

FROM ubuntu

ENV RUNNER_VERSION=2.164.0

RUN useradd -m actions \
    && apt-get update && apt-get install -y wget

RUN cd /home/actions && mkdir actions-runner && cd actions-runner \
    && wget https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \
    && tar xzf ./actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz
WORKDIR /home/actions/actions-runner

RUN chown -R actions ~actions && /home/actions/actions-runner/bin/installdependencies.sh

USER actions

# Add the script and make it the entrypoint
COPY entrypoint.sh .
ENTRYPOINT ["./entrypoint.sh"]

Let's build and test it:


$ docker run -ti --rm actions-image
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"./entrypoint.sh\": permission denied": unknown.

Woops! I forgot to make the script executable. Make the script executable to fix:

$ chmod +x entrypoint.sh

$ docker build -t actions-image .
Sending build context to Docker daemon  70.66kB
Step 1/9 : FROM ubuntu
 ---> 775349758637
...
Step 9/9 : ENTRYPOINT ["./entrypoint.sh"]
 ---> Using cache
 ---> a6535399773a
Successfully built a6535399773a
Successfully tagged actions-image:latest

$ docker run -ti --rm actions-image ${OWNER} ${REPO} ${TOKEN} my-runner

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication


√ Connected to GitHub

# Runner Registration

Enter the name of runner: [press Enter for a05fbb6da3db]
√ Runner successfully added
√ Runner connection is good

# Runner settings

Enter name of work folder: [press Enter for _work]
√ Settings Saved.


√ Connected to GitHub

2020-02-01 21:28:51Z: Listening for Jobs

Great, it all works as expected! There are a few problems however:

You need to manually run a container for each runner you wish to start.
Each runner you start needs to be manually cleaned up.
Any runner that is started can take any job for a new commit if it's idle, so cleaning up runners has the potential to accidentally kill builds that are in progress.
The registration token is a pain to retrieve. Recently github released an API to generate this token. In the next post we will automate the retrieval of this token.
In the current state runners will not unregister, meaning you must do this from the UI.

The code up to this point can be seen here.

Conclusion

In this post we have created a very simple docker image for automating self-hosted action runner registration. In the next post we will improve this by looking into the actions runner source code which contains many hidden parameters that will allow us to automate a lot more than we have so far.

JVM Internals: Memory Overview

wayofthepie — Wed, 22 Jan 2020 03:30:47 +0000

Introduction

This was originally posted on my own blog a few years back, but I'd rather continue the series here, so, re-posting.

What is this post about?

The goal of this post is to give an overview of the heap and non-heap memory regions of the JVM - with some small introduction to both - and also to show what happens in the event of a heap/non-heap memory issue within a docker container. I assume some
basic knowledge of Java, the JVM, docker and linux. You will need docker and openjdk 8 installed on a linux system (I used ubuntu 16.04 to write this post).

Containerizing a java app

To start I'm going to keep things super simple. Let's build a program which prints "Hello world!" and waits forever:

// HelloWorld.java
public class HelloWorld {
    public static void main(String[] args) throws Exception {
        System.out.println("Hello world!");
        System.in.read();
    }
}

Now, a simple Dockerfile:

FROM openjdk:8-jdk
ADD HelloWorld.java .
RUN javac HelloWorld.java
ENTRYPOINT java HelloWorld

With that we can build and launch our application in a container:

$ docker build --tag jvm-test . 
$ docker run -ti --rm --name hello-jvm jvm-test
Hello world!

You can use CTRL-C to kill the container when you are done. Right, now we have a simple program running, what can we do? Let's analyze the JVM.

Basic JVM analysis

Lets get a list what objects we have on the heap within our application. First, get into the container (assuming it's still running from above) and get the JVM processes PID.

$ docker exec -ti hello-jvm bash
root@5f20ae043968:/ $ ps aux|grep [j]ava
root         1  0.1  0.0   4292   708 pts/0    Ss+  12:27   0:00 /bin/sh -c java HelloWorld
root         7  0.2  0.1 6877428 23756 pts/0   Sl+  12:27   0:00 java HelloWorld

From the above, we see the PID is 7. For analysis, the openjdk comes with a number of tools. jmap is one such tool which allows us to view heap information about a JVM process. To get a list of objects, their number of instances and the space they take up in the heap you can use jmap -histo <JVM_PID>.

root@5f20ae043968:/ $ jmap -histo 7

 num     #instances         #bytes  class name
----------------------------------------------
   1:           422        2256744  [I
   2:          1600         141520  [C
   3:           364          58560  [B
   4:           470          53544  java.lang.Class
   5:          1204          28896  java.lang.String
   6:           551          28152  [Ljava.lang.Object;
   7:           110           7920  java.lang.reflect.Field
   8:           258           4128  java.lang.Integer
   9:            97           3880  java.lang.ref.SoftReference
  10:           111           3552  java.util.Hashtable$Entry
  11:           133           3192  java.lang.StringBuilder
  12:             8           3008  java.lang.Thread
  13:            75           2400  java.io.File
  14:            54           2080  [Ljava.lang.String;
  15:            38           1824  sun.util.locale.LocaleObjectCache$CacheEntry
  16:            12           1760  [Ljava.util.Hashtable$Entry;
  17:            55           1760  java.util.concurrent.ConcurrentHashMap$Node
  18:            27           1728  java.net.URL
  19:            20           1600  [S
  ...
  222:             1             16  sun.reflect.ReflectionFactory
Total          6583        2642792

As you can see above there are 6583 instances of a mixture of 222 different classes, taking up over 2.6MB of the heap, for our simple HelloWorld program! When I first saw this it raised a lot of questions - what is [I, why is there a java.lang.String and a [Ljava.lang.String?

What are all these classes?

The single letter class names you see above are all documented under Class.getName().

Encoding	Element Type
Z	boolean
B	byte
C	char
LclassName	class/interface
D	double
F	float
I	int
J	long
S	short

If you look back to the jmap output, the first few instances all have [ prefixing them - e.g. [I. [ denotes a 1 dimensional array of the type proceeding it - [I denotes an array of int e.g. new int[3]. [[I denotes a 2D array, new int[2][3] and so on. Also in the jmap output above were instances of [L.java.lang.String which is just an array of String's - new String[3].

To see this for yourself:

// InstanceName.java
public class InstanceName {
    public static void main(String[] args) throws Exception {
      int[] is = new int[3];
      System.out.println(is.getClass().getName());

      boolean[][][] bs = new boolean[2][5][4];
      System.out.println(bs.getClass().getName());

      String[] ss = new String[3];
      System.out.println(ss.getClass().getName());
    } 
}

Compiling and running this we get:

$ javac InstanceName.java 
$ java InstanceName 
[I
[[[Z
[Ljava.lang.String;

That's a quick overview of one way to look at what's loaded on the heap. I mentioned other memory regions in the JVM earlier, what are these?

Heap and Non-Heap memory

The JVM can be divided into many different memory segments (segments/regions/areas, I'll use these words interchangeably, but generally they mean the same thing),
if we take a high level view first we have two segments - memory used for objects on the heap and non-heap memory.

If we zoom in, the heap has different areas within which we can talk about, depending on what we want to discuss - there is the Eden space, where most new objects are initially created, the Survivor space, where objects go if they survive an Eden space garbage collection (GC) and the Old Generation which contains objects that have lived in Survivor space for a while. Specifically, it contains objects that have been initialized - e.g. List<String> s = new ArrayList<String>(); will create an ArrayList object on the heap, and s will point to this.

In the previous section I ran through what objects are loaded into the heap for our HelloWorld program, so what about non-heap memory?

Non-Heap Memory

If you have ever written a non-trivial java application with jdk8 you have probably heard of Metaspace. This is an example of non-heap memory. It's where the JVM will store class definitions, static variables, methods, classloaders and other metadata. But there are many other non-heap memory regions the JVM will use. Let's list them!

To do so, first we need to enable native memory tracking in our java app:

FROM openjdk:8-jdk
ADD HelloWorld.java .
RUN cat HelloWorld.java
RUN javac HelloWorld.java
ENTRYPOINT java -XX:NativeMemoryTracking=detail HelloWorld

Now build and re-run:

$ docker build --tag jvm-test . 
$ docker run -ti --rm --name hello-jvm jvm-test
Hello world!

In another terminal, exec into the container and get a summary of overall memory usage with jcmd's VM.native_memory command:

$ docker exec --privileged -ti hello-jvm bash
root@aa5ae77e1305:/ $ jcmd 
33 sun.tools.jcmd.JCmd
7 HelloWorld

root@aa5ae77e1305:/ $ jcmd 7 VM.native_memory summary
7:

Native Memory Tracking:

Total: reserved=5576143KB, committed=1117747KB
-                 Java Heap (reserved=4069376KB, committed=920064KB)
                            (mmap: reserved=4069376KB, committed=920064KB) 

-                     Class (reserved=1066121KB, committed=14217KB)
                            (classes #405)
                            (malloc=9353KB #178) 
                            (mmap: reserved=1056768KB, committed=4864KB) 

-                    Thread (reserved=20646KB, committed=20646KB)
                            (thread #21)
                            (stack: reserved=20560KB, committed=20560KB)
                            (malloc=62KB #110) 
                            (arena=23KB #40)

-                      Code (reserved=249655KB, committed=2591KB)
                            (malloc=55KB #346) 
                            (mmap: reserved=249600KB, committed=2536KB) 

-                        GC (reserved=159063KB, committed=148947KB)
                            (malloc=10383KB #129) 
                            (mmap: reserved=148680KB, committed=138564KB) 

-                  Compiler (reserved=134KB, committed=134KB)
                            (malloc=3KB #37) 
                            (arena=131KB #3)

-                  Internal (reserved=9455KB, committed=9455KB)
                            (malloc=9423KB #1417) 
                            (mmap: reserved=32KB, committed=32KB) 

-                    Symbol (reserved=1358KB, committed=1358KB)
                            (malloc=902KB #85) 
                            (arena=456KB #1)

-    Native Memory Tracking (reserved=161KB, committed=161KB)
                            (malloc=99KB #1559) 
                            (tracking overhead=61KB)

-               Arena Chunk (reserved=175KB, committed=175KB)
                            (malloc=175KB)

A lot more regions than just the heap! Our hello world program just got even more complex...

What does all this mean? ¹

Java Heap : heap memory.
Class : is the Metaspace region we previously spoke about.
Thread : is the space taken up by threads on this JVM's.
Code : is the code cache - this is used by the JIT to cache compiled code.
GC : space used by the garbage collector.
Compiler : space used by the JIT when generating code.
Symbols : this is for symbols, by which I believe field names, method signatures fall under. ²
Native Memory Tracking : memory used by the native memory tracker itself.
Arena Chunk : not entirely sure what this gets used for. ³

Practical memory issues

Ok, so why should you care about any of the above? Let's create an app that eats a tonne of memory.

// MemEater.java
import java.util.Vector;

public class MemEater {
    public static final void main(String[] args) throws Exception {
        Vector<byte[]> v = new Vector<byte[]>();
        for (int i = 0; i < 400; i++) {
            byte[] b = new byte[1048576]; // allocate 1 MiB
            v.add(b);
        }
        System.out.println(v.size());
        Thread.sleep(10000);
    }
}

This will create a Vector which contains 400 byte arrays of size 1 MiB ⁴, so this will use ~400MiB memory on the heap. It will then sleep for 10 seconds so we can get the memory usage easily while it runs. Let's constrain the heap to 450MiB and run this locally we can see the actual memory usage of the process. RSS Resident Set Size ⁵ is how this is measured, note that this value also contains pages mapped from shared memory, but we can gloss over that for this post.

So, lets compile our app, run in the background and get its RSS:

$ javac MemEater.java 
$ nohup java -Xms450M -Xmx450M MemEater & 
$ ps aux | awk 'NR==1; /[M]emEater/' 
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
chaospie 18019 10.5  3.0 3138368 494448 pts/19 Sl   16:06   0:00 java -Xms450M -Xmx450M MemEater

In total, the JVM process needs about 500 MiB to run (RSS is 494448 KiB). What happens if we set the heap to a size lower than it needs?

$ java -Xms400M -Xmx400M MemEater
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at MemEater.main(MemEater.java:7)

If you have used java (or any JVM language) before, you have more than likely come across this. It means that the JVM ran out of heap space to allocate objects. There are quite a few other types of OutOfMemoryError the JVM can throw in certain situations ⁶, but I won't go into more detail right now.

Now we know what happens if the JVM does not have enough heap space, what about the case where you are running in a container and hit the overall memory limit for that container?

The simplest way to reproduce this is to package up our MemEater program into a docker image and run it with less memory than it needs.

FROM openjdk:8-jdk
ADD MemEater.java .
RUN cat MemEater.java
RUN javac MemEater.java
ENTRYPOINT java -Xms450M -Xmx450M MemEater

Again, we need to build the image. However this time when we are running we limit the memory the container is allowed to use to 5M:

$ docker build --tag jvm-test .
$ docker run -ti --rm --memory 5M --memory-swappiness 0 --name memeater jvm-test
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Killed

After a few seconds you should see the output above, Killed. What happened? Before we dive into that, lets have a look at the --memory and --memory-swappiness flags used by docker.

Limiting memory with docker

Lets digress for a second, and look at the two docker flags I used above for controlling memory settings ⁷. First, for these flags to work, your kernel will need to have cgroup support enabled and the following boot parameters set (assuming grub):

$ cat /etc/default/grub
...
GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"
...

--memory sets an upper bound on the sum of all processes memory usage within a container, the smallest this can go is 4MiB, above we set it to 5m which is 5MiB. When this is set, the containers cgroup memory.limit_in_bytes is set to the value. I can't find the code that does this in docker, however we can see it as follows:

$ docker run -d --rm --memory 500M --memory-swappiness 0 --name memeater jvm-test 
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
812dbc3417eacdaf221c2f0c93ceab41f7626dca17f959298a5700358f931897
$ CONTAINER_ID=`docker ps --no-trunc | awk '{if (NR!=1) print $1}'`
$ echo $CONTAINER_ID
812dbc3417eacdaf221c2f0c93ceab41f7626dca17f959298a5700358f931897
$ cat /sys/fs/cgroup/memory/docker/${CONTAINER_ID}/memory.swappiness 
0
$ cat /sys/fs/cgroup/memory/docker/${CONTAINER_ID}/memory.limit_in_bytes
524288000

# Again, this time without limits to see the difference
$ docker run -d --rm --name memeater jvm-test 
d3e25423814ee1d79759aa87a83d416d63bdb316a305e390c2b8b98777484822
$ CONTAINER_ID=`docker ps --no-trunc | awk '{if (NR!=1) print $1}'`
$ echo $CONTAINER_ID
d3e25423814ee1d79759aa87a83d416d63bdb316a305e390c2b8b98777484822
$ cat /sys/fs/cgroup/memory/docker/${CONTAINER_ID}/memory.swappiness 
60
$ cat /sys/fs/cgroup/memory/docker/${CONTAINER_ID}/memory.limit_in_bytes
9223372036854771712

Note the WARNING, I'm not entirely sure why this appears as swap support is enabled, and seems to work. You can ignore this for now.

--memory-swappiness sets the swappiness level of the cgroup herarchy the container runs in. This maps directly to the cgroup setting memory.swappiness (at least in version 17.12 of docker ⁸ ) as seen above. Setting this to 0 disables swap for the container.

What kills the container?

So, why was the container killed? Lets run it again:

$ docker run -ti --rm --memory 5M --memory-swappiness 0 --name memeater jvm-test
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Killed

To see the cause of this kill, run journalctl -k and search for oom-killer, you should see logs like the following:

$ journalctl -k
...
Feb 18 17:34:47  kernel: java invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null),  order=0, oom_score_adj=0
Feb 18 17:34:47  kernel: java cpuset=35f18c48d432510c76e76f2e7a962e64a1372de1dc4abd830417263907bea6e0 mems_allowed=0
Feb 18 17:34:47  kernel: CPU: 0 PID: 16432 Comm: java Tainted: G           OE   4.13.0-32-generic #35~16.04.1-Ubuntu
Feb 18 17:34:47  kernel: Hardware name: Dell Inc. Precision 5520/0R6JFH, BIOS 1.3.3 05/08/2017
Feb 18 17:34:47  kernel: Call Trace:
Feb 18 17:34:47  kernel:  dump_stack+0x63/0x8b
Feb 18 17:34:47  kernel:  dump_header+0x97/0x225
Feb 18 17:34:47  kernel:  ? mem_cgroup_scan_tasks+0xc4/0xf0
Feb 18 17:34:47  kernel:  oom_kill_process+0x219/0x420
Feb 18 17:34:47  kernel:  out_of_memory+0x11d/0x4b0
Feb 18 17:34:47  kernel:  mem_cgroup_out_of_memory+0x4b/0x80
Feb 18 17:34:47  kernel:  mem_cgroup_oom_synchronize+0x325/0x340
Feb 18 17:34:47  kernel:  ? get_mem_cgroup_from_mm+0xa0/0xa0
Feb 18 17:34:47  kernel:  pagefault_out_of_memory+0x36/0x7b
Feb 18 17:34:47  kernel:  mm_fault_error+0x8f/0x190
Feb 18 17:34:47  kernel:  ? handle_mm_fault+0xcc/0x1c0
Feb 18 17:34:47  kernel:  __do_page_fault+0x4c3/0x4f0
Feb 18 17:34:47  kernel:  do_page_fault+0x22/0x30
Feb 18 17:34:47  kernel:  ? page_fault+0x36/0x60
Feb 18 17:34:47  kernel:  page_fault+0x4c/0x60
Feb 18 17:34:47  kernel: RIP: 0033:0x7fdeafb0fe2f
Feb 18 17:34:47  kernel: RSP: 002b:00007fdeb0e1db80 EFLAGS: 00010206
Feb 18 17:34:47  kernel: RAX: 000000000001dff0 RBX: 00007fdea802d490 RCX: 00007fdeac17b010
Feb 18 17:34:47  kernel: RDX: 0000000000003bff RSI: 0000000000075368 RDI: 00007fdeac17b010
Feb 18 17:34:47  kernel: RBP: 00007fdeb0e1dc20 R08: 0000000000000000 R09: 0000000000000000
Feb 18 17:34:47  kernel: R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000000000
Feb 18 17:34:47  kernel: R13: 00007fdeb0e1db90 R14: 00007fdeafff851b R15: 0000000000075368
Feb 18 17:34:47  kernel: Task in /docker/35f18c48d432510c76e76f2e7a962e64a1372de1dc4abd830417263907bea6e0 killed as a result of limit of /docker/35f18c48d432510c76e76f2e7a962e64a137
Feb 18 17:34:47  kernel: memory: usage 5120kB, limit 5120kB, failcnt 69
Feb 18 17:34:47  kernel: memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Feb 18 17:34:47  kernel: kmem: usage 1560kB, limit 9007199254740988kB, failcnt 0
Feb 18 17:34:47  kernel: Memory cgroup stats for /docker/35f18c48d432510c76e76f2e7a962e64a1372de1dc4abd830417263907bea6e0: cache:176KB rss:3384KB rss_huge:0KB shmem:144KB mapped_fil
Feb 18 17:34:47  kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Feb 18 17:34:47  kernel: [16360]     0 16360     1073      178       8       3        0             0 sh
Feb 18 17:34:47  kernel: [16426]     0 16426   609544     3160      47       4        0             0 java
Feb 18 17:34:47  kernel: Memory cgroup out of memory: Kill process 16426 (java) score 2508 or sacrifice child
Feb 18 17:34:47  kernel: Killed process 16426 (java) total-vm:2438176kB, anon-rss:3200kB, file-rss:9440kB, shmem-rss:0kB
...

The kernels OOM killer killed the application because it violated it's cgroup memory limit. From the logs above: memory: usage 5120kB, limit 5120kB, failcnt 69 shows it hit the limit, Killed process 16426 (java) total-vm:2438176kB, anon-rss:3200kB, file-rss:9440kB, shmem-rss:0kB shows that it decided to kill process 16426 which was our java process. There is a lot more information in the logs which can help identify the reason why the OOM killer killed your process, however in our case we know why - we violated the container memory limit.

With a heap issue, if we hit an out of memory error with Java Heap Space as the cause, we know immediately that the cause is the heap and we are either allocating too much, or we need to increase the heap (actually identifying the underlying cause of this overallocation in the code is another issue...). When the OOM killer kills our process, it's not so straightforward - it could be direct buffers, unconstrained non-heap memory areas (Metaspace, Code cache etc...) or even another process within the container. There is quite a bit to cover when investigating. On that note, I'll finish this post.

Conclusion

There is quite a lot more that could be said about heap/non-heap memory in the JVM, docker and the oom-killer - but I want to keep this initial post short, it's just meant to be a basic intro to JVM memory usage. Hopefully, if you took anything away from this post, it's that there is much more to think about than just the heap when using the JVM, especially in memory bound containers.

See NMT details. ↩
This one I need to look up more in-depth, as I have not been able to find solid information on it. ↩
Arena Chunk seems to be related to malloc arenas, will definitely look into this in-depth. ↩
1 MiB = 1024 KiB = 1048576 bytes. Why use MiB? Because MB is ambiguous and can mean 1000 KB or 1024 KB, whereas MiB is always 1024 KiB. ↩
See this great answer for a description of RSS. ↩
A detailed description of them can be found here. ↩
The docker documentation on this subject is excellent - see resource constraints. ↩
See docker memory swappiness. ↩

Forem: wayofthepie

Decode a certificate

Introduction

Table of Contents

Some missing things

Read a certificate

x509-parser API

Validate the file is a certificate

Error handling

Refactor

Parse the der encoded cert

Print the certificate

Tiny refactor

Conclusion

Three bytes to an integer

The problem

A solution

A better solution

Morning silence and broken drivers

Silence

Still silence

Silence and broken graphics

Conclusion

Rust and GitHub Actions

Introduction

Table of Contents

Build and test

Clippy lint

Block PR's on failing checks

Clippy deny lints

Protected branch rule

Clean up and merge

Security audit

Conclusion

A Rust CLI for decoding certs

Introduction

Table of Contents

What are we building?

Some constraints to simplify the cli

Test list

Validate args

A note on this workflow

Add an error message

Check argument is a path that exists

Factor out IO

Check argument is a file

Another small refactor

In the next post

Structure of an SSL (X.509) certificate

Introduction

Information in a certificate

Side note on the openssl command

A breakdown of the main fields

Certificate

Signature and Signature Algorithm

TBSCertificate

Version

Serial Number

Signature Algorithm

This field is duplicated, why?

Issuer

Validity

Certificate validity and the Brazilian government

Subject and Subject Public Key Info

Conclusion

Improving Our GitHub Actions Runner Orchestrator

Table of Contents

Github checks api

Making use of this new information

Using kubernetes jobs to schedule actions runners

Storing our token properly

Running our script as a kubernetes cronjob

Assigning the correct permissions

Fixing a bug in our logic

Conclusion

Hacking Together A GitHub Actions Runner Orchestrator

Table of Contents

Run a single build and cleanup

A simple orchestrator

Detecting new commits