Forem: k-mack

Techniques for Writing Docs in Markup Languages

k-mack — Fri, 24 Dec 2021 18:18:33 +0000

Software projects need to have good documentation. This makes the software more approachable and impacts its users and contributors. More importantly, it increases the software's signal-to-noise ratio, allowing developers to better understand if the software meets their needs. This is not revolutionary: high-level software developed today (i.e., using the C programming language or above) presumably uses third-party code, open-source or not. Writing software documentation, however, is challenging and thoughtful work. This is the reason why technology companies hire technical writers and developer advocates: they know good documentation is critical for their product.

Markup languages, such as Markdown and AsciiDoc, have become essential in developing software documentation. Their similarity to programming, with toolchains and a lightweight syntax, aligns with the developer mindset more so than traditional word processors. Documents written in them have a low barrier to read (e.g., you need only a text viewer installed, like more) and easy to read diffs, making their history easy to track using a version control system.

Like software source code, markup languages allow authors to write the same content a thousand different ways. With this type of flexibility and creativity in the documentation process, it helps to employ techniques to make writing documentation as enjoyable as writing code and to make the documentation source files as aesthetically pleasing, consistent, and efficient as well-styled source code. This post shares four such techniques that have improved my writing experience.

But first, a short overview of my journey with writing technical documentation :).

Personal Context

When I joined the company I am at now in 2012, Microsoft Word was primarily used throughout the company for authoring software documentation. It was not used for software library API documentation; tools like Doxygen and Javadoc were used for that. Word was for creating user and administrator guides, tutorials, frequently asked questions (FAQs), etc. The primary reason for using Word was that customers expected documentation as Word documents, and the customers were application users, not developers.

This was back when Subversion was the dominate version control system in the company, and the slack to invest in tooling around documentation generation and transformation didn't exist. Combining these with some other old-school mentalities meant that managing documentation was a bit painful: The latest and stable version of the document was maintained by one person. If a person was to modify the document, the person had to obtain a copy of the latest version, rename the file such that its name was suffixed with the author's initials (e.g., User_Guide-km.docx), turn on Word's Track Changes feature, make the modifications, and send the modified file to the maintainer. The maintainer was responsible for merging the edits of all the contributors into the document.
The document was then sent to all contributors for review to ensure no edit was lost. If an error was found in the document during review, then the workflow's recursion kicked in until everyone was satisfied the document.

This was a bit of gut punched to me. I joined the company after finishing graduate school where I wrote my reports and thesis in LaTeX. I was accustomed to treating documentation like code: use a toolchain to transform source files to an output format, automate document generation (e.g., with GNU Make), and commit changes to version control. My brain was wired to think about content first and its presentation second (e.g., *.cpp -> *.o -> {libmylib.a, libmylib.so}).

After the company made the move to Git and when Markdown became a hit with the staff, teams started to use markup languages to write software documentation. Contributors were branching and merging edits to Markdown files in Git. Each change to the Markdown files could be easily seen via git diff and git show. It was contributor-friendly, and the world seemed right and just :P. The last step was to convert the final, peer-reviewed Markdown content into a Word document. Tools like pandoc were found to be helpful for this. As time went on, things kept getting better: customers became more open to PDF and HTML, teams moved from Markdown to Asciidoctor to create richer documents, and the company-hosted GitLab instance made reviewing changes a delight.

The world, however, isn't black and white, and as is usual for software developers, technique and style became a topic of discussion. One of the nice things about Word is that it puts all authors into the same frame of mind. The structure of Word is uniform and universal. It matches how we are taught to write from an early age: pull out a blank piece of paper and start writing from left to right, top to bottom. Made a mistake? Erase, use Wite-Out, or start over. There is no commenting out sentences or variable substitution. What you write is what you get, and Word follows this. At least to my knowledge, you can't separate content from presentation in Word: Sentences are delimited by a period and a space. Paragraphs are delimited by a newline. Ordered and unordered lists use a default icon and scheme, set by the person who created the Word document. Looking back, this rigid word processing environment eased the contribution and merging process in the sense that, for example, you didn't have Bob breaking his sections into separate Word documents while Nancy confined her contributions to one document but wrote each sentence on a separate line. Transitioning from Word to markup introduced "developer chaos" in managing documentation contributions. This chaos mirrors that which is experienced with coding best practices and styles.

After a bit of back in forth about how to manage the flexibility of markup languages, I found some core techniques that helped me write good and maintainable documentation.

Techniques for Writing Docs

In 2015, I found and watched Dan Allen's presentation at Devnexus titled "Discover the Zen of Writing with Asciidoctor." The full presentation is great whether you use Asciidoctor or not. However, the gold for authors is his section on Zen techniques.

Full disclosure: The techniques below are from Dan Allen's 2015 Devnexus presentation (see links above). Full credit goes to him. I am merely echoing them in an attempt to share them and attest to how awesome they are.

After giving credit to where credit is due, let's dive into some of the techniques that Dan shares in his presentation. Note that this post could be titled something like "4 Writing Techniques Every Programmer Should Know," but it sounds like clickbait to me. Then again, I am just generally not good with creating titles.

1. Like Code, Don't Repeat Yourself

Since writing technical documents in a markup language is like writing code, don’t repeat yourself. Just like copy-and-pasting code throughout an application can result in inconsistencies, duplicating the same text throughout your documents can leave them fragmented when one section is modified but the others are not.

There are tools that allow authors to import documents into other documents. This makes it easy to pull out licenses, introductions, appendices, images, etc. into their own documents and then import them into others. Changes in these then fan out to all the other documents.

2. Like Code, Write a Statement Per Line

Most statements in code are given their own line. Apply this same principle to your documents: every sentence or significant clause (or semi-colon, colon, or list) starts a new line. Most documentation formats require a blank line to introduce a paragraph. Therefore, contiguous statements per line will be rendered as a single paragraph. This is a powerful writing technique for a number of reasons:

It taps into your programming mindset and aligns with the technical, structured workflow that you know and use everywhere else.
If a change occurs, it is localized to that line (i.e., no wrapping) which means the diff is incredibly easy to read.
Long sentences run over 80 columns, which means you’re probably ranting or meandering. Technical documents are not novels; they should be concise and to the point. Of course, some sentences are going to be over 80 characters, but if you’re pushing 100 characters, consider revising. In other words, be judicious. (A thesaurus can help you say more with less.)
Sentences can be easily moved around.
Sentences can be easily commented out (if your documentation syntax supports comments).

3. Like Code, Write Comments

Commenting is powerful in code and just as powerful in documents. Comments in your document allow you to save off chunks of text without being rendered in your generated document. This is important for keeping around thoughts and objectives for paragraphs, sections, etc. This helps the other authors and editors understand what you are trying to accomplish.

Not all markup formats support comments. If the one you use supports them, then don’t be afraid to use them.

4. Like Code, Use Variables

A number of documentation formats support variables that can encapsulate information used repeatedly throughout a document. This is also a way to shorten sentences to fit on a single line and to keep from repeating yourself. I like to use Asciidoctor's attributes to store the long and abbreviated names of the application I am writing about:

// attributes
:app-abbr: App
:app-name: Long Application Name Here
:app-uri-downloads: https://sourceforge.net/
:app-uri-downloads-link: {app-uri-downloads}[Downloads]

== Introduction

{app-name} ({app-abbr}) is an application that *blah blah blah*.

// ...

== Getting Started

Get started with {app-abbr} by downloading a pre-built binary from {app-uri-downloads-link}.

// ...

== FAQs

[quote]
Where can I download {app-abbr}?

Pre-built binaries can be downloaded from {app-uri-downloads-link}.

// ...

Conclusion

The goal of this post was to share techniques to help make writing good software documentation enjoyable with a markup language. The four techniques shared were credited to Dan Allen of the Asciidoctor project.

Fin.

Groovy's @CompileStatic and Methods with the Same Name

k-mack — Sat, 06 Nov 2021 23:22:29 +0000

Application programming interfaces (APIs) can get whacky, but compiled languages help users to get things semantically correct. And dynamic languages? Their ergonomic "dynamic sauce" ladled over a codebase can sometimes be less than helpful. This post is about how Groovy's @CompileStatic can help to demystify what is happening at the call site of a method that has the same name and descriptor as another.

Part I: The Java API

Let's start with what we know we cannot do in Java: No two methods in one class file may have the same name and descriptor.¹ The Java compiler simply does not let us do the following:

public class MyClass {
  public String something() {}
  public static String something() {}
}

When the above class is compiled, an error message complaining that method static void something() is already defined.

$ echo "public class MyClass{\n public String something(){}\n public static String something(){}\n}" > /tmp/MyClass.java          
$ "$JDK11_HOME"/bin/javac /tmp/MyClass.java
/tmp/MyClass.java:3: error: method something() is already defined in class MyClass
  public static String something() {}
                     ^
1 error

Java 8 updated the language to permit interfaces to have static methods, so we can update our API such that it has a concrete class with an instance and a static method that use the same name and descriptor. I would not say this is a common thing to see in an API, but I have seen it. In fact, this post is based on my experience with using one :).

Let's refactor the above class to use interfaces to give the impression that it has two methods with the same name and descriptor. One interface will define String something(); another interface will define static String something(); and a concrete class will implement the two interfaces. The concrete class will compile because it actually no longer has static String something() as part of its implementation. When calling the static method, it must be through the interface, not the concrete class, because the static method is part of the interface. The code for these three files is below.

// InterfaceWithSomething.java
public interface InterfaceWithSomething {
  String something();
}

// InterfaceWithStaticSomething.java
public interface InterfaceWithStaticSomething {
  static String something() {
    return "Interface's static method";
  }
}

// Implementation.java
public class Implementation implements InterfaceWithSomething, InterfaceWithStaticSomething {
  @Override
  public String something() {
    return "Implementation's instance method";
  }
}

We will use the above classes as our Java API. Let's get groovy.

Part II: The Groovy App

Since the JVM is our development platform, we can mix JVM languages. Let's use our Java API in some Groovy code.

// GroovyCode.groovy
class GroovyCode {
  static void main(String[] args) {
    def impl = new Implementation()
    println impl.something()
  }
}

Running this produces:

groovy -cp . GroovyCode.groovy 
Interface's static method

Cool! Wait, what? We are creating a new instance of Implementation and invoking the instance method String something(). Why is Groovy invoking the static method with the same name, especially since Java does not permit us to invoke Implementation.something() as that method is only part of InterfaceWithStaticSomething?

I actually do not know the answer to this question! What I do know is that there must be ambiguity at the call site, where or what, again, I do not know. The Groovy runtime decides to invoke InterfaceWithStaticSomething.something(). If we use the Groovy Console to inspect the AST and view the bytecode generated from GroovyCode, we see this:

public static varargs main([Ljava/lang/String;)V
 L0
  INVOKESTATIC GroovyCode.$getCallSiteArray ()[Lorg/codehaus/groovy/runtime/callsite/CallSite;
  ASTORE 1
 L1
  LINENUMBER 4 L1
  ALOAD 1
  LDC 0
  AALOAD
  LDC LImplementation;.class
  INVOKEINTERFACE org/codehaus/groovy/runtime/callsite/CallSite.callConstructor (Ljava/lang/Object;)Ljava/lang/Object; (itf)
  LDC LImplementation;.class
  INVOKESTATIC org/codehaus/groovy/runtime/ScriptBytecodeAdapter.castToType (Ljava/lang/Object;Ljava/lang/Class;)Ljava/lang/Object;
  CHECKCAST Implementation
  ASTORE 2
 L2
  ALOAD 2
  POP
 L3
  LINENUMBER 5 L3
  ALOAD 1
  LDC 1
  AALOAD
  LDC LGroovyCode;.class
  ALOAD 1
  LDC 2
  AALOAD
  ALOAD 2
  INVOKEINTERFACE org/codehaus/groovy/runtime/callsite/CallSite.call (Ljava/lang/Object;)Ljava/lang/Object; (itf)
  INVOKEINTERFACE org/codehaus/groovy/runtime/callsite/CallSite.callStatic (Ljava/lang/Class;Ljava/lang/Object;)Ljava/lang/Object; (itf)
  POP
 L4
  LINENUMBER 6 L4
  RETURN
  LOCALVARIABLE args [Ljava/lang/String; L0 L4 0
  LOCALVARIABLE impl LImplementation; L2 L4 2
  MAXSTACK = 4
  MAXLOCALS = 3

Near the bottom of L3 we see the INVOKEINTERFACE instruction being used twice. At both occurrences, it is calling an interface method on CallSite, which is implemented with Groovy meta code. These two instructions dynamically invoke String something() (on the Implementation object) and println.

Ultimately, whatever the ambiguity is that is causing CallSite to select the static method over the instance method, we need to remove it. Instead of relying on the "dynamic sauce" of Groovy's runtime, we need to break through it. We need the rigid static compilation that Java gives us. We need @CompileStatic!

Part III: @CompileStatic

Here is what Groovy's documentation says about @CompileStatic.

This will let the Groovy compiler use compile time checks in the style of Java then perform static compilation, thus bypassing the Groovy meta object protocol.

When a class is annotated, all methods, properties, files, inner classes, etc. of the annotated class will be type checked. When a method is annotated, static compilation applies only to items (closures and anonymous inner clsses [sic]) within the method.

Source: https://docs.groovy-lang.org/2.4.2/html/gapi/groovy/transform/CompileStatic.html

This is exactly what we need, and the really nice thing is that we only need to annotate the method main(String[]). Let's do that and see what happens.

First, we annotate what we want to be statically compiled, which is GroovyCode's main(String[]):

class GroovyCode {
  @groovy.transform.CompileStatic
  static void main(String[] args) {
    Implementation impl = new Implementation()
    println impl.something()
  }
}

Next, let's run the Groovy code:

groovy -cp . GroovyCode.groovy 
Implementation's static method

Excellent. We are now calling the method of our Java API that we originally set out to call.

Lastly, let's look at the bytecode using the Groovy Console again:

public static varargs main([Ljava/lang/String;)V
 L0
  LINENUMBER 4 L0
  NEW Implementation
  DUP
  INVOKESPECIAL Implementation.<init> ()V
  ASTORE 1
 L1
  ALOAD 1
  POP
 L2
  LINENUMBER 5 L2
  LDC LGroovyCode;.class
  ALOAD 1
  INVOKEVIRTUAL Implementation.something ()Ljava/lang/String;
  INVOKESTATIC org/codehaus/groovy/runtime/DefaultGroovyMethods.println (Ljava/lang/Object;Ljava/lang/Object;)V
  ACONST_NULL
  POP
 L3
  LINENUMBER 6 L3
  RETURN
  LOCALVARIABLE args [Ljava/lang/String; L0 L3 0
  LOCALVARIABLE impl LImplementation; L1 L3 1
  MAXSTACK = 2
  MAXLOCALS = 2

We can see there is less bytecode generated. This makes sense as static compilation removes the runtime metaprogramming. Also, we can see that the two INVOKEINTERFACE instructions we noticed before have been replaced. The first is now INVOKEVIRTUAL and invokes Implementation.something() -- the API call we have been after this whole time. The second is now INVOKESTATIC and invokes the println method. Both of which are more efficient that hopping through the call site metaprogramming that was there before.

Conclusion

Sometimes you need to cut through dynamic invocation magic to ensure what you intend to happen actually happens. The example discussed in this post was attempting to invoke an instance method that had the same name as a static method provided by a Java Interface. Groovy's runtime metaprogramming invoked the static method instead of the instance method, even though the call site looked unambiguous. We corrected the behavior of the Groovy code by using the @CompileStatic annotation to statically compile the call site.

Fin.

https://docs.oracle.com/javase/specs/jvms/se11/html/jvms-4.html#jvms-4.6 ↩

HTTPS Client Certificate Authentication With Java

k-mack — Sun, 07 Mar 2021 23:26:45 +0000

I recently had to develop a Java client to interface with an internal service over HTTPS that required client certificate authentication. It is not often that I need to dive into SSL certificates, and doing so usually requires me to step back and relearn some things. This situation was no different, but in an attempt to burn this stuff into my brain, I am writing about it here.

It's a Trust Thing

First thing's first: the client needs to trust the HTTPS connection that the service wants to establish. The server certificate used by the service is signed by an internal certificate authority (CA). Since the client code runs on the Java Virtual Machine (JVM), it is by default subject to the collection of trusted CA certificate chains (Chain of Trust) used the JVM, which -- and rightly so -- does not include the CA that signed the service's server certificate.

The JVM's Chain of Trust is contained inside the file cacerts located in the Java Development Kit's (JDK) security directory. You can use keytool, which comes with the Java Development Kit (JDK), if you are curious about which CAs are trusted by your JVM. In fact, keytool has the command-line option -cacerts to short-circuit specifying the JDK's cacerts as the command's keystore. For example, if you wanted to know if your JDK trusts certificates generated by Let's Encrypt, you would want to know that it trusts IdenTrust’s "DST Root CA X3":

PS C:\Program Files\Java\zulu11.41.23-jdk11.0.8> .\bin\keytool.exe -list -cacerts | Select-String "IdenTrust"
Enter keystore password:

identrustcommercial [jdk], Jan 16, 2014, trustedCertEntry,
identrustdstx3 [jdk], Sep 30, 2000, trustedCertEntry,      # <-- Bingo!
identrustpublicca [jdk], Jan 16, 2014, trustedCertEntry,

Client code using the default JVM Chain of Trust will not allow a secure connection to be established with the service. The service's server certificate, inspected by the client's JVM during the SSL/TLS handshake, is seen as untrustworthy since it is signed by a CA not found in cacerts. This handshaking and rejection of trust is taken care of by the JVM networking and security code running beneath the client code.

As a quick demonstration, the following (Spock) test asserts that the client JVM code fails to create an SSL connection with the service. Note that I chose to use Vert.x Web Client to handle interacting with the service, but don't let this decision distract from the core content of this post. Nevertheless, if you haven't used Vert.x, I encourage you to try it out -- especially for building server-side network applications.

class ClientTest extends Specification {
  Vertx vertx
  WebClientOptions webClientOptions
  WebClient webClient
  AsyncConditions conditions

  def setup() {
    vertx = Vertx.vertx()
    webClientOptions = new WebClientOptions().setSsl(true)
    conditions = new AsyncConditions(1)
  }

  def "cannot establish secure connection with an untrusted CA"() {
    given:
    webClient = WebClient.create(vertx, webClientOptions)

    expect:
    webClient.get(443, "service.cn.local", "/api/endpoint").send({ event ->
      conditions.evaluate {
        assert event.failed()
        assert SSLHandshakeException.class.isInstance(event.cause())
        assert "Failed to create SSL connection" == event.cause().message
      }
    })
  }

  def cleanup() {
    webClient?.close()
    vertx.close()
  }
}

The solution is straight-forward: we need to tell the JVM that it can trust the service's server certificate.

Trust Me, I Got This

There are a couple of ways to have the JVM trust that the connection attempted to be made by the service really is secure:

Add the CA certificate chain to the JVM's cacerts.
Create a separate keystore and use that in the client code.

Although the first option provides a quick solution to the problem, it will effectively allow any future code that runs on the JVM with the modified cacerts to trust certificates signed by the internal CA. This may be perfectly acceptable given the CA;
e.g., if it's the root CA of your organization, then it probably makes sense to include it in your JVM's default Chain of Trust. In my case, the internal CA is for a dedicated and isolated system,
and I actually do not want to create an environment where any Java application running on the JVM can interface with the internal service. I only want the one application that I am developing to have this capability. Additionally, the client code does not need to interface with any other system over a secure connection, so I do not need to trust any SSL certificates besides those that are signed by the internal CA. Therefore, the second option suits me better.

Create a Java Keystore

The good people at IT sent me the internal CA certificate chain and a client certificate for authenticating with the service. The CA certificate chain was a PFX file and the client certificate was a P12 file. I installed both on my Windows machine to enable my web browsers to trust and authenticate with the internal service.

From this setup, a separate Java keystore containing the internal CA certificate chain can be created. First, export the root CA certificate and its intermediate certificates. These can be found on a Windows machine at Control Panel > Internet Options > Content (Tab) > Certificates (Button). From there,

find and export the trusted root CA as a DER encoded binary X.509 (CER), and
find and export each intermediate certificates as a DER encoded binary X.509 (CER).

Use keytool to create a new keystore file that trusts the exported certificates (add the root CA first):

PS C:\Program Files\Java\zulu11.41.23-jdk11.0.8> .\bin\keytool.exe -importcert -alias internal-ca-root -file C:\Temp\internal-ca-root.cer --storetype JKS -keystore C:\Temp\internal.jks
Enter keystore password:
Re-enter new password:

Trust this certificate? [no]:  yes
Certificate was added to keystore

PS C:\Program Files\Java\zulu11.41.23-jdk11.0.8> .\bin\keytool.exe -importcert -alias internal-ca-intermediate -file C:\Temp\internal-ca-intermediate.cer --storetype JKS -keystore C:\Temp\internal.jks
Enter keystore password:
Re-enter new password:

Trust this certificate? [no]:  yes
Certificate was added to keystore

Because the first command creates the keystore, the password you enter when prompted will set the password for it. When prompted for the password after entering the second command, the same password must be used.

Configure Client to Use Keystore

The created Java keystore containing the Chain of Trust can now be used in the client code:

class ClientTest extends Specification {

  // ...

  def "can establish secure connection with separate keystore"() {
    given:
    webClientOptions
        .setTrustStoreOptions(new JksOptions()
            .setPath(CUSTOM_KEYSTORE_PATH)
            .setPassword(CUSTOM_KEYSTORE_PASSWORD))

    webClient = WebClient.create(vertx, webClientOptions)

    when:
    webClient.get(443, "service.cn.local", "/api/endpoint").send({ event ->
      conditions.evaluate {
        assert event.succeeded()
        assert CONNECTED_BUT_NOT_AUTHENTICATED == event.result().bodyAsString()
      }
    })

    then:
    conditions.await(5)
  }
}

Although a secure connection is now established between the client and service, the two-way SSL authentication is not yet satisfied. Hence, the service responds with a message indicating the client cannot reach the desired endpoint because it was not authenticated.

Certificate, Please

As mentioned at the beginning of the post
-- and the original motivation for writing this --
the internal web service uses two-way SSL authentication. This means a client is authenticated when it presents a client certificate to the service that is issued by the root CA. If a client certificate is not provided (like in the test above) or it is not signed by the root CA, then the service denies the client's request.

The final step, therefore, is to configure the client code to authenticate with the server using the client certificate:

class ClientTest extends Specification {

  // ...

  def "can establish secure connection and authenticate with client certificate"() {
    given:
    webClientOptions
        .setTrustStoreOptions(new JksOptions()
            .setPath(CUSTOM_KEYSTORE_PATH)
            .setPassword(CUSTOM_KEYSTORE_PASSWORD))
        .setPfxKeyCertOptions(new PfxOptions()
            .setPath(CLIENT_CERT_PATH)
            .setPassword(CLIENT_CERT_PASSWORD))

    webClient = WebClient.create(vertx, webClientOptions)

    when:
    webClient.get(443, "service.cn.local", "/api/endpoint").send({ event ->
      conditions.evaluate {
        assert event.succeeded()
        assert CONNECTED_AND_AUTHENTICATED == event.result().bodyAsString()
      }
    })

    then:
    conditions.await(5)
  }
}

That's it. The client-server connection is secured over HTTPS, and the service authenticates the client. Note that Vert.x Web Client makes this easy, but I assume it is just as easy with other HTTP client libraries that support HTTPS.

Conclusion

Sometimes you need to establish secure connections with servers that have not been signed by a globally trusted entity. In these situations, it is can be desirable to create a custom keystore containing the Chain of Trust associated with the server(s) you need to interface with. Java's keytool can be used to do this.
If a server employs two-way SSL authentication, you can use an HTTP library, like Vert.x Web Client, to add a client certificate to your HTTPS connection.

Fin.