Implementing the Language Server Protocol

Ian G Schweer — Wed, 06 Jul 2022 17:32:59 +0000

How we got here

Before the end of 2021, my work scheduled a hackathon for my organization. With the requirement being that it be useful for us or our players. I decided to attempt to develop a language server to help engineers get their services into our production stack.

Our stack is pretty complex and takes a lot of learning. Although the language one interacts with is yaml, it's really a DSL and requires a lot of support. The goal of this tool was to help cut down the amount of assistance someone would need who is new / generally unfamiliar with the stack.

I found it pretty difficult to get up and running, so I wanted to share my work and progress for any other developer productivity enthusiasts!

What is a Language Server

A language server is a standard developed by Microsoft that defines a language server and language client; which handle the language syntax parsing, linting, code hints (etc) and the actual interfacing with the developer respectively. Two language server clients are Visual Studio Code, and recently Neovim on version 0.6 (written entirely in Lua!). Some language servers include yamlls, sourcegraph-go and rust-analyzer. The language server protocol then defines the glue between these two components. The glue happens over JSON-rcp.

For our custom LSP, we'll start by forking the yaml-language-server maintained by redhat.

Getting started

The first thing you'll need to do is enable the LSP for your editor of choice. I'm very deep into the vim camp, so I did neovim. This is the typical setup code needed to run the yamlls

local nvim_lsp = require('lspconfig')
local servers = { 
  'yamlls',  
}
for _, lsp in ipairs(servers) do
  nvim_lsp[lsp].setup {
    on_attach = on_attach,
    flags = {
      debounce_text_changes = 150,
    }
  }
end

Which just tells the neovim LSP to setup the client to talk to the language server. The yaml-language-server runs a binary on your local computer. The default is with the --stdio flag, which would accept standard input and validate: yaml-language-server --stdio. If you run :LspInfo from vim, you'll see this info

1 client(s) attached to this buffer: yamlls

  Client: yamlls (id 1)
    root:      /Users/ischweer/dev/shards
    filetypes: yaml
    cmd:       yaml-language-server --stdio

There tends to be a lot of config you can set as well, you could also run the language server separately with an --rpc-port and --rpc-host option if you chose to run the language server in one window, and test it in the other.

A small demo of what I built

You're ready to start developing. In my case, I forked the yaml-language-server, and started adding some addition logic to fetch application specs, their configuration, and WAN setup from our central services. A service at Riot has this sort of kube-esque definition, just with more business DSL ontop

application-instance:
  name: some.app
  network:
    inbound:
      - name: another.app
        location: usw2

Pulling all these down over http is fairly straight forward. Once we have all the yamls, they are cached

  async downloadApps() {
    // we want to go through all the apps in the env, and get
    // the yaml'd app definitions
    await this.getDiscoverous();

    // now we can get the lol-services 710e env
    const url = `https://${this.gandalf_host}/api/v1/environments/lol-services/${env.LATEST_VERSION}`;
    const resp: XHRResponse = await xhr({ url: url, headers: { authorization: `Bearer ${this.gandalf_token}` } });
    console.log('Successfully grab 710e instance');

    const cache_builder: Array<Promise<boolean>> = [];

    for (const appMetadata of environmentInstance['environment']['applications']) {
      cache_builder.push(this.downloadApp(appMetadata));
    }

    await Promise.all(cache_builder);
  }

  async downloadApp(app: { name: string; version: string }): Promise<any> {
    const url = `https://${this.gandalf_host}/api/v1/applications/${app.name}/${app.version}`;
    const resp = await xhr({ url: url, headers: { authorization: `Bearer ${this.gandalf_token}` } });
    const _d = new YAML.Document();
    _d.contents = JSON.parse(resp.responseText);

    return new Promise(() => {
      fsp
        .writeFile(`${homedir()}/.cache/gandalf/${app.name}.yaml`, _d.toString(), { flag: 'wx' })
        .then()
        .catch((err) => {
          console.log(`Did not write file because it exists already ${app.name}.yaml`);
        });
    });
  }

In order to fully understand this yaml parsing, you'll need to read the subsequent blog post on the riot eng blog :) however, once it's downloaded, we have now have some cached files that help us answer questions like "Is service X actually specified talk to service Y, or is it a typo?". That looks like this in the typescript

    for (const defined_outbound of appSpec.outbounds || []) {
      let found = false;
      for (const _instanced_outbound of app_instance_outbounds.items || []) {
        const instanced_outbound = _instanced_outbound as YAMLMap;
        if (instanced_outbound.get('service') == defined_outbound) {
          found = true;
          break;
        }
      }
      if (!found) {
        errors.push({
          message: `Missing required outbound for ${defined_outbound}`,
          location: { start: app_instance_outbounds.range[0], end: app_instance_outbounds.range[2], toLineEnd: true },
          severity: 1,
          source: 'Gandalf',
          code: ErrorCode.Undefined,
        } as YAMLDocDiagnostic);
      }
    }

With that in place, we can see nice little errors in our editor when services are expected to talk to each other, but not specified to do so.

Actually doing the work

LSP is all over RPC, and you can see all the events specified above. Everything is async, and the client needs to be able to handle json serialized text for everything. You need to learn your editors internals to fully comprehend how the LSP is working :').

For neovim, always check the lsp logs: tail -n 10000 ~/.cache/nvim/lsp.log | bat. They will be extremely valueable, as most times you'll have serialization errors or unknown property issues. I found adding the following "always log" approach helped when debugging (though it slows the editor down)

console.error = (arg) => {
  if (arg === null) {
    connection.console.info(arg);
  } else {
    connection.console.error(arg);
  }
};