Forem: Christophe Willemsen

Write a kubectl plugin in Java with jbang and fabric8

Christophe Willemsen — Sat, 26 Dec 2020 12:48:09 +0000

If you use kubectl frequently, you might be find yourself running the same sequence of commands over and over again.

A frequent use case yielding such sequences is for viewing a secret which requires to find the secret and decode its base64 form.

In this tutorial, I'll go through how to create a kubectl plugin in Java. The plugin will do the following after we execute the kubectl lp command :

List the pods with their name, status and a default empty message
If the pod is not in running state, display a 🔥 icon in front of its line, otherwise display a ✅ icon
Display a message why the pod is not running

Jbang

To write the plugin in Java, I will use jbang.
Jbang is a framework that brings scripting to the Java land.

If you don't have Jbang yet installed, just run the following :

brew install jbangdev/tap/jbang

Or check the documentation to find the instructions for your operating system.

The next step is to create a template with the initializer, I'll call the script lp for list pods :

jbang init --template=cli lp.java

It will create a fully functional script lp.java with the following content :

///usr/bin/env jbang "$0" "$@" ; exit $?
//DEPS info.picocli:picocli:4.5.0

import picocli.CommandLine;
import picocli.CommandLine.Command;
import picocli.CommandLine.Parameters;

import java.util.concurrent.Callable;

@Command(name = "lp", mixinStandardHelpOptions = true, version = "lp 0.1",
        description = "lp made with jbang")
class lp implements Callable<Integer> {

    @Parameters(index = "0", description = "The greeting to print", defaultValue = "World!")
    private String greeting;

    public static void main(String... args) {
        int exitCode = new CommandLine(new lp()).execute(args);
        System.exit(exitCode);
    }

    @Override
    public Integer call() throws Exception { // your business logic goes here...
        System.out.println("Hello " + greeting);
        return 0;
    }
}

Which is a simple template that greets you, try it with the following command :

$ jbang lp.java Chris
Hello Chris
$

Adding dependencies to your scripts is as simple as adding a comment in the beginning of the file with the dependency artifact :

//DEPS io.fabric8:kubernetes-client:4.13.0

Here, we added the fabric8 kubernetes client library. While we're at it, let's see how this client works.

Fabric8

The fabric8 kubernetes client is a library that provides access to the Kubernetes Rest API via a simple and human friendly DSL.

For eg, here is how you find only the pods that are not in a RUNNING status :

kc.pods().list().getItems()
                .stream()
                .filter(pod -> !PodStatusUtil.isRunning(pod))
                .collect(Collectors.toList());

For our scenario, we will use the client to retrieve all the pods and create a new domain model PodInfo with the name, the state and an optional message :

private static List<PodInfo> getPods() {
        KubernetesClient kc;
        try {
            kc = new DefaultKubernetesClient();
        } catch (Exception e) {
            throw new RuntimeException("Unable to create default Kubernetes client", e);
        }

        return kc.pods().list().getItems().stream().map(pod -> {
            PodInfoState state = PodStatusUtil.isRunning(pod) ? PodInfoState.RUNNING : PodInfoState.FAILING;
            String message = null;
            if (!state.equals(PodInfoState.RUNNING)) {
                message = PodStatusUtil.getContainerStatus(pod).get(0).getState().getWaiting().getMessage();
            }

            return new PodInfo(pod.getMetadata().getName(), state, message);
        }).collect(Collectors.toList());
    }

    static class PodInfo {

        private final String name;
        private final PodInfoState state;
        private final String message;

        public PodInfo(String name, PodInfoState state, String message) {
            this.name = name;
            this.state = state;
            this.message = message;
        }

        public String getName() {
            return name;
        }

        public PodInfoState getState() {
            return state;
        }

        public String getMessage() {
            return message;
        }
    }

    enum PodInfoState {
        RUNNING,
        FAILING
    }

We'll also add the j-text-utils java library so we can display everything in a table

//DEPS com.massisframework:j-text-utils:0.3.4

private static void printTable(List<PodInfo> list) {
        final Object[][] tableData = list.stream()
                .map(podInfo -> new Object[]{
                        podInfo.getState().equals(PodInfoState.RUNNING) ? CHECK_MARK : FIRE,
                        podInfo.getName(),
                        podInfo.getState(),
                        podInfo.getMessage()
                })
                .toArray(Object[][]::new);
        String[] columnNames = {"", "name", "state", "message"};
        new TextTable(columnNames, tableData).printTable();
    }

The full code can be found here : https://github.com/ikwattro/kubectl-plugin-java-jbang

If we run our script, we can see it in action :

$ jbang lp.java
[jbang] Building jar...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
________________________________________________________________________________________
|   | name                             | state  | message                               |
|=======================================================================================|
| ✅ | neo4j-neo4j-core-0               | RUNNING|                                       |
| ✅ | nginx-app                        | RUNNING|                                       |
| 🔥| nginx-deployment-77fbdb7874-8gkcr| FAILING| Back-off pulling image "nginx2:1.16.1"|
| 🔥| nginx-deployment-77fbdb7874-8jps7| FAILING| Back-off pulling image "nginx2:1.16.1"|
| 🔥| nginx-deployment-77fbdb7874-vx27r| FAILING| Back-off pulling image "nginx2:1.16.1"|
| ✅ | postgres                         | RUNNING|                                       |
| ✅ | coredns-f9fd979d6-qwgfm          | RUNNING|                                       |
| ✅ | etcd-minikube                    | RUNNING|                                       |
| ✅ | kube-apiserver-minikube          | RUNNING|                                       |
| ✅ | kube-controller-manager-minikube | RUNNING|                                       |
| ✅ | kube-proxy-s5bcc                 | RUNNING|                                       |
| ✅ | kube-scheduler-minikube          | RUNNING|                                       |
| ✅ | storage-provisioner              | RUNNING|                                       |
$

Extending kubectl

Extending kubectl is very easy as it has mainly 3 conventions :

You need to place your script in your path
The name of the script should start with kubectl- and not have an extension
The script should be executable

We will need to pay attention, because when generating the jbang template, the very first line was commented for playing nicely with IDEs when editing the script.

Just uncomment the line :

#!/usr/bin/env jbang

Secondly, we will copy the script to our path :

cp lp.java /usr/local/bin/kubectl-lp

And make it executable :

chmod +x /usr/local/bin/kubectl-lp

You can now run kubectl lp for listing your pods :

$ kubectl lp
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
________________________________________________________________________________________
|   | name                             | state  | message                               |
|=======================================================================================|
| ✅ | neo4j-neo4j-core-0               | RUNNING|                                       |
| ✅ | nginx-app                        | RUNNING|                                       |
| 🔥| nginx-deployment-77fbdb7874-8gkcr| FAILING| Back-off pulling image "nginx2:1.16.1"|
| 🔥| nginx-deployment-77fbdb7874-8jps7| FAILING| Back-off pulling image "nginx2:1.16.1"|
| 🔥| nginx-deployment-77fbdb7874-vx27r| FAILING| Back-off pulling image "nginx2:1.16.1"|
| ✅ | postgres                         | RUNNING|                                       |
| ✅ | coredns-f9fd979d6-qwgfm          | RUNNING|                                       |
| ✅ | etcd-minikube                    | RUNNING|                                       |
| ✅ | kube-apiserver-minikube          | RUNNING|                                       |
| ✅ | kube-controller-manager-minikube | RUNNING|                                       |
| ✅ | kube-proxy-s5bcc                 | RUNNING|                                       |
| ✅ | kube-scheduler-minikube          | RUNNING|                                       |
| ✅ | storage-provisioner              | RUNNING|                                       |
$

Conclusion

The Kubectl tool is easily extensible and Jbang allows to write scripts in Java in no time, where we would probably have turned to Python before that tool existed.

References :

The code from this blog post : https://github.com/ikwattro/kubectl-plugin-java-jbang
kubectl : plugins documentation : https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/
jbang : https://github.com/jbangdev/jbang#installation
fabric8 : https://github.com/fabric8io/kubernetes-client

Handling synonyms in Neo4j Full Text Search

Christophe Willemsen — Mon, 11 Nov 2019 11:44:19 +0000

So you have followed the Deep Dive into Neo4j's Full Text Search tutorial, learned even how to create custom analyzers and finally watched the Full Text Search tips and tricks talk at the Neo4j Nodes19 online conference?

Still, searching for boat does not yield results containing yacht or ship, and you're wondering how to make your search engine a bit more relevant for your users?

Don't go any further, you'll learn how to do it, now!

Synonyms

A synonym is a word or phrase that means exactly or nearly the same as another word or phrase.

Why synonyms ?

It's all about recall! In other words, to help your users find the content they're interested in without them having to know specific terms.

A user searching for coffee should probably be seeing results containing latte macchiato, espresso or even ristretto.

Lists of synonyms

You can find 3rd party word lists for synonyms, such as WordNet or ConceptNet5, howeveer, appropriate word lists are domain/application/use-case dependent, and the best fit is generally a self-curated synonyms word list.

How to use them ?

The first thing to do, is to create a word list with the following format :

coffee,latte macchiato,espresso,ristretto
boat,yacht,sailing vessel,ship
fts,full text search, fulltext search

The next step is to create a custom analyzer using the synonym filter. Since we're using an analyzer the first question that might come to mind is :

Do I have to reindex all the documents when my synonyms list change ?

The answer is yes, using a query time synonym filter is very bad(TM), for the following reasons :

The QueryParser tokenizes before giving the text to the analyzer, so if a user searches for sailing vessel, the analyzer will be given the words sailing and vessel separately, and will not know they match a synonym
Multi-Word synonyms will also not work in phrase queries
The IDF of rare synonyms will be boosted

More information can be found in the Solr documentation.

Let's create our custom analyzer for synonyms then :

@Service.Implementation(AnalyzerProvider.class)
public class SynonymAnalyzer extends AnalyzerProvider {

    public static final String ANALYZER_NAME = "synonym-custom";

    public SynonymAnalyzer() {
        super(ANALYZER_NAME);
    }

    @Override
    public Analyzer createAnalyzer() {
        try {
            String synFile = "synonyms.txt";
            Analyzer analyzer = CustomAnalyzer.builder()
                    .withTokenizer(StandardTokenizerFactory.class)
                    .addTokenFilter(StandardFilterFactory.class)
                    .addTokenFilter(SynonymFilterFactory.class, "synonyms", synFile)
                    .addTokenFilter(LowerCaseFilterFactory.class)
                    .build();

            return analyzer;
        } catch (Exception e) {
            throw new RuntimeException("Unable to create analyzer", e);
        }
    }

    @Override
    public String description() {
        return "The default, standard analyzer with a synonyms file. This is an example analyzer for educational purposes.";
    }
}

A very important note is that the LowerCaseFilter comes after the SynonymFilter, in some use cases it causes synonyms to not be recognized, for example with the following list :

GB,gibabyte

If the lowercase filter is applied before synonyms, then the tokens will not match.

Create a synonyms.txt file with your synonyms list in the conf/ directory of your Neo4j instance :

conf/synonyms.txt

coffee,latte macchiato,espresso,ristretto
boat,yacht,sailing vessel,ship
fts,full text search, fulltext search

Build your analyzer jar and put it in the plugins directory of Neo4j and restart the database if needed.

Create the Index

CALL db.index.fulltext.createNodeIndex(
  'syndemo', 
  ['Article'], 
  ['text'], 
  {analyzer:'synonym-custom'}
)

Create an Article node with some text :

CREATE (n:Article {text: "This is an article about Full Text Search and Neo4j, let's go !"})

Query the index :

CALL db.index.fulltext.queryNodes('syndemo', 'fts')

╒══════════════════════════════════════════════════════════════════════╤══════════════════╕
│"node"                                                                │"score"           │
╞══════════════════════════════════════════════════════════════════════╪══════════════════╡
│{"text":"This is an article about Full Text Search and Neo4j, let's go│1.2616268396377563│
│ !"}                                                                  │                  │
└──────────────────────────────────────────────────────────────────────┴──────────────────┘

Similarly, a search for fulltext will return the result as well. But let's get fancy, heuu fuzzy ! :

CALL db.index.fulltext.queryNodes('syndemo', 'fullt*')

No results, no records

Prefix and synonyms ?

There is one limitation : prefix,fuzzy,.. queries do not use the analyzer, they produce term or multiterm queries instead.

But there is a trick you can use, add an NgramFilter to your analyzer and use a phrase query, so fts and its synonyms will have their ngrams tokenized and stored/retrieved in the index :

Analyzer analyzer = CustomAnalyzer.builder()
                    //...
                    .addTokenFilter(NGramFilterFactory.class, "minGramSize", "3", "maxGramSize", "5")
                    .build();

            return analyzer;

The NgramTokenFilter will tokenize the inputs into n-grams of the given sizes, here min 3 and max 5. So for the following input :

fulltext search

The index will contain the n-grams ful, full, fullt, ull, ullt, ullte, lte, ltex, ltext.

You can also use the EdgeNgramFilter will will produce n-grams only from the beginnig of the token, for the same example as above the n-grams will be ful, full, fullt.

Re-deploy your plugin, restart the database, drop and recreate the index and now :

CALL db.index.fulltext.queryNodes('syndemo', '"fullt*"')

╒══════════════════════════════════════════════════════════════════════╤═══════════════════╕
│"node"                                                                │"score"            │
╞══════════════════════════════════════════════════════════════════════╪═══════════════════╡
│{"text":"This is an article about Full Text Search and Neo4j, let's go│0.04872262850403786│
│ !"}                                                                  │                   │
└──────────────────────────────────────────────────────────────────────┴───────────────────┘

To finalize, let's try some other phrase queries :

CALL db.index.fulltext.queryNodes('syndemo', '"article fullte*"~2')

╒══════════════════════════════════════════════════════════════════════╤══════════════════╕
│"node"                                                                │"score"           │
╞══════════════════════════════════════════════════════════════════════╪══════════════════╡
│{"text":"This is an article about Full Text Search and Neo4j, let's go│2.3429081439971924│
│ !"}                                                                  │                  │
└──────────────────────────────────────────────────────────────────────┴──────────────────┘

Conclusion

Synonyms are a valuable asset when building search engines, offering a better recall and thus a better user experience.

You can find all the code from this blog post on this example Github repository