Forem: Raj Saxena

Continuously deploying to Kubernetes with Github Actions.

Raj Saxena — Tue, 25 Aug 2020 12:00:00 +0000

Background

We are a young Berlin-based startup that is on the Google Cloud Platform. We find GCP to be much more developer-friendly compared to the other cloud providers that we had used at our previous companies. Google Cloud offers gcloud, a CLI tool that can be used to interact & manage cloud resources on the platform. It can be used to authorize and configure credentials for various other tools like docker and kubectl. GCP has a concept of Service-Accounts that can be used to manage services by assigning the appropriate IAM permissions.

Why Github Actions?

One of the biggest pain point of the SRE/DevOps/Platform team at a previous company was managing the Jenkins cluster running CI/CD jobs. A typical self-managed installation has the following challenges:

Auto-scaling with the needs of the day. You need increased capacity to run jobs immediately during the core business hours and reduced capacity otherwise.
Jobs might need different versions of software installed. Example - one service might need Java 11 & the other service might have adapted Java 15 already.
If the jobs running on the same shared worker node are not isolated properly, they can interfere with each other by relying on the same files and libraries and overwriting them.
Jobs running on the same worker node can compete for resources leading to an increase in the build times or sometimes even timeouts causing flakiness in the pipeline.
Constant software maintenance of the main server and the worker nodes. This includes upgrading base OS, patching, or manually updating installed software.

When we were discussing the architecture of the platform, people involved brought up their (painful) experiences with CD servers that they had in the past. We all agreed that we didn't want to manage a CI/CD server cluster of our own.
To solve all of the above problems and for a few other reasons, we decided to go with Github Actions

Github Actions provide a simple YAML based syntax to configure jobs that can trigger on any Github event like push, merge to the main branch, etc. These jobs run on one of the available servers on Microsoft Azure.

We started by having a simple job to continuously integrate, build & test, create a docker container & push to the container registry. Here's a sample from a java service.

on:
  push:
    branches:
      - master

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2 # checkout repo
      - uses: actions/setup-java@v1 # Set up latest Java 14
        with:
          java-version: 14

      - name: Build with Gradle
        run: ./gradlew clean build

      - uses: GoogleCloudPlatform/github-actions/setup-gcloud@master # Setup gcloud
        with:
          service_account_key: ${{ secrets.GCP_SA_KEY }} # ServiceAccount key with necessary rights added as a secret on Github.

        # Configure credentials for docker
      - run: gcloud auth configure-docker

        # Build the Docker image
      - run: docker build -t gcr.io/example.com/${{ github.event.repository.name }}:${{ github.sha }} .

        # Push the Docker image to Google Container Registry
      - run: docker push gcr.io/example.com/${{ github.event.repository.name }}:${{ github.sha }}

Deploying apps on GKE

Another key piece in the architecture is our Kubernetes cluster where the services are deployed. We are proudly using Google Kubernetes Engine and so far the experience has been positive as it is easy to manage and scale and eliminates the operational overhead.
Kubernetes is managed with kubectl & gcloud CLI can be used to configure credentials for it by generating the necessary kubeconfig.

Applications are packaged as Helm charts and configured as per the environment needs. It's important to mention this because helm utilises kubeconfig to interact with the cluster.

Deployments are created and updated with the latest image from the container registry as follows:

# Creating a deployment
helm install -f values.yml --set image.version=<latest> example-service example-service

# Updating a deployment
helm upgrade -f values.yml --set image.version=<latest> example-service example-service

The GKE hardening guide suggests restricting the network access to the Kubernetes cluster and creating it as a private cluster. This isolates the nodes from the public internet and the access to Kubernetes API server can be further restricted to specific trusted network IPs only.

When we started, we created an allow-list with the IP of a trusted machine used to manage the cluster and deploy upgrades manually. We also built a (very cool) UI tool that could be used to deploy, revert & scale deployments.

At this point, this is how it all looked together

For a long time, this worked really well until we started feeling the need for Continuous Deployment. I share the argument for it here.

The challenge

The simplest way to add Continuous Deployment to the pipeline was to add another step in the Github Action pipeline that would deploy the latest image to the cluster. Let's update the action:

    # ...
    # same as before
    # ...

      - uses: GoogleCloudPlatform/github-actions/setup-gcloud@master # Setup gcloud
        with:
          service_account_key: ${{ secrets.GCP_SA_KEY }} # ServiceAccount key with necessary rights added as a secret on Github.

        # Configure credentials for docker
      - run: gcloud auth configure-docker

        # Build the Docker image
      - run: docker build -t gcr.io/example.com/${{ github.event.repository.name }}:${{ github.sha }} .

        # Push the Docker image to Google Container Registry
      - run: docker push gcr.io/example.com/${{ github.event.repository.name }}:${{ github.sha }}

        # Generate kubeconfig entry
      - run: gcloud container clusters get-credentials <cluster-name> --zone <zone> --project <project>

        # Install helm
      - uses: azure/setup-helm@v1
        id: install

        # Deploy latest version
      - run: helm upgrade -f values.yaml --set image.version=${{ github.sha }} example-service example-service

Unfortunately, this doesn't work & the pipeline fails at the last step. This is because the Kubernetes server is unreachable over the public internet.

Possible solutions (& the bag of problems they come with)

Make the cluster public One easy way of making the setup work, would be to make the cluster reachable over the internet. However, this would be at the expense of exposing the cluster and would have made the setup less secure. The trade-off would not be worth it.
Add the Action Runner IPs to the Allow-list While the idea seemed good in the beginning, when I dug a little deeper, I found that the servers that run the job could be any from the 5 Azure regions in the US. The Github page lists a way to download a JSON file containing the IP ranges that is updated periodically. Going down this path would have meant constantly fetching the list & updating the allow-list.

An automated solution would have been the best but I didn't want to parse the file. Luckily, Azure cloud allows to get the IP ranges for Service Tags. The problem is that all the IP ranges combined for all 5 regions are 1100+ (the last time I checked) and GKE can only have 50 entries for authorized networks.
<insert-image--banging-head-against-the-wall>
Have a light proxy server running that only allows access from the Azure Ip ranges. There wasn't an easy, out of the box solution available that could be configured programmatically to keep updated with the list of IPs.
I considered if I should create a publicly accessible API that can be called to trigger deploy? Maybe, I could authenticate with Basic Auth and call it secure. (It's not!). It's just didn't feel right to go down this path.

All of these seemed unnatural solutions for various reasons. Unable to programmatically configure the firewall, I seemed to have hit a wall.

The Solution

I kept thinking for a solution at the back of my head while working on other tasks. I didn't want to manage CD servers, I didn't want to risk exposing the cluster & I wanted helm to reach the Kubernetes control server.
I stared at the action's YAML looking for a solution more times than I would like to admit, till it finally hit me - I only needed to run the deployment step from a trusted IP.

I split the job into 2 - build & deploy.

The build part is the CPU & memory intensive part that compiles code, runs tests, creates artifacts, builds docker image & then pushes it to the container registry.
The deploy part just deploys the latest image with helm. Once build is done on Github hosted action runner, deploy is run on a self-hosted action-runner.

This self-hosted runner is a compute machine inside Google Cloud having a static IP. I could now configure this static IP into the allow-list of the Kubernetes authorized IP list.

The updated setup looks like this:

Here's the updated actions.yaml:

on:
  push:
    branches:
      - master

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2 # checkout repo
      - uses: actions/setup-java@v1 # Set up latest Java 14
        with:
          java-version: 14

      - name: Build with Gradle
        run: ./gradlew clean build

      - uses: GoogleCloudPlatform/github-actions/setup-gcloud@master # Setup gcloud
        with:
          service_account_key: ${{ secrets.GCP_SA_KEY }} # ServiceAccount key with necessary rights added as a secret on Github.

        # Configure credentials for docker
      - run: gcloud auth configure-docker

        # Build the Docker image
      - run: docker build -t gcr.io/example.com/${{ github.event.repository.name }}:${{ github.sha }} .

        # Push the Docker image to Google Container Registry
      - run: docker push gcr.io/example.com/${{ github.event.repository.name }}:${{ github.sha }}

  deploy:
    runs-on: self-hosted
    needs: [build] # to block for the build step to complete successfully
    steps:
      - uses: actions/checkout@v2

      - uses: GoogleCloudPlatform/github-actions/setup-gcloud@master # Setup gcloud
        with:
          service_account_key: ${{ secrets.GCP_SA_KEY }}

        # Generate kubeconfig entry
      - run: gcloud container clusters get-credentials <cluster-name> --zone <zone> --project <project>

        # Install helm
      - uses: azure/setup-helm@v1
        id: install

        # Deploy latest version
      - run: helm upgrade -f values.yaml --set image.version=${{ github.sha }} example-service example-service

I tried it and it worked perfectly 🎉.

Since then, we have updated all of the services to use a similar pattern of splitting build & deploy jobs & are now happily shipping code to all environments with each change.

If you have read so far, I am curious to learn from you?

What do you think of the solution?
Do you use Github Actions as well?
Do you deploy continuously to Kubernetes as well?
How do you do it now? And if you can, how would you change it?

Disclaimer

The setup shown here is a simplified version that excludes other key components present in the actual environments to focus on the topic discussed above.

A brief argument for Continuous Deployment

Raj Saxena — Tue, 11 Aug 2020 19:55:41 +0000

I work at a young startup. The engineering team consists of experienced and smart engineers who come from different backgrounds. We understood the importance of continuously integrating from the start. I want to share our journey of how we moved from Continuous Delivery to Continuous Deployment.

Let's do a quick refresher of the differences between Continuous Delivery & Continuous Deployment

Continuous Delivery

Continuous Delivery is when you integrate your code often, run the automated test suite and if they pass, proceed to run the build step to create artifacts with each merge to master. Examples of this includes compiling Java code into JARs, building JS bundles, packaging them into docker containers or creating golden machine images. The main purpose is to be able to deploy easily, at any time to any of the environments.

Continuous Deployment

Continuous Deployment takes it one step further and means taking each change all the way through the deployment pipeline and releasing it in all the environments automatically. The key principle is to release things in smaller chunks so that there are less chances of things breaking because of multiple changes aggregated over since the last release. This helps to reduce the Mean Time To Recovery (MTTR) as you have a narrow set to inspect in case something breaks.

Although it is not a novel idea, people inexperienced with it are sometimes apprehensive about it if they don't have experience with it. These concerns are valid and having them is understandable as there are multiple technical, organisational, operational or legal challenges at the company level that might prevent having a Continuous Deployment pipeline.

It's a shift in thinking and what's important is to understand that Continuous Deployment when done right actually improves stability. Once developers are used to the idea, they are more conscious when they merge code to the main branch, to test & ensure that the change is working as expected. In case, if/when something happens to brake, developers have the changes fresh in memory & are already in the context to root-cause, fix and recover.

A Continuous Deployment pipeline ensures that the latest changes are always deployed & keeps the different environments always in sync. It also avoids having to chase and remind people to deploy the latest versions.

Let's take an example to contrast with the alternative of not deploying immediately - A developer works on a task and merges the code to master. But, it's 5 PM and they don't want to risk deploy the changes to production now. They promise to do it first thing in the morning. Now, there are a lot of things that can happen between the time that code got merged to when it is actually deployed.

Some other dev working on a different task could have merged to master and acting responsibly to test their changes could have unknowingly deployed the latest version that includes the untested change of the previous task.
The original dev could have fallen sick. In this case, other members of the team would not know if the original developer has verified the changes or not.
The original dev could have moved on to a high priority bug fix. By the time, the developer came back to test the changes, they could have forgotten all the edge cases.

If something breaks now, it will take more time for the developer to get back into the context and resolve, compared to when they were already working on it and the knowledge was fresh.

Convinced, yet?

Continuous Deployment requires having sufficient automated tests (which you should be doing anyway, right?) to have enough confidence and investment in automated tooling to enable pipelines.

I understand that there cases where you actually want to be able to change a risky change in a non-production environment. For this, we first built a mechanism to 🔒 lock deployments from proceeding beyond a certain stage if we weren't comfortable deploying them all the way. This ensured that nobody else could accidentally deploy either to push a different change.

We also invested into a feature-toggle service that allows toggling things on/off at runtime without redeploying.

A reasonable approach to decision making.

Because people came from different experiences, they had concerns, fears and mixed opinions about the Continuous Deployment idea. We discussed and at the end, instead of relying on opinions & thoughts, we relied on data to help us make our decision.
We measured the percentage of non-locked deployments and the percentage of locked deployments over a period of time and concluded to optimise for the normal use-case rather than the exception.

If you want to convince people in your team & organisation, perhaps a similar approach could help you.

What is your experience?

References:

https://martinfowler.com/bliki/ContinuousDelivery.html

Originally published on https://suspendfun.com

SpringBoot fat-jar fails, Shadow 🕵🏽‍♂️ to the rescue.

Raj Saxena — Mon, 06 Jul 2020 19:57:55 +0000

Problem

I faced a peculiar problem where a fat-jar wouldn't run on remote runners. I described the details in my previous post - Dataflow + SpringBoot app fails to run when Dockerized.

To unblock myself and deliver on time, I hacked together a solution that during the build step instead of compiling the source to create a jar and containerize the jar, I packaged the source repository and used the gradle wrapper to run that code when the container starts.
It worked but it had a few problems:

❗️The images were almost 1 GB. Our typical service is around 350 MB out of which 195 MB is the Distroless Java 11 itself.
❗️The startup was slow as gradle was starting first and then booting the app.
❗️The image wasn't self-sufficient and I had to inject Artifactory credentials at runtime to start the container so that gradle can authenticate and validate dependencies.

All of these made the solution less than ideal.

In the following days, when I had more time, I kept looking into the issue and found the following issues on Github and Apache BEAM project

The above seemed related but still focused on different problems and not related specifically to SpringBoot so I also created a 🐞 BUG ticket with details.

Solution

Reading through different comments, I saw a suggestion to use maven-shade-plugin. We use gradle and I learnt that there is something similar for it called Shadow Gradle plugin

Sharing what it is directly from their page:

Shadow is a Gradle plugin for combining a project's dependency classes and resources into a single output Jar. The combined Jar is often referred to a fat-jar or uber-jar. Shadow utilizes JarInputStream and JarOutputStream to efficiently process dependent libraries into the output jar without incurring the I/O overhead of expanding the jars to disk.

Shadowing a project output has 2 major use cases:

Creating an executable JAR distribution.

Bundling and relocating common dependencies in libraries to avoid classpath conflicts

👨🏻‍💻I gave it a try and it worked. ✨

Configuration

This is my build.gradle.kts configuration:

// shadow config
tasks.withType<ShadowJar> {
    isZip64 = true
    // Required for Spring
    mergeServiceFiles()
    append("META-INF/spring.handlers")
    append("META-INF/spring.schemas")
    append("META-INF/spring.tooling")
    transform(PropertiesFileTransformer().apply {
        paths = listOf("META-INF/spring.factories")
        mergeStrategy = "append"
    })
}

With this configuration in place, the build pipeline runs ./gradlew build in the step to compile and create the merged jar. Shadow compiles the file to a jar with -all.jar suffix which is then copied to the Docker image.
This is the Dockerfile:

FROM gcr.io/distroless/java:11
VOLUME /tmp
COPY build/libs/dataExtractor-0.0.1-all.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom", "-Dspring.profiles.active=${ENV_SPRING_PROFILE}", "-jar", "/app.jar"]

This solved the 3 problems:

✅ The image is 340 MB and that is close to expected.
✅ Startup times are similar to expected.
✅ No more injected Artifactory credentials. I rotated the credentials that were used for the hack.

This was an interesting problem because the app worked every time from the IDE or when run locally via gradle but failed when it was containerized and deployed.

Please let me know if you think there is something that can be improved.

Originally published on https://suspendfun.com

Dataflow + SpringBoot app fails to run when Dockerized.

Raj Saxena — Mon, 06 Jul 2020 19:57:42 +0000

The Setup and Problem

We have a Microservices architecture with each service having its separate CloudSQL instance (like how it should be). These databases are accessible from only inside the Virtual Private Cloud (VPC) network and Hashicorp's Vault is used to generate dynamic credentials for them.

We needed a DataWarehouse for analysis and reporting. However, the setup made it a bit tricky as we didn't want to use an external ETL service or have long-living database credentials.
We agreed that the data needs to be extracted to a BigQuery Dataset from where we would expose it with Metabase.

The extraction needs to happen incrementally and periodically so the service would need to:

1) Connect to Vault to generate the read-only credentials for each database.
2) Read the required tables and data. Apply any transformations like type conversion, anonymization or pseudonymization.
3) Write to the BigQuery dataset.

Since data could be huge, I decided to use Dataflow which is Google's Stream and Batch Processing service.
Dataflow is built on Apache Beam's programming and execution model that expects you to define a Pipeline that connects the source, and sink of the data and allows you to apply any transformations or processing logic between the data flow.

It then executes the Pipeline job over an auto-scaling fleet of compute instances.

The Solution

My solution is a SpringBoot service (fondly named as data-extractor) that would run periodically and create the necessary Dataflow Pipelines. I used Gradle's KotlinDSL for dependency management.

The microservice would need access permissions to the following Google services - BigQuery, Google storage (used to store temporary files), Dataflow, Compute instances. So, I created a dedicated service-account with the necessary permissions. Service accounts are a special kind of account used by an application or a virtual machine (VM) instance, not a person.

I built the service and tested multiple runs which were all successful when I ran the service from the IDE or when I did ./gradlew run from the command line.

The service would successfully boot, connect to Vault to get the credentials, create the Dataflow job which would define the Pipeline and create necessary compute instance(s) that would process the data and then shut them down after the ETL job was done.

This roughly took 5 minutes for the entire job. The Google console UI shows a nice graph of the entire data flow with multiple user-defined and internal stages and has the logs and metrics for each step.

The Trouble:

We deploy all of our microservices into Google's Kubernetes Engine(GKE) and so once the service was in good shape, I did the usual process of building an uber fat-jar and dockerizing it.
Here's the Dockerfile that I copied from one of our other SpringBoot repositories.

FROM gcr.io/distroless/java:11
VOLUME /tmp
COPY build/libs/*.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-Djava.security.egd=file:/dev/./urandom", "-Dspring.profiles.active=${ENV_SPRING_PROFILE}", "-jar", "/app.jar"]

Assuming that ./gradlew build has run before successfully, it copies the fat-jar inside the image and then runs it when the container is started.

Pretty standard, right?

This is where everything started to fail. When I ran the container (locally as well as in the GKE), I observed that the service would boot successfully, connect to Vault to read the credentials, create the Dataflow job, start the instance but the job would just remain stuck.
The logs would just say that

Checking permissions granted to controller Service Account.

There would be a log of this printed every 5-6 minutes for up to an hour at which point the job would fail.

Here started the painful process of debugging. I desperately searched for any known bugs or issues or relevant StackOverflow answers, or anything I could find in Google documents but couldn't find anything.

The job would work successfully from IDE or with Gradle but will fail when Dockerized.

This happened 10/10 times or rather the hundreds of times that I tried.

The logs hinted at a possible permissions issue with the service-account. However, it did not make total sense as the execution was successful from within the IDE or when ran with Gradle. I made sure that there were no other GoogleCredentials present in my local environment and the service-account was the only credential injected as an environment variable. All the environment variables were the same except the part of containerization.

I enabled debug logs but that didn't give any useful insight either. The job would get stuck with the same error. Google console displayed failed job with the message:

Workflow failed. Causes: The Dataflow job appears to be stuck because no worker activity has been seen in the last 1h. Please check the worker logs in Stackdriver Logging. You can also get help with Cloud Dataflow at https://cloud.google.com/dataflow/support.

Frustrated, I tried everything from ssh-ing into the machine to checking network connectivity to reading syslogs but there was no clue anywhere.
I even opened a Google support case from our account but didn't get a helpful answer. Mostly, because they couldn't find anything wrong on their side.

I joked during the standup about this that I might have to deploy my laptop.

Between other tasks, I wasted 2 days on this thing - getting more and more frustrated and tired.

The Experiment

Working on a tight deadline, I decided to Dockerize my whole repo and run /gradlew run from inside it.
Creating a fat-jar and Dockerizing it

This shouldn't work. Right?

To my surprise it did.

It now worked 10/10 times. I was flabbergasted.

I was now more puzzled than before but also relieved that I was unblocked. I had wasted 2 days stuck on the issue - an issue I did not understand, unable to deploy the app. Those were 2 days of my life that I wasn't gonna get back.

Turns out there is something about the generated fat-jar from ./gradlew build that makes it fail to run on Dataflow.
I verified this by directly running the fat jar from the command line as well:

# data-extractor.jar is the fat uber jar containing all dependencies.
java -jar data-extractor.jar

It failed like the Docker container and at the same point.

Checking permissions granted to controller Service Account.

It had been a wild journey. I decided that I am going to deploy what's working while in parallel, I try to understand and figure out what's going on.

Here's my updated Dockerfile that copies the code into the container and runs gradlew classes to fetch the dependencies and compile the code.

FROM adoptopenjdk:11-jdk-hotspot

VOLUME /app/data-extractor
WORKDIR /app/data-extractor
# Copy source code
COPY src /app/data-extractor/src
# Copy gradle configuration and runner
COPY *.gradle.kts /app/data-extractor/
COPY gradlew /app/data-extractor/
COPY gradle /app/data-extractor/gradle

# This will install the dependencies in the image
RUN ./gradlew  --no-daemon classes

EXPOSE 8080
ENTRYPOINT ["./gradlew" "--no-daemon" "run"]

I don't know why a fat-jar doesn't work here. This is the first time I faced this and I would want to understand better. Perhaps, someone else in the community knows why and share it.

Are there more such cases?
How do people workaround them?
Is there something I can do better?

Sharing this so that someone else working on something similar can save time and frustration.

Conclusion

Computers and modern software are complex and hard. In the entire feat of integrating a mix of complex technologies like Vault, Kubernetes, BigQuery, Dataflow and the service logic itself, this little thing caused me the most pain. Parts that you think you understand well, can fail in unexpected ways when working together.
In the cloud-native environment with multiple moving parts, logs can sometimes be misleading. When that happens, it is important to inspect even the smallest things carefully. For most of the time, I suspected something around permissions and did a thorough inspection of roles and permissions. Taking a break and looking in a different direction helps.
I am sure there's a perfectly reasonable explanation of this. Maybe, it is even documented. But, I couldn't find it easily nor was the error obvious. Sharing such experiences is a good way of learning from each other without making the same mistakes. I have learnt and accepted that I don't know everything and it's fine.

If you have made so far, please let me know what you think.

Originally published on https://suspendfun.com

Java 9 to Java 13 - Top features

Raj Saxena — Fri, 25 Oct 2019 17:05:05 +0000

Programming tools, frameworks are becoming more and more developer friendly and offer better and modern features to boost developer productivity.

Java was for a long time infamous for having slow release trains.
However, keeping up with times, Java has moved to a cadence of releasing new features with a version upgrade every 6 months (every March and September).
Since then there have been a lot of cool features and tools that have been added to every java developer's toolset.

This is a quick summary of the latest features introduced between Java 9 to Java 13.

Java 9

Module system: Helps in modularisation of large apps. This helps to limit exposure of classes that are public in the module vs the true public api of the module.
Explicitly define dependencies and exports in module-info.java. Eg:

module chef {
  exports com.tbst.recipe;

  requires kitchen;
}

Here, the module chef depends on module kitchen and exports module recipe.

Jlink: Having explicit dependencies and modularized JDK means that we can easily come up with the entire dependency graph of the application. This, in turn, enables having a minimal runtime environment containing only the necessary to run the application. This can help to reduce the overall size of the executable jar.

jshell: An interactive REPL (Read-Eval-Print-Loop) for quickly playing with java code. Just say jshell on the terminal after installing JDK 9+.

Collection factory methods: Earlier, one had to initialize an implementation of a collection (Map, Set, List) first and then add objects to it. Finally, it's possible to create the immutable collections with static factory methods of the signature <Collection>.of(). This is achieved by having a bunch of static methods in each of the respective interfaces. Eg:

// Creates an immutable list
List<String> abcs = List.of("A", "B", "C");

// Creates an immutable set
Set<String> xyzs = Set.of("X", "Y", "Z");

// Creates an immutable Map
Map<String, String> mappings = Map.of("key1", "value1", "key2", "value2");

Other features
- Stream API gets more functions like dropWhile, takeWhile, ofNullable.
- Private interface methods to write clean code and keep things DRY when using default methods in interfaces.
- new HTTP2 API that supports streams and server based pushes.

Java 10

Local-Variable Type Inference: This enables us to write more modern Kotlin/Scala/Typescript like syntax where you don't have to explicitly declare the variable type without compromising type safety. Here, the compiler is able to figure out the type because of the type of the value on the right hand side in case of assignments. Eg:

var list = new ArrayList<String>();  // infers ArrayList<String>
var stream = list.stream();          // infers Stream<String>

In cases where the compiler cannot infer the value or it's ambiguous, you need to explicitly declare it. More details here

Parallel Full GC for G1: Improves G1 worst-case latencies by making the full GC parallel.

Experimental Java-Based JIT Compiler: Enables the Java-based JIT compiler, Graal, to be used as an experimental JIT compiler on the Linux/x64 platform.

Heap Allocation on Alternative Memory Devices: Enables the HotSpot VM to allocate the Java object heap on an alternative memory device, such as an NV-DIMM, specified by the user.

Root certificates in JDK: Open-source the root certificates in Oracle's Java SE Root CA program in order to make OpenJDK builds more attractive to developers, and to reduce the differences between those builds and Oracle JDK builds.

Java 11

New String methods: String class gets new methods like isBlank(), lines(), repeat(int), unicode aware strip(), stripLeading() and stripTrailing().

New File Methods: writeString(), readString() and isSameFile().

Local-Variable Syntax for Lambda Parameters: Allow var to be used when declaring the formal parameters of implicitly typed lambda expressions.
This is introduced to have uniformity with use of var for local variables. Eg:

(var x, var y) -> x.process(y)   // implicit typed lambda expression

// One benefit of uniformity is that modifiers, notably annotations, can be applied
// to local variables and lambda variables without losing brevity.
(@Nonnull var x, @Nullable var y) -> x.process(y)

JEP 328: Flight Recorder: JFR is a profiling tool used to gather diagnostics and profiling data from a running Java application.
Its performance overhead is negligible and that’s usually below 1%. Hence it can be used in production applications.

Removed the Java EE and CORBA Modules: Following packages are removed: java.xml.ws, java.xml.bind, java.activation, java.xml.ws.annotation, java.corba, java.transaction, java.se.ee, jdk.xml.ws, jdk.xml.bind

Implicitly compile and run No need to compile files with javac first. You can directly use java command and it implicitly compiles. This is done to run a program supplied as a single file of Java source code, including usage from within a script by means of "shebang" files and related techniques. Of course for any project bigger than a file, you would use a build tool like gradle, maven, etc.

Java 12 - Released March 19, 2019

Switch expression😎 The new switch expression expects a returns value. Multiple matches can go on the same line separated by comma and what happens on match is marked with ->.
Unlike the traditional switch, matches don't fall through to the next match. So you don't have to use break; and this helps prevent bugs. Eg:

String status = process(..., ..., ...);
var isCompleted = switch (status) {
    case "PROCESSED", "COMPLETED" -> true;
    case "WAITING", "STUCK" -> false;
    default -> throw new InconsistentProcessingStateException();
};

The switch expression is introduced as a preview and requires --enable-preview flag to the javac or enabling it in your IDE.

File byte comparison with File.mismatch(). (From Javadoc) Finds and returns the position of the first mismatched byte in the content of two files, or -1L if there is no mismatch.

Collections.teeing Streams API gets a new function that applies 2 functions (consumers) on the items and then merges/combines the result of those 2 functions using a third function to produce the final result.

From Javadoc - Returns a Collector that is a composite of two downstream collectors. Every element passed to the resulting collector is processed by both downstream collectors, then their results are merged using the specified merge function into the final result.

String methods: indent(int n), transform(Function f).

Smart cast instanceOf can be used now to do a smart cast as below:

...
} catch (Exception ex) {
    if(ex instanceOf InconsistentProcessingStateException ipse) {
        // use ipse directly as InconsistentProcessingStateException
    }
}

JVM improvments: Low pause GC with Shenandoah, micro-benchmark capabilities, constants API and other improvements.

Java 13 - Released September 17, 2019

Multi-line texts: It's now possible to define multiline strings without ugly escape sequences \ or appends. Eg:

var jsonBody = """
    {
        "name": "Foo",
        "age": 22
    }
""";

This is introduced as a preview and requires --enable-preview flag to the javac or enabling it in your IDE.

String gets more methods like formatted(), stripIndent() and translateEscapes() for working with multi-line texts.

Switch expression Still in preview and based on feedback now supports having : yield syntax in addition to -> syntax. Hence, we can write

String status = process(..., ..., ...);
var isCompleted = switch (status) {
    case "PROCESSED", "COMPLETED": yield true;
    case "WAITING", "STUCK": yield false;
    default: throw new RuntimeException();
};

Socket API reimplemented with modern NIO implementation. This is being done to overcome limitations of legacy api and build a better path towards Fiber as part of Project Loom

Z GC improved to release unused memory.

Wow, it's getting crazy out there.
If you are a developer that started a decent size Java project recently in the hopes that you would use the latest features and keep yourself and the project updated with the latest versions, do you feel a pressure to catchup with these frequent releases?

Add your comments below or tweet them to me.

A parting gift - I use Jenv to easily switch between different jdk versions locally while switching between different projects. It's pretty cool to manage multiple jdk versions.

Please note, the features that I talk about here are the ones that, I believe, either add cool features or increase developer productivity the most.

This is not an exhaustive list.

References and good articles:

I originally posted it here
Java 9 - pluralsite
Java 10 - techworld, dzone
Java 11 - geeksforgeeks, journaldev
Java 12 - journaldev, stackify
Java 13 - jaxenter, infoworld, dzone