Forem: Eyal Katz

Using the Redis Command-Line Correctly

Eyal Katz — Wed, 07 Jul 2021 08:21:50 +0000

Zora Cooper is a web dev and technical writer who I have had worked with on a number of projects. She wrote a perfect guide on how to properly use the Redis command-line including examples and a step by step tutorial.

Redis is an open-source in-memory data store. You can use Redis as a database, message broker, or cache. It also supports Lua script evaluation so you can build automation scripts or custom operations on top of your key-value Redis store.

Typically, developers use language-specific client libraries when building apps with Redis. These clients abstract Redis’ command-line interface (CLI), making it easier to work with but more work to set up. Client libraries are great when you’re already using Redis in a project but not as useful for trying things out or debugging a production data store. For that, the Redis CLI is your best bet.

The Redis CLI is a great choice for prototyping before implementation, troubleshooting, and experimentation because it’s faster to get started with and more flexible. It provides tab completion, a history of your commands, and all the functionality you need to read, write, update, delete, or debug data in Redis.

In this article, you’ll learn everything you need to know about using the Redis command-line correctly:

Along the way, I’ll offer some context to help you understand when you might use each of these features.

Connecting to Redis

Before you start using the Redis command-line, make sure you have it installed. You can find installation instructions here, and once complete, you can check if your installation was successful by running:

$ redis-cli --help
redis-cli 6.2.3
Usage: redis-cli [OPTIONS] [cmd [arg [arg ...]]]
...

Redis connects to a server at 127.0.0.1 on port 6379 by default. You can change the hostname using the -h flag and the port using the -p flag. To connect to a Redis server at example.com:3001, run the following:

$ redis-cli -h example.com -p 3001
example.com:3001>

If a password and username have been set for the server, you can provide them with the -u flag for username and -a for password. Using these flags is not recommended as anyone who retrieves your bash history will be able to see your password, so it’s usually better to use the AUTH command in interactive mode.

$ redis-cli
AUTH <username> <password>

You can also set the password using the REDISCLI_AUTH environment variable.

Selecting a Redis Database

A Redis server has 16 databases by default. The databases are zero-indexed, with the first database being 0. You can check the actual number by running redis-cli config get databases.

In interactive mode, the database number is displayed in the prompt within square braces. For example, 127.0.0.1:6379[13] shows that the 13th database is in use.

To select a database when connecting, pass its database number with the -n flag:

$ redis-cli -n 1
127.0.0.1:6379[1]>

If you’d like to combine all of the above commands, use the -u flag to specify a complete server URI:

$ redis-cli -u redis://<username>:<password>@<host>:<port>/<database-number>
$ redis-cli -u redis://<password>@<host>:<port>/<database-number>
$ redis-cli -u redis://<host>:<port>/<database-number>

CRUD Operations Using the Redis Command-Line

The commands used for CRUD operations differ depending on the type of data being handled, but first, you have to understand a bit about how Redis stores this data.

Redis keys are distinct identifiers associated with a value. They are used to perform operations on values (deletes, reads, and updates, for example). The values we will look at include strings, lists, sets, sorted sets, and hashes:

Keys and string values can take on the form of numbers, characters, or files.
A list is a string collection sorted by the order the elements were inserted.
A set is an unsorted list where elements must be unique.
A sorted set is just a set where each element is assigned a score, and the set is sorted by score.
Hashes are maps containing field-value string pairs.

Redis also supports streams, hyperloglogs, and bit arrays, which you can read more about in the documentation.

Adding Data to Redis

Keys with string values are created with the SET command.

$ redis-cli set cat-count 10
OK

To create a key with a list value and add elements to it, use either RPUSH or LPUSH commands. LPUSH adds elements to the beginning of a list, and RPUSH adds them to the end.

$ redis-cli rpush cat-breeds persian ragdoll bengal
(integer) 3
$ redis-cli lpush hairless-cats sphynx
(integer) 1

SADD (short for “set-add”) will create a key with a set value and add members to it.

$ redis-cli sadd cities london zurich berlin 
(integer) 3

For keys with sorted set values, use the ZADD command.

$ redis-cli zadd fruits 52 apple 47 orange
(integer) 2
HSET and HMSET create keys with hash values and add fields to them. HMSET sets multiple hash fields while HSET sets just one at a time.

$ redis-cli hset cat1 breed norwegian
(integer) 1
$ redis-cli hmset cat2 breed bobtail coat long
OK

Updating Data Using the Redis CLI

Updating string values in a Redis store can be done using one of several commands:

The SET command can update an existing key with a string value
APPEND appends a value
DECR decrements a value
GETSET sets a new value and returns the old value
INCR increments a value
SETNX sets a value if it exists Other update commands for keys with string values can be found using the help @string command in Redis’s interactive mode.

There are also several commands for updating a list value:

LPUSH and RPUSH (which I mentioned above) can be used to add new values to a list
RPUSHX appends an element to the end of a list if the key exists
LPUSHX adds an element to the beginning of a list if the key exists
LINSERT inserts a new element between existing elements in a list

Similarly, you can use help @list in interactive mode to see all the Redis commands that can manage list values.

Adding and updating data manually in Redis is useful when you want to replicate production conditions in a test environment. For example, if you have a specific list that is causing errors in production, you could add it to your local environment and trace the error through your code.

Reading Data from Redis

Reading data from Redis is a good way to figure out if specific values are causing application errors. To retrieve a key with a string value, use the GET command.

$ redis-cli get cat-count
"10"

If a value does not exist for this key, GET replies with a nil value.

Since GET only works with string values, you need the LRANGE command to get list values.

$ redis-cli lrange cat-breeds 0 -1
1) "persian"
2) "ragdoll"
3) "bengal"

“0” in the example above specifies the start index of the list and “-1” the end. This command can also be helpful when attempting to fetch a subset of elements from a large list.

The SMEMBERS command will fetch all the members in a set, and the ZRANGE command returns members between a start and end index in a sorted set.

$ redis-cli zrange fruits 0 -1
1) "orange"
2) "apple"

Finally, HGETALL, HGET, and HMGET will retrieve hash values. HGETALL will return all the field-value pairs in a hash if you would like to see it entirely. If you would like to just see one field-value pair, you would use HGET. HMGET would be useful if you’d like to view multiple specific field-value pairs of a hash.

$ redis-cli hgetall cat1
1) "breed"
2) "norwegian"

Reading all these values in the command line might be good for small lists or sets, but if you’re dealing with thousands of rows, this won’t work. Using the --csv flag, you can get CSV output that can be opened in a text editor or spreadsheet.

Deleting Data

Another common operation in Redis is deleting data. You might use this to delete problematic caches, clear up a backlogged message queue, or reset your local Redis data.

The DEL command will delete one or more values by their keys. If you try to delete a non-existent key, it is ignored, and the server will reply with the number of successfully deleted keys.

$ redis-cli del cats
(integer) 1

If you realize you made a mistake, you can restore a key using the RESTORE command.

There are other type-specific delete commands to remove elements, members, or fields from a key:

To remove specific elements from a list instead of deleting the whole key, you would use LREM
To remove a subset of elements in a list, you could use LTRIM
For sets, SREM would remove one or more members
SPOP would be useful for removing a member from a set and returning it as output
ZREM will remove a single or multiple members from a sorted set
HDEL would help delete a field from a hash For a full list use help @ in interactive mode.

Bulk Data Insertion Using the Redis CLI

If you want to migrate data from one Redis instance to another or upload a large sample data set, you could write a bash script to run SET repeatedly, or you can add data in bulk using the --pipe flag. This flag takes the path of the file containing all your Redis operations.

Since different data types are created using different commands, you have to specify the command when inserting data, even in bulk. For example, consider the insert-data.txt file below:

SET counter 1
SET tracker 2
LPUSH cities Paris Berlin
LPUSH countries France Germany
RPUSH fruits apple orange 
RPUSH vegetables carrot cucumber
SADD tiers s a b c d
ZADD scores 52 jane 47 paul
HMSET user01 name Jane age 23
HMSET user02 name Reggie age 20

You can insert the data above by running:

$ cat insert-data.txt | redis-cli --pipe
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 10

It’s slightly arduous to get all these commands right, but it will be faster than connecting to Redis multiple times to loop over a shell command.

Running Lua Scripts with Redis CLI

Lua is a lightweight scripting language that you can use to interact with Redis servers. If you’d like more flexibility when using Redis and take advantage of features such as comments, variables, conditional statements, functions, and loops, Lua scripts can be pretty helpful.

Lua scripts are evaluated when the --eval flag is used. This flag takes a path to a Lua file, but you can also provide keys or other arguments to the script.

For example, this add-counters.lua script adds five counters to the Redis database:

for i=1, 5 do
    redis.call('set', 'counter' .. i, i)
end

To run it:

$ redis-cli --eval add-counters.lua
(nil)

Keys are made available within the script in the KEYS table and arguments in the ARGV table. Here is an example of how you would use them:

for i=1, #KEYS do
    redis.call('hmset', 'user0' .. i, 'name', KEYS[i], 'age', ARGV[i])
end

The keys and arguments are passed to the script as a comma-separated list:

$ redis-cli --eval add-users.lua Jane , 23 , John , 20 , Paula , 30
(nil)

Redis provides a Lua debugger that’s especially useful for debugging complicated Lua scripts. To use it, pass the --ldb flag.

$ redis-cli --ldb --eval add-users.lua Jane , 23 , John , 20 , Paula , 30
Lua debugging session started, please use:
quit    -- End the session.
restart -- Restart the script in debug mode again.
help    -- Show Lua script debugging commands.
* Stopped at 1, stop reason = step over
-> 1   for i=1, #KEYS do
lua debugger>

The debugger will save you a lot of time as you’re tweaking your Lua scripts.

Interactive vs Command-Line Arguments

The Redis CLI can be called using command-line arguments or by starting an interactive shell.

When you specify command-line arguments using redis-cli, the Redis client sends them to the server, prints the results, and then exits. You will have to run redis-cli again with new commands to issue more. That’s how all the examples in this tutorial have worked so far.

In interactive mode, after a command is processed by the server, results are printed, and you can continue issuing further commands in the prompt. You do not need to keep typing redis-cli. The CLI can be run in interactive mode by executing redis-cli without any arguments:

$ redis-cli
127.0.0.1:6379[2]>

In interactive mode, the Redis CLI includes command completion, hints, and better help information, so you might spend a lot of time here when you’re debugging Redis.

Seeing a Summary of Redis Storage Contents

Some issues in Redis arise when your server is running a different version in testing than in production. The INFO command provides information and statistics about the Redis server, including version, operating system, uptime, clients, memory consumption, commands processed, and keyspace information.

$ redis-cli info
# Server
redis_version:6.2.3
redis_git_sha1:00000000
redis_git_dirty:0
redis_mode:standalone
os:Linux 5.8.0-53-generic x86_64
arch_bits:64

You can check the number of keys in a database using the DBSIZE command. The -n flag specifies the database you’d like to check. This can be especially useful when debugging and tracking keys added to your database.

$ redis-cli -n 0 dbsize

For real-time information and stats about the server, use the --stat flag. Every second it prints out the number of clients connected, requests, connections, and keys.

$ redis-cli --stat
------- data ------ --------------------- load -------------------- - child -
keys       mem      clients blocked requests            connections          
1          855.56K  1       0       122 (+0)            36          
1          855.56K  1       0       123 (+1)            36          
1          855.56K  1       0       124 (+1)            36          
1          855.56K  1       0       125 (+1)            36

This can come in handy when you need to find out the number of clients connected to a server, how requests are being processed, or how many connections have been made recently.

More Redis Command-Line Features

While I offered an overview of some of the most important features Redis’s command-line interface has to offer, there are many other features you might want to learn.

For example, you can use the MONITOR command to view all server requests in real-time. Redis also provides support for pub/sub messaging, which you can access via the command-line.

Knowing how to use the Redis CLI enables you to take full advantage of its powerful features without the setup time required when using client libraries in your code. The Redis CLI is especially helpful for learning how Redis works, solving server issues, testing commands quickly, and Redis server monitoring, so it’s worth learning how to use it correctly.

How to Debug Race Conditions Between Threads in Java

Eyal Katz — Mon, 28 Jun 2021 07:11:24 +0000

Derrick Wadek is a gifted full-stack Android engineer with a wealth of experience and wisdom to share. He wrote this excellent guide on Java debugging to remedy complex race conditions.

How many days off have been marred by debugging race conditions and deadlocks in complex multithreaded, Java code? You’ve probably vowed, Never again and embarked on a quest to always catch race condition errors early by writing tests and debugging.

Multithreaded applications are a great way to improve performance, but they also make routine tasks like debugging a little more complicated. In this tutorial, you’ll learn an overview of what race conditions are, what multithreaded Java code is used for, and finally how to debug race conditions in Java using a few different methods.

What Is Multithreading?

Multithreading is the ability for a program to run two or more threads concurrently, where each thread can handle a different task at the same time.

More specifically, a program is divided into two or more sub programs, which can be implemented at the same time in parallel:

private static void computeCounter(){
       Counter counter = new Counter() ;
       Thread threadOne = new Thread(getRunnable(counter , "Thread one count is: "));
       Thread threadTwo = new Thread(getRunnable(counter , "Thread two count is: "));
       threadOne.start();
       threadTwo.start();
   }

Benefits of a Multithreaded Application

It enables you to do multiple things at a time.
You can divide a long process into threads and execute them in parallel, which will eventually increase the speed of program execution.
Simultaneous access to multiple applications.
It improves performance by optimizing available resources, especially when your PC has multiple CPUs.

By default, all your program code runs on the main thread, which means if you have a 4-core CPU, only one core will be used to execute your code. This means that code will be executed one after the other, and only until the previous program is finished will the next be executed. On the other hand, if you use multithreading with three threads running concurrently, each thread will occupy separate cores and the program will be executed in parallel, reducing the amount of time to execute your program.

The following is a simple program that computes flight charges for each flight category. The category will be checked and a charge set on each.

public class FlightCategories {

    static final int FIRST_CLASS = 1;
    static final int BUSINESS_CLASS = 2;
    static final int ECONOMY_CLASS = 3;
    public static void main(String[] args){
        System.out.println("===== Application Running =====");
        try {
            FlightCharges firstClass = new FlightCharges(FIRST_CLASS);
            firstClass.computeFlightCharges();
            FlightCharges businessClass = new FlightCharges(BUSINESS_CLASS);
            businessClass.computeFlightCharges();
            FlightCharges economyClass = new FlightCharges(ECONOMY_CLASS);
            economyClass.computeFlightCharges();
            System.out.println("First class charges are "+firstClass.getCharges());
            System.out.println("Business class charges are "+businessClass.getCharges());
            System.out.println("Economy class charges are "+economyClass.getCharges());
        } catch (Exception e){
            System.out.println("Encountered error "+e.getMessage());
        }
    }
}

public class FlightCharges {
    private int category = 0;
    private double charges = 0;
    public FlightCharges(int category) {
        this.category = category;
    }
    public double getCharges() {
        return charges;
    }
    private void setCharges(double charges) {
        this.charges = charges;
    }
    public void computeFlightCharges(){
        try {
            switch (category){
                case 1:
                    setCharges(500.0);
                    break;
                case 2:
                    setCharges(250.0);
                    break;
                case 3:
                    setCharges(150.0);
                    break;
            }
        } catch (Exception e){
            System.out.println("Exception "+e.getMessage());
        }
    }
}

Observe the time it took to execute it.

It took roughly 443 ms to execute this simple program. Now repeat the same process using multithreading as shown below:

public class FlightCategoriesMultithreading {
    static final int ECONOMY_CLASS = 3;
    static final int BUSINESS_CLASS = 2;
    static final int FIRST_CLASS = 1;
    public static void main(String[] args){
        System.out.println("===== Application Running =====");
        try {
            FlightChargesMultithreading firstClass = new FlightChargesMultithreading(FIRST_CLASS);
            Thread t1 = new Thread(firstClass);
            t1.start();
            FlightChargesMultithreading businessClass = new FlightChargesMultithreading(BUSINESS_CLASS);
            Thread t2 = new Thread(businessClass);
            t2.start();
            FlightChargesMultithreading economyClass = new FlightChargesMultithreading(ECONOMY_CLASS);
            Thread t3 = new Thread(economyClass);
            t3.start();
            System.out.println("First class charges are "+firstClass.getCharges());
            System.out.println("Business class charges are "+businessClass.getCharges());
            System.out.println("Economy class charges are "+economyClass.getCharges());
        } catch (Exception e){
            System.out.println("Encountered error "+e.getMessage());
        }
    }
}

public class FlightChargesMultithreading implements Runnable{
    private int category = 0;
    private double charges = 0;
    public FlightChargesMultithreading(int category) {
        this.category = category;
    }
    public double getCharges() {
        return charges;
    }
    private void setCharges(double charges) {
        this.charges = charges;
    }
    @Override
    public void run() {
        try {
            switch (category){
                case 1:
                    setCharges(500.0);
                    break;
                case 2:
                    setCharges(250.0);
                    break;
                case 3:
                    setCharges(150.0);
                    break;
            }
        } catch (Exception e){
            System.out.println("Exception "+e.getMessage());
        }
    }
}

It only took 150 ms to execute the same program! For a simple program like this, this might not seem much, but the more complex a program gets, multithreading becomes crucial to performance.

Challenges of Multithreaded Applications

Concurrency greatly improves performance but at the detriment of debugging. Why? Because it’s easy to catch errors in a single thread but notoriously difficult to replicate bugs when tracking multiple threads.

Take, for example, the previous code that simply computed flight charges depending on flight categories. When logging the code, it will take some time to check the category of flight and set charges to it. The extra time could often fix the issue—most multithreading bugs are sensitive to the timing of events in your application, so running your code under a debugger changes the timing. Your debugger may fail to replicate the bug.

The bottom line is that multithreading is an important and useful aspect of development, but comes with its fair share of challenges.

What Are Race Conditions?

Detecting race conditions is particularly difficult. A race condition is a behavior that’s dependent on a “race” between two threads as to which one will be executed first. Two or more threads access the same variable or data in a way where the final result stored in the variable depends on how thread access to the variable is scheduled.

Thread (t1) in the FlightCategoriesMultithreading.java example above will certainly finish first in most ideal situations and this will be your expected result. The problem occurs in rare cases when one of the remaining threads (t2 or t3) finishes first. In a more complex multithreaded application running hundreds of threads, this is a potential debugging nightmare and replicating these bugs would be hard!

Essentially, the two threads read and write the same variables or data concurrently instead of consecutively, using either of these patterns:

Check, then act.
Read, modify, write, where the modified value depends on the previously read value.

Because the thread access to the variables or data is not atomic, it can cause some issues. Here’s a diagram to illustrate this:

What Should Happen

First the CPU 1 thread will read the value of the counter class into a local CPU register where the thread is running, increment the value by 1, and write that value back to main memory. When the second thread (CPU 2) reads the count, it reads the value 1 incremented by 1 and then writes the value 2 back to main memory.

What Actually Happens

Instead of sequential access as shown in the diagram above, the threads use interleaved access. Thread 1 reads the value 0 from the counter into a CPU register, and at the exact same time, thread 2 reads the value 0 from the counter into a local CPU register for thread 2.

This causes both threads to increment the value that they have stored in their own CPU and write it back. So thread 1 will increment from 0 to 1 and write back 1, but thread 2 will also increment from 0 to 1 and write back 1. So, we expect the counter to read 2, but because of the interleaved access, it reads 1.

As an example, the following code snippet simply iterates over the counter a million times and prints out the value of both threads to the console:

public class RaceConditionExample {
    private static Runnable getRunnable(Counter counter , String output) {
        return () -> {
            for (int i = 0 ; i < 1_000_000 ; i++){
               counter.counterThenGet();
            }
              System.out.println(output + counter.getCount());
        } ;
    }
    public static void main(String[] args) {
        computeCounter();
    }
    private static void computeCounter(){
        Counter counter = new Counter() ;
        Thread threadOne = new Thread(getRunnable(counter , "Thread one count is: "));
        Thread threadTwo = new Thread(getRunnable(counter , "Thread two count is: "));
        threadOne.start();
        threadTwo.start();
    }
}
The following code increments the counter after each iteration:

public class Counter {
    private int count = 0 ;
    public void counterThenGet() {
        this.count++ ;
    }
    public long getCount() {
        return count;
    }
}

How to Prevent Race Conditions

A critical section is where the race condition occurs.

public void counterThenGet() {
        this.count++ ; //CRITICAL SECTION
    }
The way to fix the critical section is to make it *atomic*, meaning that only one thread can execute within the critical section at a given time. Making the critical section atomic yields the *sequential access* to counter.

To make it atomic, wrap the section in a synchronized lock:

public class CounterWithSynchronizedLock {
private int count = 0 ;
private final Object lock = new Object() ;
public void counterThenGet() {
synchronized (lock){
this.count++ ;
}
}
public long getCount() {
return count;
}
}


Only a single thread can execute within the *lock** at a given time, avoiding a situation where two threads read the value of the counter, then increment it. Whatever thread is going to be executed will read the value, increment it, and write it back as a single atomic operation.

## The Importance of Visibility in Multithreading
The importance of *visibility* (the memory that an executing thread can see once it’s written) in multithreaded applications cannot be belabored. When thread 1 writes to the memory before thread 2 reads it, the result is a race condition. It’s therefore imperative for Java developers to design, write, and test multithreaded Java applications with the basic tools of thread synchronization:

- **Using a synchronized keyword**: A synchronized keyword ensures that an unlock happens before every subsequent lock on the same monitor.
- **Using a volatile keyword**: A volatile keyword ensures that a write to a variable happens before subsequent reads of that variable.
- **Static initialization**: Static initialization ensures that the entire program is executed once when the class is first accessed and creates an instance of it. This is done by the class loader so that the JVM guarantees thread safety.

## Debugging Race Conditions
There are a couple of ways to debug multithreaded Java applications, but we’ll focus on the three most common ways here.

### 1. Run Code as Sequential Code
Not every bug is related to race conditions, so it’s possible that debugging in a sequential environment would be easier. Alter your code so that it runs sequentially in order to make sure that the bug does not appear in a sequential run.

For example, `#program omp` parallel for:

public void doSomething(){
for (int i = 0; i < n; i++)
{
// some code here
}
}
Becomes #pragma omp parallel for num_thread(N)

public void doSomething(){
OMP.setNumThreads(N);
//omp parallel for
for (int i = 0; i < n; i++)
{
// some code here
}
}

### 2. Use Log Statements
From within a thread, use logging statements to output debug information to the debugger so that you can form a post-mortem chain that can be followed. For example:

LOG.debug(PrintStringDataFromObject)




### 3. Use IntelliJ Debugging Tools
Using the [IntelliJ IDEA debugger](https://lightrun.com/debugging/how-to-debug-remotely-in-intellij/), you can **test multithreaded code and reproduce race condition bugs**.

In our multithreading example, the two threads share a counter. Each thread makes a million iterations while incrementing the counter at each iteration consequentially. This would mean that both threads should give you 2 million iterations in total.

To test this scenario, use breakpoints as provided by IntelliJ IDEA.

Set a breakpoint at the getter that fetches the counter.

![Public Class Counter](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sftlmd6jbkncjr7al4xz.png)

You will then configure the breakpoint to only suspend the thread in which it was hit. This suspends both threads at the same line. To do this, right-click on the breakpoint, then click **Thread**.

![Public Long GetCount](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cy4escuwgbrcomf0f5tu.png)

To initiate debugging, click **Run** near the **RaceConditionExample** class and select **Debug**.

![Run Race Condition Example](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zvj0jdlb2amwmfra0j9i.png)

Thread 1 prints a result to the window console that is not 2 million. This is an expected result.

![Race Condition Example](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k66bctjvz598tthl5bzb.png)

Resume the thread by pressing F9 or clicking the **Resume** button in the upper left of the **Debug** window.

![Race Condition Example Solution 1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ppmtwilzh2chqyxjert8.png)

From the results, you can see that race conditions are realized. The second thread should be 2 million but instead 1.9 million is printed.

Now run the second code with atomic implementation of the critical section and observe the differences.

![Race Condition Example Solution 2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7st4kqlkc8vgxk7prbe4.png)
![Race Condition Example Solution 3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3aiq8658upxevc6diuua.png)
From the images above, notice that your race condition bug has been solved and your second iteration totals to 2 million, a result that your code anticipated as the correct output.

## Conclusion

Multithreading has lots of advantages, but can quickly become a debugging nightmare. Mastering the art of debugging is one of the most useful skills that you as a Java developer can use to make your development process smooth.

How Culture Impacts Technology Choice: A Review of Netflix’s Use of Microservices

Eyal Katz — Mon, 31 May 2021 09:56:28 +0000

My friend Itiel Shwartz is the CTO and co-founder of Komodor, a startup building the next-gen troubleshooting platform for Kubernetes. He wrote an article examining how Netflix's use of microservices has completely transformed and revolutionized the streaming industry as we know it. He is an avid public speaker that loves talking about things like infrastructure, Kubernetes, Python observability, and R&D culture.

I recently had the opportunity to read the book “No Rules Rules: Netflix and the Culture of Reinvention” by Reed Hastings and Erin Meyer of Netflix, and it dawned on me that while this book wasn’t at all focused on Netflix’s technology, the global company-wide culture had a significant impact on its technology choices. The book focuses on the many times Netflix had to reinvent itself and transform its business in order to revolutionize the entertainment industry. This revolution was made possible by an equally radical and formerly unheard of approach to a company culture that not only influenced how we build technology today but also thinks about company culture on a global scale.

Needless to say, Netflix’s technology choices went on to influence the way the entire industry thinks about architecture built for massive scale. In this post, I’d like to take a look at how Netflix’s culture was the backbone that enabled its bleeding edge technology decisions — and specifically microservices. Some of what I write here is actually the direct influence of my own personal love-hate relationship with microservices. This book afforded me a newfound perspective on how to succeed at building challenging technology architectures to enable the business, by ensuring the right cultural mechanisms are in place first. I’ll posit that if not for Netflix’s unique approach to company culture, their technology choices would not have been possible either, and ultimately neither would their success.

My Backstory

So just to dive into my inspiration for this post a bit more, my love-hate relationship with microservices began when I was a software engineer at eBay trying to tame a crazy monster of a monolith. It was so complex and vast and no one really understood all of it, that I learned how truly difficult it is to break up a monolith that is the beating heart of a business.

Before kickstarting Komodor, I had a couple more opportunities to work on building microservices. The first was a gradual breakdown of a monolith, and the other was building a microservices architecture from the ground up. All of this experience provided me the opportunity to build the architecture that would drive our business from day one at my new venture, and I try to apply everything I have learned all the time.

And just so we’re on the same page, if you need a refresher on microservices, a great place to start would be this foundational piece by Martin Fowler of ThoughtWorks. Another good read would be this newer piece by Segment, which provides a much more practical perspective on migrating to microservices and the challenges involved.

Netflix’s 6 Principles of Building Culture AND Great Technology

So how does this all come together? In their book Reed Hastings and Erin Meyer outline the core cultural principles that made their success possible, I’d like to compare these side by side with the cultural practices that enable engineering organizations to successfully deploy and maintain microservices architecture — which wasn’t actually the premise behind the book, but it certainly got me thinking about the places that engineering and culture intersect.

1. Principle One: Build Up Talent Density

Netflix did not compromise on talent when it came to hiring. It hired the best possible people for each position, which was the foundation to their success. This is true by orders of magnitude when it comes to microservices architecture. Microservices require a great capacity for learning, as the people maintaining the architecture will continuously encounter new challenges and problems they’ve never confronted before. If we’ll be completely honest, it is just not possible for all developers and DevOps engineers to be the best in their field.

While some companies have some pretty great engineers, Netflix has world-class engineers. This made it possible for Netflix to overcome challenges that many other engineers likely wouldn’t have had the capacity to work through as pioneers of complex architecture. It had some of the best engineers in the world working on the newly discovered challenges microservices surfaced, they overcame these and later enabled the industry to do so as well by sharing from this experience.

2. Second Principle: Begin Removing Controls

Netflix’s radical approach to culture can be seen in the most basic of company guidelines from expenses through vacation approvals, employee trust was built-in to the culture as a foundational principle. That said, this was only made possible by having the right infrastructure in place to support this culture as well. We’ll get into that later.

This cultural practice is one of the core factors that can influence the success of managing microservices architecture at companies that are not Netflix. I’ve found that even companies that have made the decision to go down the microservices route do so while old processes and mechanisms still exist, and this highly hinders the ability to properly take advantage of the migration. If companies still maintain rules like approvals for each new change or commit introduced into a system, or even reverting changes, they essentially have a distributed monolith culture.

So on the one hand, they have an architecture that can scale autonomously — but their people can’t. You need to trust your people to make the right decisions in order to extract the real benefits and velocity microservices make possible. Without this inherent trust and autonomy, you’ll essentially have the worst of both worlds. Netflix took this approach with all things, not just engineering, and it repaid them kindly in achieving success.

3. Third Principle: Maximize Candor Using Feedback Circles

Many times the decision to move to microservices is a top-down decision. What oftentimes happens is that the decision-makers don’t really have any concept of the complexity of the task at hand — breaking up their architecture into smaller and smaller pieces. On the flip side, those who need to execute on the top-down decision don’t have the agency to influence the process. The move to microservices often surfaces a lot of complications, where things that previously worked suddenly stop working. A critical piece in moving quickly and resolving these issues, is short feedback loops.

At Netflix, they were required to raise a flag and communicate problems when they surfaced, whereas other companies often do not enable a culture of such candor. Companies that don’t have an atmosphere that enables this kind of communication and candor will be set up for failure with complex architectures, where their teams will constantly be firefighting problems instead of architecting more long-term solutions built on collaborative thinking and communication as oftentimes the voices of the people in the trenches aren’t heard.

4. Fourth Principle: Build Supporting Infrastructure

To be successful, you need to know that as a first step you’ll be spending a lot of time in building supporting tools for your developers and DevOps teams to enable them to be productive in this transformation.

Like all “shift left” practices, if you only remember to do so after the fact, you will find it costing you 100x in time, resources, and even budget to build the supporting mechanisms to enable the teams to succeed. Succeeding with a loosely coupled architecture, that requires a lot of trust in your engineers’ decision-making, requires you to provide those at the bottom of the chain with more “powers.” To support their innovative approach to expense reporting and vacation requests for example, Netflix built the proper checks and balances into the system in advance, to ensure on the one hand that it couldn’t be abused, but that it could also scale with them as they grow and maintain this critical piece of their culture.

5. Fifth Principle: Celebrate Success and Sunshine Failures

I think the industry as a whole learned the importance of celebrating successes, and today nearly every organization that prides itself on its company culture has some way of demonstrating appreciation for good work. However, on the flip side, one thing that companies can still learn from Netflix is their approach to failures. At Netflix failures are not brushed under the rug, they are analyzed, to be able to learn from each experience in order for the organization to grow and evolve, and not make the same mistakes twice.

When you migrate to microservices you are going to have A LOT of failures at first, and if you don’t take the opportunity to learn from these, you will be destined to repeat the same mistakes. In the spirit of this pillar at Netflix, in a landscape of many partial successes and common failures — it is important to be self-aware and have the sensitivity to learn from these failures to figure out the root cause, and ultimately how to fix it, so it doesn’t repeat itself.

6. Sixth Principle: Leading with Context, not Control

With monolithic architecture, it is very easy to have control, as there is only one service and everyone for the most part understands what is happening in their domain and aligns themselves to the practices in place. With microservices engineering management simply cannot understand everything that is happening in the system as a whole, and each team needs to have a high degree of autonomy. That’s why transparency is extremely important coupled with providing a lot of context around what is happening, when it comes to being able to make decisions in this fast-moving engineering environment.

With microservices, it is impossible to micromanage and therefore you have to provide the general vision and direction and trust that your teams will align to these shared goals. Netflix understood how to do this really well, and this instilled a bottom-up joint purpose that fueled much of their success.

What We Have Learned

So basically to wrap it up, microservices has proven to be a very powerful architecture to power modern, complex business, but you need to remember that to achieve success with technical architecture, it is no less important to “engineer your culture” (yet another great Netflix mantra). You need to have a good understanding of the culture you need to have in place to support distributed, advanced microservices architecture and for everything to ultimately work as expected. Having the technology without the culture will just result in many unhappy and overworked engineers.

Great culture without modern engineering and architecture practices results in frustration in the inability to move at the desired velocity and deliver services as rapidly as engineers would like. Advanced architecture without the culture sets you up for failure, as you will not be able to capitalize on the benefits of microservices. So if you put in the effort to craft a truly great culture from the ground up, this will be foundational in enabling your people to thrive.

So eventually when we build great culture alongside an excellent product, this will drive those who are together with us in our purpose to get creative, and will repay us generously in both innovation and business.

8 tips to an effective code review checklist

Eyal Katz — Sun, 16 May 2021 11:23:01 +0000

Uri Shamay is a close friend and wunderkind with a wealth of experience in the devops field both as a founder and lead developer for a number of organizations. He wrote an excellent post detailing some of the most important aspects to remember when creating a code review checklist, and why it is so important in the development cycle.

Code review is the best way to maintain a high level of code quality. The code review acts not only as a gatekeeper for bad code but also as an incentive for coders to improve their skills, learn, and scrutinize their own work. Code giants like Google use code review on everything added to the codebase.

Most coders don’t use a checklist. At least not a formalized one. And definitely not a checklist as a tool produced by a team or a company that is used uniformly by all coders and reviewers. They might be using a mental checklist they have created for themselves. But this checklist may have major flaws if it is not discussed, designed, and maintained by multiple people. Checklists you can find on the internet are too long and too abstract for anyone to actually use. That is why, if you’re going to use a checklist, you should build your own.

Why use a checklist?

Code reviews need to produce changes. If they do not, they might as well not happen. This can lead to one of the biggest pitfalls of code reviews. Falling into a nitpicking session in search of something to change.

By using a checklist, the reviewers can stick to what’s important, and find those changes in the big things. Checklists also help reviewers make sure they don’t forget anything important.

When should you use a code review checklist?

A checklist can be a tool you build to improve your code reviews, or it can be more of an established company-wide practice. Either way, a checklist can be used both by the coder submitting code for review and by the person doing the review.

Preparing a code for review is an important step and a checklist allows the coder to take a step back and look at their code more objectively before handing the code on to the reviewer. I’d go so far as to say that a checklist can benefit the coder more than the reviewer.

Top tips to building an effective code review checklist

1. Mind the length

When building a code review checklist it is important to consider the length. If a checklist is too short it is unlikely to be a true checklist and cover the important things. But if a checklist is too long? It is just going to be ignored, as it would be too tedious to use.

The right size for your team might not be the same as for another. However, as a rule of thumb, 3-5 major issues and another 7-10 more entries focused or smaller issues would be a size I’d recommend.

2. Start from the basics

There are many code review checklists out there. Some are a long list of questions and some are broad topics you should make sure to touch on. Most of those issues are simply good design principles any coder should be aware of and manage while writing code. It makes sense that those would be the focus of checklists.

However, I find it unhelpful to have a checklist item that asks if the code in question is good design. Instead, I recommend including items that encourage focusing on the reasons the code was written in the first place, or give a different lens through which to review the code.

3. Prepare your code

Before you even get any explanation from the coder, ask yourself “Is this code self-explanatory?” I’m a great proponent of self-documenting code. But even if your coding style incorporates comments the code should be human-readable.

Having to read the code without any explanation forces the reviewer to make sure they can understand the code without someone explaining it to them. If any part of the code requires explanation to be understood, it might be best to break it down into smaller pieces of code or add a comment. With this item on the checklist, the reviewer can simply tell the coder that they did not understand the code. The coder can clarify, but they will know that they need to make the code clearer.

4. Affective vs effective code

Once you understand what the code does, you must ask: does the code change achieve its goal simply and effectively? People are very protective of their code, and as such will be reluctant to throw out their work and find a different solution. But if you can suggest a more effective code change? You should bring that to the attention of the coder.

5. Communicate effectively

How you communicate your proposal in a checklist is very important to whether or not it would be accepted. I find it best to ask a guiding question and let developers come up with alternative solutions. That way it is still their idea, and it will be met with less resistance. Remember, code reviews are not about showing your colleague how smart you are, it is about making the code as good as possible.

6. Don’t neglect dependencies

An often overlooked part of code review are Dependencies. It is easy to overlook a new dependency or a code in a package it doesn’t belong in. Putting this item on a checklist makes sure that this issue is not missed. This can also make the code review faster as any issues of dependencies could be resolved by the coder before submitting the code for review.

7. Consider company-specific issues

I could go on and list an endless number of issues that may or may not apply to your specific codebase. At the end of the day, your checklist needs to be tailor-made for your needs.

Does your company hold valuable information for customers? Could secrets have made their way into your codebase? Make sure you check security. Is your code highly modular, components used by many different areas of the code? Make sure everything is properly decoupled. If your company uses unit testing? Then you must make sure that the scope of the automated testing is comprehensive. Is the code change in a performance-intensive area of the software? Profiling could be a key item on your checklist.

And so on and so forth.

8. Iterate and improve

Like any design in software or otherwise, a checklist should improve over time. Put the checklist itself on the agenda of a meeting. Get feedback from everyone using it to find ways to improve it. Don’t forget to also get input from anyone that is not using it. This can help you find out why and how it can be made more accessible to them.

Sometimes a one size fit all solution will not work. Maybe another department needs a different checklist, and maybe someone in your team needs a different approach to help them improve the quality of their code review. Either way, checklists are not usually written in stone and, as such, grow and evolve with your codebase.

Creating the right code review checklist you need to do some research about what checklists are available, and take them apart. Find the items on them that speak to your company’s design and integrate them.

But most importantly, keep improving your code review checklist. The chances of you getting a perfect checklist the first time you build one are slim. Find out which bugs have slipped through the review process and add the necessary checklist items to prevent them. And cut out items that don’t produce any changes. Think of it as freeing some unused objects in your reviewer’s memory.

How to Debug Remotely in IntelliJ

Eyal Katz — Mon, 10 May 2021 09:36:55 +0000

Milan Bhardwaj is a colleague and an AI/ML software engineering adept with a wealth of experience solving real-world challenges with scalable solutions. He has driven many development cycles from inception to completion and wrote this expert and well-thought-out guide to remote debugging in IntelliJ environments.

Debugging is one of the most critical aspects of software development cycles. Developers not only leverage it to find and fix errors, but also to uncover potential performance issues in the code. Being able to debug is a core skill every developer needs to have in order to provide valuable, scalable solutions.

For applications running on remote servers and computers, developers turn to remote debugging to identify issues that can’t be produced locally. For example, you might run a smaller instance of your database locally to test with, but when pushed to a production load, bugs might surface related to large data sets. Or you might not run the same permission and firewall restrictions when developing locally that you do in production. This can lead to bugs that are impossible to reproduce locally without a significant investment in tooling.

With remote debugging, you can remotely control a target utilizing a debugger instance running on a different machine. The debugger instance, also referred to as a debug engine, runs on the same machine as the routine you’re debugging. Your local machine runs the debugger’s UI while the remote system runs both the debug engine and the target routine.

Remote debugging is particularly useful for a few reasons:

Controlling the flow of execution: It’s vital that developers understand the context of a bug as it manifests, and remote debugging affords that with features like pausing, resuming, and executing the program step by step.
Debugging in production environments: Troubleshooting problems on remote testing and production servers is no easy task. Running a debugger directly on the server where the application is running comes with its own set of limitations. Production servers usually run in strict environments where convenient developer tools are absent. Remote debugging is the perfect solution.
Catering to multiple deployment scenarios: Developers can debug a variety of complex deployment scenarios, from a single, standalone application to distributed applications and clusters.

Debugging remotely in practice

Most programming languages support some kind of remote debugging with the right tooling in place. Java includes several options for remote debugging, including integrations with most IDEs.

In this tutorial, you’ll see how to set up remote debugging on IntelliJ so you can effectively debug applications deployed to staging and production machines when necessary.

Setting up remote debugging on IntelliJ

JetBrains’ IntelliJ is one of the most intelligent integrated development environments (IDEs), used by more than 62 percent of Java developers. It offers developers out-of-the-box Java debugging functionalities that facilitate efficient, sustainable solutions.

Creating a New Project and Class

The first step is to properly set up and configure your Java project in IntelliJ. If you have not set up a Java program before, read the IntelliJ documentation to understand how you can create and configure a new Java project.

In order to demonstrate remote debugging, create a new class. Click src on the sidebar, hover on New, then click Java class. For this example, we’ll create a class named My_Numbers.

Once you have created the Java class, it’s time to write some code. Populate the Main method with the below example:

public class My_Numbers { public static void main(String[] args) { System.out.println("Initiating"); for (int num = 0; num < 100; num++) { try { Thread.sleep(1000); } catch (InterruptedException e) { e.printStackTrace(); } System.out.println(num); } System.out.println("Done"); } }

This code is a basic example that prints numbers from 0 to 100. While it’s a simple example, it will be enough to showcase remote debugging without adding too much complexity. You’ll add a [(https://www.jetbrains.com/help/idea/using-breakpoints.html)debug breakpoint] in later steps.

Configuring Remote Debugging

Now that you have a Java project and test class ready, you need to set up a debugging configuration in IntelliJ. Navigate to Run, then Edit Configurations. Click Add New Configuration (+) and choose Remote JVM Debug.

Fill in the following information to describe your remote debugging environment:

Name: In this case, My_Numbers_remote_debug.
Host: This field is for the address of the machine where the host app will run. Since we’re running on the same server in this example, it should be localhost, but if you want to debug a remote server (eg, 192.168.18.46), you must enter the IP address here.
Port: 5005 in this case. This port should be available, so pick one that your server supports.
Command-line arguments: You can copy the generated command-line arguments and paste them to the command line where JVM is active. You can copy the string for now (eg, -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005) and store it for later use.

Now IntelliJ knows where to access your remote (or local) application for debugging, but you still need to package and run the Java app to enter a debugging session.

Running the application

Set up the host application by packaging your application as a JAR and use the previously generated argument string directly in the command line.

<code><span class="hljs-keyword">java </span>-<span class="hljs-keyword">jar </span>-agentlib:<span class="hljs-keyword">jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 </span>remote-<span class="hljs-built_in">debug</span>-application-sample.<span class="hljs-keyword">jar</span>
</code>

Next, click Run, then Edit Configuration, then select My_Numbers from under the Application.

In Modify Options, select Add VM options.

Under the VM options field, paste the argument string that you copied previously.

With the necessary setup in place, you are ready to run the application. Right-click in the text editor and click Run My_Numbers.main().

In the program output, you’ll see Listening for transport dt_socket at address: 5005, followed by your app’s output. The first line states that the debug agent is active and the program is ready to accept incoming debugger connections.

Adding a Breakpoint

Now that your Java program is running, you can add a breakpoint and attach it to the process. Breakpoints allow you to set a place in the code where execution will pause so you can inspect memory, variables, and the state of your program.

You can set a line breakpoint by clicking on the gutter at the line of your choice. For this example, click line 10 and set the breakpoint with the condition num = 50. The program is suspended as soon as the breakpoint condition meets the condition.

At this point, you can perform relevant debugging actions, including expression evaluation and stepping, among other tasks. These actions will work whether you’re running locally on your machine or connecting to a remote instance of your Java application.

There is no denying that with distributed applications, it has become easier to manage, scale, and divide the workload. However, the architecture makes it much difficult to detect/reproduce bugs since it’s challenging to trace anomalies back to the source or replicate complex production environments. Remote debugging with IntelliJ can help developers identify bugs and reproduce errors across every stage of the software deployment lifecycle.

Getting Started with Spring Boot Actuator

Eyal Katz — Thu, 22 Apr 2021 13:16:11 +0000

Vikram Arucharmy wrote an engaging and well-constructed post about implementing and using the spring boot actuator. For anyone creating a production application or otherwise, it's a must.

Any production application needs to be monitored for its uptime. Let’s say you’ve developed a stock market statistics application, for example, using Spring Boot for your client. This application has to be up all the time while the stock market is open. If it’s down at a crucial time, it could mean huge losses for relevant stakeholders. In addition to being monitored for uptime, an application like this would probably even need a hotline support team that can be notified immediately if the application goes down.

By using the Spring Boot Actuator as a dependency, you can enable health status indicators to monitor your application’s status without any explicit implementation. We’ll get into many of its other functionalities as well in this tutorial, along with tips on how to use it.

What is the Spring Boot Actuator?

In mechanical terms, an actuator is a mechanical device used to move or control something. Similarly, the Spring Boot Actuator can be used to monitor and manage your Spring Boot application. All the actuator features are exposed to end users using the HTTP endpoints, communication channels that allow two software programs to communicate with each other.

The Spring Boot Actuator has several production-ready features available:

Monitoring: Checking the health status of the application to ensure that it’s running as expected without any problems. Actuator lets you check the health of the application by using the /health/ endpoint.
Logging: Writing log messages about events that occur during the runtime of the application. Actuator helps you view log messages and configure log levels during runtime using the /loggers/ endpoint.
Tracing: Logging all the HTTP requests to your application. It can be useful to understand how users are interacting with your application. Actuator helps you trace the HTTP requests by using the /httptrace/ endpoint.
Metrics: Measuring the performance and managing the memory footprint of your application. Actuator provides the useful metrics of your application by using the /metrics/ endpoint.
Auditing: Logging the authentication and authorization events of the application when Spring security is in action. This can help ensure the security and compliance of the application as well as define the lockdown mechanisms after n number of invalid login attempts. You can read more about Audit events here.

Importing and Configuring the Spring Boot Actuator

Import the Spring Boot Actuator to your Spring application, and let’s see how to enable and expose the endpoint to end users.

The spring-boot-actuator module helps you enable all the production-ready features in your Spring Boot application. Hence, you need to add this module in your dependencies. You can add the following code to the <dependencies> section of the pom.xml file, if you’re using Maven-based projects:

<dependency>  
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

If you’re using Gradle, you can use the following declaration:

dependencies { 
implementation 'org.springframework.boot:spring-boot-starter-actuator' 
}

Next, you’ll see how to enable and expose the endpoints.

Enabling Endpoints

All the available Actuator endpoints are enabled by default except the shutdown endpoint.

You can enable the Actuator endpoints using the Spring Boot Application properties file. To enable the endpoint, set the property management.endpoint.<id>.enabled in the Spring Boot properties file. The <id> must be replaced with the actual endpoint name, which needs to be enabled or disabled.

Use the below property in the properties file to enable the shutdown endpoint:

management.endpoint.shutdown.enabled=true

<id> is replaced with the endpoint name shutdown to enable the shutdown endpoint.

If you want to disable an endpoint, you can use the same property and assign the value false in place of true. This will disable the endpoint and remove the service from the application context.

If you don’t want to enable all the endpoints by default, you can disable that by using the property enabled-by-default as shown below:

management.endpoints.enabled-by-default=false

Now all the endpoints will be disabled, and you can enable each desired endpoint by setting its enabled property to true.

Exposing and Hiding Endpoints

Exposing an endpoint means making it accessible to the end users using the different technologies, namely JMX and Web (HTTP endpoints). Actuator endpoints may consist of sensitive information about the application and data. Hence, utmost care needs to be taken as to which endpoints should be exposed using which technology to make the application secure.

Similar to enabling the endpoints, you need to use the Spring Boot application.properties file to expose the endpoints. The property to expose the endpoint is management.endpoints.<technology>.exposure.include. Here technology must be replaced with either jmx or web to expose the endpoints in that specific technology.

You can find the endpoints exposed by default here. Only the enabled endpoints are exposed by default.

The property to hide the endpoint is management.endpoints.<technology>.exposure.exclude. Once again, technology must be replaced with either jmx or web to hide the endpoints in that specific technology.

A few things to note:

All the enabled endpoints are exposed in JMX by default, whereas only
health and info endpoints are exposed in Web
The exclude property takes precedence over the include property. If you have by any chance configured both exclude and include for any of the endpoints, then the endpoint will be excluded.
Now when you build your Spring Boot application, you’ll have all the relevant monitoring features readily available. You can easily consume it by using the defined endpoints.

Calling an Endpoint

In this section, you’ll learn how to call an endpoint using your browser and what will be the output of each endpoint. The following example assumes that your Spring Boot application is running in your local machine and the default port 8080. Hence the application base URL would be http://localhost:8080/.

First, you must know what endpoints are available as part of the Spring Boot Actuator in order to call it. The Spring Boot Actuator has a hypermedia (a discovery page) endpoint available under the URL <application_base_user>/actuator/ where all the available endpoints will be displayed.

To call the hypermedia endpoint /actuator/, use http://localhost:8080/actuator. You’ll see the following output with all the available endpoints displayed as a JSON text:

{
   "_links":{
      "self":{
         "href":"http://localhost:8080/actuator",
         "templated":false
      },
      "beans":{
         "href":"http://localhost:8080/actuator/beans",
         "templated":false
      },
      "caches-cache":{
         "href":"http://localhost:8080/actuator/caches/{cache}",
         "templated":true
      },
      "caches":{
         "href":"http://localhost:8080/actuator/caches",
         "templated":false
      },
      "health":{
         "href":"http://localhost:8080/actuator/health",
         "templated":false
      },
      "health-path":{
         "href":"http://localhost:8080/actuator/health/{*path}",
         "templated":true
      },
      "info":{
         "href":"http://localhost:8080/actuator/info",
         "templated":false
      },
      "conditions":{
         "href":"http://localhost:8080/actuator/conditions",
         "templated":false
      },
      "configprops":{
         "href":"http://localhost:8080/actuator/configprops",
         "templated":false
      },
      "env":{
         "href":"http://localhost:8080/actuator/env",
         "templated":false
      },
      "env-toMatch":{
         "href":"http://localhost:8080/actuator/env/{toMatch}",
         "templated":true
      },
      "loggers":{
         "href":"http://localhost:8080/actuator/loggers",
         "templated":false
      },
      "loggers-name":{
         "href":"http://localhost:8080/actuator/loggers/{name}",
         "templated":true
      },
      "heapdump":{
         "href":"http://localhost:8080/actuator/heapdump",
         "templated":false
      },
      "threaddump":{
         "href":"http://localhost:8080/actuator/threaddump",
         "templated":false
      },
      "metrics-requiredMetricName":{
         "href":"http://localhost:8080/actuator/metrics/{requiredMetricName}",
         "templated":true
      },
      "metrics":{
         "href":"http://localhost:8080/actuator/metrics",
         "templated":false
      },
      "scheduledtasks":{
         "href":"http://localhost:8080/actuator/scheduledtasks",
         "templated":false
      },
      "mappings":{
         "href":"http://localhost:8080/actuator/mappings",
         "templated":false
      }
   }
}

Next, let’s walk through the different endpoints and their defined purposes.

Health

Use the health endpoint to monitor the health of your Spring Boot application. It’ll return the current application status. This endpoint can also be used by monitoring tools to keep track of application status and notify the team when the application goes down.

This health endpoint uses the HealthContributor defined in the application context to collect the application health. When more than one HealthContirbutors is available, the final system health is derived by the StatusAggregator, which aggregates all the available health statuses from an ordered list. The first available status will be considered the overall health status.

You can check all the auto-configured HealthContributors here.

Use http://localhost:8080/actuator/health to consume the health endpoint. You’ll see the following JSON output:

{"status":"UP","groups":["custom"]}

Let’s break that down:

status: Provides the status of application.
UP: The application is running fine.
groups: Provides the name of the group to which this status is assigned. Groups organize different health indicators.
custom: The current health status’s group.

Metrics

Metrics can be used to measure the performance of your production application. Spring Boot Actuator supports dependency management and auto configuration for the Micrometer, a metrics façade that supports most popular monitoring systems.

To know all the available metrics in your Spring boot application, use http://localhost:8080/actuator/metrics. You’ll see all the available metrics names as given:

{
   "names":[
      "http.server.requests",
      "jvm.buffer.count",
      "jvm.buffer.memory.used",
      "jvm.buffer.total.capacity",
      "jvm.classes.loaded",
      "jvm.classes.unloaded",
      "jvm.gc.live.data.size",
      "jvm.gc.max.data.size",
      "jvm.gc.memory.allocated",
      "jvm.gc.memory.promoted",
      "jvm.gc.pause",
      "jvm.memory.committed",
      "jvm.memory.max",
      "jvm.memory.used",
      "jvm.threads.daemon",
      "jvm.threads.live",
      "jvm.threads.peak",
      "jvm.threads.states",
      "logback.events",
      "process.cpu.usage",
      "process.start.time",
      "process.uptime",
      "system.cpu.count",
      "system.cpu.usage",
      "tomcat.sessions.active.current",
      "tomcat.sessions.active.max",
      "tomcat.sessions.alive.max",
      "tomcat.sessions.created",
      "tomcat.sessions.expired",
      "tomcat.sessions.rejected"
   ]
}

Now, you’ll learn how to consume a specific metric from the available metrics. Let’s take http.server.requests and jvm.memory.used metrics as an example.

http.server.requests returns the number of http server requests processed by the application. Use http://localhost:8080/actuator/metrics/http.server.requests to find out the number of http requests. You’ll see the following output:

{
   "name":"http.server.requests",
   "description":null,
   "baseUnit":"seconds",
   "measurements":[
      {
         "statistic":"COUNT",
         "value":19.0
      },
      {
         "statistic":"TOTAL_TIME",
         "value":1.923151205
      },
      {
         "statistic":"MAX",
         "value":0.0
      }
   ],
   "availableTags":[
      {
         "tag":"exception",
         "values":[
            "None",
            "ResponseStatusException"
         ]
      },
      {
         "tag":"method",
         "values":[
            "GET"
         ]
      },
      {
         "tag":"uri",
         "values":[
            "/actuator/metrics/{requiredMetricName}",
            "/employees/{id}",
            "/employees",
            "NOT_FOUND",
            "/actuator/health",
            "/actuator/health/{*path}",
            "/actuator/metrics"
         ]
      },
      {
         "tag":"outcome",
         "values":[
            "CLIENT_ERROR",
            "SUCCESS"
         ]
      },
      {
         "tag":"status",
         "values":[
            "404",
            "200"
         ]
      }
   ]
}

COUNT shows the number of HTTP requests handled by your application and TOTAL_TIME shows the total time taken to process them.

jvm.memory.used returns the used memory metric of your application. Use http://localhost:8080/actuator/metrics/jvm.memory.used to find the used memory. You’ll see the following output:

{
   "name":"jvm.memory.used",
   "description":"The amount of used memory",
   "baseUnit":"bytes",
   "measurements":[
      {
         "statistic":"VALUE",
         "value":5.6238416E7
      }
   ],
   "availableTags":[
      {
         "tag":"area",
         "values":[
            "heap",
            "nonheap"
         ]
      },
      {
         "tag":"id",
         "values":[
            "Survivor Space",
            "Eden Space",
            "Metaspace",
            "Code Cache",
            "Tenured Gen"
         ]
      }
   ]
}

VALUE shows the value of the JVM memory used by your Spring boot application in Bytes.

You can explore all the available metrics by appending the metric name in the URL http://localhost:8080/actuator/metrics/ to find the different useful metrics of your application.

Conclusion

For any business-critical application, monitoring, tracing, logging, and auditing are crucial to ensure uptime and that the application is consumed appropriately by users. With Spring Boot Actuator, you can enable these functionalities in your application in a matter of minutes, simply by adding it as a dependency in your Maven POM file.

8 Top Git Security Issues & What To Do About Them

Eyal Katz — Sun, 04 Apr 2021 19:05:04 +0000

Git is the most popular software version control (SVC) standard used by developers today. That doesn’t make it the most secure. Whether you’re using GitLab, GitHub, or a locally hosted Git server; there are many security issues that can sneak up on you and start a snowball effect of unpleasant repercussions.

In this post, we’ll review just how secure Git is (or rather isn’t). We will demonstrate why and how serious Git security issues can be. Then, we’ll list the eight most common Git security issues, and what you can do about them.

How secure is Git?

At its core, Git is not built for security but for collaboration. As such, it is not secure but can be made secure through the use of tools and best practices.

Self-hosting a Git server is a security nightmare. If you are not an experienced maven in Git server configuration? You are probably not qualified to maintain a self-hosted Git solution hosting sensitive data. There are too many opportunities to exploit a misconfigured or unpatched Git server. So you may very well end up leaving a lot of holes for hackers to exploit.

Even hosted Git services such as GitHub or GitLab offer limited security. Such services offer an easy-to-use interface with enhanced access controls. However, their convenience and ease-of-use can prove to be a hindrance as well, often leading to human error. This especially true when code-commits are not properly screened by secret detection tools.

Why Git security matters

With many companies relying on Git for code management, Git has become a popular attack vector for hackers. There are numerous cautionary tales depicting the outcome of badly configured or insecure Git management. These are just the tip of the iceberg:

Two databases and a Spreadsheet

An employee at the Albert Einstein Hospital in Sao Paulo accidentally committed a sensitive spreadsheet file to a public GitHub repository. The spreadsheet in question included login credentials to two governmental databases. The first database contained private information on patients suffering from mild COVID-19 conditions. The second database held full patient hospitalization data.

Overall, the leak exposed personally identifying medical records of over 16 million Brazilian patients. The list included high-profile patients such as the Brazilian President, his family, 7 Ministers, and 17 state Governors.

Nissan takes a wrong turn

Automotive giant Nissan’s North America division suffered a massive data breach because of bad password hygiene. The company’s self-hosted Git server was misconfigured to use the default “admin/admin” password. This left the door completely open for hackers to step right in.

(Source: ZDNet)
The leak was only discovered after the source code behind Nissan’s mobile apps, websites and internal tools surfaced on hacking forums and Telegram groups. Thus, potentially leading to future exploits based on vulnerabilities hackers may discover within the pilfered code.

Don’t leave the doors open, Mercedes

A Swiss software engineer discovered a GitLab instance hosting onboard logic unit source code used in Daimler’s Mercedes Benz vans. The badly configured GitLab instance allowed anyone to register a developer account.

With a developer account in hand, over 580 Git repositories became easily accessible. This potentially enabled new and dangerous remote-takeover attacks based on vulnerabilities in the logic unit’s leaked source code.

Pay, or your Git gets it

Nearly 400 private Git repositories were held up for ransom by a hacker looking to cash out on poor security practices. The hacker scanned the internet, searching for websites with exposed Git configuration files that included login credentials.

They could then correlate these credentials with accounts on multiple Git hosting services where users were insecurely using the same login credentials. The malefactor encrypted the repositories and demanded a ransom by Bitcoin, threatening to take the vulnerable repositories public if the victims made no payment.

Healthcare providers are full of leaks

A Netherland security researcher, wondering if private patient healthcare information was easy to access online, did not take long to discover that healthcare developers did not take care of their code’s health.

The search took only 10 minutes using fairly simple searches on public GitHub repositories. With it, the researcher discovered hardcoded passwords enabling access to over 150,000 patient records across 9 healthcare entities in the United States (AccQData, MaineCare, MedPro Billing, Physician House Calls, Shields Health Care Group, Texas VirMedica, Waystar, Xybion).

Top 8 Git security issues & what to do about them

1. Hardcoded sensitive data

It’s all too convenient for a developer to store passwords, tokens, and authentication keys right in the code where such credentials are used. It’s just so tempting to save them where they are most accessible in case an issue arises.

While convenient, hardcoded secrets are possibly the worst security practice currently plaguing software development. No one is perfect. People make mistakes and forget past actions (such as temporarily storing passwords in code). People also often share code. Sometimes, they do so accidentally and at other times, intentionally as part of a collaboration. In such cases, long-forgotten secrets, still embedded in code, can easily leak and even get indexed online search engines.

There is no technical reason to use hardcoded authentication credentials. To prevent leaks, be sure to train developers to use secure coding practices. At the same time, you should ensure that security tools are integrated into the development process. These can monitor the workflow to prevent secrets from accidentally being committed.

2. Insecure Directories (.git/config)

When self-hosting a Git server, it is vitally important to secure the “.git” directory. A publicly accessible Git directory can allow malicious actors to clone the repository. Then they can scan it for secrets in the code or within historical records.

Other dangers lie in possible exploits residing in the source code itself. For example, SQL injection attacks are a lot easier to develop when you can review the source code of the target application. Furthermore, it is not enough to just deny directory listing access. Hackers with knowledge of Git’s directory structure can potentially bypass directory listing limitations, easily accessing secure files directly.

The only way to validate if a self-hosted Git installation is secure? Is by trying to penetrate it using the same techniques a hacker would. You must verify the “.git” directory and its sub-directories (especially “.git/config”) are not publicly accessible in any way or form and that all communication with the repository is performed over secure-HTTP (HTTPS).

3. Ignored .gitignore

The “.gitignore” file is an important security feature utilized by Git. The .gitignore configuration file informs Git of files that should or should not be included when the developer commits code to a repository.

Unfortunately, some developers are not trained in using .gitignore correctly. For example, a developer may add “.gitignore” to a folder name, assuming Git would ignore the folder. Another example would be a developer dropping an empty “.gitignore” file inside a folder, thinking the entire folder would be ignored.

To properly secure code commits, you must first understand how “.gitignore” works. Then, the best “.gitignore” strategy to use is one of inclusion rather than exclusion. You do not want to keep a list of files that must be skipped during a code commit. This is because more files may be added to the project over time, enabling secrets to leak if the “.gitignore” file is not consistently updated.

By using a “.gitignore” inclusion commit strategy only the specified files are committed to the repository. This helps stops secrets from becoming accidentally exposed due to lacking “.gitignore” maintenance.

4. Unsigned commits

When committing code to a Git repository, you can easily see the author committing the code. However, unless the author used a GPG key to cryptographically sign the commit, you simply can’t trust what you’re seeing.

It is fairly trivial for a developer with access to a repository to assign a code-commit they themselves performed to another developer on the project. A disgruntled employee can do this to inject a backdoor into the code, covering their track by assigning ownership of the code to another developer.

Another exploit would be to assign a code commit to a more respected or higher-ranking member of the project’s development, hoping the new code would be integrated with less oversight.

When code commits are signed, a “verified” icon appears next to the commit log entry. This ensures every member of the project knows the code was committed by the original author and was not tampered with in any way.

5. Insecure pipeline configuration

An insecure CD/CI pipeline can lead to secrets leaking or placed at risk by various processes such as pull requests coming from forks of your repository or CD/CI VMs left operating unattended.

Securing the pipeline requires holding secrets with very limited exposure. Beyond training developers to use proper security practices when storing secrets; you should integrate tools or online services into the CD/CI pipeline to provide an additional layer of protection.

You should employ Tools and Online Services to secure the build process and help store secrets as encrypted data, utilizing “just-in-time” decryption to limit exposure during storage and transport. This additional security layer is even more important when protecting extremely sensitive materials such as code-signing certificates.

6. Git vulnerabilities

While known vulnerabilities in Git are usually resolved quickly by the Git development team? When self-hosting Git, it is up to the site administrator to patch Git with the latest security updates as soon as they are released.

It is quite easy to locate Git servers through a web search, and unpatched servers are an easy mark for any hacker. To emphasize the dangers of an unpatched server, you only need to look at CVE-2017-14867. The vulnerability allowed attackers with Git-shell access to execute OS-level commands on unpatched systems – a recipe for a complete system takeover.

7. Unpatched software

Git is often used in combination with other tools or services to automate, secure, and provide analytics throughout the CD/CI pipeline. These days, hackers are not limiting themselves to directly hacking a target. It’s often easier and even more lucrative to perform a supply-chain attack on tools or services to compromise multiple entities that employ them.

To limit exposure to supply chain attacks, it is vitally important to apply tool-chain security patches as soon as they are released. You should also limit online service access to the minimum required for reliable operations, and of course, perform regular backups.

8. Inaccurate access permissions

As shown by the Mercedes example, badly configured permissions can provide an access point to every Git repository on the server. In the Mercedes case, the server automatically granted full access to anyone who just signed up for a developer account. In other cases, more subtle access permissions configuration errors may result in persons accessing data they are not authorized to.

When setting up access permissions, you must define access roles on a per-repository basis to ensure only developers with valid access credentials are allowed to interact with the repository.

Do not take Git security lightly. You have a lot to lose when your source code or intellectual data are compromised. To use Git in a production environment where code must remain secure? Secrets should never leak and security practices and policies must be consistently enforced.

Microservices vs APIs: One Doesn’t Always Imply the Other

Eyal Katz — Tue, 16 Mar 2021 17:12:18 +0000

Allan McGregor, a 15-year veteran software engineer, entrepreneur, and good friend wrote an engaging and insightful article on the oft-misconstrued relationship between Microservices and APIs.

When it comes to conversations around application architecture or working with integrations between applications, you’ve likely heard a couple terms pop up a few times: microservice and APIs.

You might also have run across the common misconception that microservices are just a way to implement APIs so they can communicate with each other. As you’ll see in this article, there are alternative ways to architect our microservice applications.

Let’s explore the concepts of microservices and APIs individually—what they are and how they’re used—and then I’ll show you how to combine both microservices and APIs into a more robust architecture.

What Are APIs?

APIs, or application programming interfaces, are what allow an application to communicate with other applications. Strictly speaking, an API is a group of protocols and standards that define how two applications share and modify each other’s data.

The definition for API has evolved and expanded over time. The term first originated to describe an interface meant only for end-user-facing programs, known as application programs. But API has come to mean something much broader. The term includes hardware interfaces and, of course, web APIs. In fact, web API is the most common application of the term nowadays.

Web APIs

If you’re using any web application today, you use APIs. They’re what enable the software integrations, allowing for many different systems to talk to each other (B2B communication). From banking and online shopping to social media and online streaming, APIs are powering most of our digital lives.

Now, APIs usually send and receive data through HTTP requests. There are several different types of protocols and API styles, like:

REST
SOAP
GraphQL
gRPC

The main difference between each resides in the architecture and, in cases like gRPC, in the communication protocol. Let’s take a closer look at one of the oldest and most common types of API: RESTful APIs.

RESTful APIs

The term REST, or Representation State Transfer, was defined by Roy Fielding in his 2000 Ph.D. dissertation “Architectural Styles and the Design of Network-based Software Architectures.” Fielding outlined six architectural constraints required to define a RESTful system:

Client-Server Architecture
Stateless
Cacheable
Uniform Interface
Layered System
Code on Demand

For an API to be RESTful, it must include these aspects:

A base URI, such as https://api.example.com

Use of semantic HTTP methods (GET, PUT, POST, and DELETE)
Media type definition for state transition of data elements

For a real-life example, think of shopping online. When you’re ready to check out, you see that the store gives you a couple of options to estimate shipping, commonly USPS, FedEx, or UPS.

Since each one of those shipping carriers is a separate company with different rates, the online store will make an API call with the contents of your shopping cart (weight, volumetric size) and retrieve the available rates for you to select during checkout. Once your order is placed and ready to be fulfilled, the store might make another API call to create the final shipment with the shipping vendor.

With APIs, applications can offer robust functionality without the need to implement everything from scratch. Other examples of this include using your social media credentials for account creation or login, buying online tickets, or even ordering a ride-share.

External and Internal APIs

APIs can be classified into two different categories: external and internal. The examples I’ve covered up to this point have been external, meaning APIs for third-party software developers and external use.

That said, APIs don’t have to be publicly available. Internal APIs are used for communication between internal applications, for example, internal microservices. With that in mind, let’s take a look at microservices next.

What Are Microservices?

Microservices are:

Microservice architecture—a variant of the service-oriented architecture (SOA) structural style—arranges an application as a collection of loosely coupled services. In a microservices architecture, services are fine-grained, and the protocols are lightweight.

Microservices are a style of software architecture used to divide different application functions into smaller “services.” These services are:

Highly maintainable and testable
Loosely coupled
Independently deployable
Organized around business logic

Microservice architecture is used to enable rapid, frequent, and reliable delivery of large and complex applications. The granularity of these services might vary greatly from a couple of services with large boundaries to hundreds of smaller services with particular functions, in the case of some larger companies.

To illustrate how a microservice architecture behaves in action, let’s take a step back and compare two different architectures for a flight booking app: monolith vs microservice.

Monolith

According to Wikipedia, a monolith is:

A monolithic application is self-contained and independent from other computing applications. The design philosophy is that the application is responsible not just for a particular task but can perform every step needed to complete a particular function.

In essence, a monolith is a single application that contains all the elements and components of an application or service, from the presentation layer and business logic all the way down to the database communication.

As you can see, our monolith is responsible for everything, including exposing an external API for third parties.

Monoliths are not necessarily bad or obsolete, and in specific scenarios, like a proof of concept or a simple web service, it might make sense to use a monolith. But as soon as our application starts to grow in responsibilities and scope, it makes sense to move to a microservice architecture.

Let’s take a look at the same application but this time using a microservice-based architecture:

There are a few key differences from our monolith approach:

We have broken down the business logic into different and separate services.
The flight-booking service doesn’t have any dependencies with the database nor the frontend.
Each service has its own API, and the other services use that API to communicate with each other.

Microservices vs. APIs

Before moving forward, let’s do a quick recap of what we’ve learned so far:
An API is the part of an application that allows other applications to request and send data.
There are many different styles of APIs and protocols.
Microservices is a software architecture style that separates an application into smaller boundaries.
Microservices can communicate to each other through APIs.

That last point is often where confusion or misunderstanding can set in. Note that the APIs implemented by each service don’t have to be RESTful, or even use HTTP for that matter; there are more efficient and faster alternatives.

One of these alternatives is gRPC, a universal RPC framework developed by Google. gRPC is a high-performance, open-source, universal RPC framework that can run in any environment.

Microservices could also communicate using the Event/Message system, which is entirely asynchronous and doesn’t need to wait for a response, thus avoiding coupling between our services.

For each one of these communication methods, there are a few pros and cons to consider.

REST

Pros:

Easy to understand and implement without previous experience
Requests and responses are usually human-readable (JSON)
Widely adopted standard with native support from many frameworks and libraries
Cons:
No standardized way to do things like versioning, search, and logical breakdown of endpoints
Poor support and applicability for streaming
No standard schema for endpoints

gRPC

Pros:
Great speed due to the binary format
Great support for streaming data bidirectionally
Supported by many languages (Java, Python, Scala) and allows for cross-language communication
Cons:
Higher learning curve, and additional knowledge required (eg, Protobuf)
Payloads are not human-readable and require additional tools for debugging
Event/Message Driven
Pros:
Fully asynchronous communication between services
Supports many different message formats: JSON, protobuf, AVRO
Can facilitate service decoupling
Cons:
Application design using messages may be more complicated
Additional message queue like Kafka, RabbitMQ or Pulsar is needed
The asynchronous nature might make it harder to debug

Making It All Work

Those are a lot of options, but here’s the best part: they are not mutually exclusive. They can all work together in your application’s service architecture.

Let’s take a final pass on our application diagram and apply what we’ve learned so far:

Microservices application with more efficient communication

There are a few changes from our previous microservice diagram:

We moved our REST API implementation to a fourth service.
REST is now only used for communicating with our frontend app and any external developer or services.
All the services are internally communicating using gRPC for better performance and handling large volumes of traffic.
Additionally, the booking service and billing service are communicating using an Event/Message system for operations that can be asynchronous.

As you can see, it’s possible to combine each communication style to leverage their benefits and strengths.

Conclusions

So now that we’ve gone over the definitions of microservices and APIs, hopefully, you’re more familiar with the connection between them.

** APIs allow our applications to send and receive data with other applications or even external developers; there are many different communication protocols and styles of APIs. Microservices are an architectural pattern that allows us to organize our application into smaller services. By segmenting our application into smaller services each with a single responsibility, we can quickly scale, maintain, and develop complex systems. **

Typically, microservices and APIs are applied together. For example, each microservice could contain an API. Through the API, the microservices will communicate with each other. However, this isn’t strictly necessary—services could use different communication strategies, like a message queue. Similarly, APIs don’t need to be implemented through microservices; for example, they could be part of a monolith application.

Knowing the difference and understanding the synergy between both concepts should allow you to leverage both efficiently when architecting and building applications. Companies like Netflix, Spotify, and Shopify have adopted this approach already.

The Complete Guide to Java String Replace

Eyal Katz — Mon, 01 Mar 2021 09:48:23 +0000

A colleague of mine Vikram Aruchamy over at LightRun wrote an excellent, concise, and easy-to-use tutorial on using the String.Replace() functionality in Java. I asked his permission to post his article here, and he graciously agreed. His post is a great resource to make sure you understand the utility behind these Java functionalities including examples.

One of the most commonly used functionalities for String objects in Java is String replace. With replace(), you can replace an occurrence of a Character or String literal with another Character or String literal.

You might use the String.replace() method in situations like:

Replacing special characters in a String (eg, a String with the German character ö can be replaced with oe). This won’t change the meaning of the word, but it will help make the string more universally understood.
Reading data from a CSV file with comma-separated values. The delimiters can be replaced with a space using the replace method to split the words.
Handling URLs in Java. You may need to encode the string to replace special characters (eg, a space needs to be replaced with %20).

In Java, keep in mind that String objects are immutable, which means the object cannot be changed once it’s created. StringBuffer and StringBuilder objects, on the other hand, are mutable, meaning they can be changed after they’re created. So, in some situations, String.replace() might not be the best way to replace a character in a string.

In this tutorial, you’ll learn:

How String.replace(), StringBuilder.replace(), StringBuffer.replace() can be used to replace a specific character or a substring in a String and when to use each of them. What happens to the immutable String object when replace() or replaceAll() is used.
How String.replaceAll() can replace all occurrences of a specific character or a substring in a String.

Using String.replace()

String.replace() is used to replace all occurrences of a specific character or substring in a given String object without using regex. There are two overloaded methods available in Java for replace(): String.replace() with Character, and String.replace() with CharSequence.

String.replace() with Character

This method accepts the oldChar and newChar, then replaces all the oldChar in the given string literal with the newChar and returns a resultant String object.

public String replace(char oldChar, char newChar)

String.replace() with CharSequence

This method accepts the target CharSequence and the replacement CharSequence, then replaces all the target CharSequence with the replacement CharSequence and returns a resultant String object.

CharSequence is an interface in Java implemented by String, StringBuffer, and StringBuilder. You can use any of these implementations in place of CharSequence.

public String replace(CharSequence target, CharSequence replacement)
While the signatures are different, these two implementations of the replace() method work much the same way.

String.replace() Examples

Let’s use the replace() method with the Char types parameter to replace a single character. You’ll use only single quotes in the replace() method parameters to denote oldChar and newChar as Characters. Double quotes make the parameters a String.

/** * Method to demonstrate the Replace method using the Chars type parameter */
private static void useStringReplaceWithChar() {
  String sourceString = "The world is schön. This is a String with special Character ö";
  String replacedString = sourceString.replace('ö', 'o');
  System.out.println("Source String before replace : " + sourceString + "\n");
  System.out.println("Replaced String : " + replacedString + "\n");
  System.out.println("Source String after replace : " + sourceString);
}

When you execute the program above, you’ll see this output:

Source String before replace : The world is schön. This is a String with special Character ö 
Replaced String : The world is schon. This is a String with special Character o 
Source String after replace : The world is schön. This is a String with special Character ö

A few things to note:

The first line printed the source String with the special character ö
When we used the replace() method, all occurrences of the special character ö are replaced with o.
With the replace() method, the original String is not modified. It always returns a new String object.

If you use the replace() method with CharSequence type parameters (denoted by double quotes instead of single):

... 
String replacedString = sourceString.replace("ö", "oe"); 
...

You’ll see similar output and the unmodified original String:

Source String before replace : The world is schön. This is a String with special Character ö 
Replaced String : The world is schoen. This is a String with special Character oe 
Source String after replace : The world is schön. This is a String with special Character ö

You can also use the replace() method with the StringBuilder implementation in place of CharSequence types”:

...
StringBuilder sourceStrBuilder = new StringBuilder("ö");
StringBuilder targetStrBuilder = new StringBuilder("oe");
String replacedString = sourceString.replace(sourceStrBuilder, targetStrBuilder);
...

When you execute this variation, you’ll see the same output as the previous example.

While the replace() method might be the obvious choice for most string replacement operations, it’s not the only option. In the next section, I’ll introduce the replace() method in StringBuilder.

Using StringBuilder.replace()

StringBuilder is a mutable sequence of characters. It is not thread-safe; methods are not synchronized. It’ll replace the substring sequence in the range passed as index parameters with the replacement String.

This is the method to use if you want to perform the replace operation in a single-threaded Java program. You can also use it in multiple thread programs as well, but that’s not recommended because the methods are not synchronized.

First, create a StringBuilder initialized with a String with the special character ö, using the following:

String sourceString = "The world is schön. This is a String with special Character ö";
StringBuilder builder = new StringBuilder(sourceString);

Next, use the replace() method with StringBuilder.

The replace() method accepts three parameters: startIndex, endIndex, and the replacementString.

startIndex is the beginning index of the replacement in a String literal. It’s inclusive, which means if you specify 1, the index 1 of the String will be included in the replacement.
endIndex is the ending index of the replacement in a String literal. It’s exclusive, which means if you specify 5, the index 5 of the String will not be included in the replacement. Replacements are made only through index 4.
replacementString is the actual replacement String. The specified index range in the String literal will be replaced with this String. The index of the character to be replaced is a required parameter, but you can use the StringBuilder.indexOf() method to identify the index of the special character as shown here:

int indexOfUmlauts = builder.indexOf("ö");

Now you have the index of the special characters in the int, which can be passed to the replace method. The following example shows the full Java code for using the StringBuilder.replace():

/** * Demonstrating the Replace method available in StringBuilder */
private static void useStringBuilderReplace() {
  String sourceString = "The world is schön. This is a String with special Character ö";
  StringBuilder builder = new StringBuilder(sourceString);
  int indexOfUmlauts = builder.indexOf("ö");
  System.out.println("Source String before replace : " + sourceString + "\n");
  System.out.println("Using StringBuilder.replace() " + builder.replace(indexOfUmlauts, indexOfUmlauts + 1, "oe") + "\n");
  System.out.println("String builder after replace : " + builder);
}

When you execute the above code, you’ll see this output:

Source String before replace : The world is schön. This is a String with special Character ö 
Using StringBuilder.replace() : The world is schoen. This is a String with special Character ö 
String builder after replace : The world is schoen. This is a String with special Character ö

A few things to note:

The first line printed the source String with the special String literal ö.
When the StringBuilder.replace() method is used, only the characters specified in the startIndex and endIndex are replaced with o. It doesn’t replace all the occurrences. You can see the character ö at the end is not replaced.
When the StringBuilder.replace() method is used, the original StringBuilder is modified. This is because StringBuilder is mutable, and it’s changed directly.

Mutability might be desired in some cases, but be careful. It could be an unintended side effect.

Using StringBuffer.replace()

StringBuffer is a mutable sequence of characters. Similar to StringBuilder.replace(), It’ll replace the substring sequence in the range passed as index parameters with the replacement String, but it’s thread-safe. Use this method when you want to perform the replace operation in a multi-threaded Java program.

To begin, create a StringBuffer initialized with a String with the special character ö similar to how you did with StringBuilder above:

String sourceString = "The world is schön. This is a String with special Character ö";
StringBuffer buffer = new StringBuffer(sourceString);
The replace() method accepts three parameters:

startIndex is the beginning index of the replacement in a String literal. It’s inclusive, which means if you specify 1, the index 1 of the String will be included in the replacement.
endIndex is the ending index of the replacement in a String literal. It’s exclusive, which means if you specify 5, the index 5 of the String will not be included in the replacement. Replacements are made only through index 4.
replacementString – The actual replacement String. The specified index range in the String literal will be replaced with this String.

The index of the Character to be replaced is a required parameter, so you can use the indexOf() method to find it. The following example shows the full Java code to use the StringBuffer.replace():

/** * Demonstrating the Replace method available in StringBuffer */
private static void useStringBufferReplace() {
  String sourceString = "The world is schön. This is a String with special Character ö";
  StringBuffer buffer = new StringBuffer(sourceString);
  int indexOfUmlauts = buffer.indexOf("ö");
  System.out.println("Source String before replace : " + sourceString + "\n");
  System.out.println("Using StringBuffer.replace() " + buffer.replace(indexOfUmlauts, indexOfUmlauts + 1, "oe") + "\n");
  System.out.println("String buffer after replace : " + buffer);
}

Just like in the StringBuilder case above, StringBuffer.replace() does not replace all instances of the special character and it does mutate the original string buffer:

Source String before replace : The world is schön. This is a String with special Character ö 
Using StringBuffer.replace() The world is schoen. This is a String with special Character ö 
String buffer after replace : The world is schoen. This is a String with special Character ö

Using String.replaceAll()

The last method for replacing strings in Java that I’ll cover is the String.replaceAll() method. It uses a regular expression to check if a matching substring needs replaced. The signature of the String.replace() method is shown here:

public String replaceAll(String regex, String replacementString)

Where:

regex is the regular expression to be matched to find the substring.
replacementString is the String that will replace the matched substring. ## The Difference Between String.replaceAll() and String.replace() Based on the name alone, you might guess that replaceAll() will replace all occurrences of a character or a substring, and you’d be correct, but String.replace() also replaces all occurrences of a character or a substring.

The only difference between the two is that with replaceAll(), you can use a regular expression to match the substring that needs to be replaced, whereas with replace(), you cannot use a regex. It accepts only Character or CharSequence.

Therefore, use replaceAll() when you don’t know the exact substring to be replaced but you can find it using a regular expression.

For example, you may get an HTML String response when you consume a RESTful service from your Java program. Within the HTML, you may want to extract only the text and ignore all the HTML tags.

The following example demonstrates how to use replaceAll() with a regex to remove all the HTML tags from the response:

/** * Demonstrating the replaceAll method using a regular expression */
private static void userStringReplaceAll() {
  String sourceString = "<body><h1>this is heading1</h1></body>";
  String replacedString = sourceString.replaceAll("<(.|\n)+?>", "");
  System.out.println("Source String before replace : " + sourceString + "\n");
  System.out.println("Text without HTMl Tags : " + replacedString + "\n");
  System.out.println("Source String after replace : " + sourceString);
}

If you’re not familiar with regex syntax, learn how to generate it here.

When you execute the program above, you’ll see the following output:

Source String before replace : <body><h1>this is heading1</h1></body> 
Text without HTML tags : this is heading1 
Source String after replace : <body><h1>this is heading1</h1></body>

As you can see, the original string was not modified, but the replaced string has all the tags stripped away. Regex is very useful when you have complicated string replacement rules, so replaceAll() will be a good method to keep in your toolbox.

Conclusion

In this tutorial, you learned how to use the String.replace(), StringBuilder.replace(), and StringBuffer.replace() methods to replace all occurrences of a substring. You also learned which Char implementation should be used in case of a single-threaded or multi-threaded Java program, and how to use replaceAll() with a regular expression.

If you’re having trouble which string replacement method to use or you have your own suggestions, feel free to leave them in the comments. I look forward to hearing what you think.

Observability vs. Monitoring in DevOps

Eyal Katz — Wed, 10 Feb 2021 12:58:42 +0000

If you strip the buzzwords and TLAs from the definition of DevOps? You’ll find that the roles and tasks involved aim mostly for more uptime and less downtime in the SDLC (software development lifecycle). The first step to achieving that is becoming aware of downtime as it happens with the help of monitoring solutions. Only then can you respond and resolve the issue in a timely manner that minimizes the dreaded and expensive downtime of software development teams.

The high cost of downtime

The importance of ensuring uptime cannot be understated. This is because when DevOps teams are slow to resolve a critical incident? The costs are high – very high.

The average cost per ticket in North America is $15.56. This cost increases as the ticket gets escalated, with an L3 ticket coming at approximately $80-$100 and more. In a survey conducted by ITIC, 33% of respondents noted that the cost of one hour of downtime can reach $1 million, and sometimes even up to $5 million.

Data center outages are costly as well. Every minute in the data recovery journey can cost from $137 to $427. The average hourly cost of an infrastructure failure is estimated at $100K, and the average cost of a critical application failure is between $500K to $1M per hour.

It’s clear that when time costs that much money? You need to respond at lightning speed. Which means you need to be aware of any outage or downtime in real or near-real-time. This is where monitoring and observability come into play.

Let’s take a look at each, how they are different from each other, and why every DevOps team needs both in their incident management and resolution arsenal.

What is monitoring in DevOps?

Monitoring systems enable DevOps engineers to view and understand the state of a system using a predefined set of metrics and logs. By monitoring the behavior of various components in a system, DevOps engineers can be first on the scene to detect failures when they occur.

Moreover, monitoring plays a vital role in enabling the analysis of long-term trends, dashboard design, and alerting.

Monitoring in DevOps supports three main incident management goals:

Issue detection of incidents such as outages, service degradations, bugs, and unauthorized activity; alerting to such issues when they arise and presenting relevant data via dashboards.
Issue resolution by answering the question “what” and “where,” and delivering information that supports troubleshooting and root cause analysis.
Continuous improvement of the software delivery process by providing insights that support enhanced capacity and financial planning, trending, performance engineering, and reporting.

Ultimately, monitoring in the DevOps CI-CD pipeline drives the collection of the data required to support automated problem detection and alerting, manual debugging when necessary, and overall system health analysis – each of which is critical to accelerated incident resolution.

What is observability in DevOps?

If monitoring is a tree in DevOps incident resolution? Then observability is the forest. That is, if monitoring lets you know when and where something has gone wrong, observability helps you see the bigger picture (i.e. the forest for the trees), by answering ‘why did it go wrong?’.

Having observability means being able to extract actionable insights from the monitoring tool logs. With these insights, you can have a fuller picture of the health and performance of your systems, applications, and infrastructure.

The primary components of observability are:

Logging for keeping a record of incidents so that teams can learn from previous events to accelerate finding the source and reason for an error. This plays a critical role in debugging.
Tracing, which is regarded by some as the most important part of observability, as it enables the understanding of the relationship between the cause and effect of an issue. Traces are often visualized through a waterfall graph, enabling developers to understand the time taken in a system, across queues, network hops, and servers. Ultimately, it makes the observable system more effective and drives root cause identification.
Metrics are the quantitative data that is collected, and enables developers to spot trends that emerge over days, weeks, and months.

Among the key benefits of observability is in the process of transforming masses of data into insights that are actionable and accessible to anyone. Without the proper tools, you find yourself with more and more devices monitored, and a growing mass of logs that are created every second. In such a case data can be more of a bane than a benefit.

Employing observability provides access to knowledge on how to solve anomalies. The more data that is collected and analyzed, the higher the likelihood that this data can be leveraged to accelerate incident resolution. It may even help in pre-empting issues before they arise.

Observability vs monitoring: what is the difference?

Monitoring is a subset of observability. Only a system that is observable can be monitored. Monitoring tools track the overall health of an application, aggregate data on how it’s performing, and when something goes wrong, alert you to what is happening and where.

Observability, on the other hand, is not something that you do. Rather, observability is something that you have. As opposed to monitoring, observability is proactive, leveraging the logs of monitoring together with machine learning and causation to deliver visibility and understanding of not only what is happening and where, but why and how to fix it.

In addition, it’s important to note that ‘observability’ is not just a fancy word for monitoring. It is the whole that is greater than the sum of its monitoring (and other) parts.

To know that something has gone wrong is important – and that’s what we have monitoring for. But, when it comes down to it, knowing when something has gone wrong is simply not enough in today’s software development world.

What DevOps teams need are insights – better, wider, more accurate. The kind that are only possible with observability.

By knowing the difference between monitoring and observability and the goal of each, the symbiotic relationship between the two can be leveraged for powerful incident resolution acceleration.

Top 10 Most Common Java Vulnerabilities You Need to Prevent

Eyal Katz — Tue, 15 Dec 2020 12:56:48 +0000

It’s easy to think that our code is secure. Vulnerabilities or potential exploits are often the things we think about last. Most of the time, our thoughts are preoccupied with sprints, scrums, meeting notes, and whatever the latest pivots marketing got approved are.

In a world where development speed takes precedence over code security, this can be a genuine issue. A breach or a hack can cost a business big dollars, if not kill it altogether. According to IBM’s 2020 Cost of Data Breach Report, the average total cost of a breach sits at 3.86 million US dollars.

The worst part of this is that it takes an average of 280 days to identify and contain the breach. Data is digital gold, and your code is the vessel that holds it.

While Java is considered relatively safe because it is a server-side language, there are still multiple ways to attack and access things you want to remain private.

To help you get a head start on the exploits your code may develop, we’ve listed the top 10 most common Java vulnerabilities, and how you can (and should) prevent them.

1. Code Injections

Every application that accepts input is vulnerable to code injections. A code injection occurs when the data passed through the input causes unintended side-effects to how your program runs or returns data.

If you think about it, a form is a two-way process. Someone enters the data, the application consumes the data and returns a result. When that result is not what you expect it to be but something else, it can leave your application in a vulnerable state.

Code injections happen a lot, and are far easier than you think to execute. In 2010, a Japanese developer noticed that you could send HTML as tweets. With a sprinkle of JavaScript inside the HTML, he sent a little Twitter worm off around midnight when no one was really supposed to be awake.

What did this little piece of code do?

Every time a user would hover over it, they would instantly retweet it. So when one of their followers was scrolling through, they might just retweet that little worm. This resulted in a chain effect of over 3000 retweets within a few minutes.

Though Twitter doesn’t only use Java in its stack, the cautionary event can be applied to protecting your input forms.

The easiest method is to apply input validation with output sanitizing and escaping. This means that any attempts at sending HTML code will be parsed or rejected, depending on what your application is doing.

2. Command Injections

An OS command injection, or also commonly known as a shell injection, is a security vulnerability that allows attackers to execute shell commands on the server that’s running your application.

PHP is often the target for command injections because it calls a sh/bash/cmd by default. Java, however, performs a fork() of the given command to create child processes and pass it the given arguments.

However, this doesn’t guarantee that your code is safe.

Your application may be segmented into different levels of legacy code stitched together to form the final product. These legacy products can act as entry ways for shell command injections.

Sometimes, you might need to issue a command line to your server – like sending a confirmation email. Rather than using Runtime.exec() to tap into a server, it’s better to use the available Java API located at javax.mail.*

3. Connection String Injection

Connection strings are a set of definitions that are used to connect an application to a data source. It may connect to your relational databases, LDAP directories and files.

For a database connection string injection, there are four parameters that a malicious user would need: the data source, the initial catalog, the user id, and password.

Connection string attacks happen when a bad actor gains access by injecting parameters into the connect strings using semicolons as separators.

The issue here is that some database providers have no limit cap and run on a ‘last one wins’ algorithm. This means that an attacker can run multiple connection injection strings, pollute them with duplicate parameters and the database will accept the valid combination.

As a result, the attacker ends up bypassing the normal authentication process without getting locked out.

For example, an injected connection string can look something like this:


Data Source = myDataSource; Initial Catalog = db; Integrated Security = no; User ID = myUsername; Password = XXX; Intergrated Security = true; Data Source = myDataSource; Initial Catalog = db; Integrated Security = no; User ID = myUsername; Password = XXX; Intergrated Security = no;

Once in, the malicious user can hijack the credentials and modify them to whatever they want.

4. LDAP Injection

An LDAP injection exploits input validations and injects executable queries. LDAP is the Lightweight Directory Access Protocol, which is an open and cross-platform protocol used for directory service authentication.

LDAP is a communication language that applications can use to access directory servers. These directory servers often store usernames, passwords, account details and other information that can be shared with other entities on the network.

LDAP injections can happen when an application inserts unsanitized inputs directly into an LDAP statement. When this occurs, the attacker can use the LDAP filter syntax causing the server to execute other queries and LDAP statements.

The easiest way to prevent LDAP injection is to ensure that LDAP special characters – ( ) ! | & * are escaped or rejected at validation.

5. Reflected XSS

A reflected XSS, or reflected cross site scripting, is the process of adding malicious scripts that is activated through a link. The request then sends the user to somewhere else.

For example, a reflected XSS can be embedded to blend in with the rest of the site in a user comment section. The user may click on it and end up going to a 3rd party site and then redirected back to the original site.

Whilst at the 3rd party, malicious activities such as cookie or session stealing may occur. Although it is hard to monitor reflected XSS, spam filters on links submitted can help reduce the frequency.

6. Resource Injection

Resource injections occur when the attacker successfully changes the resource identifiers used by the application to perform malicious tasks. This might be changing the port number, modifying the file name, and gaining the ability to execute or access other resources.

How can this happen? Usually when the application defines the resource via user input.

For example, imagine that an attacker gains access to a shopping site via connection string injection, or has successfully stolen your user’s details via XSS. They can now modify or query details using resource injection. Your attacker can cause havoc by ordering things from the site, modifying or stealing more information about your customer without them knowing.

7. SQL Injection

SQL injection is the process of injecting SQL within data requests that results in the backend application giving back confidential data or executing malicious scripting content on the database.

This can result in a complete compromise on the host, access to data and breaches of privacy. Not only this, SQL injection can result in data loss or corruption, and potentially lock you out of your own database. When this happens, the injection has fully taken control.

The easiest fix to this is to ensure that you validate on the server-side. Frontend inputs can be easily bypassed and the backend is the backstop to prevent unwanted characters, such as spaces and ‘ ‘ quote marks, to trickle through.

[Source: https://xkcd.com/327/ ]

8. Second Order SQL Injection

Second order SQL injection is a two-step process. First, the attacker adds something to your application but doesn’t immediately execute it. They might be waiting for more data or waiting for a trigger activity.

This is where second-order SQL injection differs from normal SQL injections.

The attacker injects code into the table row, which is deemed as a trusted source. That table row is then called, causing the attack to move from its dormant state to active execution.

Testing for second-order SQL is harder because of its often covert nature.

For example, the malicious user signs up with the username ‘ or ‘hacker’=’hacker

This means that ‘ or ‘hacker’=’hacker gets stored in the database. The database may use the following query to verify a user’s identity:

SELECT * from creditcards WHERE username = '<usernamehere>';

So when the username is ‘ or ‘hacker’=’hacker, the final query looks something like this:

SELECT * from creditcards WHERE username = '' or 'hacker'='hacker';

Then when you go to login, the user name completes the validation query, giving access to other users and their account details.

9. Stored XSS

A stored XSS, or persistent XSS attack takes place when an attacker injects a script into the content of a website or app. Unlike reflected XSS where third party links are embedded, store XSS is more dangerous in that it doesn’t require the user to interact with it.

Social media sites are particularly vulnerable to stored XSS attacks because of the platform’s nature. Users are encouraged to post and interact.

XSS is also commonly known as web worms, where the user ends up with the offending element and it executes on the browser. This attack can lead to stolen cookies, account information or some additional function through account impersonation.

XSS can be triggered in four places and is often done through JavaScript – @post .title, post.url, @post .id and @post .footer_attr

To prevent this, rejecting or escaping special characters likes < > and @ before parsing it can be useful.

10. XPath Injection

While JSON is the rising star for data structuring, XML documents are still popular and widely used. XPath is the syntax used for defining parts of an XML document. The idea behind XPath injections is similar to SQL injections.

The only difference between SQL injection and XPath is that the latter is in an XML format. This process of sending malformed data can be easily achieved if the attacker figures out the XML structure.

This allows the attacker to navigate XML documents, gaining access to various information such as username and password details.

Often, an XPath injection occurs when the query is built on unvalidated inputs. The trick to prevent XPath injections is to use precompiled XPaths. Avoid taking in full expressions from an unsecured source. If you have to parametrize your XPath, isolate it to string only parameters to prevent your query from getting hijacked.

For most injections, validating user inputs before you consume them is the easiest way to prevent potential attacks. It’s easy to offload the task over to the frontend, but they are only the first line of defense that’s not always guaranteed to hold.

While Java can be frontend and backend simultaneously, it’s still good practice to check that whatever your user gives you is what you expect it to be. Setting up your validation parameters may be a matter of identifying and specifying what’s allowed rather than trying to figure out and eliminate everything else.

21 Best IntelliJ Plugins for 2019 (100% Free)

Eyal Katz — Thu, 21 Nov 2019 13:39:24 +0000

Like other IDEs or CMSs before it, IntelliJ has quickly become more than an IDE. With its vast plugin marketplace, IntelliJ has turned into a full-fledged platform for all Java related coding. Have no doubt, JetBrain’s decision to open source and allow external developers to develop solutions on top of it is a strategic one.

Keep that in mind because if you use IntelliJ this will affect you. In fact, at Codota we like to consider ourselves data based swamis and are therefore predicting that in about 2 years from now, you’ll be staring at your IDE wondering how did IntelliJ become all about the plugins so quickly?!

We also like to help our users stay ahead of the curve and that’s why we put together this comprehensive list of all the plugins that will lead this shift. We based our choices on a variety of different factors - be it the plugin’s popularity in the JetBrains directory, the search volume that the problem they look to solve gets, or their aggregated review score. All these factors, and others, were taken into consideration when putting together this list.

Top 21 JetBrains IntelliJ Plugins for 2019

1. Maven Helper

The Maven Helper plugin provides an easy way to find and exclude conflicting dependencies, actions to run/debug maven goals for a module that contains the current file or on the root module, and actions to run/debug the current test file. It's really a great time saver for solving dependency issues.

2. Codota

Shameless plug - got to do it but before you judge listen in… the Codota plugin helps you code faster and with fewer pesky errors. Codota completes lines of code based on millions of Java programs, along with your own unique context (open it from within the IDE using Ctrl + Shift + O). You can also find commonly used open-source Java code snippets for the class or method you're using in the code search.

Installation Instructions

3. Add to gitignore

With Add to gitignore you can create an automated protocol to ignore a specific path. You can track .gitignore files in your repository named .gitignore. Git uses it to determine which files and directories to ignore before you commit. This handy little plugin lets you do just that straight from your IDE.

4. String Manipulation

This plugin is pretty straightforward. Use String Manipulation to - well, manipulate strings. Perform a variety of different tasks on strings such as converting to camel case, capitalizing, escaping string in Java, and more.

5. Grep Console

Grep Console enables you to define a series of regular expressions which will be tested against the console output. Every expression that matches a line will change the style of the whole line/parts of it.

6. Nyan Progress Bar

Nothing wrong with a little bit of fun in your IDE and nothing is more fun than Nyan cat. So how about a Nyan progress bar for those endless waits for rebuilds.

7. CamelCase

Great plugin that switches easily between CamelCase, camelCase, snake_case, SNAKE_CASE, well...you get the point. Wish they had one for blog writing too.

8. Rainbow Brackets

I love Rainbow Brackets / Parentheses for IntelliJ. This useful tool saves you the confusion of selecting which bracket needs to be closed. Each pair of brackets/parentheses has a different color. Pretty simple, but cool. Plus, this plugin adds some color to your otherwise pale code.

9. CheckStyle-IDEA

This plugin provides both real-time and on-demand scanning of Java files with CheckStyle from within IDEA. Checkstyle is a static code analysis tool used for checking if Java source code is compiling correctly. This awesome plugin connects to CheckStyle you do it from your IDE.

10. AWS Toolkit

AWS Toolkit is an open-source plugin used to make AWS application development easier. Use the toolkit to create, test, and debug apps built on AWS.

11. EduTools

You don't have to be Joshua Bloch or Herbert Schildt to teach Java. EduTools helps you teach (and learn) IntelliJ based programming languages right in your IDE. This is done in the form of coding tasks with instant verification and feedback.

12. BashSupport

Bash language support for IntelliJ. The open source software supports run configurations, rename refactoring, quick fixes, documentation lookup, syntax highlighting, inspections, and plenty more.

13. Python

The Python plugin adds full-scale functionality for Python development. Integrate Python into your IntelliJ platform. What more could you ask for?!

14. JRebel for IntelliJ

JRebel enhances developer productivity - by reloading code changes instantly. It skips the rebuilding, restarting, and redeploying cycle that is common within Java development.

15. Docker Integration

This plugin allows you to download and build Docker images, create and start Docker containers, and do other related tasks. Docker lets developers deploy applications inside containers for testing code in an environment identical to production.

16. CSV Plugin

Lightweight CSV plugin that allows you to edit files in CSV/TSV format.

17. Eclipse code formatter

Basically exactly what it sounds like. Use Eclipse Code Formatter to fix the problem of maintaining a common style for environments including Eclipse & IntelliJ developers.

18. FindBugs-IDEA

Provides static byte code analysis to look for bugs in Java code from within IntelliJ IDEA. FindBugs is a defect detection tool for Java that uses static analysis to look for more than 200 bug patterns, such as null pointer dereferences, infinite recursive loops, bad uses of the Java libraries and deadlocks.

19. AceJump

AceJump allows you to quickly navigate the caret to any position visible in the editor. If no matches can be found on-screen, AceJump will scroll to the next match it can find.

20. Zero Width Character Locator

We're getting close to the end of our very comprehensive list of IntelliJ plugins. I really like plugins like this one that offer a simple solution to a very common problem. The Zero Width Character Locator plugin adds an inspection that prevents some hard to find bugs related to invisible zero width characters in source code and resources.

21. IdeaVim

Finally, the IdeaVim plugin also made our list. IdeaVim provides Vim emulation in IntelliJ based IDEs. It includes many of the popular features Vim has to offer: normal/insert/visual modes, motion keys, deletion/changing, marks, registers, some Ex commands, and much more.

Thanks for making it all the way down! It may seem like a long list but with the thousands of plugins available for IntelliJ it was not an easy list to compile. So, if we left out your favorite plugin please let us know in the comments below. Happy coding!