Reversee, yet another web debugger?

Gal Ben Ami — Mon, 13 Jan 2020 17:10:57 +0000

A while ago I have started a project named Reversee. On first sight, it looks just like another web debugger. So, why bother developing yet another web debugger?

At the time, I used Charles Proxy and WireShark on a mac and Fidler on windows. Those tools are great!

But, Fiddler was never great on platforms other than Windows. WireShark felt like using a surgical knife to slice bread.

All of the above tools sometimes just give you too much information, for some, it takes too long to get them running. If you are dealing with non standard clients, like CLIs, microservices, web applications etc, you must know how to configure a proxy for them to work.

All I needed is a tool to debug apis between client and server, I wanted it to take 2 seconds to start and get some data, and as a side bonus, I wanted it to work on Mac Windows and Linux. That is why I decided to write my own as a side project.

So, I fired up my computer and started writing. Luckily, Electron was there when I started. So, after the first night (I work during the days), I had the basic functionality running!

What can you do with Reversee?
The core functionality is something like the network tab on chrome. You have a client, a server, and Reversee in the middle. You can see http requests, body, headers, responses. Basically, what you would expect from a web debugger.

So, what is the difference?

In Reversee, reverse proxy is a first class citizen. Internally Reversee uses reverse proxy to get the requests from the client and fire them to the server, just like NGINX does.

Charles and Fidler mostly uses forward proxy, which has its benefits and drawbacks.

This gives you the ability to simplify many things, consider the following use case.

You are developing a web service that consumes info from some other web service (can be a public service, your own lambda function or whatever).

We will call the first web service the client, and the other one the server. Assume the server can only accept https. Now, you want to see the traffic between those services.

Using Charles, Wireshark or Fiddler, you will have to add some certificates for SSL/TLS proxy.

Using Reversee you only need to set Reversee up to listen on http traffic and send https traffic, and to configure the client to use http://localhost:. No certificates headache is involved.

Another cool feature of Reversee is request and response interceptors, you can run Javascript code on requests and responses. For example, assume you need to rename a header, change the response body, go to a different URL, etc. All can be easily achieved with a couple lines of code.

Let's see an example how to use Reversee.
We will work with https://jsonplaceholder.typicode.com/ as an example API.

Start Reversee.
Select http for the client protocol.
Select a random high port that is not in use on your computer. We will use 9000 in our example.
Select https for the server protocol.
Switch it to on.

See below:

We will use cURL for our example.

Run the following command from commandline:

curl http://localhost:9000/todos/1

Look in Reversee and see your request:

This shows you how fast it is to start using Reversee. Notice that we are debugging an https endpoint. In most tools, this is not trivial! In other tools, you will have to install certificate in your machine and trust it. Here, we are using a trick. the client is using http and Reversee is using https to talk to the server, that is how we can see traffic going out from the client and coming back from the server.

For clients that will not support http, but only https, you can also install Reversee's root certificate, but this is for another time.

Request/Response modifications - With Reversee you can change the Request or the Response using a few lines of code. This is useful for example if you want to add/modify header, change the request or response body, add a request query parameter, etc.

For example, assume you are developing the client in this client-server and you need to develop a new feature that requires a new field in the response named rating. We can add it easily to the response.

If you haven't done so already, download Reversee and start (Web) debugging: https://reversee.ninja

Auto-MAT - Analyse Java Heap Dumps From Commandline

Gal Ben Ami — Sat, 28 Sep 2019 19:03:34 +0000

A while ago, I had to analyse a large java heap dump, generated by one of our servers after a memory leak. This was hard.
I usually use Eclipse MAT for the job, but this time, it was so large the tool could not parse the heap dump in an acceptable time, and required more and more memory.

I eventually took a strong machine with GUI in the cloud and got it to work, but the experience was bad. I had to use GUI for that, because all the tools I knew back then required GUI to operate.

Running a server with GUI in the cloud is not something I do every day, and it took some time to configure the server, start it, connect with remote desktop etc'.

I figured there must be a better option. So I looked around, and discovered that you can let Eclipse MAT parse the heap dump from commandline!

You can use a script that comes with the Linux installation of Eclipse MAT. The script will calculate indexes for later use in eclipse MAT and can also generate some nice HTML reports that are usually good enough to understand the problem.
The index files will help Eclipse MAT open in seconds, instead of long hours.

This is much better than the experience I had, I could run the script on a remote server from terminal, which is easy to set up, and I usually have a couple of servers ready to do some work for me.

I've tried the script and it worked great. But, it comes only with the linux installation. What if I wanted to run it on Mac or Windows? Also, Eclipse MAT requires Java to be installed.

So, I decided to wrap it in a docker container, and that is how Auto-MAT got created.
Now every time I need to analyse a heap dump I could just run something like:


 run -it --mount src=$(pwd),target=/data,type=bind docker.bintray.io/jfrog/auto-mat heap1.hprof 11g suspects,overview

This will parse the heap dump, create indexes and will generate nice 2 HTML reports. Suspects and Overview.

After running this, it will usually be enough to open the HTML reports to understand the problem. But if you need a deeper drill down into the memory, you could open the heap dump with MAT and, since the indexes are already there, Eclipse MAT will open in a snap.

Well, that was great, but if I needed a strong machine to run Auto-MAT, I would still have to find a machine, upload the heap dump, install docker if it is not installed, and then download everything back or store it somewhere.

So, I decided to build an automation around it. First, I've asked my colleague to build a Jenkins job that will take a heap dump from a production server, and will upload it to Artifactory.

Later I built a second Jenkins job that will download the heap dump from Artifactory and run Auto-MAT on it. The artifacts of Auto-MAT run will be uploaded back to artifactory.

The second Jenkins job can be triggered by the first, or manually after someone has uploaded the heap dump to Artifactory.

In this way, anyone in the organisation can analyse heap dumps with a click of a button.

If you are planning to work with heap dumps in automation, I recommend to compress them, I use pigz which is a faster version of gzip. I used Artifactory to store the files for my automation, you can use anything else, but if you are using Artifactory check out jfrog-cli, and especially the parallel download and upload features.

Auto-MAT is open source and you can visit the github page for more: https://github.com/jfrog/auto-mat

Forem: Gal Ben Ami

Reversee, yet another web debugger?

Auto-MAT - Analyse Java Heap Dumps From Commandline