Connor Shade

Writing a Custom Nmap Scan

Published 28.07.2023 , Last Edited 07.08.2023

Welcome back to the second part of “service-to-shell”. In this post I’ll outline the process of making a custom nmap match directive that can be used to detect services that may not have an existing nmap directive. This can be especially useful when dealing with custom applications. Here’s an overview of what I’ll cover:

Reporting a banner

By default, uvicorn will use the Server header of “uvicorn” in the response. This says nothing about the application running behind uvicorn, and in some cases that’s a good thing. Security through obscurity isn’t, but such a bland banner does mean nobody can query Shodan and see the specific exploitable device you’re running (at least, not from the Server header) - curl -v http://127.0.0.1:8000:

 1*   Trying 127.0.0.1:8000...
 2* Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0)
 3> GET / HTTP/1.1
 4> Host: 127.0.0.1:8000
 5> User-Agent: curl/7.81.0
 6> Accept: */*
 7> 
 8* Mark bundle as not supporting multiuse
 9< HTTP/1.1 200 OK
10< date: Sun, 30 Jul 2023 12:26:47 GMT
11< server: uvicorn
12< content-length: 33
13< content-type: application/json
14< 
15* Connection #0 to host 127.0.0.1 left intact
16{"message":"Working DNSSearcher"}

My goal is to get the server: uvicorn to say something like server: dnssearcher v0.1.0. My first though was to modify the FastAPI app, but I wasn’t able to get that working - I don’t think FastAPI is aware of the server header being returned.

Reverse Proxy

My next attempt was to put the whole application behind a proxy and rewrite the server banner using the proxy. The idea works, but the implementation required using a massive Dockerfile that compiled nginx with the ngx_http_headers_more_filter_module module. This allowed me to set the header I wanted inside of the nginx config file:

 1# nginx.config
 2load_module modules/ngx_http_headers_more_filter_module.so;
 3events {}
 4
 5http {
 6    server_tokens off;
 7    server {
 8
 9        more_set_headers "Server: DNS Searcher v0.1.0";
10        listen 80;
11        server_name dnssearcher;
12
13        location / {
14            proxy_pass http://dnssearcher:8000;
15        }
16    }
17}

This works, but then I realized something: uvicorn is between FastAPI and the proxy, uvicorn is the application. Is there a way to modify uvicorn so that it returns a server header?

Uvicorn

After going through all that trouble I thought to do a Google search for modifying the server banner of a uvicorn service. It turns out, the solution was just adding --header "Server: dnssearcher v0.1.0" to the uvicorn start arguments:

1uvicorn main:app --reload --header "Server: dnssearcher v0.1.0"

Now, the headers are correct when I curl -v localhost:8000 (I don’t even need the proxy):

 1*   Trying 127.0.0.1:8000...
 2* Connected to localhost (127.0.0.1) port 8000 (#0)
 3> GET / HTTP/1.1
 4> Host: localhost:8000
 5> User-Agent: curl/7.81.0
 6> Accept: */*
 7> 
 8* Mark bundle as not supporting multiuse
 9< HTTP/1.1 200 OK
10< date: Sun, 30 Jul 2023 12:45:47 GMT
11< server:  DNS Searcher v0.1.0 uvicorn
12< content-length: 33
13< content-type: application/json
14< 
15* Connection #0 to host localhost left intact
16{"message":"Working DNSSearcher"}

Now that the banner actually reports useful information, we can get started on parsing the banner. First I should go over the nmap probe file structure.

nmap-services-probe file

The nmap-services-probe is a flat file with simple grammar. I highly recommend reading through the full page of the nmap manual on the nmap-services-probe file before making your own probes, but I’ll do my best to distill the info here.

Layout

Each file has:

Here is a sample, taken directly from the nmap documentation and slightly modified:

 1Exclude T:9100-9107
 2
 3# This is the NULL probe that just compares any banners given to us
 4##############################NEXT PROBE##############################
 5Probe TCP NULL q||
 6# Wait for at least 5 seconds for data.  Otherwise an Nmap default is used.
 7totalwaitms 5000
 8match ftp m/^220[ -]Microsoft FTP Service\r\n/ p/Microsoft ftpd/
 9match ftp m/^220 ProFTPD (\d\S+) Server/ p/ProFTPD/ v/$1/
10softmatch ftp m/^220 [-.\w ]+ftp.*\r\n$/i
11
12Probe TCP GenericLines q|\r\n\r\n|
13ports 80
14sslports 443
15softmatch http m|^HTTP/\d\.\d|
16match http m|^HTTP/1\.[01] \d\d\d.*?\r\nServer: nginx\r\n|s p/nginx/ cpe:/a:igor_sysoev:nginx/

The above example has every section you should need to get started. On line 1 you can see the Exclude directive - TCP ports 9100-9107 are excluded, since those are common printer ports and may print anything sent to them. Line 5 holds the first Probe directive for the “NULL” probe. On lines 8 and 9, there are match directives for two separate FTP services, while line 10 has a generic softmatch if the response code is 220 and the banner has “ftp” in it.

The next Probe on line 12 has port and sslports specified. There is also as softmatch on line 15 to match any generic HTTP service - the scan will return a service of “http” with no additional information if the softmatch hits but nothing else does. Hitting this softmatch will also limit future match objects to the same service, which in this case is “http”.

How does it work

What is the link between a Probe, a match, and a softmatch when writing a services-probe file? As an example, we’ll take a look at the flow of Calibre which has a softmatch and match across different probes:

1# version available with GetRequest
2softmatch http m|^HTTP/1\.0 400 Bad Request\r\nContent-Length: 40\r\nContent-Type: text/plain; charset=UTF-8\r\nDate: .*\r\n\r\nMultiple leading empty lines not allow
3ed| p/Calibre Content Server httpd/ cpe:/a:kovid_goyal:calibre/

This softmatch is after the Probe “GenericLines”. softmatch on the service means that only match directives which have a service of http will be tried. It also identifies that, even if no other match directives hit, the CPE data for Application is “kovid_goyal:calibre”. The actual definition for GenericLines is below - notice the q|\r\n\r\n| - this string of characters is what was sent to Calibre, prompting the “Multiple leading empty lines not allow” error message:

1Probe TCP GenericLines q|\r\n\r\n|
2rarity 1
3ports 21,23,35,43,79,98,110,113,119,199,214,264,449,505,510,540,587,616,628,666,731,771,782,1000,1010,1040-1043,1080,1212,1220,1248,1302,1400,1432,1467,1501,1505,1666
4,1687-1688,2010,2024,2600,3000,3005,3128,3310,3333,3940,4155,5000,5400,5432,5555,5570,6112,6432,6667-6670,7144,7145,7200,7780,8000,8138,9000-9003,9801,11371,11965,137
520,15000-15002,18086,19150,26214,26470,31416,30444,34012,56667
6sslports 989,990,992,995

Much further down the services-probe file is another calibre match definition,which expects a well-formatted HTTP response with a server header. The regex extracts the Calibre version from the header, putting it in the CPE:

1match http m|^HTTP/1\.1 \d\d\d (?:[^\r\n]*\r\n(?!\r\n))*?Server: calibre ([\d.]+)\r\n|s p/Calibre Content Server httpd/ v/$1/ cpe:/a:kovid_goyal:calibre:$1/

This is underneath the GetRequest probe, which sends a proper HTTP GET request:

1Probe TCP GetRequest q|GET / HTTP/1.0\r\n\r\n|
2rarity 1
3ports 1,70,79,80-85,88,113,139,143,280,497,505,514,515,540,554,591,620,631,783,888,898,900,901,1026,1080,1042,1214,1220,1234,1314,1344,1503,1610,1611,1830,1900,2001,2002,2030,2064,2160,2306,2396,2525,2715,2869,3000,3002,3052,3128,3280,3372,3531,3689,3872,4000,4444,4567,4660,4711,5000,5427,5060,5222,5269,5280,5432,5800-5803,5900,5985,6103,6346,6544,6600,6699,6969,7002,7007,7070,7100,7402,7776,8000-8010,8080-8085,8088,8118,8181,8530,8880-8888,9000,9001,9030,9050,9080,9090,9999,10000,10001,10005,11371,13013,13666,13722,14534,15000,17988,18264,31337,40193,50000,55555
4sslports 443,993,995,1311,1443,3443,4443,5061,5986,7443,8443,8531,9443,10443,14443,44443,60443

And above everything is the NULL probe, which initiates the TCP connection.

So from top to bottom, the process is:

  1. A TCP connection from the NULL probe, which waits for 6000ms, in case the server sends banners to a new connection.

    • Some services will send banners once a TCP connection is established, without waiting for a request. Calibre is not one of them, so the NULL probe returns nothing

  2. The banner is run against all softmatch and match lines within the NULL probe.

    • In this case, there is no banner and therefore no matches.

  3. A TCP Probe for GenericLines that just sends \r\n\r\n is run

    • This returns an error message from Calibre. This is implementation-specific. Using nc to connect to dnssearcher and send \r\n\r\n doesn’t get a response of any kind, but sending a basic GET / HTTP/1.1\r\n gets this response:

      1GET / HTTP/1.1
2
3HTTP/1.1 200 OK
4date: Tue, 01 Aug 2023 22:53:18 GMT
5server: uvicorn
6content-length: 33
7content-type: application/json
8
9{"message":"Working DNSSearcher"}

  4. Nmap searches top-down through all matches under this Probe for either a softmatch or a match

    • A softmatch on the error banner that identifies that the product is “Calibre Content Server httpd” and the cpe string is “cpe:/a:kovid_goyal:calibre/” - this also limits further match attempts to http matches.

  5. A TCP Probe for GetRequest is sent, the format is a standard get request for /

    • I believe that the GetRequest probe is sent because it: A) Is next in the file ; B) Has at least one http match directive

  6. Nmap searches for matches under this probe that have a service of http and finds a matching one that extracts Calibre-specific data - this gets the version information.

Adding a detection to nmap-service-probes

Now that we’ve covered a little bit about how nmap performs its lookups, we can answer the question: What is the best way to extract dnssearcher version data using nmap?

The easiest method would be to add a match to an existing Probe directive. Based on the tests with nc and dnssearcher, we know that it won’t respond to the GenericLines probe but it will respond to the GetRequest probe - we could simply add a match somewhere within the GetRequest probe.

The idea of adding a custom version detection match is covered in the nmap manual. To start with, we can make a temp directory to hold a temp copy of the nmap-service-probes file. The default location for the nmap services probe file for me is /usr/share/nmap/nmap-service-probes. If you’re unsure of the location, you can run the following command:

1sudo find / -type f -name "*nmap*probe*" 2>/dev/null

From within my service-to-shell project directory I’ll make a new folder for nmap stuff, copy the nmap-service-probes file there, and set an environment variable for nmap to use:

1mkdir ./custom-nmap
2cp /usr/share/nmap/nmap-service-probes ./custom-nmap
3# keep in mind, the env variable is only good for the lifetime of this terminal session
4export "NMAPDIR=$PWD/custom-nmap"

Now we’re good to modify the temporary service probe file at ~/custom-nmap/nmap-service-probes. We need to know what the full banner is in order to write a match, and we can actually use nmap for that. If we run an nmap version scan nmap -sV -p 8000 127.0.0.1 against this service, the service detection will fail and print out the data it got for the GetRequest probe (Actually it prints out the data for every probe it tries, but we only need the GetRequest results):

 1# Truncated response
 21 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service :
 3SF-Port8000-TCP:V=7.92%I=7%D=7/25%Time=64BFC608%P=x86_64-pc-linux-gnu%r(Ge
 4SF:tRequest,AA,"HTTP/1\.1\x20200\x20OK\r\ndate:\x20Tue,\x2025\x20Jul\x2020
 5SF:23\x2012:54:32\x20GMT\r\nserver:\x20\x20dnssearcher\x20v0\.1\.0\r\ncont
 6SF:ent-length:\x2033\r\ncontent-type:\x20application/json\r\n\r\n{\"messag
 7SF:e\":\"Working\x20DNSSearcher\"}")
 8# Cleaned up banner
 9"HTTP/1\.1\x20200\x20OK\r\ndate:\x20Tue,\x2025\x20Jul\x2020
10SF:23\x2012:54:32\x20GMT\r\nserver:\x20\x20dnssearcher\x20v0\.1\.0\r\ncont
11SF:ent-length:\x2033\r\ncontent-type:\x20application/json\r\n\r\n{\"messag
12SF:e\":\"Working\x20DNSSearcher\"}"

This output is more useful than the netcat output because it includes the escaped whitespace characters and hex representation of whitespace. The regex for this is pretty simple - it starts with an HTTP status code,has the date, then the server. After the server header we can stop processing:

1# HTTP status section
2^HTTP/\d\.\d \d{3} \w+\r\n
3# date section
4date: [ \w:,]+\r\n
5# server identifier section
6[sS]erver:\s+dnssearcher v([\d\.]+)
7# all together in a match:
8match http m|^HTTP/\d\.\d \d{3} \w+\r\n[dD]ate: [ \w:,]+\r\n[sS]erver:\s+dnssearcher v([\d\.]+)| p/DNSSearcher/ v/$1/ cpe:/a:connorshade:dnssearcher:$1/a

By dropping the match line just underneath the sslports section of the TCP GetRequest Probe, we can test out the probe by running nmap again - nmap -sV -p 8000 127.0.0.1:

 1Starting Nmap 7.80 ( https://nmap.org ) at 2023-08-01 19:58 EDT
 2Nmap scan report for localhost (127.0.0.1)
 3Host is up (0.000078s latency).
 4
 5PORT     STATE SERVICE VERSION
 68000/tcp open  http    DNSSearcher 0.1.0
 7
 8Read from /home/connor/git/service-to-shell/custom_nmap: nmap-service-probes.
 9Read from /usr/bin/../share/nmap: nmap-payloads nmap-services.
10Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
11Nmap done: 1 IP address (1 host up) scanned in 11.23 seconds

There it is, a successful nmap scan using a custom match object. But in the beginning of this post I promised a custom scan with a custom Probe - we can do that too, and also increase the speed of scanning.

A Minimal nmap-service-probes file

What is the smallest nmap probe file we can make that will still successfully scan DNSSearcher? We know from how does it work that some important directives are:

Lets start from a pretty small nmap-service-probes file:

1Exclude T:9100-9107
2Probe TCP NULL q||
3totalwaitms 6000
4tcpwrappedms 3000
5Probe TCP GetRequest q|GET / HTTP/1.0\r\n\r\n|
6rarity 1
7ports 1,70,79,80-85,88,113,139,143,280,497,505,514,515,540,554,591,620,631,783,888,898,900,901,1026,1080,1042,1214,1220,1234,1314,1344,1503,1610,1611,1830,1900,2001,2002,2030,2064,2160,2306,2396,2525,2715,2869,3000,3002,3052,3128,3280,3372,3531,3689,3872,4000,4444,4567,4660,4711,5000,5427,5060,5222,5269,5280,5432,5800-5803,5900,5985,6103,6346,6544,6600,6699,6969,7002,7007,7070,7100,7402,7776,8000-8010,8080-8085,8088,8118,8181,8530,8880-8888,9000,9001,9030,9050,9080,9090,9999,10000,10001,10005,11371,13013,13666,13722,14534,15000,17988,18264,31337,40193,50000,55555
8sslports 443,993,995,1311,1443,3443,4443,5061,5986,7443,8443,8531,9443,10443,14443,44443,60443
9match http m|^HTTP/\d\.\d \d{3} \w+\r\n[dD]ate: [ \w:,]+\r\n[sS]erver:\s+dnssearcher v([\d\.]+)| p/DNSSearcher/ v/$1/ cpe:/a:connorshade:dnssearcher:$1/a

Running this by itself we see a speed improvement from 11.23 seconds to 6.17 seconds, and it only takes up 9 lines. Not bad if you know exactly what you need to scan and just want it done as fast as possible. But if you know exactly what you’re scanning, do we need the port information?

1Probe TCP NULL q||
2totalwaitms 6000
3tcpwrappedms 3000
4Probe TCP GetRequest q|GET / HTTP/1.0\r\n\r\n|
5rarity 1
6match http m|^HTTP/\d\.\d \d{3} \w+\r\n[dD]ate: [ \w:,]+\r\n[sS]erver:\s+dnssearcher v([\d\.]+)| p/DNSSearcher/ v/$1/ cpe:/a:connorshade:dnssearcher:$1/a

No, we actually need none of the port information. This runs perfectly fine - which makes sense, the NULL probe has no ports, so any port directives must function as a filter if they exist, otherwise a probe is run against all ports. Do we need the NULL probe?

1Probe TCP GetRequest q|GET / HTTP/1.0\r\n\r\n|
2match http m|^HTTP/\d\.\d \d{3} \w+\r\n[dD]ate: [ \w:,]+\r\n[sS]erver:\s+dnssearcher v([\d\.]+)| p/DNSSearcher/ v/$1/ cpe:/a:connorshade:dnssearcher:$1/a

The NULL probe is not needed, and this scan runs pretty fast (0.16 seconds) - that’s the fastest service detection I’ve seen in nmap. There you have it - a bare-bones probe that could be the basis for your own detections.

In most cases you should probably just use nmap’s default service-probe file (or one with additions), but if you have to test a bunch of devices in serial, and every second spent on scanning matters, then you can run a stripped probe file for a 100x speed increase.

Wrapping up

At this point, we have:

Check out the two sections below for things I didn’t cover, or ideas for future tooling, and be sure to check out the next post in the series where we’ll make a custom metasploit module to exploit dnssearcher.

Stuff to test

There are some things I didn’t cover in this post:

Possible Tooling

My first experience actually digging into the nmap-service-probes file was when I was trying to manually parse lists of existing banners. I don’t believe that nmap has a way to feed banners in, so I wrote some Python to extract the regular expressions and associated version data from nmap and run banners through these regular expressions.

The original code I wrote was pretty rough. Now that I have a little more experience with nmap, and a stronger desire to find the correct place to put probes and matches, I might write some nmap tooling later. I’m thinking something like sysmon_utils but for nmap - a CLI tool that allows for:

Keep an eye on my GitHub if that sounds useful or interesting.

Prev: Making a Vulnerable Webapp Next: Making a Custom Metasploit Module