nginx request fingerprint module

Adds the $req_fp variable that encodes a compact fingerprint of every HTTP request/response pair.

---

Overview

ngx_http_req_fp_module is an nginx C module that computes a compact, structured fingerprint string for every HTTP request/response pair and exposes it as the $req_fp nginx variable. The fingerprint captures 17 fixed-position fields — request structure, parameter shapes, header patterns, client context, and response outcome — encoded as a short underscore-separated string (~50–200 bytes).

5FX6Li0LMETnpsF7YNY_ge_11_02_aae_ssz_4-7-25_0000_07_aacchuu_enus_n_c_s_r_200_712

Because it is a native nginx variable it is available inline — loggable, passable to upstreams, usable in map blocks, or injectable as a request header — with no external tooling required for collection.

---

Why not existing tools?

Tool	Limitation
hfinger	Offline only — consumes pcap files after the fact; no live path
JA4	TLS layer only — sees the handshake, not HTTP structure or parameters
JA4H	HTTP layer — limited to header patterns, not parameter shapes or response details
ModSecurity + CRS	Rule-based anomaly scoring, not fingerprinting; heavyweight; not designed for ML consumption
Zeek / Bro	Separate network-tap infrastructure required; operates on raw packets
Akamai HTTP/2 FP	Closed research; HTTP/2 SETTINGS only; doesn't capture application-layer signals

This module sits inside nginx, requires no separate infrastructure, captures both network-layer signals (HTTP version, header patterns) and application-layer signals (parameter shapes, content types, auth presence) in a single pass, and produces structured output that needs zero further parsing to feed an AI or ML pipeline.

---

Why not store full requests and responses?

Storage cost — A full request with headers, body, and response can be kilobytes to megabytes. A fingerprint is ~100–200 bytes. At 10 000 req/s that is the difference between ~10 GB/day and ~150 MB/day of log data.

Privacy and compliance — Full requests contain passwords, session tokens, PII, health data, and payment information. The fingerprint captures shape (param count, value types, value lengths) without capturing content. You can retain fingerprints indefinitely under most data-handling policies; you cannot do that with raw requests.

ML/AI readiness — Full HTTP logs require complex parsing, tokenization, and normalization before any model can consume them. Fingerprint fields are already structured, typed, and bounded — they load directly into a feature matrix.

Noise reduction — Most bytes in a raw request are irrelevant to anomaly detection (the actual values of normal parameters, response HTML, etc.). The fingerprint discards that noise at source.

---

Goal

Identify requests that fall outside the expected behavior patterns of a specific application.

A given web application has a stable fingerprint distribution: specific paths receive specific methods, specific parameter shapes, specific header sets, and specific content types. Legitimate browsers and API clients produce consistent, predictable fingerprints. Attackers, scanners, misconfigured clients, and bots do not.

By collecting fingerprints over time you build a behavioral baseline per endpoint — what does a normal POST /api/login look like? What header set does a normal browser send? What parameter types does /search always receive? — and then flag anything that deviates: missing headers, unexpected auth methods, wrong content types for the method, unusual path depths, parameters that are suddenly objects instead of strings.

The computation is a single cheap pass over already-parsed nginx structures, so the baseline is continuously maintained with no sampling tradeoff.

---

Fingerprint format

{path_b62}_{method2}_{http_ver}_{path_depth}_{param_keys}_{param_types}_{param_lens}_{req_ctype}_{hdr_count}_{hdr_list}_{accept_lang}_{auth_type}_{cookie}_{cookie_fields}_{referer}_{status}_{body_bytes}

17 underscore-separated fields, grouped by purpose:

Request line

#	Field	Description
1	path_b62	URI path bytes as a big-endian integer, base62-encoded
2	method2	First 2 chars of HTTP method, lowercased (ge, po, pu, de, …)
3	http_ver	2-digit protocol version: 09 10 11 20 30
4	path_depth	Number of / chars in URI, zero-padded 2 digits, max 99

Parameters (query string + application/x-www-form-urlencoded body, combined and sorted)

#	Field	Description
5	param_keys	First char of each param name, sorted; nil if none
6	param_types	Type code per value (same order); nil if none
7	param_lens	Decoded value lengths, dash-separated; 0 if none
8	req_ctype	First 4 alpha chars of request Content-Type subtype; 0000 if absent (e.g. json, html, xwww, form)

Request headers

#	Field	Description
9	hdr_count	Total request header count, zero-padded 2 digits, max 99
10	hdr_list	Sorted first-char initials of all header names (limit 32); nil if none

Client context

#	Field	Description
11	accept_lang	First 4 lowercase alpha chars of primary Accept-Language tag (hyphens/digits stripped); 0000 if absent
12	auth_type	n none · b Basic · t Bearer/token · d Digest · o other
13	cookie	c if Cookie header present, n if absent
14	cookie_fields	Sorted first-char initials of cookie field names (limit 32); nil if none
15	referer	n no Referer · s same-domain · x cross-domain

Response

#	Field	Description
16	status	HTTP response status code; 0 if not yet sent
17	body_bytes	Response body bytes; 0 if none

Value type codes (field 5)

Code	Type	Example values
i	integer	42, -7
f	float	3.14, -0.5
s	string	hello, foo-bar
c	char (single char)	x (len == 1, no other match)
b	bool	true, false
t	time	14:30:00
d	date	2026-02-19
z	datetime with timezone	2026-02-19T22:54:13+05:00, 2026-01-01T00:00:00Z
e	empty	`` (zero-length value)
o	object	{…} (starts with {)
l	list / array	[…] (starts with [)

Examples

GET /path/file.ext?a1=test&a2=foo-bar&endTime=2026-02-19T22:54:13%2B05:00
  (7 headers, no request body, Accept-Language: en-US,
   no auth, session cookie, came from same domain)
→ 5FX6Li0LMETnpsF7YNY_ge_11_02_aae_ssz_4-7-25_0000_07_aacchuu_enus_n_c_s_r_200_712

GET /
  (5 headers, no Accept-Language, no auth, no cookie, no referer)
→ l_ge_11_01_nil_nil_0_0000_05_achuu_0000_n_n_nil_n_302_0

Field-by-field breakdown of example 1:

5FX6Li0LMETnpsF7YNY  path /path/file.ext base62
ge                   GET
11                   HTTP/1.1
02                   path depth: /path/file.ext has 2 slashes
aae                  params: a1, a2, endTime → first chars sorted
ssz                  types:  string, string, datetime+tz
4-7-25               lengths: 4, 7, 25
0000                 no request Content-Type (GET has no body)
07                   7 request headers
aacchuu              header initials: accept, accept-encoding, cache-control,
                       content-type, host, user-agent (example)
enus                 Accept-Language: en-US → first 4 alpha chars lowercase
n                    no Authorization header
c                    cookie present
s                    cookie field "session" → first char 's'
r                    referer is same domain
200                  HTTP 200
712                  712 bytes response body

---

Building

Static (built into nginx)

./configure --add-module=/path/to/nginx_req_fp
make
make install

Dynamic (DSO)

./configure --add-dynamic-module=/path/to/nginx_req_fp
make modules
# copy objs/ngx_http_req_fp_module.so to your nginx modules directory

Load the dynamic module in nginx.conf:

load_module modules/ngx_http_req_fp_module.so;

---

Usage

http {
    log_format fp '$req_fp $remote_addr "$request"';
    access_log /var/log/nginx/fp.log fp;

    server {
        listen 80;
        location / {
            # $req_fp is also usable in headers, map blocks, etc.
            add_header X-FP $req_fp;
            proxy_pass http://backend;
        }
    }
}

---

Implementation notes

• Base62 alphabet: 0-9A-Za-z
• Parameter source: URL query string (r->args) and request body for methods that carry one (POST, PUT, PATCH, etc.). Body params are parsed when Content-Type: application/x-www-form-urlencoded is present and the body has already been read into memory by nginx (e.g. via proxy_pass, fastcgi_pass, or client_body_buffer_size keeping the body in RAM). Body params are appended to query params, then the combined list is sorted. File-backed body chunks (bodies spilled to disk beyond client_body_buffer_size) are skipped.
• Max params: 32. Additional params beyond the limit are silently ignored.
• URL decoding: param names and values are decoded before processing; + is treated as a space per the application/x-www-form-urlencoded convention.
• Body counting: a body filter accumulates the actual bytes flowing through the filter chain, so the count reflects the real wire size (after any sub-filters have transformed the content). If the body filter hasn't run yet (e.g. when the variable is accessed in a header directive), the declared Content-Length is used as a fallback.
• URI path limit: URIs longer than 2048 bytes are truncated to that length before base62 encoding.
• Key dependency: r->request_body is only populated if something upstream already called ngx_http_read_client_request_body — which proxy_pass, fastcgi_pass, uwsgi_pass, and similar directives do automatically. If you need it in a pure nginx location (no upstream), add lua_need_request_body on; (OpenResty) or a custom rewrite-phase read in the config.

---

Testing the base62 encoding

/   → 0x2F = 47 → 47 / 62 = 0 rem 47 → base62[47] = 'l'  ✓

For multi-byte paths, treat the raw bytes as a big-endian unsigned integer and encode in base62 using the alphabet above.

---

License

Licensed under the Apache License 2.0 with the Commons Clause condition.

• You may use, copy, modify, and distribute this software freely under the terms of the Apache 2.0 license.
• The Commons Clause restricts selling the software or offering it as a hosted/managed service for a fee.

See LICENSE for the full text.