Skip to content

Update with lots of fixes and improvements (speed, out-of-bound reads, new output) - #255

Open
ifsnop wants to merge 67 commits into
CroatiaControlLtd:masterfrom
ifsnop:master
Open

ifsnop wants to merge 67 commits into
CroatiaControlLtd:masterfrom
ifsnop:master

Conversation

@ifsnop

@ifsnop ifsnop commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

2.8.11-ifsnop.1 (python_v0.7.11 unchanged)
First version of the fork https://github.com/ifsnop/asterix, based on CroatiaControlLtd/asterix 2.8.11.
The original project has not been updated since December 2025.
Security fixes (found with AddressSanitizer/UBSan and fuzzing):
Out-of-bounds reads when computing item lengths of truncated or malformed records
Out-of-bounds reads in PCAP link/IP/UDP headers and ORADIS headers, endless loop with ORADIS byte count 0
Stack overflow parsing BDS entries of -LF filter files
Memory leak with ORADIS input
Invalid casts of Repetitive sub-formats (BDS) to Fixed
Non printable characters in ASCII fields copied to the output
Stack and static buffer overflows building file names in the disk device
TCP client socket used after close, definition object leaked when asterix.ini is missing
Source IP of each record in TXT, JSON and XML outputs (UDP multicast input and GPS files)
New output format -jx / --json-hex: compact JSON, one record per line, with hex and mask of each field

2.8.11-ifsnop.2 (python_v0.7.11 unchanged)
-jx / --json-hex made to change a value and write it back to ASTERIX:
"fspec" (FSPEC bytes) and "hexdata" (data items after the FSPEC): the record is fspec+hexdata
"pos" in every field: byte offset of its hex in hexdata (first byte of hexdata is pos 0)
"mask" whenever a field does not fill its hex bytes, so hexdata[pos..] = (hex & ~mask) | (value & mask)
Unique keys: FX bits numbered by extent (FX0, FX1, ...), other repeated names numbered (spare, spare1)
-j record header changed (incompatible with the original): "fspec" printed separately and "hexdata" holds
only the data items after the FSPEC (in the original, hexdata was the whole record)

2.8.11-ifsnop.3 (python_v0.7.11 unchanged)
Faster decoding with byte-identical output: -jx about 8x faster (51 MB GPS file: 68 s -> 8.5 s),
other output formats 2-2.6x faster:
format() formats once into a stack buffer instead of twice through the heap
-jx fields written without allocations (hex, mask and pos built in place, mask built once per field)
Items written straight into the output string instead of copied through temporary strings
-jx unique keys computed once per format instead of a std::set per field
Cached FRN -> item and Compound presence-bit lookups
Disk input: no ftell/fstat/select system calls per packet
make rebuilds objects when an included header changes (dependency targets were never matched);
run make clean once after updating

2.8.11-ifsnop.4 (python_v0.7.11 unchanged)
Record header with the FSPEC apart in -jh, -je and the default text output, as in -j and -jx (incompatible):
"fspec" (JSON) / "Fspec:" (text) has the FSPEC bytes and "hexdata" / "HexData:" only the data
items after the FSPEC: the record is fspec+hexdata. XML keeps the whole record in hexdata.
-h documents the default text output and lists -G and -xh in the usage line
Format strings: text from input data or file names is never used as a printf format
(a definitions file name with % made the program crash); format/argument type mismatches fixed
Python publish workflow runs only by hand; CodeQL workflow updated (codeql-action v4, make build)
Text fields (ASCII encoding: callsign, aircraft type, airports, runway, stand, SID/STAR, CAT004
designators...) only print A-Z, 0-9, space and '.' (radio channel names, I205/090); any other
byte is printed as '@'. A quote or backslash in damaged data made -j/-jx invalid JSON.
Visible on captures that pad these fields with control characters: they used to print as
spaces and now print as '@' (e.g. callsign DLHEGN + 0x01 -> DLHEGN@). -jx "hex" keeps the raw bytes.
Malformed records: decoding goes on after them as before, and a WARNING on stderr gives the record
number and the whole data block in hex (CAT, LEN, data) for later analysis. Decoder messages
are no longer cut at 1024 characters.
Fixes found by the new test suite:
HDLC input from files: files under 4 KB decoded nothing and longer files lost their last
partial 4 KB buffer
HDLC: a frame with an Asterix block length of 0 looped forever
PCAP: a corrupt packet length (incl_len) allocated up to 4 GB and could abort (std::bad_alloc);
lengths over 262144 bytes or past the end of the file are rejected
FINAL: a byte count smaller than the record header wrapped around to almost 4 GB (same abort)
XMLParser: a definitions file that failed inside leaked that category
Test suite (tests/, see tests/README.md), run by GitHub Actions on every push and pull request:
golden outputs: 13 inputs x 8 output formats x with and without -LF, including synthetic FINAL,
HDLC and ORADIS files generated from the real samples (tests/tools/gen_samples.py)
-jx consistency (fspec, length, crc, hex/mask/pos/val, unique keys) and agreement of -j, -jh,
-je, text and XML; write-back of edited values through hex/mask/pos
stdin, UDP multicast and loop mode input; command line options
malformed inputs (one per fixed bug) and mutated inputs under AddressSanitizer/UBSan, valid JSON
whatever the input; unit tests (bit fields, getLength contract of every item of every category,
parser, UAP selection, record headers); libFuzzer; line coverage floor (65%, from 48%)
CMake: minimum version 3.10; the decoder is a static library; options ASTERIX_TESTS, ASTERIX_FUZZ and
ASTERIX_COVERAGE. install/test/test.sh prints the diff of a failed case.

2.8.11-ifsnop.5 (python_v0.7.11 unchanged)
-j, -jh, -x and -xh about twice as fast as in ifsnop.4, output byte for byte the same: the field key,
numeric value and text value no longer go through format() (51 MB GPS file: -j 15.4 s -> 7.7 s,
39.9 s before any optimization; -jh 15.6 -> 8.5 s; -x 17.6 -> 8.8 s; -xh 22.0 -> 10.8 s)
README: build and coverage badges of this fork; the coverage job publishes the line coverage to the
badges branch on each push to master

ifsnop and others added 30 commits February 16, 2026 23:01
- DataBlock: print source IP as dotted quad; %lu with an unsigned int
  argument was undefined behaviour on 64-bit.
- python_parser: pass nIP to parse_next_data_block, whose signature
  gained the parameter.
- CUdpDevice: initialise _nSrcIP and clear it at the start of each
  Read() so a stale address is never reported.
- CBaseDevice::getLastSourceIP: return 0 instead of false.
- Regenerate the data outputs in install/sample_output; the only
  differences are the new source IP lines. filter.txt is left as is:
  "Test filter" already fails on master (see trail learning).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
New output format ASTERIX_JSONX (EJSONX). It has the same structure as
-j (one record per line, no trailing comma, so lines can be parsed
independently), but each field is an object with its value and the hex
of the bytes that hold it, plus the bit mask when the field is not a
whole number of bytes (the same rule as -je):

  "SAC":{"val":25,"hex":"19"}
  "V":{"val":0,"hex":"02","mask":"10000000"}

Unlike -je it is valid JSON (':' instead of '='), single line, and has
no name/meaning. -j, -jh and -je are unchanged. EJSONX is appended after
EGPS so existing format numbers do not move.

Checked on the PCAP, ORADIS and GPS samples: every line parses on its
own, and removing hex/mask gives exactly the -j line. New test
"Test jsonx CAT_062_065"; valgrind clean.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
SonarCloud reported "unannotated fall-through between switch labels"
after EJSONX was added below the "/* no break */" comment. Sonar does
not accept the comment, and [[fallthrough]] needs C++17 (the project
is C++11). Instead, all four JSON labels now share one block, and the
"\n\t\t" prefix is added with an if for EJSONH/EJSONE only.

No behaviour change: the -j, -jh, -je and -jx outputs of the four
sample files are byte-identical before and after.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
SonarCloud flagged the explicit delete[] in appendJsonHexMask. The hex
and mask strings are now held in std::unique_ptr<unsigned char[]>.

For that to be safe, the error paths of the bit-field text getters
(six-bit, hex, full-byte hex, mask, octal, ASCII) now return "???"
allocated with new[] (newErrorString<T>) instead of strdup. Callers
release these results with delete[], so the old malloc-based error
result was undefined behaviour for every text output, not only -jx.

No output change: -j, -jh, -je and -jx are byte-identical on the four
sample files; valgrind clean.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Source IP in decoded output and new -jx (compact JSON with hex/mask) format
In the ORADIS branches of the raw, GPS and PCAP subformats:
- The temporary AsterixData whose blocks are spliced into the packet's
  data was never deleted (32 bytes leaked per extra block).
- The 6-byte ORADIS header was read before checking 6 bytes remained.
- A byte count below 6 was not rejected: byteCount - 6 became negative
  (huge length to parsePacket), and a count of 0 never advanced the
  loop.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
item_name is 128 bytes but was read with %128s, which writes up to 129
(the terminating NUL). Triggered by -LF with BDS entries.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The sub-format of a Repetitive item can be DataItemFormatBDS (e.g.
Mode S MB data), so the C-style casts to DataItemFormatFixed were
undefined behaviour (reported by UBSan). Only virtual methods of
DataItemFormat are used, so the base pointer is enough.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The printable test looked at *pData (the first byte of the item)
instead of the character being copied. Control characters were copied
into the output whenever the first byte was printable, and a NUL byte
cut the string short. Now each non printable character becomes a space.
A NULL result from getBits is also handled.

Visible in the output only for ASCII fields holding such bytes (e.g.
the 0x01 trailing byte of some CAT048 callsigns or a NUL in CAT062 RDS).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The failure is only a warning (as in Init()), but the socket was closed
and then used by bind() and connect().

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The headers were read without comparing to incl_len: a short packet
read past the buffer, an IP header length below 5 moved the pointer
backwards, a UDP length below 8 underflowed, and a UDP length larger
than the captured bytes (small snaplen) made parsePacket read past the
buffer. Such packets are now rejected with an error.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
DataItemFormat::getLength(pData) had no limit, so with truncated or
malformed data the length calculation itself read past the end of the
buffer (ASan heap-buffer-overflow in Repetitive::getLength and in
Fixed::isLastPart called from Variable). DataItem::parse compared the
length with the remaining bytes only after it had been computed.
Variable::getLength could also run without limit when the FX chain
continued past the parts defined for the item.

getLength now takes nMaxLength and never reads more than that many
bytes. When the item does not fit it returns a value greater than
nMaxLength, so the existing "needed length > available" checks in
DataItem::parse and DataRecord reject it as before.

- Variable: checks each part fits before reading its FX bit, and a
  zero-length part cannot loop forever.
- Compound: primary and each secondary part are bounded by what is
  left; getText stops and closes the output if a part does not fit.
- Repetitive / Explicit: the repetition / length byte must exist.
- Fixed, BDS: unchanged (no data is read).
- getText of Compound, Explicit and Repetitive, and the Wireshark and
  Python wrappers, pass the item length as the bound.

No output change on valid data: the 81 sample outputs (10 inputs x 8
formats + -L) are byte-identical. With 960 mutated inputs, master has
sanitizer errors on 522 and this branch on none. The Wireshark wrapper
was compiled with -DWIRESHARK_WRAPPER. The Python wrapper was only
syntax-checked against a stub Python.h, because the headers are not
installed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
- DoneWithFile: sfxFormat was 10 bytes but sprintf wrote ".ci<pid>_
  %y%m%d%H%M%S" into it (up to 24 bytes), a stack overflow with any
  pid. It is now 32 bytes; strftime uses sizeof(suffix).
- NextFileName: newName is static, so strncpy without a terminator let
  strcat append after the previous call's name, and the extension
  strcat was unbounded. Built with snprintf now, and truncation is
  logged.
- DoneWithFile / DoneAll: strncpy + strcat replaced with snprintf
  sized to the destination.

These paths are only used by the MARKDONE, PACKETFILE and disk output
modes, which the CLI does not build. They removed the
-Wstringop-truncation warnings: diskdevice.cxx now builds clean with
-Wall -O2. The new NextFileName gives the same names as before for
normal paths (checked on 21 cases), and the 81 sample outputs are
unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Fix buffer overflows, leaks and invalid casts in the decoder
The original CroatiaControlLtd/asterix has not been updated since
December 2025, and its decoder had memory-safety problems that are
fixed in this fork.

- version.h: version 2.8.11-ifsnop.1 ("<upstream>-ifsnop.<n>"), plus
  the upstream version and the fork URL. asterix -V / -h now print the
  fork origin under the version line. The Croatia Control copyright
  notice is kept.
- README.md and README: fork notice with the reasons and the changes.
  README.md also says that the PyPI package and the badges belong to
  the original project.
- HISTORY: entry for 2.8.11-ifsnop.1.
- The Python module version (0.7.11) is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Mark as fork: version 2.8.11-ifsnop.1 and fork notice in README
-jx is meant for changing a value and writing it back to ASTERIX at
minimal cost. Until now it lost information and gave no position:

- Unique keys: Variable items repeated FX (and some items repeated
  spare) in the same object, so standard JSON parsers kept only the
  last one (110 of 281 sample records). FX bits are now numbered by
  extent (FX0, FX1, ...) and other repeated names get the first free
  number (spare, spare1).
- mask whenever the field does not fill its hex bytes. The -je rule
  (mask only when the bit length is not a multiple of 8) dropped the
  mask of 8*n-bit fields that start mid-byte (BDS res1), so the byte
  they share with the next field could not be told apart.
- "fspec" (bytes while the last bit is 1) is split from "hexdata",
  which now holds only the data items: the record is fspec + hexdata.
- "pos" on every field: byte offset of its hex in hexdata. A value is
  written back with hexdata[pos..] = (hex & ~mask) | (value & mask),
  with no knowledge of the item structure (Compound, Repetitive,
  Explicit, BDS). DataRecord stores each item's offset (m_nOffset),
  and DataItemBits::JsonPosition carries it while the item is written.

-j, -jh and -je are unchanged (byte-identical on all samples).

Checked on all samples (451 records, 18850 fields): every pos points at
its hex, fspec + hexdata equals the old hexdata, values are identical
to -j, and there are no duplicate keys. Round trip: each of 10408 fields
inverted inside its mask, re-encoded and decoded again. 10268 change
only that field, 0 are wrong, and 140 change the record structure
(BDS register number, CAT001 I020 TYP UAP selector) as expected. ASan/
UBSan and 960 mutated inputs are clean.

The trail learning on -jx is replaced, since its mask rule and the
missing-structure limits no longer hold.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
- "fspec" is now printed from the FSPEC bytes stored at parse time
  (DataRecord::m_pFSPECData / m_nFSPECLength) instead of being cut from
  the hexdata string. The FSPEC is never recomputed. -jx output is
  byte-identical.
- -j gets the same record header as -jx: "fspec" apart, and "hexdata"
  with only the data items after the FSPEC. In -jx, "pos" is a byte
  offset in this hexdata (its first byte is pos 0).

Breaking change for -j, decided by the user: in the original project
the hexdata of -j was the whole record, and now the whole record is
fspec + hexdata. Documented in README.md and README. The expected
outputs of the two -j tests were regenerated (header lines only).
-jh, -je, XML and TXT are unchanged.

trail check reports a conflict with the -jx learning on master
(l-ce031d62), which says -j is unchanged. This branch replaces that
learning (now l-5bf7ab58), but the replacement only counts once it is
merged, so the conflict is expected until then.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The -j record header change (fspec split from hexdata) is incompatible
with 2.8.11-ifsnop.1, so the fork revision goes up. HISTORY lists the
-jx write-back changes (fspec, pos, full masks, unique keys) and the -j
header change.

trail check still reports the conflict on cat_062_065_json.txt: the
learning on master says -j does not change, and this branch replaces it
(l-5bf7ab58), which only counts once PR #4 is merged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
-jx: edit a value and write it back to ASTERIX (fspec, pos, full masks, unique keys); -j header with fspec; version 2.8.11-ifsnop.2
The rules did generate header dependencies but they never matched. gcc
wrote "../../obj/<dir>/release/X.o: X.cpp X.h ..." and a sed then
rewrote the target as "$(OBJ_DIR_SED)/X.o $(OBJ_DIR_SED)/X.d :".
OBJ_DIR_SED was never defined, so the included .d files held rules for
/X.o at the filesystem root and the real objects only depended on their
source file. Editing a header (version.h, a class layout in UAP.h)
left stale objects: -V kept the old version, and objects built against
two layouts of the same class crashed at run time.

The five compile rules now use -MMD -MP -MF $(OBJ_DIR)/X.d: gcc names
the target after -o, so no sed is needed; -MP adds empty rules for the
headers, so deleting or renaming one does not break the build. System
headers are left out of the .d files.

Checked: second make does nothing; touching asterix/UAP.h, main/version.h,
engine/basedevice.hxx or asterix/Utils.h rebuilds exactly the objects
that include them, the libraries and the executable; removing a header
and its include builds; make debug gets the same .d files; -jx output
and install/test unchanged. Run make clean once after updating so old
.d files are replaced.

Replaces the learning on version bumps, whose forced-rebuild warning
this fixes (l-b9b7da47 -> l-949a9c1c).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
make: rebuild objects when an included header changes
format() called vsnprintf twice (once to measure, once to write) and
copied the result through a heap buffer, for every fragment of output:
about 45% of the -jx run time. It now formats into a 512-byte stack
buffer and only takes the old two-pass path for longer results.

New helpers write exactly what the matching printf conversion would:
appendHexByte/appendHexBytes ("%02X"), writeLong/appendLong ("%ld") and
appendDouble7 ("%.7lf", through std::to_chars when built as C++17,
which is specified to match printf; snprintf under C++11).

-jx of a 51 MB GPS file: 68 s -> 50 s, output identical.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Every -jx bits field went through getBits (new[]), sprintf("%02X") per
byte and three format() calls for hex, mask and pos, plus getBits again
in getUnsigned.

- getUnsigned reads the (at most 5) bytes of the field and shifts the
  value out; the old getBits path stays for out-of-range requests.
- appendJsonHexMask builds ,"hex":"..."[,"mask":"..."],"pos":N in a
  stack buffer and appends it once. The hex is the bytes holding the
  field, the mask depends only on m_nFrom/m_nTo and is built once per
  field (rebuilt if they change).
- Key, "val" and closing text are appended directly.

Same text as getHexBitStringFullByte/getHexBitStringMask and the
format() calls they replace. -jx of a 51 MB GPS file: 50 s -> 25 s,
output identical.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Each level built its text in a temporary string that the level above
copied again: DataItem -> DataRecord -> AsterixData, and the parts of
Variable and Compound items (Variable also through substr).

Now each level appends to the caller's string and, when the item prints
nothing, cuts it back to where it started. Separators and braces are
the same: Variable parts are written as {...} and the braces trimmed in
place, the record header is removed when no item prints.

The record header writes the long fspec/hexdata strings directly
instead of through one format() call, and the record hexdata is built
with a table instead of snprintf per byte.

All output formats identical on the samples (with and without -LF).
-jx of a 51 MB GPS file: 25 s -> 20 s.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
jsonUniqueKey filled a std::set<std::string> for every field of every
record (a new set per JSON object) to number repeated names (spare,
spare1...) and FX bits (FX0, FX1...). The keys depend only on the
format: for a Fixed item on its own, and for each extent of a Variable
item (the parts before an extent are always the same ones).

- DataItemFormatFixed::makeJsonPrefixes runs the same jsonUniqueKey
  algorithm and stores each key as its ready-made "key":{ prefix.
- A Fixed item keeps its prefixes; a Variable item keeps them per
  extent (each extent its own object when the item has one sub item,
  keys accumulated across parts otherwise, as before).
- DataItemBits::getText takes the prefix instead of the bare key.

-jx of a 51 MB GPS file: 20 s -> 15 s, output identical.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
ifsnop and others added 29 commits November 7, 2026 19:33
Fix CodeQL alerts: format strings and publish workflow permissions
-j and -jx already printed "fspec" (the FSPEC bytes) and a "hexdata"
holding only the data items after it, so the record is fspec + hexdata.
-jh, -je and the default text output now use the same record header:
- text: a "Fspec: ..." line before "HexData: ...", which loses the
  FSPEC bytes;
- -jh / -je: "fspec":"..." before "hexdata":"...", same layout as the
  rest of their header.
Only XML (-x/-xh) keeps the whole record in hexdata. The data block
"HexData:" line of the text output (CAT+LEN of the block) is unchanged.

Incompatible for readers of hexdata in -jh, -je and text: use
fspec + hexdata to get the whole record.

The FSPEC hex and the items hex are computed once per record for the
text and JSON formats (DataRecord::getText).

Expected outputs regenerated for the affected tests only (filter.txt
left as it is). Checked by rebuilding the old output from the new one
(old hexdata == new fspec + new hexdata, all other lines equal) on all
samples and a slice of a 51 MB GPS file, with every FSPEC well formed;
-l, -x, -xh, -j and -jx are byte-identical; ASan/UBSan clean.

Replaces the learning on the hex JSON output (l-5bf7ab58 ->
l-565293c9): it said -jh, -je and text kept the whole record.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
- -h: a "(default)" entry under "Output format" describes the text
  output used when no output format option is given (data block and
  record headers, Fspec + HexData, items with values, units and
  meanings); -j, -jh, -je and -jx note the fspec/hexdata record header;
  the usage line lists -G and -xh, which were only in the option list.
- README usage block mirrors -h (its usage line had the same gaps).
- README.md: the record header item now covers -j, -jh, -je and text.
- HISTORY: 2.8.11-ifsnop.4, including the changes merged after
  2.8.11-ifsnop.3 (format strings, CodeQL and publish workflows).

The fork revision goes up because the -jh/-je/text hexdata change is
incompatible with 2.8.11-ifsnop.3.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
FSPEC apart in -jh, -je and text output; document the default output in -h; 2.8.11-ifsnop.4
"Test filter" compares `asterix -L` (every configured item) with
install/sample_output/filter.txt, which was never regenerated after the
XML definitions changed:
- CAT015 was added (148 items missing from filter.txt);
- CAT001 I100 QB1/QD1 were renamed QB4/QD4.

filter.txt is regenerated with -L. It is also the -LF input of the
filtered CAT001/002 and CAT034/048 tests; their expected outputs were
regenerated too and did not change, because the samples contain no
CAT001 I100 and no CAT015 records.

install/test/test.sh now passes completely ("All tests OK").

Replaces the learning that kept this test failing on purpose
(l-2ef9bc59 -> l-5bec4220).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Tests: refresh filter.txt so "Test filter" passes
The HDLC reader always asked the device for MAX_RXBUF (4096) bytes, and
a file device only returns complete reads: an HDLC file under 4 KB
decoded nothing, and longer files lost their last partial buffer
(a 5 KB file with 200 records gave 155).

When the device knows how many bytes are left (files), the reader asks
for at most that. Devices that do not know (serial, stdin: 0) keep
reading MAX_RXBUF as before. Read() itself is unchanged: the ORADIS and
GPS readers rely on it failing when a block is incomplete.

Found by the synthetic HDLC sample of the new test suite.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
A <Category> is handed to the definitions at </Category>. When a
definitions file failed to parse inside a category (syntax error,
unknown element), that Category and its data items were leaked.
~XMLParser() now deletes it; after a successful handover the pointer
is already NULL.

Found by LeakSanitizer in the new unit tests (definitions file with a
syntax error).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
proc_hdlc_frame() walks the Asterix blocks of a frame by their length
field. A block length of 0 never advanced, so a frame with a valid FCS
and such a block looped forever (reachable from serial input and from
HDLC files of 4 KB or more; smaller files did not get that far before
the previous fix). The frame is now rejected when a block length is
under 3 (its own header) or when no room is left for a header.

Found by the mutated-input run of the new test suite; the regression
input is tests/data/bad/hdlc_block_len_zero.hdlc.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The packet buffer was allocated with incl_len straight from the file.
A corrupt value asked for up to 4 GB: depending on the memory available
the program aborted with an uncaught std::bad_alloc, or read on with a
huge buffer. incl_len must now be between 1 and libpcap's
MAXIMUM_SNAPLEN (262144) and, for files, not more than what is left;
otherwise the packet is reported and skipped like any other read error.

Found by the mutated-input run of the new test suite; the regression
input is tests/data/bad/pcap_huge_incl_len.pcap.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The record length was computed as byte count - header - padding in an
unsigned variable, so a byte count under 12 wrapped around to almost
4 GB and the "neededLen <= 0" check could never fire: the buffer
allocation then aborted with std::bad_alloc (or read on with a huge
buffer when memory allowed). The byte count must now be larger than
header + padding and, for files, fit in what is left.

Found by the robustness run of the new test suite with a memory limit;
the regression input is tests/data/bad/final_count_too_small.final.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
tests/run_tests.sh [binary] [sections] checks the command line tool:
- golden: 13 inputs (tests/cases.txt) x 8 output formats x with and
  without -LF, against tests/golden (UPDATE=1 rewrites them). Wall-clock
  timestamps of raw/ORADIS input are masked.
- consistency (tools/check_outputs.py): every -jx record has a well
  formed fspec, fspec+hexdata of "length" bytes and a matching crc;
  every field's hex is at pos in hexdata, its mask fits and an integer
  val equals the masked bits; keys are unique; fspec/hexdata agree in
  -j, -jh, -je, text and XML; -j values equal -jx val.
- roundtrip (tools/roundtrip.py): data fields are edited with
  hexdata[pos..] = (hex & ~mask) | (value & mask), the records rebuilt
  and decoded again: exactly the edited values change.
- inputs: stdin, UDP multicast on loopback, loop mode -o.
- robustness: malformed inputs (tests/data/bad), optional mutated
  inputs (MUTANTS=dir, tools/mutate.py), a % in the definitions file
  name; runs must end in time, without signal or sanitizer report
  (MEMLIMIT_KB caps memory so huge allocations fail everywhere).
- cli: -h, -V, option errors, -L against filter.txt.

tests/tools/gen_samples.py builds the synthetic inputs from the real
samples: the CAT001/002 blocks re-wrapped as raw, ORADIS, FINAL and
HDLC files, a stream with every category, and malformed inputs (each
of them a regression of a fixed bug). tests/data/filter_subset.txt is
an -LF filter that really filters (filter.txt lists every item).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
… as a library

CMake (minimum 3.10, as CMake 4 refuses older minimums):
- asterix_decoder: static library with the decoder (src/asterix without
  the packet formats), linked by the application, the unit tests and
  the fuzz target.
- -DASTERIX_TESTS=ON: tests/unit (asterix_unit_tests) and ctest, which
  also runs tests/run_tests.sh with the CMake binary.
- -DASTERIX_FUZZ=ON: tests/fuzz/fuzz_parse, libFuzzer with clang (the
  decoder gets -fsanitize=fuzzer-no-link), a plain program that runs the
  given files with other compilers.
- -DASTERIX_COVERAGE=ON: gcov build; tests/tools/gcov_flush.c turns
  SIGTERM into exit() so stopped runs keep their data.

Unit tests (tests/unit, own single-header harness, C++11): bit fields
and encodings through getText, the -jx hex/mask/pos text, the
getLength contract (no over-read, > nMaxLength when it does not fit,
same length on its own bytes) on every item of every category, parser
edge cases, CAT001 UAP selection, record headers of every format,
format()/append helpers against printf, regressions (Tracer %, XML
errors, truncated BDS).

tests/tools/coverage_report.py reports line coverage of src/ and fails
under tests/coverage_min.txt (63%; 64.6% measured, 48% before).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
.github/workflows/tests.yml:
- make: release and debug build, install/test (with valgrind),
  tests/run_tests.sh, and a check that tests/data matches
  gen_samples.py.
- cmake: C++11 build with gcc and clang, ctest (unit tests and
  run_tests.sh with that binary).
- sanitizers: ASan+UBSan build, unit tests with leak detection,
  run_tests.sh with 520 mutated inputs, the decoder on every input.
- fuzz: libFuzzer for 60 s on each push, 30 min weekly; the crashing
  input is kept as an artifact.
- coverage: gcov build, all tests, report in the job summary, fails
  under tests/coverage_min.txt.

tests/README.md describes the layers, how to run them locally and what
to update when an output changes on purpose. install/test/test.sh now
prints the diff of a failed case.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Text fields (callsign, aircraft type, airports, runway, stand, SID/STAR,
CAT004 designators...) printed every printable character as is, so a
quote or backslash in damaged data made -j/-jx invalid JSON (12 lines
over 1950 mutated inputs), and '<' or '&' would break XML.

Per the ASTERIX definitions these fields are upper-case alphanumeric
padded with spaces (e.g. I062/390: "up to seven upper-case
alphanumeric characters ... padded with space characters"; ICAO
aircraft identification: alphanumeric, no symbols). The radio channel
name of I205/090 also has a decimal point (121.100). Every other
character, control characters included, is now printed as '@', which no
output format has to escape. -jx "hex" keeps the raw bytes.

Golden outputs: 582 characters changed in 18 files, all of them a
control character used as padding in the samples (e.g. callsign
DLHEGN + 0x01, runway NU2 0x00) that was printed as a space and is
now '@'.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The output is unchanged: after a malformed record, decoding goes on
from where that record stopped (most records after it decode correctly,
some may be wrong, as ASTERIX records carry no length). DataBlock now
reports it once per block, so the data can be checked later:

  WARNING: ... malformed record 4 in data block CAT001 (59 bytes),
  0 record(s) decoded after it may be wrong. Block: 01003BF7C6...

The block is printed in hex as received (CAT, LEN and data). Tracer
and the application's debug_trace accept messages of any length (they
were cut at 1024 characters), and debug_trace prints decoder messages
starting with "Warning: " as WARNING instead of ERROR.

Tests: unit tests for the warning (whole block, no warning on a good
block, messages over 1024 characters); run_tests.sh checks the warning
on tests/data/bad/raw_broken_record_in_block.raw and that -j/-jx output
of malformed and mutated inputs is valid JSON.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Test suite run on every push and pull request; 5 decoder fixes it found
Text fields keep only A-Z, 0-9, space and '.'; warning with the whole block on malformed records
PR #12 was merged into its base branch test_suite after PR #11 had
already been merged into master, so its commits never reached master.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…fsnop.3

2.8.11-ifsnop.4 was set in version.h by PR #9 but never tagged; this
release also includes everything merged after it:
- record header with the FSPEC apart in -jh, -je and text (PR #9);
- format-string fixes and workflow updates (PRs #7, #8);
- the test suite run on every push and pull request (PR #11) and the
  five decoder fixes it found (HDLC files and block length 0, PCAP
  packet length, FINAL byte count, XMLParser leak);
- text fields limited to A-Z, 0-9, space and '.', and the warning with
  the whole block on malformed records (PR #12).

README.md lists the test suite and the fixes and explains how to run
the tests; README gets a TESTS section and the CMake options; HISTORY
details every change of 2.8.11-ifsnop.4.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
- build: state of the Tests workflow on master (shields.io, label
  "build").
- coverage: on each push to master, the coverage job writes the line
  coverage as a shields.io endpoint JSON (COVERAGE_BADGE in
  coverage_report.py) to badges/coverage.json. The badges branch holds
  that one file in a single commit, force-pushed every time. That step
  is the only one with write permission (job-level contents: write).

The README now says which badges are this fork's and which belong to
the original project.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
README: build and coverage badges
The -jx path of DataItemBits::getText was rewritten with direct appends
in the speed-up work, but the field key, the numeric value and the text
value of -j, -jh, -x and -xh still went through format() (vsnprintf):
61% of the run time of -j. They now use the existing helpers
(appendLong for %ld, appendDouble7 for %.7lf, direct appends for keys,
quoted strings, XML tags and the separator), writing the same text.

51 MB GPS file (asterix_enaire.ini, -G), output identical (md5):
  -j    15.4 s -> 7.7 s   (39.9 s before any optimization)
  -jh   15.6 s -> 8.5 s
  -x    17.6 s -> 8.8 s
  -xh   22.0 s -> 10.8 s

The 208 golden outputs and install/test pass unchanged; unit tests and
the ASan run with 520 mutated inputs are clean.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
-j, -jh, -x and -xh about twice as fast, same output
Includes since 2.8.11-ifsnop.4:
- -j, -jh, -x and -xh about twice as fast, same output (PR #14);
- build and coverage badges in the README (PR #13).

No output change, so it is compatible with ifsnop.4; the revision goes
up to mark the new release, as requested.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
(cherry picked from commit 43fede5)
The upstream fix (previous commit, CroatiaControlLtd#254) only
touched asterix/config, used by the Python module. Copy it to
install/config so the C++ binary decodes CAT048 RE too, and regenerate
install/sample_output/filter.txt with -L for the new CAT048:IRE:VAL.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
3.1% Duplication on New Code (required ≤ 3%)
D Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant