Add --frame-packing option to emit the frame packing arrangement SEI

[danielcamposramos]

x265 defines the SEI payload type FRAME_PACKING but had no way to write it.
An HEVC encode of side-by-side or top-and-bottom 3D video could not carry the in-stream mark that players and 3D televisions read.
This adds the option, as discussed in #970.

What it adds

--frame-packing <integer> with the values H.265 defines in Table D.8.
3 is side by side, 4 is top and bottom, 5 is temporal interleaving.
It is off by default, and the bitstream does not change when it is not given.
Other values are rejected with an error, because H.265 does not define them.
Frame 0 is always signalled as the left view, as x264 does.

Where the SEI is written

It is a prefix SEI NAL unit written in FrameEncoder::compressFrame(), next to the alternative transfer characteristics SEI.
Types 3 and 4 write it with each IRAP picture, with the persistence flag set.
Type 5 writes it with every picture, because current_frame_is_frame0_flag alternates, with the persistence flag clear.
The syntax is H.265 D.2.16 and the semantics are D.3.16.
The new SEIFramePacking class is in sei.h.

API

The new member framePacking is at the end of x265_param, with X265_BUILD raised to 218.
Parsing, default, copy, the parameter string, the CLI help, cli.rst and the test lists are updated.

How it was tested

The SEI NAL units were compared byte for byte with an independent writer that follows the D.2.16 syntax table field by field.
They matched for types 3 and 4 on every IRAP picture, and for type 5 on every picture with the flag alternating in output order.
The same writer, in its H.264 form, reproduces the 16-byte message x264 writes.
Over 60-frame clips with several GOPs, with and without B-frames, open and closed GOP, there is one SEI with each IRAP picture and none elsewhere for types 3 and 4, and one on every picture for type 5.
Without the option the output is byte-identical to the unpatched build (--frame-threads 1 --pools 1).
The 8-bit and 10-bit builds have no new warnings.
FFmpeg 9.0.2 reads the message and reports Stereo 3D side data with the right type on the pictures (and the left and right view alternating for type 5).
A stock FFmpeg with libx264 carries the layout through a transcode to H.264.
The clips were also played on a Linux stereo desktop: the player reads the SEI from the stream and declares the video as stereo, and each eye gets its own view.
The captures are attached (hevc-sbs-capture.png for --frame-packing 3, hevc-tb-capture.png for --frame-packing 4): LEFT in the left eye, RIGHT in the right eye, the same frame number in both.

Reference SEI

The prefix SEI NAL units the option writes, without the start code, for the byte-for-byte comparison:

Option	NAL unit
--frame-packing 3	4E 01 2D 06 81 81 00 00 03 00 02 80
--frame-packing 4	4E 01 2D 06 82 01 00 00 03 00 02 80
--frame-packing 5, frame 0	4E 01 2D 04 82 81 10 00 80
--frame-packing 5, frame 1	4E 01 2D 04 82 81 00 00 80

Notes

I did not run TestBench, as no primitives are touched.
The three lines in regression-tests.txt and the one in smoke-tests.txt use big_buck_bunny_360p24.y4m, which I have not run through the CI harness.

[vunguyen1989]

Hi @danielcamposramos , I noticed something unexpected with frame-packing type 3 when using --single-sei.

I tested commit c1362f8 with an SBS Y4M input:

  x265 --y4m --preset ultrafast --frames 48 \
    --frame-threads 1 --pools none \
    --keyint 12 --min-keyint 12 --no-scenecut --bframes 0 \
    --no-info --no-aud --no-repeat-headers \
    --frame-packing 3 --single-sei \
    --input input.y4m --output single-sei.h265

  ffmpeg -hide_banner -nostdin -i single-sei.h265 \
    -map 0:v:0 -c:v copy -bsf:v trace_headers -f null -

With --no-single-sei, all four FPA messages before POC 0/12/24/36 are valid. With --single-sei, only the first is valid; FFmpeg reports Invalid SEI message: payload_size too large for the other three (log is attached)

For example, before POC 12, the prefix SEI bytes excluding the start code are:

Expected: 4E 01 2D 06 81 81 00 00 03 00 02 80
Actual: 4E 01 D0 59 FE 20 EC 2D 06 81 81 00 00 03 00 02 80

The extra five bytes match the beginning of the preceding picture’s slice-header payload. Could you take a look at whether the buffer is being reset correctly before collecting the prefix SEI messages?

single-sei_header.log

[danielcamposramos]

Thank you @vunguyen1989, good catch, and thank you for the exact command and bytes.

You were right: m_bs was not reset before the prefix SEI messages were collected.
In single SEI NAL mode it is reset only by the access unit delimiter or by the repeated stream headers, so with --no-aud, on a picture without repeated headers, it still held the previous picture's slice header.

This is older than --frame-packing.
On master, without our option, --hrd --single-sei --no-aud --no-repeat-headers gives the same FFmpeg error on the picture timing SEI of every picture after the first (47 of 48 SEI NAL units in a 48 picture encode).
The frame packing message made it visible in one more combination.

The fix is a separate commit, 4712634, which resets m_bs in that case and in no other.
With your command, the four messages before POC 0/12/24/36 are now all 4E 01 2D 06 81 81 00 00 03 00 02 80, and trace_headers reports no errors.

What I checked, 8 and 10 bit:

• types 3, 4 and 5 with --single-sei in all four --aud/--no-aud and --repeat-headers/--no-repeat-headers combinations: every prefix SEI NAL unit parses exactly, and every frame packing payload matches an independent reference writer;
• --single-sei with --hrd and --max-cll in the same NAL units: clean;
• streams that were already correct do not change: with --aud, without --single-sei, and on keyframes with --repeat-headers the output is byte-identical to before the fix, and without --frame-packing it is byte-identical to master.

Our earlier tests ran --single-sei only together with --repeat-headers, which hides the problem on keyframes, so your case is now part of our test matrix.

If you prefer the fix as its own PR, since it also affects master, I can split it out.

[vunguyen1989]

Thanks for the detailed follow-up and for verifying all those combinations.

Since the m_bs issue also affects master independently of frame packing, I think splitting the fix into its own PR would be cleaner and make the frame-packing PR easier to review.

Glad the test case helped uncover it.

[danielcamposramos]

Thank you, done: the fix is now its own pull request, #988, on current master, and this one is back to the single frame packing commit.
Until #988 is merged, --single-sei --no-aud --no-repeat-headers writes the same stale bytes with or without --frame-packing.