Add VFR support to ratecontrol and slicetype, fix HRD timing.

[cubicibo]

Internal:

• Each frame now conveys a duration.
• Slicetype determines the cpbRemovalDelay and dpbOutputDelay for HRD conformance.
• Slicetype places keyframes on the min/max interval in $ClockTick$ units rather than an absolute frame counts.
• Ratecontrol weights each frame base on its $cpbDuration$ and $frameDuration$.
• VBV Lookahead considers the frame duration.
• CuTree uses the $frameDuration$ to give more weight to long-standing frames/blocks.

User interface:

• --pic-struct and --frame-dup now produce conformant bitstreams.
• Clean-up interaction between both option, and reject values that would produce orphaned fields.
• Add --psfile to specify arbitrary pulldown patterns ("VFR-in-CFR" container).

Limitation:

• VFR encode (or with structures / de-dup) is not well suited with how --frames (totalFrames) is defined, because this does not convey a duration.

[cubicibo]

@mcw-Lavanya Ready for review.

I don't know why only some debug VC smoke test fails. The same test pass on non-debug binaries.
Furthermore, it's only the tests that specify --bitrate. I cannot build or debug Windows binaries so I cannot look into it.

[mcw-Lavanya]

Hi @cubicibo, thanks for the detailed writeup on this — really nice work threading frame duration through the whole ratecontrol/slicetype/cuTree chain.

I dug into the Windows Debug + --bitrate smoke test failures a bit, and I have a suspicion about where it's coming from. Wanted to flag it humbly since I could be missing context on your side.

In ratecontrol.cpp, the constructor (RateControl::RateControl()) used to have:

cpp
m_fps = (double)m_param->fpsNum / m_param->fpsDenom;

In commit bc7e03c ("ratecontrol: support arbitrary timebases for VFR"), this line looks like it was replaced with:

cpp
m_timebase = (double)m_param->fpsDenom / m_param->fpsNum;

But m_fps (declared in ratecontrol.h, no default initializer) still seems to be read in several places later in the file, e.g.:

ratecontrol.cpp:389 – X265_MIN(m_param->keyframeMax, (int)(m_fps + 0.5))
ratecontrol.cpp:845 – vbvMaxBitrate * (m_fps / m_param->reconfigWindowSize)
ratecontrol.cpp:1128 – same keyframeMax clamp
ratecontrol.cpp:1888 – m_param->totalFrames <= 2 * m_fps
ratecontrol.cpp:1935/1937 – curBitrate/projectedBitrate AQ calc
ratecontrol.cpp:2332 – m_framesDone < (int)(m_fps + 0.5)
ratecontrol.cpp:3518/3525 – forwardMasking() scenecut windows

If I'm reading it right, m_fps is never reassigned anywhere else, so those reads are of an uninitialized member. That would explain the pattern you're seeing pretty well — MSVC's debug CRT poisons freshly allocated memory, so an uninitialized double there would come back with garbage in Debug but often "accidentally work" in Release. And since m_fps is only touched in the ABR/VBV/CBR paths, it'd only bite --bitrate runs, not CRF/CQP ones.

Could well be I'm missing an initializer somewhere else in the flow — if so, apologies for the noise! But if it checks out, restoring something like:

cpp
m_fps = (double)m_param->fpsNum / m_param->fpsDenom; // or 1.0 / m_timebase

right next to the new m_timebase line seems like it should clear it up.

[cubicibo]

Thank you, a bad oversight of mine. Unfortunately that does not solve it.

[mcw-Lavanya]

Thank you, a bad oversight of mine. Unfortunately that does not solve it.

0001-ratecontrol-initialize-m_durationDone.patch

Hi @cubicibo ,
kindly apply this patch on this PR and let us know if it is not affecting the PR behaviour and at the same time fixing the smoke test output issue

[mcw-Lavanya]

Hi @cubicibo ,
The build failures are due to some warnings present in code previously, that is fixed from our end and raised a PR, once that PR is merged I shall intimate you , you can commit empty changes to trigger the Ci pipeline again for the CI tests to pass.

Thank you

[mcw-Lavanya]

Hi @cubicibo,
Kindly rebase the PR with current master, a warning is being fixed in master commit.
Thank you

[mcw-Lavanya]

Hi @cubicibo ,
Can we get the test setup, CLI and benchmarking results of this PR to verify from our end?

Thank you

[cubicibo]

I have found one more conformance issue. However it is present in the master branch, it does not come from this PR. x265 bitstreams sometime breach Equation C-18 (§ C.4). This is not a problem for most decoders using the H.222 leak method... but it still needs fixing. I will try to address it in another PR first.

Regarding your question:

Test set-up & toolings

• Python HEVC HRD Verifier: implements all CPB equations of section H.265 C.x, and DPB output timing.
• MT8581-based STB

I have included a readme to the Python HRD verifier. You are free to make any use of it.

Command lines

1. --psfile: Specify pulldown information through file.

• psf.txt to encode 4 framerates section: 23.976, 29.97, 59.94 and 47.95 in 59.94 HEVC container.
• Test purpose: verify --keyint 60 is indeed at most 60 clock ticks apart, regardless of the section's framerate.

./x265 --y4m - -o ~/output.265 --vbv-maxrate 10000 --vbv-bufsize 8000 --fps 60000/1001 --psfile ~/psf.txt --hrd --aud --repeat-headers --keyint 60 --bitrate 6500

2. --prepulldown-fps prepulldown.patch (not commited yet) : This lets you specify the input framerate, prior to pulldown to --fps.

• Purpose: offer more versatile equivalent to x264 --pulldown argument.
• Useful for broadcast, where movies are 24p and must be delivered via 50p or 60p.

./x265 --y4m - -o ~/output.265 --vbv-maxrate 10000 --vbv-bufsize 8000 --fps 60000/1001 --prepulldown-fps 24000/1001 --hrd --aud --repeat-headers --keyint 60 --bitrate 6500

Evaluation

1. A STB plays the bitstream at the appropriate framerate after muxing to a Transport Stream thanks to picture-structure markings.
2. You can evaluate the DPB output timing, and CPB usage with the provided Python HRD verifier.

python3 client.py --dpb timing.txt --plot vbv.png --logfile hss.txt output.265

With command line 2) on an arbitrary video:
Example DPB timing output - As you can see, every picture is alternatively doubled and tripled to attain 59.94 fps
Example VBV plot

[mcw-Lavanya]

I have found one more conformance issue. However it is present in the master branch, it does not come from this PR. x265 bitstreams sometime breach Equation C-18 (§ C.4). This is not a problem for most decoders using the H.222 leak method... but it still needs fixing. I will try to address it in another PR first.

Regarding your question:

Test set-up & toolings

• Python HEVC HRD Verifier: implements all CPB equations of section H.265 C.x, and DPB output timing.
• MT8581-based STB

I have included a readme to the Python HRD verifier. You are free to make any use of it.

Command lines

1. --psfile: Specify pulldown information through file.

• psf.txt to encode 4 framerates section: 23.976, 29.97, 59.94 and 47.95 in 59.94 HEVC container.
• Test purpose: verify --keyint 60 is indeed at most 60 clock ticks apart, regardless of the section's framerate.

./x265 --y4m - -o ~/output.265 --vbv-maxrate 10000 --vbv-bufsize 8000 --fps 60000/1001 --psfile ~/psf.txt --hrd --aud --repeat-headers --keyint 60 --bitrate 6500

2. --prepulldown-fps prepulldown.patch (not commited yet) : This lets you specify the input framerate, prior to pulldown to --fps.

• Purpose: offer more versatile equivalent to x264 --pulldown argument.
• Useful for broadcast, where movies are 24p and must be delivered via 50p or 60p.

./x265 --y4m - -o ~/output.265 --vbv-maxrate 10000 --vbv-bufsize 8000 --fps 60000/1001 --prepulldown-fps 24000/1001 --hrd --aud --repeat-headers --keyint 60 --bitrate 6500

Evaluation

1. A STB plays the bitstream at the appropriate framerate after muxing to a Transport Stream thanks to picture-structure markings.
2. You can evaluate the DPB output timing, and CPB usage with the provided Python HRD verifier.

python3 client.py --dpb timing.txt --plot vbv.png --logfile hss.txt output.265

With command line 2) on an arbitrary video: Example DPB timing output - As you can see, every picture is alternatively doubled and tripled to attain 59.94 fps Example VBV plot

Thank you @cubicibo , we will verify from our end and get back to you

[mcw-Lavanya]

Hi @cubicibo ,
Kindly update the help page of x265 for using --psfile in cli which is added in cli.rst.
code LGTM

[mcw-Lavanya]

Hi @cubicibo ,
May I know why this line of code is repeated

[mcw-Lavanya]

Hi @cubicibo ,
Is it possible to create the prepulldown related commits in new PR so that it would be easier for understanding and reference,
psfile related commits we can push separately,
Let us know if it is possible, or we shall review the new code here
Thank you

[cubicibo]

Apologies, I will revert that commit. I had figured it was an easier entry-point to most users contrary to the --ps-file.