Add VFR support to ratecontrol and slicetype, fix HRD timing.

[cubicibo]

This PR changes the way the Picture Timing SEI CPB and DPB offset are set, to fix #883.

Once SlicetypeDecide() decides the slice type of an input picture, it computes the HRD timing (AU cpb_removal_delay and dpb_output_delay) prior to handing off the picture to the encoder. This data is then used in frameencoder to write the Picture Timing delays.

Verification
Bitstream with a random pic-struct sequence. The CPB delay of each picture is extended given the (concurrently displayed) frame repetition. In 1 second, there is an average of 25 pictures as expected (equal mixture of tripling, doubling, and single frames).

This was before the change: at 50 fps, 50 frames were removed from the CPB, despite the doubling at the display. Please ignore the different bitrate, I used different VBV parameters.

Pending implementation:

• Each access unit now has a plannedCpbDuration. This value should be used in ratecontrol to determine how much bits a frame can receive. The decoder removes the frame twice slower from the CPB with frame doubling: each frame can have more bits assigned.

Newly forbidden

• Forbid the usage of frame doubling or tripling with temporal layers.
  • This requires dropping access units in the higher layers, which is not supported by the current architecture.
  • --frame-dup feature is thereby incompatible with sub temporal layers as well.
• Forbid the usage of picture structures leading to orphaned fields.

[mcw-Lavanya]

Hi @cubicibo , Could you please share the tool on which you performed the test for above PR and the benchmarking results if available.

Thank you

[mcw-Lavanya]

Hi @cubicibo , Could you please share the tool on which you performed the test for above PR and the benchmarking results if available.

Thank you

Hi @cubicibo ,
A gentle reminder for providing the tool used for the test and the bemchmarking results.
Thank you

[cubicibo]

Tool is MTS 4 Elementary Stream Analyzer.

However I am not done with this PR, I still need to modify the ratecontrol, I am also writing a combined CPB and DPB timing verifier as MTS4 is only checking the CPB timing, not the DPB one.

[mcw-Lavanya]

Tool is MTS 4 Elementary Stream Analyzer.

However I am not done with this PR, I still need to modify the ratecontrol, I am also writing a combined CPB and DPB timing verifier as MTS4 is only checking the CPB timing, not the DPB one.

Well, Thank you for acknowledging, let us know once it is completed.
Thank you

[cubicibo]

As you can see, the ratecontrol aspects are now functional. The buffer fills is not limited to a single clock tick. However a few things in ratecontrol.cpp are now brittle:

• Anything with the m_totalFrames: a difference to a frame number carries no notion of time: a frame can last for more than a clock tick. It's difficult to claim if we are "far" or "close" to the beginning or end of the encode in this situation.
• Other components using a constant FPS for tuning should probably switch to a moving average. Examples:
  • tuneQScaleForGrain uses bitstream timebase, probably needs a moving average.
  • forwardMasking uses the total average framerate.

Two more notes:

• The cpbInitialRemovalDelay of subsequent I-frames is wrong, this is also the case with trunk, I'm looking into it. Fixed, probably.
• Keyframe placement should be done according to the VUI timebase, not as an absolute frame count difference. In any broadcast environment, what matters is the GOP temporal duration. I suggest to modify slicetype keyframe placement logic to use the newly introduced m_displayPicCount of a Frame object. This change is transparent to all users encoding CFR content.

Early feedback welcome. I know this is invasive and some topics probably require a discussion. Happy to answer questions.

I see 3 candidates CLI options for the end user:

• --pulldown, akin to x264. <- will implement
• --psfile to specify a file with the pic-struct value of every frame (dynamic pulldown in a CFR container) <- will implement
• --tcfile-in for timecode v1/v2/v3 files, to do VFR encodes (typically with a timebase of 1 ms). <- open for discussion.