[FFmpeg-devel] [PATCHv2] avcodec/nvenc: Reconfigure resolution on-the-fly

Oliver Collyer ovcollyer at mac.com
Wed Mar 13 12:40:29 EET 2019


>>> Can you explain the actual intended use-case for this?
>>> 
>>> The current way to handle resolution changes (or any other stream change of similar magnitude, like a pixel format change) is to flush the existing encoder and then make a new one with the new parameters.  Even ignoring the trivial benefit that all encoders handle this with no additional code, it has the significant advantage that all of the stream metadata, including parameter sets, can be handled correctly.  The handling here does rather badly at this - stream metadata will be misleading, and if you take one of these streams and try to put it into some container with global headers you may well end up with a quite broken file.
>>> 
>> 
>> I’m not sure I follow; your logic seems contradictory here - clearly if you are trying to do this in a stream with global headers you’re going to run into trouble, but during writing to such a stream whether you 1) flush, delete and re-create, or 2) reconfigure the encoder is going to have the same effect iand won’t change anything since it’s not the encoder writing any such global stream headers it’s the muxer? Or did you mean something else?
> 
> Yeah, that wasn't very clear, sorry.  The main point I'm trying to make on the stream metadata is that if you flush and re-create then all of that /is/ available to the user.  They can of course then throw it all away to get back to the state you have if they wish, but there was at least the option to make use of it (e.g. if new global headers can be handled, maybe by starting a new fragment).
> 

Ok, this is probably where I'm a bit sketchy as to how things work, so please bear with me (or don't worry about this if you don't have time!!!) - so if you flush, and re-create the encoder it would write the new width/height to the output stream?

In my code I roughly do this:

- allocate output format context
- for each input stream I am interested in, I create an output stream
- find an encoder for each stream and allocate the context
- set up the encoder with the parameters I need (so width, height etc) and any other options
- open the encoder
- copy the codec parameters to the stream (avcodec_parameters_from_context)
- write header for the output format
- start encoding my data

I believe I originally took and adapted these steps from the ffmpeg sample code.

So now, if I want to change width/height, I just change it directly in the encoder context width/height, and with this patch it picks it up and since I am only interested in mpegts it works fine. Presumably, the reconfigure does not trigger any kind of change to the output stream meta data. 

Whereas, you are saying that when I want to change width/height if I instead:

- flush the encoder
- free encoder context
- create a new encoder context
- set up the encoder with the new parameters
- open the encoder

This would give me a different result? So does opening the encoder (as opposed to reconfiguring it) fill in a bunch of output stream meta data then? Wouldn't I still need to do the step above "copy the codec parameters to the stream" and then write a new header for the output format myself?

> I guess it's not clear to me that adding this new external interface for the special case is really a good trade-off.
> 
>> I’m using it in a server app where I want to quickly and efficiently change the video size/bitrate of a transport stream sent over long distance either when the remote user requests or in response to changing network conditions, with as little disruption to the viewing experience as possible.
> 
> Does the stop-start actually take longer?  Hardware encoders generally try to minimise the amount of state they need to initialise and carry (because TDM), so recreating it should not be a costly operation.  (Of course, I have no idea what proprietary blobs will do.)
> 

That I do not know, but when I was looking at adding support for this into my project I noticed that it already handles bitrate and DAR changes so it was trivial to add resolution changes too (with the exception of needing to specify a maximum width/height).

>> For the record when this patch is used in conjunction with encoding into an mpegts stream it plays fine in VLC/libVLC, which defects the video changes in the stream and recreates the vout/resizes the video accordingly.
> 
> Yes, I can see that it should be fine in MPEG-TS.
> 
>>> Given that, I think there should be some justification for why you might want to do this rather than following the existing approach.  Some mention in the API (avcodec.h) to explain what's going on might be a good idea, since that is currently assuming that parameters like width/height are fixed and must be set before opening the encoder.  Relatedly, if there isn't any support for retrieving new metadata then it should probably display a big warning if the user tries to make a stream like this with global headers, since the global headers provided to the user on startup won't actually work for the whole stream.
>>> 
>> 
>> I think the fact this functionality is only accessible programmatically means that the bar would be quite high, ie the user likely knows what they are doing, but I can certainly put a comment next to the width/height avcodecctx members along the lines of “In some limited scenarios such as when using nvenc the width/height can be changed during encoding and will be detected by the encoder causing it to reconfigure itself to the new picture sIze. Extreme care should be taken when doing this with a format that uses global headers as the global headers will no longer correspond to the adjusted picture size!”
> 
> Yeah, that's probably more sensible than what I suggested.
> 
>> Alternatively, maybe this is all a bit too obscure and it’s better left in my own customised ffmpeg branch? That would be quite ok, although the code does already handle dynamic bitrate and DAR changes so I figured to offer you the patch...
> 
> Maybe.  If your use-case ends up genuinely better somehow then I think it probably would be sensible to include (and maybe investigate doing it in other cases so that nvenc isn't a surprising special-case here), but adding complexity to the library interface to save a little bit of code on the user side in one case seems a bit excessive.
> 

To me it adds negligible complexity and the comment hopefully explains what to be aware of, and we already have precedent as mentioned above regarding dynamic bitrate and DAR for the nvenc encoder.

However, I'm totally fine with whatever you guys decide, or happy to amend the patch further if you think it would be useful.

Regards

Oliver

> Thanks,
> 
> - Mark
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel



More information about the ffmpeg-devel mailing list