September 29th, 2015, GSoC 2015 results

FFmpeg participated to the latest edition of the Google Summer of Code Project. FFmpeg got a total of 8 assigned projects, and 7 of them were successful.

We want to thank Google, the participating students, and especially the mentors who joined this effort. We're looking forward to participating in the next GSoC edition!

Below you can find a brief description of the final outcome of each single project.

Basic servers for network protocols, mentee: Stephan Holljes, mentor: Nicolas George

Stephan Holljes's project for this session of Google Summer of Code was to implement basic HTTP server features for libavformat, to complement the already present HTTP client and RTMP and RTSP server code.

The first part of the project was to make the HTTP code capable of accepting a single client; it was completed partly during the qualification period and partly during the first week of the summer. Thanks to this work, it is now possible to make a simple HTTP stream using the following commands:

    ffmpeg -i /dev/video0 -listen 1 -f matroska \
    -c:v libx264 -preset fast -tune zerolatency http://:8080
    ffplay http://localhost:8080/

The next part of the project was to extend the code to be able to accept several clients, simultaneously or consecutively. Since libavformat did not have an API for that kind of task, it was necessary to design one. This part was mostly completed before the midterm and applied shortly afterwards. Since the ffmpeg command-line tool is not ready to serve several clients, the test ground for that new API is an example program serving hard-coded content.

The last and most ambitious part of the project was to update ffserver to make use of the new API. It would prove that the API is usable to implement real HTTP servers, and expose the points where more control was needed. By the end of the summer, a first working patch series was undergoing code review.

Browsing content on the server, mentee: Mariusz Szczepańczyk, mentor: Lukasz Marek

Mariusz finished an API prepared by the FFmpeg community and implemented Samba directory listing as qualification task.

During the program he extended the API with the possibility to remove and rename files on remote servers. He completed the implementation of these features for file, Samba, SFTP, and FTP protocols.

At the end of the program, Mariusz provided a sketch of an implementation for HTTP directory listening.

Directshow digital video capture, mentee: Mate Sebok, mentor: Roger Pack

Mate was working on directshow input from digital video sources. He got working input from ATSC input sources, with specifiable tuner.

The code has not been committed, but a patch of it was sent to the ffmpeg-devel mailing list for future use.

The mentor plans on cleaning it up and committing it, at least for the ATSC side of things. Mate and the mentor are still working trying to finally figure out how to get DVB working.

Implementing full support for 3GPP Timed Text Subtitles, mentee: Niklesh Lalwani, mentor: Philip Langdale

Niklesh's project was to expand our support for 3GPP Timed Text subtitles. This is the native subtitle format for mp4 containers, and is interesting because it's usually the only subtitle format supported by the stock playback applications on iOS and Android devices.

ffmpeg already had basic support for these subtitles which ignored all formatting information - it just provided basic plain-text support.

Niklesh did work to add support on both the encode and decode side for text formatting capabilities, such as font size/colour and effects like bold/italics, highlighting, etc.

The main challenge here is that Timed Text handles formatting in a very different way from most common subtitle formats. It uses a binary encoding (based on mp4 boxes, naturally) and stores information separately from the text itself. This requires additional work to track which parts of the text formatting applies to, and explicitly dealing with overlapping formatting (which other formats support but Timed Text does not) so it requires breaking the overlapping sections into separate non-overlapping ones with different formatting.

Finally, Niklesh had to be careful about not trusting any size information in the subtitles - and that's no joke: the now infamous Android stagefright bug was in code for parsing Timed Text subtitles.

All of Niklesh's work is committed and was released in ffmpeg 2.8.

libswscale refactoring, mentee: Pedro Arthur, mentors: Michael Niedermayer, Ramiro Polla

Pedro Arthur has modularized the vertical and horizontal scalers. To do this he designed and implemented a generic filter framework and moved the existing scaler code into it. These changes now allow easily adding removing, splitting or merging processing steps. The implementation was benchmarked and several alternatives were tried to avoid speed loss.

He also added gamma corrected scaling support. An example to use gamma corrected scaling would be:

    ffmpeg -i input -vf scale=512:384:gamma=1 output

Pedro has done impressive work considering the short time available, and he is a FFmpeg committer now. He continues to contribute to FFmpeg, and has fixed some bugs in libswscale after GSoC has ended.

AAC Encoder Improvements, mentee: Rostislav Pehlivanov, mentor: Claudio Freire

Rostislav Pehlivanov has implemented PNS, TNS, I/S coding and main prediction on the native AAC encoder. Of all those extensions, only TNS was left in a less-than-usable state, but the implementation has been pushed (disabled) anyway since it's a good basis for further improvements.

PNS replaces noisy bands with a single scalefactor representing the energy of that band, gaining in coding efficiency considerably, and the quality improvements on low bitrates are impressive for such a simple feature.

TNS still needs some polishing, but has the potential to reduce coding artifacts by applying noise shaping in the temporal domain (something that is a source of annoying, notable distortion on low-entropy bands).

Intensity Stereo coding (I/S) can double coding efficiency by exploiting strong correlation between stereo channels, most effective on pop-style tracks that employ panned mixing. The technique is not as effective on classic X-Y recordings though.

Finally, main prediction improves coding efficiency by exploiting correlation among successive frames. While the gains have not been huge at this point, Rostislav has remained active even after the GSoC, and is polishing both TNS and main prediction, as well as looking for further improvements to make.

In the process, the MIPS port of the encoder was broken a few times, something he's also working to fix.

Animated Portable Network Graphics (APNG), mentee: Donny Yang, mentor: Paul B Mahol

Donny Yang implemented basic keyframe only APNG encoder as the qualification task. Later he wrote interframe compression via various blend modes. The current implementation tries all blend modes and picks one which takes the smallest amount of memory.

Special care was taken to make sure that the decoder plays correctly all files found in the wild and that the encoder produces files that can be played in browsers that support APNG.

During his work he was tasked to fix any encountered bug in the decoder due to the fact that it doesn't match APNG specifications. Thanks to this work, a long standing bug in the PNG decoder has been fixed.

For latter work he plans to continue working on the encoder, making it possible to select which blend modes will be used in the encoding process. This could speed up encoding of APNG files.