Improving AV1-ISOBMFF open source tools

[cconcolato]

This issue is here to collect the list of open-source tools that do not process AV1 streams but could if they found developers or had funding; or open source tools that process AV1 but have missing/buggy features and that could be improved/fixed if they found developers or had funding. The focus here is on AV1-ISOBMFF, or AVIF, implementations but we can also hear about AV1-implementations in general.

There is no commitment to fund them but the idea is to make it easier for whoever wants to fund such work to find the list of interesting features; and for developers who want to find interesting projects to work on.

If you know of any open source software that fits this category, please add it to this issue, indicating the link to the source code, a description of the bugs to be fixed/features to be implemented.

[farindk]

libheif

This HEIF / AVIF image encoder and decoder C-API library is widely used (GIMP, Krita, ImageMagick, libvips, Gnome, Darktable, digiKam, ...), has many language bindings (JavaScript, Go, C++, NodeJS, Python, Rust, Swift, C#), and is cross-platform. Still, there are a number of features missing that are requested from application side as they depend on them being implemented in libheif.

Features where interested developers can contribute:

• Faster color space conversions. Currently, these are mostly non-optimized, straight-forward implementations. Having SIMD optimized versions would significantly speed up the conversions. As an alternative to implementing new conversions, it is also possible to make use of libraries like libyuv or libswscale for this.
• Faster geometric transformations (rotation / mirroring) of the images.
• Build an automatic testing framework with a set of images that cover the possible range of inputs so that we can be sure not to introduce regressions in new releases.

Features where funding would be welcome. These are tasks that require deeper, architectural changes:

• While libheif can decode tiled images, there is no support yet for accessing individual tiles in the decoder, or for encoding tiled images. This is essential for large images.
• Support for modifying existing images without re-encoding. Also APIs for adding EXIF/XMP data to existing images. Adding additional images/thumbnails. Or cropping existing images.
• Better control over the encoding process: manually specify geometric transformations, encode overlay images.
• Support for image sequences (both decoding and encoding).

[rafaelcaricio]

GStreamer does have elements that already have some support for AV1. There is a list of bugs/feature requests in its issue tracker:

• GStreamer monorepo: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/?sort=created_date&state=opened&search=av1&first_page_size=20
• Rust plugins: https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/?sort=created_date&state=opened&search=av1&first_page_size=20