JPEG XS: Difference between revisions
→top: mark article as containing language that sounds subjective and/or like an advertisement; fix some citations and add archive links |
m unpiped links using script. grammar. WP:DUPLINK. add link. |
||
(10 intermediate revisions by 7 users not shown) | |||
Line 13: | Line 13: | ||
| released = {{start date and age|2019|5|1}} |
| released = {{start date and age|2019|5|1}} |
||
| owner = [[Joint Photographic Experts Group]] |
| owner = [[Joint Photographic Experts Group]] |
||
| genre = [[ |
| genre = [[Lossy]] and [[lossless]] [[image compression]] [[image format|format]] |
||
| container for = |
| container for = |
||
| contained by = |
| contained by = |
||
Line 23: | Line 23: | ||
}} |
}} |
||
'''JPEG XS''' (ISO/IEC 21122) is an interoperable, visually lossless, [[low-latency]] and lightweight [[Image compression|image]] and [[Video codec|video coding]] system used in professional applications.<ref name="Fraunhofer IIS">{{Cite news|url=https://www.iis.fraunhofer.de/en/ff/amm/prod/bildkompr/jpegxs.html|title=JPEG XS - the new low complexity codec standard for professional video production|date=2019-01-09|work=Fraunhofer Institute for Integrated Circuits IIS}}</ref><ref name="JPEG XS">{{Cite web|url=https://jpeg.org/jpegxs/ |title=Overview of JPEG XS |website=jpeg.org |access-date=2019-07-31}}</ref><ref>{{Cite news|url=https://geeko.lesoir.be/2018/04/19/le-jpeg-xs-futur-standard-de-compression-pour-la-realite-virtuelle/|title=Le JPEG XS, futur standard de compression pour la réalité virtuelle|date=2018-04-19|work=LeSoir.be}}</ref><ref>{{Cite news|url=https://geeko.lesoir.be/2019/01/10/le-nouveau-standard-jpeg-xs-est-belge/|title=Le nouveau standard JPEG XS est belge|date=2019-01-10|work=LeSoir.be}}</ref><ref>T. Richter, J. Keinert, S. Foessel, A. Descampe, G. Rouvroy and J. -B. Lorent, "JPEG-XS -- A High-Quality Mezzanine Image Codec for Video Over IP," in SMPTE Motion Imaging Journal, vol. 127, no. 9, pp. 39-49, Oct. 2018, doi: 10.5594/JMI.2018.2862098 [https://ieeexplore.ieee.org/document/8470198 link].</ref> Applications of the standard include streaming high |
'''JPEG XS''' (ISO/IEC 21122) is an interoperable, visually lossless, [[low-latency]] and lightweight [[Image compression|image]] and [[Video codec|video coding]] system used in professional applications.<ref name="Fraunhofer IIS">{{Cite news|url=https://www.iis.fraunhofer.de/en/ff/amm/prod/bildkompr/jpegxs.html|title=JPEG XS - the new low complexity codec standard for professional video production|date=2019-01-09|work=Fraunhofer Institute for Integrated Circuits IIS}}</ref><ref name="JPEG XS">{{Cite web|url=https://jpeg.org/jpegxs/ |title=Overview of JPEG XS |website=jpeg.org |access-date=2019-07-31}}</ref><ref>{{Cite news|url=https://geeko.lesoir.be/2018/04/19/le-jpeg-xs-futur-standard-de-compression-pour-la-realite-virtuelle/|title=Le JPEG XS, futur standard de compression pour la réalité virtuelle|date=2018-04-19|work=LeSoir.be}}</ref><ref>{{Cite news|url=https://geeko.lesoir.be/2019/01/10/le-nouveau-standard-jpeg-xs-est-belge/|title=Le nouveau standard JPEG XS est belge|date=2019-01-10|work=LeSoir.be}}</ref><ref>T. Richter, J. Keinert, S. Foessel, A. Descampe, G. Rouvroy and J. -B. Lorent, "JPEG-XS -- A High-Quality Mezzanine Image Codec for Video Over IP," in SMPTE Motion Imaging Journal, vol. 127, no. 9, pp. 39-49, Oct. 2018, doi: 10.5594/JMI.2018.2862098 [https://ieeexplore.ieee.org/document/8470198 link].</ref> Applications of the standard include streaming high-quality content for [[virtual reality]], drones, autonomous vehicles using cameras, gaming, and broadcasting (SMPTE [[SMPTE 2022|ST 2022]] and [[ST 2110]]).<ref name="JPEG XS"/><ref>{{Cite web|url=https://www.tinynews.be/jpeg-xs-intopix-belgique/|title=Les belges d'intoPIX révolutionnent la compression d'image et de vidéo avec la nouvelle norme JPEG XS|date=January 10, 2019}}</ref><ref>{{Citation|url= https://actu.epfl.ch/news/a-new-jpeg-format-for-virtual-reality-drones-and-s/ |title= A new JPEG format for virtual reality, drones and self-driving cars |first= Laure-Anne |last= Pessina |date= September 4, 2018 |via= actu.epfl.ch |archive-url=https://web.archive.org/web/20230605095051/https://actu.epfl.ch/news/a-new-jpeg-format-for-virtual-reality-drones-and-s/ |archive-date=2023-06-05 |url-status=live}}</ref><ref>{{Cite web|url= https://www.ibc.org/introduction-to-jpeg-xs/2452.article |title= Introduction to JPEG XS |first1= Joachim |last1= Keinert |first2= Jean-Baptiste |last2= Lorent |first3= Antonin |last3= Descampe |first4= Gaël |last4= Rouvroy |first5= Siegfried |last5= Fößel |website= IBC |archive-url=https://web.archive.org/web/20230402121749/https://www.ibc.org/introduction-to-jpeg-xs/2452.article |archive-date= 2023-04-02 |url-status= live}}</ref> It was the first ISO codec ever designed for this specific purpose. JPEG XS, built on core technology from both intoPIX and Fraunhofer IIS, is formally standardized as ISO/IEC 21122 by the [[Joint Photographic Experts Group]] with the first edition published in 2019. Although not official, the XS acronym was chosen to highlight the ''eXtra Small'' and ''eXtra Speed'' characteristics of the codec. Today, the JPEG committee is still actively working on further improvements to XS, with the second edition<ref name="ieee_jpeg_xs_2021">A. Descampe et al., "JPEG XS--A New Standard for Visually Lossless Low-Latency Lightweight Image Coding," in Proceedings of the IEEE, doi: 10.1109/JPROC.2021.3080916 [https://dial.uclouvain.be/pr/boreal/object/boreal:248275 PDF].</ref> scheduled for publication (beginning of 2022) and initial efforts being launched towards a third edition. |
||
==Features of JPEG XS== |
==Features of JPEG XS== |
||
Line 30: | Line 30: | ||
# '''[[Transparency (data compression)|Visually transparent compression]]''': XS compressed content is indistinguishable from the original uncompressed content (passing ISO/IEC 29170-2 tests) for compression ratios up to 3 bit per pixel (bpp). |
# '''[[Transparency (data compression)|Visually transparent compression]]''': XS compressed content is indistinguishable from the original uncompressed content (passing ISO/IEC 29170-2 tests) for compression ratios up to 3 bit per pixel (bpp). |
||
# '''[[ |
# '''[[Low latency]]''': The total end-to-end latency, introduced by the XS compression-decompression cycle, is minimal. Depending on the configuration, XS typically imposes only between 1 and 32 lines of additional end-to-end latency, when compared to the same system using uncompressed video. |
||
# '''Lightweight''': XS is designed to have low computational and memory complexity, allowing for efficient low-power and low-resource implementations on various platforms such as [[ |
# '''Lightweight''': XS is designed to have low computational and memory complexity, allowing for efficient low-power and low-resource implementations on various platforms such as [[CPU]], [[GPU]], [[FPGA]] and [[ASIC]]. |
||
Relying on these key features, JPEG XS is suitable to be used in any application where uncompressed content is now the norm, yet still allowing for significant savings in the required bandwidth usage, preserving quality and low latency. Among the targeted use cases are video transport over professional video links (like [[Serial digital interface|SDI]] and [[Ethernet]]/[[Internet Protocol|IP]]), real-time video storage, memory buffers, omnidirectional video capture and rendering, and image sensor compression (for example in cameras and in the automotive industry). Typical compression ratios go up to 10:1 but can also be higher depending on the nature of the image or the requirements of the targeted application.<ref>{{Cite web|url=http://ds.jpeg.org/whitepapers/jpeg-xs-whitepaper.pdf|title=JPEG White paper: JPEG XS, a new standard for visually lossless low-latency lightweight image coding system}}</ref> JPEG XS favors visually lossless quality in combination with low latency and low complexity, over crude compression performance. Hence, it is not a direct competitor to alternative image codecs like [[JPEG 2000]] and [[JPEG XL]] or video codecs like [[AV1]], [[ |
Relying on these key features, JPEG XS is suitable to be used in any application where uncompressed content is now the norm, yet still allowing for significant savings in the required bandwidth usage, preserving quality and low latency. Among the targeted use cases are video transport over professional video links (like [[Serial digital interface|SDI]] and [[Ethernet]]/[[Internet Protocol|IP]]), real-time video storage, memory buffers, omnidirectional video capture and rendering, and image sensor compression (for example in cameras and in the automotive industry). Typical compression ratios go up to 10:1 but can also be higher depending on the nature of the image or the requirements of the targeted application.<ref>{{Cite web|url=http://ds.jpeg.org/whitepapers/jpeg-xs-whitepaper.pdf|title=JPEG White paper: JPEG XS, a new standard for visually lossless low-latency lightweight image coding system}}</ref> JPEG XS favors visually lossless quality in combination with low latency and low complexity, over crude compression performance. Hence, it is not a direct competitor to alternative image codecs like [[JPEG 2000]] and [[JPEG XL]] or video codecs like [[AV1]], [[AVC/H.264]] and [[High Efficiency Video Coding|HEVC/H.265]]. |
||
Other important features are: |
Other important features are: |
||
Line 39: | Line 39: | ||
# '''Exact bitrate allocation''': JPEG XS allows to accurately set the targeted bitrate to perfectly match the available bandwidth (also referred to as [[constant bitrate]] or [[Constant bitrate|CBR]]). |
# '''Exact bitrate allocation''': JPEG XS allows to accurately set the targeted bitrate to perfectly match the available bandwidth (also referred to as [[constant bitrate]] or [[Constant bitrate|CBR]]). |
||
# '''Multi-generation robustness''': JPEG XS allows for at least 10 encoding-decoding cycles, without significant quality degradation.<ref>{{Cite book|last1=Richter|first1=Thomas|last2=Keinert|first2=Joachim|last3=Descampe|first3=Antonin|last4=Rouvroy|first4=Gael|last5=Willeme|first5=Alexandre|title=2017 IEEE International Symposium on Multimedia (ISM) |chapter=Multi-generation-robust Coding with JPEG XS |date=December 2017|chapter-url=https://ieeexplore.ieee.org/document/8241574|pages=6–13|doi=10.1109/ISM.2017.12|isbn=978-1-5386-2937-6|s2cid=32986247}}</ref> This feature allows for example transparently chaining of multiple devices that recompress the signal, without any significant quality degradation taking place. |
# '''Multi-generation robustness''': JPEG XS allows for at least 10 encoding-decoding cycles, without significant quality degradation.<ref>{{Cite book|last1=Richter|first1=Thomas|last2=Keinert|first2=Joachim|last3=Descampe|first3=Antonin|last4=Rouvroy|first4=Gael|last5=Willeme|first5=Alexandre|title=2017 IEEE International Symposium on Multimedia (ISM) |chapter=Multi-generation-robust Coding with JPEG XS |date=December 2017|chapter-url=https://ieeexplore.ieee.org/document/8241574|pages=6–13|doi=10.1109/ISM.2017.12|isbn=978-1-5386-2937-6|s2cid=32986247}}</ref> This feature allows for example transparently chaining of multiple devices that recompress the signal, without any significant quality degradation taking place. |
||
# '''Multi-platform interoperability''': The algorithms used in JPEG XS |
# '''Multi-platform interoperability''': The algorithms used in JPEG XS allow for efficient implementations on different platforms, like CPU, GPU, FPGA and ASIC. Each of these platform architectures is best exploited when a specific degree of parallelism is available in the implementation. For instance, a multi-core CPU implementation will leverage a coarse-grained parallelism, while GPU or FPGA will work better with a fine-grained parallelism. Moreover, the choice of parallelism used in the implementation at the encoder will not affect that of the decoder. This means that real-time encoding and decoding between platforms is possible, without sacrificing the low complexity, low latency or high-quality properties. |
||
# '''Support for mathematical [[Lossless compression|lossless coding]] (MLS)''': JPEG XS is also capable of coding images in a mathematically lossless way, |
# '''Support for mathematical [[Lossless compression|lossless coding]] (MLS)''': JPEG XS is also capable of coding images in a mathematically lossless way, to achieve perfect reconstruction at the decoder side (new profile supported by 2nd edition). |
||
# '''Support for [[ |
# '''Support for [[High Dynamic Range]] (HDR) content''': The current version of JPEG XS supports bit-depths of up to 16 bits per component, and it provides several parameterizable non-linear transforms (NLTs) to efficiently compress HDR content. |
||
# '''Support for RAW Bayer/CFA compression''': JPEG XS also |
# '''Support for RAW Bayer/CFA compression''': JPEG XS has also the capability to compress [[Color filter array|Color Filter Array]] (CFA) content, such as RAW [[Bayer filter|Bayer]] content produced by digital cameras. A special color transform, called Star-Tetrix, allows for efficient and direct compression of the original RAW sample values, without the need for converting the Bayer samples to RGB samples first.<ref>T. Richter, S. Fößel, A. Descampe and G. Rouvroy, "Bayer CFA Pattern Compression With JPEG XS," in IEEE Transactions on Image Processing, vol. 30, pp. 6557-6569, 2021, doi: 10.1109/TIP.2021.3095421 [https://ieeexplore.ieee.org/document/9488288 link].</ref> |
||
#'''Accurate flow control''': A JPEG XS encoder continuously monitors the amount of bits |
#'''Accurate flow control''': A JPEG XS encoder continuously monitors the amount of bits sent out, and adjusts its rate allocation process to neither overflow nor underflow a normatively<ref>{{Cite web|last=International Organization for Standardization|title=ISO/IEC 21122-2:2019|url=https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/07/45/74536.html|access-date=2021-09-07|website=ISO|language=en}}</ref> defined decoder input buffer. |
||
==Application domains== |
==Application domains== |
||
This section lists them main application domains where JPEG XS is actively used. New and other application domains are subject to be added in the future, |
This section lists them main application domains where JPEG XS is actively used. New and other application domains are subject to be added in the future, for example, framebuffer compression or AR/VR applications. |
||
===Transport over video links and IP networks === |
===Transport over video links and IP networks === |
||
Video bandwidth requirements are growing continuously, as video resolutions, frame rates, bit depths and the amount of video streams are constantly increasing. Likewise, the capacities of video links and communication channels are also growing, yet at a slower pace than what is needed to address the huge video bandwidth growth. In addition, the investments to upgrade the capacity of links and channels are significant and need to be amortized over several years. |
Video bandwidth requirements are growing continuously, as video resolutions, frame rates, bit depths, and the amount of video streams are constantly increasing. Likewise, the capacities of video links and communication channels are also growing, yet at a slower pace than what is needed to address the huge video bandwidth growth. In addition, the investments to upgrade the capacity of links and channels are significant and need to be amortized over several years. |
||
Moreover, both the broadcast and pro-AV markets are shifting towards AV-over-IP |
Moreover, both the broadcast and pro-AV markets are shifting towards AV-over-IP-based infrastructure, with a preference going to 1 Gigabit Ethernet links for remote production or 10G Ethernet networks for in-house facilities. Both 1G, 2.5G, and 10G Ethernet are cheap and ubiquitous, while 25G or better links are usually not yet affordable. Given the available bandwidth and infrastructure cost, relying on uncompressed video is therefore no longer an option, as 4K, 8K, increased bit depths (for HDR), and higher framerates need to be supported. |
||
JPEG XS |
JPEG XS is a light-weight compression that visually preserves the quality compared to an uncompressed stream, at a low cost, targeted at compression ratios of up to 10:1. With XS, it is for example possible to repurpose existing SDI cables to transport 4K60 over a single 3G-SDI (at 4:1), and even over a single HD-SDI (at 8:1). Similar scenarios can be used to transport 8K60 content over various SDI cable types (e.g. 6G-SDI and 12G-SDI). Alternatively, XS enables transporting 4K60 content over 1G Ethernet and 8K60 over 5G or 10G Ethernet, which would be impossible without compression. The following table shows some expected compression ranges for some typical use cases. |
||
{| class="wikitable" border="1" style="border-collapse: collapse;" |
{| class="wikitable" border="1" style="border-collapse: collapse;" |
||
Line 78: | Line 78: | ||
===Real-time video storage and playout=== |
===Real-time video storage and playout=== |
||
Related to the transport of video streams is the storage and retrieval of high |
Related to the transport of video streams is the storage and retrieval of high-resolution streams where bandwidth limitations similarly apply. For instance, video cameras use internal storage like SSD drives or SD cards to hold large streams of images, yet the maximum data rates of such storage devices are limited and well below the uncompressed video throughput. |
||
===Sensor compression=== |
===Sensor compression=== |
||
As stated, JPEG XS |
As stated, JPEG XS has built-in support for the direct compression of RAW [[Bayer filter|Bayer]]/CFA images using the Star-Tetrix Color Transform. This transform takes a RAW Bayer pattern image and [[Decorrelation|decorrelates]] the samples into a 4-component image with each component having only a quarter of the resolution.<ref>{{Cite journal|last1=Richter|first1=Thomas|last2=Fößel|first2=Siegfried|last3=Descampe|first3=Antonin|last4=Rouvroy|first4=Gaël|date=2021|title=Bayer CFA Pattern Compression With JPEG XS|journal=IEEE Transactions on Image Processing|volume=30|pages=6557–6569|doi=10.1109/TIP.2021.3095421|pmid=34270422|bibcode=2021ITIP...30.6557R|s2cid=236001190|issn=1941-0042|doi-access=free}}</ref> This means that the total amount of samples to further process and compress remains the same, yet the values are decorrelated similarly to a classical Multiple Component Transform. |
||
Avoiding such conversion prevents loss |
Avoiding such conversion prevents information loss and allows this processing step to be done outside of the camera. This is advantageous because it allows to defer [[demosaicing]] the Bayer content from the moment of capturing to the production phase, where choices regarding artistic intent and various settings can be better made. Recall that the demosaicing process is irreversible and requires certain choices, like the choice of interpolation algorithm or the level of noise reduction, to be made upfront. Moreover, the demosaicing process can be power-hungry and will also introduce extra latency and complexity. The ability to push this step out of the camera is possible with JPEG XS and allows to use more advanced algorithms resulting in better quality in the end. |
||
==Standards== |
==Standards== |
||
Line 94: | Line 94: | ||
{| class="wikitable" border="1" style="border-collapse: collapse;" |
{| class="wikitable" border="1" style="border-collapse: collapse;" |
||
|- |
|- |
||
! scope="col" | Part !! scope="col" | |
! scope="col" | Part !! scope="col" | 1st edition !! scope="col" | 2nd edition (in force) |
||
!3rd edition!! scope="col" | Title |
|||
|- |
|- |
||
| 1 || [https://www.iso.org/standard/74535.html 2019] || [https://www.iso.org/standard/81551.html 2022] || Core coding system |
| 1 || [https://www.iso.org/standard/74535.html 2019] || [https://www.iso.org/standard/81551.html 2022] |
||
|scheduled 2024|| Core coding system |
|||
|- |
|- |
||
| 2 || [https://www.iso.org/standard/74536.html 2019] || [https://www.iso.org/standard/81552.html 2022] || Profiles and buffer models |
| 2 || [https://www.iso.org/standard/74536.html 2019] || [https://www.iso.org/standard/81552.html 2022] |
||
|scheduled 2024|| Profiles and buffer models |
|||
|- |
|- |
||
| || || [https://www.iso.org/standard/84584.html 2022/Amd1] || Profile and sublevel for 4:2:0 content |
| 2|| -|| [https://www.iso.org/standard/84584.html 2022/Amd1] |
||
| -|| Profile and sublevel for 4:2:0 content |
|||
|- |
|- |
||
| 3 || [https://www.iso.org/standard/74537.html 2019] || [https://www.iso.org/standard/81553.html 2022] || Transport and container formats |
| 3 || [https://www.iso.org/standard/74537.html 2019] || [https://www.iso.org/standard/81553.html 2022] |
||
|scheduled 2024|| Transport and container formats |
|||
|- |
|- |
||
| 4 || [https://www.iso.org/standard/74538.html 2020] || [https://www.iso.org/standard/82568.html 2022] || Conformance testing |
| 4 || [https://www.iso.org/standard/74538.html 2020] || [https://www.iso.org/standard/82568.html 2022] |
||
|in development|| Conformance testing |
|||
|- |
|- |
||
| 5 || [https://www.iso.org/standard/74539.html 2020] || [https://www.iso.org/standard/82569.html 2022] || Reference software |
| 5 || [https://www.iso.org/standard/74539.html 2020] || [https://www.iso.org/standard/82569.html 2022] |
||
|in development|| Reference software |
|||
|} |
|} |
||
'''Part 1''', formally designated as ISO/IEC 21122-1, describes the core coding system of JPEG XS. This standard defines the syntax and, similarly to other JPEG and MPEG image codecs, the decompression process to reconstruct a continuous-tone digital image from its encoded codestream. Part 1 does provide some guidelines of the inverse process that compresses a digital image into a compressed codestream, or more simply called the encoding process, but leaves implementation |
'''Part 1''', formally designated as ISO/IEC 21122-1, describes the core coding system of JPEG XS. This standard defines the syntax and, similarly to other JPEG and MPEG image codecs, the decompression process to reconstruct a continuous-tone digital image from its encoded codestream. Part 1 does provide some guidelines of the inverse process that compresses a digital image into a compressed codestream, or more simply called the encoding process, but leaves implementation-specific optimizations and choices to the implementers. |
||
'''Part 2''' (ISO/IEC 21122-2) builds on top of Part 1 to segregate different applications and uses of JPEG XS into reduced coding tool subsets with tighter constraints. The definition of profiles, levels and sublevels allows reducing the complexity of implementations in particular application use cases, while also safeguarding interoperability. Recall that lower complexity typically means less power consumption, lower production costs, easier constraints, etc. Profiles represent interoperability subsets of the codestream syntax specified in Part 1. In addition, levels and sublevels provide limits to the maximum throughput in respectively the encoded (codestream) and the decoded (spatial and pixels) image domains. Part 2 furthermore also specifies a buffer model, consisting of a decoder model and a transmission channel model, |
'''Part 2''' (ISO/IEC 21122-2) builds on top of Part 1 to segregate different applications and uses of JPEG XS into reduced coding tool subsets with tighter constraints. The definition of profiles, levels, and sublevels allows for reducing the complexity of implementations in particular application use cases, while also safeguarding interoperability. Recall that lower complexity typically means less power consumption, lower production costs, easier constraints, etc. Profiles represent interoperability subsets of the codestream syntax specified in Part 1. In addition, levels and sublevels provide limits to the maximum throughput in respectively the encoded (codestream) and the decoded (spatial and pixels) image domains. Part 2 furthermore also specifies a buffer model, consisting of a decoder model and a transmission channel model, to enable guaranteeing low latency requirements to a fraction of the frame size. |
||
'''Part 3''' (ISO/IEC 21122-3) specifies transport and container formats for JPEG XS codestreams. It defines carriage of important metadata, like color spaces, mastering display metadata (MDM), and EXIF, |
'''Part 3''' (ISO/IEC 21122-3) specifies transport and container formats for JPEG XS codestreams. It defines the carriage of important metadata, like color spaces, mastering display metadata (MDM), and EXIF, to facilitate transport, editing, and presentation. Furthermore, this part defines the XS-specific ISOBMFF boxes, an Internet Media Type registration, and additional syntax to allow embedding XS in formats like MP4, MPEG-2 TS, or the HEIF image file format. |
||
'''Part 4''' (ISO/IEC 21122-4) is a supporting standard of JPEG XS that provides conformance testing and buffer model verification. This standard is crucial to implementers of XS and appliance conformance testing. |
'''Part 4''' (ISO/IEC 21122-4) is a supporting standard of JPEG XS that provides conformance testing and buffer model verification. This standard is crucial to implementers of XS and appliance conformance testing. |
||
Finally, '''Part 5''' (ISO/IEC 21122-5) represents a reference software implementation (written in ISO C11) of the JPEG XS Part 1 decoder, conforming to the Part 2 profiles, levels and sublevels, as well as an exemplary encoder implementation. |
|||
A second edition of all five parts is in the making and will be published at latest in the beginning of 2022. It provides additional coding tools, profiles and levels, and new reference software to add support for efficient compression of 4:2:0 content, RAW Bayer/CFA content, and mathematically lossless compression. |
A second edition of all five parts is in the making and will be published at the latest in the beginning of 2022. It provides additional coding tools, profiles and levels, and new reference software to add support for efficient compression of 4:2:0 content, RAW Bayer/CFA content, and mathematically lossless compression. |
||
===RFC9134 - RTP Payload Format for ISO/IEC 21122 (JPEG XS)=== |
===RFC9134 - RTP Payload Format for ISO/IEC 21122 (JPEG XS)=== |
||
Line 125: | Line 132: | ||
RFC 9134<ref>{{cite IETF |RFC=9134 |title=RTP Payload Format for ISO/IEC 21122 (JPEG XS)}}</ref> describes a payload format for the [[Real-time Transport Protocol|Real-Time Transport Protocol]] (RTP, RFC 3550<ref>{{cite IETF |RFC=3550 |title=RTP: A Transport Protocol for Real-Time Applications}}</ref>) to carry JPEG XS encoded video. In addition, the recommendation also registers the official Media Type Registration for JPEG XS video as {{code|video/jxsv}}, along with its mapping of all parameters into the [[Session Description Protocol]] (SDP). |
RFC 9134<ref>{{cite IETF |RFC=9134 |title=RTP Payload Format for ISO/IEC 21122 (JPEG XS)}}</ref> describes a payload format for the [[Real-time Transport Protocol|Real-Time Transport Protocol]] (RTP, RFC 3550<ref>{{cite IETF |RFC=3550 |title=RTP: A Transport Protocol for Real-Time Applications}}</ref>) to carry JPEG XS encoded video. In addition, the recommendation also registers the official Media Type Registration for JPEG XS video as {{code|video/jxsv}}, along with its mapping of all parameters into the [[Session Description Protocol]] (SDP). |
||
The RTP Payload Format for JPEG XS in turn enables using JPEG XS in SMPTE [[ |
The RTP Payload Format for JPEG XS in turn enables using JPEG XS in SMPTE [[ST 2110]] environments using SMPTE ST 2110-22 for CBR compressed video transport. |
||
===MPEG-TS for JPEG XS=== |
===MPEG-TS for JPEG XS=== |
||
ISO/IEC 13818-1:2022, known as MPEG-TS 8th edition, specifies carriage support for JPEG XS in MPEG Transport Streams.<ref>{{cite web | url=https://www.iso.org/standard/83239.html | title=Iso/Iec 13818-1:2022 }}</ref> See also [[MPEG-2]]. Note that AMD1 (Carriage of LCEVC and other improvements) of ISO/IEC 13818-1:2022 contains some additional corrections, improvements and clarifications regarding embedding JPEG XS in MPEG-TS.<ref>{{cite web | url=https://www.iso.org/standard/83253.html | title=ISO/IEC 13818-1:2022/Amd 1:2023 }}</ref> |
ISO/IEC 13818-1:2022, known as MPEG-TS 8th edition, specifies carriage support for JPEG XS in MPEG Transport Streams.<ref>{{cite web | url=https://www.iso.org/standard/83239.html | title=Iso/Iec 13818-1:2022 }}</ref> See also [[MPEG-2]]. Note that AMD1 (Carriage of LCEVC and other improvements) of ISO/IEC 13818-1:2022 contains some additional corrections, improvements, and clarifications regarding embedding JPEG XS in MPEG-TS.<ref>{{cite web | url=https://www.iso.org/standard/83253.html | title=ISO/IEC 13818-1:2022/Amd 1:2023 }}</ref> |
||
===VSF TR-07 and TR-08=== |
===VSF TR-07 and TR-08=== |
||
Line 139: | Line 146: | ||
===JPEG XS in IPMX=== |
===JPEG XS in IPMX=== |
||
Internet Protocol Media Experience (IPMX) is a proposed set of open standards and specifications to enable the carriage of compressed and uncompressed video, audio and data over IP networks for the pro AV market. JPEG XS is supported under IPMX via VSF TR-10-8<ref>[https://vsf.tv/download/technical_recommendations/VSF_TR-10-8_2023-04-14.pdf TR-10-8:2023-Internet Protocol Media Experience (IPMX): NMOS Requirements - DRAFT]</ref> and TR-10-11.<ref>[https://vsf.tv/download/technical_recommendations/VSF_TR-10-11_2023-04-14.pdf TR-10-11:2023-Internet Protocol Media Experience (IPMX): Constant Bit-Rate Compressed Video - DRAFT]</ref> |
Internet Protocol Media Experience (IPMX) is a proposed set of open standards and specifications to enable the carriage of compressed and uncompressed video, audio, and data over IP networks for the pro AV market. JPEG XS is supported under IPMX via VSF TR-10-8<ref>[https://vsf.tv/download/technical_recommendations/VSF_TR-10-8_2023-04-14.pdf TR-10-8:2023-Internet Protocol Media Experience (IPMX): NMOS Requirements - DRAFT]</ref> and TR-10-11.<ref>[https://vsf.tv/download/technical_recommendations/VSF_TR-10-11_2023-04-14.pdf TR-10-11:2023-Internet Protocol Media Experience (IPMX): Constant Bit-Rate Compressed Video - DRAFT]</ref> |
||
==History== |
==History== |
||
The JPEG committee started the standardization activity in 2016 with an open call for a high-performance, low |
The JPEG committee started the standardization activity in 2016 with an open call for a high-performance, low-complexity image coding standard. The best-performing candidates formed the basis for the new standard. First implementations were demonstrated in April 2018 at the [[NAB Show]] and later that year at the [[International Broadcasting Convention]].<ref>{{Cite web|url=https://uclouvain.be/fr/sciencetoday/actualites/jpeg-iso-hd-compression-photographie-numerique-jpeg-xs-resolution-pixels.html|title=JPEG XS, le nouveau standard de compression made in UCLouvain|website=UCLouvain}}</ref> XS was also presented at [[Consumer Electronics Show|CES]] in 2019. |
||
==Technical overview== |
==Technical overview== |
||
Line 149: | Line 156: | ||
===Core coding=== |
===Core coding=== |
||
The JPEG XS standard is a classical wavelet-based still-image codec without any frame buffer. While the standard<ref>{{Cite web|last=International Organization for Standardization|title=ISO/IEC 21122-1:2019|url=https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/07/45/74535.html|access-date=2021-09-07|website=ISO|language=en}}</ref> defines JPEG XS on |
The JPEG XS standard is a classical wavelet-based still-image codec without any frame buffer. While the standard<ref>{{Cite web|last=International Organization for Standardization|title=ISO/IEC 21122-1:2019|url=https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/07/45/74535.html|access-date=2021-09-07|website=ISO|language=en}}</ref> defines JPEG XS based on a hypothetical reference coder, JPEG XS is easier to explain through the steps a typical encoder performs:<ref name=":0">{{Cite journal|last1=Descampe|first1=Antonin|last2=Richter|first2=Thomas|last3=Ebrahimi|first3=Touradj|last4=Foessel|first4=Siegfried|last5=Keinert|first5=Joachim|last6=Bruylants|first6=Tim|last7=Pellegrin|first7=Pascal|last8=Buysschaert|first8=Charles|last9=Rouvroy|first9=Gaël|date=September 2021|title=JPEG XS—A New Standard for Visually Lossless Low-Latency Lightweight Image Coding|url=https://ieeexplore.ieee.org/document/9443088|journal=Proceedings of the IEEE|volume=109|issue=9|pages=1559–1577|doi=10.1109/JPROC.2021.3080916|s2cid=236727596|issn=1558-2256}}</ref> |
||
'''Component up-scaling and optional component decorrelation:''' In the first step, the DC gain of the input data is removed and it is upscaled to a bit-precision of 20 bits. Optionally, a multi-component generation, identical to the JPEG 2000 RCT, is applied. This transformation is a lossless approximation of an RGB to YUV conversion, generating one luma and two chroma channels. |
'''Component up-scaling and optional component decorrelation:''' In the first step, the DC gain of the input data is removed and it is upscaled to a bit-precision of 20 bits. Optionally, a multi-component generation, identical to the JPEG 2000 RCT, is applied. This transformation is a lossless approximation of an RGB to YUV conversion, generating one luma and two chroma channels. |
||
'''Wavelet transformation:''' Input data is spacially decorrelated by a 5/3 [[Daubechies wavelet]] filter. While a five-stage transformation is performed in horizontal direction, only 0 to 2 transformations are run in vertical direction. The reason for this asymmetrical filter is to minimize latency. |
'''Wavelet transformation:''' Input data is spacially decorrelated by a 5/3 [[Daubechies wavelet]] filter. While a five-stage transformation is performed in the horizontal direction, only 0 to 2 transformations are run in the vertical direction. The reason for this asymmetrical filter is to minimize latency. |
||
'''Prequantization''': The output of the wavelet filter is converted to a sign-magnitude representation and pre-quantized by a |
'''Prequantization''': The output of the wavelet filter is converted to a sign-magnitude representation and pre-quantized by a dead zone [[quantizer]] to 16-bit precision. |
||
'''Rate control and quantization''': The encoder determines by a non-normative process<ref name=":0" /> the rate of each possible quantization setting and then quantizes data by either a |
'''Rate control and quantization''': The encoder determines by a non-normative process<ref name=":0" /> the rate of each possible quantization setting and then quantizes data by either a dead zone quantizer or a data-dependent uniform quantizer. |
||
'''Entropy coding:''' JPEG XS uses minimalistic [[Entropy encoding]] for the quantized data which proceeds in up to four passes over horizontal lines of quantized wavelet coefficients. The steps are: |
'''Entropy coding:''' JPEG XS uses minimalistic [[Entropy encoding]] for the quantized data which proceeds in up to four passes over horizontal lines of quantized wavelet coefficients. The steps are: |
||
* '''Significance coding''': In the (optional) first pass, the significance of 32 consecutive wavelet coefficients is coded by a single bit. |
* '''Significance coding''': In the (optional) first pass, the significance of 32 consecutive wavelet coefficients is coded by a single bit. |
||
* '''Bitplane count coding''': In the second pass, the number of non-zero bitplanes of groups of four coefficients each, the so-called "bitplane count", is entropy coded through a [[Golomb coding|Golomb]] type code. This step may optionally use the bitplane counts of the preceding line as source for prediction ([[Differential pulse-code modulation]]) and then |
* '''Bitplane count coding''': In the second pass, the number of non-zero bitplanes of groups of four coefficients each, the so-called "bitplane count", is entropy coded through a [[Golomb coding|Golomb]] type code. This step may optionally use the bitplane counts of the preceding line as the source for prediction ([[Differential pulse-code modulation]]) and then encode only the prediction difference. |
||
* '''Data coding''':The third pass inserts the raw bitplane values into the codestream without further coding. |
* '''Data coding''':The third pass inserts the raw bitplane values into the codestream without further coding. |
||
* '''Sign coding''': In the last optional coding pass, the sign bits of all non-zero coefficients |
* '''Sign coding''': In the last optional coding pass, the sign bits of all non-zero coefficients are inserted into the codestream. If this coding pass is not present, sign bits are included in the data coding pass for all coefficients. |
||
'''Codestream packing''': All entropy |
'''Codestream packing''': All entropy-coded data are packed into a linear stream of bits (grouped in byte multiples) along with all of the required image metadata. This sequence of bytes is called the codestream and its high-level syntax is based on the typical JPEG markers and marker segments syntax.<ref>{{Cite web|url=https://github.com/thorfdbg/codestream-parser|title=codestream-parser|date=June 16, 2022|via=GitHub}}</ref> |
||
===Profiles, levels and sublevels=== |
===Profiles, levels and sublevels=== |
||
JPEG XS defines profiles (in ISO/IEC 21122-2) that define subsets of coding tools |
JPEG XS defines profiles (in ISO/IEC 21122-2) that define subsets of coding tools that conforming decoders shall support, by limiting the permitted parameter values and allowed markers. The following table represents an overview of all the profiles along with their most important properties. Please refer to the standard for a complete specification of each profile. |
||
{| |
{|class="wikitable" border="1" style="border-collapse: collapse;" |
||
|- |
|||
⚫ | ! scope="col"|Profile !! scope="col"|P<sub>pih</sub> !! scope="col"|B[i] !! scope="col"|N<sub>bpp,max</sub> !! scope="col"|B<sub>w</sub> !! scope="col"|B<sub>r</sub> !! scope="col"|F<sub>q</sub> !! scope="col"|Q<sub>pih</sub> !! scope="col"|Horizontal DWT !! scope="col"|Vertical DWT !! scope="col"|Chroma sampling formats !! scope="col"|C<sub>pih</sub> !! scope="col"|Edition |
||
|- |
|||
|Light 422.10||{{code|0x1500}}||8, 10||20||20||4||8||0||1 to 5||0, 1||4:0:0, 4:2:2||0||1 |
|||
|- |
|||
⚫ | |||
|- |
|- |
||
|Light-Subline 422.10||{{code|0x2500}}||8, 10||20||20||4||8||0, 1||1 to 5||0||4:0:0, 4:2:2||0||1 |
|||
⚫ | |||
|- |
|- |
||
| |
|Main 420.12||{{code|0x3240}}||8, 10, 12||18||20||4||8||0, 1||1 to 5||1||4:2:0||0||1 |
||
|- |
|- |
||
| |
|Main 422.10||{{code|0x3540}}||8, 10||20||20||8||4||0, 1||1 to 5||0, 1||4:0:0, 4:2:2||0||1 |
||
|- |
|- |
||
| |
|Main 444.12||{{code|0x3A40}}||8, 10, 12||36||20||4||8||0, 1||1 to 5||0, 1||4:0:0, 4:2:2, 4:4:4||0, 1||1 |
||
|- |
|- |
||
| |
|Main 4444.12||{{code|0x3E40}}||8, 10, 12||48||20||4||8||0, 1||1 to 5||0, 1||4:0:0, 4:2:2, 4:4:4, 4:2:2:4, 4:4:4:4||0, 1||1 |
||
|- |
|- |
||
| |
|High 420.12||{{code|0x4240}}||8, 10, 12||18||20||4||8||0, 1||1 to 5||1, 2||4:2:0||0||2 |
||
|- |
|- |
||
| |
|High 444.12||{{code|0x4A40}}||8, 10, 12||36||20||4||8||0, 1||1 to 5||0, 1, 2||4:0:0, 4:2:2, 4:4:4||0, 1||1 |
||
|- |
|- |
||
| |
|High 4444.12||{{code|0x4E40}}||8, 10, 12||48||20||4||8||0, 1||1 to 5||0, 1, 2||4:0:0, 4:2:2, 4:4:4, 4:2:2:4, 4:4:4:4||0, 1||1 |
||
|- |
|- |
||
| |
|CHigh 444.12||{{code|0x4A44}}||8, 10, 12||36||20||4||8||0, 1||1 to 5||0, 1, 2||4:0:0, 4:2:2, 4:4:4||0, 1||3 |
||
|- |
|- |
||
|TDC 444.12 |
|||
| High 420.12 || 8, 10, 12 || 20 || 8 || 0, 1 || 1 to 5 || 1, 2 || 4:2:0 || 0 || {{code|0x4240}} |
|||
|{{code|0x4A45}} |
|||
|8, 10, 12 |
|||
|36 |
|||
|20 |
|||
|4 |
|||
|8 |
|||
|0, 1 |
|||
|colspan="2"|(3, 0) and (4, 0) if not 4:2:0, |
|||
(4, 1), (5, 1), (5, 2) otherwise |
|||
|4:0:0, |
|||
4:2:0, |
|||
4:2:2, |
|||
4:4:4 |
|||
|0, 1 |
|||
|3 |
|||
|- |
|- |
||
|TDC MLS 444.12 |
|||
⚫ | |||
|{{code|0x6A45}} |
|||
|8, 10, 12 |
|||
|36 |
|||
|B[i] |
|||
|4 |
|||
|0 |
|||
|0, 1 |
|||
|colspan="2"|(3, 0) and (4, 0) if not 4:2:0, |
|||
(4, 1), (5, 1), (5, 2) otherwise |
|||
|4:0:0, |
|||
4:2:0, |
|||
4:2:2, |
|||
4:4:4 |
|||
|0, 1 |
|||
|3 |
|||
|- |
|- |
||
|MLS.12||{{code|0x6EC0}}||8, 10, 12 |
|||
| High 4444.12 || 8, 10, 12 || 20 || 8 || 0, 1 || 1 to 5 || 0, 1, 2 || 4:0:0, 4:2:2, 4:4:4, 4:2:2:4, 4:4:4:4 || 0, 1 || {{code|0x4E40}} |
|||
|48||B[i]||4||0||0, 1||1 to 5||0, 1, 2||4:0:0, |
|||
4:2:0, |
|||
4:2:2, |
|||
4:4:4, |
|||
4:2:2:4, |
|||
4:4:4:4 |
|||
|0, 1||2 |
|||
|- |
|- |
||
|MLS.16||{{code|0x6ED0}}||8, 10, 12, 14, 16 |
|||
| MLS.12 || 8, 10, 12 || same as input || 0 || 0, 1 || 1 to 5 || 0, 1, 2 || 4:0:0, 4:2:0, 4:2:2, 4:4:4, 4:2:2:4, 4:4:4:4 || 0, 1 || {{code|0x6EC0}} |
|||
|64||B[i]||5||0||0, 1||1 to 5||0, 1, 2||4:0:0, |
|||
4:2:0, |
|||
4:2:2, |
|||
4:4:4, |
|||
4:2:2:4, |
|||
4:4:4:4 |
|||
|0, 1||3 |
|||
|- |
|- |
||
| |
|LightBayer||{{code|0x9300}}||10, 12, 14, 16||64||18, 20||4||6, 8||0, 1||1 to 5||0||Bayer||3||2 |
||
|- |
|- |
||
| |
|MainBayer||{{code|0xB340}}||10, 12, 14, 16||64||18, 20||4||6, 8||0, 1||1 to 5||0, 1||Bayer||3||2 |
||
|- |
|- |
||
| |
|HighBayer||{{code|0xC340}}||10, 12, 14, 16||64||18, 20||4||6, 8||0, 1||1 to 5||0, 1, 2||Bayer||3||2 |
||
|} |
|} |
||
In addition, JPEG XS defines levels to represent a lower bound on the required throughput that conforming decoders need to support in the decoded image domain (also called the spatial domain). The following table lists the levels as defined by JPEG XS. The maximums are given in the context of the sampling grid, so they refer to a per |
In addition, JPEG XS defines levels to represent a lower bound on the required throughput that conforming decoders need to support in the decoded image domain (also called the spatial domain). The following table lists the levels as defined by JPEG XS. The maximums are given in the context of the sampling grid, so they refer to a per-pixel value where each pixel represents one or more component values. However, in the context of Bayer data JPEG XS internally interprets the Bayer pattern as an interleaved grid of four components. This means that the number of sampling grid points required to represent a Bayer image is four times smaller than the total number of Bayer sample points. Each group of 2x2 (four) Bayer values gets interpreted as one sampling grid point with four components. Thus sensor resolutions should be divided by four to calculate the respective width, height and amount of sampling grid points. For this reason, all levels also bear double names. Please refer to the standard for a complete specification of each level. |
||
{| class="wikitable mw-collapsible mw-collapsed" border="1" style="border-collapse: collapse;" |
{| class="wikitable mw-collapsible mw-collapsed" border="1" style="border-collapse: collapse;" |
||
Line 234: | Line 291: | ||
|} |
|} |
||
Similarly to the concept of levels, JPEG XS defines sublevels to represent a lower bound on the required throughput that conforming decoders need to support in the encoded image domain. Each sublevel is defined by a nominal |
Similarly to the concept of levels, JPEG XS defines sublevels to represent a lower bound on the required throughput that conforming decoders need to support in the encoded image domain. Each sublevel is defined by a nominal bit-per-pixel (Nbpp) value that indicates the maximum amount of bits per pixel for an encoded image of the maximum permissible number of sampling grid points according to the selected conformance level. Thus, decoders conforming to a particular level and sublevel shall conform to the following constraints derived from Nbpp: |
||
* The maximum codestream size in bytes (from SOC to EOC, including all markers) is <math>S_{sl,max}=\bigg\lfloor \frac{L_{max}\times N_{bpp}}{8} \bigg\rfloor</math>. |
* The maximum codestream size in bytes (from SOC to EOC, including all markers) is <math>S_{sl,max}=\bigg\lfloor \frac{L_{max}\times N_{bpp}}{8} \bigg\rfloor</math>. |
Revision as of 16:25, 10 July 2024
This article contains content that is written like an advertisement. (January 2024) |
Filename extension | .jxs |
---|---|
Internet media type |
image/jxsc, video/jxsv[1] |
Magic number | 0xFF10 FF50 |
Developed by | Joint Photographic Experts Group |
Initial release | May 1, 2019 |
Type of format | Lossy and lossless image compression format |
Standard | ISO/IEC 21122 |
Website | jpeg |
JPEG XS (ISO/IEC 21122) is an interoperable, visually lossless, low-latency and lightweight image and video coding system used in professional applications.[2][3][4][5][6] Applications of the standard include streaming high-quality content for virtual reality, drones, autonomous vehicles using cameras, gaming, and broadcasting (SMPTE ST 2022 and ST 2110).[3][7][8][9] It was the first ISO codec ever designed for this specific purpose. JPEG XS, built on core technology from both intoPIX and Fraunhofer IIS, is formally standardized as ISO/IEC 21122 by the Joint Photographic Experts Group with the first edition published in 2019. Although not official, the XS acronym was chosen to highlight the eXtra Small and eXtra Speed characteristics of the codec. Today, the JPEG committee is still actively working on further improvements to XS, with the second edition[10] scheduled for publication (beginning of 2022) and initial efforts being launched towards a third edition.
Features of JPEG XS
Three main features are key to JPEG XS:
- Visually transparent compression: XS compressed content is indistinguishable from the original uncompressed content (passing ISO/IEC 29170-2 tests) for compression ratios up to 3 bit per pixel (bpp).
- Low latency: The total end-to-end latency, introduced by the XS compression-decompression cycle, is minimal. Depending on the configuration, XS typically imposes only between 1 and 32 lines of additional end-to-end latency, when compared to the same system using uncompressed video.
- Lightweight: XS is designed to have low computational and memory complexity, allowing for efficient low-power and low-resource implementations on various platforms such as CPU, GPU, FPGA and ASIC.
Relying on these key features, JPEG XS is suitable to be used in any application where uncompressed content is now the norm, yet still allowing for significant savings in the required bandwidth usage, preserving quality and low latency. Among the targeted use cases are video transport over professional video links (like SDI and Ethernet/IP), real-time video storage, memory buffers, omnidirectional video capture and rendering, and image sensor compression (for example in cameras and in the automotive industry). Typical compression ratios go up to 10:1 but can also be higher depending on the nature of the image or the requirements of the targeted application.[11] JPEG XS favors visually lossless quality in combination with low latency and low complexity, over crude compression performance. Hence, it is not a direct competitor to alternative image codecs like JPEG 2000 and JPEG XL or video codecs like AV1, AVC/H.264 and HEVC/H.265.
Other important features are:
- Exact bitrate allocation: JPEG XS allows to accurately set the targeted bitrate to perfectly match the available bandwidth (also referred to as constant bitrate or CBR).
- Multi-generation robustness: JPEG XS allows for at least 10 encoding-decoding cycles, without significant quality degradation.[12] This feature allows for example transparently chaining of multiple devices that recompress the signal, without any significant quality degradation taking place.
- Multi-platform interoperability: The algorithms used in JPEG XS allow for efficient implementations on different platforms, like CPU, GPU, FPGA and ASIC. Each of these platform architectures is best exploited when a specific degree of parallelism is available in the implementation. For instance, a multi-core CPU implementation will leverage a coarse-grained parallelism, while GPU or FPGA will work better with a fine-grained parallelism. Moreover, the choice of parallelism used in the implementation at the encoder will not affect that of the decoder. This means that real-time encoding and decoding between platforms is possible, without sacrificing the low complexity, low latency or high-quality properties.
- Support for mathematical lossless coding (MLS): JPEG XS is also capable of coding images in a mathematically lossless way, to achieve perfect reconstruction at the decoder side (new profile supported by 2nd edition).
- Support for High Dynamic Range (HDR) content: The current version of JPEG XS supports bit-depths of up to 16 bits per component, and it provides several parameterizable non-linear transforms (NLTs) to efficiently compress HDR content.
- Support for RAW Bayer/CFA compression: JPEG XS has also the capability to compress Color Filter Array (CFA) content, such as RAW Bayer content produced by digital cameras. A special color transform, called Star-Tetrix, allows for efficient and direct compression of the original RAW sample values, without the need for converting the Bayer samples to RGB samples first.[13]
- Accurate flow control: A JPEG XS encoder continuously monitors the amount of bits sent out, and adjusts its rate allocation process to neither overflow nor underflow a normatively[14] defined decoder input buffer.
Application domains
This section lists them main application domains where JPEG XS is actively used. New and other application domains are subject to be added in the future, for example, framebuffer compression or AR/VR applications.
Transport over video links and IP networks
Video bandwidth requirements are growing continuously, as video resolutions, frame rates, bit depths, and the amount of video streams are constantly increasing. Likewise, the capacities of video links and communication channels are also growing, yet at a slower pace than what is needed to address the huge video bandwidth growth. In addition, the investments to upgrade the capacity of links and channels are significant and need to be amortized over several years.
Moreover, both the broadcast and pro-AV markets are shifting towards AV-over-IP-based infrastructure, with a preference going to 1 Gigabit Ethernet links for remote production or 10G Ethernet networks for in-house facilities. Both 1G, 2.5G, and 10G Ethernet are cheap and ubiquitous, while 25G or better links are usually not yet affordable. Given the available bandwidth and infrastructure cost, relying on uncompressed video is therefore no longer an option, as 4K, 8K, increased bit depths (for HDR), and higher framerates need to be supported.
JPEG XS is a light-weight compression that visually preserves the quality compared to an uncompressed stream, at a low cost, targeted at compression ratios of up to 10:1. With XS, it is for example possible to repurpose existing SDI cables to transport 4K60 over a single 3G-SDI (at 4:1), and even over a single HD-SDI (at 8:1). Similar scenarios can be used to transport 8K60 content over various SDI cable types (e.g. 6G-SDI and 12G-SDI). Alternatively, XS enables transporting 4K60 content over 1G Ethernet and 8K60 over 5G or 10G Ethernet, which would be impossible without compression. The following table shows some expected compression ranges for some typical use cases.
Video stream | Video throughput | Link type | Available throughput | Compression ratio |
---|---|---|---|---|
2k 60 fps 4:2:2 10 bpc | 2.7 Gbit/s | HD-SDI | 1.33 Gbit/s | ~2 |
4k 60 fps 4:2:2 10 bpc | 10.6 Gbit/s | 3G-SDI | 2.65 Gbit/s | ~4 |
2k 60 fps 4:2:2 10 bpc | 2.7 Gbit/s | 1G Ethernet | 0.85 Gbit/s | ~3 |
2k 60 fps 4:4:4 12 bpc | 4.8 Gbit/s | 1G Ethernet | 0.85 Gbit/s | ~6 |
4k 60 fps 4:4:4 12 bpc | 19.1 Gbit/s | 10G Ethernet | 7.96 Gbit/s | ~2.2 |
8k 60 fps 4:2:2 10 bpc | 42.5 Gbit/s | 10G Ethernet | 7.96 Gbit/s | ~6 |
8k 120 fps 4:2:2 10 bpc | 84.9 Gbit/s | 25G Ethernet | 21.25 Gbit/s | ~4 |
Real-time video storage and playout
Related to the transport of video streams is the storage and retrieval of high-resolution streams where bandwidth limitations similarly apply. For instance, video cameras use internal storage like SSD drives or SD cards to hold large streams of images, yet the maximum data rates of such storage devices are limited and well below the uncompressed video throughput.
Sensor compression
As stated, JPEG XS has built-in support for the direct compression of RAW Bayer/CFA images using the Star-Tetrix Color Transform. This transform takes a RAW Bayer pattern image and decorrelates the samples into a 4-component image with each component having only a quarter of the resolution.[15] This means that the total amount of samples to further process and compress remains the same, yet the values are decorrelated similarly to a classical Multiple Component Transform.
Avoiding such conversion prevents information loss and allows this processing step to be done outside of the camera. This is advantageous because it allows to defer demosaicing the Bayer content from the moment of capturing to the production phase, where choices regarding artistic intent and various settings can be better made. Recall that the demosaicing process is irreversible and requires certain choices, like the choice of interpolation algorithm or the level of noise reduction, to be made upfront. Moreover, the demosaicing process can be power-hungry and will also introduce extra latency and complexity. The ability to push this step out of the camera is possible with JPEG XS and allows to use more advanced algorithms resulting in better quality in the end.
Standards
JPEG XS (ISO/IEC 21122)
The JPEG XS coding system is an ISO/IEC suite of standards that consists of the following parts:
Part | 1st edition | 2nd edition (in force) | 3rd edition | Title |
---|---|---|---|---|
1 | 2019 | 2022 | scheduled 2024 | Core coding system |
2 | 2019 | 2022 | scheduled 2024 | Profiles and buffer models |
2 | - | 2022/Amd1 | - | Profile and sublevel for 4:2:0 content |
3 | 2019 | 2022 | scheduled 2024 | Transport and container formats |
4 | 2020 | 2022 | in development | Conformance testing |
5 | 2020 | 2022 | in development | Reference software |
Part 1, formally designated as ISO/IEC 21122-1, describes the core coding system of JPEG XS. This standard defines the syntax and, similarly to other JPEG and MPEG image codecs, the decompression process to reconstruct a continuous-tone digital image from its encoded codestream. Part 1 does provide some guidelines of the inverse process that compresses a digital image into a compressed codestream, or more simply called the encoding process, but leaves implementation-specific optimizations and choices to the implementers.
Part 2 (ISO/IEC 21122-2) builds on top of Part 1 to segregate different applications and uses of JPEG XS into reduced coding tool subsets with tighter constraints. The definition of profiles, levels, and sublevels allows for reducing the complexity of implementations in particular application use cases, while also safeguarding interoperability. Recall that lower complexity typically means less power consumption, lower production costs, easier constraints, etc. Profiles represent interoperability subsets of the codestream syntax specified in Part 1. In addition, levels and sublevels provide limits to the maximum throughput in respectively the encoded (codestream) and the decoded (spatial and pixels) image domains. Part 2 furthermore also specifies a buffer model, consisting of a decoder model and a transmission channel model, to enable guaranteeing low latency requirements to a fraction of the frame size.
Part 3 (ISO/IEC 21122-3) specifies transport and container formats for JPEG XS codestreams. It defines the carriage of important metadata, like color spaces, mastering display metadata (MDM), and EXIF, to facilitate transport, editing, and presentation. Furthermore, this part defines the XS-specific ISOBMFF boxes, an Internet Media Type registration, and additional syntax to allow embedding XS in formats like MP4, MPEG-2 TS, or the HEIF image file format.
Part 4 (ISO/IEC 21122-4) is a supporting standard of JPEG XS that provides conformance testing and buffer model verification. This standard is crucial to implementers of XS and appliance conformance testing.
Finally, Part 5 (ISO/IEC 21122-5) represents a reference software implementation (written in ISO C11) of the JPEG XS Part 1 decoder, conforming to the Part 2 profiles, levels and sublevels, as well as an exemplary encoder implementation.
A second edition of all five parts is in the making and will be published at the latest in the beginning of 2022. It provides additional coding tools, profiles and levels, and new reference software to add support for efficient compression of 4:2:0 content, RAW Bayer/CFA content, and mathematically lossless compression.
RFC9134 - RTP Payload Format for ISO/IEC 21122 (JPEG XS)
RFC 9134[16] describes a payload format for the Real-Time Transport Protocol (RTP, RFC 3550[17]) to carry JPEG XS encoded video. In addition, the recommendation also registers the official Media Type Registration for JPEG XS video as video/jxsv
, along with its mapping of all parameters into the Session Description Protocol (SDP).
The RTP Payload Format for JPEG XS in turn enables using JPEG XS in SMPTE ST 2110 environments using SMPTE ST 2110-22 for CBR compressed video transport.
MPEG-TS for JPEG XS
ISO/IEC 13818-1:2022, known as MPEG-TS 8th edition, specifies carriage support for JPEG XS in MPEG Transport Streams.[18] See also MPEG-2. Note that AMD1 (Carriage of LCEVC and other improvements) of ISO/IEC 13818-1:2022 contains some additional corrections, improvements, and clarifications regarding embedding JPEG XS in MPEG-TS.[19]
VSF TR-07 and TR-08
See VSF TR-07[20] and TR-08,[21] published by the Video Services Forum
NMOS with JPEG XS
A Networked Media Open Specifications that enables registration, discovery, and connection management of JPEG XS endpoints using the AMWA IS-04 and IS-05 NMOS Specifications. See AMWA BCP-006-01,[22] published by Advanced Media Workflow Association.
JPEG XS in IPMX
Internet Protocol Media Experience (IPMX) is a proposed set of open standards and specifications to enable the carriage of compressed and uncompressed video, audio, and data over IP networks for the pro AV market. JPEG XS is supported under IPMX via VSF TR-10-8[23] and TR-10-11.[24]
History
The JPEG committee started the standardization activity in 2016 with an open call for a high-performance, low-complexity image coding standard. The best-performing candidates formed the basis for the new standard. First implementations were demonstrated in April 2018 at the NAB Show and later that year at the International Broadcasting Convention.[25] XS was also presented at CES in 2019.
Technical overview
Core coding
The JPEG XS standard is a classical wavelet-based still-image codec without any frame buffer. While the standard[26] defines JPEG XS based on a hypothetical reference coder, JPEG XS is easier to explain through the steps a typical encoder performs:[27]
Component up-scaling and optional component decorrelation: In the first step, the DC gain of the input data is removed and it is upscaled to a bit-precision of 20 bits. Optionally, a multi-component generation, identical to the JPEG 2000 RCT, is applied. This transformation is a lossless approximation of an RGB to YUV conversion, generating one luma and two chroma channels.
Wavelet transformation: Input data is spacially decorrelated by a 5/3 Daubechies wavelet filter. While a five-stage transformation is performed in the horizontal direction, only 0 to 2 transformations are run in the vertical direction. The reason for this asymmetrical filter is to minimize latency.
Prequantization: The output of the wavelet filter is converted to a sign-magnitude representation and pre-quantized by a dead zone quantizer to 16-bit precision.
Rate control and quantization: The encoder determines by a non-normative process[27] the rate of each possible quantization setting and then quantizes data by either a dead zone quantizer or a data-dependent uniform quantizer.
Entropy coding: JPEG XS uses minimalistic Entropy encoding for the quantized data which proceeds in up to four passes over horizontal lines of quantized wavelet coefficients. The steps are:
- Significance coding: In the (optional) first pass, the significance of 32 consecutive wavelet coefficients is coded by a single bit.
- Bitplane count coding: In the second pass, the number of non-zero bitplanes of groups of four coefficients each, the so-called "bitplane count", is entropy coded through a Golomb type code. This step may optionally use the bitplane counts of the preceding line as the source for prediction (Differential pulse-code modulation) and then encode only the prediction difference.
- Data coding:The third pass inserts the raw bitplane values into the codestream without further coding.
- Sign coding: In the last optional coding pass, the sign bits of all non-zero coefficients are inserted into the codestream. If this coding pass is not present, sign bits are included in the data coding pass for all coefficients.
Codestream packing: All entropy-coded data are packed into a linear stream of bits (grouped in byte multiples) along with all of the required image metadata. This sequence of bytes is called the codestream and its high-level syntax is based on the typical JPEG markers and marker segments syntax.[28]
Profiles, levels and sublevels
JPEG XS defines profiles (in ISO/IEC 21122-2) that define subsets of coding tools that conforming decoders shall support, by limiting the permitted parameter values and allowed markers. The following table represents an overview of all the profiles along with their most important properties. Please refer to the standard for a complete specification of each profile.
Profile | Ppih | B[i] | Nbpp,max | Bw | Br | Fq | Qpih | Horizontal DWT | Vertical DWT | Chroma sampling formats | Cpih | Edition |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Light 422.10 | 0x1500 |
8, 10 | 20 | 20 | 4 | 8 | 0 | 1 to 5 | 0, 1 | 4:0:0, 4:2:2 | 0 | 1 |
Light 444.12 | 0x1A00 |
8, 10, 12 | 36 | 20 | 4 | 8 | 0 | 1 to 5 | 0, 1 | 4:0:0, 4:2:2, 4:4:4 | 0, 1 | 1 |
Light-Subline 422.10 | 0x2500 |
8, 10 | 20 | 20 | 4 | 8 | 0, 1 | 1 to 5 | 0 | 4:0:0, 4:2:2 | 0 | 1 |
Main 420.12 | 0x3240 |
8, 10, 12 | 18 | 20 | 4 | 8 | 0, 1 | 1 to 5 | 1 | 4:2:0 | 0 | 1 |
Main 422.10 | 0x3540 |
8, 10 | 20 | 20 | 8 | 4 | 0, 1 | 1 to 5 | 0, 1 | 4:0:0, 4:2:2 | 0 | 1 |
Main 444.12 | 0x3A40 |
8, 10, 12 | 36 | 20 | 4 | 8 | 0, 1 | 1 to 5 | 0, 1 | 4:0:0, 4:2:2, 4:4:4 | 0, 1 | 1 |
Main 4444.12 | 0x3E40 |
8, 10, 12 | 48 | 20 | 4 | 8 | 0, 1 | 1 to 5 | 0, 1 | 4:0:0, 4:2:2, 4:4:4, 4:2:2:4, 4:4:4:4 | 0, 1 | 1 |
High 420.12 | 0x4240 |
8, 10, 12 | 18 | 20 | 4 | 8 | 0, 1 | 1 to 5 | 1, 2 | 4:2:0 | 0 | 2 |
High 444.12 | 0x4A40 |
8, 10, 12 | 36 | 20 | 4 | 8 | 0, 1 | 1 to 5 | 0, 1, 2 | 4:0:0, 4:2:2, 4:4:4 | 0, 1 | 1 |
High 4444.12 | 0x4E40 |
8, 10, 12 | 48 | 20 | 4 | 8 | 0, 1 | 1 to 5 | 0, 1, 2 | 4:0:0, 4:2:2, 4:4:4, 4:2:2:4, 4:4:4:4 | 0, 1 | 1 |
CHigh 444.12 | 0x4A44 |
8, 10, 12 | 36 | 20 | 4 | 8 | 0, 1 | 1 to 5 | 0, 1, 2 | 4:0:0, 4:2:2, 4:4:4 | 0, 1 | 3 |
TDC 444.12 | 0x4A45
|
8, 10, 12 | 36 | 20 | 4 | 8 | 0, 1 | (3, 0) and (4, 0) if not 4:2:0,
(4, 1), (5, 1), (5, 2) otherwise |
4:0:0,
4:2:0, 4:2:2, 4:4:4 |
0, 1 | 3 | |
TDC MLS 444.12 | 0x6A45
|
8, 10, 12 | 36 | B[i] | 4 | 0 | 0, 1 | (3, 0) and (4, 0) if not 4:2:0,
(4, 1), (5, 1), (5, 2) otherwise |
4:0:0,
4:2:0, 4:2:2, 4:4:4 |
0, 1 | 3 | |
MLS.12 | 0x6EC0 |
8, 10, 12 | 48 | B[i] | 4 | 0 | 0, 1 | 1 to 5 | 0, 1, 2 | 4:0:0,
4:2:0, 4:2:2, 4:4:4, 4:2:2:4, 4:4:4:4 |
0, 1 | 2 |
MLS.16 | 0x6ED0 |
8, 10, 12, 14, 16 | 64 | B[i] | 5 | 0 | 0, 1 | 1 to 5 | 0, 1, 2 | 4:0:0,
4:2:0, 4:2:2, 4:4:4, 4:2:2:4, 4:4:4:4 |
0, 1 | 3 |
LightBayer | 0x9300 |
10, 12, 14, 16 | 64 | 18, 20 | 4 | 6, 8 | 0, 1 | 1 to 5 | 0 | Bayer | 3 | 2 |
MainBayer | 0xB340 |
10, 12, 14, 16 | 64 | 18, 20 | 4 | 6, 8 | 0, 1 | 1 to 5 | 0, 1 | Bayer | 3 | 2 |
HighBayer | 0xC340 |
10, 12, 14, 16 | 64 | 18, 20 | 4 | 6, 8 | 0, 1 | 1 to 5 | 0, 1, 2 | Bayer | 3 | 2 |
In addition, JPEG XS defines levels to represent a lower bound on the required throughput that conforming decoders need to support in the decoded image domain (also called the spatial domain). The following table lists the levels as defined by JPEG XS. The maximums are given in the context of the sampling grid, so they refer to a per-pixel value where each pixel represents one or more component values. However, in the context of Bayer data JPEG XS internally interprets the Bayer pattern as an interleaved grid of four components. This means that the number of sampling grid points required to represent a Bayer image is four times smaller than the total number of Bayer sample points. Each group of 2x2 (four) Bayer values gets interpreted as one sampling grid point with four components. Thus sensor resolutions should be divided by four to calculate the respective width, height and amount of sampling grid points. For this reason, all levels also bear double names. Please refer to the standard for a complete specification of each level.
Level | Max width | Max height | Max pixels (Lmax) | Max pixel rate (Rs,max) | Plev High Byte |
---|---|---|---|---|---|
Unrestricted | 65535 | 65535 | - | - | 0x00
|
1k-1, Bayer2k-1 | 1280 | 5120 | 2621440 | 83558400 | 0x04
|
2k-1, Bayer4k-1 | 2048 | 8192 | 4194304 | 133693440 | 0x10
|
4k-1, Bayer8k-1 | 4096 | 16384 | 8912896 | 267386880 | 0x20
|
4k-2, Bayer8k-2 | 4096 | 16384 | 16777216 | 534773760 | 0x24
|
4k-3, Bayer8k-3 | 4096 | 16384 | 16777216 | 1069547520 | 0x28
|
8k-1, Bayer16k-1 | 8192 | 32768 | 35651584 | 1069547520 | 0x30
|
8k-2, Bayer16k-2 | 8192 | 32768 | 67108864 | 2139095040 | 0x34
|
8k-3, Bayer16k-3 | 8192 | 32768 | 67108864 | 4278190080 | 0x38
|
10k-1, Bayer20k-1 | 10240 | 40960 | 104857600 | 3342336000 | 0x40
|
Similarly to the concept of levels, JPEG XS defines sublevels to represent a lower bound on the required throughput that conforming decoders need to support in the encoded image domain. Each sublevel is defined by a nominal bit-per-pixel (Nbpp) value that indicates the maximum amount of bits per pixel for an encoded image of the maximum permissible number of sampling grid points according to the selected conformance level. Thus, decoders conforming to a particular level and sublevel shall conform to the following constraints derived from Nbpp:
- The maximum codestream size in bytes (from SOC to EOC, including all markers) is .
- The maximum admissible encoded throughput in bits per second is .
The following table lists the existing sublevels and their respective nominal bpp values. Please refer to the standard for a complete specification of each level.
Sublevel | Nominal bpp (Nbpp) | Plev Low Byte |
---|---|---|
Unrestricted | - | 0x00
|
Full | Native image bpp | 0x80
|
Sublev12bpp | 12 | 0x10
|
Sublev9bpp | 9 | 0x0C
|
Sublev6bpp | 6 | 0x08
|
Sublev4bpp | 4 | 0x06
|
Sublev3bpp | 3 | 0x04
|
Sublev2bpp | 2 | 0x03
|
Patents and RAND
JPEG XS contains patented technology which is made available for licensing via the JPEG XS Patent Portfolio License (JPEG XS PPL). This license pool covers essential patents owned by Licensors for implementing the ISO/IEC 21122 JPEG XS video coding standard and is available under RAND terms.[29]
References
- ^ https://www.iana.org/assignments/media-types/video/jxsv [bare URL plain text file]
- ^ "JPEG XS - the new low complexity codec standard for professional video production". Fraunhofer Institute for Integrated Circuits IIS. 2019-01-09.
- ^ a b "Overview of JPEG XS". jpeg.org. Retrieved 2019-07-31.
- ^ "Le JPEG XS, futur standard de compression pour la réalité virtuelle". LeSoir.be. 2018-04-19.
- ^ "Le nouveau standard JPEG XS est belge". LeSoir.be. 2019-01-10.
- ^ T. Richter, J. Keinert, S. Foessel, A. Descampe, G. Rouvroy and J. -B. Lorent, "JPEG-XS -- A High-Quality Mezzanine Image Codec for Video Over IP," in SMPTE Motion Imaging Journal, vol. 127, no. 9, pp. 39-49, Oct. 2018, doi: 10.5594/JMI.2018.2862098 link.
- ^ "Les belges d'intoPIX révolutionnent la compression d'image et de vidéo avec la nouvelle norme JPEG XS". January 10, 2019.
- ^ Pessina, Laure-Anne (September 4, 2018), A new JPEG format for virtual reality, drones and self-driving cars, archived from the original on 2023-06-05 – via actu.epfl.ch
- ^ Keinert, Joachim; Lorent, Jean-Baptiste; Descampe, Antonin; Rouvroy, Gaël; Fößel, Siegfried. "Introduction to JPEG XS". IBC. Archived from the original on 2023-04-02.
- ^ A. Descampe et al., "JPEG XS--A New Standard for Visually Lossless Low-Latency Lightweight Image Coding," in Proceedings of the IEEE, doi: 10.1109/JPROC.2021.3080916 PDF.
- ^ "JPEG White paper: JPEG XS, a new standard for visually lossless low-latency lightweight image coding system" (PDF).
- ^ Richter, Thomas; Keinert, Joachim; Descampe, Antonin; Rouvroy, Gael; Willeme, Alexandre (December 2017). "Multi-generation-robust Coding with JPEG XS". 2017 IEEE International Symposium on Multimedia (ISM). pp. 6–13. doi:10.1109/ISM.2017.12. ISBN 978-1-5386-2937-6. S2CID 32986247.
- ^ T. Richter, S. Fößel, A. Descampe and G. Rouvroy, "Bayer CFA Pattern Compression With JPEG XS," in IEEE Transactions on Image Processing, vol. 30, pp. 6557-6569, 2021, doi: 10.1109/TIP.2021.3095421 link.
- ^ International Organization for Standardization. "ISO/IEC 21122-2:2019". ISO. Retrieved 2021-09-07.
- ^ Richter, Thomas; Fößel, Siegfried; Descampe, Antonin; Rouvroy, Gaël (2021). "Bayer CFA Pattern Compression With JPEG XS". IEEE Transactions on Image Processing. 30: 6557–6569. Bibcode:2021ITIP...30.6557R. doi:10.1109/TIP.2021.3095421. ISSN 1941-0042. PMID 34270422. S2CID 236001190.
- ^ RTP Payload Format for ISO/IEC 21122 (JPEG XS). RFC 9134.
- ^ RTP: A Transport Protocol for Real-Time Applications. RFC 3550.
- ^ "Iso/Iec 13818-1:2022".
- ^ "ISO/IEC 13818-1:2022/Amd 1:2023".
- ^ TR-07: Transport of JPEG XS Video in MPEG-2 TS over IP
- ^ TR-08: Transport of JPEG XS Video in ST 2110-22
- ^ AMWA BCP-006-01: NMOS With JPEG XS
- ^ TR-10-8:2023-Internet Protocol Media Experience (IPMX): NMOS Requirements - DRAFT
- ^ TR-10-11:2023-Internet Protocol Media Experience (IPMX): Constant Bit-Rate Compressed Video - DRAFT
- ^ "JPEG XS, le nouveau standard de compression made in UCLouvain". UCLouvain.
- ^ International Organization for Standardization. "ISO/IEC 21122-1:2019". ISO. Retrieved 2021-09-07.
- ^ a b Descampe, Antonin; Richter, Thomas; Ebrahimi, Touradj; Foessel, Siegfried; Keinert, Joachim; Bruylants, Tim; Pellegrin, Pascal; Buysschaert, Charles; Rouvroy, Gaël (September 2021). "JPEG XS—A New Standard for Visually Lossless Low-Latency Lightweight Image Coding". Proceedings of the IEEE. 109 (9): 1559–1577. doi:10.1109/JPROC.2021.3080916. ISSN 1558-2256. S2CID 236727596.
- ^ "codestream-parser". June 16, 2022 – via GitHub.
- ^ "JPEG XS Patent Pool". JPEG XS Patent Pool.