Associate Professor Jona Ballé explains the next frontier in image coding

This January, the first image coding standard based on machine learning technology, JPEG AI, was sent for publication as an International Standard. JPEG AI leverages JPEG with major trends in imaging technologies and provides an efficient standardized solution for image coding, with nearly 30% improvement over the most advanced solutions, making this a significant milestone.

Associate Professor of Electrical and Computer Engineering Jona Ballé has done foundational research that contributed to the development of JPEG AI. We caught up with them to learn more.

Q: What is JPEG AI?

A: You may know JPEG as a way to store your photos on a computer. Most likely, you've seen photos stored with the filename ending ".jpg" or ".jpeg" at some point. But it is also used when photos are transmitted, for example when you are looking at a photo on a website, or on other digital devices, such as a smartphone or a digital camera. JPEG stands for Joint Photographic Experts Group — as the name implies, it's a group of experts on digital imaging. They've been working on standards for how to represent digital photographs on computers, for a long time — at least since the 1980s. That's longer ago than I am old!

JPEG keeps improving their recommendations on exactly how images are represented (or "coded") digitally. We also say that an image is "compressed" in a digital file, because we try to use as few bits or bytes to store them as possible. That makes it possible to store more photos on the same device, or load a website faster. JPEG regularly uses new research insights to improve the average compression efficiency — that is, how precise the image is represented vs. how many bytes it requires to store it. They can also make use of improvements in hardware, for example, faster chips, to improve efficiency. JPEG AI is the latest of their recommendations, which for the first time makes use of machine learning.

In machine learning, the algorithm that is used to compress the image is not entirely specified by humans, but part of the algorithm is "learned," that is it is optimized by running it on a large number of examples, and the machine figures out by itself how to best tune the algorithm.

Q: What does it mean to become an international standard?

A: An international standard is an agreement. In this case, on how to compress the image (take sensor readings from a camera and store it as bytes) and how to decompress it (read the bytes and make it visible on a display). This is very useful, since we can pass these digital representations of images around (put them on a disk, send them by email, etc.) and rest assured that whoever we give them to can access the image.

So if many of our devices support the same standard, it makes our digital lives a lot easier.

Q: Why is that considered such a milestone?

A: With each new standard, we improve the way we handle digital imagery. Sometimes, as in this case, it is mostly about making things more efficient. Other times, a new standard can handle entirely new media. For example, MPEG is a similar group that was formed to work on moving images — that is, video. There are also more recent efforts to standardize 3D representations of images, like 3D movies or entire virtual reality scenes that you can walk around in, and interact with.

Q: As a layperson, I sometimes see images labeled JPEG; will this development affect me at all?

A: Before a new standard is actually used, it needs to be supported by our devices. So right now, it will not noticeably affect you. JPEG AI is still in the process of being officially published. But two or three years from now, we might see technology that supports JPEG AI.

Q: Was your work foundational to this (or in what ways did it contribute)?

A: I am very proud to say that my original research papers were the inspiration for JPEG AI to be developed. Its fundamental structure is based on some of my research that I published starting in 2016. Back then, I was a post-doc researcher at NYU and among a few researchers worldwide who considered using machine learning in image compression. Around 2020, after I had transitioned to a role in Google Research, I was contacted by a few researchers working on standardization, because they wanted to evaluate my published method — a working prototype of sorts — for developing a new standard. It turns out that it performed so well that they decided it was worth pursuing a new JPEG standard. Over the next few years, many experts across the world tweaked my method, added new components to it, solved existing issues, re-evaluated it, and so forth. Recently, they voted it mature enough to publish.

In my research, I like to stay a bit ahead of these developments. I'm already working on the next improvement that — hopefully one day — will be considered substantial enough to be used in a standard.