Image processing and the beauty of mathematics

Image processing is the manipulation of or a group of techniques that manipulate images. It could be manipulating images to enhance content in these images, like removing noise or deblurring it, or it could be processing or manipulating images to quantify things in images, like segmenting objects in these images, estimating shapes of these objects, color or things like this. This is what image processing does, developing methods that can do this in an automated fashion without, or with only minimal, human intervention.

Segmentation, so isolating objects and images, tracking them and so on, and usually the development of these methodologies, which are very generic in themselves, is linked to quite specific real-world applications, where a lot of the interdisciplinary work comes in that we do, in fields like biology, medicine, material sciences, cultural heritage, all the way to conservation and even forensics.

This, to a certain extent, is based on diffusion. This is describing the diffusion of the milk over time, and you can describe this by the heat equation. Another example is, heat tells you something about the distribution of heat in a room. It can tell you, if I have a room with two sources of heat like my two radiators here, how the heat from these sources distributes into the room. This can be nicely used, instead of with heat, with an intensity function, diffusing color and gray values in an image, which, for instance, you could use to smooth an image and get rid of things like noise. You can think of it as a component that is very rough in this image, in this color intensity function that you want to smooth out with this diffusion process.

The issues that you have usually with this are that noise added to an image is random. You sometimes know what the distribution of this noise is, which can help you develop methods to subtract the noise, but it's not that you know what the intensity of this noise is in every pixel. If you did, you could just subtract it and then have your clean image. It's a random thing that has been done to your image. Now the goal is to decompose this random component; usually, if you look at the rainbow, you can imagine that this is a very highly oscillatory component – intensities are changing randomly from one pixel to the next – and you want to subtract this random component from the clean, structured, somewhat smooth image underneath.

Of course, it will also kill some very sharp features that I have; it's blurring the image everywhere. This is why mathematical methods and image processing exist as a field – to develop differential equations that go beyond the heat equation, but that can weigh diffusion by the amount of content that there is underneath. You diffuse less when you have sharp features that you would like to preserve, and you diffuse more in regions where your intensity, your color function, is kind of constant or nearly constant. As such, you can get a crisp image that doesn't have noise.

The main goal of image inpainting – which is the restoration of digital images – is to replace the snowball with something that is visually plausible, given what you see from the rest of the image or your experience of what is potentially behind the snowball.

We use the intensity values on the boundary of the snowball, on the edges of the snowball, and then we diffuse these into the inpainting domain, which is defined by the snowball.

Computer tomography — X-ray tomography — measures line integrals through your human body. From these you want to reconstruct an image. By the way, line integrals, also called the Radon transform, are named after an Austrian mathematician, Johann Radon, who was looking at the Radon transform totally separately from computer tomography, which didn't exist at the time that he was alive; this shows the importance of sometimes blue-sky mathematical research that doesn't have an immediate application. This is where mathematical imaging can help to get the best image. Best, of course, depends on what you want to do later with the image. In a lot of non-medical imaging like photography and so on, the best image is going to be the visually most pleasing image. In medical imaging, the best image often is the one that allows me to, in the most accurate way, make a diagnostic decision or a prognosis on the individual that I'm examining or helps me characterize a certain type of disease in the most accurate or meaningful way. The best image is in the eye of the observer or the user. This is where mathematics comes in, and it comes in because there is not just one unique answer, one image that fits these measurements. These measurements that are taken, as any kind of real-world measurements, are going to be messy. They're going to be noisy, imperfect; there might be gaps. In medical imaging, what also happens is that the coverage of all the measurements that uniquely define the image that I can reconstruct from it might not be complete in this transform domain, like Fourier or Radon. There is some ambiguity, like in the inpainting case. You can reconstruct multiple images that fit the measurements in the same way. You want to reconstruct the one that is best.

It might allow you to look more at things that dynamically happen in the human body. You need a certain number of measurements for each image snapshot in time, but if you want to look at things dynamically, you have less time per time snapshot. This is just one example in this whole medical imaging field where mathematics can help to optimize the trade-off between how much effort, how much money and how much time you put into measurements. In computer tomography and X-ray imaging, this could also relate to radiation, how much you radiate the patient. The less you radiate, the more noise you usually have in these measurements, so you need to optimize this trade-off between this and the quality of the image. For instance, in a collaboration that we had with a former postdoc of mine and colleagues at Addenbrooke's Hospital, we developed mathematical methods that could help speed up the process of magnetic resonance imaging acquisition by an order of magnitude. Instead of being there for tens of minutes – half an hour or more – you could reduce this to just a couple of minutes without losing information content.

That is the beauty of this: a lot of these methods translate from one field to the next in terms of application domain, and then it's a matter of choosing. There are lots of examples where images appear. They appear in healthcare; they appear in cultural heritage; they appear in environmental monitoring, in material sciences. A lot of research – and also everyday life in how we operate nowadays – is dependent on automated interpretation of imaging data, loads of digital imaging data.

On the other hand, we have the emergence of black-box models like deep neural networks that can characterize information content in data sets – imaging data sets – to a very high accuracy, but without the accountability, trustability and interpretability features that we have with classical methods. Finding the sweet spot between the two that gives us the best of both worlds is one of the main challenges mathematical imaging people are facing nowadays.

Editor’s note: This article has been faithfully transcribed from the original interview filmed with the author, and carefully edited and proofread. Edit date: 2025