Mathematical imaging and deep learning

Mathematical image processing has been hit tremendously by the emergence and success of deep learning, and this is really when machine learning has become an extremely powerful tool, all of a sudden, for image processing. Before, we had more low-level machine learning in image processing, but it was not really something that was overshadowing everything else. The work that mathematical imaging people had done before used model-driven approaches based on mathematical equations, deriving these equations from first principles of understanding what the image is all about — what the problem with this image is, for instance, for denoising or deconvolution — and then thinking about how to turn it into a mathematical model that is well posed, that gives you a solution that is robust, that is generalizable and so on.

In the work that I do, the philosophical implications of deep learning and the kind of things that I think we should worry about, or maybe sometimes not worry about, are always triggered by the fact that we often use imaging as a tool to look at things that we otherwise can't look at. In medical imaging, we often look at people because we don't want to cut them open. In scientific imaging, we look at things that are often so tiny that we can't see them with the naked eye. In these settings, it's really important that we can trust in the result of the image reconstruction algorithm – image denoising, image deconvolution, image inpainting algorithm.

The key insight here is maybe to think a little bit about, in addition to the really impressive results that neural networks can achieve, what the limitations are. I always like to think about this by putting the two side by side. Deep learning can give you really accurate, very impressive results in detection and images, segmentation, also in image denoising and so on, but usually what you train it with is what you get as a result. If you train it only on images of cats, it will also only work on images of cats.

The exciting thing is to bring them together. I think this is where the field of mathematical imaging as a whole is moving more and more, putting together the best of both worlds – getting the accuracy from neural networks, and the tremendous capability that neural networks have to characterize information content in imaging data, together with the principled way of mathematical equations-driven approaches, explainability and so on. Looking at this from the point of view of medical and scientific imaging, we may not have so much training data available, like in scientific imaging, where we often use imaging as a tool to explore things we have never seen before.

A key insight in mathematical imaging, and why mathematical imaging people are thinking about these different methodologies to, for instance, reconstruct a computer tomography image from measurements that the machine has taken, is that there are various different reconstructions, infinitely many reconstructions, that fit the measurements in the same way. The art is to pick out the one that is most appropriate for what you want to do later, that is maybe most accurate in highlighting what is hidden in this image.

Before deep learning came in, we tried to model this prior information in terms of mathematical equations. What are the characteristic features in these abdominal scans? What is really important?

There are scenarios and settings where pure deep learning approaches can go wrong, in particular if we are in settings where we rely too much on this highly accurate prior information, in settings where the data is very limited, where we have much less data than we would need to reconstruct an image from it confidently. One technique that comes to mind, used in the biomedical domain, is limited angle tomography, where you have a whole viewpoint of the object you're looking at that you haven't measured. In these settings where you have huge gaps, like in the inpainting example, when we think about image inpainting for images (restoration of images), if we think of this in the transform in the medical domain, if we have large gaps in the data, this is where deep learning can go wrong – if it takes too much of the prior information into account and doesn't respect the information content in the data. It might make up things that are not there. If you look at this example that we have in the bottom left image and compare that with the ground truth, you see that it has totally made up things like bones that shouldn't be there. The whole shape of the body is different, so this is where you see cats where there are no cats.

If you try to pin down or formalize what deep learning is, it's basically optimizing – minimizing a loss function, a mathematical function that is fed with training data, under the constraint that what you're optimizing are parameters in a neural network that has a certain architecture. It is already a mathematical problem. It's a constrained optimization problem where the constraint is that you're optimizing parameters in a neural network with a certain architecture and so on. It lends itself to mathematical analysis. The problem is that it's super complex, very high dimensional, and the deepness of neural networks – where you have nested layers in which one operation is nested within another one and within another one – is what makes it such a complex mathematical problem.

In the context of imaging in particular, I think the main way forward and the main area where mathematical imaging people are at the moment influencing the field is structure-preserving deep learning: imposing constraints on neural networks that might come in terms of equations or symmetries of the solutions that you expect, constraints that help you impose certain properties of the solution that you know the solution should have, or that help you to impose adversarial stability, making the deep learning process more robust.