Back in early 2017 ARM announced the Mali-C71, their first homegrown ISP design. The C71 targets the automotive industry. To that end, it incorporates many extra safety-critical features to meet rigorous safety and reliability requirements. Today, ARM is extending the Mali Camera family with two more image signal processors – the Mali-C52 and the Mali-C32, this time for intelligent vision devices at the edge.
The motivation here is simple. ARM is seeing increasing demand for higher image quality largely driven by computer vision. In the next three years, they are predicting the traditional markets (e.g., security surveillance, video conferencing and IP camera) to grow at a CAGR of 20% to 500 million units shipped. Likewise, they see emerging markets (drones, robots, action cameras, and smart assistant devices) growing at a CAGR of 15% to over 150 million units shipments by 2021. With an explosion in the number of smart devices, ARM is seeing a significant growth opportunity, growth that aligns quite nicely with their vision of enabling one trillion connected devices.
Mali-C52 and Mali-C32
Key Differentiators Under the Hood
The two ISPs are built based on a brand new pixel pipeline architecture that processes the pixel through over 25 different algorithms in real time. With the new Mali ISPs, ARM says they are introducing a number of unique differentiators. Those include localized dynamic range compression, HDR precision management, noise management, and color management. Much of this functionality we have first seen when they introduced the Mali-C71.
Local Tone Mapping
One of the examples they provided is a fairly challenging scene. This kind of scene is found everywhere such as a camera looking onto a street on a bright day. This type of scene would typically require 24 bit or so to faithfully try to represent the picture, however with a typical 10-bit or 12-bit ADC, you’re are restricted to substantially less with nowhere nearly as much data to work with. Note that the goal here is to try to match what the human eye actually sees. In term, your final output is also bounded by this lower precision.
To address this, the typical solution is to the take multiple pictures and combine them to obtain more cumulated data. These multiple exposures are then combined by the image signal processor to create a high dynamic range image featuring something that resembles a 24-bit representation. The HDR data output can then undergo dynamic range compression which is accomplished through tone mapping. The most common solution on the market is global tone mapping. Although this solution allows dark parts to become more visible, global tone mapping causes a loss of contrast. This means some details are actually lost in the process. In the example on the slide below, some shadow details are lost and the highlights in the clouds have also been clipped in the process. While many parts of the picture are now visible, others have been lost.
One of the features ARM added to their new ISPs is local tone mapping operations. We have first seen this back when the Mali-C71 was announced, actually. The technology goes under the name ARM Iridix. Here, contextual features are extracted from surrounding pixels. This allows for varying contrasts across the image. Using the example photo again, with local tone mapping the contrast in the clouds are enhanced and the highlights are now visible again. The same is true for the various shadows.
Another feature ARM talked about involving HDR is precision management where particular areas of focus in the picture can be enhanced such as a license plate or human face of a bystander. The technology was first announced with the C71 for autonomous vehicles and now we are seeing it making its way to embedded and edge devices.
Noise Reduction & Color Enhancement
Like the C71, the two new Mali ISPs also inherit the noise reduction capabilities feature it had. Sensor clipping can destroy the hue and tone of some details. Another feature the new Mali ISPs are bringing is 3D color enhancement which can use a 3D lookup table in order to reconstruct some of the clipped areas. Both the Mali-C52 and Mali-C32 implement a 9x9x9 color LUT (look up table) in-hardware in the RGB domain. In real-time, they are capable of performing one-to-one mapping, eliminating the need for various post-processing operations.
Putting the Pieces Together
Ultimately, the new Mali ISPs are one more piece of ARM’s bigger IP ecosystem. For the emerging intelligent devices market such as robots, you’d have the Mali-C52/C32 ISP for processing the imaging input along with a smaller core such as a Cortex-M for processing that various other sensor input. All of these data acquisition circuitries are then fed somewhere which processes it all. ARM sees their machine learning framework the ideal platform for this which includes their NN software libraries that work with their GPUs, Cortex cores, and their recent ML processor.