ARM Announces the Mali-C52 and Mali-C32 ISPs for Intelligent Vision Devices

Back in early 2017 ARM announced the Mali-C71, their first homegrown ISP design. The C71 targets the automotive industry. To that end, it incorporates many extra safety-critical features to meet rigorous safety and reliability requirements. Today, ARM is extending the Mali Camera family with two more image signal processors – the Mali-C52 and the Mali-C32, this time for intelligent vision devices at the edge.

The motivation here is simple. ARM is seeing increasing demand for higher image quality largely driven by computer vision. In the next three years, they are predicting the traditional markets (e.g., security surveillance, video conferencing and IP camera) to grow at a CAGR of 20% to 500 million units shipped. Likewise, they see emerging markets (drones, robots, action cameras, and smart assistant devices) growing at a CAGR of 15% to over 150 million units shipments by 2021. With an explosion in the number of smart devices, ARM is seeing a significant growth opportunity, growth that aligns quite nicely with their vision of enabling one trillion connected devices.

Mali-C52 and Mali-C32

Two ISPs are being announced, the Mali-C52 and the Mali-C32. ARM says the IP is already available and has already been licensed to a number of partners. Initial products are expected to come from traditional markets while emerging markets will follow a little after. The difference between the two boils down to the type of optimization you are looking for. It’s a compromise between area, image quality, and performance. The Mali-C32 is a smaller version of the C52. It is optimized for area in lower-power for entry-level cost-sensitive devices. The key emphasis here is that there is a great amount of overlap between the two. Both ISPs target both traditional and emerging markets and both can use the same software. This provides ARM customers with great flexibility in terms of use cases. Both ISPs come with the software drivers and libraries to control the ISP, sensor, including the 3A libraries (auto-white balance, auto-focus, and auto-exposure) and the calibration and tuning tools. Within their constraints, both ISPs push what’s possible as far as pixel quality, resolution and frame rate go. On the 16-nanometer node, they are capable of upward of 600 megapixels/sec with a maximum resolution of 16 MPiexels supported per frame, supporting four exposures for HDR, and can sustain real-time up to 4K resolution at 60 frames per second. Both ISPs support 4 cameras in time division multiplex format (on frame by frame basis). For example, you can have a single camera with a 16-megapixel resolution at 30 FPS or two cameras with 8-megapixel resolution at 4K at 30 FPS.

Key Differentiators Under the Hood

The two ISPs are built based on a brand new pixel pipeline architecture that processes the pixel through over 25 different algorithms in real time. With the new Mali ISPs, ARM says they are introducing a number of unique differentiators. Those include localized dynamic range compression, HDR precision management, noise management, and color management. Much of this functionality we have first seen when they introduced the Mali-C71.

Local Tone Mapping

One of the examples they provided is a fairly challenging scene. This kind of scene is found everywhere such as a camera looking onto a street on a bright day. This type of scene would typically require 24 bit or so to faithfully try to represent the picture, however with a typical 10-bit or 12-bit ADC, you’re are restricted to substantially less with nowhere nearly as much data to work with. Note that the goal here is to try to match what the human eye actually sees. In term, your final output is also bounded by this lower precision.

To address this, the typical solution is to the take multiple pictures and combine them to obtain more cumulated data. These multiple exposures are then combined by the image signal processor to create a high dynamic range image featuring something that resembles a 24-bit representation. The HDR data output can then undergo dynamic range compression which is accomplished through tone mapping. The most common solution on the market is global tone mapping. Although this solution allows dark parts to become more visible, global tone mapping causes a loss of contrast. This means some details are actually lost in the process. In the example on the slide below, some shadow details are lost and the highlights in the clouds have also been clipped in the process. While many parts of the picture are now visible, others have been lost.

One of the features ARM added to their new ISPs is local tone mapping operations. We have first seen this back when the Mali-C71 was announced, actually. The technology goes under the name ARM Iridix. Here, contextual features are extracted from surrounding pixels. This allows for varying contrasts across the image. Using the example photo again, with local tone mapping the contrast in the clouds are enhanced and the highlights are now visible again. The same is true for the various shadows.

Another feature ARM talked about involving HDR is precision management where particular areas of focus in the picture can be enhanced such as a license plate or human face of a bystander. The technology was first announced with the C71 for autonomous vehicles and now we are seeing it making its way to embedded and edge devices.

Noise Reduction & Color Enhancement

Like the C71, the two new Mali ISPs also inherit the noise reduction capabilities feature it had. Sensor clipping can destroy the hue and tone of some details. Another feature the new Mali ISPs are bringing is 3D color enhancement which can use a 3D lookup table in order to reconstruct some of the clipped areas. Both the Mali-C52 and Mali-C32 implement a 9x9x9 color LUT (look up table) in-hardware in the RGB domain. In real-time, they are capable of performing one-to-one mapping, eliminating the need for various post-processing operations.

Putting the Pieces Together

Ultimately, the new Mali ISPs are one more piece of ARM’s bigger IP ecosystem. For the emerging intelligent devices market such as robots, you’d have the Mali-C52/C32 ISP for processing the imaging input along with a smaller core such as a Cortex-M for processing that various other sensor input. All of these data acquisition circuitries are then fed somewhere which processes it all. ARM sees their machine learning framework the ideal platform for this which includes their NN software libraries that work with their GPUs, Cortex cores, and their recent ML processor.