High Resolution Automotive Image Sensors for Higher Levels of Autonomy
More vehicles on the road today are equipped with some level of advanced driver assistance systems (ADAS) or autonomous driving (AD) systems offering autopilot, lane keeping, blind spot alarm and other functions. Many of these functions are enabled by cameras using onsemi automotive image sensors.
www.onsemi.com
In most ADAS and AD system designs, we see many automotive cameras forming a 360-degree sensing cocoon as shown in Figure 1. Narrow and wide field-of-view (FOV) cameras are used to detect various objects including other vehicles and vulnerable road users (VRUs) such as pedestrians, bicyclists and motorbikes. These cameras are also used to detect road lanes, curbs, traffic light colors, and to extract information from traffic signs.
These cameras must be able to capture detailed imagery in different lighting conditions - from sunny days to pitch-dark nights. These lighting conditions are so broad that the captured images often span high dynamic range (HDR) well beyond the capability of human eye. Typically, the human eye is capable of 80 to 90 dB HDR perception, while a daytime street scenery with direct sun often exceeds 140 dB HDR, which is 100 thousand to 1 million times more. In addition, maximum vehicle speed limits dictate the object detection often exceeding 200 m or even 300 m at highway speed of 130 km/h (80 mi/h). Long range detection thus requires images with high resolution. All these challenges define the requirements for the latest automotive image sensors.
onsemi was first to invent 120 dB HDR sensors to enable initial ADAS designs that started to appear between 2009 and 2015. These early ADAS solutions used 1- and 2-megapixel (MP) HDR image sensors from onsemi, which are implemented in the majority of ADAS systems in the cars running on road today.
The European Automobile Manufacturers’ Association (ACEA) reported that the number of road fatalities in the European Union (EU) has halved since 2008 (Figure 2) [Source: Road fatalities and motor vehicles in the EU, 2008-2020 trend]. ADAS systems equipped with HDR cameras were a large contributor to this trend. In constant pursuit of better and safer car designs equipped with ADAS, onsemi was the first to introduce an 8.3 MP automotive-grade sensor to the market.
Figure 2: European Union (EU) Road Fatalities 2008-2020.
The first-generation high-resolution sensor enabled the detection of distant objects. With a 2.1 µm dual-gain pixel, the 8.3 MP AR0820AT sensor was optimized with 140 dB HDR and provided good low-light performance. These traits then became de facto industry standards. The performance of this first high-resolution automotive image sensor enabled object detection at long distances, as shown in Figure 3, and opened a path to increased road safety. One of the sensor operational modes was designed for very low light conditions by employing an in-pixel summing technique that combines four pixels together with very low readout noise. This mode reduced resolution from 8.3 MP to 2 MP but increased very low light signal-to-noise ratios (SNR) more than twice and enabled dark night sub-lux imaging. ADAS often uses on-fly switching from 8.3 MP resolution with full image details to lower 2 MP resolution with much better low light image capture. The active sensor mode switching between these two resolutions is supported by the sensor context switching mechanism.
Figure 3: Long-distance object detection enabled by AR0820AT high-resolution sensor.
One example of the 140 dB HDR imaging being used for the detection of VRUs is shown in Figure 4. The image was captured by AR0820AT in 4-exposure HDR mode. The blind spot camera clearly showed a motorbike and its headlights despite the very bright lighting conditions. Even when the sun was very low, the motorbike would be visible and could be detected. The AR0820AT was also the first automotive image sensor that introduced two non-traditional new color filter arrays (CFA).
Figure 4: Detection of VRUs with AR0820AT 4-exposure HDR mode and RYYCy CFA.
The first CFA included red, yellow, and cyan (RYYCy) pixels. Advanced color processing engines can effectively convert RYYCy raw captures into nice and vivid color images like Figure 4. By allowing more photons to enter the pixel, the RYYCy pattern improved low-light performance. The other CFA included red, clear, and blue (RCCB) pixels with improved low-light performance while providing good color reproduction. Both CFA patterns on the AR0820AT are widely used in modern ADAS and AD systems.
In the effort to support Level 3 and higher levels of autonomy, which enables highway and urban autopilot capabilities in mass-produced vehicles, and continue the development of advanced safety capabilities, onsemi pushed performance even further with the HyperluxTM AR0823AT 8.3 MP sensor. The second-generation super-exposure AR0823AT significantly improved the performance in low light conditions and delivered unmatched 150 dB ultra HDR with LED flicker mitigation (LFM). The sensor captures crisp details within the image across a wide range of lighting conditions, arming system solutions with unparalleled capabilities. Figure 5 shows an image with details of objects inside of a dark tunnel as well as vehicles at ~400 m in the direct sun. The sensor maintains stable performance across the entire range of automotive temperatures and can handle the most demanding corner cases to truly enable level 3 and higher autonomous driving.
Figure 5: Image captured from a tunnel facing direct sun by AR0823AT with 150 dB HDR.
The HDR LFM operation of the sensor is becoming more important as LED-powered lights and signshave entered the automotive space. Traffic lights and signs, headlights, and brake lights use LEDs with pulse width modulation that may cause some missing or illegible information in captured images.
The AR0823AT is one of the first automotive image sensors with in-pixel HDR LFM capability that guarantees capturing bright traffic lights and signs without missing information. The AR0820AT relies on HDR multi-exposure operation but that resulted in LED flicker effects in images when part of LED traffic signs may be missing or degraded. On the other hand, the AR0823AT in super exposure LFM operation covers up to 120 dB flicker-free range, preserving all the information of the LED traffic signs. Figure 6 shows images of a 90 km/h speed limit sign captured by the AR0820AT (left) with some degradation and the AR0823AT (right) with crisp and true details.
Figure 6: LFM performance comparison.
150 dB HDR is one of the most important features that AR0823AT brings to the table. The ultra HDR becomes handy, especially in corner situations with direct sun or sun reflections, as well as detecting traffic light colors. We use examples of direct sun as captured by the AR0823AT and one of the competing solutions. The competing sensor is capable of only 130 dB HDR. The AR0823AT’s range is 10 times better than the competition and allows the ADAS or AD system to clearly capture the signage information. Comparatively, the detecting ability of the competing solution is compromised, as the image (Figure 7) is saturated around the sun. When it comes to reading traffic lights, AR0823AT can capture true colors even in very bright sunny conditions while the other sensor may fail and result in false color detection. We do want our cars to “see” traffic light colors accurately, independent of the positions of light sources like the sun or headlights.
Figure 7: HDR and color performance comparison.
The superior capabilities of AR0823AT, including sharp and true color images, enhanced low light performance and the 150 dB HDR LFM, are the key attributes for long-distance object detection, during sunny days and dark nights. onsemi has been studying the impacts of image quality at the levels of object detection using the latest Hyperlux sensors on test vehicles. Millions of real-life images and videos were collected and processed via the “You Only Look Once” (YOLO) real time object detection engine.
Figure 8 displays some results of object detection enabled by AR0823AT. The left images show a sunny day on a highway with good lighting conditions. In this scenario, cars, traffic signs, lights and others can be reliably detected at up to 400 m. In the nighttime scenario on a dark city street (right image), reliable object detection is up to 250 m. The results of these studies confirmed the readiness of AR0823AT to support level 3 and higher of autonomous driving vehicles at highway speeds up to 130 km/h.
Figure 8: Object detections enabled by AR0823AT images during day and nighttime.
As the automotive industry transitions en masse to level 3 of autonomous driving, all these studies give reassurance that high-resolution sensors from onsemi provide a good foundation for various automotive applications. We foresee that, in the future, the resolution of image sensors will go beyond 8.3 MP in order to achieve even higher levels of autonomy. We work diligently to deliver these future sensors without compromising on performance and cost. With the largest intellectual property portfolio in imaging sensing and unique resilient dual supply manufacturing, onsemi continues supporting development for safer roads and higher levels of autonomy.
www.onsemi.com
