Socionext and Osaka University Develop New Deep Learning Method for Object Detection in Low-Light Conditions

Langen/Germany -- August 24, 2020 --- Socionext Inc. and a research group at Osaka University Institute for Datability Science, led by Professor Hajime Nagahara, have jointly developed a new method of deep learning, which enables image recognition and object detection in extremely low-light conditions. By merging multiple models, the new method enables the detection of objects without the generation of huge datasets, a task previously thought to be essential.

Socionext plans to incorporate this new method into the company’s image signal processors to develop new SoCs, as well as new camera systems around such SoCs, for applications including automotive, security, industrial and others that require high-performance image recognition. The research work will be presented at European Conference on Computer Vision (ECCV) 2020, held online from August 23 through 28 (British Summer Time). https://eccv2020.eu/

A major challenge throughout the evolution of computer vision technology, has been to improve the image recognition performance for applications such as in-vehicle cameras and surveillance systems under poor lighting conditions. Previously, a deep learning method using RAW image data from sensors has been developed, called “Learning to See in the Dark” ^[1]. However, this method requires a dataset of more than 200,000 images with more than 1.5 million annotations ^[2] for end-to-end learning. Preparing such a large dataset with RAW images is both costly and time-prohibitive.

Fig.1 Learning to See in the Dark / Challenges for RAW image recognition

The joint research team proposed the new domain adaptation method, which builds a required model using existing datasets by utilizing machine learning techniques such as Transfer Learning and Knowledge Distillation. The new method resolves the challenge through the following steps: (1) building an inference model with existing datasets, (2) extracting knowledge from the aforementioned inference model, (3) merging the models by glue layers, and (4) building generative model by knowledge distillation. It enables the learning of a desired image recognition model using the existing datasets (Fig.2).

Fig.2 Domain Adaptation Method

Using this domain adaptation method, the team has built an object detection model "YOLO in the Dark" using RAW images taken in extreme dark conditions, with the YOLO model ^[3] (Fig. 3). Learning of the object detection model with RAW images can be achieved with the existing dataset, without generating additional datasets. In contrast to the case where the object cannot be detected by brightness enhancement of images with existing YOLO model (a), the proposed new model made it possible to recognize RAW images and detect objects (b). The amount of computing resources needed in this new model is about the half of the baseline model, which uses the combination of previous models (c).

Fig.3 YOLO in the Dark

This "direct recognition of RAW images" by the method is expected to be used for object detection in extremely dark conditions, along with many other applications. Socionext plans to incorporate this new method into the company’s image signal processors to develop new SoCs, as well as new camera systems around such SoCs, and offer leading-edge solutions for applications including automotive, security, industrial and others that require high-performance image recognition.

[European Conference on Computer Vision (ECCV) 2020]

Held online from August 23 to 28 (British Summer Time)

https://eccv2020.eu/

Presentation by Yukihiro Sasagawa, Socionext and Hajime Nagahara, Osaka University

"YOLO in the Dark - Domain Adaptation Method for Merging Multiple Models -"

About Socionext

Socionext is a global, innovative enterprise that designs, develops and delivers System-on-Chip solutions to customers worldwide. The company is focused on technologies that drive todayʼs leadingedge applications in consumer, automotive and industrial markets. Socionext combines world-class expertise, experience, and an extensive IP portfolio to provide exceptional solutions and ensure a better quality of experience for customers. Founded in 2015, Socionext Inc. is headquartered in Yokohama, and has offices in Japan, Asia, United States and Europe to lead its product development and sales activities. For more information, visit www.socionext.com.

Notes:

[1] “Learning to See in the Dark” : CVPR2018, Chen et al.

[2] MS COCO dataset as an example (https://cocodataset.org/)

[3] YOLO (You Only Look Once): One of the deep learning object detection methods