A Multimode Vision Sensor With Temporal Contrast Pixel and Column-Parallel Local Binary Pattern Extraction for Dynamic Depth Sensing Using Stereo Vision

Min-Yang Chiu; Guan-Cheng Chen; Tzu-Hsiang Hsu; Ren-Shuo Liu; Chung-Chuan Lo; Kea-Tiong Tang; Meng-Fan Chang; Chih-Cheng Hsieh

doi:10.1109/JSSC.2023.3292051

A Multimode Vision Sensor With Temporal Contrast Pixel and Column-Parallel Local Binary Pattern Extraction for Dynamic Depth Sensing Using Stereo Vision

Min-Yang Chiu, Guan-Cheng Chen, Tzu-Hsiang Hsu, Ren-Shuo Liu, Chung-Chuan Lo, Kea-Tiong Tang, Meng-Fan Chang Chih-Cheng Hsieh

IEEE Journal of Solid-State Circuits, Vol.58(10), pp.2676-2677

01/10/2023

: https://doi.org/10.1109/JSSC.2023.3292051

: WOS:001035956700001

Feature extraction

frame difference (FD)

global shutter

local binary pattern (LBP)

Motion detection

Pulse width modulation

region of interest (ROI)

saliency detection

Sensors

Stereo vision

vision sensor

Vision sensors

Voltage

This article presents a 0.56/0.8 V multimode vision sensor for temporal and spatial information extraction with processing in sensor (PIS) technique. The proposed six transistors and one capacitor (6T1C) temporal contrast pixel (TCP) structure using the exposure compensation scheme (ECS) can realize in-pixel temporal contrast calculation and motion event reporting (ER) with global shutter exposure and seamless frame difference (FD) based on the pulsewidth modulation (PWM) operation. In addition, the column-parallel local binary pattern (LBP) extraction provides further spatial feature information without the power-consuming analog-to-digital conversion to meet the power-efficient requirement for always-on-edge devices. According to the temporal and spatial information extracted from FD and LBP modes, this work realizes the PIS operations for dynamic depth calculation of a stereo vision system, which filters out the static scene and provides the depth information of the dynamic objects only. To further alleviate the data transfer bandwidth, the region of interest (ROI) capability is also embedded in the sensor for data windowing and object locating. The prototyped vision sensor was fabricated in TSMC's standard 0.18 μ m CMOS process and verified. The vision sensor provides five operating modes that can adapt to different tasks, which include the raw image mode (IM), FD mode, ER mode, LBP mode, and ROI mode. The measurement results show a maximum frame rate of 540/819/540 frames/s at a power consumption of 390/162.6/151.9 μ W with an achieved iFoM of 45.5/12.5/17.7 pJ/pixel ⋅ frame in IM/ER/LBP mode, respectively.

See more details

: A Multimode Vision Sensor With Temporal Contrast Pixel and Column-Parallel Local Binary Pattern Extraction for Dynamic Depth Sensing Using Stereo Vision
: Min-Yang Chiu - Novatek Microelectronics (Taiwan)
Guan-Cheng Chen - National Tsing Hua University
Tzu-Hsiang Hsu - MediaTek (Taiwan)
Ren-Shuo Liu - National Tsing Hua University, College of Semiconductor Research
Chung-Chuan Lo - National Tsing Hua University, Department of Life Science
Kea-Tiong Tang - National Tsing Hua University, College of Semiconductor Research
Meng-Fan Chang - National Tsing Hua University, College of Semiconductor Research
Chih-Cheng Hsieh - National Tsing Hua University, College of Semiconductor Research
: IEEE
: 11
: MOST111-2218-E-007-009; 111-2221-E-007-113-MY3 / Ministry of Science and Technology (MOST), Taiwan Qualcomm through the Taiwan University Research Collaboration Project Taiwan Semiconductor Research Institute (TSRI) Novatek Microelectronics Corp
: Journal article
: 01/10/2023
: IEEE Journal of Solid-State Circuits, Vol.58(10), pp.2676-2677
: English