Hybrid Event–Frame Sensors: Modeling, Calibration, and Simulation
1AI Thrust HKUST(GZ) · 2Robotics and Perception Group, University of Zurich · 3CUHK · 4AlpsenTek
Abstract
Hybrid event–frame sensors place an APS and an EVS on the same chip, combining high dynamic range and low latency events with dense intensity frames. However, the mixed circuit introduces complex, poorly modeled noise in both modalities.
We propose a unified statistical noise model that jointly describes APS and EVS, covering photon shot, dark current, fixed-pattern and quantization noise, and linking event noise to illumination. Based on this model, we build a calibration pipeline on real hybrid sensors and introduce H-ESIM, a simulator that generates RAW frames and events with calibrated noise.
Experiments on two industrial hybrid sensors show that our model fits measured noise well and that networks trained or fine-tuned on H-ESIM generalize to real data for video interpolation and deblurring.
Method
Unified Noise Model
APS and EVS share the same latent signal \(I_c(t)\). APS integrates \(I_c\) over exposure to produce RAW intensity, while EVS reads log-intensity differences and fires events when the change exceeds a threshold.
- APS output is modeled as \(I_c\) plus illumination-, exposure- and fixed-dependent noise.
- EVS voltage differences are decomposed into signal and Gaussian noise terms.
- ON/OFF event probabilities are expressed with the Q-function of the signal-to-noise ratio.
Calibration from Real Sensors
We collect dark and static illuminated scenes under varied exposure times on two hybrid sensors (GEN2 and Eiger). APS calibration recovers clean intensity and per-position variance; EVS calibration uses APS intensity as a proxy for brightness to fit an intensity-dependent noise model.
H-ESIM Simulator
Given high-frame-rate video, H-ESIM synthesizes noisy RAW frames and events. APS synthesis uses an sRGB→RAW inverse pipeline plus calibrated noise and quantization. EVS synthesis maps intensity to voltage and uses calibrated thresholds and noise parameters to sample ON/OFF events.
Results
Noise Calibration
APS calibration shows that hybrid layouts amplify row and fixed-pattern noise and that variance depends on brightness, exposure and CFA position. A second-order variance model matches measured noise across GEN2 and Eiger.
EVS calibration confirms that noise event probability grows with brightness and follows the Q-function prediction, with polarity asymmetry explained by threshold–brightness coupling.
Simulation Quality
From high-speed RGB videos, H-ESIM generates RAW frames under different exposures and matching events, with events aligning to motion edges and staying sparse in static regions.
Downstream Tasks
We evaluate temporal and spatial tasks on real hybrid data: video frame interpolation (TimeLens, CBMNet, TimeLens-XL, HR-INR) and RAW deblurring (eSL, MADE, EFNet). Models fine-tuned on H-ESIM show sharper details and better perceptual scores.
Citation
If you find this work useful, please cite:
@article{lu2025hybrid,
title = {Hybrid Event--Frame Sensors: Modeling, Calibration, and Simulation},
author = {Yunfan Lu and Nico Messikommer and Xiaogang Xu and Liming Chen
and Yuhan Chen and Nikola Zubic and Davide Scaramuzza and Hui Xiong},
note = {Preprint},
year = {2025}
}