The EU-project DENSE seeks to eliminate one of the most pressing problems of automated driving: the inability of current systems to sense their surroundings under all weather conditions. Severe weather – such as snow, heavy rain or fog – have long been viewed as one of the last remaining technical challenges preventing self-driving cars from being brought to the market. This can only be overcome by developing a fully reliable environment perception technology.

This homepage provides a variety of projects and datasets that have been developed during the project.

• Gated2Depth: Imaging framework which converts three images from a gated camera into high-resolution depth maps with depth accuracy comparable to pulsed lidar measurements.
• Seeing Through Fog: Novel multi-modal datatset with 12,000 samples under different weather and illumination conditions and 1,500 measurements acquired in a fog chamber. Furthermore, the dataset contains accurate human annotations for objects annotated with 2D and 3D bounding boxes and frame tags  describing the weather, daytime and street condition.
• Gated2Gated: Self-supervised imaging framework, which estimates absolute depth maps from three gated images and uses cycle and temporal consistency as training signals. This work provides an extension of the Seeing Through Fog dataset with synchronized temporal history frames for each keyframe. Each keyframe contains synchronized lidar data and fine granulated scene and object annotations.
• Pixel Accurate Depth Benchmark: Evaluation benchmark for depth estimation and completion under varying weather conditions using high-resolution depth.
• Image Enhancement Benchmark: A benchmark for image enhancement methods.

We present an imaging framework which converts three images from a low-cost CMOS gated camera into high-resolution depth maps with depth accuracy comparable to pulsed lidar measurements. The main contributions are:

• We propose a learning-based approach for estimating dense depth from gated images, without the need for dense depth labels for training.
• We validate the proposed method in simulation and on real-world measurements acquired with a prototype system in challenging automotive scenarios. We show that the method recovers dense depth up to 80m with depth accuracy comparable to scanning lidar.
• Dataset: We provide the first long-range gated dataset, covering over 4,000km driving throughout Northern Europe.

To capture the dataset, we have equipped a test vehicle with sensors covering the visible, mm-wave, NIR, and FIR band, see Figure below. We measure intensity, depth, and weather conditions. For more information, please refer to our dataset paper.

We present a self-supervised imaging framework, which estimates dense depth from a set of three gated images without any ground truth depth for supervision. By predicting scene albedo, depth and ambient illumination, we reconstruct the input gated images and enforce cycle consistency as can be seen in the title picture. In addition, we use view synthesis to introduce temporal consistency between adjacent gated images.
The main contributions are:

• We propose the first self-supervised method for depth estimation from gated images that learns depth prediction without ground truth depth.
• We validate that the proposed method outperforms previous state of the art including fully supervised methods in challenging weather conditions as dense fog or snowfall where lidar data is too cluttered to be used for direct supervision.
• Dataset: We provide the first temporal gated imaging dataset, consisting of around 130,000 samples, captured in clear and adverse weather conditions as an extension of the labeled Seeing Through Fog dataset which provides detailed scene annotations, object labels and corresponding multi modal data.

We model four typical automotive outdoor scenarios from the KITTI dataset. Specifically, we setup the following four realistic scenarios: pedestrian zone, residential area, construction area and highway. In total, this benchmark consists of 10 randomly selected samples of each scenario (day/night) in clear, light rain, heavy rain and 17 visibility levels in fog (20-100 m in 5 m steps), resulting in 1,600 samples in total.

To acquire realistic automotive sensor data, which serves as input for the depth evaluation methods assessed in this work, we equipped a research vehicle with a RGB stereo camera (Aptina AR0230, 1920x1024, 12bit) and a lidar (Velodyne HDL64-S3, 905 nm). All sensors run in a robot operating system (ROS) environment and are time-synchronized by a pulse per second (PPS) signal provided by a proprietary inertial measurement unit (IMU).

Due to the ill-posed problem of collecting data within adverse weather scenarios, especially within fog, most approaches in the field of image de-hazing are based on synthetic datasets and standard metrics that mostly originate from general tasks as image denoising or deblurring. To be able to evaluate the performance of such a system, it is necessary to have real data and an adequate metric. We introduce a novel calibrated benchmark dataset recorded in real, well defined weather conditions. Therefore, we have captured moving calibrated reflectance targets at continuously changing distances within a challenging illumination setting in controlled environment settings.  This enables a direct calculation of contrast in between the respective reflectance targets and an easy interpretable enhancement result in terms of visible contrast.

Summarized our main contributions are:

• We provide a novel benchmark dataset recorded under defined and adjustable real weather conditions.
• We introduce a metric describing the performance of image enhancement methods in a more intuitive and interpretable way.
• We compare a variety of state-of-the-art image enhancement methods and metrics based on our novel benchmark dataset.

To capture the dataset we have placed our research vehicle within a fog chamber and moved 50cmx50cm large diffusive Zenith Polymer Targets along the red line below.  For more information please refer to our paper. The used static scenario is currently part of the Pixel Accurate Depth benchmark.

Above we show example images for clear conditions (left) and foggy images (right) for a fog visibility 30m-40m.