Automotive Collision Avoidance System Field Operational Test Program
FIRST ANNUAL REPORT

5 FORWARD VISION SENSOR (Task B2)

Objectives

The overall goal of the Forward Vision Sensor Task is to facilitate the development of a robust, real-time forward looking lane tracking system to enhance the overall forward Path Estimation and Target Selection algorithms (Task C2). Additional objectives are to:

  1. Integrate the selected vision system with other subsystems
  2. Support FOT deployment.

Approach

The system will consist of two components. A video camera, mounted behind the windshield of the vehicle, will acquire images of the roadway ahead of the host. A remotely located image processing unit will then detect and track the position of the lane boundaries in the images, and will provide a model of the changing road geometry. In addition to road shape, the lane tracking system will provide estimates of lane width and of the host's heading and lateral position in the lane. In the Data Fusion Module (Task C1) this information will be fused with road and host data from other sources, such as Scene Tracking and GPS Map, to provide more accurate estimates of road and host state to the Target Selection Module.

Although many different vision-based lane detection and tracking systems have been developed worldwide, their primary focus has been on applications such as lane departure warning and lane keeping, where the required range of operation is usually less than 25 meters. Host heading and lateral lane position derived from such systems can be used to reduce the effects of driver hunting and host lane changes on the task of in-path target selection, but the more serious problems associated with curve entry/exit scenarios remain. To address these, an accurate prediction of the roadway geometry up to 100 meters ahead of the host is desired. The goal of this task is to develop a vision-based lane tracking system that will provide these long-range road curvature estimates as well as complement the Scene Tracking and GPS approaches under development in Tracking and Identification Task (Task C2).

To develop the robust vision system required for this program, and to take advantage of existing automotive vision technology, three short-range real-time lane tracking systems were identified as potential starting points for this task. Selection of these systems was based on their developer's demonstrated competency in the development, integration, and road testing of these systems, and on their willingness to extend their system to meet the goals of this program. Teams from the University of Pennsylvania (U-Penn), Ohio State University (OSU), and the University of Michigan – Dearborn (UM-D) were each contracted by DDE1 to further the development of their respective systems. During the first fourteen months of development, DDE is providing technology direction and evaluating the progress of the three competing university teams. Based upon the results of this activity and an official technology down-select process, one approach will be identified for further development and final integration into the FOT vehicles.

Milestones and Deliverables through June 2000

The Lane Tracking System Requirements Document was prepared and delivered to NHTSA and to the university teams. It defined performance and interface requirements, specifying output data content and accuracies, system update rate and latencies, range and realm of operation, and road and marker types.

Work Accomplished

An initial vision kick-off meeting was held in August 1999 in conjunction with the overall Program kick-off. This review was intended to provide the Team and NHTSA an opportunity to formally assess the status of each lane sensing system (i.e., level of performance, capability, maturity) and overall current system design. In an open forum, each university team presented a top level overview of their vision system’s architecture, their performance design requirements (requirements to which their baseline system was originally designed), and a real-time lab demonstration of their current baseline lane tracking system operating on video taped imagery.

Private meetings where also held with each university team to discuss, in detail, their plans for enhancing their system to meet the preliminary system requirements. Each contractor presented a System Analysis Review in which they described anticipated algorithmic changes and challenges, calibration issues, and various vehicle interface requirements (i.e., desired camera features, vehicle sensors, diagnostics). This task has been completed.

One objective was for the vision teams to work with the Team members to define specific performance and interface requirements for the vision subsystem. In general, the performance requirements flow down from the overall FCW system and, specifically, from the needs of the Target Selection Module. The interface requirements have been selected to conform to the overall system design as well as to the specific needs of the Data Fusion Module, which is the primary user of the vision system data.

In summary, the requirements state that the system should provide host and road state estimates to within these specified one-sigma accuracy requirements:

    1. Lateral position in lane: < 0.2 meters
    2. Lane width: < 0.2 meters
    3. Heading: < 0.2°
    4. Road Geometry: < 0.75 meters at 75 meter range2

The Forward Vision Sensor should produce confidence estimates (which may be a function of range) for the road-geometry and host vehicle state. The system should also report the number of lane markers (i.e. left, right or none) that it has acquired as well as some indication of when a lane change event has occurred. The minimum update rate is 10 Hz with an initial maximum acquisition time of 5 seconds. The system should work on the freeways, freeway transitions, expressways and parkways where the minimum horizontal radius of curvature is 300 meters, and when the host speed is between 25 and 75 mph. The system will operate in clement weather, in both day and night conditions, and under natural and artificial lighting. The road surface should be paved, clear, and free from glare, and the road markings should have good contrast. The lane markings can be of single or double lines that are either solid or dashed. This task has been completed.

Research Findings

The bulk of the lane tracking system development falls on the shoulders of the university teams, who continue to develop, test, and enhance their algorithms to meet the specified performance and interface requirements. In support of that work, DDE efforts have concentrated on defining requirements, implementing the video data acquisition system, constructing and coordinating use of the Vision EDV, managing the university teams, and setting priorities and providing technology direction when appropriate. DDE and HRL have been working with the universities to define confidence measures appropriate to each system and which are meaningful to the Data Fusion Task. The following paragraphs describe some of these activities in more detail.

Video Data Acquisition System

To facilitate algorithm development/iteration, system evaluation, and the eventual migration to the final platform, a simple method of collecting video imagery and correlated inertial data was devised. In the FOT FCW system, the vision subsystem will communicate with other subsystems via the CAN bus. Both vehicle speed and yaw rate are available on the bus, as well as the radar scan index, which can be used like a system clock to synchronize data from various sources. A system was designed and implemented to collect this data and store it on the audio track of videotape. Then, on video playback in the laboratory, the audio track is decoded, and converted back to the original CAN messages. Thus, the vision system can be used in the vehicle and on the bench without modification. A diagram of the system is shown in Figure 5.1.

Each vision team was provided with an audio encoder/decoder for their vehicle and laboratory setups. Having adopted this system, encoded videotape can be provided to each vision team for system evaluation. By outputting the scan index with their system's results we are able to compare the different vision system's performance on identical scenarios.

Vision EDV

A Vision EDV was configured as a test bed for the development and evaluation of the lane tracking systems. GM supplied a 1996 Buick which was outfitted by DDE with a CCD-camera, CAN bus, speed and yaw rate sensors, a vehicle interface processor to format and transmit the vehicle data on the CAN bus, and the video encoder system described above. This vehicle was provided for the shared use of all vision teams, and has been driven by each to collect the video scenarios that are currently being used for system refinement and validation. During the down-select process, each of the vision systems can be integrated into the vehicle, and data collected from each simultaneously.

Figure 5.1 System to Collect and Replay Video

Figure 5.1 System to Collect and Replay Video

 

Definition of Confidence Measures

The correct interpretation and degree of reliability of confidence measures generated by the vision subsystem (as well as the GPS and Scene Tracking subsystems) is critical to a successful Fusion task. HRL and DDE have been working with the university teams to define a common output format and confidence measures that will allow the vision system results to be readily compared to those from other vision and non-vision subsystems, and to ground truth data. The resulting Output Format Specification defines the expected road curvature model, common units on all variables, and five levels of confidence in up to four range zones. The university teams have adapted their systems to use this output format.

Algorithm Development

Initial system development has been carried out in the laboratory with data sets collected in the Vision EDV. Since the start of the program, each university team has extended its system to process image data out to the required ranges, and to provide the specified host and road state variables and confidence measures. Efforts continue to improve the system performance on real-world scenarios in which complications such as suspension rock, lane changes, dashed lane markers, freeway exits and intersections, vertical roadbed curvature, and traffic provide challenges to all.

The rest of this section contains a brief progress report from each vision team describing their approach, progress since program inception, and future plans. All three university teams continue to improve the performance of their systems. The milestones for the hardware/software requirements and final performance requirements are complete. The in-lab development and in-vehicle development are progressing according to schedule.

University of Pennsylvania

Approach

At the start of this project our lane tracking system could be broken into three stages: a feature extraction stage which found candidate lane markers in the imagery using a variant of a matched filter, a lane fitting stage which fit straight lines to the extracted features in the near field, and a Kalman filter which combined the resulting line estimates with the inertial measurements obtained from the yaw rate and velocity sensors to produce a final estimate for lane state. The system was capable of estimating the vehicle's orientation and offset with respect to the lane along with the lane width and curvature in the near field.

Improvements

During this project we have extended our approach to incorporate lane tracking in the far field (out to 75m). In order to do this, the lane tracking system has been extended to include two new parameters, the curvature of the road at 100 meters and the pitch of the camera with respect to the road surface. In order to incorporate the expansion in the state vector the current implementation is based on a technique referred to as particle filtering rather than a Kalman filter. In this framework, multiple hypotheses for the lane state are maintained, propagated and scored over time to approximate the evolution of a joint probability distribution over the parameter space. In addition to allowing us to model the effects of far field curvature and pitch, this approach also allows us to characterize better the uncertainty associated with our estimates for these parameters. Two additional techniques have been exploited to improve the convergence of the estimation system. Firstly, a factored sampling approach has been used to split the search for lane parameters into two connected pieces, one concerned with estimating the vehicle’s position, orientation, and pitch, and the lane width and the curvature parameters. Secondly, an importance sampling approach is employed where the results from a Hough Transform analysis are used to bias the hypothesis proposal process.

Summary Results and Future Plans

The current system is able to track highway roads at a rate of 10Hz out to the 75meters. The system has the ability to handle lane changes and partial occlusions of the lane markings. Our plan is to focus our efforts on exploring the tradeoffs associated with the factored sampling and importance sampling schemes and on experimenting with various lane marker extraction methods to improve performance in low light conditions and on concrete road surfaces.

Ohio State University

Approach

The road ahead of the camera/car is modeled as a clothoid. The clothoid parameterizes curvature of the road as a linear function of distance from the camera. Starting with an initial estimate of the curvature (usually zero, which corresponds to a straight road), a search area is defined in the image in which potential lane marker candidates are identified. Once this is done, an optimization scheme using dynamic programming is used to select a final set of lane marker candidates. This is done separately for the left and right lanes. The optimization scheme takes into account the structure of the lane candidates involved, the proximity to the estimated lane position, and the local smoothness of the lane boundary contour that is being constructed. Finally, using the generated left and right lanes, a centerline is constructed taking into account the confidence that the system has in the two lanes. The parameters associated with the centerline are then Kalman filtered to remove any minor variations that might arise, and to make a smooth prediction. This filtered set of parameters forms the estimate of lane centerline for the current image being analyzed. This is projected into the next image frame and the entire process is repeated.

Improvements

  1. Initially, all geometry was confined to the image plane. This means that the models used were for pixel locations and had no correlation to the real world geometry that the camera is in. We have now implemented a perspective-mapping scheme that translates image coordinates of candidate lane markers into real world locations, and only after this are the other processes in the algorithm carried out. The procedure of coordinate translation allows for more realistic modeling of lane boundaries.
  2. A simple quadratic was used to model the lane boundary contours. The clothoid model is a comparatively better way of modeling lane contours, since the clothoid is used in road construction and highway models.
  3. The matched filter used to identify lane marker candidates in the image was of constant dimensions. This means that the same matched filter was used to search for all markers, even though the size of markers in the image diminishes with distance from the camera. We have now modified the matched filter. The scale of the matched filter decreases with geometric distance from the camera, to account for reducing size of lane markers in the image as we move away from the camera.
  4. We now use confidence measures to assess relative validity of the left and right lanes, and to combine the two into a single centerline estimate.

Summary Results and Future Plans

  1. The clothoid based lane marker identification system has been designed and implemented. Performance of the system is good in terms of candidate lane markers identified.
  2. Currently, the left and right lanes are identified separately and combined only at the end of analysis into a single centerline. This does not yet take into account some geometric realities like the fact that the left and right lanes are always parallel. We are investigating the performance of a scheme in which the centerline itself is identified in the dynamic programming procedure, using supporting lane markers on the left and right sides.

University of Michigan

Approach

The Likelihood Of Image Shape (LOIS) Lane Detector, developed by Dr. Karl Kluge of the University of Michigan and Prof. Sridhar Lakshmanan of the University of Michigan-Dearborn, applies a deformable template approach to the problem of estimating lane shape using computer vision. The set of possible lane edges in the image plane consists of a parametric family of curves corresponding to a model in which lane edges are concentric circular arcs on a flat ground plane. A simple matching function (based on how much the image brightness changes near the lane edges) measures how well a particular hypothetical pair of lane edges matches a given input image. A discrete Metropolis optimization method is used to find the pair of lane edges, which maximizes that matching function for each successive image captured by a forward-looking video camera mounted on a car. The parameter estimates from the LOIS lane detector are tracked from frame-to-frame in order to: (a) provide a good initial guess as to where the lanes are in any given frame based on where the lanes were detected in past frames, and (b) signal a lane change.

Improvements

Since the start of the program, U of M's focus has been on:

  1. Improvements to execution speed: As a result of algorithm improvements and porting to a single board computer, we have achieved a four-fold improvement in speed.
  2. Development of estimator confidence measures: A new estimator confidence measure has been developed based on the curvature of the shape/image matching function surface.
  3. Far range distraction problem: LOIS, like other lane detection/tracking systems, has unacceptable lane estimation errors in far ranges. A systematic study as to why this problem occurs, and what measures can be taken to alleviate this problem has been done. This includes changes to the LOIS matching function, use of a better optimization method, and data trimming.
  4. Testing on large data sets: Large lane image data sets were collected using the FOT vehicle. LOIS' performance was tested on these data sets, as well as those provided by DDE.

Summary Results and Future Plans

  1. LOIS currently runs at approximately 8 frames per second on images with 320 (Columns) x 240 (Rows) resolution.
  2. LOIS currently provides an acceptable error rate up to 40m range. Effort is being made to further extend this range of acceptable performance.
  3. LOIS currently provides an off-line estimator confidence measure. Computation of this measure is being incorporated into LOIS, so that it too is real-time.
  4. LOIS' performance on large data sets is currently being determined by visually inspecting the graphical overlay of the detected lanes on the processed image. An effort is being made to determine system performance in a more coherent and repeatable manner.

All three university teams continue to improve the performance of their systems. The milestones for the hardware/software requirements and final performance requirements are complete. The in-lab development and in-vehicle development are progressing according to schedule.

Plans through December 2000

DDE will manage the technology down-select activities in order to identify the vision system that best meets the agreed-upon lane tracking system performance requirements. The selected vision team will then work with the other FOT team members in order to integrate their system with the full portfolio of CW subsystems. The final down select is scheduled for the end of the second year of development. Adhering to this schedule would mean that the Data Fusion and other tasks would have to characterize and design interfaces to all three vision systems until the final down select was completed. To reduce the amount of parallel effort, and more effectively concentrate on the many issues that arise in extended on-road operation, the teams will be subjected to an early down select. This process is scheduled to begin in October 2000, with a meeting in which each vision team will present their work and a bench demonstration of the current system.

DDE and HRL have been working together to formulate a test plan for the lane tracking system down select. As part of this plan, a suite of test scenarios have been defined to evaluate the lane tracking system performance against the specified subsystem requirements. First-round situational videotape was created, and consists of a series of calibration images followed by six driving sequences, each 7–11 minutes in length. The data set exhibits variations in sun angle, traffic densities, road curvature, lane marker quality, and driving patterns (include lane changes and weaving). Vehicle data was encoded on the tape as described above, and for some scenarios, correlated high-accuracy GPS data was collected to aid in determining ground truth.

The vision teams will soon begin processing the first-round tape. They have been asked to provide a system log following the guidelines in the Output Format Specification, and videotape displaying their system's graphical output. The results will initially be compared against ground truth determined using post-processed yaw rate and GPS data. During the next six months, we will complete the technology down-select process and begin focusing on tuning the performance of the selected system.

Figure 5.2 Task B2 Schedule

Figure 5.2 Task B2 Schedule


1UM-D contact: Sridhar Lakshmanan; OSU contact: Umit Ozguner; U-Penn contact: C.J. Taylor.

2Estimates should be such that the error in calculation of lateral displacement of the lane from the host current position should have standard deviation no greater than 0.75 m at any point starting at 15 m and continuing out to a distance of 75 m from the front of the vehicle.


[TITLE PAGE]      [TABLE OF CONTENTS]
[1 Executive Summary]     [2 Introduction]     [3 System Integration]     [4 Forward Radar Sensor]
[5 Forward Vision Sensor]     [6 Brake Control System]     [7 Throttle Control System]
[8 Driver-Vehicle Interface]      [9 Data Fusion]     [10 Tracking & Identification]     [11 CW Function]   
 [12 ACC Function]     [13 Fleet Vehicle Build]      [14 Field Operational Test]
[Appendix A]     [Acronyms]