How do you make a vehicle drive itself? What challenges do you have to solve to make a Level 5 autonomous vehicle? In the Part 1 of this series, we examined the potential impact of ubiquitous vehicle autonomy in society. In this post, let's look at both the problems that need to be solved to put self-driving cars on the roads, and various approaches to solving those problems.

These issues fall loosely into two categories: architectural and social (otherwise known as functional and non-functional requirements). Architectural challenges deal with the requirements to deploy a self-driving vehicle such as understanding and navigating our road systems, responding to road conditions in real-time both predictable conditions such as traffic signs and lanes as well as unpredictable events; like jay-walkers, debris and human drivers.

Social challenges deal with integrating autonomous vehicles into human society to focus their disruption positively.

Architectural Challenges

Road Systems

Map Courtesy

The map above is an outline of Toronto; or more accurately, the Greater Toronto Area. Roads are shown as yellow or black lines on this map. This map illustrates the first challenge: successfully navigating our road systems. This road system has been developed over a couple centuries that involved a few changes in powered transportation. From horses, to horse-drawn carriages, to the first automobiles, through to today's modern incarnation of the same. There's a lot of legacy built into the city to accommodate old patterns of use and the shifts to new patterns. There's also complexities built in by local government to accommodate the desires of residents; like street parking on some roads, temporary and permanent bike lanes, shifts in lane usage due to construction, road-work or just daily traffic patterns. There is even Jarvis St, where the direction of travel of a lane changes with traffic lights.

Humans have a hard enough time understanding signage (Image courtesy:

The point is our road systems are complex and built for human consumption. Heck, even humans have a hard time understanding road signs sometimes. So the first challenge is to to know your location in the road network and the laws, constraints, traffic signs applicable to that location.

The typical solution is a combination of maps, GPS and/or cellular mapping for finding location. Video cameras, image recognition and AI are combined to undertand traffic signs. The AI component may even include text-recognition and natural language processing to understand unique signs such as signs posted with local speed restrictions or signs indicating road works ahead.


Illustration by Tim Gunther, National Geographic

Closely related to your geographic location is your precise physical location. The vehicle must know your exact location in space to the nearest centimeter. Your real-time precise location is important in knowing which lane you occupy and your proximity to surrounding obstacles like traffic, pedestrians, parked vehicles, debris etc. The best positional resolution that civilian GPS can provide is +/- 3.7 meters. That means any GPS reading places your location somewhere in a circle that is at least 8 meters in diameter. Given that the width of a lane ranges from 2.5 to 3.5 meters, you could be well outside your designated lane and heading into oncoming traffic if you relied solely on GPS. GPS resolution cannot guarantee the positioning of the car in a specific lane; or even on the road for that matter. Its good enough to place you in a general area, so that other sensors can refine your location to greater accuracy.

Other techniques are needed to significantly improve on precision of location. This usually involves merging readings from multiple sensors such as video cameras, LIDAR, and RADAR.

Know Your Environment & Your Place in it

Object Detection using simple single-purpose AI model
LIDAR point cloud (courtesy Alex Kushleyev)

Apart from the vehicles location, you also need to know the precise location, speed and direction of obstacles at that location. This allows the vehicle to build a map of the surrounding environment and project that map into the future to predict the precise location of the environment at that point in time. Humans do that instinctively in everyday life. Think of the coordination involved in catching a ball. Our brains are wired to project observations forwards in time so that we can predict with a high degree of accuracy, where that object (ball) will be at a given point in time so that we can intercept (catch) it. We're not concerned with where it is, but where it will be in the future. Self-driving cars need to be taught that same skill and given measurements that map the environment so that they can navigate it in space and time.

Of course, a real-world environment is not complete without the human element. Imagine a case where a traffic cop is at an intersection directing traffic due to some special circumstance. Humans learn to obey the traffic cop in preference to the traffic signals. Or who hasn't been in traffic at a major intersection with all traffic signals are down. Understanding these rare conditions would require some form of intelligent engine to navigate them. These corner cases may be exceptions, but are a huge part of the difference between a Level 5 and Level 6 autonomous vehicle. Your self-driving car cannot be stalled in traffic or on the side of the road because it doesn't know how to navigate these cases.

Behavior Modeling

We hinted at behavior in the previous section when discussing environmental corner cases. Behavior includes those cases as well as the planned behavior of the autonomous system.

Take for instance driving style. Everyone has their own unique way of driving; from the calm to the wanna-be Le Mans racer. Each driving style comes has a certain level of comfort, or lack thereof. Zooming down the highway passing everything is sight may feel great to the driver (who is in control) but nightmarish to their passengers. Imagine if there's no driver at all and your car behaves like brakes are optional...or worse yet; resevered only for two seconds before disaster strikes. Conversely, if your car is very conservative and stays well below the speed limit in the right lane no matter what, your driving experience would be equally poor. There has to be some balance.

Path and Behaviour Planning in Action

This is where behavior planning comes in. The vehicle has to have the smarts to create an enjoyable driving experience while safely reacting to any incidents in time and obeying the rules of the road. All the input from location, sensors, traffic, maps etc are all processed simultaneously to determine exactly what speed, acceleration (or braking) and steering are needs at each point in time in the near future. There also must be limits on speed to obey the law (except in emergencies where needed) and acceleration or braking to keep passengers comfortable. The planned behavior must also be updated continuously to ensure that the most up-to-date picture of the world is being used.

Putting this all together

Autonomous Vehicle Architecture (courtesy Udacity Self-Driving Car Program)

Putting all these pieces together requires expertise in several areas including:

  • Control Systems - Steering, acceleration, braking
  • Computer Vision - Video and Image processing
  • Mathematics & Statistics - Sensor Fusion, Localization, Trajectory Generation
  • Computer Science - Behaviour Planning, Trajectory Planning
  • Artificial Intelligence - Traffic, Obstacle, Signal Recognition, Behaviour Planning, Situational Awareness

All this makes for a very complex system with huge requirements on sensing storage and processing. In fact Intel estimates that an autonomous vehicle would generate about 4,000 gigabytes of data every day. To put that in perspective, an HD movie download would be less than 4 gigabytes; so that's over 1,000 HD movies generated daily.

The complexity extends beyond the expertise required to build a system and the size of the computing problem. The behaviour modeling presents the biggest challenge. To be able to react to corner cases goes beyond programmed behaviour. The volume of possible corner cases is driven by the variety in human behaviour and environments (including road rules, weather, etc) in which autonomous vehicles can be deployed. The best approach to finding these corner cases is experience; a fact Waymo knows only too well with over 300,000 miles of autonomous driving since 2009.

Supporting Architecture

Just like any other disruptive technology, autonomous vehicles will have far-reaching implications for the way our world functions. Think of how much the internet has changed our way of life. We can communicate and work over vast distances with complete strangers, from the comfort of our homes. We no longer need to visit a bank to make basic transactions and we certainly don't need to visit a mall to purchase necessities such as clothing or food.

Similarly, before fast, self-powered vehicles became widely available, there was no need for driving instruction, licensing requirements or automotive insurance. Today, it is illegal in most countries to drive without an approved government-issued certificate, and a minimum amount of insurance coverage to minimize the impact of an accident on ourselves and any third-parties involved.

A new self-driving paradigm could usher in a slew of changes both intended, required and side-effects. Lets examine some scenarios that could lead to new or changed architecture in the supporting infrastructure for autonomous vehicles.

Autonomous vehicles will invoke new considerations in our social and legal structures. In the event of an accident involving a self-driving car, who is responsible for damage or injury caused? If the car is responsible, then ultimately the car manufacturer is liable. However, what part does the quality of the vehicle maintenance have to play in the ultimate assignment of liability? If the owner of the vehicle is responsible for its up-keep, are they also at least partially liable in the event of an accident if maintenance was sub-standard? This may mandate coverage to protect owners against lapses such as this.

Auto insurance companies will play an important role in the supporting future of supporting architecture. This is due to the potential impact of autonomous vehicles on the nature of their business and their profitability. The first crop of self-driving vehicles will be seen as a significant risk until they prove themselves on the roads in regular traffic; even if these vehicles have been exercised for thousands of miles during testing. This may lead insurers to lobby for regulations that restrict autonomous vehicles; possibly using any of the following:

  • geo-fencing - limiting vehicles to a given locale
  • approved route/time combinations - aka avoiding rush hour or peak times
  • locale certifications - a per-locale vehicle license certifying that the vehicle 'understands' that locale
  • vehicle black boxes - immutable log that tracks every aspect of vehicle operation and navigation

As autonomous vehicles (AVs) become more common-place and prove themselves trustworthy or advantageous, these regulations could shift in favor of AVs. Ubiquity could likely force popular changes to reinforce it. Its entirely possible that future laws will restrict the usage of non-autonomous vehicles; maybe even outlaw them altogether. This change is even more feasible if public infrastructure is changed to accommodate autonomous vehicles. Even if regulations don't reinforce AVs, economic factors such as ubiquitous autonomous public transportation or expense auto insurance premiums for human drivers, could lead to the same result.

What about warranty on self-driving systems or vehicles? All physical systems, including electronics, are subject to wear and tear. This means that manufacturers will warrant their systems for a finite period based on the expected lifetime. Once the system is past warranty but still functional, there will be a temptation to continue using it. In this case the manufacturer is no longer liable so some other form of coverage is required.


Our road systems and signals are built for human drivers. Autonomous driving can be made immediately feasible if the roads and signals are adapted to meet the needs of machine control and navigation. Some possible changes are:

  • Adding a network of beacons to aid in localization
  • Adapting traffic signals for direct communication with autonomous vehicles. Instead of image recognition, the vehicle can know the signal state via direct communication
  • Sharing local traffic signals, road conditions and the state of surrounding traffic via a dedicated vehicle network; possibly via 5G. Vehicles could use this data as the primary reference with on-board sensors as the fallback in case of network congestion or failure.

This is not an insignificant investment, but could speed the adoption of self-driving vehicles; especially in 'legacy' cities with inconsistent layouts, rules and patterns.


In the last installment, we'll examine the current state of autonomous development and adoption.