Self-driving vehicles: What are they and how do they work?

I wrote this paper for a seminar at my university and I thought I should share it with the world since I really put a lot of work into it and the outcome isn't that bad. As a bonus, it even fits the general theme of the blog, self-driving vehicles are still mobile "gadgets" - only far more complex than laptops, phones or bicycles. 

Introduction

Despite popular belief, the idea behind self-driving (autonomous) vehicles has been around for quite some time, although in different forms. People just like to see themselves in the driver’s seat of their car, while they do anything else but pay attention to the road, waiting to reach their destination. But this was not always the case. Some 50 years ago, a car was considered autonomous if it only could keep its lane or match the speed of the vehicle in front of it. People today even like to brag about their new ’self-driving car’ that can park itself. This general confusion or lack of knowledge sparked the need for a unified definition of an autonomous vehicle, a way to tell for sure if a vehicle is indeed ’self-driving’ or if it only has some cool features. Blyth et al elaborate: ”It is essential to clarify the current technological definitions of self-driving vehicle technology. The self-driving vehicle is an arrangement of technologies whose aim is to remove some or all of a driver’s actions and interventions. Consequently, self-driving vehicles are expected to navigate their environment with little or no driver input” [5]. The solution comes from the Society of Automotive Engineers (short SAE) in form of a standard that means to show the clear line between self-driving vehicles and normal ones. This standard presents itself as a table which is attached below as Figure 1. Perhaps the most intriguing aspect of autonomous vehicles is the way they manage to navigate around. ”Driverless cars are essentially autonomous decision-making systems that process a stream of observations from onboard sensors such as radars, LIDARs, cameras, GPS/INS units, and odometry. These observations, together with prior knowledge about the road network, rules of the road, vehicle dynamics, and sensor models, are used to automatically select values for controlled variables governing the vehicle's motion” [12]. Although a lot of progress has been made in developing new technologies to help engineer such vehicles in recent years, there is still a lot of work to be done before we will be able to purchase and use them.

Putting the main concerns against the advantages

Of course, it would be uninspired not to also debate on the advantages and disadvantages of self-driving vehicles. Let’s start with the advantages. The World Economic Forum estimates that the transition from normal (Levels 0-2 in Figure 1) vehicles to autonomous ones could generate more than $60 billion for the automotive industry and even more money from fields such as public or cargo/ delivery transportation and ride-sharing services (for example, a study at the University of Texas found that ”each autonomous vehicle can replace about11 conventional vehicles” [7]). Moreover, cars are parked about 95% of their lifetime [9]. Introducing a wide-spread car-sharing program will reduce ownership costs (humans would only have to ’call’ their ride and pay for it only as long as they use it - this would also allow everyone to use new, eco-friendly models, thus reducing pollution) and solve the parking problem that almost all cities have, as well as eliminate traffic congestion. By mitigating traffic congestion, people would spend less time stuck in traffic and more time with their families, in some cases at work, further augmenting the social and economic benefits. They could drastically reduce the number of fatal car crashes thus improving safety. Automated vehicles can detect and react in dangerous situations much faster than humans and are not distracted as easily by conversations, phone calls or fatigue. Self-driving vehicles could also help to mitigate traffic congestion that is plaguing most major cities around the globe by operating in conjunction with a ’Smart Traffic Monitoring System’ to optimize the traffic flow. This leads us the next benefit: reducing air pollution. Not only can a computer-operated vehicle improve fuel economy, but by allowing the traffic to flow, pollution levels will sink (a 2016 study shows that ”pollution levels inside cars at red lights or in traffic jams are up to 40% higher than when the traffic is moving” [15] ). Finally, having more people sharing the same car to get to different destinations should reduce the number of cars on the road at the given time. This benefits the factors mentioned above: fewer cars reduce traffic congestion, have a beneficial impact on the environment and lower the chance of accidents.

All of the benefits mentioned above are quite convincing if they are not set side-by-side with the issues and challenges posed by autonomous vehicles. Vehicles, in general, rely on data to function and the manufacturers rely on the data the vehicles send over to develop new features and technologies. Despite multiple data protection and transparency laws, some manufacturers do not fully disclose the full extent of the data they are collecting, for what purposes they are using it and how secure it is with them. Strong regulation needs to be implemented here to protect the consumers and allow them to opt-out of these data collection schemes.

Security is another main concern since autonomous vehicles rely on ’Vehicle-to-Vehicle’ (V2V), ’Vehicle-to-Infrastructure’ (V2I) communication and on the locally stored data (the algorithm and services framework and the user data) in order to function properly. Potential hackers could tamper with and compromise the software or communication to cause potentially deadly events (as an experiment conducted in 2015 on a Jeep Cherokee demonstrates: hackers ”can target Jeep Cherokees and give the attacker wireless control, via the Internet, to any of thousands of vehicles. Their code is [...] software that lets hackers send commands through the Jeeps entertainment system to its dashboard functions, steering, brakes, and transmission, all from a laptop” https://www.wired.com/2015/07/hackers-remotely-kill-jeep-highway/). Governments need to enforce a very strong policy that pushes manufacturers to monitor their vehicles for potential vulnerabilities, implement fail-safe systems and release constant software security patches in a timely manner.

Autonomous vehicles could also impact the commercial transport. The freight industry could save billions of dollars by employing self-driving vehicles instead of truck drivers. But this would only spark social and ethical problems similar to those from the late 18th century during the industrial revolution [10]. furthermore, ”self-driving vehicles may change the political economy of employment in sectors such as delivery where higher speeds for delivery vehicles could enable an increase in the distance travelled by goods purely for profit. As a result, retailers could have increased choice in terms of potential suppliers. This could result in more competition thus driving down prices paid, for example, to local farmers”[5], making the aforementioned problem even worse. Further aggravating the social issue is the idea that autonomous vehicles could be shared between people, similarly to how current car-sharing schemes work. Depending on the pricing of this program, two things could happen. On the one hand, if the pricing is low, then people would use this commodity for more and more of their daily routines (it is, after all, more comfortable to drive to work than to travel by public transit or by bike). If manufacturers see the rising demand and act (like they are compelled to - following the economic principle) to increase the supply, we could see more cars on the streets instead of less, lowering the praised positive ecological impact. In this situation, cities will also be built less and less with pedestrians and cyclists in mind, but with large streets and boulevards to accommodate the high amounts of cars. On the other hand, if travelling by autonomous vehicle will be made a premium, society will end up getting divided between those who can afford this and those who have to rely on public transit or their bikes and feet in what would become a very similar situation to the introduction of the first mass-produced automobiles a century ago [5].

Current road infrastructure has been designed with human drivers in mind humans should be able to easily read traffic signs quickly (there are of course exceptions here), but there is still a lot of progress to be made in AI vision before any program will be able to match and surpass human image recognition. Changing the traffic sign layout and design could help solve this problem, but it would require new standards to be agreed upon by international commissions and massive investments in the infrastructure (development and replacement costs need to be considered here), possibly resulting in fund-cuts in other important segments. Not only that, but most companies envision self-driving vehicles as battery-powered, with an electric propulsion system. Since electric vehicles have a lower autonomy than ones powered by gas, governments and car manufacturers would have to invest immense sums of money to create charging standards (a unified standard would be the best, that way buyers would not be tied to a single type of station), charging stations and perhaps even wireless charging lanes. At the same time, self-driving vehicles need to always be connected to the infrastructure and with each other. Although there already are wireless communication standards that are specially designed for this task, the issue of setting up access points on roads still exists. On top of that, if manufacturers decide to use something like road-side unit localization schemes (see the ”Route Planning” section of this paper) instead of GPS (which they are likely to), it would mean that governments would have to invest insane sums of money to build these hubs on every road every couple of hundreds of meters.

Perhaps the most argued upon are the ethical concerns. Most public debates often only discuss the so-called ”Trolley-Problem” (if a car is about to hit (and kill) a person but could try to avoid and hit a wall while killing the passengers, what would (should) it choose?). Given the fact that the underlying algorithms of self-driving cars are based upon machine learning, we need to take into consideration the way the AI is ’thought’ how to drive in order to avoid biases of all sorts. The AI of a car should not, for example, choose to run over a pedestrian based on skin colour or other factors that have little relevance in this situation. Researchers still need to work on ways to avoid biases while still feeding the AI the data it needs in order to learn how to do its job.

This leads to the issue of liability in case of an accident. Although it is predicted that autonomous vehicles will drastically lower the number of fatal crashes every year, they are far from being that good at actually driving and accidents will happen until their development is ’complete’. Real-world testing is a very important part of the development process; however, should an accident happen resulting in a (very expensive) lawsuit against a company, that company could cease development, recall the vehicles and (in the worst case) the lawsuit would deem autonomous vehicles as ”unreasonably dangerous” or say that they have always had a ”defective condition” practically suddenly ending any research efforts. Then again, not all design concepts exclude the human driver. So one could make the argument that the human driver is responsible for any incident, even if in reality the human did nothing wrong, ”putting him at a significant disadvantage” [5]. According to N. A. Greenblatt [9], this could be solved if we started treating AIs who caused accidents just like humans: ”Instead of applying design-defect laws to computer drivers, use ordinary negligence laws. That is, a computer driver should be held liable only if a human driver who took the same actions in the same circumstances would be held liable. The circumstances include the position and velocity of the vehicles, weather conditions, and so on. The ’mind’ of the computer driver need not be examined any more than a human’s mind should be. The robot-driver’s private ’thoughts’ (in the form of computer code) need not be parsed. Only its conduct need be considered” [9].

Last but not least, there is the issue of the propulsion system. Most manufacturers are developing vehicles based on battery-electric propulsion, a propulsion system that has not proven to be reliable in the long term. Batteries have limited life-spans (they can only be charged and discharged a limited number of times, as the chemical reaction that generates electricity wears out the inner reactants to the point where they cannot be used at all anymore), the materials they are made of are toxic metals and recycling them is difficult. Replacement is not cheap, nor is manufacture easy. To sum up: the proud owners of electric cars will either have to send their car in for battery replacement every 2 or 3 years and pay a large sum of money for this (battery failures are as of now excluded from any manufacturer warranty) or replace the vehicle altogether, increasing his carbon footprint.

Route Planning

According to Paden et al. ([12]), route planning sits at the ”highest level” of an autonomous vehicle’s ”decision-making system”. The vehicle needs to be able to generate a valid and efficient route from its position to its destination. Finding the best possible route to a certain destination can be solved by representing the road as a ”graph with edge weights corresponding to the cost of traversing a road segment” [12](also see Figure 2). Algorithms that can solve this problem already exist, however they may not be efficient enough as navigating in congested urban areas is subject to a high number of variables that could change the ’weight’ of a given segment as the vehicle is still traversing it. Clever uses of V2I and V2V systems can help in overcoming this obstacle.

At the same time, a navigating vehicle needs always know its precise location, so it can determine the next steps for reaching the destination. Standard GPS tracking solutions are too inaccurate for this task. According to the US Government Information about the Global Positioning System (https://www.gps.gov/systems/gps/performance/accuracy/), ”GPS systems are typically accurate within a 4.9 m [which is about twice as large as a standard highway lane anyway] radius under the open sky. However, their accuracy worsens near buildings and trees” and the interference generated by a large number of cars using the same satellites for triangulation in a small area (if the vehicles would be in a very congested area) renders GPS tracking useless in this scenario. Maps would also need to be updated very often and be centimeter-accurate and be kept both locally (in the vehicle’s local storage) and in an open cloud-like system, thus requiring cars to always be connected to a network. Areas that are impossible to be reached by radio waves would be rendered un-driveable. A new approach called ’Roadside Unit-aided Localization Scheme’ - RSUL - aims to solve these problems. Roadside-Units (RSUs) are like access points to the internet for cars. They use V2V and V2I for communication [2]. In a nutshell, the RSUs would be placed at fixed, rather short intervals on both sides of the road and would work similarly to GPS to triangulate the position of all vehicles equipped with a beacon-like radio antenna within a reduced area, then broadcast the positioning data to all vehicles within said area, ”the geometry of single lanes as well as a topology, i.e. their interconnectedness within the (road) network”[17]. Of course, other information, like speed-limits, traffic sign information etc. could be transmitted as well, reducing the need for traffic signs and mitigating the need for traffic sign recognition algorithms. By being in such proximity to the vehicles and by relying on either the 802.11p or (and?) 5G short-range wireless standards, communication is fast, reliable and can be used even in places where GPS signals are poor [2].

Behavioural Decision-Making & Collision Avoidance

Paden et al’s second layer is called the ”behavioural decision-making” [12] level. Basically, the computer is responsible for taking all decisions that help the vehicle reach its destination. This includes selecting the appropriate actions and reactions while following all traffic rules (follow any traffic signs, traffic lights, road markings), avoiding any collisions or accidents by looking for, assessing and taking action against potential dangers (like a pedestrian crossing the street when and where they are not supposed to or (motor-)bikes cutting lanes, even other vehicles suddenly taking sudden evasive action against obstacles like potholes, etc.). Systems that recognize and predict the potential trajectories or actions of all other participants in traffic (including, but not limited to: pedestrians, cyclists, scooters, motorbikes, cars, vans, trucks, buses and so on) are needed. High-level data-fusion methods augmented by machine learning algorithms and neural networks (especially Convolutional Neural Networks - CNNs - and subsequent versions) have proven most effective so far. Using the right hardware among the right software are key here. ”Behavior prediction of human drivers has widely been investigated. [...] Human driver prediction within the vehicle for which the safety assessment is performed can be conducted with the help of learning mechanisms such as neural networks. The same approaches can be applied to the traffic scenes that are observed from a fixed location, such as intersections, which allows typical behaviour patterns of the specific scene to be learned” [3]. The situation assessment is divided into two categories: the deterministic and the probabilistic approach. The deterministic approach computes various threat measures, such as time to collision, time to brake, deceleration to safety time, predicted minimum distance and so on. The problem with the deterministic approach is that it does not consider the unexpected events that could occur during the driving task. The probabilistic method has been used to account for those uncertainties (Bayesian networks [16], fuzzy logic and the Markov process [3]) [11]. Every self-driving vehicle would, therefore, need a safety-check loop that constantly checks using (preferably) both deterministic and probabilistic approaches if the vehicle’s trajectory intersects with any other objects or with the trajectories of other vehicles. Figure 3 depicts how this system could look.

Current approaches use a full array of different radars, ultrasonic sensors and LiDARs (light detection and ranging systems). However, they are still far from perfect and have multiple issues, like the ”slow scanning speed and the low” [14] depth. Another issue is the interference between the same types of sensors. This comes to light especially when there are multiple cars moving closer together in the same direction [14]. Camera systems, on the other hand, are less expensive and are crucial to identifying objects, road lanes or traffic signs, even if their performance is highly dependent on the weather conditions and image processing, in general, requires more computational power [14]. At the other end of this behavioural layer, CNNs are being used to extract and classify data from still images, but with the right amount of computing power, data can be extracted even from videos (as long as the processor can keep up with the frame rate). A basic diagram depicting how this works can be found in Figure 4. CNNs need to be trained using complex machine learning algorithms and need to be fed with huge amounts of data to perform well. Gathering all this data could prove more problematic than initially thought as testing autonomous vehicles is not legal in the majority of urban areas as well as in most rural areas, but companies like Tesla have overcome this by equipping all of their cars with all the necessary hardware for up to level 4 autonomous driving, but limits the feature to level 3 while the sensors gather the data required for analysing human behaviour in different driving scenarios.

As defined by the SAE J3016 document, level 4 cars still need the driver to intervene when the vehicle encounters a situation it cannot handle (yet). This means that the system needs to be able to first analyse every single situation it finds itself in, determine if it can take the necessary actions (to avoid unpleasant situations such as collisions) and if it can not or has not been trained to handle that situation to first warn the driver and if he does not respond within a very short amount of time to safely pull over, pause the driving task and wait for driver input - all of this without disrupting traffic or causing any collisions. Figure 5 aims to help the understanding of this. Making this matter more complicated, level 5 vehicles would not even be able to request driver intervention, as most concepts describe a design that lack any external, passenger accessible, drive controls (like a steering wheel, throttle, brake and clutch pedals). These vehicles would need to have the computational power that allows them to make the best decision even in unknown circumstances without putting anyone (or anything) in danger.

Motion Planning and Vehicle Control

In Paden et al.’s model [12], motion planning and vehicle control are the bottom-most layers in an autonomous car’s design. Both layers heavily rely upon the Advanced Driver Assistance Systems that are explained below. At the roots of every autonomous car, bus or truck lie the Advanced Driving Assistance Systems (short ADAS). ADAS is ”an umbrella term for a collection of systems which are developed for driver convenience, safety, traffic assistance as well as multi-purpose lateral and longitudinal control systems” [18]. These systems have existed for quite some time and they are pretty well optimized and stable. In the SAE J3016 classification, every single automation level starting from 1 has some of these features implemented. Figure 6 shows a collection of examples of these systems. Most of these systems have existed since the 1970s (”with the introduction of the electronic anti-lock brakes” [8] - today this system is referred to as ABS), while others are still in development. Nevertheless, these systems need to work together for a vehicle to be truly autonomous. Driver assistance systems help the driver in different situations and perform certain manoeuvres. They also can compensate for the imperfections or shortfalls of a person’s driving by displaying relevant information for the driver. Therefore they exist to increase both comfort and safety of the passengers. These systems need information about the environment, vehicle and driver to function [16]. Vehicle Platooning is a great example of a system that would be very important for self-driving vehicles and is an implementation of a multitude of ADAS systems (like ACC, motion controls) as well as of the V2V communication scheme: A few autonomous vehicles drive close to each other. The leading vehicle would transmit information about the road conditions ahead to the other vehicles in that platoon. Sharing information about the driving environment would provide a worthwhile solution to traffic congestion and in some cases even pollution [4]. The ADASs are key elements of a multi-layered behavioural plan [12] upon which self-driving cars need to rely. ”Collision avoidance system is a key component in self-driving vehicles and obstacle detection is one of the main tasks of this system” [14].

When the behavioural layer decides that a certain action (ex. turning at an intersection, changing the lanes, overtaking, etc.) is required, a lower-level system needs to translate that action into a realistic trajectory that needs to be possible for the vehicle (the system needs to know what that exact model of vehicle is capable of, using the same exact system for many different vehicles should be therefore avoided), comfortable for the passengers and safe to execute (the system should not collide with another traffic participant while avoiding another collision). After that, the vehicle control level executes that manoeuvre and sends feedback back to the behavioural decision-making layer [12].

Conclusion

This paper defined the notion of a self-driving vehicle, then discussed the advantages and disadvantages of self-driving vehicles and outlined the main concepts behind the way they work: Self-driving vehicles operate on multiple layers. The first one is the route-planning, followed by decision-making and collision avoidance and motion planning and vehicle control. Autonomous vehicles are still prone to errors, which is why more research into the complex algorithms behind them is required as well as more real-life testing. The current design splits the vehicle’s software framework into 5 layers [12] with different functionality that work together to make navigation safe and reliable. ”Designing a car is only a small part of the” ’autonomous revolution’. Because if they prove to be successful (as promised), their widespread introduction and commercialization will lead, by a domino effect, to new legislation frameworks, infrastructure (roads with RSUs and wireless charging lanes for EVs), transportation conventions (signs) - in fact to a new lifestyle. The new possibilities also have a powerful socio-economic, ecological and cultural impact [13] while facing resistance from people who are concerned that they will lose their jobs because of the self-driving vehicles or who are too afraid to put a computer basically in control of their lives. Cars will indeed transform: the driver and the driving UX will no longer be in focus - he will no longer own the car per s´e - the cars will instead belong to the network and the focus will shift on the passengers and on their comfort.

Nevertheless, self-driving cars are still at least a few years from becoming a reality, that gives carmakers, governments, and the population enough time to iron out any difficulties.

References

• [1] Automated driving: levels of driving automation are defined in new sae international standard j3016. http://www.sae.org/misc/pdfs/automated_ driving.pdf.
• [2] M. Ali, H. Artail, and Y. Nasser. A cooperative and roadside unit-aided localization scheme for vehicular ad hoc networks. In 2017 International Conference on Electrical and Computing Technologies and Applications (ICECTA), pages 1–5, Nov 2017.
• [3] M. Althoff, O. Stursberg, and M. Buss. Model-based probabilistic collision detection in autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 10(2):299–310, June 2009.
• [4] C. Bergenhem, S. Shaldover, E. Coelingh, C. Englund, and S. Tsugawa. Overview of platooning systems. In Proceedings of the 19th ITS World Congress, 22-26. October 2012.
• [5] P. Blyth, M. Mladenovic, B. Nardi, H. Ekbia, and N. Su. Expanding the design horizon for self-driving vehicles: Distributing benefits and burdens. IEEE Technology and Society Magazine, 35(3):44–49, Sept 2016.
• [6] J. Borenstein, J. Herkert, and K. Miller. Self-driving cars: Ethical responsibilities of design engineers. IEEE Technology and Society Magazine, 36(2):67–75, June 2017.
• [7] D. Fagnant and K. Kockelman. The travel and environmental implications of shared autonomous vehicles using agent-based model scenarios. In 93rd annual meeting of the Transportation Research Board in Washington, D.C., January 2014.
• [8] N. Goodall. Can you program ethics into a self-driving car? IEEE Spectrum, 53(6):28–58, June 2016.
• [9] N. Greenblatt. Self-driving cars and the law. IEEE Spectrum, 53(2):46–51, Feb 2016.
• [10] M. J. Hanna and S. C. Kimmel. Current us federal policy framework for self-driving vehicles: Opportunities and challenges. Computer, 50(12):32–40, December 2017.
• [11] S. Noh, K. An, and W. Han. High-level data fusion based probabilistic situation assessment for highly automated driving. In 2015 IEEE 18th International Conference on Intelligent Transportation Systems, pages 1587–1594, Sept 2015.
• [12] B. Paden, M. p, S. Z. Yong, D. Yershov, and E. Frazzoli. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on Intelligent Vehicles, 1(1):33–55, March 2016.
• [13] C. Pozna and C. Antonya. Issues about autonomous cars. In 2016 IEEE 11th International Symposium on Applied Computational Intelligence and Informatics (SACI), pages 13–18, May 2016.
• [14] G. Prabhakar, B. Kailath, S. Natarajan, and R. Kumar. Obstacle detection and classification using deep learning for tracking in high-speed autonomous driving. In 2017 IEEE Region 10 Symposium (TENSYMP), pages 1–6, July 2017.
• [15] T. Schlossberg. Stuck in traffic, polluting the inside of our cars. New York Times, 29. August 2016.
• [16] J. Schneider, A. Wilde, and K. Naab. Probabilistic approach for modeling and identifying driving situations. In 2008 IEEE Intelligent Vehicles Symposium, pages 343–348, June 2008.
• [17] C. Stiller and J. Ziegler. 3d perception and planning for self-driving and cooperative automobiles. In International Multi-Conference on Systems, Sygnals Devices, pages 1–7, March 2012.
• [18] A. Zakir, P. Konstantin, Z. Hairi, G. Djahid, A. Rahman, and M. Azizi. Current collision mitigation technologies for advanced driver assistance systems a survey. 6:78–90, 12 2016.