Add Thesis

Safety of Machine Learning Systems in Autonomous Driving

Written by Fadi Al-Khoury

Paper category

Master Thesis






Thesis: Safety issues when using learning systems for autonomous driving. A pioneering work in this field was [13] in 1989, in which a neural network uses cameras and laser ranging inputs to calculate the steering angle of the vehicle. This method was later called end-to-end learning because the algorithm was not designed by hand. After introducing convolutional neural networks (CNN), LeCun et al. used end-to-end learning in a well-known project. [14], driving a model truck on unknown open terrain while avoiding obstacles. Recently, due to improvements in computing resources and network design with more hidden layers, CNN has made progress in visual tasks. Networks with a large number of layers are called "deep" networks in the literature, and the corresponding phrase "deep learning" is also used. [15] Use deep CNN for end-to-end steering in lane and road following. The main attraction of the end-to-end approach is that domain expertise is not required to understand how the system should solve the problem. Although selecting the appropriate training data set and loss function for the problem requires some expertise, the process of solving the problem is implicitly inferred from the data. Related to the concept of interpretability mentioned earlier, the end-to-end system can be regarded as unexplainable, black boxes, calculations. Considering that the use of visual information to solve the main problem of controlling a vehicle is composed of several logical sub-problems, such as: Lane tracking, scene understanding, vehicle state perception, coordination with other vehicles, action strategies, etc. The main problem solved by the end-to-end logic system is not understood, and the correctness of the solution to each logical sub-problem is not understood. This in itself brings security issues and exacerbates the aforementioned challenge of sufficient training data. To be safe, the training data needs to cover the rare cases of running all the logic subsystems of the end-to-end system. To illustrate this challenge, consider a problem consisting of two sub-problems: PA and PB. Since in the end-to-end system, the solution to the sub-problem is not checked individually, the failure in a sub-problem may be masked in the aggregation system. Consider the fault FA that affects the PA's solution. Due to the masking effect, this particular fault may not always be detected in the aggregation system. Instead of simply asking the training data to consider FA, in the end-to-end system, the data needs to perform FA so that it can be detected in the aggregation system. The rarity of the required data may be much higher than in the case of examining the sub-problems individually. As the complexity of end-to-end system issues increases, it becomes increasingly difficult to mitigate the security issues associated with sufficient training data. 2.3 Security monitoring methods Security monitoring is not a new paradigm, although other terms have also been used to describe this method. An early work in this area was [19] in 1987, which demonstrated a formal method approach that generates supervisors from a system model and a constraint model, both of which are expressed as automata. Supervisors issue restraints to prevent dangerous behavior. An example provided is when two users should not access shared resources at the same time. The goal is to prohibit conversion to meet synchronization requirements. The author provides a formal method to prove that the required supervisor exists and the corresponding comprehensive problem can be solved. An automated tool for verifying the properties of automata models has been developed, called a model checker. The model checker checks the reachability status of the model in detail. If the attribute is true, the model checker confirms it completely, otherwise it provides some diagnostic information. Model checking has been used in the design of safety monitors. Siminiceanu and Ciardo [20] used their SMART model checker to verify the design and expose potential problems with NASA's airport runway safety monitor, which detects intrusions and alerts pilots. With the increasing complexity of system behavior and its environment, more descriptive models are needed and the size of their state space increases. This poses a scalability challenge when using the model checker, because all reachable states need to be checked. This is called the "state space explosion" problem. In [20], the popular NuSMV and SPIN model checkers were found to be unsuitable for runway safety monitoring applications with large state spaces. Another consideration is that embedded systems exhibit both event-driven and time-driven phenomena. In order to represent such a system, a mixed model including discrete variables and continuous variables is used. There are model checkers for hybrid systems, especially [21], but the computational complexity of model-checking hybrid models is greater than that of discrete models, which presents further scalability difficulties. In applications that use implicit discrete-time views and other continuous variables (for example, due to sampling and quantization), it may be appropriate to use discrete models. Run-time verification is another paradigm related to model checking. The safety monitor does not use the model checker to verify the online safety monitor, but online checks the execution of the system. In fact, not all possible system executions are thoroughly checked, but only those encountered during system operation. This provides a fault detection mechanism. A key difference from model checking is that no system model is required. In [3], a runtime verification architecture is proposed for safety-critical embedded systems. Read Less