Runtime Analysis with R2U2: A Tool Exhibition Report

With the Dragoneye setup (A) we perform online monitoring directly on-board the Dragoneye UAS during a flight. The communication between the flight computer and the R2U2 framework, running on the Parallella board, uses a read-only UART interface. An operator is responsible for launching, landing, and might have to control the UAS manually via the GCS over a radio link. This scenario consumes a lot of effort both, in terms of planning and conducting the tests. Besides, every flight is a potential risk for the aircraft and its environment, especially when testing airborne fault injections.

Injecting faults during a UAS flight poses a threat to the UAS and its environment. An option is to mitigate the consequences of the injected faults. Nonetheless, a complete roll back might not be possible, or there can be unforeseen side effects.

Another disadvantage of this setup is that it is practically impossible to perform two totally equal flights, thus, reproducibility of experiments is heavily limited.

(B) illustrates the processor-in-the-loop (PITL) or “Ironbird” setup. The Ironbird is a UAS laboratory configuration, consisting of the actual Dragoneye hardware like the flight computer, the sensors, the actuators, and the communication devices. Compared to the Dragoneye, the Ironbird is not able to fly and has no engines. However, it offers direct access to the aircraft’s hardware components. Thereby, the Ironbird setup enables to read data or simulate certain conditions (e.g., fault injections, or simulate certain sensor measurements) by tapping into the component’s interfaces. Thus, injected faults or attack scenarios can be tested without risk. Since the hardware components of both, the Ironbird, and the Dragoneye are equal, the same binary executables of the FSW are executed. Therefore, the setup offers a realistic testing environment for the flight software. Furthermore, due to the laboratory environment, reproducibility is improved compared to real test flights. However, simulating a realistic test flight requires a huge effort, since all the sensor readings (e.g., GPS, airspeed, barometric altimeter) have to be simulated and have to react accordingly to the actuator’s actions. Communication with the (test) operator can either be performed via the radio link, or directly by means of a USB connection to the flight computer. All components are the same as the Dragoneye setup, also the communication to the R2U2 framework is identical. Hence, the data is transmitted via a read-only UART interface.

The software-in-the-loop (SITL) test setup (C) is a simulation environment for the NASA Dragoneye UAS on a Desktop PC using the Ardupilot SITL Simulator
[http://ardupilot.org/dev/docs/sitl-simulator-software-in-the-loop.html#]
.

As illustrated in the Figure above, the Dragoneye UAS flight behavior is simulated by the Open Source JSBSim flight dynamics model. The flight software (executable autopilot) is compiled from the same sources as for the Dragoneye and the Ironbird setup. However, the SITL executable uses specific low-level drivers, which emulate all of the aircraft’s hardware components and communicate directly with the JSBSim flight dynamics model via local UDP connections. The Open Source MAVLink proxy
[http://dronecode.github.io/MAVProxy/]
is connected via local TCP connections to the autopilot software. This enables to connect multiple ground control stations to the UAS simulation. Thus, for our case studies, an attacker GCS is connected besides the operator GCS in order to simulate the injection of malicious MAVLink packets to the UAS.

The SITL low-level drivers are used to simulate fault injections or malicious attacks on sensors like, for example, sensor failures, GPS spoofing, or radio jamming. Thus, compared to the Dragoneye setup, faults can be injected without the risk of damaging the aircraft during a real test flight. Triggering of such fault injections or simulated attacks can be achieved by sending specifically designed MAVLink packets via the MAVLink proxy.

The simulated UART SITL drivers are directly streaming the extracted monitoring data on a physical UART of the PC. Hence, the data transmission to the R2U2 framework is equal to the Dragoneye or the Ironbird setup.

In order to provide realistic test flights, the simulation uses random sensor noise and random environmental conditions like wind. However, since all parameters are controllable and the simulated missions can be carried out automatically by predefined scripts, the test flights are reproducible.

A manual on how to setup the SITL Simulation can be found here:

We provide dataflash logfiles of the autopilot software for different case studies we presented in [RV15]:

You can go through our simulation dataflash logfiles using APM Planner 2.0. Start APM Planner 2.0, and as shown in the figure below, switch to the "GRAPHS" view, go on "Open Log", and select the apm_dataflash*.bin file from the casestudy directory.

Now you can select the signals you want to display in the right pane. Scaling can be changed by doubleclick on the vertical axis. For example, to see some interesting signals for the GPS Spoofing casestudy, display the signals as can be seen in the next figure.

In order to run the SITL Simulation, please refer to the manual here. If you intend to run the simulation scripts from our casestudies, you will need to install the MAVProxy modifications. Simply overwrite the files in the MAVProxy installation directory with the files you can download here. Once the SITL simulation is running, you can execute this scripts using the modified MAVProxy application by issuing the command "script <path_to_script>". To reproduce our results, you can follow the directions from our flight_plan.ods, which is located in each casestudy directory.

The LTL/MTL formulas to be evaluated are defined in an text file as shown below:

In Lines 2-5, the results from the preprocessing stage / the atomic proposition checkers are mapped to labels, in order to use them in the formulas. The assigned values refer to the index of the outputs from the preprocessing. The labels can then be used in the formulas.

The file is split into a section with past time formulas and future time formulas. Each section can include an arbitrary number of formulas. Currently, we support up to 256 formulas, however, the maximum limit can be extended by defining a larger results memory in the R2U2 FPGA design parameters.

Each formula can have label as can be seen, for example, in line 12 (bad_packets). These labels are used in order to map the result of a formula to a node within a R2U2 Bayesian network model. Furthermore, these labels are used for in the Matlab vizualisation scripts of the results.

The supported operators / syntax is described in further details here.

These files are then compiled to the R2U2 binary program using the LTL2R2U2 compiler.

For designing the R2U2 Bayesian network models, we use the free graphical design environment Samiam from the Automated Reasoning Group of the University of California.

Please refer to their documentation for details on how to install/use the tool.

An example of an R2U2 model for the DoS Hijack casestudy from [RV15] can be downloaded here.

To define different states, doubleclick on a node, select the tab "Properties", and add the states as shown in the figure below.

To define the CPT tables, double click on a node, select the tab "Probabilities" and enter the Probabilities for the different combinations.

The tool offers a query mode where the model can be evaluated. To enable the query mode select menu→mode→query mode.

Next, select: query→show monitors→show all in order to display the monitors of all nodes.

As can be seen in the figure below, an observation can be simulated by selecting the observed stated on a node. The probabilities of the other nodes are instantenously updated.