mobile | classic
Dataweek Electronics & Communications Technology Magazine

Follow us on:
Follow us on Facebook Share via Twitter Share via LinkedIn


Electronics Buyers' Guide

Electronics Manufacturing & Production Handbook 2017


A fault tolerant, fully distributed RTOS
19 July 2017, Computer/Embedded Technology

As the world is moving towards ‘smarter’ systems, often embedded, and our life and society is becoming dependent on their uninterrupted operation, fault tolerance is becoming a prime requirement. Just think about autonomous driving. Will it safely bring you home all the time? In case of a system fault, the reaction time is less than 100 milliseconds. No time for a reboot.

A consequence of the fine-grain space partitioning support in the VirtuosoNext real-time operating system (RTOS) is the capability to recover from runtime faults within a few microseconds. The combination of fine-grain concurrency and this fast recovery effectively provides support for non-stop hard real-time processing even when faults occur, without a complex and costly system design. VirtuosoNext’s non-stop capability means that fault tolerance comes within reach in a cost-efficient manner, as well as in terms of development effort as in terms of compute resources.

Why do systems fail?

From time to time, the public is made aware of catastrophic failures induced by software errors. Such failures happen all the time but do not always make it to the news. A failing car is considered an annoyance, a failing mission to Mars or an airplane crashing is a costly disaster.

It is no wonder that in safety critical sectors much attention is given to the engineering process and in particular to the correctness of the software. Formal methods, rigorous processes, and external reviews by experts all aim at bringing the probability of failure to the lowest acceptable level.

Still, systems can fail spectacularly. Think about several missions to Mars, the first Ariane-5 rocket launch, the failing Patriots during the first Gulf war, etc. Sometimes the failure is not even induced by the software or the processors but by a failing power supply, as recently happened at British Airways.

Why do such systems still fail? A first obvious reason is human error. Even when using formal techniques, it is very difficult to obtain a complete set of specifications that take into account all real-world circumstances. Formal models might be proven correct, but remain incomplete models of a much more complex world.

Or the software engineer made a subtle mistake that no toolchain can catch, or worse: that will never be tested for because the mistake is unknown. And last but not least, software is an imperfect execution on an imperfect piece of hardware. In a digital world, software data types have a limited precision even if the real-world variables are continuous and can have any value. In addition, the bits that represent that data in the processor can become corrupted, for example because a cosmic ray induced extra electric charges in the circuit, or because a power glitch caused a bit to change its value. As chip features are continuously shrinking and chips operate at lower voltages, this becomes more probable.

Moreover, embedded systems work with real-world sampled data. The precision and range of the sensor introduces numeral imprecision and noise that, when not properly taken into account, can result in large deviations when used in long calculations. Any of these examples, and there are many more, will most likely result in a failure of the software and ultimately in a failing of the system.

A smart world needs to be fault tolerant

Computers have taken over the world, but they went into hiding as well and we call them embedded. Many of our systems (cars, energy networks, telecommunication systems, etc.) would not even be able to offer the service they provide without their ‘embedded’ computers.

Today, these systems are becoming ‘smart’ and adaptive. They can do this by constantly monitoring our environment and by interpreting what their sensors measure. An example of an emerging market is autonomous systems, e.g. self-driving cars and drones. Such systems have control loops measured in milliseconds. If they fail, the impact can be catastrophic because of avalanche effects.

To reduce these risks to a minimum, they must be able to continue to operate even when faults occur. We call this an ARRL-3 system, or with extra hardware redundancy an ARRL-4 or -5 system (see The novel non-stop capability of the VirtuosoNext RTOS provides higher safety at a much lower development and implementation cost. Essentially, the non-stop capability offers ARRL-4 in an ARRL-3 package (when using the right hardware such as the presence of an MPU or MMU and error correcting logic for e.g. program memory).

Current remedies

So, what do engineers do to deal with these unlikely but inevitable failures? The most common approach is known by all of us when our PC presents us with a frozen screen: switch off the power and reboot. As anyone knows, this is brutally simple, but it can take a serious amount of time, from tens of seconds to minutes. In complex systems like airplanes it can take a lot more time as the rebooting results in mandatory system checks. Therefore, the reboot option is hardly a valid one when dealing with safety-critical real-time systems.

Another option is to use partitioning. Derived from the server world, partitioning is based on very trustworthy software that runs on top of the hardware, typically called a hypervisor. The latter will assign applications to specific memory regions and typically share the time between time partitions taking tens of milliseconds. If a failure occurs, the whole partition is re-initialised and the application is rebooted, leaving the rest of the system intact. While this avoids the complete reboot of the whole system, the impact on the failing application can still be substantial as outlined in the previous paragraph. It is not compatible with hard real-time when safety and security risks are present.

VirtuosoNext changes this in a disruptive and very convenient way. It allows applications to recover in a few microseconds without disrupting the continued execution of the application. VirtuosoNext Designer uses a combination of code generators and a formally developed, small but distributed real-time kernel.

Partitioning starts when the application is written. An application is very naturally written as a number of tasks that synchronise and communicate by using so-called hub entities. In practice these hub entities will present themselves as traditional RTOS services like events, semaphores, resource locks, mailboxes, etc. From the top level application model description, code generators generate the data structures, the initialisation code and static program images that can’t be modified at runtime as they reside in read-only memory.

A key element is the semantics of these hub services. The services pass their data by copying and with strict hand shaking. The data is passed on to the kernel task which in turn then delivers the data to the corresponding task. While it is possible to pass address pointers, this is only allowed for so-called trusted code, for example the kernel task and driver tasks.

First line of defence: decoupling

The first line of defence is that the underlying implementation fully decouples and strictly separates the application tasks from each other. If the program is correctly developed (hence not using shared memory unless under strictly controlled circumstances), no task will access the memory of another task. As the scheduling is not time based but event and priority based, small timing differences have no impact on the correct operation of the program. This is further reinforced by the use of an underlying packet-switching mechanism that also supports transparent programming across multiple processors, be it that the CPUs reside on the same chip (many/multicore processors) or across a network.

Second line of defence: fine grain partitioning at task level

The second line of defence is that VirtuosoNext Designer will allocate each task its own private memory space, fully protected by the memory protection logic of the processors. Also the code itself is protected and ‘read-only’, avoiding that it can be unwillingly modified. No task can read or write in the memory of another task, even as a result of a programming error or a fault in the hardware.

If such a fault would happen then the task will fail, but the rest of the system will continue to execute, unless another task was dependent on the execution of the failing task. In this case, the system will not crash but will stall, waiting for its data to arrive (over one of the hub entities). In many cases even this will not happen as using a hub service like a blackboard, the task can just read the latest data from that hub and continue. Note also that for security risks, this effectively limits the risks as well.

To learn about the third and fourth lines of defence offered by VirtuosoNext, and the benefits it provides for embedded security, read the full version of this article at, under the Computer/Embedded Technology category.

For more information contact Eric Verhulst, Altreonic,

  Share on Facebook Share via Twitter Share via LinkedIn    

Further reading:

  • Development kits for students and hobbyists
    14 November 2018, RS Components (SA), Computer/Embedded Technology
    RS Components has introduced new ranges of tools from Seeed Studio, Arduino and Bare Conductive that target academic institutions, including educators and students of electronics, as well as inspiring ...
  • Accelerator cards for AI and data centres
    14 November 2018, Avnet South Africa, Computer/Embedded Technology
    At the recently held Xilinx Developer Forum (XDF), Xilinx launched the Alveo portfolio of powerful accelerator cards designed to increase performance in industry-standard servers across cloud and on-premise ...
  • CompactPCI Serial board for networking
    14 November 2018, Rugged Interconnect Technologies, Computer/Embedded Technology
    Pentek’s SY8-Cyclone is a CompactPCI Serial peripheral board, equipped with a powerful FPGA and up to 10 RJ45 connectors for 100BASE-TX Ethernet. With its PCI Express x4 interface, the Cyclone-V FPGA ...
  • OpenVPX board for radar and DRFM
    14 November 2018, Rugged Interconnect Technologies, Computer/Embedded Technology
    Pentek introduced its first product in the Quartz architecture family, the Model 5950 – an eight-channel A/D and D/A converter, 3U OpenVPX board based on the Xilinx Zynq UltraScale+ RFSoC FPGA. The low ...
  • FPGA and SoC modules
    14 November 2018, RS Components (SA), Computer/Embedded Technology
    RS Components has added a new portfolio of advanced modules based on FPGAs and system-on-chip (SoC) ICs from Trenz Electronic, a leading developer and manufacturer of module-based solutions for the industrial ...
  • Evaluation kit for Vishay sensors
    14 November 2018, RS Components (SA), Computer/Embedded Technology
    The Vishay SensorXplorer evaluation kit is based on a universal USB-to-I²C board for connecting individual sensor boards to a host PC. All software and connectors are included with the kit, which is backwards-compatible ...
  • Arduino shield for energy measurement
    14 November 2018, Altron Arrow, Computer/Embedded Technology
    Analog Devices’ EV-ADE9153ASHIELDZ is an Arduino shield compatible with Arduino Uno, Arduino Zero, or ESP8266. The shield has an on-board shunt resistor for line current measurement and enables quick ...
  • Dev platform for IoT devices with advanced sound and vision
    14 November 2018, RS Components (SA), Telecoms, Datacoms, Wireless, Computer/Embedded Technology
    RS Components is boosting opportunities to create smarter IoT devices by introducing Sony Spresence, featuring Arm Cortex-M4F performance and extensive sensing, audio and imaging capabilities. The low-power ...
  • Top 10 development boards of 2018
    10 October 2018, Altron Arrow, This Week's Editor's Pick, Computer/Embedded Technology
    In order to support the most innovative year yet in 2018, Altron Arrow presents its top 10 recommended development boards and kits.
  • Rugged COTS-based computing platform
    10 October 2018, Rugged Interconnect Technologies, Computer/Embedded Technology
    North Atlantic Industries has leveraged its modular and adaptable COSA architecture to provide application-ready systems for harsh air, land and sea environments. These rugged COTS, preconfigured and ...
  • Development kits for students and hobbyists
    10 October 2018, RS Components (SA), Computer/Embedded Technology
    RS Components has introduced new ranges of tools from Seeed Studio, Arduino and Bare Conductive that target academic institutions, including educators and students of electronics, as well as inspiring ...
  • Accelerator cards for AI and data centres
    10 October 2018, Avnet South Africa, Computer/Embedded Technology
    At the recently held Xilinx Developer Forum (XDF), Xilinx launched the Alveo portfolio of powerful accelerator cards designed to increase performance in industry-standard servers across cloud and on-premise ...

Technews Publishing (Pty) Ltd
1st Floor, Stabilitas House
265 Kent Ave, Randburg, 2194
South Africa
Publications by Technews
Dataweek Electronics & Communications Technology
Electronic Buyers Guide (EBG)

Hi-Tech Security Solutions
Hi-Tech Security Business Directory

Motion Control in Southern Africa
Motion Control Buyers’ Guide (MCBG)

South African Instrumentation & Control
South African Instrumentation & Control Buyers’ Guide (IBG)
Terms & conditions of use, including privacy policy
PAIA Manual


    Classic | Mobile

Copyright © Technews Publishing (Pty) Ltd. All rights reserved.