In another blog I’ve talked about improving systems safety by controlling unintended software interactions. That’s one way to improve availability – remove an entire class of potential failures. Another way to improve availability is to have a second copy of software running, ready to take over in the event that the first copy fails. Termed hot standby, this technique creates redundancy by operating primary and standby systems simultaneously. Data is mirrored to the standby system in real time so that both systems contain identical information. Other standby approaches include warm and cold standby configurations. “Warm standby” mirrors data at intervals that are not realtime, while cold standby mirrors data at an even longer interval than warm standby.
To make hot standby a viable realtime technique for improving system availability, the primary and standby systems must be isolated from each other. Using a separate Virtual Machines (VM) for the primary and standby systems ensures the best isolation.
Hot standby via virtualization employing Intel® Virtual Technology (VT) works by isolating the Virtual Machines (VM) from each other, and conceptually from the underlying hardware. A standby instance of critical applications remains loaded and ready to run – data is transferred in realtime from the primary instance of the application to the hot standby instance. In some systems, when a critical fault occurs within in the primary VM, the Virtual Machine Monitor (VMM) handles the fault exception and signals for a switch to the hot standby. Examples of this type of fault includes memory and instruction execution exceptions. Other systems approaches to hot standby employ a process that monitors a defined failure mechanism, such as a “heart beat” from the primary and hot standby systems, and switches to the hot standby if the primary system fails to delivery its heart beat on time. For hot standby operation, the standby copy of the application runs using copies of the primary application’s data. Both applications operate simultaneously on a continuous basis. In most cases, there must be modifications to the application so that communications between the running application copies can occur.
One way of implementing hot standby is by using a virtualized OS like VLX from Intel® Embedded Alliance Associate member Virtuallogix.
VirtualLogix employs paravirtualization techniques to virtualize OS partitions within VLX. Paravirtualization means that some modification of the guest OS kernel has been done by VirtualLogix. This contrasts with full virtualization in which no OS modifications are required for the OS to operate in the virtual environment. The most often cited reason for employing paravirtualization is that the OS will run more efficiently, particularly in those portions of the application that access I/O.
Paravirtualization focuses on communication between the guest OS and the hypervisor to improve performance and efficiency. Paravirtualization requires modifying the OS kernel to replace nonvirtualizable instructions with hypercalls. These hypercalls communicate directly with the virtualization layer hypervisor. The hypervisor also provides hypercall interfaces for other critical kernel operations such as memory management, interrupt handling and time keeping.
VLX enables multiple Operating Systems, called guest OS's, to run simultaneously on the same single-core or multi-core processor. Guest OS's are isolated from each other, but communication mechanisms exist to permit transfer of data between guests. In this figure two or more RTOS partitions contain operating copies of the application, designated partition 1 and an implied partition 2. The Rich OS, designated partition n, contains user interface and other non-realtime code.
Using VLX to enable hot standby requires at least two separate RTOS partitions each with a running copy of the application, and communications from the primary application to the standby application in realtime.
Other Hypervisor configurations are also available. Intel® Embedded Alliance Affiliate member Real-Time Systems GmcH offers a fully virtualized VMM. Apart from providing access to a wider variety of RTOSes, their Hypervisor explicitly provides communication between operating systems. The Real-Time Systems solution provides a configurable user-shared memory as well as a TCP/IP based virtual network driver. Hypervisor is also said by the company to have zero overhead with hard real-time performance.
Hot standby has been a staple in the server arena to improve availability for many years. As embedded processors have gained increased memory, more processing power, and expended I/O capabilities, more advanced embedded systems designs have become possible. Hot Standby is one of these techniques. Although two companies have been highlighted in this blog, there are other possibilities for hypervisor or RTOS alternatives – including a “roll your own” approach to the hypervisor functions. Intel® Embedded Alliance Affiliatemember Greens Hills Software also provides the INTEGRITY® real-time operating system which can meet all the requirements for supporting a hot standby embedded system.
Regardless of your choice of virtualization technology, VT can be used to improve a variety of technical qualities. For a growing number of products no quality is more important than systems availability. Can your embedded designs benefit from hot standby?