Skip navigation
2011

Many embedded systems are deployed remotely, far from any technical staff that might be able to fix a problem with a malfunctioning computer. Certainly military and &aerospace applications rely on remote systems that operate reliably, but even digital signage, industrial, and retail applications locate business-critical systems that are far from the tech staff. Systems designed for such applications need some ability to automatically recover from faults along with remote-management capabilities. Intel® Architecture (IA) processors can readily support such applications and manufacturers of modular- and system-level IA-based products that target embedded applications often add reliability layers

.

The watchdog timer is among the most commonly used techniques that can allow a remote system to reboot itself upon some type of system fault. Most manufacturers of IA-based computer modules, single-board computers (SBCs), and embedded-targeted systems include a watchdog timer.  For example, the small Advantech* ARK-6310 ruggedized system based on a Mini-ITX board includes a watchdog timer that can be programmed with an interval ranging from 1 to 255 seconds.

 

Design teams typically utilize the timer to automatically reset a system. A properly-functioning system includes a recurring task that resets the timer on a regular basis. A malfunctioning system that is hung on a task and fails to reset the timer will allow the timer to count all of the way to zero and trigger a system reset.

 

While most all systems include the watchdog functionality, teams may want to add features that allow a combination of preventive maintenance, remote management, and automatic restoration. Intel® Active Management Technology (AMT) supports remote-management capabilities and as I covered in a recent post is one of several IA features that can deliver mission-critical reliability.

 

Let’s consider how a couple of third parties have added system reliability features. LiPPERT Embedded Computers**, for example, has a technology suite called LEMT (LiPPERT Enhanced Management Technology) that it is supporting on all of its new SBCs. For example, LEMT is available on the recently announced CoreExpress ECO2 computer-on-module (COM) product that is based on the CoreExpress standard that was originally developed by LiPPERT and that now is being promulgated by the Small Form Factor Special Interest Group (SFF-SIG). The ECO2 (pictured below) is based on the Intel® Atom™ E6xx processor family. The company also supports LEMT on its E6xx-based Toucon-TC  COM Express module.

 

cem-eco2.jpg

 

The LEMT technology is based on the combination of a System Management Controller (SMC) IC and an application-layer program that can be accessed locally or remotely via a network connection. The SMC IC is a microcontroller that combines power-sequencing functions needed at boot time with the ability to monitor and control elements of  the system.

 

The LEMT technology essentially enables preventive actions that can keep a system running reliably. The LEMT application can provide details of operating parameters such as system voltages, watchdog status, the current CPU and board temperatures, fan speeds, maximum temperature over a period of operation, and other data.

 

LEMT works with Windows and Linux systems. It allows a technical team to preempt costly failures or at the very least prepare to replace systems that are vulnerable. Moreover, LEMT combined with AMT will allow a remote team to change system settings and perhaps keep a system operational until it can be replaced.

 

There are also technologies that help a remote team diagnose a failure and perhaps restore system operation after what would be a fatal fault in many cases. Advantech, for example, has a software product that it calls Advantech eSOS – emergency secondary OS for system recovery.  eSOS is a Linux-based secondary OS that is stored in ROM in an embedded system and that is unaffected by any problems that may have impacted the execution of the primary OS. The eSOS doesn’t have the complex set of hardware dependencies that are present in the primary OS and can often be booted by a failed system.

 

In a typical implementations, an eSOS-based system would first try and reboot when triggered by the watchdog timer. If the

boot to the primary OS fails, the system will boot into eSOS. The eSOS then performs a hardware analysis on the system and emails a detailed report to the remote technical team.

 

esos.jpg

 

The remote team can connect with the system via telnet or ftp, and attempt to restore system operation. In fact the system will allow a complete restore of a Windows-based operating system. At a minimum, eSOS allows the team to determine the cause of failure and simplify the repair process.

 

The eSOS technology can be used with a variety of Windows operating systems including XP and Windows Embedded.  The technology was initially deployed on the Advantech PCM-9361 SBC and the SOM-5761 COM products. Both are based on the Atom N270 processor. Advantech plans to support other Atom-based products and perhaps other OSs going forward.

 

Please share you experience with remote mission-critical systems with other followers of the Intel® Embedded Community.  Your comments will be greatly appreciated. How have you handle remote monitoring and maintenance? What challenges did you face? What do you think about the LiPPERT and Advantech technologies covered here?

 

Maury Wright

Roving Reporter (Intel Contractor)

Intel® Embedded Alliance

 

*Advantech is a Premier member of the Intel® Embedded Alliance

** LiPPERT Embedded Computers is an Affiliate member of the Alliance

With fourteen currently available members of the Atom family, there is a variety of choices to meet hardware needs. But, choice sometimes can complicate software development. In the case of Atom processors, there are differences not only in execution performance, but also in the native support for some functions like full motion video. Properly structured and conceived software can make porting to a new Atom environment quick and easy. With forethought and following some simple rules you can save development costs and time, speeding new products to market.

 

Structuring software to make porting code across the Atom family can be simplified by choosing the right development environment. One solution is based on supported, commercial software tools such as the Eclipse environment which provides the ability to customize your software development environment.  Software development tools from Intel, Wind River Systems(1), and Green Hills Software(2) use the Eclipse framework to permit third party tools and user customization of the environment.

The Eclipse framework includes a Concurrent Versions System (CVS) repository. This is a persistent information store used to “manage” multi-user access to projects and their contents. Different software versions are maintained by the use of branches which spring off the major development line. Projects in a repository can be of two forms: immutable (a project version) or modifiable (a project in a branch). Since CVS is not a rigidly defined system, it’s possible to manipulate how the tool gets used – creating unique strengths and weaknesses.

 

CVSworkflow.jpg

 

The Eclipse Foundation documentation defines an ideal work flow:

 

1.   Start fresh. Before starting work, update the resources in the workspace with the current branch state. If you are sure that you have no local work that you care about, the fastest way to get caught up is to select the projects you are interested in from the branch (or HEAD) and select Checkout (or Replace with > Latest from Repository if the projects already exist locally). This will overwrite your local resources with those from the branch.

2.   Make changes. Work locally in your Workbench, creating new resources, modifying existing ones, saving locally as you go.

3.   Synchronize. When you are ready to commit your work, synchronize with the repository.

a.   Update. Examine incoming changes and add them to your local Workbench. This allows you to determine if there are changes which might affect the integrity of what you are about to commit. Resolve conflicts. Test again, run integrity checkers (for example, check for broken hypertext links, ensure your code compiles, and so on).

b.   Commit. Now that you are confident that your changes are well integrated with the latest branch contents, commit your changes to the branch. To be prudent, you may repeat the previous step if there are new incoming changes.

 

This ideal workflow can be compromised by accident or on purpose. Or, a different approach can be defined by the development team. Usually the integrity of development software is assured through company-wide or development team programming standards and practices. These standards are also where the specifics of how to use CVS are kept, although there is not a guarantee of automatic checking for adherence to the standard.

 

Some companies do not allow embedded applications-specific software to change the development environment as a way to control uniformity of development. In these cases more traditional tools are still available: programming language macro pre-processors and source configuration and control software.

 

C and C++ both include standardly defined macro preprocessors as part of the pre-compilation process. The standard C preprocessor (CPP) has basic symbol-based processing that can control compilation:

 

#ifdef _WIN32 // _WIN32 is defined by all Windows 32 and 64 bit compilers

#include <windows.h>

#else

#include <linux.h>

#endif

 

In this case we are selectively choosing between header information specific to an operating environment depending on whether or not the environment is a version of Microsoft® (3) Windows™ or something else. In this case the software definition either selects the Windows header, or the Linux header if WIN32 is false. It’s important to recognize that this structure really selects between the true value of WIN32 and the false value. If there were more than two choices for operating environment possible, we would need to have additional conditional compilation steps or use a different form of conditional compilation.

 

The two feature-set options of critical importance for products using E6xx Atom processors are: the use of the standard IOH and the presence of integrated graphics support including full motion video. We could define two different boot mechanisms based on the presence or absence of the standard IOH:

 

#ifdef std_hub_type

#include <stdboot.c>

#else

#include <bldk_boot.c>

#endif

 

 

Configuring software for the presence (or absence) of processor-based capabilities is similar to selecting between two different boot mechanisms. However, for processors that include full motion video capabilities and versions that have a limited set of graphics support, there are some other considerations. Members of the Atom family are supported by two different drivers for graphics: The Intel Embedded Graphics Drivers (IEGD) and Intel Embedded Media and Graphics Driver (EMGD).  Assuming that your development can use these drivers, the primary issue is selecting which one to compile as part of your application. This may be accomplished using the same compile time if-then-else structure used to select between a Windows environment and a Linux environment.

 

Another approach to managing different versions of software for different processors is to add third party tools to the Eclipse environment. Subclipse is a pre-configured Subversion plug-in for the Eclipse workbench.  Subversion is a relational database model of configuration control that adheres strictly to the “all or nothing” philosophy of updating source files. By doing so, there is little danger of some part of an application remaining uncompiled without your knowledge due to conflicts. Installing Subversion is a simple process on an Eclipse workbench, requiring a small number of mouse clicks.

 

Each of the commercial software development packages includes a software project management “wizard” that can also be used to configure software. As an example, Green Hills’ MULTI Project Builder uses a template that defines a specific hardware system (board) so that necessary software modules are automatically included in the compilation process.

 

GHSWizard.jpg

 

Project management facilities can provide some control over what software is included in a specific build.  Selecting between which capabilities a specific member of the Atom family supports can be achieved through the Board Support Package, or template. One template may be defined for Atom-based systems that include full motion video, and another version for Atom processors that do not have full hardware acceleration.

 

Developing software that can work on all Atom processors is easy, but care must be taken when choosing algorithms to simplify the porting task. Porting issues within the Atom family are related to the presence or absence of hardware acceleration on the processor. The generic software porting issues are eliminated for software that remains targeted to Atom-based processors. Some guidelines for developing software to span multiple members of a single processor family include:

 

  1. Manage software configuration at the highest level that your tools support. By keeping as many of the implementation alternatives out of the code proper you minimize the chances of an incompatibility sneaking past undetected.
  2. Create a single Application Programming Interface specification for different processor hardware capabilities. Special function hardware homologation should be kept compartmentalized behind the defined interface.
  3. Establish minimum performance requirements for key algorithms and define systems level variables to control the configuration management to generate a warning when trying to configure the application to a processor that has too      little performance.
  4. Clearly document performance-oriented programming techniques that are key to an algorithm’s performance or function – it’s amazing how often the original author forgets why a particular technique was used. This is especially important for programming techniques or styles that are processor-specific.
  5. Use the performance analysis tools provided by the software tool chain to explore alternative algorithms and implementations. Despite the natural tendency to peephole optimize software, the best performance is often found by large scale algorithmic choices. 

 

The Atom processor family offers alternatives for many embedded applications. Anticipating the future choices that will exist can be made easier by reviewing the Intel Atom family roadmap.

 

How will future Atom processors affect your software development process?

 

_________________________________________________________________________________________________________

1. Wind River Systems is an Associate member of the Intel Embedded Alliance

2. Green Hills Software is an Affiliate member of the Intel Embedded Alliance

3. Microsoft® Corporation is an Associate member of the Intel Embedded Alliance

 

Henry Davis

Roving Reporter (Intel Contractor)

Intel® Embedded Alliance