Testing for security flaws is difficult because it fundamentally differs from the functional testing performed on most applications. Most functional testing involves “positive testing” to ensure correct responses to expected inputs. In contrast, security testing revolves around “negative testing,” which subjects the system to unexpected inputs and conditions to see if it will fail (Figure 1). The inputs needed for this negative testing can’t easily be derived from requirement documents alone.



Figure 1. Security testing focuses on the unexpected and unplanned.


To make things more challenging, security vulnerabilities may not be apparent outside a real-time environment. Consider the denial-of-service (DoS) attack as a simple example. These attacks saturate a system with inputs (typically network traffic) so that the system resources are overloaded and cannot function as intended. The key point about DoS attacks is that each individual input may appear to be a valid, expected event. It’s only when you accumulate the tasks in real time that the problem becomes visible.


Security testing is also hindered by the fact that testing tools are typically designed for developers, not testers. For example, code coverage tools used by developers are often too “heavy,” complicated, and slow for quality assurance (QA) testing. What’s more, these tools typically require instrumented code rather than production code. On the other hand, QA-oriented testing tools often treat code as a black box, making it difficult to determine exactly where the security vulnerabilities are.


These problems are exacerbated for projects that employ agile or iterative development, where code is updated frequently, often close to the launch date. Simply testing the entire code base on every revision can be impractical under these circumstances, yet testers must maintain comprehensive coverage to ensure that updates to one part of the code have not exposed new vulnerabilities in other functions.


Wind River Test Management was designed to address challenges like these. It provides three capabilities that are critical to catching security flaws:

  • Code coverage analysis of real-time production code
  • Automated performance regression testing
  • Binary code change analysis to minimize re-testing and flag suspicious code
  • The ability to simulate attacks thorough both pinpoint fault injection and “brute force” Fuzz testing


Let’s start with code coverage. Unlike typical code coverage tools, Wind River Test Management does not require a special build. This is a critical distinction, because most coverage methods work by inserting special instructions into the code. This instrumentation can cause a large performance hit. In addition to slowing testing—potentially stretching run times from hours to days—instrumentation can cause time-sensitive tests to fail, mask problems, or introduce bugs.


In contrast, Wind River Test Management takes production binary code—no special builds required—and analyzes the structure to instrument the code on the fly. Once a section of code has been covered, the instrumentation for that code is removed. Thus, the performance impact of instrumentation is minimized—in principle, there is no performance hit to steady-state execution.


This low-impact approach makes it practical to measure code coverage on production code. This is critical for negative testing, because non-covered code is inherently suspicious. Unexecuted functions might be “dead code” that contains security vulnerabilities. For example, developers can forget to remove test or debug APIs. Leaving this code in is a bad idea. Debug code is usually subject to less scrutiny—and thus more likely to have holes—when the development team assumes it will be removed.


It’s also fairly common for developers to leave debug code in intentionally, e.g., to aid in diagnostics after shipping. This is also a bad practice because hackers can take advantage of this back door. Finally, unexecuted functions could be malicious code inserted by a disgruntled coder, an infected development machine, etc.


Next, let’s consider performance profiling. Wind River Test Management’s real-time testing capabilities enable realistic performance profiling. Among other benefits, this feature enables testers to identify performance degradation across builds. Such anomalies can indicate that good code has been replaced with malware. To illustrate the significance of this feature, consider StuxNet. This worm overwrote code that was supposed to monitor motor speeds, replacing it with code that oscillated the motors from 2 Hz-1 kHz. The design of the worm made it difficult to detect in the field, but oscillating motors requires significantly more processor cycles than just monitoring them. Thus, performance regression can help testers identify malicious code.


Code change analysis is another key feature. Wind River Test Management also automatically inspects the binary code to identify which sections changed between builds. The system can use this “binary build differencing” and its structural code analysis to determine which test cases must be re-run, and to flag tests that may need to be enhanced to verify the changed code (see Figure 2). This information helps minimize testing on new builds, which is critical to achieving comprehensive coverage. It can also alert testers to unauthorized changes. For example, suppose functions A, B, and C all change, but only A and B are on authorized change list. This throws up a red flag—why did C change? Has the code been compromised?



Figure 2. Automated change-driven testing.


Finally, the tool includes features that help simulate attacks, including both pinpoint and brute-force method. The pinpoint method uses fault injection to mimic a hacker forcing the system into edge condition. For example, fault injection can be used to simulate a DoS attack that fills memory, causing the system to crash and leave vulnerable files on disk. Rather than force the tester to actually fill memory, the tool lets testers simulate the condition by injecting fault signals into the system (Figure 3). These faults could include conditions like memory full, database full, or no network connection. Because the tool gives testers white-box visibility, they can watch how the system responds to make sure any failures are graceful and do not expose vulnerabilities.



Figure 3. Fault injection can simulate an attack.


Wind River Test Management’s Security Pack extends attack simulation with fuzz testing licensed from Codenomicon. This technology deliberately feeds the software malformed inputs. Unlike fault injection, which requires knowledge of fault conditions, fuzz testing doesn’t require knowledge of the code. Instead, it uses a brute-force approach in an attempt to locate unknown risks.


The bottom line for testers is that security is an increasing concerns for embedded software, but test software is often poorly suited to detecting vulnerabilities. Wind River Test Management provides an alternative approach that both addresses security concerns and solves many common problems in QA testing.



Of course, even the best testing may not reveal every vulnerability. That's why it is wise to chose a hardware platform with built-in security features. One such platform is the 2nd generation Intel® Core™ processor family, which features the Intel® vPro™ suite of hardware-assisted security and management technologies. These technologies include:


  • Intel® Active Management Technology (Intel® AMT), which provides remote diagnosis and repair
  • Intel® Trusted Execution Technology (Intel® TXT), which supplies security protection over and above ordinary software solutions
  • Intel® Virtualization Technology (Intel® VT), which enables secure task separation, e.g., for systems that mix open-source and high-security software.


Together, these technologies provide a foundation for proactive threat deterrence – an approach that stops threats before they breach your system and isolates compromised systems so the damage is contained. See my recent Intel Vpro overview for details.

security.pngFor more on securing connected devices, see intel.com/go/embedded-security.


Wind River is an Associate member of the Intel® Embedded Alliance.


Kenton Williston

Roving Reporter (Intel Contractor), Intel® Embedded Alliance

Editor-In-Chief, Embedded Innovator magazine

Follow me on Twitter: @kentonwilliston