Skip navigation
2013

Last time, in part 1, we explained how the growing M2M market is motivating a move to standards-based solutions, and how the Intel® Intelligent Systems Framework (Intel® ISF) and Digi International M2M Solution Builder Kit meet this need. This time we take a closer look at the elements of Digi’s Kit.

 

Long History in H/W Gateways

Digi International started over 25 years ago as a commercial networking company focusing mostly on hardware. Their original product was a multiport serial controller for the PC, which evolved by first adding wired Ethernet and LAN connections, then Wi-Fi, RF (such as ZigBee), and finally cellular radios. Today the company emphasizes embedded modules—of their own manufacture and by third parties such as Kontron—integrated with ready-to-go software into wireless M2M gateway products and services.

 

The company created the scalable iDigi Device Cloud and iDigi Manager Pro software to remotely manage configurations of thousands of connected M2M devices, manage software and firmware updates,  set security policies, and even reboot,  all with centralized access and control of devices or Digi gateways (Figure 2).  Both the iDigi Device Cloud and iDigi Manager Pro software can be used on Digi's or third party products using the iDigi Device Connector (Figure 3).

 

Roving_Reporter_Digi_Figure 2.png

Figure 2: iDigi Manager Pro shown centrally managing groups of remote devices spread out geographically.

 

Roving_Reporter_Digi_Figure 3.png

Figure 3: iDigi Connector software can bring most M2M devices into the iDigi Cloud, depending upon the operating system flavor.

 

Under the Hood: Hardware

A residential example of a typical Digi design starts with the M2M Solution Builder kit acting as a home gateway or concentrator, bridged between myriad wireless devices and the home's Ethernet LAN or the on-board 3G cellular radio providing Internet access to the cloud. An 802.15.4-enabled smart meter would report energy usage and time-load statistics, while a Wi-Fi equipped thermostat, washer/dryer and heater blower would accept remote instructions while providing updates on their operational state. The heater blower, for instance, might report increased current draw as a result of a dirty filter. Smart appliances can be scheduled to turn on during off-peak electricity times, or lights can turn on as the homeowner's smartphone broadcasts a text message saying "I'm nearly home" based upon GPS coordinates.

 

The Digi M2M Solution Builder Kit solves the "connectivity" portion of ISF with many options. Myriad flavors of 2G/3G cellular radios provide high bandwidth data at up to 14.4 Mbits/s (HSUPA at 900 MHz), and Wi-Fi options include 802.11a/b/g/n with WPA2 security.  There's also Gigabit Ethernet plus 802.15.4, although ZigBee remains a future add-on, as is Bluetooth 4.0 and 4G/LTE. Other I/O includes USB 2.0 and audio/video, plus up to four PCI Express mini card connectors for user-added I/O or processing.

 

But the heart of the hardware remains the Intel Atom E620T 600 MHz CPU and the x86 architecture. Instead of blindly transferring M2M data to the cloud, the Intel Atom Processor allows local processing and computing decision-making. Data can be routed between the gateway I/O (say, from a Wi-Fi connected sensor to USB disk). Image and audio processing can be done on a camera video/sound feed, and only clips or alerts that meet certain criteria need to be broadcast to the cloud. With the power of the E620T, the companion Intel Platform Controller Hub E620T, 1 GB of DDR2 SDRAM, and additional processing that a designer adds to the open PCI Express slots, the Solution Builder is a powerful platform to test-bed an M2M gateway. Future connectivity such as ZigBee or Bluetooth can be plugged in, while the Atom concurrently handles all protocols.

 

But besides the performance of the CPU itself is the ubiquity of the Intel x86 architecture, instruction set, and huge PC-based ecosystem. Instead of using any old embedded architecture with the challenge of creating device drivers and a custom boot sequence, the x86 architecture and its BIOS-like boot process make development a snap. Linux drivers are available for plenty of components and add-in boards, and existing x86-based applications from other designs can be easily and quickly transported into the Atom/Linux environment. Time-to-market is dramatically reduced from months of embedded development down to weeks or even days.

 

 

M2M promises to be the next wave of the Internet, bringing new services, productivity, and connections between previously "dumb" devices that will soon sport intelligence while broadcasting exabytes of data. Digi International's M2M Solution Builder Kit with Intel Atom 620T CPU is a flexible design platform from which to create an M2M gateway design or test platform. Compliant with Intel's Intelligent Systems Framework and connected to the iDigi Device Cloud, the Kit promises interoperability, manageability, and secure data transmission with other M2M nodes.

 

Intel connectivity logo thumbnail.jpgFor more on extending the Internet to embedded devices, see intel.com/go/embedded-connectivity

 

Kontron is a Premier member of the Intel® Intelligent Systems Alliance. Eurotech, Digi International, McAfee, Microsoft, and Wind River are Associate members of the Alliance.

 

 

Chris Ciufo

Roving Reporter (Intel Contractor), Intel® Intelligent Systems Alliance

Debugging Simics -- on Simics

Posted by jennysuh Jan 16, 2013

By Jakob Engblom, Senior Technical Manager, Tools & Lifecycle Solutions at Wind River

 

Blurb

I often write and talk about how useful Simics is debugging concurrency bugs and glitches in multithreaded and multicore systems. Recently, we had a case where we proved this on a very complex application, namely, Simics itself. This nicely demonstrated both the recursive completeness of Simics, and its usefulness for conquering tricky bugs in complex software.


The beginning of this story is a bug in Simics, triggered by a certain Simics configuration. The Simics target is a Power Architecture machine, running some bare-metal test code testing the processor simulation. Occasionally, this setup would crash Simics, due to some bug in Simics or the models. It was a difficult bug to track down, as it only happened in one run out of 50 or so. When attaching a debugger to try to diagnose it, it invariably did not happen (a classic Heisenbug).

Simics-on-simics-1

Simics is the perfect tool to diagnose these kinds of issues, but in order to do that, we had to get the failing program into Simics. I.e., running Simics on Simics. The first step was to create a duplicate of the development host, inside of Simics. This was fairly simple, just a matter of installing a Fedora 16 standard Linux on an 8-core Intel target. Once the Linux was installed and booted, a checkpoint of the system was taken.

Next, the development code tree from the host was packaged up as a tar file and put on a DVD image file. Simics was started from the checkpoint of the booted target system, and the DVD image inserted into the virtual DVD drive and mounted by the Fedora Linux running on Simics. The tar file was copied to the file system on the target, and unpacked. A new checkpoint was taken after the Simics installation was thus completed and Simics could run on Simics. The result at this point was a completely self-contained, controllable, and repeatable environment.

The screenshot below shows Simics running on Simics, with the same desktop wallpaper being used for both the host and outer Simics Fedora system:

Simics-on-simics-screenshot-600px

The next step was to replicate the bug inside of Simics. To this end, a shell command was used that repeatedly ran the inner Simics until the bug hit (obviously, this session was started from the checkpoint after the Simics installation).

The result was this setup, ready to run Simics until the bug hit:

Simics-on-simics-2

To recap, we have Simics running on Simics. The “inner Simics” is configured with the Power Architecture setup that resulted in a crash on the host, and the “outer Simics” is running Fedora 16, providing a virtual replica of the development host (but inside of Simics).

Additional scripting in the outer Simics was used to make the search for and replication of the bug more efficient.

Simics-on-simics-3

  • The Simics script varied the time slices given to the processors in the IA target system. This caused greater variation in scheduling of concurrent processes and threads in the Simics-simulated Fedora 16 OS, which in turn helped provoke the bug so that it appeared faster (after fewer runs of the inner Simics).
  • A checkpoint was taken after the inner Simics had been started and the timing variation applied to the IA processors – but before it had started executing the test case. This meant that a checkpoint would be available that led straight to the bug, with no need to do any warm-up of the target or particular configuration of Simics. The checkpoint would in effect be a self-contained bug report for the issue.
  • A magic instruction (blue star) was planted in the segfault handler of the inner Simics, making it very simple to catch the crash of the inner Simics. Often, using a magic instruction like this is simpler than trying to capture the right page fault or putting a breakpoint at the right place. A magic instruction is a live marker in the code that will always trigger, regardless of debug information or OS awareness. Furthermore, it has no overhead until it hits.

Eventually, after some 20 runs of the inner Simics, the bug was triggered. Thanks to the checkpoint and Simics repeatability, reproducing the bug is trivial. The Simics crash could now be reproduced any number of times, and it was time to go debug and figure out why Simics crash. An occasional Heisenbug had been converted into a 100% reproducible Bohrbug.

 

The first step of debugging was to figure out the mapping of the many dynamically loaded modules in the inner Simics. This was done by running the outer Simics and sending a Ctrl-Z to the Fedora shell, pausing the inner Simics. Then, the /proc file system on the Fedora Linux running on Simics was interrogated to find the load addresses. Since the checkpoint was taken after Simics was started, we know that this is the mapping in the software setup found in the checkpoint. Every time the checkpoint is opened, the same mapping applies – so the information was saved and used to setup symbolic debug information for the Simics modules used.

 

The next step of debugging was to open the checkpoint again, turn on reverse execution, and run forward until the magic instruction hit. Then, OS awareness was used to back up until the last time that the inner Simics was running prior to hitting the segfault handler. This placed the execution of the outer Simics at the precise instruction where the inner Simics crashed.

 

It turned out that Simics was trying to execute code in a location (BCDE) where no code was to be found.

 

Stepping back one instruction led to a JMP instruction to the location BCDE.

 

So where did this JMP BCDE come from? It was clearly not part of the static code of Simics, but something that was generated at run time by Simics itself (Simics contains a JIT compiler and thus modifying running code at run time is perfectly expected behavior).

To find out how the bad JMP was created, a memory write breakpoint was put on the instruction (JMP BCDE), and execution reversed. Simics stopped at the point where the “JMP” part of the instruction was written to memory. Doing a stack back trace at this point showed the code that was trying to write a five-byte “JMP XYZQ” instruction into the JIT-generated code stream. Since the breakpoint had hit on the write of the byte containing the JMP instruction code, indicating that the other four bytes (containing the actual JMP target location of XYZQ) were yet to be written when the instruction got executed and Simics crashed.

 

Stepping forward (on the processor) revealed that a thread switch happened in the inner Simics, and that the incoming thread immediately executed the five-byte JMP instruction, such as it was. Since only the JMP byte had been written, this was a jump to location BCDE, rather than the intended XYZQ (it would also have been OK to execute the original ABDCE code). Thus, the issue was diagnosed to be a read-write race condition, with the twist that the read was an execution of the memory as code and the write a regular data write. As soon as the problem was identified it was of course very easy to fix.

 

With the same setup, another race condition in Simics was also found and fixed, involving the more common case of multiple concurrent threads updating and reading a shared data structure without sufficient synchronization.

 

In summary, this blog post has described one instance where Simics was used to find and fix concurrency bugs in a real-world complex software system called Simics. The key to the success was the repeatability that Simics provides, even for timing-related occasional events, along with checkpoints, scripting, reverse execution, and debug facilities.

 

For additional information from Wind River, visit us on Facebook.

We're all more connected to the Internet than ever before.  The telecommunications giant Cisco estimated  that in 2012 over 100 million smartphone users will each consume over 1 GB per month and that global data traffic will increase 18 fold between 2011 and 2016. That's a 78 percent CAGR topping out at 10.8 exabytes (10E18) per month by 2016. [February 2012 "Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2011-2016"]


But here's the killer stat: M2M traffic will grow faster (22-fold) than human-initiated data during that timeframe, representing 5 percent of the total. Analyst firm IDC drills down even further: 1.8 billion intelligent systems (not smartphones or tablets) in 2010 doubles to 4 billion by 2015, representing trillions of intelligent connections, many autonomous M2M transactions created without human intervention via the Internet cloud. The point: M2M represents the next wave in embedded systems, with machines talking to machines more often than to humans. [IDC, "Intelligent Systems: The Next Big Opportunity", August 2011.]


Yet despite the tendency to focus on the design task of connecting all those machines and intelligent systems, the real challenge -- and business opportunity -- rests with harnessing the value the M2M data flowing across the Internet. Digi International, a member of the Intel Intelligent Systems Alliance, is a systems integrator focused on providing hardware, software and services designed to connect these systems. Digi recently started shipping  (December 2012) the Digi M2M Solution Builder Kit based upon Intel's Intelligent Systems Framework (ISF) and an Intel Atom processor running Wind River Workbench and Wind River Linux.


The Kit provides a comprehensive out-of-the-box cloud-enabled solution to help M2M developers rapidly get their hardware connected to the cloud by "Making Wireless M2M Easy" (Figure 1). But most importantly, the kit focuses on what developers are going to do with their machine's data before it hits the cloud, how the machine is remotely managed, and what options are available to secure the machine and its data. Not coincidentally, these are all core tenets of Intel's Intelligent Systems Framework.

 

1-Roving_Reporter_Digi_Figure 1 JPG small.jpg

Figure 1: Digi's M2M Solution Builder Kit makes "Wireless M2M Easy". It includes hardware, software and connectivity to the iDigi Device Cloud. It also supports Intel's Intelligent Systems Framework and is interoperable with other ISF solutions.

 

Digi International and Intel's ISF

The emphasis on services and systems is what sets Digi apart from most integrators and is why Intel's Intelligent Systems Framework is so appealing. ISF is a "set of recipes" for connected devices, solving the problem of connectivity -- that is, how devices attach to the cloud -- and so much more. The framework brings together hardware, operating systems, tools and more with a focus on: connectivity, security and manageability.

The key ingredients, according to Intel, are:

  • Processor platforms (including Intel® Atom™, Intel® Core™ and Intel® Xeon® ) and related tech like Intel® vPro and Intel® TXT, as well as a range of I/O for flexible communications
  • OSs including Microsoft* Windows*, Wind River* Linux*, and Wind River* VxWorks*
  • Security including McAfee Embedded Control and McAfee Deep Defender
  • Remote manageability capabilities that support third-party management consoles

 

More importantly, Intel is assuring compatibility between compliant solutions between suppliers. That means Digi's M2M Solution Builder Kit and the iDigi Device Cloud are compatible with ISF compliant products from other Intel Intelligent Systems Alliance partners like Eurotech, McAfee, Wind River and others.  In fact, the Builder Kit itself is composed of Alliance member components: the hardware is Kontron's M2M Smart Services Developer Kit, which when sold by Digi uses a module from Telit Wireless that's based upon an Intel RF component.

 

Intel's plan with ISF is to take the focus off of the computers that make up M2M and shift it to the task of intelligent computing. For example, previously a DVD rental vending machine designer might've focused exclusively on connecting their machine to the Internet and creating a rudimentary data exchange protocol. The data gathered by the machine is essential for billing and in-machine inventory management.

 

But there's no reason the video rental machine can't also communicate with other close-by rental machines or even retail stores, advising the customer where to locate an out-of-stock DVD and the best route to get there. Taken further, why not have all similarly branded DVD vending machines share data with close-by soda machines or fast food restaurants to offer customers incentives for food and drinks to go with their movie? Databases of renters' habits could make intelligent suggestions, increasing the revenue per transaction for the cooperative retailers, in this example.

 

For this to become reality, it's essential that the machines interoperate, for sure. But the future of M2M is monetizing the data flowing into the cloud by parsing it in new ways: aggregating machine- and human-generated data to create new services and increase productivity. Intel's goal with ISF focuses on connectivity, manageability, and security with an emphasis on off-the-shelf hardware, software, and services from ISF-compatible vendors like Digi.

 

Next time, in part 2, we’ll look under the hood of Digi’s kit to see how they achieve this goal.

 

Intel connectivity logo thumbnail.jpgFor more on extending the Internet to embedded devices, see intel.com/go/embedded-connectivity


Kontron is a Premier member of the Intel® Intelligent Systems Alliance. Eurotech, Digi International, McAfee, Microsoft, and Wind River are Associate members of the Alliance.


Chris Ciufo
Roving Reporter (Intel Contractor), Intel® Intelligent Systems Alliance


 

 

Late last year, in the shadows of the big splash launch of the Intel® Intelligent Systems Framework (http://www-ssl.intel.com/content/www/us/en/embedded/intelligent-systems-framework.html), another more subtle yet very significant announcement was made by the Intelligent Systems Group (ISG) of Intel.  At the Intel® Intelligent Systems Summit in Taipei last October (http://www.intel-events.com/iiss), ISG announced future availability of a product called the Intel® Firmware Support Package (Intel® FSP . . . http://www.intel.com/fsp).  Details of this announcement and technical presentation can be found here:  https://www.intel-event.com/iiss/iiss/d/track4/4-3.pdf

 

In 2010, Intel announce availability of a royalty-free system firmware solution called the Intel® Boot Loader Development Kit (Intel® BLDK . . . http://www.intel.com/go/bldk).   However, in October (2012), Intel announce it would be evolving the Intel BLDK solution, and transition to the Intel FSP solutions to enable royalty-free system firmware solutions.

 

Why you ask is this significant?  In addition to the Intel FSP solution being available on more Intel platforms, and not just the Intel® Atom™ Processors, the Intel FSP has standardized APIs that allow for interface of Intel silicon initialization code into any customized system firmware implementation.  The implication is that open source firmware solutions, such as Coreboot (http://www.coreboot.org/) and U-Boot (http://www.denx.de/wiki/U-Boot) can be utilized on Intel silicon for the first time.  Intel has historically held their initialization code very “close to the chest,” and prohibited any integration of silicon initialization code combined with open source code.  This has made it very difficult for embedded developers to utilize or develop anything other than a BIOS for an embedded system based on Intel® Architecture.  Now with the Intel FSP, embedded developers can use Intel Architecture for the first time ever with open source firmware solutions.

 

Want to find out more about the Intel FSP? Please plan to attend technical presentations at the Real-Time & Embedded Computing Conferences (http://www.rtecc.com/) at these locations and dates:

 

Santa Clara, CA       January 24

Boston, MA            May 9

San Diego, CA         October 15

Irvine, CA            October 17

 

Hope to see you there!