The Intel® 5100 Memory Controller Hub chipset (Intel® 5100 MCH) provides tremendous flexibility when designing embedded systems. Some of that flexibility comes in the form of multiple PCI Express (PCIe) ports on the Intel® 5100 MCH and on the companion Intel® 82801IR I/O Controller Hub (Intel® ICH9R). You'll find these chipsets used in high-performance embedded products such as Advantech's PCE-7214-a PICMG 1.3, full-sized, server-grade system host board (SHB) featuring dual Intel® Quad/Dual-Core Xeon® computing power-Kontron's CP6014 CompactPCI processor board featuring two Quad-Core Intel® Xeon® LV L5408 processors, and Kontron's CP6016 CompactPCI CPU board based on the Intel® CoreTM2 Duo processor T9400. Advantech and Kontron are both Premier members of Intel® Embedded and Communications Alliance.
The Intel® 5100 MCH provides six x4 Gen 1 PCIe ports and the Intel® ICH9R provides an additional six x1 Gen 1 PCIe ports. These six x1 Gen 1 PCIe ports can also be configured as one x2 port and four x1 ports. Although all of these ports on the Intel® 5100 MCH and Intel® ICH9R are Gen 1 PCIe, these ports vary somewhat in their I/O performance. To maximize overall system I/O performance you must consider the way your system design will allocate these PCIe ports while balancing the unique characteristics of each PCIe port available from the two chipset hubs and the corresponding peripheral-device performance requirements. The figure below shows how Advantech's PCE-7214 system host board breaks out the various PCIe ports from the Intel® 5100 MCH and the Intel® ICH9R to the board's connectors. Note that two of the Intel® ICH9R PCIe x1 ports are actually used to carry two Gigabit Ethernet ports.
Advantech PCE-7124 System Design with the Intel® 5100 MCH and Intel® ICH9R chipset
The first thing to do in this design process is to consider PCIe port width. The six x4 PCIe ports from the Intel® 5100 MCH can be programmably configured as one x16 port and one x8 port, as three x8 ports, or as six x4 ports. Similarly, you should note that the six x1 PCIe ports on the Intel® ICH9R can be programmably configured as one x4 and two x1 ports. Of course PCIe port bandwidth increases with port width, but even applications that do not need the full bandwidth of a wide PCIe port can still realize a performance boost due to lower transaction latency. A PCI Express packet's transmission time-and therefore its transaction latency-decreases as PCIe port width increases. So you may choose to connect a PCIe peripheral device with a wide PCIe link (formed from two connected PCIe device ports) just to get lower I/O latency even if that peripheral device doesn't need the full bandwidth available from a wide PCIe link.
It's relatively easy to choose between the PCIe ports provided by the Intel® 5100 MCH and those provided by the Intel® ICH9R. Place peripheral devices with the most stringent performance requirements on the Intel® 5100 MCH's PCIe ports and put lower-performance devices on the PCIe ports associated with the Intel® ICH9R because the PCIe ports on the I/O Controller Hub have more than double the latency, as shown in the table below:
In addition, the Intel® 5100 MCH's PCIe ports provide slightly more bandwidth than the Intel® ICH9R's PCIe ports, as shown in the following table:
(Note: The relative latency measurement in the first table was measured with only one outstanding transaction request while the relative bandwidth measurements shown in the second table were conducted with as many as 32 outstanding PCIe I/O requests, which tends to mask latency's impact on PCIe bandwidth.)
Finally, you should be aware that the Intel® 5100 MCH contains two I/O units (IOUs, called IOU0 and IOU1) and that the Intel® 5100 MCH's six x4 PCIe ports are distributed across the two internal IOUs as shown in the figure below:
I/O Unit (IOU) distribution and x4 PCIe port allocation ordering for the Intel® 5100 MCH
For best overall system performance, your embedded design should balance the I/O transaction load on the Intel® 5100 MCH's two IOUs. The circled numbers in the above figure show the assignment order for the various x4 PCIe ports on the Intel® 5100 MCH. To ensure that your system design gets the maximum PCIe performance from the Intel® 5100 MCH, you should follow this assignment order instead of first using all of the PCIe ports associated with one IOU before using the other IOU's PCIe ports.
So how many PCIe ports are enough for your system? Are the six x4 PCIe ports on the Intel® 5100 MCH sufficient or do you need more and wider ports? Does the relatively small bandwidth difference between the PCIe ports on the Intel® 5100 MCH and the Intel® ICH9R make a significant difference in your embedded design's performance?
Note: For more details, download the corresponding article Configuring and Tuning for Performance on Intel® 5100 Memory Controller Hub Chipset Based Platforms
(http://download.intel.com/technology/itj/2009/v13i1/pdf/ITJ-02-Tuning-Intel-5100.pdf), which appeared in the Intel Technology Journal, Volume 13, Issue 1, 2009.
Roving Reporter (Intel Contractor)
Intel® Embedded and Communications Alliance