7 Replies Latest reply on Mar 24, 2017 7:23 AM by Carlos_A

    i210 PCIe read random hang

    Axurit Community Member

      Hi,

       

      We are validating a custom board embedding Intel i210 product, which connect the processor through PCIe.

      Our custom board system is linux 4.1.15 which contains i210 driver natively.

      i210 is connected to a serial flash, which we already programmed through SPI with a SPI programmer.

      We programmed our serial flash with Dev_Start_I210_Copper_NOMNG_4Mb_A2_3.25_0.03.bin (provided by one of your distributor).

      During linux boot up, i210 driver is correctly initializing itself (Device ID 0x1533).

      It performs a lot of read through PCIe, but it hangs at a random PCIe read : "value = readl(&hw_addr[reg]);" in "igb_rd32(struct e1000_hw *hw, u32 reg)" in "igb_main.c".

       

      In attached files :

      - log of i210 initialization process,

      - log of lspci -vv

       

      Can you give us some hint ?

       

      In advance, thank you for your support.

        • Re: i210 PCIe read random hang
          Carlos_A Brown Belt

          Hello, Axurit

           

          Thank you for contacting the Intel Embedded Community.

           

          In order to have a better idea of this situation, we would like to address the following questions:

           

          Could you please tell us if the affected design is a third party one or it has been developed by you? In case that it is your project, could you please tell us if it has been developed based on the suggestions stated in the Intel(R) Ethernet Controller I210: Design Guide document # 513305, Intel(R) Ethernet Controller I210-AT/IT: Layout Review Checklist document # 495298, and Intel(R) Ethernet Controller I210-IS: Layout Review Checklist document # 495299? Also, could you please confirm us if it has been verified by Intel?

           

          Thanks in advance for your collaboration to solve this inconvenience.

           

          Best regards,

          Carlos_A.

            • Re: i210 PCIe read random hang
              Axurit Community Member

              Hello Carlos

               

              Thanks for your quick answer.

              The design was not reviewed by Intel but the schematic was reviewed by the AVNET specialist.

              We review the schematic using the check list you sent and the main differences are that :

              • we have 220 nf capacitors on the i210 Receive lines these capacitors are near the iMX6 processor
              • we also have 100 nF capacitors on the Clock lines (the capacitors are recommended in the iMX ref design).

               

              We don't review for the moment the Ethernet side  because our issue is the PCIe interface.

               

              We make some changes :

              • replacing the 220 nF capacitors on the RX line with 100 nF capacitors but the issue remains the same.
              • Replacing the 100 nF capacitors with 0 W

               

              The last thing we try is to remove the 100 nF (replaced with 0W resistor) on the TX lines of the i210 with the 100 nF capacitors on the RX and clock lines (these is the way the PCIe interface is implemented on the iMX ref design).

              The first time we launch the driver all seems good but we can't ping our CPU using this Ethernet channel.

              The other times the driver stops but seem to go further.

               

               

               

              We have added traces in the igb driver and the drivers always hangs on read instruction but never at the same address.

              We think we have some timing issues on the PCIe interface but we cannot find how to correct that.

              The imx6 has some capabilities modifying timing at the PCIe phy level but it seem it mainly affect the transmit line of the processor.

              The i210 and the iM6 are very close in our design which should minimize the timming issue.

               

              Do you have any idea?

              You will find here attached the 3 schematic sheets of our design which concern the iMX6 and i210 PCIe interface if you want to review them.

               

              One other issue we have was with EEPROMARMTOOL that couldn't run and stops with bus error.

              We have a look on several discussion on embedded community that have the same problem but we can't find the way they solve this issue (it seems that the problem is due to memory mapping).

              So we use an AARDVARK USB to SPI interface to program the i210 external Flash and we found we have a bug on the AARDVARK interface with the flash (you can see on the schematic that the pin 5

              of the connector is connected to Flash DO0 instead of flash DO1) this issue was corrected on the board so we can program the flash but the flash read hangs using the external probe.

              Is there any link with the driver issue?

               

              Thank for your help, if you need some more information on our design or the whole schematic don't hesitate to ask me.

               

               

              Best regards

                • Re: i210 PCIe read random hang
                  Carlos_A Brown Belt

                  Hello, Axurit:

                   

                  Thanks for your detailed reply.

                   

                  Based on your previous communication, we want to address the following questions:

                   

                  Could you please verify that your implementation fulfills with the layout requirements stated at the documents listed on our previous communication? In case that there are discrepancies please correct them, try to reproduce the problem, and let us know the consequences of these changes.

                   

                  Could you please confirm if the system hangs, or just the part? in case that it is the part when it hangs, does it come back after a reset or is a power cycle needed? or does it hang during boot up, or after the system has been up for a bit? please give us a detailed explanation.

                   

                  Thanks again for your cooperation.

                   

                  Best regards,

                  Carlos_A.

                    • Re: i210 PCIe read random hang
                      Axurit Community Member

                      Hello Carlos

                       

                      We will review the lauout next week with our specialist and if there is somme issues on the layout we will modify the board and make a new pcb.

                       

                      In fact it's not the system that hangs its the igb driver.

                      If we launch the baord without loading the driver all works fine and the i210 is detected using the lspci (you have the lspci -vv result attached withthe first message).

                      When we launch the driver, rhe driver hangs and it's why the whole system hangs.

                      We add traces in the driver abd the last action before the driver hangs is a register read but never at the same address (you also have a trace of these prints attached with the first message).

                      It seems that there is someting wrong (perhaps a timing issue or someting like that) whent the imX6 tries to read the i210 registers.

                      Modifying the Ac coupling capacitor on these lines (replacing them by 0 Hom resistor) seems to make some diffrences.

                      The first time we start the driver after this modification all seems to work. The driver launch doesn't hang and we see the the eth link and can configure it using ifconfig but we cant ping the board.

                      so we restart the baord and the driver hangs again at each launch.

                       

                      We realy think tere is some issu on the read lines or in the driver.

                       

                      Best regards