7 Replies Latest reply on Apr 21, 2017 6:31 PM by AdolfoS

    i801_smbus interrupt timeout on Broadwell-DE

    arios Community Member

      Hello,

      I am running Debian Jesse on an Intel Broadwell-DE and I am having difficulties enabling the SMBUS drv=i801_smbus) interrupts.

      I removed the "i2c_i801 disable_features-0x10" in the options.config file under '/etc/modprobe.d' but interrupts are not working.

      The i801_smbus driver is setup to enable the 'INTREN' bit of SMBUS Host control register (SMBUS-D31:f3).

      According to spec, when this bit is enable the completion of a command is supposed to interrupt the processor.

      I can see the interrupt pending in the PCI configuration space of the SMBUS device but its seems the CPU never sees this interrupt.

      The interrupt counters in '/proc/interrupts' never get updated and i2c operations timeout, see printouts below.

       

      As shown below, i enabled the apic=debug via the kernel command line but it seems the registers displayed are the values

      before the SMBUS driver gets a chance to allocate its interrupt.

      Do you know of any utilities/commands to display the local-APIC and IO-APIC contents?

      Any ideas why processor is not seeing the interrupt? Perhaps, there is a MSR type register in the CPU core that needs to be enabled,

      sort of like a global interrupt enabled bit??

       

      I would appreciate your help in this matter.

       

      thank you,

       

       

      #####

      :~$ cat /proc/interrupts

                 CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7      

        0:         11          0          0          0          0          0          0          0  IR-IO-APIC   2-edge      timer

        4:       5399          0          0          0          0          0          0          0  IR-IO-APIC   4-edge      serial

        9:          0          0          0          0          0          0          0          0  IR-IO-APIC   9-fasteoi   acpi

      1. 18:          0          0          0          0          0          0          0          0  IR-IO-APIC  18-fasteoi   i801_smbus

      24:       3535          0          0          0          0          0          0          0  IR-PCI-MSI 512000-edge      ahci[0000:00:1f.2]

      25:          1          0          0          0          0          0          0          0  IR-PCI-MSI 2621440-edge      eth0

      26:      65305          0          0          0          0          0          0          0  IR-PCI-MSI 2621441-edge      eth0-TxRx-0

      ########

       

      The PCI config space of the SMBUS device shows interrupt pending on PCISTST register (bit3, offset6-7):

       

      ########

      :~$ sudo lspci -s 0000:00:1f.3 -vv -xxx -k

      00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)

          Subsystem: Intel Corporation Device 7270

          Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

          Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+

          Interrupt: pin C routed to IRQ 18

          Region 0: Memory at fe41a000 (64-bit, non-prefetchable) [size=256]

          Region 4: I/O ports at 2020 [size=32]

          Kernel driver in use: i801_smbus

      00: 86 80 22 8c 03 00 88 02 05 00 05 0c 00 00 00 00

      10: 04 a0 41 fe 00 00 00 00 00 00 00 00 00 00 00 00

      20: 21 20 00 00 00 00 00 00 00 00 00 00 86 80 70 72

      30: 00 00 00 00 00 00 00 00 00 00 00 00 07 03 00 00

      40: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      60: 03 04 04 00 00 00 08 08 00 00 00 00 00 00 00 00

      70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      80: 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      f0: 00 00 00 00 00 00 00 00 b1 0f 06 08 00 00 00 00

      #########

       

      The device driver (i801_smbus) times-out with any I2c operation:

       

      #########

      :~$ sudo i2cdetect -y 0

           0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f

      00:          i801_smbus 0000:00:1f.3: Timeout waiting for interrupt!

      i801_smbus 0000:00:1f.3: Transaction timeout

      -- i801_smbus 0000:00:1f.3: Timeout waiting for interrupt!

      i801_smbus 0000:00:1f.3: Transaction timeout

      -- i801_smbus 0000:00:1f.3: Timeout waiting for interrupt!

      i801_smbus 0000:00:1f.3: Transaction timeout

      -- i801_smbus 0000:00:1f.3: Timeout waiting for interrupt!

      i801_smbus 0000:00:1f.3: Transaction timeout

      -- i801_smbus 0000:00:1f.3: Timeout waiting for interrupt!

      #############

       

      Following are details of the platform:

       

      ############

      :~$ inxi -F

      System:    Host: debian Kernel: 4.9.8-platina-mk1-amd64 x86_64 (64 bit)

                 Console: tty 0 Distro: Debian GNU/Linux 8

      Machine:   Mobo: Intel model: Camelback Mountain Platina DC v: 1.0

                 Bios: coreboot v: 6a724f1-dirty date: 03/14/2017

      CPU:       Quad core Intel Xeon D-1527 (-HT-MCP-) cache: 6144 KB

                 Clock Speeds: 1: 2194 MHz 2: 2194 MHz 3: 2194 MHz 4: 2194 MHz

                 5: 2194 MHz 6: 2194 MHz 7: 2194 MHz 8: 2194 MHz

      Graphics:  Card: Failed to Detect Video Card!

                 Display Server: N/A driver: N/A

                 tty size: 80x24 Advanced Data: N/A out of X

      Network:   Card-1: Broadcom Device b960

                 IF: N/A state: N/A speed: N/A duplex: N/A mac: N/A

                 Card-2: Broadcom Device b960

                 IF: N/A state: N/A speed: N/A duplex: N/A mac: N/A

                 Card-3: Intel Device 15ab

                 IF: N/A state: N/A speed: N/A duplex: N/A mac: N/A

                 Card-4: Intel Device 15ab

                 IF: N/A state: N/A speed: N/A duplex: N/A mac: N/A

                 Card-5: Intel I210 Gigabit Network Connection driver: igb

                 IF: eth0 state: up speed: 1000 Mbps duplex: full

                 mac: 02:46:8a:00:0b:86

      Drives:    HDD Total Size: 128.0GB (17.9% used)

                 ID-1: /dev/sda model: SanDisk_SD8SMAT1 size: 128.0GB

      Partition: ID-1: / size: 102G used: 6.2G (7%) fs: ext4 dev: /dev/sda2

                 ID-2: swap-1 size: 17.06GB used: 0.00GB (0%) fs: swap dev: /dev/sda3

      RAID:      No RAID devices: /proc/mdstat, md_mod kernel module present

      Sensors:   None detected - is lm-sensors installed and configured?

      Info:      Processes: 110 Uptime: 6:22 Memory: 69.7/16014.7MB

                 Init: systemd runlevel: 5 Client: Shell (bash) inxi: 2.1.28

      ########

       

      Enabling the "apic=debug" gives a view of the Local-APIC and IO-APIC during initialization, see 'dmesg' log below.

      It seems this view is displayed before the i801_smbus has a chance to allocation the interrupt via its request_irq() system call

      since pin12 (irq12 = 18decimal) is disabled.

       

      ######

      [2.079044] printing local APIC contents on CPU#0/0:
      [2.079046] ... APIC ID:  00000000 (0)
      [2.082155] ... APIC VERSION: 01060015
      [2.084882] ... APIC TASKPRI: 00000000 (00)
      [2.084883] ... APIC PROCPRI: 00000000
      [2.084885] ... APIC LDR: 00000001
      [2.084886] ... APIC SPIV: 000001ff
      [2.084887] ... APIC ISR field:
      [2.084890] 0000000000000000000000000000000000000000000000000000000000000000
      [2.084899] ... APIC TMR field:
      [2.084901] 0000000000000000000000000000000000000000000000000000000000000000
      [2.084910] ... APIC IRR field:
      [2.084912] 0000000000000000000000000000000000000000000000000000000000008000
      [2.084921] ... APIC ESR: 00000000
      [2.084923] ... APIC ICR: 000008fb
      [2.084924] ... APIC ICR2: 00000004
      [2.084926] ... APIC LVTT: 000400ef
      [2.084927] ... APIC LVTPC: 00000400
      [2.084928] ... APIC LVT0: 00010700
      [2.084930] ... APIC LVT1: 00000400
      [2.084931] ... APIC LVTERR: 000000fe
      [2.084932] ... APIC TMICT: 00000000
      [2.084934] ... APIC TMCCT: 00000000
      [2.084935] ... APIC TDCR: 00000000
      [2.084947] number of MP IRQ sources: 15.
      [2.084949] number of IO-APIC #8 registers: 24.
      [2.084951] number of IO-APIC #9 registers: 24.
      [2.084952] testing the IO APIC.......................
      [2.089246] IO APIC #8......
      [2.089247] .... register #00: 08000000
      [2.089248] .......: physical APIC id: 08
      [2.089250] .......: Delivery Type: 0
      [2.089251] .......: LTS      : 0
      [2.089252] .... register #01: 00170020
      [2.089253] ....... : max redirection entries: 17
      [2.089255] ....... : PRQ implemented: 0
      [2.089256] ....... : IO APIC version: 20
      [2.089257] .... IRQ redirection table:
      [2.089258] IOAPIC 0:
      [2.089266]  pin00, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089272]  pin01, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089278]  pin02, enabled , edge , high, V(02), IRR(0), S(0), remapped, I(8001),  Z(0)
      [2.089284]  pin03, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089290]  pin04, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089295]  pin05, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089301]  pin06, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089306]  pin07, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089312]  pin08, disabled, edge , high, V(C1), IRR(0), S(0), physical, D(5B), M(2)
      [2.089318]  pin09, enabled , level, high, V(09), IRR(0), S(0), remapped, I(40008),  Z(0)
      [2.089323]  pin0a, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089329]  pin0b, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089334]  pin0c, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089340]  pin0d, disabled, edge , high, V(59), IRR(0), S(0), remapped, I(3210642),  Z(2)
      [2.089346]  pin0e, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089351]  pin0f, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089357]  pin10, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089362]  pin11, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089368] pin12, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089373]  pin13, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089379]  pin14, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089384]  pin15, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089390]  pin16, disabled, edge , high, V(1C), IRR(0), S(0), logical , D(FC), M(2)
      [2.089396]  pin17, disabled, edge , high, V(BB), IRR(0), S(0), remapped, I(35CE6B9C),  Z(2)
      [2.089399] IO APIC #9......
      [2.089400] .... register #00: 09000000
      [2.089402] .......: physical APIC id: 09
      [2.089403] .......: Delivery Type: 0
      [2.089404] .......: LTS      : 0
      [2.089405] .... register #01: 00170020
      [2.089406] ....... : max redirection entries: 17
      [2.089407] ....... : PRQ implemented: 0
      [2.089409] ....... : IO APIC version: 20
      [2.089410] .... register #02: 00000000
      [2.089411] ....... : arbitration: 00
      [2.089412] .... register #03: 00000001
      [2.089413] ....... : Boot DT: 1
      [2.089414] .... IRQ redirection table:
      [2.089415] IOAPIC 1:
      [2.089420]  pin00, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089424]  pin01, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089428]  pin02, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089432]  pin03, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089436]  pin04, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089440]  pin05, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089444]  pin06, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089448]  pin07, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089452]  pin08, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089455]  pin09, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089459]  pin0a, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089464]  pin0b, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089467]  pin0c, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089471]  pin0d, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089475]  pin0e, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089479]  pin0f, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089483]  pin10, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089487]  pin11, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089491]  pin12, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089495]  pin13, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089499]  pin14, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089503]  pin15, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089507]  pin16, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089511]  pin17, disabled, edge , high, V(00), IRR(0), S(0), physical, D(00), M(0)
      [2.089512] IRQ to pin mappings:
      [2.089514] IRQ0 -> 0:2
      [2.089518] IRQ1 -> 0:1
      [2.089521] IRQ3 -> 0:3
      [2.089524] IRQ4 -> 0:4
      [2.089527] IRQ5 -> 0:5
      [2.089530] IRQ6 -> 0:6
      [2.089533] IRQ7 -> 0:7
      [2.089536] IRQ8 -> 0:8
      [2.089539] IRQ9 -> 0:9
      [2.089542] IRQ10 -> 0:10
      [2.089545] IRQ11 -> 0:11
      [2.089548] IRQ12 -> 0:12
      [2.089551] IRQ13 -> 0:13
      [2.089554] IRQ14 -> 0:14
      [2.089557] IRQ15 -> 0:15
      [2.089560] IRQ16 -> 0:16
      [2.089562] IRQ19 -> 0:19
      [2.089566] .................................... done.

      ###########

       

       

       

       

        • Re: i801_smbus interrupt timeout on Broadwell-DE
          AdolfoS Brown Belt

          Hello, aureliorio,

           

           

          Please try to reproduce the issue on a different Linux distribution, to check that this issue is not OS dependent.

           

           

          Could you please clarify if the board is manufactured by your company or by a third party vendor?

           

           

          Have you tried to increase the dmesg buffer size to see if you can check more information?

          If not try with "log_buf_len=n" where n is the size of the buffer.

           

           

          Best regards,

          Adolfo Sanchez

            • Re: i801_smbus interrupt timeout on Broadwell-DE
              arios Community Member

              Hi Adolfo,

              i tried an older kernel but it did the same thing, see below.

               

              And yes to your other question. The board was designed and manufactured by us.

              It is based on Intel's Camelback Mountain CRB.

               

              I have not tried increasing the size of dmesg buffer. Not sure what the default is.

              I will experiment with this....

               

              i tried adding the "enable_irq(dev->irq);" to the driver code, thinking that maybe it was not,

              but it did the same thing....

               

              thanks,

               

               

               

              :~$ inxi -F

              System:    Host: debian Kernel: 4.6.0-rc2-platina-mk1-amd64 x86_64 (64 bit)

                         Console: tty 0 Distro: Debian GNU/Linux 8

              Machine:   Mobo: Intel model: Camelback Mountain Platina DC v: 1.0

                         Bios: coreboot v: 6a724f1-dirty date: 03/14/2017

              CPU:       Quad core Intel Xeon D-1527 (-HT-MCP-) cache: 6144 KB

                         Clock Speeds: 1: 2194 MHz 2: 2194 MHz 3: 2194 MHz 4: 2194 MHz

                         5: 2194 MHz 6: 2194 MHz 7: 2194 MHz 8: 2194 MHz

                • Re: i801_smbus interrupt timeout on Broadwell-DE
                  AdolfoS Brown Belt

                  Hello, Aurelio,

                   

                  Thanks for your feedback.

                   

                  Could you please try with a different Linux distribution like Fedora or Ubuntu, and not just with an older kernel version of Debian?

                   

                  Please make sure that the microcode and firmware (sps, fsp) are updated to the latest version.

                   

                  I will be waiting for your feedback.

                   

                  Best regards,

                  Adolfo Sanchez

                    • Re: i801_smbus interrupt timeout on Broadwell-DE
                      arios Community Member

                      Hi Adolfo,

                       

                      I will try the ubutu but this might take a bit longer (i'm actually a hardware buy).

                      Our build environment is for debian and not sure how to build a ubuntu from scratch.

                      I figure I might need to modify I2c driver. So, better be able to build it from scratch.

                       

                      We are using the latest FSP (gold_001). How do i find the latest microcode (SPS?

                      we're using:

                       

                      CONFIG_CPU_MICROCODE_HEADER_FILES="3rdparty/intel-blobs/microcode/cpu/broadwell_de/M1050663_07000009.h 3rdparty/intel-blobs/microcode/cpu/broadwell_de/M1050662_0000000F.h 3rdparty/intel-blobs/microcode/cpu/broadwell_de/MFF50661_F1000008.h"

                        • Re: i801_smbus interrupt timeout on Broadwell-DE
                          AdolfoS Brown Belt

                          Hello, Aurelio,

                           

                           

                          Thanks for your feedback.

                           

                           

                          Could you please clarify what do you mean by your build environment? Are you customizing the Debian distro in some way to suit your board?

                           

                           

                          Is it possible for you to test with a Ubuntu Live USB? This way would not require any changes to the original OS of your board.

                           

                           

                          Are you developing your own firmware or you are working with a BIOS vendor?

                           

                           

                          Have you signed a CNDA with Intel?

                           

                           

                          To check the SPS code you can use the SPSInfo tool that comes with the SPS kit, if you don't have the kit, there are other third party tools available that can be used to read information from the spi. However since this are not officially approved by Intel I cannot recommend any in particular.

                           

                           

                          Best regards,

                          Adolfo Sanchez

                            • Re: i801_smbus interrupt timeout on Broadwell-DE
                              arios Community Member

                              Hi Adolfo,

                              Not much customization of Debian (Other than some new I2c driver/i2c interrupt handlers, perhaps). Sometimes we need to update its kernel. So, we're able to build it from scratch. The software team builds/develops our apps in this environment. I am task with coming up with a driver to handle SMBUS_ALERT interrupt going into the Broadwell-DE cpu and that's the reason I am playing with the I2c_i801 driver.

                               

                              Our product comes with Ubuntu 14.04 installed from our manufacturer. I played with it and it turns out Ubuntu is not using the i2c_i801 module. It is blacklisted in Ubuntu and never sets up any interrupts, see picture below. Ubuntu is using a alternative way for the i2c. Maybe, it is the i2c-algo-bit.ko module. in any case, it doesn't configure any interrupts.

                               

                              we're using open-source coreboot for the CPU BIOS.

                               

                              Yeah, i am pretty sure we have an NDA in place.

                               

                              I thought coreboot was loading the microcode?? maybe, that's something different. We're can i find info about the SPS code? i search datasheets/manuals but couldn't find any description.

                               

                              thanks,

                               

                                • Re: i801_smbus interrupt timeout on Broadwell-DE
                                  AdolfoS Brown Belt

                                  Hello, Aurelio,

                                   

                                   

                                  Regarding your consultation about the MCU, indeed coreboot loads the MCU, however it is necessary to provide the most uptodate MCU file to coreboot.

                                   

                                   

                                  As for SPS: "Server Platform Services(SPS) is an essential Firmware part of all Intel® Server platforms running inside PCH. The SPS Firmware not only provides service that allows the platform to boot, but also provides value added services to customers such as advanced power management on single and multi-node platforms, monitoring of CPU utilization, efficient thermal management assist capabilities, RAS Management (aka Dengate) and MCTP transport over PCIe."

                                   

                                   

                                  Maybe you are more familiar with the term Managemente Engine? It serves the same purpose.

                                   

                                   

                                  Are you the person in charge of the firmware part also? If not, I think you should consult this with your BIOS developer.

                                   

                                   

                                  For what I have seen the i2c-algo-bit.ko is dependent on a driver called i2c-core.ko that seems to have its own interrupt handler:http://lxr.free-electrons.com/source/drivers/i2c/i2c-core.c

                                   

                                   

                                  I would recommend also that you consult with the i2c driver developer community, here is the mailing list: linux-i2c@vger.kernel.org

                                   

                                   

                                  Best regards,

                                  Adolfo Sanchez