27 Replies Latest reply on Sep 25, 2015 9:56 AM by Varun

    Problems running DPDK KNI example

    Varun Green Belt

      I'm new to DPDK. I'm trying to learn about KNI. I compiled and fired up the KNI example provided with the library.

       

      • I used the ifconfig commands to bring up the vEth interfaces that spin up corresponding to each port and assign ipaddress and netmasks to them.
      • I tried to ping and hping (tcp ping) other machines on the same network.
      • Through tcpdump, I can see that the KNI interfaces send out the right frames but don't seem to be receiving anything.
      • I don't see any errors in dmesg or the console running the KNI test script.
      • I proceeded to try and debug by printing out mbufs entries. The KNI Rx doesn't seem find anything in the ring buffer corresponding to the Eth port which is supposed to receive the data/frames. However, the ifconfig messages do show up on the buffers and are accordingly acted upon by the kernel.

       

      It would be very helpful if someone can advise me on how to move forward with this and get this to work as it is supposed to. I have been spinning my wheels on this for a while. I have tried this on both VMs and physical machine with multiple network interfaces.

       

       

      Thanks

        • Re: Problems running DPDK KNI example
          gabriel.thomas Brown Belt

          Hello Varun,

           

          Welcome to the Intel Embedded Community.

           

          We are checking your case and will contact you as soon as possible.

           

          Best regards,

           

          Gabriel Thomas

          • Re: Problems running DPDK KNI example
            Carlos_A Brown Belt

            Hello Varun,

             

            Thank you for contacting the Embedded Design Community.

             

            The information that may help you is stated in the KNI libraryhttp://dpdk.org/doc/api/rte__kni_8h.htmlweb site or its source code.

             

            Please let us know if this information is useful to you.

             

            Best Regards,

            Carlos A.

              • Re: Problems running DPDK KNI example
                Varun Green Belt

                Hi Carlos,

                I did look at the documentation (links you provided). From what I can say, the example script should spin up virtual ethernet interfaces that handover frames to the Kernel network stack. I'm running an unmodified version of the KNI example that is included in the dpdk repo. I see that the interfaces send out traffic just fine but don't seem to be receiving frames. Do I understand the example script wrong?

                 

                Thanks

                Varun

                  • Re: Problems running DPDK KNI example
                    Carlos_A Brown Belt

                    Hello Varun,

                     

                    Thanks for your reply.

                     

                    We would like to address the following questions to better understand your consultation:

                     

                    Could you please tell us the DPDK version associated to the cited issue?

                     

                    Could you please tell us all the information of the Network Interface Cards (NICs) related to this situation?

                     

                     

                    Please let us know all the requested information to help you in the best way possible.

                     

                    Thanks in advance for your collaboration.

                     

                    Best Regards,

                    Carlos A.

                      • Re: Problems running DPDK KNI example
                        Varun Green Belt

                        I'm using DPDK-2.0.

                        Network interface cards: Intel 82580 Gigabit Network Connection.

                         

                        Does that help? Let me know if you need any more specific information.

                         

                        Thanks

                          • Re: Problems running DPDK KNI example
                            Carlos_A Brown Belt

                            Hello Varun,

                             

                            Thanks for your reply.

                             

                            Could you please confirm that the DPDK related to this situation is the stated at the following web site?

                             

                            dpdk - Data Plane Development Kit

                             

                             

                            Thanks again for your collaboration.

                             

                            Best Regards,

                            Carlos A.

                              • Re: Problems running DPDK KNI example
                                Varun Green Belt

                                Yes it is.

                                I downloaded it from this exact link: http://dpdk.org/browse/dpdk/snapshot/dpdk-2.1.0.tar.gz

                                  • Re: Problems running DPDK KNI example
                                    Carlos_A Brown Belt

                                    Hello Varun,

                                     

                                    Thanks for your reply.

                                     

                                    Actually the web site that you have mentioned is different that the one stated on our previous message, which is the following:

                                     

                                    http://dpdk.org/browse/dpdk/tree/drivers/net/e1000/

                                     

                                    Could you please try this and let us know if the problem persists?

                                     

                                    Thanks again for your cooperation.

                                     

                                    Best Regards,

                                    Carlos A.

                                      • Re: Problems running DPDK KNI example
                                        Varun Green Belt

                                        Oh ok.. didn't know that.. will try this out and let you know..

                                        • Re: Problems running DPDK KNI example
                                          Varun Green Belt

                                          I can just clone it from here right?

                                          git://dpdk.org/dpdk

                                          • Re: Problems running DPDK KNI example
                                            Varun Green Belt

                                            I just tried out the branch you pointed me at. seems like I still have the same issue.

                                              • Re: Problems running DPDK KNI example
                                                muthurajanjayakumar Green Belt

                                                Thanks Carlos Thanks Varun

                                                Our design engineer asked to check on the following please:

                                                You might have already done some of these.

                                                Kindly can we request you to let us know your input for these questions.

                                                In case if you need some more clarifications in the following questions, please do not hesitate to let us know.

                                                 

                                                Thank you so much Carlos.

                                                 

                                                 

                                                 

                                                 

                                                • What KNI mode is used? lo_mode_none, lo_mode_fifo, lo_mode_fifo_skb modes are used for performance analysis so should not be used for this case. Only normal mode works.
                                                • For the test you performed from vEth to remote server, have you tried to add kernel debug information in kni_net_tx() to see whether it is invoked by TCP/IP stack?
                                                • Can you ping from another server to the IP of KNI vEth? When KNI works in normal mode, "netif_rx(skb);" in kni_net_rx_normal() should be invoked to push DPDK received packets to TCP/IP stack routed to KNI vEth.
                                                • Can you send the scripts as well as network topology for us to diagnose?

                                                  

                                                  • Re: Problems running DPDK KNI example
                                                    Varun Green Belt
                                                    • I'm running KNI in normal mode. When I insmod the module, I do not pass any arguments. Also the code I'm trying to run is the KNI example script provided with DPDK. As far as I know the code, it doesn't do anything fancy with the mode.
                                                    • Yes I have. Trying to reach remote server from vEth, the kernel TCP/IP stack is invoked and the traffic is seen at the remote server. The remote server responds, which seems to be lost. I tried debugging the methods which poll the interface buffer for new frames, and seems like they never find anything new in the ring buffer.
                                                    • No, I can't ping from a remote server to the vEth interface. I can see the remote server send the frames out, but that seems to be never received by the vEth interface. I put debug instructions in the code where it polls the ring buffer at the physical interface for new received frames. From what I can see there is never anything new there, except when I issue PMD commands through ifconfig.
                                                    • I'm using the short and simple KNI example script provided in the DPDK repo. This one: dpdk - Data Plane Development Kit. The network topology is pretty simple. I have an interface on my machine controlled by DPDK KNI, that interface is physically connected by wire to another linux machine. I'm trying to ping one interface from the other. I know the network setup should work because when I pull out the interface from DPDK control and hand it over to the kernel, I can ping it just fine.

                                                     

                                                    Hope that answered your questions. Let me know if you need more information.

                                                    Thanks a lot for helping me with this.

                                                     

                                                    Varun

                                                      • Re: Problems running DPDK KNI example
                                                        Carlos_A Brown Belt

                                                        Hello Varun,

                                                        Thanks for your reply.

                                                         

                                                        It seems that the promiscuous mode is off, then the NIC drops packet, make sure the “-P” flag is set when starting kni application.

                                                        kni [EAL options] -- -P -p PORTMASK --config="(port,lcore_rx,lcore_tx[,lcore_kthread,...])”

                                                         

                                                         

                                                        In case that the issue persists, please check is if iommu=pt kernel parameter is used when using igb_uio. Please refer to the information stated in section 5.6, on page 19 of the DPDK Getting Started Guide for Linux Release 2.0.0.

                                                         

                                                        Please let us know if this information is useful to you.

                                                         

                                                        Best Regards,
                                                        Carlos A.

                                                          • Re: Problems running DPDK KNI example
                                                            Varun Green Belt

                                                            I have already tried all 3 of those. However just to make sure I will try again with the new repo that you provided me. And keep you updated.

                                                             

                                                            Thanks

                                                            • Re: Problems running DPDK KNI example
                                                              Varun Green Belt

                                                              I went over all the suggestions again and it didn't work work me.

                                                              any other suggestions?

                                                                • Re: Problems running DPDK KNI example
                                                                  Carlos_A Brown Belt

                                                                  Hello Varun,

                                                                   

                                                                  Thanks for your reply.

                                                                   

                                                                  Could you please run testpmd (one in tx_only mode) on two servers to see whether it works or not? Please give us a detailed description of the results.

                                                                   

                                                                  Thanks again for your collaboration to solve this inconvenience.

                                                                   

                                                                  Best Regards,

                                                                  Carlos A.

                                                                    • Re: Problems running DPDK KNI example
                                                                      Varun Green Belt

                                                                      I'm not sure how to do that. Can you give me a little more detailed instructions? This is the first time I'm doing anything with DPDK.

                                                                       

                                                                      I ran "sudo ./testpmd -c 3 -n 2" and got the following result:

                                                                      (NOTE: I have 4 interfaces controlled by DPDK, but only one is connecting out to my other box, that is the one you can see is live)

                                                                       

                                                                      EAL: Requesting 2048 pages of size 2MB from socket 0

                                                                      EAL: TSC frequency is ~3292523 KHz

                                                                      EAL: Master lcore 0 is ready (tid=11324900;cpuset=[0])

                                                                      EAL: lcore 1 is ready (tid=a7e7700;cpuset=[1])

                                                                      EAL: PCI device 0000:03:00.0 on NUMA socket -1

                                                                      EAL:   probe driver: 8086:150e rte_igb_pmd

                                                                      EAL:   PCI memory mapped at 0x7fc911271000

                                                                      EAL:   PCI memory mapped at 0x7fc91126d000

                                                                      PMD: eth_igb_dev_init(): port_id 0 vendorID=0x8086 deviceID=0x150e

                                                                      EAL: PCI device 0000:03:00.1 on NUMA socket -1

                                                                      EAL:   probe driver: 8086:150e rte_igb_pmd

                                                                      EAL:   PCI memory mapped at 0x7fc9111ed000

                                                                      EAL:   PCI memory mapped at 0x7fc9111e9000

                                                                      PMD: eth_igb_dev_init(): port_id 1 vendorID=0x8086 deviceID=0x150e

                                                                      EAL: PCI device 0000:03:00.2 on NUMA socket -1

                                                                      EAL:   probe driver: 8086:150e rte_igb_pmd

                                                                      EAL:   PCI memory mapped at 0x7fc911169000

                                                                      EAL:   PCI memory mapped at 0x7fc911165000

                                                                      PMD: eth_igb_dev_init(): port_id 2 vendorID=0x8086 deviceID=0x150e

                                                                      EAL: PCI device 0000:03:00.3 on NUMA socket -1

                                                                      EAL:   probe driver: 8086:150e rte_igb_pmd

                                                                      EAL:   PCI memory mapped at 0x7fc8b3b80000

                                                                      EAL:   PCI memory mapped at 0x7fc911161000

                                                                      PMD: eth_igb_dev_init(): port_id 3 vendorID=0x8086 deviceID=0x150e

                                                                      Configuring Port 0 (socket 0)

                                                                      PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance, consider setting the TX WTHRESH value to 4, 8, or 16.

                                                                      PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fc813ded9c0 hw_ring=0x7fc813defa00 dma_addr=0x21b1efa00

                                                                      PMD: eth_igb_rx_queue_setup(): sw_ring=0x7fc813ddd440 hw_ring=0x7fc813ddd880 dma_addr=0x21b1dd880

                                                                      PMD: eth_igb_start(): <<

                                                                      Port 0: 90:E2:BA:0D:EE:5C

                                                                      Configuring Port 1 (socket 0)

                                                                      PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance, consider setting the TX WTHRESH value to 4, 8, or 16.

                                                                      PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fc813dcb1c0 hw_ring=0x7fc813dcd200 dma_addr=0x21b1cd200

                                                                      PMD: eth_igb_rx_queue_setup(): sw_ring=0x7fc813dbac40 hw_ring=0x7fc813dbb080 dma_addr=0x21b1bb080

                                                                      PMD: eth_igb_start(): <<

                                                                      Port 1: 90:E2:BA:0D:EE:5D

                                                                      Configuring Port 2 (socket 0)

                                                                      PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance, consider setting the TX WTHRESH value to 4, 8, or 16.

                                                                      PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fc813da89c0 hw_ring=0x7fc813daaa00 dma_addr=0x21b1aaa00

                                                                      PMD: eth_igb_rx_queue_setup(): sw_ring=0x7fc813d98440 hw_ring=0x7fc813d98880 dma_addr=0x21b198880

                                                                      PMD: eth_igb_start(): <<

                                                                      Port 2: 90:E2:BA:0D:EE:5E

                                                                      Configuring Port 3 (socket 0)

                                                                      PMD: eth_igb_tx_queue_setup(): To improve 1G driver performance, consider setting the TX WTHRESH value to 4, 8, or 16.

                                                                      PMD: eth_igb_tx_queue_setup(): sw_ring=0x7fc813d861c0 hw_ring=0x7fc813d88200 dma_addr=0x21b188200

                                                                      PMD: eth_igb_rx_queue_setup(): sw_ring=0x7fc813d75c40 hw_ring=0x7fc813d76080 dma_addr=0x21b176080

                                                                      PMD: eth_igb_start(): <<

                                                                      Port 3: 90:E2:BA:0D:EE:5F

                                                                      Checking link statuses...

                                                                      Port 0 Link Up - speed 1000 Mbps - full-duplex

                                                                      Port 1 Link Down

                                                                      Port 2 Link Down

                                                                      Port 3 Link Down

                                                                      Done

                                                                      No commandline core given, start packet forwarding

                                                                        io packet forwarding - CRC stripping disabled - packets/burst=32

                                                                        nb forwarding cores=1 - nb forwarding ports=4

                                                                        RX queues=1 - RX desc=128 - RX free threshold=32

                                                                        RX threshold registers: pthresh=8 hthresh=8 wthresh=0

                                                                        TX queues=1 - TX desc=512 - TX free threshold=0

                                                                        TX threshold registers: pthresh=32 hthresh=0 wthresh=0

                                                                        TX RS bit threshold=0 - TXQ flags=0x0

                                                                      Press enter to exit

                                                                        • Re: Problems running DPDK KNI example
                                                                          Carlos_A Brown Belt

                                                                          Hello Varun,

                                                                          Could you please tell us what is the DPDK version and Operating System (OS) related to your last communication?

                                                                          Thanks again for your cooperation to solve this inconvenience.

                                                                          Best Regards,
                                                                          Carlos A.


                                                                          Engineer is under contract to Intel.

                                                                            • Re: Problems running DPDK KNI example
                                                                              Varun Green Belt

                                                                              I'm using DPDK checked out from dpdk - Data Plane Development Kit

                                                                               

                                                                              The OS is Ubuntu 14.04.

                                                                                • Re: Problems running DPDK KNI example
                                                                                  Carlos_A Brown Belt

                                                                                  Hello Varun,

                                                                                   

                                                                                  Thanks for your reply.

                                                                                   

                                                                                  Could you please tell us what is the NIC SKU that you are using? Because it seems that you are using a 1 G NIC instead of  a 10 G.

                                                                                   

                                                                                  By the way, we suggest you to submit this issue through DPDK.org, which is the official channel for DPDK technical consultations.

                                                                                  Thanks again for your cooperation to solve this inconvenience.

                                                                                  Best Regards,
                                                                                  Carlos A.

                                                                                    • Re: Problems running DPDK KNI example
                                                                                      Varun Green Belt

                                                                                      Hi Carlos,

                                                                                      I'm out of town this week. So don't have the SKU handy with me. I will get back to you as soon as I get back. But yes, I'm using 1G NIC.

                                                                                       

                                                                                      When you say DPDK.org, do you mean the mailing lists?

                                                                                       

                                                                                      Thanks

                                                                                      Varun

                                                                                        • Re: Problems running DPDK KNI example
                                                                                          Carlos_A Brown Belt

                                                                                          Hello Varun,

                                                                                           

                                                                                          Thanks for your reply.

                                                                                           

                                                                                          Could you please tell us NIC device ID, kernel version, platform, and DPDK version?

                                                                                           

                                                                                          Thanks again for your cooperation to solve this inconvenience.

                                                                                           

                                                                                          Best Regards,
                                                                                          Carlos A.

                                                                                           

                                                                                            • Re: Problems running DPDK KNI example
                                                                                              Varun Green Belt

                                                                                              Hi Carlos,

                                                                                              Sorry for the late reply. I have been out of town with spotty internet connection. The info you required:

                                                                                              NIC Device: Ethernet controller: Intel Corporation 82580 Gigabit Network Connection

                                                                                              Kernel: 3.13.0-62-generic

                                                                                              platform: Ubuntu 14.04 64 bit

                                                                                              DPDK version: 2.1.0

                                                                                               

                                                                                              Does that help?

                                                                                               

                                                                                              Thanks

                                                                                              Varun

                                                                                                • Re: Problems running DPDK KNI example
                                                                                                  Carlos_A Brown Belt

                                                                                                  Hello Varun,

                                                                                                   

                                                                                                  Thanks for your reply.

                                                                                                   

                                                                                                  Actually KNI is independent of any specific NIC, as it is just a channel between user space and kernel space.

                                                                                                   

                                                                                                  On our system, the kernel bridge works well with using X710 (i40e). We have created two KNI instances in kernel space, which added to a bridge (brctl addbr <>), then it can receive packets from user space and then forward the packets back from another KNI instance.

                                                                                                   

                                                                                                  Due to these facts, we would like to address the following questions:

                                                                                                   

                                                                                                  Could you please check if KNI works well with other NICs (I350, 82599, X710 etc.) on your board?

                                                                                                   

                                                                                                  Could you please verify that the testpmd or l2fwd, l3fwd works with the 82580?

                                                                                                   

                                                                                                  Could you please tell us if you have previous experience with KNI?

                                                                                                   

                                                                                                  By the way, customers sometimes are struggling on using KNI with network managers. Generally we just disable network manager before running KNI application.

                                                                                                   

                                                                                                  Thanks again for your cooperation to solve this inconvenience.

                                                                                                   

                                                                                                  Best Regards,

                                                                                                  Carlos_A

                                                                                                    • Re: Problems running DPDK KNI example
                                                                                                      Varun Green Belt

                                                                                                      HI Carlos,

                                                                                                      Thanks for the reply. I will try out the steps you recommended. I'm running a little busy, so I will get back to you in a couple of days.

                                                                                                       

                                                                                                      On you question about my experience with KNI, I don't have any experience with KNI or even DPDK. This is the first project I undertook.

                                                                                                       

                                                                                                      Thanks

                                                                                                      Varun