8 Replies Latest reply on Oct 24, 2014 8:42 AM by Danny

    Need help on DPDK uses Linux hugepages

    HuBugui Community Member

      rte_mempool_create needs continues memory and we use 2MB hugepage. We want to get 1GB continues memory or more, currently want 1GB.

       

      We tried reserve 1024 pages of 2 MB and dpdk code for 32-bit (currently we are) limited to using 1 GB so only takes the first 512 pages:

      #ifndef RTE_ARCH_X86_64

        /* for 32-bit systems, limit number of hugepages to 1GB per page size */

        hpi->num_pages[0] = RTE_MIN(hpi->num_pages[0],

        RTE_PGSIZE_1G / hpi->hugepage_sz);

      #endif

      And we found this 512 pages will be continues so we get 1 GB continues memory. But the other 512 pages will be waste since our application will not use it.

       

      Then we try change to reserve only 512 pages, but this time we cannot get the 1GB continues. Actually even we reserve 850 pages or 750 pages some number smaller than 1024 pages, we cannot got 1 GB continues memory.

       

      Any one know why? And any way to help it? We only want 1GB continues memory but we do not want to over reserve pages.

        • Re: Need help on DPDK uses Linux hugepages
          Leon Brown Belt

          Hi HuBugui,

           

          Welcome to the Intel® Embedded Community.

           

          We are working on getting an answer to your question. Have a great day and we'll be talking with you soon! 

           

           

          Best Regards,

          Leon

          • Re: Need help on DPDK uses Linux hugepages
            muthurajanjayakumar Green Belt

            Hi
            Thank you for using DPDK.

             

            A General observation regarding huge page -

             

            The allocation of hugepages should be done at boot time or as soon as possible after system boot to prevent memory from being fragmented in physical memory. To reserve hugepages at boot time, a parameter is passed to the Linux* kernel on the kernel command line.

             

            As you may be aware of, there are 2 Huge page sizes Intel Processor supports  - 1) 2 Meg and 2) 1 Gig

            Since you want 1 Gig continuous memory, can we please recommend you to use 1 Gig Huge page size (instead of 2 Meg)

            Specify in the GRUB for 1 Page of 1 Gig Huge page .

             

            For 1G pages, the size must be specified explicitly and can also be optionally set as the default hugepage size for the system. For example, to reserve 1G of hugepage memory in the form of one 1G page, the following options should be passed to the kernel:

             

            default_hugepagesz=1G hugepagesz=1G hugepages=1

             

            Please kindly refer to section 2.3 and subsections there of in http://dpdk.org/doc/intel/dpdk-start-linux-1.7.0.pdf

             

            Also, let us know the following information

             

            1) What is your kernel version?

            2) How many sockets your machne has?

            3) Plese let us know the output of cat /proc/cpuinfo

             

            Thanks,

              • Re: Need help on DPDK uses Linux hugepages
                HuBugui Community Member

                Hi,

                 

                Thanks a lot for kindly help.

                 

                We are running virtual machine on KVM. Kernel version is 2.6.32-220.el6.adx.x86_64.

                It seems KVM guest unable to get pdpe1gb flag from host CPU, so it does not support 1GB hugepage size. :-(

                I found someone also saw this issue: Bug #1248959 “pdpe1gb flag is missing in guest running on Intel ...” : Bugs : QEMU ()

                Even in KVM I set "Copy host CPU configuration" and saw pdpe1gb flag is showed as "require".

                 

                The guest is 1 socket for 2 cores.

                 

                [root@VirtualADX ~]# cat /proc/cpuinfo

                processor       : 0

                vendor_id       : GenuineIntel

                cpu family      : 6

                model           : 42

                model name      : Intel Xeon E312xx (Sandy Bridge)

                stepping        : 1

                cpu MHz         : 2594.210

                cache size      : 4096 KB

                fpu             : yes

                fpu_exception   : yes

                cpuid level     : 13

                wp              : yes

                flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx f16c rdrand hypervisor lahf_lm xsaveopt fsgsbase smep

                bogomips        : 5188.42

                clflush size    : 64

                cache_alignment : 64

                address sizes   : 46 bits physical, 48 bits virtual

                power management:

                 

                 

                processor       : 1

                vendor_id       : GenuineIntel

                cpu family      : 6

                model           : 42

                model name      : Intel Xeon E312xx (Sandy Bridge)

                stepping        : 1

                cpu MHz         : 2594.210

                cache size      : 4096 KB

                fpu             : yes

                fpu_exception   : yes

                cpuid level     : 13

                wp              : yes

                flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx f16c rdrand hypervisor lahf_lm xsaveopt fsgsbase smep

                bogomips        : 5188.42

                clflush size    : 64

                cache_alignment : 64

                address sizes   : 46 bits physical, 48 bits virtual

                power management:

                 

                Thanks

                  • Re: Need help on DPDK uses Linux hugepages
                    HuBugui Community Member

                    The host is 12 core machine with same kernel. But it may change.

                    [root@centos-139277 ~]# cat /proc/cpuinfo

                    processor       : 0

                    vendor_id       : GenuineIntel

                    cpu family      : 6

                    model           : 62

                    model name      : Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz

                    stepping        : 4

                    cpu MHz         : 1200.000

                    cache size      : 15360 KB

                    physical id     : 0

                    siblings        : 12

                    core id         : 0

                    cpu cores       : 6

                    apicid          : 0

                    initial apicid  : 0

                    fpu             : yes

                    fpu_exception   : yes

                    cpuid level     : 13

                    wp              : yes

                    flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 x2apic popcnt aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms

                    bogomips        : 5188.42

                    clflush size    : 64

                    cache_alignment : 64

                    address sizes   : 46 bits physical, 48 bits virtual

                    power management:

                      • Re: Need help on DPDK uses Linux hugepages
                        LynnZ Brown Belt

                        Hi!  Would you post your qemu/kvm command-lines to start VMs, and make sure you used the options below to assign host huge pages to guest as guest huge pages.  -memm-paht /mnt/huge -mem-prealloc ?

                          • Re: Need help on DPDK uses Linux hugepages
                            HuBugui Community Member

                            I tried using virt-manager and also command line qemu-kvm with mem-path and mem-prealloc, it seems not help:

                            On host:

                            HugePages_Total:       6

                            HugePages_Free:        3

                            HugePages_Rsvd:        1

                            HugePages_Surp:        0

                            Hugepagesize:    1048576 kB

                            DirectMap4k:        6144 kB

                            DirectMap2M:     2058240 kB

                            DirectMap1G:    14680064 kB

                             

                            start guest:

                            /usr/libexec/qemu-kvm -cpu host -drive file=/root/img/centtos.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1  -m 4096 -smp 2,cores=1,threads=1,sockets=2 --enable-kvm -name "os2" -nographic -vnc :3 -monitor unix:/tmp/vm1monitor,server,nowait -net none -no-reboot -mem-path /mnt/huge_1GB -mem-prealloc -netdev type=tap,id=net1,script=no,downscript=no,ifname=ovsvhost80,vhost=on -device  virtio-net-pci,netdev=net1,mac=00:00:00:00:00:01,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off  -netdev type=tap,id=net2,script=no,downscript=no,ifname=ovsvhost81,vhost=on -device virtio-net-pci,netdev=net2,mac=00:00:00:00:00:02,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off

                             

                            The guest using 2M hugepage with 512 pages, still only get 313 pages continues memory.