10 Replies Latest reply on Jan 6, 2012 9:06 AM by george_mccollister

    EMGD . memory leak and streaming issue

    Community Member

      Hi,

      I am working on ATOM z530 +us15W chipset with Emgd enabled.I am succesfully ported EMGD 1.6 with debian and working well for dual sdvo and lvds display. when i tried to play some local video files using gstreamer its succefully plays..Then i have integrated splitted desktop vaapi library and trying to stream 720 p and 1080p. During this i have following  issues. can anyone please hellp me for solving the problems

       

      1. streaming 720p hangs after 5-10 mins .

      2. streaming 1080p hangs before 1 mins

      3.If streaming hangs and restarted using "ctrl+c" i am seeing a memory leak of <40Mb virtual memory(vmalloc).

      4. when i am trying local playback and closing using "ctl+c" inbetween i can recreate issue 3. This will happen only once in 10 times or so.

       

      I have one more question regarding memory allocation. is it possible to use static virtual memory instead of dynamic memory allocation every time? if possible how?

       

      Note:I am using vaapi from spliieted desktop and emgd1.6 with debian.

       

      Hope i will get a support.I am in a deliverble stage of my project and i am having really bad time with issue.Please help me ..

       

      Regards  and Thanks

      Thomas Scariah

        • Re: EMGD . memory leak and streaming issue
          Kirk Brown Belt

          Thomas,

           

          Does anyone at your company have an Intel Premier Support Account (we call it Quad internally)?  The is the best way to get support from us.  The EDC forum is a "best effort" type system.

           

          Is the reason you are using such an older version of EMGD because of support for an older kernel/X version?  Is there a way to try at least 1.8 version now shipping?

           

          Splitted Desktop sometimes provides VA API libraries beyond the recommneded ones for a particular driver.  I am thinking that for 1.6 you may need to use a version of VA API earlier (like .29) than the latest one (.31 or later) but the release notes with the EMGD driver should talk about it.

           

          If the VA API is crashing, it would not surprise me that memory allocations get left open- 40MB sounds like decode buffers that would not be closed properly if the driver panics.  After a crash like that, a reboot is always a good idea.

           

          What specific feature are you trying to use to "stream".  There are some interfaces that are not supported and if you are trying to use those, it might explain your issues (long shot as they should fail immediately).  I';ve been very successful taking video from ethernet from an E600 encoding video to another E600 decoding it to the screen.  Here is the command line we use to do this:

           

          TX to other

          gst-launch v4l2src ! videorate ! ffmpegcolorspace ! 'video/x-raw-yuv,width=640,height=480,framerate=30/1' ! MixVideoEncoder encode-format=video/mpeg profile=2 rate-control=CBR bit-rate=15000000 need-display=0 ! queue ! tcpclientsink host=192.168.0.108 port=5000 protocol=1


          TX to Self

          gst-launch v4l2src ! videorate ! ffmpegcolorspace ! 'video/x-raw-yuv,width=640,height=480,framerate=30/1' ! MixVideoEncoder encode-format=video/mpeg profile=2 rate-control=CBR bit-rate=15000000 need-display=0 ! queue ! tcpclientsink host=192.168.0.109 port=5000 protocol=1

           

          Receive from other

          gst-launch tcpserversrc host=192.168.0.109 port=5000 protocol=1 ! MixVideoDecoder ! MixVideoSink

           

          Obviously your IP Addresses and port addresses would be different- we hard code them in a stand-alone demo fashion...

           

          Hope this helps.

            • Re: EMGD . memory leak and streaming issue
              Community Member

              Hi Kirk/All,

              Thanks for your immediate reply. I am attaching error log from emgd with debug enabled..

              I am using ubuntu natty with emgd 1.6 and libva0.31 ,libav 0.6.2 & vaapi 0.2.5

              The sender is encoding video to h264 format(DM6467) and streaming.The receiver is decoding the frames using US15W hardware decoder

               

              I have tested the stream by decoding/playing using a standard PC+windows+vlc player and standard PC+ubuntu+gstreamer+ffdec_h264, and i am able to see the video and the setup is working for more than 6 hours On the Intel ATOM+ubuntu+gstremer+vaapi i am able to play the video but the system crashes after sometime(after 20min) with the below message

               

              [h264 @ 0x99d0c00]number of reference frames exceeds max (probably corrupt input), discarding one

                  Last message repeated 16 times

              [h264 @ 0x99d0c00]get_buffer() failed (-1 1 2 (nil))

              [h264 @ 0x99d0c00]decode_slice_header error

              [h264 @ 0x99d0c00]mmco: unref short failure

               

              if i restart gstreamer pilepline(without rebooting system) it works for some time and again crashes i also displays the message

               

               

              Since the Intel ATOM+ubuntu+gstremer+vaapi system is working for some thim it cannot be encoder-decoder incompatiability can you tell where cound be the problem and where should i check for the solution

               

              command on Intel ATOM+ubuntu+gstremer+vaapi for playing the stream gst-launch-0.10 gstrtpbin udpsrc port=5434 ! "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, profile-level-id=(string)428028, sprop-parameter-sets=(string)\"Z0KAKPQCgC2IAA\\=\\=\\,aN48gAA\\=\", payload=(int)96" ! rtpbin.recv_rtp_sink_0 rtpbin. ! rtph264depay ! "video/x-h264, framerate=(fraction)25/1, width=(int)1280, height=(int)720" !  vaapidecode ! vaapisink sync=false async=false

               

              Regards and Thanks

              Thomas Scariah

                • Re: EMGD . memory leak and streaming issue
                  Kirk Brown Belt

                  OK, that makes a bit more sense.  What is happening is that the decode is running out of memory in the middle of the decode process.  When that happens, Linux bails out on the driver.  You could likely correct this by making sure the graphics aperature setting in your bios sets it to the maximum (or at least 256MB) to allow the driver to allocate enough memory to the graphics.  You may also be running out of free runtime memory that may require you to add more ram to the system.

                   

                  The out of memory failure is not handled very elegantly, but much of that is out of our control.  Video decode, especially of high bit rate content done with less than efficient encoder is a very high memory use case.

                   

                  Hope that helps.

                    • Re: EMGD . memory leak and streaming issue
                      Community Member

                      Hi kirk,

                      I have ported 1.8 EMGD and checked the same.The issue still remains.

                      My aperture size setting is 256MB and RAM size is 1GB.Instead of streaming i tried playback and closed using ctrl+c. Sometime when closed using ctrl+c i am getting below error and if that error happens memory will get leaked(>40MB).

                      [h264 @ 0xa104de0]number of reference frames exceeds max (probably corrupt input), discarding one
                          Last message repeated 19 times
                      [h264 @ 0xa104de0]get_buffer() failed (-1 1 2 (nil))
                      [h264 @ 0xa104de0]decode_slice_header error

                       

                      Please find the log where  i have done the testing. Actually i had a privilage user access but i lost the password and i am not able to recover it .

                      Below is the playback test I have done and teh memory info FYI..please note this is a playback test only ..

                       

                      root@usbharddisk-lht:~# cat /proc/meminfo
                      MemTotal:         764316 kB
                      MemFree:          666224 kB
                      Buffers:            4712 kB
                      Cached:            28888 kB
                      SwapCached:            0 kB
                      Active:            19172 kB
                      Inactive:          24640 kB
                      Active(anon):      10308 kB
                      Inactive(anon):       28 kB
                      Active(file):       8864 kB
                      Inactive(file):    24612 kB
                      Unevictable:           0 kB
                      Mlocked:               0 kB
                      SwapTotal:             0 kB
                      SwapFree:              0 kB
                      Dirty:               144 kB
                      Writeback:             0 kB
                      AnonPages:         10164 kB
                      Mapped:            31032 kB
                      Shmem:               104 kB
                      Slab:               8868 kB
                      SReclaimable:       1956 kB
                      SUnreclaim:         6912 kB
                      KernelStack:         600 kB
                      PageTables:          568 kB
                      NFS_Unstable:          0 kB
                      Bounce:                0 kB
                      WritebackTmp:          0 kB
                      CommitLimit:      382156 kB
                      Committed_AS:     134776 kB
                      VmallocTotal:     262144 kB
                      VmallocUsed:       18936 kB
                      VmallocChunk:     239660 kB
                      DirectMap4k:       40020 kB
                      DirectMap4M:      737280 kB
                      root@usbharddisk-lht:/home/release/test_app# gst-launch-0.10 filesrc location=./Jeans.mp4 ! qtdemux name=demux demux.video_00 ! queue ! vaapidecode ! vaapisink demux.audio_00 ! queue ! faad ! alsasink device=plug:hdmi
                      libva: libva version 0.32.0
                      libva: va_getDriverName() returns 0
                      libva: Trying to open /usr/lib/dri/emgd_drv_video.so
                      Intel(R) Embedded Media and Graphics Driver 1.8 Build 2032
                      Using XCB based dispatch table.
                      libva: va_openDriver() returns 0
                      Setting pipeline to PAUSED ...
                      libva: libva version 0.32.0
                      libva: va_getDriverName() returns 0
                      libva: Trying to open /usr/lib/dri/emgd_drv_video.so
                      Intel(R) Embedded Media and Graphics Driver 1.8 Build 2032
                      Using XCB based dispatch table.
                      libva: va_openDriver() returns 0
                      Pipeline is PREROLLING ...
                      Pipeline is PREROLLED ...
                      Setting pipeline to PLAYING ...
                      New clock: GstAudioSinkClock
                      ^CCaught interrupt -- handling interrupt.
                      Interrupt: Stopping pipeline ...
                      Execution ended after 1752461761 ns.
                      Setting pipeline to PAUSED ...
                      Setting pipeline to READY ...
                      [h264 @ 0x95aee30]number of reference frames exceeds max (probably corrupt input), discarding one
                          Last message repeated 19 times
                      [h264 @ 0x95aee30]get_buffer() failed (-1 1 2 (nil))
                      [h264 @ 0x95aee30]decode_slice_header error
                      [h264 @ 0x95aee30]mmco: unref short failure
                      Setting pipeline to NULL ...
                      Freeing pipeline ...

                       

                      root@usbharddisk-lht:/home/release/test_app# cat /proc/meminfo
                      VmallocTotal:     262144 kB
                      VmallocUsed:       71704 kB
                      VmallocChunk:     181512 kB
                      root@usbharddisk-lht:/home/release/test_app# gst-launch-0.10 filesrc location=./Jeans.mp4 ! qtdemux name=demux demux.video_00 ! queue ! vaapidecode ! vaapisink demux.audio_00 ! queue ! faad ! alsasink device=plug:hdmi
                      libva: libva version 0.32.0
                      libva: va_getDriverName() returns 0
                      libva: Trying to open /usr/lib/dri/emgd_drv_video.so
                      Intel(R) Embedded Media and Graphics Driver 1.8 Build 2032
                      Using XCB based dispatch table.
                      libva: va_openDriver() returns 0
                      Setting pipeline to PAUSED ...
                      libva: libva version 0.32.0
                      libva: va_getDriverName() returns 0
                      libva: Trying to open /usr/lib/dri/emgd_drv_video.so
                      Intel(R) Embedded Media and Graphics Driver 1.8 Build 2032
                      Using XCB based dispatch table.
                      libva: va_openDriver() returns 0
                      Pipeline is PREROLLING ...
                      Pipeline is PREROLLED ...
                      Setting pipeline to PLAYING ...
                      New clock: GstAudioSinkClock
                      ^CCaught interrupt -- handling interrupt.
                      Interrupt: Stopping pipeline ...
                      Execution ended after 751977361 ns.
                      Setting pipeline to PAUSED ...
                      Setting pipeline to READY ...
                      [h264 @ 0xa104de0]number of reference frames exceeds max (probably corrupt input), discarding one
                          Last message repeated 19 times
                      [h264 @ 0xa104de0]get_buffer() failed (-1 1 2 (nil))
                      [h264 @ 0xa104de0]decode_slice_header error
                      [h264 @ 0xa104de0]mmco: unref short failure
                      Setting pipeline to NULL ...
                      Freeing pipeline ...
                      root@usbharddisk-lht:/home/release/test_app# cat /proc/meminfo
                      MemTotal:         764316 kB
                      MemFree:          545680 kB
                      Buffers:            5556 kB
                      Cached:            43692 kB
                      SwapCached:            0 kB
                      Active:            27664 kB
                      Inactive:          32228 kB
                      Active(anon):      10712 kB
                      Inactive(anon):       28 kB
                      Active(file):      16952 kB
                      Inactive(file):    32200 kB
                      Unevictable:           0 kB
                      Mlocked:               0 kB
                      SwapTotal:             0 kB
                      SwapFree:              0 kB
                      Dirty:                12 kB
                      Writeback:             0 kB
                      AnonPages:         10632 kB
                      Mapped:           133736 kB
                      Shmem:               104 kB
                      Slab:               9784 kB
                      SReclaimable:       2184 kB
                      SUnreclaim:         7600 kB
                      KernelStack:         608 kB
                      PageTables:          684 kB
                      NFS_Unstable:          0 kB
                      Bounce:                0 kB
                      WritebackTmp:          0 kB
                      CommitLimit:      382156 kB
                      Committed_AS:     134896 kB
                      VmallocTotal:     262144 kB
                      VmallocUsed:      121800 kB
                      VmallocChunk:     133936 kB
                      DirectMap4k:       93268 kB
                      DirectMap4M:      684032 kB

                       

                       

                      Regards and Thanks

                      Thomas Scariah

                        • Re: EMGD . memory leak and streaming issue
                          Community Member

                          Hi Krik,

                           

                          I am happy to say that issue is been solved by increasing the "buffer-size" property of "udp-src" element in gstreamer ..So I am able to stream 1080p and 720p smoothly . But still, if process is not closed smoothly we are ending up in memory leak .. I don't know how to solve this .. If you can help this it's very helpful...

                           

                           

                          Regards

                          Thomas Scariah

                            • Re: EMGD . memory leak and streaming issue
                              Kirk Brown Belt

                              That is excellent news.  So basically, Gstreamer was overflowing one of its buffers, causing a panic, and making the kernal shut it down.  Video decode takes a LOT of memory to do HD at high framerates as it is a lot of data.

                               

                              My guess is that the error then memory leak is because Gstreamer has had us allocate an internal buffer for the decode when it blows up and we do not know they do not need it anymore.  I wish there was a way to detect if a framework above the driver could know that it has failed, but unfortunately there is no Linux or Gstreamer mechnaism to do that.  Maybe the open source community should lokk at the Gstreamer framework and beef up its overflow detection to prevent the failure from occring?

                               

                              We cannot just close out a buffer unless we are explicitly told to do so, thus the "memory leak" in this case.  From the driver perspective, it really is not a leak- it is a valid allocated video buffer that is open because we were told to open it, and nver told to close it.  I believe this is actually one of those times a Linux reboot is required to recover fully from such a situation.

                               

                              Hope that helps.  Kirk

                    • Re: EMGD . memory leak and streaming issue
                      Green Belt

                      I think I'm seeing the same memory leak.

                      On my system with EMGD 1.10 I've verified that gmm_unmap isn't being called for the following mapping when the X server is killed:

                      pvr/services4/3rdparty/emgd_displayclass/emgd_dc.c line 2210.

                       

                      It appears that it is supposed to be unmapped on line 1978 of the same file but emgddc_free_a_devinfo doesn't get called.

                        • Re: EMGD . memory leak and streaming issue
                          Kirk Brown Belt

                          Yes, if you are doing video decode, and kill X in the middle of the decode, then start X again and repeat, you WILL see a memory decrease becuase of a "leak".  The issue is out of our hands (and not really a true leak) because of the way video was architected by theLinux community as an add-on- it is sort of a feature right now.  The kernel does not know it needs to clean-up video memory when shutting down X and the video stuff has no way to know about its memory once X is killed and restarted.

                           

                          I hate to say it, but you either need to:

                           

                          -WAIT for decode to finish BEFORE killing X, or

                          -You DO need to reboot to recover memory if you shutdown X in the middle of a decode (and likely encode also).  We have found no way to prevent this issue from happening because of the way Linux works currently, or

                          -Don't kill X, it should not be necessary EVER during normal operation of a system.  If someone is using that use case as a test, It is a silly one, because it does not reflect a normal use case.

                           

                          Hope this helps.

                            • Re: EMGD . memory leak and streaming issue
                              Green Belt

                              I'm not doing video decoding and I'm still seeing a massive leak. Just changing the runlevel back and forth between 5 and 3 will leak my system to death on Timesys Fedora Remix. I've made a patch that fixes part of the problem, I'm working on the rest now. I'm really amazed at the poor quality of this driver. Some of the comments in the kernel portion of the code are ridiculous and give me no confidence in this driver or anything to do with it (it reflects VERY badly on Intel's corporate image). FFS do you even try starting and stopping your X server to see if your kernel driver leaks memory !?! There is even a free in the code that is commented out and without it the kernel module leaks!

                                • Re: EMGD . memory leak and streaming issue
                                  Green Belt

                                  Many hours and liters of diet pepsi later I've made a patch that fixes the leaks I've been referring to. I haven't finished because there might still be more leaks but at least I've found several HUGE ones. Feedback is appreciated as I've created this patch hastily with nearly no prior knowledge of the workings of this driver. I still believe there is some kind of emgd shutdown race condition plauging my production systems. It should be easier to track down now that the driver isn't leaking like a sieve.

                                   

                                  Patch written and tested on NANO-8045 w/ Timesys Fedora Remix 14 and EMGD 1.10.0 Build 2209.