8 Replies Latest reply on Jul 23, 2015 10:59 PM by SteveLinsell

    Intel QAT Linux Kernel Cryptographic Framework Driver Error !

    FranzCC Green Belt

      Hello again,

      i'm now testing the Linux Kernel Cryptographic Framework Driver using the V 4.3.1.

      Firstly, the driver doesn't include any documentation as described in the download section.

      But i was able to build the kernel module.

      Of course, before i changed the qat config to use the kernel CY device. (IPSEC0)

      After loading the module, these error can be seen in the kernel ring buffer.

        556.201221] icp_netkey_shim_setkey: Re-initialising previous session

      [  556.201243] [error] LacSymPerform_BufferParamCheck() - : Invalid API Param - Source and Dest buffer lengths need to be equal

      [  556.201249] perform_sym_chaining: cpaCySymPerformOp failed. (status = (-4))

      [  556.201253] alg: aead-ddst: encryption failed on test 1 for icp_qat_aes_cbc_sha1_hmac: ret=22

      [  556.201289] Quick Assist icp_authenc_aes_cbc_hmac_sha1 loaded

      [  556.201382] alg: No test for authenc(hmac(md5),cbc(aes)) (icp_qat_aes_cbc_md5_hmac)

      [  556.201397] Quick Assist icp_authenc_aes_cbc_hmac_md5 loaded

      [  556.202302] icp_netkey_shim_setkey: Re-initialising previous session

      [  556.202385] icp_netkey_shim_setkey: Re-initialising previous session

      [  556.202460] icp_netkey_shim_setkey: Re-initialising previous session

      [  556.202535] icp_netkey_shim_setkey: Re-initialising previous session

      [  556.202611] icp_netkey_shim_setkey: Re-initialising previous session

      [  556.202687] icp_netkey_shim_setkey: Re-initialising previous session

      [  556.202776] icp_netkey_shim_setkey: Re-initialising previous session

      [  556.202797] [error] LacSymPerform_BufferParamCheck() - : Invalid API Param - Source and Dest buffer lengths need to be equal

      [  556.202803] perform_sym_chaining: cpaCySymPerformOp failed. (status = (-4))

      [  556.202807] alg: aead-ddst: encryption failed on test 1 for icp_qat_aes_cbc_sha256_hmac: ret=22

      etc.

       

      Now, the kernel module doesn't provide any parameters.

      Please tell me, if this is an error, where i can set the source and dest buffer values.

      icp_qa_al also doesn't provide any buffer specific parameters.

       

      Please explain the error and what can be done to get rid of it.

       

       

      Rgds.

       

      Franz

        • Re: Intel QAT Linux Kernel Cryptographic Framework Driver Error !
          SteveLinsell Community Member


          Hi Franz,

           

          Which Linux and Kernel version are you running on?

          The QAT LKCF Driver is tested against Fedora 16 and CentOS 7 only.

          I believe an internal defect has been raised in the last few weeks for the exact same issue you are seeing. It was observed when trying to run on Fedora 20.

          My understanding is that it has not been investigated yet.

           

          The issues are caused because with the newer linux kernels more internal tests are run when you load the module. The module has been written to satisy the tests that are run on the linux versions mentioned above only, but it appears the newer kernels are running some tests that appear to be passing invalid length buffers to the QAT Driver. This will need investigation to determine whether it is because the test is invalid (doesn't make sense) with our hardware or whether it is a bug in the QAT LKCF Driver in the way it is constructing the buffers being passed to the QAT Driver (it is more likely to be the later).

           

          Kind Regards,

           

          Steve.

            • Re: Intel QAT Linux Kernel Cryptographic Framework Driver Error !
              FranzCC Green Belt

              Hi Steve,

              yes, F20, bevause we rely on kernel features of this release.

              Also, 3.1 is pretty OLD.

              Because, using the output of cy0, there were successful symmetric requests.

              +--------------------------------------------------+

              | Statistics for Instance                   IPSec0 |

              | Symmetric Stats                                  |

              +--------------------------------------------------+

              | Sessions Initialized:                         30 |

              | Sessions Removed:                             30 |

              | Session Errors:                                0 |

              +--------------------------------------------------+

              | Symmetric Requests:                           24 |

              | Symmetric Request Errors:                      0 |

              | Symmetric Completed:                          24 |

              | Symmetric Completed Errors:                    0 |

              | Symmetric Verify Failures:                     0 |

              +--------------------------------------------------+

              Now again, is this module working (regardless of the failed tests) ?

              The icp_perf_aead module is using old scheduler code, which seems not to be working under F20.

              Patching sched.h (of the kernel) to let the module compile ends up in 100% cpu util but now real enc tests happen.

              sched.h: (Addition)

              #define task_is_dead(task) ((task)->exit_state != 0)

              So, i think, rewriting for a state of the art kernel schould be no problem, doesn't it ?

               

               

              Rgds.

               

              Franz

              • Re: Intel QAT Linux Kernel Cryptographic Framework Driver Error !
                FranzCC Green Belt

                Hi again,

                sorry to bother you again ;-) but LTS kernel 3.12.44 also works.

                This saves me from backporting some features.

                 

                Fyi the dmesg output while loading icp_qat_netkey:

                [ 1005.502242] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.502348] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.502444] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.502550] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.502646] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.502743] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.502855] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.502952] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.503047] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.503141] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.503235] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.503337] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.503434] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.503563] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.503661] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.503759] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.503860] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.503957] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.504054] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.504148] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.504295] Quick Assist icp_authenc_aes_cbc_hmac_sha1 loaded

                [ 1005.507271] alg: No test for authenc(hmac(md5),cbc(aes)) (icp_qat_aes_cbc_md5_hmac)

                [ 1005.507320] Quick Assist icp_authenc_aes_cbc_hmac_md5 loaded

                [ 1005.507503] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.507598] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.507695] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.507791] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.507888] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.507983] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.508099] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.508193] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.508291] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.508386] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.508483] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.508583] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.508679] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.508791] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.508889] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.508986] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.509081] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.509176] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.509280] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.509375] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.509528] Quick Assist icp_authenc_aes_cbc_hmac_sha256 loaded

                [ 1005.509688] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.509776] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.509856] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.509934] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.510013] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.510089] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.510183] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.510260] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.510337] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.510414] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.510490] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.510569] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.510645] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.510735] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.510812] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.510888] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.510965] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.511041] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.511119] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.511196] icp_netkey_shim_setkey: Re-initialising previous session

                [ 1005.511308] Quick Assist icp_authenc_aes_cbc_hmac_sha512 loaded

                [ 1005.511381] alg: No test for authenc(hmac(sha1),cbc(des3_ede)) (icp_qat_des3_cbc_sha1_hmac)

                [ 1005.511397] Quick Assist icp_authenc_des3_cbc_hmac_sha1 loaded

                [ 1005.511459] alg: No test for authenc(hmac(md5),cbc(des3_ede)) (icp_qat_des3_cbc_md5_hmac)

                [ 1005.511475] Quick Assist icp_authenc_des3_cbc_hmac_md5 loaded

                [ 1005.511529] alg: No test for authenc(hmac(sha256),cbc(des3_ede)) (icp_qat_des3_cbc_sha256_hmac)

                [ 1005.511544] Quick Assist icp_authenc_des3_cbc_hmac_sha256 loaded

                [ 1005.511589] alg: No test for authenc(hmac(sha512),cbc(des3_ede)) (icp_qat_des3_cbc_sha512_hmac)

                [ 1005.511602] Quick Assist icp_authenc_des3_cbc_hmac_sha512 loaded

                 

                head -20 /proc/icp_c2xxx_dev0/cy/IPSec0

                +--------------------------------------------------+

                | Statistics for Instance                   IPSec0 |

                | Symmetric Stats                                  |

                +--------------------------------------------------+

                | Sessions Initialized:                         63 |

                | Sessions Removed:                             63 |

                | Session Errors:                                0 |

                +--------------------------------------------------+

                | Symmetric Requests:                           63 |

                | Symmetric Request Errors:                      0 |

                | Symmetric Completed:                          63 |

                | Symmetric Completed Errors:                    0 |

                | Symmetric Verify Failures:                     0 |

                +--------------------------------------------------+

                 

                Rgds.

                 

                Franz

                  • Re: Intel QAT Linux Kernel Cryptographic Framework Driver Error !
                    FranzCC Green Belt

                    Hi again,

                    to sum it up i did:

                     

                    Kernel: 3.12.44 (LTS)

                     

                    Since the kernel only tests internal alg i disables the autotest for external alg:

                    Disabled internal alg autotest since it is external via config option: CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y

                     

                    The testmodule output:

                     

                      227.757068] Loading AEAD Performance Test module ...
                      [  227.768769]
                      AEAD givencrypt performance: alg authenc(hmac(sha1),cbc(aes))
                      [  268.022058] [sched_delayed] sched: RT throttling activated
                      [  277.749438] -----------------------------------------------
                      [  277.749444] Number threads:                            4
                      [  277.749447] Number of requests per thread:             5000000
                      [  277.749450] Pkt Size:                                  1024
                      [  277.749453] Total number of Cycles:                    79169726139
                      [  277.749455] CPU frequency:                         1750 MHz
                      [  277.749457] Throughput:                            3621 Mbps
                      [  277.749460] -----------------------------------------------
                      [  277.749463]


                    Testing via ipsec tunnel and iperf, i nearly gain wirespeed:


                    [ ID] Interval   Transfer Bandwidth

                    [  3]  0.0-10.0 sec  1.14 GBytes   978 Mbits/sec

                     

                    --------------------------------------------------+

                    | Symmetric Requests:                       597600 |

                    | Symmetric Request Errors:                      0 |

                    | Symmetric Completed:                      597600 |

                    | Symmetric Completed Errors:                    0 |

                    | Symmetric Verify Failures:                     0 |

                    +--------------------------------------------------+

                     

                    Thanks Steve for your input.

                     

                    Rgds.

                     

                    Franz

                • Re: Intel QAT Linux Kernel Cryptographic Framework Driver Error !
                  FranzCC Green Belt

                  Hi Steve,

                  the root cause of the buffer problem is the stack protection which is enabled by default.

                  Disabling it, i use the 3.18.19 LTS kernel along with icp_qa_al and icp_qat_netkey w/o any problems so far.

                   

                  Option:

                  CONFIG_HAVE_CC_STACKPROTECTOR=y

                  # CONFIG_CC_STACKPROTECTOR is not set

                  CONFIG_CC_STACKPROTECTOR_NONE=y

                  # CONFIG_CC_STACKPROTECTOR_REGULAR is not set

                  # CONFIG_CC_STACKPROTECTOR_STRONG is not set

                   

                  Rgds.

                   

                  Franz