feedc0de/wolfssl
1
0
Fork 0
forked from wolfSSL/wolfssl
Code Pull Requests Activity

refactor AESNI implementations and *VECTOR_REGISTERS* macros to allow dynamic as-needed fallback to pure C, via WC_AES_C_DYNAMIC_FALLBACK.

Browse Source 
wolfssl/wolfcrypt/aes.h: add key_C_fallback[] to struct Aes, and remove comment that "AESNI needs key first, rounds 2nd, not sure why yet" now that AES_128_Key_Expansion_AESNI no longer writes rounds after the expanded key.

wolfcrypt/src/aes.c:
* add _AESNI or _aesni suffixes/infixes to AESNI implementations that were missing them: AES_CBC_encrypt(), AES_CBC_decrypt_by*(), AES_ECB_encrypt(), AES_*_Key_Expansion(), AES_set_encrypt_key(), AES_set_decrypt_key(), AES_GCM_encrypt(), AES_GCM_decrypt(), AES_XTS_encrypt(), and AES_XTS_decrypt().
* move key size check from to start of wc_AesSetKeyLocal().
* refactor pure-C AES setkey and cipher implementations to use aes->key_C_fallback when defined(WC_AES_C_DYNAMIC_FALLBACK).
* refactor wc_AesSetKeyLocal() to set up both AESNI and pure-C expanded keys when defined(WC_AES_C_DYNAMIC_FALLBACK).
* refactor all (haveAESNI && aes->use_aesni) conditions to just (aes->use_aesni).
* add macros VECTOR_REGISTERS_PUSH and VECTOR_REGISTERS_POP, which do nothing but push a brace level when !defined(WC_AES_C_DYNAMIC_FALLBACK), but when defined(WC_AES_C_DYNAMIC_FALLBACK), they call SAVE_VECTOR_REGISTERS2() and on failure, temporarily clear aes->use_aesni and restore at _POP().
* refactor all invocations of SAVE_VECTOR_REGISTERS() and RESTORE_VECTOR_REGISTERS() to VECTOR_REGISTERS_PUSH and VECTOR_REGISTERS_POP, except in wc_AesSetKeyLocal(), wc_AesXtsEncrypt(), and wc_AesXtsDecrypt(), which are refactored to use SAVE_VECTOR_REGISTERS2(), with graceful failure concealment if defined(WC_AES_C_DYNAMIC_FALLBACK).
* orthogonalize cleanup code in wc_AesCbcEncrypt(),  wc_AesCcmEncrypt() and wc_AesCcmDecrypt().
* streamline fallthrough software definitions of wc_AesEncryptDirect() and wc_AesDecryptDirect(), and remove special-casing for defined(WOLFSSL_LINUXKM)&&defined(WOLFSSL_AESNI).

wolfcrypt/src/aes_asm.{S,asm}:
* remove errant "movl $10, 240(%rsi)" from AES_128_Key_Expansion_AESNI.
* add _AESNI suffixes/infixes to implementations that needed them.

wolfcrypt/src/{aes_gcm_asm.{S,asm},aes_xts_asm.S}: regenerate from revisions in scripts#357 -- adds _aesni suffixes to implementations that were missing them.

wolfssl/wolfcrypt/types.h: remove DEBUG_VECTOR_REGISTER_ACCESS macros, and add dummy fallthrough definitions for SAVE_VECTOR_REGISTERS2 and WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL.

wolfssl/wolfcrypt/memory.h: adopt DEBUG_VECTOR_REGISTER_ACCESS code from types.h, and add definitions for WC_DEBUG_VECTOR_REGISTERS_RETVAL_INITVAL and WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL.

linuxkm/linuxkm_wc_port.h: add arch-specific macro definitions for SAVE_VECTOR_REGISTERS2().

wolfcrypt/benchmark/benchmark.c: add missing gates around calls to RESTORE_VECTOR_REGISTERS().

configure.ac:
* cover various interdependencies in enable-all/enable-all-crypto, for better behavior in combination with --disable-aesgcm, --disable-ecc, --disable-ocsp, --disable-hmac, --disable-chacha, --disable-ed25519, and --disable-ed448.
* inhibit aesgcm_stream in enable-all/enable-all-crypto when ENABLED_LINUXKM_DEFAULTS, because it is currently incompatible with WC_AES_C_DYNAMIC_FALLBACK.
* add -DWC_AES_C_DYNAMIC_FALLBACK when ENABLED_LINUXKM_DEFAULTS.
* add 3 new interdependency checks: "ECCSI requires ECC.", "SAKKE requires ECC.", "WOLFSSH requires HMAC."

wolfcrypt/src/asn.c: tweak gating to accommodate defined(NO_RSA) && !defined(HAVE_ECC).

wolfcrypt/src/evp.c: tweak gating to accommodate defined(NO_HMAC).

wolfcrypt/src/logging.c: remove DEBUG_VECTOR_REGISTER_ACCESS code (moved to memory.c).

wolfcrypt/src/memory.c: change #include of settings.h to types.h; adopt DEBUG_VECTOR_REGISTER_ACCESS code from logging.c; add implementation of SAVE_VECTOR_REGISTERS2_fuzzer().

wolfcrypt/src/pwdbased.c: add explanatory #error scrypt requires HMAC.

wolfcrypt/test/test.c:
* add DEBUG_VECTOR_REGISTER_ACCESS clauses to aes_xts_128_test(), aesecb_test(), aesctr_test(), aes_test() CBC section, aes256_test() CBC section, and aesgcm_default_test_helper()
* remove duplicate wc_AesEcbDecrypt() in aesecb_test().
* add gating for pbkdf2_test().
* fix cleanup code in dsa_test().
* fix gating in pkcs7authenveloped_run_vectors() to accommodate !defined(HAVE_AESGCM).
* fix gating in cryptocb_test() to accommodate defined(NO_HMAC).

wolfssl/wolfcrypt/cryptocb.h: remove gates around "pk" sub-struct of struct wc_CryptoInfo -- wc_CryptoInfo.pk.type (an int) is used unconditionally when --enable-debug, and is used with DH.

wolfssl/wolfcrypt/error-crypt.h: fix whitespace.
This commit is contained in:
Daniel Pouzzner
2023-11-17 01:15:28 -06:00
parent e395aad84b
commit a10260ca5f
20 changed files with 1934 additions and 1345 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
configure.ac
View File

@ -720,7 +720,6 @@ then
# this set is also enabled by enable-all-crypto: # this set is also enabled by enable-all-crypto:
test "$enable_atomicuser" = "" && enable_atomicuser=yes test "$enable_atomicuser" = "" && enable_atomicuser=yes
test "$enable_aesgcm" = "" && enable_aesgcm=yes test "$enable_aesgcm" = "" && enable_aesgcm=yes
test "$enable_aesgcm_stream" = "" && test "$enable_aesgcm" = "yes" && enable_aesgcm_stream=yes
test "$enable_aesccm" = "" && enable_aesccm=yes test "$enable_aesccm" = "" && enable_aesccm=yes
test "$enable_aesctr" = "" && enable_aesctr=yes test "$enable_aesctr" = "" && enable_aesctr=yes
test "$enable_aeseax" = "" && enable_aeseax=yes test "$enable_aeseax" = "" && enable_aeseax=yes
@ -741,22 +740,22 @@ then
test "$enable_hkdf" = "" && enable_hkdf=yes test "$enable_hkdf" = "" && enable_hkdf=yes
test "$enable_curve25519" = "" && enable_curve25519=yes test "$enable_curve25519" = "" && enable_curve25519=yes
test "$enable_curve448" = "" && enable_curve448=yes test "$enable_curve448" = "" && enable_curve448=yes
test "$enable_fpecc" = "" && enable_fpecc=yes test "$enable_fpecc" = "" && test "$enable_ecc" != "no" && enable_fpecc=yes
test "$enable_eccencrypt" = "" && enable_eccencrypt=yes test "$enable_eccencrypt" = "" && test "$enable_ecc" != "no" && enable_eccencrypt=yes
test "$enable_psk" = "" && enable_psk=yes test "$enable_psk" = "" && enable_psk=yes
test "$enable_cmac" = "" && enable_cmac=yes test "$enable_cmac" = "" && enable_cmac=yes
test "$enable_siphash" = "" && enable_siphash=yes test "$enable_siphash" = "" && enable_siphash=yes
test "$enable_xts" = "" && enable_xts=yes test "$enable_xts" = "" && enable_xts=yes
test "$enable_ocsp" = "" && enable_ocsp=yes test "$enable_ocsp" = "" && enable_ocsp=yes
test "$enable_ocspstapling" = "" && enable_ocspstapling=yes test "$enable_ocspstapling" = "" && test "$enable_ocsp" != "no" && enable_ocspstapling=yes
test "$enable_ocspstapling2" = "" && enable_ocspstapling2=yes test "$enable_ocspstapling2" = "" && test "$enable_ocsp" != "no" && enable_ocspstapling2=yes
test "$enable_crl" = "" && enable_crl=yes test "$enable_crl" = "" && enable_crl=yes
test "$enable_supportedcurves" = "" && enable_supportedcurves=yes test "$enable_supportedcurves" = "" && enable_supportedcurves=yes
test "$enable_tlsx" = "" && enable_tlsx=yes test "$enable_tlsx" = "" && enable_tlsx=yes
test "$enable_pwdbased" = "" && enable_pwdbased=yes test "$enable_pwdbased" = "" && enable_pwdbased=yes
test "$enable_aeskeywrap" = "" && enable_aeskeywrap=yes test "$enable_aeskeywrap" = "" && enable_aeskeywrap=yes
test "$enable_x963kdf" = "" && enable_x963kdf=yes test "$enable_x963kdf" = "" && enable_x963kdf=yes
test "$enable_scrypt" = "" && enable_scrypt=yes test "$enable_scrypt" = "" && test "$enable_hmac" != "no" && enable_scrypt=yes
test "$enable_indef" = "" && enable_indef=yes test "$enable_indef" = "" && enable_indef=yes
test "$enable_enckeys" = "" && enable_enckeys=yes test "$enable_enckeys" = "" && enable_enckeys=yes
test "$enable_hashflags" = "" && enable_hashflags=yes test "$enable_hashflags" = "" && enable_hashflags=yes
@ -771,7 +770,7 @@ then
test "$enable_md4" = "" && enable_md4=yes test "$enable_md4" = "" && enable_md4=yes
test "$enable_cryptocb" = "" && enable_cryptocb=yes test "$enable_cryptocb" = "" && enable_cryptocb=yes
test "$enable_anon" = "" && enable_anon=yes test "$enable_anon" = "" && enable_anon=yes
test "$enable_ssh" = "" && enable_ssh=yes test "$enable_ssh" = "" && test "$enable_hmac" != "no" && enable_ssh=yes
test "$enable_savesession" = "" && enable_savesession=yes test "$enable_savesession" = "" && enable_savesession=yes
test "$enable_savecert" = "" && enable_savecert=yes test "$enable_savecert" = "" && enable_savecert=yes
@ -797,6 +796,7 @@ then
if test "$ENABLED_LINUXKM_DEFAULTS" != "yes" if test "$ENABLED_LINUXKM_DEFAULTS" != "yes"
then then
test "$enable_aesgcm_stream" = "" && test "$enable_aesgcm" = "yes" && enable_aesgcm_stream=yes
test "$enable_compkey" = "" && enable_compkey=yes test "$enable_compkey" = "" && enable_compkey=yes
test "$enable_quic" = "" && enable_quic=yes test "$enable_quic" = "" && enable_quic=yes
AM_CFLAGS="$AM_CFLAGS -DHAVE_CRL_IO -DHAVE_IO_TIMEOUT" AM_CFLAGS="$AM_CFLAGS -DHAVE_CRL_IO -DHAVE_IO_TIMEOUT"
@ -836,7 +836,7 @@ then
if test "$ENABLED_FIPS" = "no" if test "$ENABLED_FIPS" = "no"
then then
test "$enable_pkcallbacks" = "" && enable_pkcallbacks=yes test "$enable_pkcallbacks" = "" && enable_pkcallbacks=yes
test "$enable_xchacha" = "" && enable_xchacha=yes test "$enable_xchacha" = "" && test "$enable_chacha" != "no" && enable_xchacha=yes
test "$enable_scep" = "" && enable_scep=yes test "$enable_scep" = "" && enable_scep=yes
test "$enable_pkcs7" = "" && enable_pkcs7=yes test "$enable_pkcs7" = "" && enable_pkcs7=yes
test "$enable_nullcipher" = "" && enable_nullcipher=yes test "$enable_nullcipher" = "" && enable_nullcipher=yes
@ -844,9 +844,9 @@ then
if test "$ENABLED_32BIT" != "yes" if test "$ENABLED_32BIT" != "yes"
then then
test "$enable_ed25519" = "" && enable_ed25519=yes test "$enable_ed25519" = "" && enable_ed25519=yes
test "$enable_ed25519_stream" = "" && enable_ed25519_stream=yes test "$enable_ed25519_stream" = "" && test "$enable_ed25519" != "no" && enable_ed25519_stream=yes
test "$enable_ed448" = "" && enable_ed448=yes test "$enable_ed448" = "" && enable_ed448=yes
test "$enable_ed448_stream" = "" && enable_ed448_stream=yes test "$enable_ed448_stream" = "" && test "$enable_ed448" != "no" && enable_ed448_stream=yes
fi fi
if test "$ENABLED_LINUXKM_DEFAULTS" != "yes" if test "$ENABLED_LINUXKM_DEFAULTS" != "yes"
@ -856,8 +856,8 @@ then
test "$enable_curl" = "" && enable_curl=yes test "$enable_curl" = "" && enable_curl=yes
test "$enable_tcpdump" = "" && enable_tcpdump=yes test "$enable_tcpdump" = "" && enable_tcpdump=yes
test "$enable_eccsi" = "" && enable_eccsi=yes test "$enable_eccsi" = "" && test "$enable_ecc" != "no" && enable_eccsi=yes
test "$enable_sakke" = "" && enable_sakke=yes test "$enable_sakke" = "" && test "$enable_ecc" != "no" && enable_sakke=yes
fi fi
fi fi
@ -908,7 +908,6 @@ if test "$ENABLED_ALL_CRYPT" = "yes"
then then
test "$enable_atomicuser" = "" && enable_atomicuser=yes test "$enable_atomicuser" = "" && enable_atomicuser=yes
test "$enable_aesgcm" = "" && enable_aesgcm=yes test "$enable_aesgcm" = "" && enable_aesgcm=yes
test "$enable_aesgcm_stream" = "" && test "$enable_aesgcm" = "yes" && enable_aesgcm_stream=yes
test "$enable_aesccm" = "" && enable_aesccm=yes test "$enable_aesccm" = "" && enable_aesccm=yes
test "$enable_aesctr" = "" && enable_aesctr=yes test "$enable_aesctr" = "" && enable_aesctr=yes
test "$enable_aeseax" = "" && enable_aeseax=yes test "$enable_aeseax" = "" && enable_aeseax=yes
@ -929,22 +928,22 @@ then
test "$enable_hkdf" = "" && enable_hkdf=yes test "$enable_hkdf" = "" && enable_hkdf=yes
test "$enable_curve25519" = "" && enable_curve25519=yes test "$enable_curve25519" = "" && enable_curve25519=yes
test "$enable_curve448" = "" && enable_curve448=yes test "$enable_curve448" = "" && enable_curve448=yes
test "$enable_fpecc" = "" && enable_fpecc=yes test "$enable_fpecc" = "" && test "$enable_ecc" != "no" && enable_fpecc=yes
test "$enable_eccencrypt" = "" && enable_eccencrypt=yes test "$enable_eccencrypt" = "" && test "$enable_ecc" != "no" && enable_eccencrypt=yes
test "$enable_psk" = "" && enable_psk=yes test "$enable_psk" = "" && enable_psk=yes
test "$enable_cmac" = "" && enable_cmac=yes test "$enable_cmac" = "" && enable_cmac=yes
test "$enable_siphash" = "" && enable_siphash=yes test "$enable_siphash" = "" && enable_siphash=yes
test "$enable_xts" = "" && enable_xts=yes test "$enable_xts" = "" && enable_xts=yes
test "$enable_ocsp" = "" && enable_ocsp=yes test "$enable_ocsp" = "" && enable_ocsp=yes
test "$enable_ocspstapling" = "" && enable_ocspstapling=yes test "$enable_ocspstapling" = "" && test "$enable_ocsp" != "no" && enable_ocspstapling=yes
test "$enable_ocspstapling2" = "" && enable_ocspstapling2=yes test "$enable_ocspstapling2" = "" && test "$enable_ocsp" != "no" && enable_ocspstapling2=yes
test "$enable_crl" = "" && enable_crl=yes test "$enable_crl" = "" && enable_crl=yes
test "$enable_supportedcurves" = "" && enable_supportedcurves=yes test "$enable_supportedcurves" = "" && enable_supportedcurves=yes
test "$enable_tlsx" = "" && enable_tlsx=yes test "$enable_tlsx" = "" && enable_tlsx=yes
test "$enable_pwdbased" = "" && enable_pwdbased=yes test "$enable_pwdbased" = "" && enable_pwdbased=yes
test "$enable_aeskeywrap" = "" && enable_aeskeywrap=yes test "$enable_aeskeywrap" = "" && enable_aeskeywrap=yes
test "$enable_x963kdf" = "" && enable_x963kdf=yes test "$enable_x963kdf" = "" && enable_x963kdf=yes
test "$enable_scrypt" = "" && enable_scrypt=yes test "$enable_scrypt" = "" && test "$enable_hmac" != "no" && enable_scrypt=yes
test "$enable_indef" = "" && enable_indef=yes test "$enable_indef" = "" && enable_indef=yes
test "$enable_enckeys" = "" && enable_enckeys=yes test "$enable_enckeys" = "" && enable_enckeys=yes
test "$enable_hashflags" = "" && enable_hashflags=yes test "$enable_hashflags" = "" && enable_hashflags=yes
@ -959,7 +958,7 @@ then
test "$enable_md4" = "" && enable_md4=yes test "$enable_md4" = "" && enable_md4=yes
test "$enable_cryptocb" = "" && enable_cryptocb=yes test "$enable_cryptocb" = "" && enable_cryptocb=yes
test "$enable_anon" = "" && enable_anon=yes test "$enable_anon" = "" && enable_anon=yes
test "$enable_ssh" = "" && enable_ssh=yes test "$enable_ssh" = "" && test "$enable_hmac" != "no" && enable_ssh=yes
if test "$ENABLED_32BIT" != "yes" if test "$ENABLED_32BIT" != "yes"
then then
@ -969,6 +968,7 @@ then
if test "$ENABLED_LINUXKM_DEFAULTS" != "yes" if test "$ENABLED_LINUXKM_DEFAULTS" != "yes"
then then
test "$enable_aesgcm_stream" = "" && test "$enable_aesgcm" = "yes" && enable_aesgcm_stream=yes
test "$enable_compkey" = "" && enable_compkey=yes test "$enable_compkey" = "" && enable_compkey=yes
fi fi
@ -983,21 +983,21 @@ then
if test "$ENABLED_FIPS" = "no" if test "$ENABLED_FIPS" = "no"
then then
test "$enable_pkcallbacks" = "" && enable_pkcallbacks=yes test "$enable_pkcallbacks" = "" && enable_pkcallbacks=yes
test "$enable_xchacha" = "" && enable_xchacha=yes test "$enable_xchacha" = "" && test "$enable_chacha" != "no" && enable_xchacha=yes
test "$enable_pkcs7" = "" && enable_pkcs7=yes test "$enable_pkcs7" = "" && enable_pkcs7=yes
test "$enable_nullcipher" = "" && enable_nullcipher=yes test "$enable_nullcipher" = "" && enable_nullcipher=yes
if test "$ENABLED_32BIT" != "yes" if test "$ENABLED_32BIT" != "yes"
then then
test "$enable_ed25519" = "" && enable_ed25519=yes test "$enable_ed25519" = "" && enable_ed25519=yes
test "$enable_ed25519_stream" = "" && enable_ed25519_stream=yes test "$enable_ed25519_stream" = "" && test "$enable_ed25519" != "no" && enable_ed25519_stream=yes
test "$enable_ed448" = "" && enable_ed448=yes test "$enable_ed448" = "" && enable_ed448=yes
test "$enable_ed448_stream" = "" && enable_ed448_stream=yes test "$enable_ed448_stream" = "" && test "$enable_ed448" != "no" && enable_ed448_stream=yes
fi fi
if test "$ENABLED_LINUXKM_DEFAULTS" != "yes" if test "$ENABLED_LINUXKM_DEFAULTS" != "yes"
then then
test "$enable_eccsi" = "" && enable_eccsi=yes test "$enable_eccsi" = "" && test "$enable_ecc" != "no" && enable_eccsi=yes
test "$enable_sakke" = "" && enable_sakke=yes test "$enable_sakke" = "" && test "$enable_ecc" != "no" && enable_sakke=yes
fi fi
fi fi
@ -2896,6 +2896,10 @@ then
if test "$ENABLED_AESNI" = "yes" || test "$ENABLED_INTELASM" = "yes" if test "$ENABLED_AESNI" = "yes" || test "$ENABLED_INTELASM" = "yes"
then then
AM_CFLAGS="$AM_CFLAGS -DWOLFSSL_AESNI" AM_CFLAGS="$AM_CFLAGS -DWOLFSSL_AESNI"
if test "$ENABLED_LINUXKM_DEFAULTS" = "yes"
then
AM_CFLAGS="$AM_CFLAGS -DWC_AES_C_DYNAMIC_FALLBACK"
fi
if test "$CC" != "icc" if test "$CC" != "icc"
then then
case $host_os in case $host_os in
@ -3951,6 +3955,10 @@ AC_ARG_ENABLE([eccsi],
if test "x$ENABLED_ECCSI" = "xyes" if test "x$ENABLED_ECCSI" = "xyes"
then then
if test "$ENABLED_ECC" = "no"
then
AC_MSG_ERROR([ECCSI requires ECC.])
fi
AM_CFLAGS="$AM_CFLAGS -DWOLFCRYPT_HAVE_ECCSI -DWOLFSSL_PUBLIC_MP" AM_CFLAGS="$AM_CFLAGS -DWOLFCRYPT_HAVE_ECCSI -DWOLFSSL_PUBLIC_MP"
fi fi
@ -3961,6 +3969,11 @@ AC_ARG_ENABLE([sakke],
[ ENABLED_SAKKE=no ] [ ENABLED_SAKKE=no ]
) )
if test "$ENABLED_SAKKE" != "no" && test "$ENABLED_ECC" = "no"
then
AC_MSG_ERROR([SAKKE requires ECC.])
fi
if test "x$ENABLED_SAKKE" = "xsmall" if test "x$ENABLED_SAKKE" = "xsmall"
then then
ENABLED_SAKKE="yes" ENABLED_SAKKE="yes"
@ -8926,6 +8939,11 @@ if test "x$ENABLED_OPENSSLCOEXIST" = "xyes"; then
fi fi
fi fi
if test "$ENABLED_WOLFSSH" = "yes" && test "$ENABLED_HMAC" = "no"
then
AC_MSG_ERROR([WOLFSSH requires HMAC.])
fi
AS_IF([test "x$ENABLED_WOLFSSH" = "xyes"],[AM_CPPFLAGS="$AM_CPPFLAGS -DWOLFSSL_WOLFSSH"]) AS_IF([test "x$ENABLED_WOLFSSH" = "xyes"],[AM_CPPFLAGS="$AM_CPPFLAGS -DWOLFSSL_WOLFSSH"])
# only allow secure renegotiation info with TLSV12 and ASN # only allow secure renegotiation info with TLSV12 and ASN

2
linuxkm/linuxkm_wc_port.h
View File

@ -185,6 +185,7 @@
#endif #endif
#ifndef SAVE_VECTOR_REGISTERS #ifndef SAVE_VECTOR_REGISTERS
#define SAVE_VECTOR_REGISTERS(fail_clause) { int _svr_ret = save_vector_registers_x86(); if (_svr_ret != 0) { fail_clause } } #define SAVE_VECTOR_REGISTERS(fail_clause) { int _svr_ret = save_vector_registers_x86(); if (_svr_ret != 0) { fail_clause } }
#define SAVE_VECTOR_REGISTERS2() save_vector_registers_x86()
#endif #endif
#ifndef RESTORE_VECTOR_REGISTERS #ifndef RESTORE_VECTOR_REGISTERS
#define RESTORE_VECTOR_REGISTERS() restore_vector_registers_x86() #define RESTORE_VECTOR_REGISTERS() restore_vector_registers_x86()
@ -193,6 +194,7 @@
#include <asm/fpsimd.h> #include <asm/fpsimd.h>
#ifndef SAVE_VECTOR_REGISTERS #ifndef SAVE_VECTOR_REGISTERS
#define SAVE_VECTOR_REGISTERS(fail_clause) { int _svr_ret = save_vector_registers_arm(); if (_svr_ret != 0) { fail_clause } } #define SAVE_VECTOR_REGISTERS(fail_clause) { int _svr_ret = save_vector_registers_arm(); if (_svr_ret != 0) { fail_clause } }
#define SAVE_VECTOR_REGISTERS2() save_vector_registers_arm()
#endif #endif
#ifndef RESTORE_VECTOR_REGISTERS #ifndef RESTORE_VECTOR_REGISTERS
#define RESTORE_VECTOR_REGISTERS() restore_vector_registers_arm() #define RESTORE_VECTOR_REGISTERS() restore_vector_registers_arm()

4
wolfcrypt/benchmark/benchmark.c
View File

@ -2125,7 +2125,9 @@ static void bench_stats_sym_finish(const char* desc, int useDeviceID,
(void)useDeviceID; (void)useDeviceID;
(void)ret; (void)ret;
#ifdef WOLFSSL_LINUXKM_USE_SAVE_VECTOR_REGISTERS
RESTORE_VECTOR_REGISTERS(); RESTORE_VECTOR_REGISTERS();
#endif
TEST_SLEEP(); TEST_SLEEP();
} /* bench_stats_sym_finish */ } /* bench_stats_sym_finish */
@ -2283,7 +2285,9 @@ static void bench_stats_asym_finish_ex(const char* algo, int strength,
(void)useDeviceID; (void)useDeviceID;
(void)ret; (void)ret;
#ifdef WOLFSSL_LINUXKM_USE_SAVE_VECTOR_REGISTERS
RESTORE_VECTOR_REGISTERS(); RESTORE_VECTOR_REGISTERS();
#endif
TEST_SLEEP(); TEST_SLEEP();
} /* bench_stats_asym_finish_ex */ } /* bench_stats_asym_finish_ex */

1464
wolfcrypt/src/aes.c
View File

File diff suppressed because it is too large Load Diff

161
wolfcrypt/src/aes_asm.S
View File

@ -30,7 +30,7 @@
#ifdef WOLFSSL_X86_64_BUILD #ifdef WOLFSSL_X86_64_BUILD
/* /*
AES_CBC_encrypt (const unsigned char *in, AES_CBC_encrypt_AESNI (const unsigned char *in,
unsigned char *out, unsigned char *out,
unsigned char ivec[16], unsigned char ivec[16],
unsigned long length, unsigned long length,
@ -38,11 +38,11 @@ AES_CBC_encrypt (const unsigned char *in,
int nr) int nr)
*/ */
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_CBC_encrypt .globl AES_CBC_encrypt_AESNI
AES_CBC_encrypt: AES_CBC_encrypt_AESNI:
#else #else
.globl _AES_CBC_encrypt .globl _AES_CBC_encrypt_AESNI
_AES_CBC_encrypt: _AES_CBC_encrypt_AESNI:
#endif #endif
# parameter 1: %rdi # parameter 1: %rdi
# parameter 2: %rsi # parameter 2: %rsi
@ -95,7 +95,7 @@ ret
#if defined(WOLFSSL_AESNI_BY4) #if defined(WOLFSSL_AESNI_BY4)
/* /*
AES_CBC_decrypt_by4 (const unsigned char *in, AES_CBC_decrypt_AESNI_by4 (const unsigned char *in,
unsigned char *out, unsigned char *out,
unsigned char ivec[16], unsigned char ivec[16],
unsigned long length, unsigned long length,
@ -103,11 +103,11 @@ AES_CBC_decrypt_by4 (const unsigned char *in,
int nr) int nr)
*/ */
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_CBC_decrypt_by4 .globl AES_CBC_decrypt_AESNI_by4
AES_CBC_decrypt_by4: AES_CBC_decrypt_AESNI_by4:
#else #else
.globl _AES_CBC_decrypt_by4 .globl _AES_CBC_decrypt_AESNI_by4
_AES_CBC_decrypt_by4: _AES_CBC_decrypt_AESNI_by4:
#endif #endif
# parameter 1: %rdi # parameter 1: %rdi
# parameter 2: %rsi # parameter 2: %rsi
@ -276,7 +276,7 @@ DEND_4:
#elif defined(WOLFSSL_AESNI_BY6) #elif defined(WOLFSSL_AESNI_BY6)
/* /*
AES_CBC_decrypt_by6 (const unsigned char *in, AES_CBC_decrypt_AESNI_by6 (const unsigned char *in,
unsigned char *out, unsigned char *out,
unsigned char ivec[16], unsigned char ivec[16],
unsigned long length, unsigned long length,
@ -284,11 +284,11 @@ AES_CBC_decrypt_by6 (const unsigned char *in,
int nr) int nr)
*/ */
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_CBC_decrypt_by6 .globl AES_CBC_decrypt_AESNI_by6
AES_CBC_decrypt_by6: AES_CBC_decrypt_AESNI_by6:
#else #else
.globl _AES_CBC_decrypt_by6 .globl _AES_CBC_decrypt_AESNI_by6
_AES_CBC_decrypt_by6: _AES_CBC_decrypt_AESNI_by6:
#endif #endif
# parameter 1: %rdi - in # parameter 1: %rdi - in
# parameter 2: %rsi - out # parameter 2: %rsi - out
@ -504,7 +504,7 @@ DEND_6:
#else /* WOLFSSL_AESNI_BYx */ #else /* WOLFSSL_AESNI_BYx */
/* /*
AES_CBC_decrypt_by8 (const unsigned char *in, AES_CBC_decrypt_AESNI_by8 (const unsigned char *in,
unsigned char *out, unsigned char *out,
unsigned char ivec[16], unsigned char ivec[16],
unsigned long length, unsigned long length,
@ -512,11 +512,11 @@ AES_CBC_decrypt_by8 (const unsigned char *in,
int nr) int nr)
*/ */
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_CBC_decrypt_by8 .globl AES_CBC_decrypt_AESNI_by8
AES_CBC_decrypt_by8: AES_CBC_decrypt_AESNI_by8:
#else #else
.globl _AES_CBC_decrypt_by8 .globl _AES_CBC_decrypt_AESNI_by8
_AES_CBC_decrypt_by8: _AES_CBC_decrypt_AESNI_by8:
#endif #endif
# parameter 1: %rdi - in # parameter 1: %rdi - in
# parameter 2: %rsi - out # parameter 2: %rsi - out
@ -761,18 +761,18 @@ DEND_8:
/* /*
AES_ECB_encrypt (const unsigned char *in, AES_ECB_encrypt_AESNI (const unsigned char *in,
unsigned char *out, unsigned char *out,
unsigned long length, unsigned long length,
const unsigned char *KS, const unsigned char *KS,
int nr) int nr)
*/ */
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_ECB_encrypt .globl AES_ECB_encrypt_AESNI
AES_ECB_encrypt: AES_ECB_encrypt_AESNI:
#else #else
.globl _AES_ECB_encrypt .globl _AES_ECB_encrypt_AESNI
_AES_ECB_encrypt: _AES_ECB_encrypt_AESNI:
#endif #endif
# parameter 1: %rdi # parameter 1: %rdi
# parameter 2: %rsi # parameter 2: %rsi
@ -925,18 +925,18 @@ EECB_END_4:
/* /*
AES_ECB_decrypt (const unsigned char *in, AES_ECB_decrypt_AESNI (const unsigned char *in,
unsigned char *out, unsigned char *out,
unsigned long length, unsigned long length,
const unsigned char *KS, const unsigned char *KS,
int nr) int nr)
*/ */
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_ECB_decrypt .globl AES_ECB_decrypt_AESNI
AES_ECB_decrypt: AES_ECB_decrypt_AESNI:
#else #else
.globl _AES_ECB_decrypt .globl _AES_ECB_decrypt_AESNI
_AES_ECB_decrypt: _AES_ECB_decrypt_AESNI:
#endif #endif
# parameter 1: %rdi # parameter 1: %rdi
# parameter 2: %rsi # parameter 2: %rsi
@ -1092,20 +1092,19 @@ DECB_END_4:
/* /*
void AES_128_Key_Expansion(const unsigned char* userkey, void AES_128_Key_Expansion_AESNI(const unsigned char* userkey,
unsigned char* key_schedule); unsigned char* key_schedule);
*/ */
.align 16,0x90 .align 16,0x90
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_128_Key_Expansion .globl AES_128_Key_Expansion_AESNI
AES_128_Key_Expansion: AES_128_Key_Expansion_AESNI:
#else #else
.globl _AES_128_Key_Expansion .globl _AES_128_Key_Expansion_AESNI
_AES_128_Key_Expansion: _AES_128_Key_Expansion_AESNI:
#endif #endif
# parameter 1: %rdi # parameter 1: %rdi
# parameter 2: %rsi # parameter 2: %rsi
movl $10, 240(%rsi)
movdqu (%rdi), %xmm1 movdqu (%rdi), %xmm1
movdqa %xmm1, (%rsi) movdqa %xmm1, (%rsi)
@ -1158,15 +1157,15 @@ ret
/* /*
void AES_192_Key_Expansion (const unsigned char *userkey, void AES_192_Key_Expansion_AESNI (const unsigned char *userkey,
unsigned char *key) unsigned char *key)
*/ */
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_192_Key_Expansion .globl AES_192_Key_Expansion_AESNI
AES_192_Key_Expansion: AES_192_Key_Expansion_AESNI:
#else #else
.globl _AES_192_Key_Expansion .globl _AES_192_Key_Expansion_AESNI
_AES_192_Key_Expansion: _AES_192_Key_Expansion_AESNI:
#endif #endif
# parameter 1: %rdi # parameter 1: %rdi
# parameter 2: %rsi # parameter 2: %rsi
@ -1249,15 +1248,15 @@ ret
/* /*
void AES_256_Key_Expansion (const unsigned char *userkey, void AES_256_Key_Expansion_AESNI (const unsigned char *userkey,
unsigned char *key) unsigned char *key)
*/ */
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_256_Key_Expansion .globl AES_256_Key_Expansion_AESNI
AES_256_Key_Expansion: AES_256_Key_Expansion_AESNI:
#else #else
.globl _AES_256_Key_Expansion .globl _AES_256_Key_Expansion_AESNI
_AES_256_Key_Expansion: _AES_256_Key_Expansion_AESNI:
#endif #endif
# parameter 1: %rdi # parameter 1: %rdi
# parameter 2: %rsi # parameter 2: %rsi
@ -1337,7 +1336,7 @@ ret
#elif defined WOLFSSL_X86_BUILD #elif defined WOLFSSL_X86_BUILD
/* /*
AES_CBC_encrypt (const unsigned char *in, AES_CBC_encrypt_AESNI (const unsigned char *in,
unsigned char *out, unsigned char *out,
unsigned char ivec[16], unsigned char ivec[16],
unsigned long length, unsigned long length,
@ -1345,11 +1344,11 @@ AES_CBC_encrypt (const unsigned char *in,
int nr) int nr)
*/ */
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_CBC_encrypt .globl AES_CBC_encrypt_AESNI
AES_CBC_encrypt: AES_CBC_encrypt_AESNI:
#else #else
.globl _AES_CBC_encrypt .globl _AES_CBC_encrypt_AESNI
_AES_CBC_encrypt: _AES_CBC_encrypt_AESNI:
#endif #endif
# parameter 1: stack[4] => %edi # parameter 1: stack[4] => %edi
# parameter 2: stack[8] => %esi # parameter 2: stack[8] => %esi
@ -1416,7 +1415,7 @@ _AES_CBC_encrypt:
/* /*
AES_CBC_decrypt_by4 (const unsigned char *in, AES_CBC_decrypt_AESNI_by4 (const unsigned char *in,
unsigned char *out, unsigned char *out,
unsigned char ivec[16], unsigned char ivec[16],
unsigned long length, unsigned long length,
@ -1424,11 +1423,11 @@ AES_CBC_decrypt_by4 (const unsigned char *in,
int nr) int nr)
*/ */
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_CBC_decrypt_by4 .globl AES_CBC_decrypt_AESNI_by4
AES_CBC_decrypt_by4: AES_CBC_decrypt_AESNI_by4:
#else #else
.globl _AES_CBC_decrypt_by4 .globl _AES_CBC_decrypt_AESNI_by4
_AES_CBC_decrypt_by4: _AES_CBC_decrypt_AESNI_by4:
#endif #endif
# parameter 1: stack[4] => %edi # parameter 1: stack[4] => %edi
# parameter 2: stack[8] => %esi # parameter 2: stack[8] => %esi
@ -1614,18 +1613,18 @@ DEND_4:
ret ret
/* /*
AES_ECB_encrypt (const unsigned char *in, AES_ECB_encrypt_AESNI (const unsigned char *in,
unsigned char *out, unsigned char *out,
unsigned long length, unsigned long length,
const unsigned char *KS, const unsigned char *KS,
int nr) int nr)
*/ */
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_ECB_encrypt .globl AES_ECB_encrypt_AESNI
AES_ECB_encrypt: AES_ECB_encrypt_AESNI:
#else #else
.globl _AES_ECB_encrypt .globl _AES_ECB_encrypt_AESNI
_AES_ECB_encrypt: _AES_ECB_encrypt_AESNI:
#endif #endif
# parameter 1: stack[4] => %edi # parameter 1: stack[4] => %edi
# parameter 2: stack[8] => %esi # parameter 2: stack[8] => %esi
@ -1791,18 +1790,18 @@ EECB_END_4:
/* /*
AES_ECB_decrypt (const unsigned char *in, AES_ECB_decrypt_AESNI (const unsigned char *in,
unsigned char *out, unsigned char *out,
unsigned long length, unsigned long length,
const unsigned char *KS, const unsigned char *KS,
int nr) int nr)
*/ */
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_ECB_decrypt .globl AES_ECB_decrypt_AESNI
AES_ECB_decrypt: AES_ECB_decrypt_AESNI:
#else #else
.globl _AES_ECB_decrypt .globl _AES_ECB_decrypt_AESNI
_AES_ECB_decrypt: _AES_ECB_decrypt_AESNI:
#endif #endif
# parameter 1: stack[4] => %edi # parameter 1: stack[4] => %edi
# parameter 2: stack[8] => %esi # parameter 2: stack[8] => %esi
@ -1969,16 +1968,16 @@ DECB_END_4:
/* /*
void AES_128_Key_Expansion(const unsigned char* userkey, void AES_128_Key_Expansion_AESNI(const unsigned char* userkey,
unsigned char* key_schedule); unsigned char* key_schedule);
*/ */
.align 16,0x90 .align 16,0x90
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_128_Key_Expansion .globl AES_128_Key_Expansion_AESNI
AES_128_Key_Expansion: AES_128_Key_Expansion_AESNI:
#else #else
.globl _AES_128_Key_Expansion .globl _AES_128_Key_Expansion_AESNI
_AES_128_Key_Expansion: _AES_128_Key_Expansion_AESNI:
#endif #endif
# parameter 1: stack[4] => %eax # parameter 1: stack[4] => %eax
# parameter 2: stack[8] => %edx # parameter 2: stack[8] => %edx
@ -2038,15 +2037,15 @@ PREPARE_ROUNDKEY_128:
/* /*
void AES_192_Key_Expansion (const unsigned char *userkey, void AES_192_Key_Expansion_AESNI (const unsigned char *userkey,
unsigned char *key) unsigned char *key)
*/ */
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_192_Key_Expansion .globl AES_192_Key_Expansion_AESNI
AES_192_Key_Expansion: AES_192_Key_Expansion_AESNI:
#else #else
.globl _AES_192_Key_Expansion .globl _AES_192_Key_Expansion_AESNI
_AES_192_Key_Expansion: _AES_192_Key_Expansion_AESNI:
#endif #endif
# parameter 1: stack[4] => %eax # parameter 1: stack[4] => %eax
# parameter 2: stack[8] => %edx # parameter 2: stack[8] => %edx
@ -2131,15 +2130,15 @@ PREPARE_ROUNDKEY_192:
/* /*
void AES_256_Key_Expansion (const unsigned char *userkey, void AES_256_Key_Expansion_AESNI (const unsigned char *userkey,
unsigned char *key) unsigned char *key)
*/ */
#ifndef __APPLE__ #ifndef __APPLE__
.globl AES_256_Key_Expansion .globl AES_256_Key_Expansion_AESNI
AES_256_Key_Expansion: AES_256_Key_Expansion_AESNI:
#else #else
.globl _AES_256_Key_Expansion .globl _AES_256_Key_Expansion_AESNI
_AES_256_Key_Expansion: _AES_256_Key_Expansion_AESNI:
#endif #endif
# parameter 1: stack[4] => %eax # parameter 1: stack[4] => %eax
# parameter 2: stack[8] => %edx # parameter 2: stack[8] => %edx

50
wolfcrypt/src/aes_asm.asm
View File

@ -47,14 +47,14 @@ ENDIF
; /* ; /*
; AES_CBC_encrypt[const ,unsigned char*in ; AES_CBC_encrypt_AESNI[const ,unsigned char*in
; unsigned ,char*out ; unsigned ,char*out
; unsigned ,char ivec+16 ; unsigned ,char ivec+16
; unsigned ,long length ; unsigned ,long length
; const ,unsigned char*KS ; const ,unsigned char*KS
; int nr] ; int nr]
; */ ; */
AES_CBC_encrypt PROC AES_CBC_encrypt_AESNI PROC
;# parameter 1: rdi ;# parameter 1: rdi
;# parameter 2: rsi ;# parameter 2: rsi
;# parameter 3: rdx ;# parameter 3: rdx
@ -117,16 +117,16 @@ LAST:
mov rdi,rax mov rdi,rax
mov rsi,r11 mov rsi,r11
ret ret
AES_CBC_encrypt ENDP AES_CBC_encrypt_AESNI ENDP
; void AES_CBC_decrypt_by4(const unsigned char* in, ; void AES_CBC_decrypt_AESNI_by4(const unsigned char* in,
; unsigned char* out, ; unsigned char* out,
; unsigned char ivec[16], ; unsigned char ivec[16],
; unsigned long length, ; unsigned long length,
; const unsigned char* KS, ; const unsigned char* KS,
; int nr) ; int nr)
AES_CBC_decrypt_by4 PROC AES_CBC_decrypt_AESNI_by4 PROC
; parameter 1: rdi ; parameter 1: rdi
; parameter 2: rsi ; parameter 2: rsi
; parameter 3: rdx ; parameter 3: rdx
@ -325,16 +325,16 @@ DEND_4:
movdqa xmm15, [rsp+112] movdqa xmm15, [rsp+112]
add rsp, 8+8*16 ; 8 = align stack , 8 xmm6-12,15 16 bytes each add rsp, 8+8*16 ; 8 = align stack , 8 xmm6-12,15 16 bytes each
ret ret
AES_CBC_decrypt_by4 ENDP AES_CBC_decrypt_AESNI_by4 ENDP
; void AES_CBC_decrypt_by6(const unsigned char *in, ; void AES_CBC_decrypt_AESNI_by6(const unsigned char *in,
; unsigned char *out, ; unsigned char *out,
; unsigned char ivec[16], ; unsigned char ivec[16],
; unsigned long length, ; unsigned long length,
; const unsigned char *KS, ; const unsigned char *KS,
; int nr) ; int nr)
AES_CBC_decrypt_by6 PROC AES_CBC_decrypt_AESNI_by6 PROC
; parameter 1: rdi - in ; parameter 1: rdi - in
; parameter 2: rsi - out ; parameter 2: rsi - out
; parameter 3: rdx - ivec ; parameter 3: rdx - ivec
@ -582,16 +582,16 @@ DEND_6:
movdqa xmm14, [rsp+128] movdqa xmm14, [rsp+128]
add rsp, 8+9*16 ; 8 = align stack , 9 xmm6-14 16 bytes each add rsp, 8+9*16 ; 8 = align stack , 9 xmm6-14 16 bytes each
ret ret
AES_CBC_decrypt_by6 ENDP AES_CBC_decrypt_AESNI_by6 ENDP
; void AES_CBC_decrypt_by8(const unsigned char *in, ; void AES_CBC_decrypt_AESNI_by8(const unsigned char *in,
; unsigned char *out, ; unsigned char *out,
; unsigned char ivec[16], ; unsigned char ivec[16],
; unsigned long length, ; unsigned long length,
; const unsigned char *KS, ; const unsigned char *KS,
; int nr) ; int nr)
AES_CBC_decrypt_by8 PROC AES_CBC_decrypt_AESNI_by8 PROC
; parameter 1: rdi - in ; parameter 1: rdi - in
; parameter 2: rsi - out ; parameter 2: rsi - out
; parameter 3: rdx - ivec ; parameter 3: rdx - ivec
@ -865,18 +865,18 @@ DEND_8:
movdqa xmm13, [rsp+112] movdqa xmm13, [rsp+112]
add rsp, 8+8*16 ; 8 = align stack , 8 xmm6-13 16 bytes each add rsp, 8+8*16 ; 8 = align stack , 8 xmm6-13 16 bytes each
ret ret
AES_CBC_decrypt_by8 ENDP AES_CBC_decrypt_AESNI_by8 ENDP
; /* ; /*
; AES_ECB_encrypt[const ,unsigned char*in ; AES_ECB_encrypt_AESNI[const ,unsigned char*in
; unsigned ,char*out ; unsigned ,char*out
; unsigned ,long length ; unsigned ,long length
; const ,unsigned char*KS ; const ,unsigned char*KS
; int nr] ; int nr]
; */ ; */
; . globl AES_ECB_encrypt ; . globl AES_ECB_encrypt
AES_ECB_encrypt PROC AES_ECB_encrypt_AESNI PROC
;# parameter 1: rdi ;# parameter 1: rdi
;# parameter 2: rsi ;# parameter 2: rsi
;# parameter 3: rdx ;# parameter 3: rdx
@ -1054,7 +1054,7 @@ EECB_END_4:
movdqa xmm12, [rsp+48] movdqa xmm12, [rsp+48]
add rsp,8+4*16 ; 8 = align stack , 4 xmm9-12 16 bytes each add rsp,8+4*16 ; 8 = align stack , 4 xmm9-12 16 bytes each
ret ret
AES_ECB_encrypt ENDP AES_ECB_encrypt_AESNI ENDP
; /* ; /*
; AES_ECB_decrypt[const ,unsigned char*in ; AES_ECB_decrypt[const ,unsigned char*in
@ -1241,17 +1241,17 @@ DECB_END_4:
movdqa xmm12, [rsp+48] movdqa xmm12, [rsp+48]
add rsp,8+4*16 ; 8 = align stack , 4 xmm9-12 16 bytes each add rsp,8+4*16 ; 8 = align stack , 4 xmm9-12 16 bytes each
ret ret
AES_ECB_decrypt ENDP AES_ECB_decrypt_AESNI ENDP
; /* ; /*
; void ,AES_128_Key_Expansion[const unsigned char*userkey ; void ,AES_128_Key_Expansion_AESNI[const unsigned char*userkey
; unsigned char*key_schedule]/ ; unsigned char*key_schedule]/
; */ ; */
; . align 16,0x90 ; . align 16,0x90
; . globl AES_128_Key_Expansion ; . globl AES_128_Key_Expansion
AES_128_Key_Expansion PROC AES_128_Key_Expansion_AESNI PROC
;# parameter 1: rdi ;# parameter 1: rdi
;# parameter 2: rsi ;# parameter 2: rsi
@ -1322,14 +1322,14 @@ PREPARE_ROUNDKEY_128:
pxor xmm1,xmm3 pxor xmm1,xmm3
pxor xmm1,xmm2 pxor xmm1,xmm2
ret ret
AES_128_Key_Expansion ENDP AES_128_Key_Expansion_AESNI ENDP
; /* ; /*
; void ,AES_192_Key_Expansion[const unsigned char*userkey ; void ,AES_192_Key_Expansion_AESNI[const unsigned char*userkey
; unsigned char*key] ; unsigned char*key]
; */ ; */
; . globl AES_192_Key_Expansion ; . globl AES_192_Key_Expansion
AES_192_Key_Expansion PROC AES_192_Key_Expansion_AESNI PROC
;# parameter 1: rdi ;# parameter 1: rdi
;# parameter 2: rsi ;# parameter 2: rsi
@ -1426,14 +1426,14 @@ PREPARE_ROUNDKEY_192:
pxor xmm3,xmm4 pxor xmm3,xmm4
pxor xmm3,xmm2 pxor xmm3,xmm2
ret ret
AES_192_Key_Expansion ENDP AES_192_Key_Expansion_AESNI ENDP
; /* ; /*
; void ,AES_256_Key_Expansion[const unsigned char*userkey ; void ,AES_256_Key_Expansion_AESNI[const unsigned char*userkey
; unsigned char*key] ; unsigned char*key]
; */ ; */
; . globl AES_256_Key_Expansion ; . globl AES_256_Key_Expansion
AES_256_Key_Expansion PROC AES_256_Key_Expansion_AESNI PROC
;# parameter 1: rdi ;# parameter 1: rdi
;# parameter 2: rsi ;# parameter 2: rsi
@ -1495,7 +1495,7 @@ AES_256_Key_Expansion PROC
mov rdi,rax mov rdi,rax
mov rsi,r11 mov rsi,r11
ret ret
AES_256_Key_Expansion ENDP AES_256_Key_Expansion_AESNI ENDP
MAKE_RK256_a: MAKE_RK256_a:
pshufd xmm2,xmm2,0ffh pshufd xmm2,xmm2,0ffh

292
wolfcrypt/src/aes_gcm_asm.S
View File

@ -180,15 +180,15 @@ L_aes_gcm_mod2_128:
.quad 0x1, 0xc200000000000000 .quad 0x1, 0xc200000000000000
#ifndef __APPLE__ #ifndef __APPLE__
.text .text
.globl AES_GCM_encrypt .globl AES_GCM_encrypt_aesni
.type AES_GCM_encrypt,@function .type AES_GCM_encrypt_aesni,@function
.align 16 .align 16
AES_GCM_encrypt: AES_GCM_encrypt_aesni:
#else #else
.section __TEXT,__text .section __TEXT,__text
.globl _AES_GCM_encrypt .globl _AES_GCM_encrypt_aesni
.p2align 4 .p2align 4
_AES_GCM_encrypt: _AES_GCM_encrypt_aesni:
#endif /* __APPLE__ */ #endif /* __APPLE__ */
pushq %r13 pushq %r13
pushq %r12 pushq %r12
@ -207,7 +207,7 @@ _AES_GCM_encrypt:
pxor %xmm6, %xmm6 pxor %xmm6, %xmm6
cmpl $12, %ebx cmpl $12, %ebx
movl %ebx, %edx movl %ebx, %edx
jne L_AES_GCM_encrypt_iv_not_12 jne L_AES_GCM_encrypt_aesni_iv_not_12
# # Calculate values when IV is 12 bytes # # Calculate values when IV is 12 bytes
# Set counter based on IV # Set counter based on IV
movl $0x1000000, %ecx movl $0x1000000, %ecx
@ -247,7 +247,7 @@ _AES_GCM_encrypt:
aesenc %xmm7, %xmm1 aesenc %xmm7, %xmm1
cmpl $11, %r10d cmpl $11, %r10d
movdqa 160(%r15), %xmm7 movdqa 160(%r15), %xmm7
jl L_AES_GCM_encrypt_calc_iv_12_last jl L_AES_GCM_encrypt_aesni_calc_iv_12_last
aesenc %xmm7, %xmm5 aesenc %xmm7, %xmm5
aesenc %xmm7, %xmm1 aesenc %xmm7, %xmm1
movdqa 176(%r15), %xmm7 movdqa 176(%r15), %xmm7
@ -255,20 +255,20 @@ _AES_GCM_encrypt:
aesenc %xmm7, %xmm1 aesenc %xmm7, %xmm1
cmpl $13, %r10d cmpl $13, %r10d
movdqa 192(%r15), %xmm7 movdqa 192(%r15), %xmm7
jl L_AES_GCM_encrypt_calc_iv_12_last jl L_AES_GCM_encrypt_aesni_calc_iv_12_last
aesenc %xmm7, %xmm5 aesenc %xmm7, %xmm5
aesenc %xmm7, %xmm1 aesenc %xmm7, %xmm1
movdqa 208(%r15), %xmm7 movdqa 208(%r15), %xmm7
aesenc %xmm7, %xmm5 aesenc %xmm7, %xmm5
aesenc %xmm7, %xmm1 aesenc %xmm7, %xmm1
movdqa 224(%r15), %xmm7 movdqa 224(%r15), %xmm7
L_AES_GCM_encrypt_calc_iv_12_last: L_AES_GCM_encrypt_aesni_calc_iv_12_last:
aesenclast %xmm7, %xmm5 aesenclast %xmm7, %xmm5
aesenclast %xmm7, %xmm1 aesenclast %xmm7, %xmm1
pshufb L_aes_gcm_bswap_mask(%rip), %xmm5 pshufb L_aes_gcm_bswap_mask(%rip), %xmm5
movdqu %xmm1, 144(%rsp) movdqu %xmm1, 144(%rsp)
jmp L_AES_GCM_encrypt_iv_done jmp L_AES_GCM_encrypt_aesni_iv_done
L_AES_GCM_encrypt_iv_not_12: L_AES_GCM_encrypt_aesni_iv_not_12:
# Calculate values when IV is not 12 bytes # Calculate values when IV is not 12 bytes
# H = Encrypt X(=0) # H = Encrypt X(=0)
movdqa (%r15), %xmm5 movdqa (%r15), %xmm5
@ -283,27 +283,27 @@ L_AES_GCM_encrypt_iv_not_12:
aesenc 144(%r15), %xmm5 aesenc 144(%r15), %xmm5
cmpl $11, %r10d cmpl $11, %r10d
movdqa 160(%r15), %xmm9 movdqa 160(%r15), %xmm9
jl L_AES_GCM_encrypt_calc_iv_1_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc %xmm9, %xmm5 aesenc %xmm9, %xmm5
aesenc 176(%r15), %xmm5 aesenc 176(%r15), %xmm5
cmpl $13, %r10d cmpl $13, %r10d
movdqa 192(%r15), %xmm9 movdqa 192(%r15), %xmm9
jl L_AES_GCM_encrypt_calc_iv_1_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc %xmm9, %xmm5 aesenc %xmm9, %xmm5
aesenc 208(%r15), %xmm5 aesenc 208(%r15), %xmm5
movdqa 224(%r15), %xmm9 movdqa 224(%r15), %xmm9
L_AES_GCM_encrypt_calc_iv_1_aesenc_avx_last: L_AES_GCM_encrypt_aesni_calc_iv_1_aesenc_avx_last:
aesenclast %xmm9, %xmm5 aesenclast %xmm9, %xmm5
pshufb L_aes_gcm_bswap_mask(%rip), %xmm5 pshufb L_aes_gcm_bswap_mask(%rip), %xmm5
# Calc counter # Calc counter
# Initialization vector # Initialization vector
cmpl $0x00, %edx cmpl $0x00, %edx
movq $0x00, %rcx movq $0x00, %rcx
je L_AES_GCM_encrypt_calc_iv_done je L_AES_GCM_encrypt_aesni_calc_iv_done
cmpl $16, %edx cmpl $16, %edx
jl L_AES_GCM_encrypt_calc_iv_lt16 jl L_AES_GCM_encrypt_aesni_calc_iv_lt16
andl $0xfffffff0, %edx andl $0xfffffff0, %edx
L_AES_GCM_encrypt_calc_iv_16_loop: L_AES_GCM_encrypt_aesni_calc_iv_16_loop:
movdqu (%rax,%rcx,1), %xmm8 movdqu (%rax,%rcx,1), %xmm8
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8 pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
pxor %xmm8, %xmm4 pxor %xmm8, %xmm4
@ -363,22 +363,22 @@ L_AES_GCM_encrypt_calc_iv_16_loop:
pxor %xmm2, %xmm4 pxor %xmm2, %xmm4
addl $16, %ecx addl $16, %ecx
cmpl %edx, %ecx cmpl %edx, %ecx
jl L_AES_GCM_encrypt_calc_iv_16_loop jl L_AES_GCM_encrypt_aesni_calc_iv_16_loop
movl %ebx, %edx movl %ebx, %edx
cmpl %edx, %ecx cmpl %edx, %ecx
je L_AES_GCM_encrypt_calc_iv_done je L_AES_GCM_encrypt_aesni_calc_iv_done
L_AES_GCM_encrypt_calc_iv_lt16: L_AES_GCM_encrypt_aesni_calc_iv_lt16:
subq $16, %rsp subq $16, %rsp
pxor %xmm8, %xmm8 pxor %xmm8, %xmm8
xorl %ebx, %ebx xorl %ebx, %ebx
movdqu %xmm8, (%rsp) movdqu %xmm8, (%rsp)
L_AES_GCM_encrypt_calc_iv_loop: L_AES_GCM_encrypt_aesni_calc_iv_loop:
movzbl (%rax,%rcx,1), %r13d movzbl (%rax,%rcx,1), %r13d
movb %r13b, (%rsp,%rbx,1) movb %r13b, (%rsp,%rbx,1)
incl %ecx incl %ecx
incl %ebx incl %ebx
cmpl %edx, %ecx cmpl %edx, %ecx
jl L_AES_GCM_encrypt_calc_iv_loop jl L_AES_GCM_encrypt_aesni_calc_iv_loop
movdqu (%rsp), %xmm8 movdqu (%rsp), %xmm8
addq $16, %rsp addq $16, %rsp
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8 pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
@ -437,7 +437,7 @@ L_AES_GCM_encrypt_calc_iv_loop:
pxor %xmm1, %xmm2 pxor %xmm1, %xmm2
pxor %xmm7, %xmm2 pxor %xmm7, %xmm2
pxor %xmm2, %xmm4 pxor %xmm2, %xmm4
L_AES_GCM_encrypt_calc_iv_done: L_AES_GCM_encrypt_aesni_calc_iv_done:
# T = Encrypt counter # T = Encrypt counter
pxor %xmm0, %xmm0 pxor %xmm0, %xmm0
shll $3, %edx shll $3, %edx
@ -512,28 +512,28 @@ L_AES_GCM_encrypt_calc_iv_done:
aesenc 144(%r15), %xmm8 aesenc 144(%r15), %xmm8
cmpl $11, %r10d cmpl $11, %r10d
movdqa 160(%r15), %xmm9 movdqa 160(%r15), %xmm9
jl L_AES_GCM_encrypt_calc_iv_2_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc %xmm9, %xmm8 aesenc %xmm9, %xmm8
aesenc 176(%r15), %xmm8 aesenc 176(%r15), %xmm8
cmpl $13, %r10d cmpl $13, %r10d
movdqa 192(%r15), %xmm9 movdqa 192(%r15), %xmm9
jl L_AES_GCM_encrypt_calc_iv_2_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc %xmm9, %xmm8 aesenc %xmm9, %xmm8
aesenc 208(%r15), %xmm8 aesenc 208(%r15), %xmm8
movdqa 224(%r15), %xmm9 movdqa 224(%r15), %xmm9
L_AES_GCM_encrypt_calc_iv_2_aesenc_avx_last: L_AES_GCM_encrypt_aesni_calc_iv_2_aesenc_avx_last:
aesenclast %xmm9, %xmm8 aesenclast %xmm9, %xmm8
movdqu %xmm8, 144(%rsp) movdqu %xmm8, 144(%rsp)
L_AES_GCM_encrypt_iv_done: L_AES_GCM_encrypt_aesni_iv_done:
# Additional authentication data # Additional authentication data
movl %r11d, %edx movl %r11d, %edx
cmpl $0x00, %edx cmpl $0x00, %edx
je L_AES_GCM_encrypt_calc_aad_done je L_AES_GCM_encrypt_aesni_calc_aad_done
xorl %ecx, %ecx xorl %ecx, %ecx
cmpl $16, %edx cmpl $16, %edx
jl L_AES_GCM_encrypt_calc_aad_lt16 jl L_AES_GCM_encrypt_aesni_calc_aad_lt16
andl $0xfffffff0, %edx andl $0xfffffff0, %edx
L_AES_GCM_encrypt_calc_aad_16_loop: L_AES_GCM_encrypt_aesni_calc_aad_16_loop:
movdqu (%r12,%rcx,1), %xmm8 movdqu (%r12,%rcx,1), %xmm8
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8 pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
pxor %xmm8, %xmm6 pxor %xmm8, %xmm6
@ -593,22 +593,22 @@ L_AES_GCM_encrypt_calc_aad_16_loop:
pxor %xmm2, %xmm6 pxor %xmm2, %xmm6
addl $16, %ecx addl $16, %ecx
cmpl %edx, %ecx cmpl %edx, %ecx
jl L_AES_GCM_encrypt_calc_aad_16_loop jl L_AES_GCM_encrypt_aesni_calc_aad_16_loop
movl %r11d, %edx movl %r11d, %edx
cmpl %edx, %ecx cmpl %edx, %ecx
je L_AES_GCM_encrypt_calc_aad_done je L_AES_GCM_encrypt_aesni_calc_aad_done
L_AES_GCM_encrypt_calc_aad_lt16: L_AES_GCM_encrypt_aesni_calc_aad_lt16:
subq $16, %rsp subq $16, %rsp
pxor %xmm8, %xmm8 pxor %xmm8, %xmm8
xorl %ebx, %ebx xorl %ebx, %ebx
movdqu %xmm8, (%rsp) movdqu %xmm8, (%rsp)
L_AES_GCM_encrypt_calc_aad_loop: L_AES_GCM_encrypt_aesni_calc_aad_loop:
movzbl (%r12,%rcx,1), %r13d movzbl (%r12,%rcx,1), %r13d
movb %r13b, (%rsp,%rbx,1) movb %r13b, (%rsp,%rbx,1)
incl %ecx incl %ecx
incl %ebx incl %ebx
cmpl %edx, %ecx cmpl %edx, %ecx
jl L_AES_GCM_encrypt_calc_aad_loop jl L_AES_GCM_encrypt_aesni_calc_aad_loop
movdqu (%rsp), %xmm8 movdqu (%rsp), %xmm8
addq $16, %rsp addq $16, %rsp
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8 pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
@ -667,7 +667,7 @@ L_AES_GCM_encrypt_calc_aad_loop:
pxor %xmm1, %xmm2 pxor %xmm1, %xmm2
pxor %xmm7, %xmm2 pxor %xmm7, %xmm2
pxor %xmm2, %xmm6 pxor %xmm2, %xmm6
L_AES_GCM_encrypt_calc_aad_done: L_AES_GCM_encrypt_aesni_calc_aad_done:
# Calculate counter and H # Calculate counter and H
pshufb L_aes_gcm_bswap_epi64(%rip), %xmm4 pshufb L_aes_gcm_bswap_epi64(%rip), %xmm4
movdqa %xmm5, %xmm9 movdqa %xmm5, %xmm9
@ -685,7 +685,7 @@ L_AES_GCM_encrypt_calc_aad_done:
xorq %rbx, %rbx xorq %rbx, %rbx
cmpl $0x80, %r9d cmpl $0x80, %r9d
movl %r9d, %r13d movl %r9d, %r13d
jl L_AES_GCM_encrypt_done_128 jl L_AES_GCM_encrypt_aesni_done_128
andl $0xffffff80, %r13d andl $0xffffff80, %r13d
movdqa %xmm6, %xmm2 movdqa %xmm6, %xmm2
# H ^ 1 # H ^ 1
@ -1104,7 +1104,7 @@ L_AES_GCM_encrypt_calc_aad_done:
aesenc %xmm7, %xmm15 aesenc %xmm7, %xmm15
cmpl $11, %r10d cmpl $11, %r10d
movdqa 160(%r15), %xmm7 movdqa 160(%r15), %xmm7
jl L_AES_GCM_encrypt_enc_done jl L_AES_GCM_encrypt_aesni_enc_done
aesenc %xmm7, %xmm8 aesenc %xmm7, %xmm8
aesenc %xmm7, %xmm9 aesenc %xmm7, %xmm9
aesenc %xmm7, %xmm10 aesenc %xmm7, %xmm10
@ -1124,7 +1124,7 @@ L_AES_GCM_encrypt_calc_aad_done:
aesenc %xmm7, %xmm15 aesenc %xmm7, %xmm15
cmpl $13, %r10d cmpl $13, %r10d
movdqa 192(%r15), %xmm7 movdqa 192(%r15), %xmm7
jl L_AES_GCM_encrypt_enc_done jl L_AES_GCM_encrypt_aesni_enc_done
aesenc %xmm7, %xmm8 aesenc %xmm7, %xmm8
aesenc %xmm7, %xmm9 aesenc %xmm7, %xmm9
aesenc %xmm7, %xmm10 aesenc %xmm7, %xmm10
@ -1143,7 +1143,7 @@ L_AES_GCM_encrypt_calc_aad_done:
aesenc %xmm7, %xmm14 aesenc %xmm7, %xmm14
aesenc %xmm7, %xmm15 aesenc %xmm7, %xmm15
movdqa 224(%r15), %xmm7 movdqa 224(%r15), %xmm7
L_AES_GCM_encrypt_enc_done: L_AES_GCM_encrypt_aesni_enc_done:
aesenclast %xmm7, %xmm8 aesenclast %xmm7, %xmm8
aesenclast %xmm7, %xmm9 aesenclast %xmm7, %xmm9
movdqu (%rdi), %xmm0 movdqu (%rdi), %xmm0
@ -1178,9 +1178,9 @@ L_AES_GCM_encrypt_enc_done:
movdqu %xmm15, 112(%rsi) movdqu %xmm15, 112(%rsi)
cmpl $0x80, %r13d cmpl $0x80, %r13d
movl $0x80, %ebx movl $0x80, %ebx
jle L_AES_GCM_encrypt_end_128 jle L_AES_GCM_encrypt_aesni_end_128
# More 128 bytes of input # More 128 bytes of input
L_AES_GCM_encrypt_ghash_128: L_AES_GCM_encrypt_aesni_ghash_128:
leaq (%rdi,%rbx,1), %rcx leaq (%rdi,%rbx,1), %rcx
leaq (%rsi,%rbx,1), %rdx leaq (%rsi,%rbx,1), %rdx
movdqu 128(%rsp), %xmm8 movdqu 128(%rsp), %xmm8
@ -1448,7 +1448,7 @@ L_AES_GCM_encrypt_ghash_128:
pxor %xmm3, %xmm2 pxor %xmm3, %xmm2
cmpl $11, %r10d cmpl $11, %r10d
movdqa 160(%r15), %xmm7 movdqa 160(%r15), %xmm7
jl L_AES_GCM_encrypt_aesenc_128_ghash_avx_done jl L_AES_GCM_encrypt_aesni_aesenc_128_ghash_avx_done
aesenc %xmm7, %xmm8 aesenc %xmm7, %xmm8
aesenc %xmm7, %xmm9 aesenc %xmm7, %xmm9
aesenc %xmm7, %xmm10 aesenc %xmm7, %xmm10
@ -1468,7 +1468,7 @@ L_AES_GCM_encrypt_ghash_128:
aesenc %xmm7, %xmm15 aesenc %xmm7, %xmm15
cmpl $13, %r10d cmpl $13, %r10d
movdqa 192(%r15), %xmm7 movdqa 192(%r15), %xmm7
jl L_AES_GCM_encrypt_aesenc_128_ghash_avx_done jl L_AES_GCM_encrypt_aesni_aesenc_128_ghash_avx_done
aesenc %xmm7, %xmm8 aesenc %xmm7, %xmm8
aesenc %xmm7, %xmm9 aesenc %xmm7, %xmm9
aesenc %xmm7, %xmm10 aesenc %xmm7, %xmm10
@ -1487,7 +1487,7 @@ L_AES_GCM_encrypt_ghash_128:
aesenc %xmm7, %xmm14 aesenc %xmm7, %xmm14
aesenc %xmm7, %xmm15 aesenc %xmm7, %xmm15
movdqa 224(%r15), %xmm7 movdqa 224(%r15), %xmm7
L_AES_GCM_encrypt_aesenc_128_ghash_avx_done: L_AES_GCM_encrypt_aesni_aesenc_128_ghash_avx_done:
aesenclast %xmm7, %xmm8 aesenclast %xmm7, %xmm8
aesenclast %xmm7, %xmm9 aesenclast %xmm7, %xmm9
movdqu (%rcx), %xmm0 movdqu (%rcx), %xmm0
@ -1522,8 +1522,8 @@ L_AES_GCM_encrypt_aesenc_128_ghash_avx_done:
movdqu %xmm15, 112(%rdx) movdqu %xmm15, 112(%rdx)
addl $0x80, %ebx addl $0x80, %ebx
cmpl %r13d, %ebx cmpl %r13d, %ebx
jl L_AES_GCM_encrypt_ghash_128 jl L_AES_GCM_encrypt_aesni_ghash_128
L_AES_GCM_encrypt_end_128: L_AES_GCM_encrypt_aesni_end_128:
movdqa L_aes_gcm_bswap_mask(%rip), %xmm4 movdqa L_aes_gcm_bswap_mask(%rip), %xmm4
pshufb %xmm4, %xmm8 pshufb %xmm4, %xmm8
pshufb %xmm4, %xmm9 pshufb %xmm4, %xmm9
@ -1710,14 +1710,14 @@ L_AES_GCM_encrypt_end_128:
pxor %xmm4, %xmm2 pxor %xmm4, %xmm2
pxor %xmm2, %xmm6 pxor %xmm2, %xmm6
movdqu (%rsp), %xmm5 movdqu (%rsp), %xmm5
L_AES_GCM_encrypt_done_128: L_AES_GCM_encrypt_aesni_done_128:
movl %r9d, %edx movl %r9d, %edx
cmpl %edx, %ebx cmpl %edx, %ebx
jge L_AES_GCM_encrypt_done_enc jge L_AES_GCM_encrypt_aesni_done_enc
movl %r9d, %r13d movl %r9d, %r13d
andl $0xfffffff0, %r13d andl $0xfffffff0, %r13d
cmpl %r13d, %ebx cmpl %r13d, %ebx
jge L_AES_GCM_encrypt_last_block_done jge L_AES_GCM_encrypt_aesni_last_block_done
leaq (%rdi,%rbx,1), %rcx leaq (%rdi,%rbx,1), %rcx
leaq (%rsi,%rbx,1), %rdx leaq (%rsi,%rbx,1), %rdx
movdqu 128(%rsp), %xmm8 movdqu 128(%rsp), %xmm8
@ -1737,16 +1737,16 @@ L_AES_GCM_encrypt_done_128:
aesenc 144(%r15), %xmm8 aesenc 144(%r15), %xmm8
cmpl $11, %r10d cmpl $11, %r10d
movdqa 160(%r15), %xmm9 movdqa 160(%r15), %xmm9
jl L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_aesenc_block_aesenc_avx_last
aesenc %xmm9, %xmm8 aesenc %xmm9, %xmm8
aesenc 176(%r15), %xmm8 aesenc 176(%r15), %xmm8
cmpl $13, %r10d cmpl $13, %r10d
movdqa 192(%r15), %xmm9 movdqa 192(%r15), %xmm9
jl L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_aesenc_block_aesenc_avx_last
aesenc %xmm9, %xmm8 aesenc %xmm9, %xmm8
aesenc 208(%r15), %xmm8 aesenc 208(%r15), %xmm8
movdqa 224(%r15), %xmm9 movdqa 224(%r15), %xmm9
L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last: L_AES_GCM_encrypt_aesni_aesenc_block_aesenc_avx_last:
aesenclast %xmm9, %xmm8 aesenclast %xmm9, %xmm8
movdqu (%rcx), %xmm9 movdqu (%rcx), %xmm9
pxor %xmm9, %xmm8 pxor %xmm9, %xmm8
@ -1755,8 +1755,8 @@ L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last:
pxor %xmm8, %xmm6 pxor %xmm8, %xmm6
addl $16, %ebx addl $16, %ebx
cmpl %r13d, %ebx cmpl %r13d, %ebx
jge L_AES_GCM_encrypt_last_block_ghash jge L_AES_GCM_encrypt_aesni_last_block_ghash
L_AES_GCM_encrypt_last_block_start: L_AES_GCM_encrypt_aesni_last_block_start:
leaq (%rdi,%rbx,1), %rcx leaq (%rdi,%rbx,1), %rcx
leaq (%rsi,%rbx,1), %rdx leaq (%rsi,%rbx,1), %rdx
movdqu 128(%rsp), %xmm8 movdqu 128(%rsp), %xmm8
@ -1801,16 +1801,16 @@ L_AES_GCM_encrypt_last_block_start:
pxor %xmm3, %xmm6 pxor %xmm3, %xmm6
cmpl $11, %r10d cmpl $11, %r10d
movdqa 160(%r15), %xmm9 movdqa 160(%r15), %xmm9
jl L_AES_GCM_encrypt_aesenc_gfmul_last jl L_AES_GCM_encrypt_aesni_aesenc_gfmul_last
aesenc %xmm9, %xmm8 aesenc %xmm9, %xmm8
aesenc 176(%r15), %xmm8 aesenc 176(%r15), %xmm8
cmpl $13, %r10d cmpl $13, %r10d
movdqa 192(%r15), %xmm9 movdqa 192(%r15), %xmm9
jl L_AES_GCM_encrypt_aesenc_gfmul_last jl L_AES_GCM_encrypt_aesni_aesenc_gfmul_last
aesenc %xmm9, %xmm8 aesenc %xmm9, %xmm8
aesenc 208(%r15), %xmm8 aesenc 208(%r15), %xmm8
movdqa 224(%r15), %xmm9 movdqa 224(%r15), %xmm9
L_AES_GCM_encrypt_aesenc_gfmul_last: L_AES_GCM_encrypt_aesni_aesenc_gfmul_last:
aesenclast %xmm9, %xmm8 aesenclast %xmm9, %xmm8
movdqu (%rcx), %xmm9 movdqu (%rcx), %xmm9
pxor %xmm9, %xmm8 pxor %xmm9, %xmm8
@ -1819,8 +1819,8 @@ L_AES_GCM_encrypt_aesenc_gfmul_last:
pxor %xmm8, %xmm6 pxor %xmm8, %xmm6
addl $16, %ebx addl $16, %ebx
cmpl %r13d, %ebx cmpl %r13d, %ebx
jl L_AES_GCM_encrypt_last_block_start jl L_AES_GCM_encrypt_aesni_last_block_start
L_AES_GCM_encrypt_last_block_ghash: L_AES_GCM_encrypt_aesni_last_block_ghash:
pshufd $0x4e, %xmm5, %xmm9 pshufd $0x4e, %xmm5, %xmm9
pshufd $0x4e, %xmm6, %xmm10 pshufd $0x4e, %xmm6, %xmm10
movdqa %xmm6, %xmm11 movdqa %xmm6, %xmm11
@ -1861,11 +1861,11 @@ L_AES_GCM_encrypt_last_block_ghash:
pxor %xmm13, %xmm14 pxor %xmm13, %xmm14
pxor %xmm8, %xmm14 pxor %xmm8, %xmm14
pxor %xmm14, %xmm6 pxor %xmm14, %xmm6
L_AES_GCM_encrypt_last_block_done: L_AES_GCM_encrypt_aesni_last_block_done:
movl %r9d, %ecx movl %r9d, %ecx
movl %ecx, %edx movl %ecx, %edx
andl $15, %ecx andl $15, %ecx
jz L_AES_GCM_encrypt_aesenc_last15_enc_avx_done jz L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_done
movdqu 128(%rsp), %xmm4 movdqu 128(%rsp), %xmm4
pshufb L_aes_gcm_bswap_epi64(%rip), %xmm4 pshufb L_aes_gcm_bswap_epi64(%rip), %xmm4
pxor (%r15), %xmm4 pxor (%r15), %xmm4
@ -1880,21 +1880,21 @@ L_AES_GCM_encrypt_last_block_done:
aesenc 144(%r15), %xmm4 aesenc 144(%r15), %xmm4
cmpl $11, %r10d cmpl $11, %r10d
movdqa 160(%r15), %xmm9 movdqa 160(%r15), %xmm9
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_aesenc_avx_last
aesenc %xmm9, %xmm4 aesenc %xmm9, %xmm4
aesenc 176(%r15), %xmm4 aesenc 176(%r15), %xmm4
cmpl $13, %r10d cmpl $13, %r10d
movdqa 192(%r15), %xmm9 movdqa 192(%r15), %xmm9
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_aesenc_avx_last
aesenc %xmm9, %xmm4 aesenc %xmm9, %xmm4
aesenc 208(%r15), %xmm4 aesenc 208(%r15), %xmm4
movdqa 224(%r15), %xmm9 movdqa 224(%r15), %xmm9
L_AES_GCM_encrypt_aesenc_last15_enc_avx_aesenc_avx_last: L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_aesenc_avx_last:
aesenclast %xmm9, %xmm4 aesenclast %xmm9, %xmm4
subq $16, %rsp subq $16, %rsp
xorl %ecx, %ecx xorl %ecx, %ecx
movdqu %xmm4, (%rsp) movdqu %xmm4, (%rsp)
L_AES_GCM_encrypt_aesenc_last15_enc_avx_loop: L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_loop:
movzbl (%rdi,%rbx,1), %r13d movzbl (%rdi,%rbx,1), %r13d
xorb (%rsp,%rcx,1), %r13b xorb (%rsp,%rcx,1), %r13b
movb %r13b, (%rsi,%rbx,1) movb %r13b, (%rsi,%rbx,1)
@ -1902,16 +1902,16 @@ L_AES_GCM_encrypt_aesenc_last15_enc_avx_loop:
incl %ebx incl %ebx
incl %ecx incl %ecx
cmpl %edx, %ebx cmpl %edx, %ebx
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_loop jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_loop
xorq %r13, %r13 xorq %r13, %r13
cmpl $16, %ecx cmpl $16, %ecx
je L_AES_GCM_encrypt_aesenc_last15_enc_avx_finish_enc je L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_finish_enc
L_AES_GCM_encrypt_aesenc_last15_enc_avx_byte_loop: L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_byte_loop:
movb %r13b, (%rsp,%rcx,1) movb %r13b, (%rsp,%rcx,1)
incl %ecx incl %ecx
cmpl $16, %ecx cmpl $16, %ecx
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_byte_loop jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_byte_loop
L_AES_GCM_encrypt_aesenc_last15_enc_avx_finish_enc: L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_finish_enc:
movdqu (%rsp), %xmm4 movdqu (%rsp), %xmm4
addq $16, %rsp addq $16, %rsp
pshufb L_aes_gcm_bswap_mask(%rip), %xmm4 pshufb L_aes_gcm_bswap_mask(%rip), %xmm4
@ -1956,8 +1956,8 @@ L_AES_GCM_encrypt_aesenc_last15_enc_avx_finish_enc:
pxor %xmm13, %xmm14 pxor %xmm13, %xmm14
pxor %xmm8, %xmm14 pxor %xmm8, %xmm14
pxor %xmm14, %xmm6 pxor %xmm14, %xmm6
L_AES_GCM_encrypt_aesenc_last15_enc_avx_done: L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_done:
L_AES_GCM_encrypt_done_enc: L_AES_GCM_encrypt_aesni_done_enc:
movl %r9d, %edx movl %r9d, %edx
movl %r11d, %ecx movl %r11d, %ecx
shlq $3, %rdx shlq $3, %rdx
@ -2009,19 +2009,19 @@ L_AES_GCM_encrypt_done_enc:
movdqu 144(%rsp), %xmm0 movdqu 144(%rsp), %xmm0
pxor %xmm6, %xmm0 pxor %xmm6, %xmm0
cmpl $16, %r14d cmpl $16, %r14d
je L_AES_GCM_encrypt_store_tag_16 je L_AES_GCM_encrypt_aesni_store_tag_16
xorq %rcx, %rcx xorq %rcx, %rcx
movdqu %xmm0, (%rsp) movdqu %xmm0, (%rsp)
L_AES_GCM_encrypt_store_tag_loop: L_AES_GCM_encrypt_aesni_store_tag_loop:
movzbl (%rsp,%rcx,1), %r13d movzbl (%rsp,%rcx,1), %r13d
movb %r13b, (%r8,%rcx,1) movb %r13b, (%r8,%rcx,1)
incl %ecx incl %ecx
cmpl %r14d, %ecx cmpl %r14d, %ecx
jne L_AES_GCM_encrypt_store_tag_loop jne L_AES_GCM_encrypt_aesni_store_tag_loop
jmp L_AES_GCM_encrypt_store_tag_done jmp L_AES_GCM_encrypt_aesni_store_tag_done
L_AES_GCM_encrypt_store_tag_16: L_AES_GCM_encrypt_aesni_store_tag_16:
movdqu %xmm0, (%r8) movdqu %xmm0, (%r8)
L_AES_GCM_encrypt_store_tag_done: L_AES_GCM_encrypt_aesni_store_tag_done:
addq $0xa0, %rsp addq $0xa0, %rsp
popq %r15 popq %r15
popq %r14 popq %r14
@ -2030,19 +2030,19 @@ L_AES_GCM_encrypt_store_tag_done:
popq %r13 popq %r13
repz retq repz retq
#ifndef __APPLE__ #ifndef __APPLE__
.size AES_GCM_encrypt,.-AES_GCM_encrypt .size AES_GCM_encrypt_aesni,.-AES_GCM_encrypt_aesni
#endif /* __APPLE__ */ #endif /* __APPLE__ */
#ifndef __APPLE__ #ifndef __APPLE__
.text .text
.globl AES_GCM_decrypt .globl AES_GCM_decrypt_aesni
.type AES_GCM_decrypt,@function .type AES_GCM_decrypt_aesni,@function
.align 16 .align 16
AES_GCM_decrypt: AES_GCM_decrypt_aesni:
#else #else
.section __TEXT,__text .section __TEXT,__text
.globl _AES_GCM_decrypt .globl _AES_GCM_decrypt_aesni
.p2align 4 .p2align 4
_AES_GCM_decrypt: _AES_GCM_decrypt_aesni:
#endif /* __APPLE__ */ #endif /* __APPLE__ */
pushq %r13 pushq %r13
pushq %r12 pushq %r12
@ -2063,7 +2063,7 @@ _AES_GCM_decrypt:
pxor %xmm6, %xmm6 pxor %xmm6, %xmm6
cmpl $12, %ebx cmpl $12, %ebx
movl %ebx, %edx movl %ebx, %edx
jne L_AES_GCM_decrypt_iv_not_12 jne L_AES_GCM_decrypt_aesni_iv_not_12
# # Calculate values when IV is 12 bytes # # Calculate values when IV is 12 bytes
# Set counter based on IV # Set counter based on IV
movl $0x1000000, %ecx movl $0x1000000, %ecx
@ -2103,7 +2103,7 @@ _AES_GCM_decrypt:
aesenc %xmm7, %xmm1 aesenc %xmm7, %xmm1
cmpl $11, %r10d cmpl $11, %r10d
movdqa 160(%r15), %xmm7 movdqa 160(%r15), %xmm7
jl L_AES_GCM_decrypt_calc_iv_12_last jl L_AES_GCM_decrypt_aesni_calc_iv_12_last
aesenc %xmm7, %xmm5 aesenc %xmm7, %xmm5
aesenc %xmm7, %xmm1 aesenc %xmm7, %xmm1
movdqa 176(%r15), %xmm7 movdqa 176(%r15), %xmm7
@ -2111,20 +2111,20 @@ _AES_GCM_decrypt:
aesenc %xmm7, %xmm1 aesenc %xmm7, %xmm1
cmpl $13, %r10d cmpl $13, %r10d
movdqa 192(%r15), %xmm7 movdqa 192(%r15), %xmm7
jl L_AES_GCM_decrypt_calc_iv_12_last jl L_AES_GCM_decrypt_aesni_calc_iv_12_last
aesenc %xmm7, %xmm5 aesenc %xmm7, %xmm5
aesenc %xmm7, %xmm1 aesenc %xmm7, %xmm1
movdqa 208(%r15), %xmm7 movdqa 208(%r15), %xmm7
aesenc %xmm7, %xmm5 aesenc %xmm7, %xmm5
aesenc %xmm7, %xmm1 aesenc %xmm7, %xmm1
movdqa 224(%r15), %xmm7 movdqa 224(%r15), %xmm7
L_AES_GCM_decrypt_calc_iv_12_last: L_AES_GCM_decrypt_aesni_calc_iv_12_last:
aesenclast %xmm7, %xmm5 aesenclast %xmm7, %xmm5
aesenclast %xmm7, %xmm1 aesenclast %xmm7, %xmm1
pshufb L_aes_gcm_bswap_mask(%rip), %xmm5 pshufb L_aes_gcm_bswap_mask(%rip), %xmm5
movdqu %xmm1, 144(%rsp) movdqu %xmm1, 144(%rsp)
jmp L_AES_GCM_decrypt_iv_done jmp L_AES_GCM_decrypt_aesni_iv_done
L_AES_GCM_decrypt_iv_not_12: L_AES_GCM_decrypt_aesni_iv_not_12:
# Calculate values when IV is not 12 bytes # Calculate values when IV is not 12 bytes
# H = Encrypt X(=0) # H = Encrypt X(=0)
movdqa (%r15), %xmm5 movdqa (%r15), %xmm5
@ -2139,27 +2139,27 @@ L_AES_GCM_decrypt_iv_not_12:
aesenc 144(%r15), %xmm5 aesenc 144(%r15), %xmm5
cmpl $11, %r10d cmpl $11, %r10d
movdqa 160(%r15), %xmm9 movdqa 160(%r15), %xmm9
jl L_AES_GCM_decrypt_calc_iv_1_aesenc_avx_last jl L_AES_GCM_decrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc %xmm9, %xmm5 aesenc %xmm9, %xmm5
aesenc 176(%r15), %xmm5 aesenc 176(%r15), %xmm5
cmpl $13, %r10d cmpl $13, %r10d
movdqa 192(%r15), %xmm9 movdqa 192(%r15), %xmm9
jl L_AES_GCM_decrypt_calc_iv_1_aesenc_avx_last jl L_AES_GCM_decrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc %xmm9, %xmm5 aesenc %xmm9, %xmm5
aesenc 208(%r15), %xmm5 aesenc 208(%r15), %xmm5
movdqa 224(%r15), %xmm9 movdqa 224(%r15), %xmm9
L_AES_GCM_decrypt_calc_iv_1_aesenc_avx_last: L_AES_GCM_decrypt_aesni_calc_iv_1_aesenc_avx_last:
aesenclast %xmm9, %xmm5 aesenclast %xmm9, %xmm5
pshufb L_aes_gcm_bswap_mask(%rip), %xmm5 pshufb L_aes_gcm_bswap_mask(%rip), %xmm5
# Calc counter # Calc counter
# Initialization vector # Initialization vector
cmpl $0x00, %edx cmpl $0x00, %edx
movq $0x00, %rcx movq $0x00, %rcx
je L_AES_GCM_decrypt_calc_iv_done je L_AES_GCM_decrypt_aesni_calc_iv_done
cmpl $16, %edx cmpl $16, %edx
jl L_AES_GCM_decrypt_calc_iv_lt16 jl L_AES_GCM_decrypt_aesni_calc_iv_lt16
andl $0xfffffff0, %edx andl $0xfffffff0, %edx
L_AES_GCM_decrypt_calc_iv_16_loop: L_AES_GCM_decrypt_aesni_calc_iv_16_loop:
movdqu (%rax,%rcx,1), %xmm8 movdqu (%rax,%rcx,1), %xmm8
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8 pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
pxor %xmm8, %xmm4 pxor %xmm8, %xmm4
@ -2219,22 +2219,22 @@ L_AES_GCM_decrypt_calc_iv_16_loop:
pxor %xmm2, %xmm4 pxor %xmm2, %xmm4
addl $16, %ecx addl $16, %ecx
cmpl %edx, %ecx cmpl %edx, %ecx
jl L_AES_GCM_decrypt_calc_iv_16_loop jl L_AES_GCM_decrypt_aesni_calc_iv_16_loop
movl %ebx, %edx movl %ebx, %edx
cmpl %edx, %ecx cmpl %edx, %ecx
je L_AES_GCM_decrypt_calc_iv_done je L_AES_GCM_decrypt_aesni_calc_iv_done
L_AES_GCM_decrypt_calc_iv_lt16: L_AES_GCM_decrypt_aesni_calc_iv_lt16:
subq $16, %rsp subq $16, %rsp
pxor %xmm8, %xmm8 pxor %xmm8, %xmm8
xorl %ebx, %ebx xorl %ebx, %ebx
movdqu %xmm8, (%rsp) movdqu %xmm8, (%rsp)
L_AES_GCM_decrypt_calc_iv_loop: L_AES_GCM_decrypt_aesni_calc_iv_loop:
movzbl (%rax,%rcx,1), %r13d movzbl (%rax,%rcx,1), %r13d
movb %r13b, (%rsp,%rbx,1) movb %r13b, (%rsp,%rbx,1)
incl %ecx incl %ecx
incl %ebx incl %ebx
cmpl %edx, %ecx cmpl %edx, %ecx
jl L_AES_GCM_decrypt_calc_iv_loop jl L_AES_GCM_decrypt_aesni_calc_iv_loop
movdqu (%rsp), %xmm8 movdqu (%rsp), %xmm8
addq $16, %rsp addq $16, %rsp
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8 pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
@ -2293,7 +2293,7 @@ L_AES_GCM_decrypt_calc_iv_loop:
pxor %xmm1, %xmm2 pxor %xmm1, %xmm2
pxor %xmm7, %xmm2 pxor %xmm7, %xmm2
pxor %xmm2, %xmm4 pxor %xmm2, %xmm4
L_AES_GCM_decrypt_calc_iv_done: L_AES_GCM_decrypt_aesni_calc_iv_done:
# T = Encrypt counter # T = Encrypt counter
pxor %xmm0, %xmm0 pxor %xmm0, %xmm0
shll $3, %edx shll $3, %edx
@ -2368,28 +2368,28 @@ L_AES_GCM_decrypt_calc_iv_done:
aesenc 144(%r15), %xmm8 aesenc 144(%r15), %xmm8
cmpl $11, %r10d cmpl $11, %r10d
movdqa 160(%r15), %xmm9 movdqa 160(%r15), %xmm9
jl L_AES_GCM_decrypt_calc_iv_2_aesenc_avx_last jl L_AES_GCM_decrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc %xmm9, %xmm8 aesenc %xmm9, %xmm8
aesenc 176(%r15), %xmm8 aesenc 176(%r15), %xmm8
cmpl $13, %r10d cmpl $13, %r10d
movdqa 192(%r15), %xmm9 movdqa 192(%r15), %xmm9
jl L_AES_GCM_decrypt_calc_iv_2_aesenc_avx_last jl L_AES_GCM_decrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc %xmm9, %xmm8 aesenc %xmm9, %xmm8
aesenc 208(%r15), %xmm8 aesenc 208(%r15), %xmm8
movdqa 224(%r15), %xmm9 movdqa 224(%r15), %xmm9
L_AES_GCM_decrypt_calc_iv_2_aesenc_avx_last: L_AES_GCM_decrypt_aesni_calc_iv_2_aesenc_avx_last:
aesenclast %xmm9, %xmm8 aesenclast %xmm9, %xmm8
movdqu %xmm8, 144(%rsp) movdqu %xmm8, 144(%rsp)
L_AES_GCM_decrypt_iv_done: L_AES_GCM_decrypt_aesni_iv_done:
# Additional authentication data # Additional authentication data
movl %r11d, %edx movl %r11d, %edx
cmpl $0x00, %edx cmpl $0x00, %edx
je L_AES_GCM_decrypt_calc_aad_done je L_AES_GCM_decrypt_aesni_calc_aad_done
xorl %ecx, %ecx xorl %ecx, %ecx
cmpl $16, %edx cmpl $16, %edx
jl L_AES_GCM_decrypt_calc_aad_lt16 jl L_AES_GCM_decrypt_aesni_calc_aad_lt16
andl $0xfffffff0, %edx andl $0xfffffff0, %edx
L_AES_GCM_decrypt_calc_aad_16_loop: L_AES_GCM_decrypt_aesni_calc_aad_16_loop:
movdqu (%r12,%rcx,1), %xmm8 movdqu (%r12,%rcx,1), %xmm8
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8 pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
pxor %xmm8, %xmm6 pxor %xmm8, %xmm6
@ -2449,22 +2449,22 @@ L_AES_GCM_decrypt_calc_aad_16_loop:
pxor %xmm2, %xmm6 pxor %xmm2, %xmm6
addl $16, %ecx addl $16, %ecx
cmpl %edx, %ecx cmpl %edx, %ecx
jl L_AES_GCM_decrypt_calc_aad_16_loop jl L_AES_GCM_decrypt_aesni_calc_aad_16_loop
movl %r11d, %edx movl %r11d, %edx
cmpl %edx, %ecx cmpl %edx, %ecx
je L_AES_GCM_decrypt_calc_aad_done je L_AES_GCM_decrypt_aesni_calc_aad_done
L_AES_GCM_decrypt_calc_aad_lt16: L_AES_GCM_decrypt_aesni_calc_aad_lt16:
subq $16, %rsp subq $16, %rsp
pxor %xmm8, %xmm8 pxor %xmm8, %xmm8
xorl %ebx, %ebx xorl %ebx, %ebx
movdqu %xmm8, (%rsp) movdqu %xmm8, (%rsp)
L_AES_GCM_decrypt_calc_aad_loop: L_AES_GCM_decrypt_aesni_calc_aad_loop:
movzbl (%r12,%rcx,1), %r13d movzbl (%r12,%rcx,1), %r13d
movb %r13b, (%rsp,%rbx,1) movb %r13b, (%rsp,%rbx,1)
incl %ecx incl %ecx
incl %ebx incl %ebx
cmpl %edx, %ecx cmpl %edx, %ecx
jl L_AES_GCM_decrypt_calc_aad_loop jl L_AES_GCM_decrypt_aesni_calc_aad_loop
movdqu (%rsp), %xmm8 movdqu (%rsp), %xmm8
addq $16, %rsp addq $16, %rsp
pshufb L_aes_gcm_bswap_mask(%rip), %xmm8 pshufb L_aes_gcm_bswap_mask(%rip), %xmm8
@ -2523,7 +2523,7 @@ L_AES_GCM_decrypt_calc_aad_loop:
pxor %xmm1, %xmm2 pxor %xmm1, %xmm2
pxor %xmm7, %xmm2 pxor %xmm7, %xmm2
pxor %xmm2, %xmm6 pxor %xmm2, %xmm6
L_AES_GCM_decrypt_calc_aad_done: L_AES_GCM_decrypt_aesni_calc_aad_done:
# Calculate counter and H # Calculate counter and H
pshufb L_aes_gcm_bswap_epi64(%rip), %xmm4 pshufb L_aes_gcm_bswap_epi64(%rip), %xmm4
movdqa %xmm5, %xmm9 movdqa %xmm5, %xmm9
@ -2541,7 +2541,7 @@ L_AES_GCM_decrypt_calc_aad_done:
xorl %ebx, %ebx xorl %ebx, %ebx
cmpl $0x80, %r9d cmpl $0x80, %r9d
movl %r9d, %r13d movl %r9d, %r13d
jl L_AES_GCM_decrypt_done_128 jl L_AES_GCM_decrypt_aesni_done_128
andl $0xffffff80, %r13d andl $0xffffff80, %r13d
movdqa %xmm6, %xmm2 movdqa %xmm6, %xmm2
# H ^ 1 # H ^ 1
@ -2840,7 +2840,7 @@ L_AES_GCM_decrypt_calc_aad_done:
pxor %xmm8, %xmm14 pxor %xmm8, %xmm14
pxor %xmm14, %xmm7 pxor %xmm14, %xmm7
movdqu %xmm7, 112(%rsp) movdqu %xmm7, 112(%rsp)
L_AES_GCM_decrypt_ghash_128: L_AES_GCM_decrypt_aesni_ghash_128:
leaq (%rdi,%rbx,1), %rcx leaq (%rdi,%rbx,1), %rcx
leaq (%rsi,%rbx,1), %rdx leaq (%rsi,%rbx,1), %rdx
movdqu 128(%rsp), %xmm8 movdqu 128(%rsp), %xmm8
@ -3108,7 +3108,7 @@ L_AES_GCM_decrypt_ghash_128:
pxor %xmm3, %xmm2 pxor %xmm3, %xmm2
cmpl $11, %r10d cmpl $11, %r10d
movdqa 160(%r15), %xmm7 movdqa 160(%r15), %xmm7
jl L_AES_GCM_decrypt_aesenc_128_ghash_avx_done jl L_AES_GCM_decrypt_aesni_aesenc_128_ghash_avx_done
aesenc %xmm7, %xmm8 aesenc %xmm7, %xmm8
aesenc %xmm7, %xmm9 aesenc %xmm7, %xmm9
aesenc %xmm7, %xmm10 aesenc %xmm7, %xmm10
@ -3128,7 +3128,7 @@ L_AES_GCM_decrypt_ghash_128:
aesenc %xmm7, %xmm15 aesenc %xmm7, %xmm15
cmpl $13, %r10d cmpl $13, %r10d
movdqa 192(%r15), %xmm7 movdqa 192(%r15), %xmm7
jl L_AES_GCM_decrypt_aesenc_128_ghash_avx_done jl L_AES_GCM_decrypt_aesni_aesenc_128_ghash_avx_done
aesenc %xmm7, %xmm8 aesenc %xmm7, %xmm8
aesenc %xmm7, %xmm9 aesenc %xmm7, %xmm9
aesenc %xmm7, %xmm10 aesenc %xmm7, %xmm10
@ -3147,7 +3147,7 @@ L_AES_GCM_decrypt_ghash_128:
aesenc %xmm7, %xmm14 aesenc %xmm7, %xmm14
aesenc %xmm7, %xmm15 aesenc %xmm7, %xmm15
movdqa 224(%r15), %xmm7 movdqa 224(%r15), %xmm7
L_AES_GCM_decrypt_aesenc_128_ghash_avx_done: L_AES_GCM_decrypt_aesni_aesenc_128_ghash_avx_done:
aesenclast %xmm7, %xmm8 aesenclast %xmm7, %xmm8
aesenclast %xmm7, %xmm9 aesenclast %xmm7, %xmm9
movdqu (%rcx), %xmm0 movdqu (%rcx), %xmm0
@ -3182,18 +3182,18 @@ L_AES_GCM_decrypt_aesenc_128_ghash_avx_done:
movdqu %xmm15, 112(%rdx) movdqu %xmm15, 112(%rdx)
addl $0x80, %ebx addl $0x80, %ebx
cmpl %r13d, %ebx cmpl %r13d, %ebx
jl L_AES_GCM_decrypt_ghash_128 jl L_AES_GCM_decrypt_aesni_ghash_128
movdqa %xmm2, %xmm6 movdqa %xmm2, %xmm6
movdqu (%rsp), %xmm5 movdqu (%rsp), %xmm5
L_AES_GCM_decrypt_done_128: L_AES_GCM_decrypt_aesni_done_128:
movl %r9d, %edx movl %r9d, %edx
cmpl %edx, %ebx cmpl %edx, %ebx
jge L_AES_GCM_decrypt_done_dec jge L_AES_GCM_decrypt_aesni_done_dec
movl %r9d, %r13d movl %r9d, %r13d
andl $0xfffffff0, %r13d andl $0xfffffff0, %r13d
cmpl %r13d, %ebx cmpl %r13d, %ebx
jge L_AES_GCM_decrypt_last_block_done jge L_AES_GCM_decrypt_aesni_last_block_done
L_AES_GCM_decrypt_last_block_start: L_AES_GCM_decrypt_aesni_last_block_start:
leaq (%rdi,%rbx,1), %rcx leaq (%rdi,%rbx,1), %rcx
leaq (%rsi,%rbx,1), %rdx leaq (%rsi,%rbx,1), %rdx
movdqu (%rcx), %xmm1 movdqu (%rcx), %xmm1
@ -3242,28 +3242,28 @@ L_AES_GCM_decrypt_last_block_start:
pxor %xmm3, %xmm6 pxor %xmm3, %xmm6
cmpl $11, %r10d cmpl $11, %r10d
movdqa 160(%r15), %xmm9 movdqa 160(%r15), %xmm9
jl L_AES_GCM_decrypt_aesenc_gfmul_last jl L_AES_GCM_decrypt_aesni_aesenc_gfmul_last
aesenc %xmm9, %xmm8 aesenc %xmm9, %xmm8
aesenc 176(%r15), %xmm8 aesenc 176(%r15), %xmm8
cmpl $13, %r10d cmpl $13, %r10d
movdqa 192(%r15), %xmm9 movdqa 192(%r15), %xmm9
jl L_AES_GCM_decrypt_aesenc_gfmul_last jl L_AES_GCM_decrypt_aesni_aesenc_gfmul_last
aesenc %xmm9, %xmm8 aesenc %xmm9, %xmm8
aesenc 208(%r15), %xmm8 aesenc 208(%r15), %xmm8
movdqa 224(%r15), %xmm9 movdqa 224(%r15), %xmm9
L_AES_GCM_decrypt_aesenc_gfmul_last: L_AES_GCM_decrypt_aesni_aesenc_gfmul_last:
aesenclast %xmm9, %xmm8 aesenclast %xmm9, %xmm8
movdqu (%rcx), %xmm9 movdqu (%rcx), %xmm9
pxor %xmm9, %xmm8 pxor %xmm9, %xmm8
movdqu %xmm8, (%rdx) movdqu %xmm8, (%rdx)
addl $16, %ebx addl $16, %ebx
cmpl %r13d, %ebx cmpl %r13d, %ebx
jl L_AES_GCM_decrypt_last_block_start jl L_AES_GCM_decrypt_aesni_last_block_start
L_AES_GCM_decrypt_last_block_done: L_AES_GCM_decrypt_aesni_last_block_done:
movl %r9d, %ecx movl %r9d, %ecx
movl %ecx, %edx movl %ecx, %edx
andl $15, %ecx andl $15, %ecx
jz L_AES_GCM_decrypt_aesenc_last15_dec_avx_done jz L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_done
movdqu 128(%rsp), %xmm4 movdqu 128(%rsp), %xmm4
pshufb L_aes_gcm_bswap_epi64(%rip), %xmm4 pshufb L_aes_gcm_bswap_epi64(%rip), %xmm4
pxor (%r15), %xmm4 pxor (%r15), %xmm4
@ -3278,23 +3278,23 @@ L_AES_GCM_decrypt_last_block_done:
aesenc 144(%r15), %xmm4 aesenc 144(%r15), %xmm4
cmpl $11, %r10d cmpl $11, %r10d
movdqa 160(%r15), %xmm9 movdqa 160(%r15), %xmm9
jl L_AES_GCM_decrypt_aesenc_last15_dec_avx_aesenc_avx_last jl L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_aesenc_avx_last
aesenc %xmm9, %xmm4 aesenc %xmm9, %xmm4
aesenc 176(%r15), %xmm4 aesenc 176(%r15), %xmm4
cmpl $13, %r10d cmpl $13, %r10d
movdqa 192(%r15), %xmm9 movdqa 192(%r15), %xmm9
jl L_AES_GCM_decrypt_aesenc_last15_dec_avx_aesenc_avx_last jl L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_aesenc_avx_last
aesenc %xmm9, %xmm4 aesenc %xmm9, %xmm4
aesenc 208(%r15), %xmm4 aesenc 208(%r15), %xmm4
movdqa 224(%r15), %xmm9 movdqa 224(%r15), %xmm9
L_AES_GCM_decrypt_aesenc_last15_dec_avx_aesenc_avx_last: L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_aesenc_avx_last:
aesenclast %xmm9, %xmm4 aesenclast %xmm9, %xmm4
subq $32, %rsp subq $32, %rsp
xorl %ecx, %ecx xorl %ecx, %ecx
movdqu %xmm4, (%rsp) movdqu %xmm4, (%rsp)
pxor %xmm0, %xmm0 pxor %xmm0, %xmm0
movdqu %xmm0, 16(%rsp) movdqu %xmm0, 16(%rsp)
L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop: L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_loop:
movzbl (%rdi,%rbx,1), %r13d movzbl (%rdi,%rbx,1), %r13d
movb %r13b, 16(%rsp,%rcx,1) movb %r13b, 16(%rsp,%rcx,1)
xorb (%rsp,%rcx,1), %r13b xorb (%rsp,%rcx,1), %r13b
@ -3302,7 +3302,7 @@ L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop:
incl %ebx incl %ebx
incl %ecx incl %ecx
cmpl %edx, %ebx cmpl %edx, %ebx
jl L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop jl L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_loop
movdqu 16(%rsp), %xmm4 movdqu 16(%rsp), %xmm4
addq $32, %rsp addq $32, %rsp
pshufb L_aes_gcm_bswap_mask(%rip), %xmm4 pshufb L_aes_gcm_bswap_mask(%rip), %xmm4
@ -3347,8 +3347,8 @@ L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop:
pxor %xmm13, %xmm14 pxor %xmm13, %xmm14
pxor %xmm8, %xmm14 pxor %xmm8, %xmm14
pxor %xmm14, %xmm6 pxor %xmm14, %xmm6
L_AES_GCM_decrypt_aesenc_last15_dec_avx_done: L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_done:
L_AES_GCM_decrypt_done_dec: L_AES_GCM_decrypt_aesni_done_dec:
movl %r9d, %edx movl %r9d, %edx
movl %r11d, %ecx movl %r11d, %ecx
shlq $3, %rdx shlq $3, %rdx
@ -3400,24 +3400,24 @@ L_AES_GCM_decrypt_done_dec:
movdqu 144(%rsp), %xmm0 movdqu 144(%rsp), %xmm0
pxor %xmm6, %xmm0 pxor %xmm6, %xmm0
cmpl $16, %r14d cmpl $16, %r14d
je L_AES_GCM_decrypt_cmp_tag_16 je L_AES_GCM_decrypt_aesni_cmp_tag_16
subq $16, %rsp subq $16, %rsp
xorq %rcx, %rcx xorq %rcx, %rcx
xorq %rbx, %rbx xorq %rbx, %rbx
movdqu %xmm0, (%rsp) movdqu %xmm0, (%rsp)
L_AES_GCM_decrypt_cmp_tag_loop: L_AES_GCM_decrypt_aesni_cmp_tag_loop:
movzbl (%rsp,%rcx,1), %r13d movzbl (%rsp,%rcx,1), %r13d
xorb (%r8,%rcx,1), %r13b xorb (%r8,%rcx,1), %r13b
orb %r13b, %bl orb %r13b, %bl
incl %ecx incl %ecx
cmpl %r14d, %ecx cmpl %r14d, %ecx
jne L_AES_GCM_decrypt_cmp_tag_loop jne L_AES_GCM_decrypt_aesni_cmp_tag_loop
cmpb $0x00, %bl cmpb $0x00, %bl
sete %bl sete %bl
addq $16, %rsp addq $16, %rsp
xorq %rcx, %rcx xorq %rcx, %rcx
jmp L_AES_GCM_decrypt_cmp_tag_done jmp L_AES_GCM_decrypt_aesni_cmp_tag_done
L_AES_GCM_decrypt_cmp_tag_16: L_AES_GCM_decrypt_aesni_cmp_tag_16:
movdqu (%r8), %xmm1 movdqu (%r8), %xmm1
pcmpeqb %xmm1, %xmm0 pcmpeqb %xmm1, %xmm0
pmovmskb %xmm0, %rdx pmovmskb %xmm0, %rdx
@ -3425,7 +3425,7 @@ L_AES_GCM_decrypt_cmp_tag_16:
xorl %ebx, %ebx xorl %ebx, %ebx
cmpl $0xffff, %edx cmpl $0xffff, %edx
sete %bl sete %bl
L_AES_GCM_decrypt_cmp_tag_done: L_AES_GCM_decrypt_aesni_cmp_tag_done:
movl %ebx, (%rbp) movl %ebx, (%rbp)
addq $0xa8, %rsp addq $0xa8, %rsp
popq %rbp popq %rbp
@ -3436,7 +3436,7 @@ L_AES_GCM_decrypt_cmp_tag_done:
popq %r13 popq %r13
repz retq repz retq
#ifndef __APPLE__ #ifndef __APPLE__
.size AES_GCM_decrypt,.-AES_GCM_decrypt .size AES_GCM_decrypt_aesni,.-AES_GCM_decrypt_aesni
#endif /* __APPLE__ */ #endif /* __APPLE__ */
#ifdef WOLFSSL_AESGCM_STREAM #ifdef WOLFSSL_AESGCM_STREAM
#ifndef __APPLE__ #ifndef __APPLE__

276
wolfcrypt/src/aes_gcm_asm.asm
View File

@ -96,7 +96,7 @@ L_aes_gcm_mod2_128 QWORD 1, 13979173243358019584
ptr_L_aes_gcm_mod2_128 QWORD L_aes_gcm_mod2_128 ptr_L_aes_gcm_mod2_128 QWORD L_aes_gcm_mod2_128
_DATA ENDS _DATA ENDS
_text SEGMENT READONLY PARA _text SEGMENT READONLY PARA
AES_GCM_encrypt PROC AES_GCM_encrypt_aesni PROC
push r13 push r13
push rdi push rdi
push rsi push rsi
@ -130,7 +130,7 @@ AES_GCM_encrypt PROC
pxor xmm6, xmm6 pxor xmm6, xmm6
cmp ebx, 12 cmp ebx, 12
mov edx, ebx mov edx, ebx
jne L_AES_GCM_encrypt_iv_not_12 jne L_AES_GCM_encrypt_aesni_iv_not_12
; # Calculate values when IV is 12 bytes ; # Calculate values when IV is 12 bytes
; Set counter based on IV ; Set counter based on IV
mov ecx, 16777216 mov ecx, 16777216
@ -170,7 +170,7 @@ AES_GCM_encrypt PROC
aesenc xmm1, xmm7 aesenc xmm1, xmm7
cmp r10d, 11 cmp r10d, 11
movdqa xmm7, OWORD PTR [r15+160] movdqa xmm7, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_calc_iv_12_last jl L_AES_GCM_encrypt_aesni_calc_iv_12_last
aesenc xmm5, xmm7 aesenc xmm5, xmm7
aesenc xmm1, xmm7 aesenc xmm1, xmm7
movdqa xmm7, OWORD PTR [r15+176] movdqa xmm7, OWORD PTR [r15+176]
@ -178,20 +178,20 @@ AES_GCM_encrypt PROC
aesenc xmm1, xmm7 aesenc xmm1, xmm7
cmp r10d, 13 cmp r10d, 13
movdqa xmm7, OWORD PTR [r15+192] movdqa xmm7, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_calc_iv_12_last jl L_AES_GCM_encrypt_aesni_calc_iv_12_last
aesenc xmm5, xmm7 aesenc xmm5, xmm7
aesenc xmm1, xmm7 aesenc xmm1, xmm7
movdqa xmm7, OWORD PTR [r15+208] movdqa xmm7, OWORD PTR [r15+208]
aesenc xmm5, xmm7 aesenc xmm5, xmm7
aesenc xmm1, xmm7 aesenc xmm1, xmm7
movdqa xmm7, OWORD PTR [r15+224] movdqa xmm7, OWORD PTR [r15+224]
L_AES_GCM_encrypt_calc_iv_12_last: L_AES_GCM_encrypt_aesni_calc_iv_12_last:
aesenclast xmm5, xmm7 aesenclast xmm5, xmm7
aesenclast xmm1, xmm7 aesenclast xmm1, xmm7
pshufb xmm5, OWORD PTR L_aes_gcm_bswap_mask pshufb xmm5, OWORD PTR L_aes_gcm_bswap_mask
movdqu [rsp+144], xmm1 movdqu [rsp+144], xmm1
jmp L_AES_GCM_encrypt_iv_done jmp L_AES_GCM_encrypt_aesni_iv_done
L_AES_GCM_encrypt_iv_not_12: L_AES_GCM_encrypt_aesni_iv_not_12:
; Calculate values when IV is not 12 bytes ; Calculate values when IV is not 12 bytes
; H = Encrypt X(=0) ; H = Encrypt X(=0)
movdqa xmm5, OWORD PTR [r15] movdqa xmm5, OWORD PTR [r15]
@ -206,27 +206,27 @@ L_AES_GCM_encrypt_iv_not_12:
aesenc xmm5, [r15+144] aesenc xmm5, [r15+144]
cmp r10d, 11 cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160] movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_calc_iv_1_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc xmm5, xmm9 aesenc xmm5, xmm9
aesenc xmm5, [r15+176] aesenc xmm5, [r15+176]
cmp r10d, 13 cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192] movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_calc_iv_1_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc xmm5, xmm9 aesenc xmm5, xmm9
aesenc xmm5, [r15+208] aesenc xmm5, [r15+208]
movdqa xmm9, OWORD PTR [r15+224] movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_encrypt_calc_iv_1_aesenc_avx_last: L_AES_GCM_encrypt_aesni_calc_iv_1_aesenc_avx_last:
aesenclast xmm5, xmm9 aesenclast xmm5, xmm9
pshufb xmm5, OWORD PTR L_aes_gcm_bswap_mask pshufb xmm5, OWORD PTR L_aes_gcm_bswap_mask
; Calc counter ; Calc counter
; Initialization vector ; Initialization vector
cmp edx, 0 cmp edx, 0
mov rcx, 0 mov rcx, 0
je L_AES_GCM_encrypt_calc_iv_done je L_AES_GCM_encrypt_aesni_calc_iv_done
cmp edx, 16 cmp edx, 16
jl L_AES_GCM_encrypt_calc_iv_lt16 jl L_AES_GCM_encrypt_aesni_calc_iv_lt16
and edx, 4294967280 and edx, 4294967280
L_AES_GCM_encrypt_calc_iv_16_loop: L_AES_GCM_encrypt_aesni_calc_iv_16_loop:
movdqu xmm8, [rax+rcx] movdqu xmm8, [rax+rcx]
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
pxor xmm4, xmm8 pxor xmm4, xmm8
@ -286,22 +286,22 @@ L_AES_GCM_encrypt_calc_iv_16_loop:
pxor xmm4, xmm2 pxor xmm4, xmm2
add ecx, 16 add ecx, 16
cmp ecx, edx cmp ecx, edx
jl L_AES_GCM_encrypt_calc_iv_16_loop jl L_AES_GCM_encrypt_aesni_calc_iv_16_loop
mov edx, ebx mov edx, ebx
cmp ecx, edx cmp ecx, edx
je L_AES_GCM_encrypt_calc_iv_done je L_AES_GCM_encrypt_aesni_calc_iv_done
L_AES_GCM_encrypt_calc_iv_lt16: L_AES_GCM_encrypt_aesni_calc_iv_lt16:
sub rsp, 16 sub rsp, 16
pxor xmm8, xmm8 pxor xmm8, xmm8
xor ebx, ebx xor ebx, ebx
movdqu [rsp], xmm8 movdqu [rsp], xmm8
L_AES_GCM_encrypt_calc_iv_loop: L_AES_GCM_encrypt_aesni_calc_iv_loop:
movzx r13d, BYTE PTR [rax+rcx] movzx r13d, BYTE PTR [rax+rcx]
mov BYTE PTR [rsp+rbx], r13b mov BYTE PTR [rsp+rbx], r13b
inc ecx inc ecx
inc ebx inc ebx
cmp ecx, edx cmp ecx, edx
jl L_AES_GCM_encrypt_calc_iv_loop jl L_AES_GCM_encrypt_aesni_calc_iv_loop
movdqu xmm8, [rsp] movdqu xmm8, [rsp]
add rsp, 16 add rsp, 16
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
@ -360,7 +360,7 @@ L_AES_GCM_encrypt_calc_iv_loop:
pxor xmm2, xmm1 pxor xmm2, xmm1
pxor xmm2, xmm7 pxor xmm2, xmm7
pxor xmm4, xmm2 pxor xmm4, xmm2
L_AES_GCM_encrypt_calc_iv_done: L_AES_GCM_encrypt_aesni_calc_iv_done:
; T = Encrypt counter ; T = Encrypt counter
pxor xmm0, xmm0 pxor xmm0, xmm0
shl edx, 3 shl edx, 3
@ -435,28 +435,28 @@ L_AES_GCM_encrypt_calc_iv_done:
aesenc xmm8, [r15+144] aesenc xmm8, [r15+144]
cmp r10d, 11 cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160] movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_calc_iv_2_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc xmm8, xmm9 aesenc xmm8, xmm9
aesenc xmm8, [r15+176] aesenc xmm8, [r15+176]
cmp r10d, 13 cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192] movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_calc_iv_2_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc xmm8, xmm9 aesenc xmm8, xmm9
aesenc xmm8, [r15+208] aesenc xmm8, [r15+208]
movdqa xmm9, OWORD PTR [r15+224] movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_encrypt_calc_iv_2_aesenc_avx_last: L_AES_GCM_encrypt_aesni_calc_iv_2_aesenc_avx_last:
aesenclast xmm8, xmm9 aesenclast xmm8, xmm9
movdqu [rsp+144], xmm8 movdqu [rsp+144], xmm8
L_AES_GCM_encrypt_iv_done: L_AES_GCM_encrypt_aesni_iv_done:
; Additional authentication data ; Additional authentication data
mov edx, r11d mov edx, r11d
cmp edx, 0 cmp edx, 0
je L_AES_GCM_encrypt_calc_aad_done je L_AES_GCM_encrypt_aesni_calc_aad_done
xor ecx, ecx xor ecx, ecx
cmp edx, 16 cmp edx, 16
jl L_AES_GCM_encrypt_calc_aad_lt16 jl L_AES_GCM_encrypt_aesni_calc_aad_lt16
and edx, 4294967280 and edx, 4294967280
L_AES_GCM_encrypt_calc_aad_16_loop: L_AES_GCM_encrypt_aesni_calc_aad_16_loop:
movdqu xmm8, [r12+rcx] movdqu xmm8, [r12+rcx]
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
pxor xmm6, xmm8 pxor xmm6, xmm8
@ -516,22 +516,22 @@ L_AES_GCM_encrypt_calc_aad_16_loop:
pxor xmm6, xmm2 pxor xmm6, xmm2
add ecx, 16 add ecx, 16
cmp ecx, edx cmp ecx, edx
jl L_AES_GCM_encrypt_calc_aad_16_loop jl L_AES_GCM_encrypt_aesni_calc_aad_16_loop
mov edx, r11d mov edx, r11d
cmp ecx, edx cmp ecx, edx
je L_AES_GCM_encrypt_calc_aad_done je L_AES_GCM_encrypt_aesni_calc_aad_done
L_AES_GCM_encrypt_calc_aad_lt16: L_AES_GCM_encrypt_aesni_calc_aad_lt16:
sub rsp, 16 sub rsp, 16
pxor xmm8, xmm8 pxor xmm8, xmm8
xor ebx, ebx xor ebx, ebx
movdqu [rsp], xmm8 movdqu [rsp], xmm8
L_AES_GCM_encrypt_calc_aad_loop: L_AES_GCM_encrypt_aesni_calc_aad_loop:
movzx r13d, BYTE PTR [r12+rcx] movzx r13d, BYTE PTR [r12+rcx]
mov BYTE PTR [rsp+rbx], r13b mov BYTE PTR [rsp+rbx], r13b
inc ecx inc ecx
inc ebx inc ebx
cmp ecx, edx cmp ecx, edx
jl L_AES_GCM_encrypt_calc_aad_loop jl L_AES_GCM_encrypt_aesni_calc_aad_loop
movdqu xmm8, [rsp] movdqu xmm8, [rsp]
add rsp, 16 add rsp, 16
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
@ -590,7 +590,7 @@ L_AES_GCM_encrypt_calc_aad_loop:
pxor xmm2, xmm1 pxor xmm2, xmm1
pxor xmm2, xmm7 pxor xmm2, xmm7
pxor xmm6, xmm2 pxor xmm6, xmm2
L_AES_GCM_encrypt_calc_aad_done: L_AES_GCM_encrypt_aesni_calc_aad_done:
; Calculate counter and H ; Calculate counter and H
pshufb xmm4, OWORD PTR L_aes_gcm_bswap_epi64 pshufb xmm4, OWORD PTR L_aes_gcm_bswap_epi64
movdqa xmm9, xmm5 movdqa xmm9, xmm5
@ -608,7 +608,7 @@ L_AES_GCM_encrypt_calc_aad_done:
xor rbx, rbx xor rbx, rbx
cmp r9d, 128 cmp r9d, 128
mov r13d, r9d mov r13d, r9d
jl L_AES_GCM_encrypt_done_128 jl L_AES_GCM_encrypt_aesni_done_128
and r13d, 4294967168 and r13d, 4294967168
movdqa xmm2, xmm6 movdqa xmm2, xmm6
; H ^ 1 ; H ^ 1
@ -1027,7 +1027,7 @@ L_AES_GCM_encrypt_calc_aad_done:
aesenc xmm15, xmm7 aesenc xmm15, xmm7
cmp r10d, 11 cmp r10d, 11
movdqa xmm7, OWORD PTR [r15+160] movdqa xmm7, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_enc_done jl L_AES_GCM_encrypt_aesni_enc_done
aesenc xmm8, xmm7 aesenc xmm8, xmm7
aesenc xmm9, xmm7 aesenc xmm9, xmm7
aesenc xmm10, xmm7 aesenc xmm10, xmm7
@ -1047,7 +1047,7 @@ L_AES_GCM_encrypt_calc_aad_done:
aesenc xmm15, xmm7 aesenc xmm15, xmm7
cmp r10d, 13 cmp r10d, 13
movdqa xmm7, OWORD PTR [r15+192] movdqa xmm7, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_enc_done jl L_AES_GCM_encrypt_aesni_enc_done
aesenc xmm8, xmm7 aesenc xmm8, xmm7
aesenc xmm9, xmm7 aesenc xmm9, xmm7
aesenc xmm10, xmm7 aesenc xmm10, xmm7
@ -1066,7 +1066,7 @@ L_AES_GCM_encrypt_calc_aad_done:
aesenc xmm14, xmm7 aesenc xmm14, xmm7
aesenc xmm15, xmm7 aesenc xmm15, xmm7
movdqa xmm7, OWORD PTR [r15+224] movdqa xmm7, OWORD PTR [r15+224]
L_AES_GCM_encrypt_enc_done: L_AES_GCM_encrypt_aesni_enc_done:
aesenclast xmm8, xmm7 aesenclast xmm8, xmm7
aesenclast xmm9, xmm7 aesenclast xmm9, xmm7
movdqu xmm0, [rdi] movdqu xmm0, [rdi]
@ -1101,9 +1101,9 @@ L_AES_GCM_encrypt_enc_done:
movdqu [rsi+112], xmm15 movdqu [rsi+112], xmm15
cmp r13d, 128 cmp r13d, 128
mov ebx, 128 mov ebx, 128
jle L_AES_GCM_encrypt_end_128 jle L_AES_GCM_encrypt_aesni_end_128
; More 128 bytes of input ; More 128 bytes of input
L_AES_GCM_encrypt_ghash_128: L_AES_GCM_encrypt_aesni_ghash_128:
lea rcx, QWORD PTR [rdi+rbx] lea rcx, QWORD PTR [rdi+rbx]
lea rdx, QWORD PTR [rsi+rbx] lea rdx, QWORD PTR [rsi+rbx]
movdqu xmm8, [rsp+128] movdqu xmm8, [rsp+128]
@ -1371,7 +1371,7 @@ L_AES_GCM_encrypt_ghash_128:
pxor xmm2, xmm3 pxor xmm2, xmm3
cmp r10d, 11 cmp r10d, 11
movdqa xmm7, OWORD PTR [r15+160] movdqa xmm7, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_aesenc_128_ghash_avx_done jl L_AES_GCM_encrypt_aesni_aesenc_128_ghash_avx_done
aesenc xmm8, xmm7 aesenc xmm8, xmm7
aesenc xmm9, xmm7 aesenc xmm9, xmm7
aesenc xmm10, xmm7 aesenc xmm10, xmm7
@ -1391,7 +1391,7 @@ L_AES_GCM_encrypt_ghash_128:
aesenc xmm15, xmm7 aesenc xmm15, xmm7
cmp r10d, 13 cmp r10d, 13
movdqa xmm7, OWORD PTR [r15+192] movdqa xmm7, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_aesenc_128_ghash_avx_done jl L_AES_GCM_encrypt_aesni_aesenc_128_ghash_avx_done
aesenc xmm8, xmm7 aesenc xmm8, xmm7
aesenc xmm9, xmm7 aesenc xmm9, xmm7
aesenc xmm10, xmm7 aesenc xmm10, xmm7
@ -1410,7 +1410,7 @@ L_AES_GCM_encrypt_ghash_128:
aesenc xmm14, xmm7 aesenc xmm14, xmm7
aesenc xmm15, xmm7 aesenc xmm15, xmm7
movdqa xmm7, OWORD PTR [r15+224] movdqa xmm7, OWORD PTR [r15+224]
L_AES_GCM_encrypt_aesenc_128_ghash_avx_done: L_AES_GCM_encrypt_aesni_aesenc_128_ghash_avx_done:
aesenclast xmm8, xmm7 aesenclast xmm8, xmm7
aesenclast xmm9, xmm7 aesenclast xmm9, xmm7
movdqu xmm0, [rcx] movdqu xmm0, [rcx]
@ -1445,8 +1445,8 @@ L_AES_GCM_encrypt_aesenc_128_ghash_avx_done:
movdqu [rdx+112], xmm15 movdqu [rdx+112], xmm15
add ebx, 128 add ebx, 128
cmp ebx, r13d cmp ebx, r13d
jl L_AES_GCM_encrypt_ghash_128 jl L_AES_GCM_encrypt_aesni_ghash_128
L_AES_GCM_encrypt_end_128: L_AES_GCM_encrypt_aesni_end_128:
movdqa xmm4, OWORD PTR L_aes_gcm_bswap_mask movdqa xmm4, OWORD PTR L_aes_gcm_bswap_mask
pshufb xmm8, xmm4 pshufb xmm8, xmm4
pshufb xmm9, xmm4 pshufb xmm9, xmm4
@ -1633,14 +1633,14 @@ L_AES_GCM_encrypt_end_128:
pxor xmm2, xmm4 pxor xmm2, xmm4
pxor xmm6, xmm2 pxor xmm6, xmm2
movdqu xmm5, [rsp] movdqu xmm5, [rsp]
L_AES_GCM_encrypt_done_128: L_AES_GCM_encrypt_aesni_done_128:
mov edx, r9d mov edx, r9d
cmp ebx, edx cmp ebx, edx
jge L_AES_GCM_encrypt_done_enc jge L_AES_GCM_encrypt_aesni_done_enc
mov r13d, r9d mov r13d, r9d
and r13d, 4294967280 and r13d, 4294967280
cmp ebx, r13d cmp ebx, r13d
jge L_AES_GCM_encrypt_last_block_done jge L_AES_GCM_encrypt_aesni_last_block_done
lea rcx, QWORD PTR [rdi+rbx] lea rcx, QWORD PTR [rdi+rbx]
lea rdx, QWORD PTR [rsi+rbx] lea rdx, QWORD PTR [rsi+rbx]
movdqu xmm8, [rsp+128] movdqu xmm8, [rsp+128]
@ -1660,16 +1660,16 @@ L_AES_GCM_encrypt_done_128:
aesenc xmm8, [r15+144] aesenc xmm8, [r15+144]
cmp r10d, 11 cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160] movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_aesenc_block_aesenc_avx_last
aesenc xmm8, xmm9 aesenc xmm8, xmm9
aesenc xmm8, [r15+176] aesenc xmm8, [r15+176]
cmp r10d, 13 cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192] movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_aesenc_block_aesenc_avx_last
aesenc xmm8, xmm9 aesenc xmm8, xmm9
aesenc xmm8, [r15+208] aesenc xmm8, [r15+208]
movdqa xmm9, OWORD PTR [r15+224] movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last: L_AES_GCM_encrypt_aesni_aesenc_block_aesenc_avx_last:
aesenclast xmm8, xmm9 aesenclast xmm8, xmm9
movdqu xmm9, [rcx] movdqu xmm9, [rcx]
pxor xmm8, xmm9 pxor xmm8, xmm9
@ -1678,8 +1678,8 @@ L_AES_GCM_encrypt_aesenc_block_aesenc_avx_last:
pxor xmm6, xmm8 pxor xmm6, xmm8
add ebx, 16 add ebx, 16
cmp ebx, r13d cmp ebx, r13d
jge L_AES_GCM_encrypt_last_block_ghash jge L_AES_GCM_encrypt_aesni_last_block_ghash
L_AES_GCM_encrypt_last_block_start: L_AES_GCM_encrypt_aesni_last_block_start:
lea rcx, QWORD PTR [rdi+rbx] lea rcx, QWORD PTR [rdi+rbx]
lea rdx, QWORD PTR [rsi+rbx] lea rdx, QWORD PTR [rsi+rbx]
movdqu xmm8, [rsp+128] movdqu xmm8, [rsp+128]
@ -1724,16 +1724,16 @@ L_AES_GCM_encrypt_last_block_start:
pxor xmm6, xmm3 pxor xmm6, xmm3
cmp r10d, 11 cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160] movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_aesenc_gfmul_last jl L_AES_GCM_encrypt_aesni_aesenc_gfmul_last
aesenc xmm8, xmm9 aesenc xmm8, xmm9
aesenc xmm8, [r15+176] aesenc xmm8, [r15+176]
cmp r10d, 13 cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192] movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_aesenc_gfmul_last jl L_AES_GCM_encrypt_aesni_aesenc_gfmul_last
aesenc xmm8, xmm9 aesenc xmm8, xmm9
aesenc xmm8, [r15+208] aesenc xmm8, [r15+208]
movdqa xmm9, OWORD PTR [r15+224] movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_encrypt_aesenc_gfmul_last: L_AES_GCM_encrypt_aesni_aesenc_gfmul_last:
aesenclast xmm8, xmm9 aesenclast xmm8, xmm9
movdqu xmm9, [rcx] movdqu xmm9, [rcx]
pxor xmm8, xmm9 pxor xmm8, xmm9
@ -1742,8 +1742,8 @@ L_AES_GCM_encrypt_aesenc_gfmul_last:
pxor xmm6, xmm8 pxor xmm6, xmm8
add ebx, 16 add ebx, 16
cmp ebx, r13d cmp ebx, r13d
jl L_AES_GCM_encrypt_last_block_start jl L_AES_GCM_encrypt_aesni_last_block_start
L_AES_GCM_encrypt_last_block_ghash: L_AES_GCM_encrypt_aesni_last_block_ghash:
pshufd xmm9, xmm5, 78 pshufd xmm9, xmm5, 78
pshufd xmm10, xmm6, 78 pshufd xmm10, xmm6, 78
movdqa xmm11, xmm6 movdqa xmm11, xmm6
@ -1784,11 +1784,11 @@ L_AES_GCM_encrypt_last_block_ghash:
pxor xmm14, xmm13 pxor xmm14, xmm13
pxor xmm14, xmm8 pxor xmm14, xmm8
pxor xmm6, xmm14 pxor xmm6, xmm14
L_AES_GCM_encrypt_last_block_done: L_AES_GCM_encrypt_aesni_last_block_done:
mov ecx, r9d mov ecx, r9d
mov edx, ecx mov edx, ecx
and ecx, 15 and ecx, 15
jz L_AES_GCM_encrypt_aesenc_last15_enc_avx_done jz L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_done
movdqu xmm4, [rsp+128] movdqu xmm4, [rsp+128]
pshufb xmm4, OWORD PTR L_aes_gcm_bswap_epi64 pshufb xmm4, OWORD PTR L_aes_gcm_bswap_epi64
pxor xmm4, [r15] pxor xmm4, [r15]
@ -1803,21 +1803,21 @@ L_AES_GCM_encrypt_last_block_done:
aesenc xmm4, [r15+144] aesenc xmm4, [r15+144]
cmp r10d, 11 cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160] movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_aesenc_avx_last
aesenc xmm4, xmm9 aesenc xmm4, xmm9
aesenc xmm4, [r15+176] aesenc xmm4, [r15+176]
cmp r10d, 13 cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192] movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_aesenc_avx_last jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_aesenc_avx_last
aesenc xmm4, xmm9 aesenc xmm4, xmm9
aesenc xmm4, [r15+208] aesenc xmm4, [r15+208]
movdqa xmm9, OWORD PTR [r15+224] movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_encrypt_aesenc_last15_enc_avx_aesenc_avx_last: L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_aesenc_avx_last:
aesenclast xmm4, xmm9 aesenclast xmm4, xmm9
sub rsp, 16 sub rsp, 16
xor ecx, ecx xor ecx, ecx
movdqu [rsp], xmm4 movdqu [rsp], xmm4
L_AES_GCM_encrypt_aesenc_last15_enc_avx_loop: L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_loop:
movzx r13d, BYTE PTR [rdi+rbx] movzx r13d, BYTE PTR [rdi+rbx]
xor r13b, BYTE PTR [rsp+rcx] xor r13b, BYTE PTR [rsp+rcx]
mov BYTE PTR [rsi+rbx], r13b mov BYTE PTR [rsi+rbx], r13b
@ -1825,16 +1825,16 @@ L_AES_GCM_encrypt_aesenc_last15_enc_avx_loop:
inc ebx inc ebx
inc ecx inc ecx
cmp ebx, edx cmp ebx, edx
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_loop jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_loop
xor r13, r13 xor r13, r13
cmp ecx, 16 cmp ecx, 16
je L_AES_GCM_encrypt_aesenc_last15_enc_avx_finish_enc je L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_finish_enc
L_AES_GCM_encrypt_aesenc_last15_enc_avx_byte_loop: L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_byte_loop:
mov BYTE PTR [rsp+rcx], r13b mov BYTE PTR [rsp+rcx], r13b
inc ecx inc ecx
cmp ecx, 16 cmp ecx, 16
jl L_AES_GCM_encrypt_aesenc_last15_enc_avx_byte_loop jl L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_byte_loop
L_AES_GCM_encrypt_aesenc_last15_enc_avx_finish_enc: L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_finish_enc:
movdqu xmm4, [rsp] movdqu xmm4, [rsp]
add rsp, 16 add rsp, 16
pshufb xmm4, OWORD PTR L_aes_gcm_bswap_mask pshufb xmm4, OWORD PTR L_aes_gcm_bswap_mask
@ -1879,8 +1879,8 @@ L_AES_GCM_encrypt_aesenc_last15_enc_avx_finish_enc:
pxor xmm14, xmm13 pxor xmm14, xmm13
pxor xmm14, xmm8 pxor xmm14, xmm8
pxor xmm6, xmm14 pxor xmm6, xmm14
L_AES_GCM_encrypt_aesenc_last15_enc_avx_done: L_AES_GCM_encrypt_aesni_aesenc_last15_enc_avx_done:
L_AES_GCM_encrypt_done_enc: L_AES_GCM_encrypt_aesni_done_enc:
mov edx, r9d mov edx, r9d
mov ecx, r11d mov ecx, r11d
shl rdx, 3 shl rdx, 3
@ -1932,19 +1932,19 @@ L_AES_GCM_encrypt_done_enc:
movdqu xmm0, [rsp+144] movdqu xmm0, [rsp+144]
pxor xmm0, xmm6 pxor xmm0, xmm6
cmp r14d, 16 cmp r14d, 16
je L_AES_GCM_encrypt_store_tag_16 je L_AES_GCM_encrypt_aesni_store_tag_16
xor rcx, rcx xor rcx, rcx
movdqu [rsp], xmm0 movdqu [rsp], xmm0
L_AES_GCM_encrypt_store_tag_loop: L_AES_GCM_encrypt_aesni_store_tag_loop:
movzx r13d, BYTE PTR [rsp+rcx] movzx r13d, BYTE PTR [rsp+rcx]
mov BYTE PTR [r8+rcx], r13b mov BYTE PTR [r8+rcx], r13b
inc ecx inc ecx
cmp ecx, r14d cmp ecx, r14d
jne L_AES_GCM_encrypt_store_tag_loop jne L_AES_GCM_encrypt_aesni_store_tag_loop
jmp L_AES_GCM_encrypt_store_tag_done jmp L_AES_GCM_encrypt_aesni_store_tag_done
L_AES_GCM_encrypt_store_tag_16: L_AES_GCM_encrypt_aesni_store_tag_16:
movdqu [r8], xmm0 movdqu [r8], xmm0
L_AES_GCM_encrypt_store_tag_done: L_AES_GCM_encrypt_aesni_store_tag_done:
movdqu xmm6, [rsp+160] movdqu xmm6, [rsp+160]
movdqu xmm7, [rsp+176] movdqu xmm7, [rsp+176]
movdqu xmm8, [rsp+192] movdqu xmm8, [rsp+192]
@ -1964,10 +1964,10 @@ L_AES_GCM_encrypt_store_tag_done:
pop rdi pop rdi
pop r13 pop r13
ret ret
AES_GCM_encrypt ENDP AES_GCM_encrypt_aesni ENDP
_text ENDS _text ENDS
_text SEGMENT READONLY PARA _text SEGMENT READONLY PARA
AES_GCM_decrypt PROC AES_GCM_decrypt_aesni PROC
push r13 push r13
push rdi push rdi
push rsi push rsi
@ -2003,7 +2003,7 @@ AES_GCM_decrypt PROC
pxor xmm6, xmm6 pxor xmm6, xmm6
cmp ebx, 12 cmp ebx, 12
mov edx, ebx mov edx, ebx
jne L_AES_GCM_decrypt_iv_not_12 jne L_AES_GCM_decrypt_aesni_iv_not_12
; # Calculate values when IV is 12 bytes ; # Calculate values when IV is 12 bytes
; Set counter based on IV ; Set counter based on IV
mov ecx, 16777216 mov ecx, 16777216
@ -2043,7 +2043,7 @@ AES_GCM_decrypt PROC
aesenc xmm1, xmm7 aesenc xmm1, xmm7
cmp r10d, 11 cmp r10d, 11
movdqa xmm7, OWORD PTR [r15+160] movdqa xmm7, OWORD PTR [r15+160]
jl L_AES_GCM_decrypt_calc_iv_12_last jl L_AES_GCM_decrypt_aesni_calc_iv_12_last
aesenc xmm5, xmm7 aesenc xmm5, xmm7
aesenc xmm1, xmm7 aesenc xmm1, xmm7
movdqa xmm7, OWORD PTR [r15+176] movdqa xmm7, OWORD PTR [r15+176]
@ -2051,20 +2051,20 @@ AES_GCM_decrypt PROC
aesenc xmm1, xmm7 aesenc xmm1, xmm7
cmp r10d, 13 cmp r10d, 13
movdqa xmm7, OWORD PTR [r15+192] movdqa xmm7, OWORD PTR [r15+192]
jl L_AES_GCM_decrypt_calc_iv_12_last jl L_AES_GCM_decrypt_aesni_calc_iv_12_last
aesenc xmm5, xmm7 aesenc xmm5, xmm7
aesenc xmm1, xmm7 aesenc xmm1, xmm7
movdqa xmm7, OWORD PTR [r15+208] movdqa xmm7, OWORD PTR [r15+208]
aesenc xmm5, xmm7 aesenc xmm5, xmm7
aesenc xmm1, xmm7 aesenc xmm1, xmm7
movdqa xmm7, OWORD PTR [r15+224] movdqa xmm7, OWORD PTR [r15+224]
L_AES_GCM_decrypt_calc_iv_12_last: L_AES_GCM_decrypt_aesni_calc_iv_12_last:
aesenclast xmm5, xmm7 aesenclast xmm5, xmm7
aesenclast xmm1, xmm7 aesenclast xmm1, xmm7
pshufb xmm5, OWORD PTR L_aes_gcm_bswap_mask pshufb xmm5, OWORD PTR L_aes_gcm_bswap_mask
movdqu [rsp+144], xmm1 movdqu [rsp+144], xmm1
jmp L_AES_GCM_decrypt_iv_done jmp L_AES_GCM_decrypt_aesni_iv_done
L_AES_GCM_decrypt_iv_not_12: L_AES_GCM_decrypt_aesni_iv_not_12:
; Calculate values when IV is not 12 bytes ; Calculate values when IV is not 12 bytes
; H = Encrypt X(=0) ; H = Encrypt X(=0)
movdqa xmm5, OWORD PTR [r15] movdqa xmm5, OWORD PTR [r15]
@ -2079,27 +2079,27 @@ L_AES_GCM_decrypt_iv_not_12:
aesenc xmm5, [r15+144] aesenc xmm5, [r15+144]
cmp r10d, 11 cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160] movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_decrypt_calc_iv_1_aesenc_avx_last jl L_AES_GCM_decrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc xmm5, xmm9 aesenc xmm5, xmm9
aesenc xmm5, [r15+176] aesenc xmm5, [r15+176]
cmp r10d, 13 cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192] movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_decrypt_calc_iv_1_aesenc_avx_last jl L_AES_GCM_decrypt_aesni_calc_iv_1_aesenc_avx_last
aesenc xmm5, xmm9 aesenc xmm5, xmm9
aesenc xmm5, [r15+208] aesenc xmm5, [r15+208]
movdqa xmm9, OWORD PTR [r15+224] movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_decrypt_calc_iv_1_aesenc_avx_last: L_AES_GCM_decrypt_aesni_calc_iv_1_aesenc_avx_last:
aesenclast xmm5, xmm9 aesenclast xmm5, xmm9
pshufb xmm5, OWORD PTR L_aes_gcm_bswap_mask pshufb xmm5, OWORD PTR L_aes_gcm_bswap_mask
; Calc counter ; Calc counter
; Initialization vector ; Initialization vector
cmp edx, 0 cmp edx, 0
mov rcx, 0 mov rcx, 0
je L_AES_GCM_decrypt_calc_iv_done je L_AES_GCM_decrypt_aesni_calc_iv_done
cmp edx, 16 cmp edx, 16
jl L_AES_GCM_decrypt_calc_iv_lt16 jl L_AES_GCM_decrypt_aesni_calc_iv_lt16
and edx, 4294967280 and edx, 4294967280
L_AES_GCM_decrypt_calc_iv_16_loop: L_AES_GCM_decrypt_aesni_calc_iv_16_loop:
movdqu xmm8, [rax+rcx] movdqu xmm8, [rax+rcx]
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
pxor xmm4, xmm8 pxor xmm4, xmm8
@ -2159,22 +2159,22 @@ L_AES_GCM_decrypt_calc_iv_16_loop:
pxor xmm4, xmm2 pxor xmm4, xmm2
add ecx, 16 add ecx, 16
cmp ecx, edx cmp ecx, edx
jl L_AES_GCM_decrypt_calc_iv_16_loop jl L_AES_GCM_decrypt_aesni_calc_iv_16_loop
mov edx, ebx mov edx, ebx
cmp ecx, edx cmp ecx, edx
je L_AES_GCM_decrypt_calc_iv_done je L_AES_GCM_decrypt_aesni_calc_iv_done
L_AES_GCM_decrypt_calc_iv_lt16: L_AES_GCM_decrypt_aesni_calc_iv_lt16:
sub rsp, 16 sub rsp, 16
pxor xmm8, xmm8 pxor xmm8, xmm8
xor ebx, ebx xor ebx, ebx
movdqu [rsp], xmm8 movdqu [rsp], xmm8
L_AES_GCM_decrypt_calc_iv_loop: L_AES_GCM_decrypt_aesni_calc_iv_loop:
movzx r13d, BYTE PTR [rax+rcx] movzx r13d, BYTE PTR [rax+rcx]
mov BYTE PTR [rsp+rbx], r13b mov BYTE PTR [rsp+rbx], r13b
inc ecx inc ecx
inc ebx inc ebx
cmp ecx, edx cmp ecx, edx
jl L_AES_GCM_decrypt_calc_iv_loop jl L_AES_GCM_decrypt_aesni_calc_iv_loop
movdqu xmm8, [rsp] movdqu xmm8, [rsp]
add rsp, 16 add rsp, 16
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
@ -2233,7 +2233,7 @@ L_AES_GCM_decrypt_calc_iv_loop:
pxor xmm2, xmm1 pxor xmm2, xmm1
pxor xmm2, xmm7 pxor xmm2, xmm7
pxor xmm4, xmm2 pxor xmm4, xmm2
L_AES_GCM_decrypt_calc_iv_done: L_AES_GCM_decrypt_aesni_calc_iv_done:
; T = Encrypt counter ; T = Encrypt counter
pxor xmm0, xmm0 pxor xmm0, xmm0
shl edx, 3 shl edx, 3
@ -2308,28 +2308,28 @@ L_AES_GCM_decrypt_calc_iv_done:
aesenc xmm8, [r15+144] aesenc xmm8, [r15+144]
cmp r10d, 11 cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160] movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_decrypt_calc_iv_2_aesenc_avx_last jl L_AES_GCM_decrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc xmm8, xmm9 aesenc xmm8, xmm9
aesenc xmm8, [r15+176] aesenc xmm8, [r15+176]
cmp r10d, 13 cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192] movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_decrypt_calc_iv_2_aesenc_avx_last jl L_AES_GCM_decrypt_aesni_calc_iv_2_aesenc_avx_last
aesenc xmm8, xmm9 aesenc xmm8, xmm9
aesenc xmm8, [r15+208] aesenc xmm8, [r15+208]
movdqa xmm9, OWORD PTR [r15+224] movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_decrypt_calc_iv_2_aesenc_avx_last: L_AES_GCM_decrypt_aesni_calc_iv_2_aesenc_avx_last:
aesenclast xmm8, xmm9 aesenclast xmm8, xmm9
movdqu [rsp+144], xmm8 movdqu [rsp+144], xmm8
L_AES_GCM_decrypt_iv_done: L_AES_GCM_decrypt_aesni_iv_done:
; Additional authentication data ; Additional authentication data
mov edx, r11d mov edx, r11d
cmp edx, 0 cmp edx, 0
je L_AES_GCM_decrypt_calc_aad_done je L_AES_GCM_decrypt_aesni_calc_aad_done
xor ecx, ecx xor ecx, ecx
cmp edx, 16 cmp edx, 16
jl L_AES_GCM_decrypt_calc_aad_lt16 jl L_AES_GCM_decrypt_aesni_calc_aad_lt16
and edx, 4294967280 and edx, 4294967280
L_AES_GCM_decrypt_calc_aad_16_loop: L_AES_GCM_decrypt_aesni_calc_aad_16_loop:
movdqu xmm8, [r12+rcx] movdqu xmm8, [r12+rcx]
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
pxor xmm6, xmm8 pxor xmm6, xmm8
@ -2389,22 +2389,22 @@ L_AES_GCM_decrypt_calc_aad_16_loop:
pxor xmm6, xmm2 pxor xmm6, xmm2
add ecx, 16 add ecx, 16
cmp ecx, edx cmp ecx, edx
jl L_AES_GCM_decrypt_calc_aad_16_loop jl L_AES_GCM_decrypt_aesni_calc_aad_16_loop
mov edx, r11d mov edx, r11d
cmp ecx, edx cmp ecx, edx
je L_AES_GCM_decrypt_calc_aad_done je L_AES_GCM_decrypt_aesni_calc_aad_done
L_AES_GCM_decrypt_calc_aad_lt16: L_AES_GCM_decrypt_aesni_calc_aad_lt16:
sub rsp, 16 sub rsp, 16
pxor xmm8, xmm8 pxor xmm8, xmm8
xor ebx, ebx xor ebx, ebx
movdqu [rsp], xmm8 movdqu [rsp], xmm8
L_AES_GCM_decrypt_calc_aad_loop: L_AES_GCM_decrypt_aesni_calc_aad_loop:
movzx r13d, BYTE PTR [r12+rcx] movzx r13d, BYTE PTR [r12+rcx]
mov BYTE PTR [rsp+rbx], r13b mov BYTE PTR [rsp+rbx], r13b
inc ecx inc ecx
inc ebx inc ebx
cmp ecx, edx cmp ecx, edx
jl L_AES_GCM_decrypt_calc_aad_loop jl L_AES_GCM_decrypt_aesni_calc_aad_loop
movdqu xmm8, [rsp] movdqu xmm8, [rsp]
add rsp, 16 add rsp, 16
pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask pshufb xmm8, OWORD PTR L_aes_gcm_bswap_mask
@ -2463,7 +2463,7 @@ L_AES_GCM_decrypt_calc_aad_loop:
pxor xmm2, xmm1 pxor xmm2, xmm1
pxor xmm2, xmm7 pxor xmm2, xmm7
pxor xmm6, xmm2 pxor xmm6, xmm2
L_AES_GCM_decrypt_calc_aad_done: L_AES_GCM_decrypt_aesni_calc_aad_done:
; Calculate counter and H ; Calculate counter and H
pshufb xmm4, OWORD PTR L_aes_gcm_bswap_epi64 pshufb xmm4, OWORD PTR L_aes_gcm_bswap_epi64
movdqa xmm9, xmm5 movdqa xmm9, xmm5
@ -2481,7 +2481,7 @@ L_AES_GCM_decrypt_calc_aad_done:
xor ebx, ebx xor ebx, ebx
cmp r9d, 128 cmp r9d, 128
mov r13d, r9d mov r13d, r9d
jl L_AES_GCM_decrypt_done_128 jl L_AES_GCM_decrypt_aesni_done_128
and r13d, 4294967168 and r13d, 4294967168
movdqa xmm2, xmm6 movdqa xmm2, xmm6
; H ^ 1 ; H ^ 1
@ -2780,7 +2780,7 @@ L_AES_GCM_decrypt_calc_aad_done:
pxor xmm14, xmm8 pxor xmm14, xmm8
pxor xmm7, xmm14 pxor xmm7, xmm14
movdqu [rsp+112], xmm7 movdqu [rsp+112], xmm7
L_AES_GCM_decrypt_ghash_128: L_AES_GCM_decrypt_aesni_ghash_128:
lea rcx, QWORD PTR [rdi+rbx] lea rcx, QWORD PTR [rdi+rbx]
lea rdx, QWORD PTR [rsi+rbx] lea rdx, QWORD PTR [rsi+rbx]
movdqu xmm8, [rsp+128] movdqu xmm8, [rsp+128]
@ -3048,7 +3048,7 @@ L_AES_GCM_decrypt_ghash_128:
pxor xmm2, xmm3 pxor xmm2, xmm3
cmp r10d, 11 cmp r10d, 11
movdqa xmm7, OWORD PTR [r15+160] movdqa xmm7, OWORD PTR [r15+160]
jl L_AES_GCM_decrypt_aesenc_128_ghash_avx_done jl L_AES_GCM_decrypt_aesni_aesenc_128_ghash_avx_done
aesenc xmm8, xmm7 aesenc xmm8, xmm7
aesenc xmm9, xmm7 aesenc xmm9, xmm7
aesenc xmm10, xmm7 aesenc xmm10, xmm7
@ -3068,7 +3068,7 @@ L_AES_GCM_decrypt_ghash_128:
aesenc xmm15, xmm7 aesenc xmm15, xmm7
cmp r10d, 13 cmp r10d, 13
movdqa xmm7, OWORD PTR [r15+192] movdqa xmm7, OWORD PTR [r15+192]
jl L_AES_GCM_decrypt_aesenc_128_ghash_avx_done jl L_AES_GCM_decrypt_aesni_aesenc_128_ghash_avx_done
aesenc xmm8, xmm7 aesenc xmm8, xmm7
aesenc xmm9, xmm7 aesenc xmm9, xmm7
aesenc xmm10, xmm7 aesenc xmm10, xmm7
@ -3087,7 +3087,7 @@ L_AES_GCM_decrypt_ghash_128:
aesenc xmm14, xmm7 aesenc xmm14, xmm7
aesenc xmm15, xmm7 aesenc xmm15, xmm7
movdqa xmm7, OWORD PTR [r15+224] movdqa xmm7, OWORD PTR [r15+224]
L_AES_GCM_decrypt_aesenc_128_ghash_avx_done: L_AES_GCM_decrypt_aesni_aesenc_128_ghash_avx_done:
aesenclast xmm8, xmm7 aesenclast xmm8, xmm7
aesenclast xmm9, xmm7 aesenclast xmm9, xmm7
movdqu xmm0, [rcx] movdqu xmm0, [rcx]
@ -3122,18 +3122,18 @@ L_AES_GCM_decrypt_aesenc_128_ghash_avx_done:
movdqu [rdx+112], xmm15 movdqu [rdx+112], xmm15
add ebx, 128 add ebx, 128
cmp ebx, r13d cmp ebx, r13d
jl L_AES_GCM_decrypt_ghash_128 jl L_AES_GCM_decrypt_aesni_ghash_128
movdqa xmm6, xmm2 movdqa xmm6, xmm2
movdqu xmm5, [rsp] movdqu xmm5, [rsp]
L_AES_GCM_decrypt_done_128: L_AES_GCM_decrypt_aesni_done_128:
mov edx, r9d mov edx, r9d
cmp ebx, edx cmp ebx, edx
jge L_AES_GCM_decrypt_done_dec jge L_AES_GCM_decrypt_aesni_done_dec
mov r13d, r9d mov r13d, r9d
and r13d, 4294967280 and r13d, 4294967280
cmp ebx, r13d cmp ebx, r13d
jge L_AES_GCM_decrypt_last_block_done jge L_AES_GCM_decrypt_aesni_last_block_done
L_AES_GCM_decrypt_last_block_start: L_AES_GCM_decrypt_aesni_last_block_start:
lea rcx, QWORD PTR [rdi+rbx] lea rcx, QWORD PTR [rdi+rbx]
lea rdx, QWORD PTR [rsi+rbx] lea rdx, QWORD PTR [rsi+rbx]
movdqu xmm1, [rcx] movdqu xmm1, [rcx]
@ -3182,28 +3182,28 @@ L_AES_GCM_decrypt_last_block_start:
pxor xmm6, xmm3 pxor xmm6, xmm3
cmp r10d, 11 cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160] movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_decrypt_aesenc_gfmul_last jl L_AES_GCM_decrypt_aesni_aesenc_gfmul_last
aesenc xmm8, xmm9 aesenc xmm8, xmm9
aesenc xmm8, [r15+176] aesenc xmm8, [r15+176]
cmp r10d, 13 cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192] movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_decrypt_aesenc_gfmul_last jl L_AES_GCM_decrypt_aesni_aesenc_gfmul_last
aesenc xmm8, xmm9 aesenc xmm8, xmm9
aesenc xmm8, [r15+208] aesenc xmm8, [r15+208]
movdqa xmm9, OWORD PTR [r15+224] movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_decrypt_aesenc_gfmul_last: L_AES_GCM_decrypt_aesni_aesenc_gfmul_last:
aesenclast xmm8, xmm9 aesenclast xmm8, xmm9
movdqu xmm9, [rcx] movdqu xmm9, [rcx]
pxor xmm8, xmm9 pxor xmm8, xmm9
movdqu [rdx], xmm8 movdqu [rdx], xmm8
add ebx, 16 add ebx, 16
cmp ebx, r13d cmp ebx, r13d
jl L_AES_GCM_decrypt_last_block_start jl L_AES_GCM_decrypt_aesni_last_block_start
L_AES_GCM_decrypt_last_block_done: L_AES_GCM_decrypt_aesni_last_block_done:
mov ecx, r9d mov ecx, r9d
mov edx, ecx mov edx, ecx
and ecx, 15 and ecx, 15
jz L_AES_GCM_decrypt_aesenc_last15_dec_avx_done jz L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_done
movdqu xmm4, [rsp+128] movdqu xmm4, [rsp+128]
pshufb xmm4, OWORD PTR L_aes_gcm_bswap_epi64 pshufb xmm4, OWORD PTR L_aes_gcm_bswap_epi64
pxor xmm4, [r15] pxor xmm4, [r15]
@ -3218,23 +3218,23 @@ L_AES_GCM_decrypt_last_block_done:
aesenc xmm4, [r15+144] aesenc xmm4, [r15+144]
cmp r10d, 11 cmp r10d, 11
movdqa xmm9, OWORD PTR [r15+160] movdqa xmm9, OWORD PTR [r15+160]
jl L_AES_GCM_decrypt_aesenc_last15_dec_avx_aesenc_avx_last jl L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_aesenc_avx_last
aesenc xmm4, xmm9 aesenc xmm4, xmm9
aesenc xmm4, [r15+176] aesenc xmm4, [r15+176]
cmp r10d, 13 cmp r10d, 13
movdqa xmm9, OWORD PTR [r15+192] movdqa xmm9, OWORD PTR [r15+192]
jl L_AES_GCM_decrypt_aesenc_last15_dec_avx_aesenc_avx_last jl L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_aesenc_avx_last
aesenc xmm4, xmm9 aesenc xmm4, xmm9
aesenc xmm4, [r15+208] aesenc xmm4, [r15+208]
movdqa xmm9, OWORD PTR [r15+224] movdqa xmm9, OWORD PTR [r15+224]
L_AES_GCM_decrypt_aesenc_last15_dec_avx_aesenc_avx_last: L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_aesenc_avx_last:
aesenclast xmm4, xmm9 aesenclast xmm4, xmm9
sub rsp, 32 sub rsp, 32
xor ecx, ecx xor ecx, ecx
movdqu [rsp], xmm4 movdqu [rsp], xmm4
pxor xmm0, xmm0 pxor xmm0, xmm0
movdqu [rsp+16], xmm0 movdqu [rsp+16], xmm0
L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop: L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_loop:
movzx r13d, BYTE PTR [rdi+rbx] movzx r13d, BYTE PTR [rdi+rbx]
mov BYTE PTR [rsp+rcx+16], r13b mov BYTE PTR [rsp+rcx+16], r13b
xor r13b, BYTE PTR [rsp+rcx] xor r13b, BYTE PTR [rsp+rcx]
@ -3242,7 +3242,7 @@ L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop:
inc ebx inc ebx
inc ecx inc ecx
cmp ebx, edx cmp ebx, edx
jl L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop jl L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_loop
movdqu xmm4, [rsp+16] movdqu xmm4, [rsp+16]
add rsp, 32 add rsp, 32
pshufb xmm4, OWORD PTR L_aes_gcm_bswap_mask pshufb xmm4, OWORD PTR L_aes_gcm_bswap_mask
@ -3287,8 +3287,8 @@ L_AES_GCM_decrypt_aesenc_last15_dec_avx_loop:
pxor xmm14, xmm13 pxor xmm14, xmm13
pxor xmm14, xmm8 pxor xmm14, xmm8
pxor xmm6, xmm14 pxor xmm6, xmm14
L_AES_GCM_decrypt_aesenc_last15_dec_avx_done: L_AES_GCM_decrypt_aesni_aesenc_last15_dec_avx_done:
L_AES_GCM_decrypt_done_dec: L_AES_GCM_decrypt_aesni_done_dec:
mov edx, r9d mov edx, r9d
mov ecx, r11d mov ecx, r11d
shl rdx, 3 shl rdx, 3
@ -3340,24 +3340,24 @@ L_AES_GCM_decrypt_done_dec:
movdqu xmm0, [rsp+144] movdqu xmm0, [rsp+144]
pxor xmm0, xmm6 pxor xmm0, xmm6
cmp r14d, 16 cmp r14d, 16
je L_AES_GCM_decrypt_cmp_tag_16 je L_AES_GCM_decrypt_aesni_cmp_tag_16
sub rsp, 16 sub rsp, 16
xor rcx, rcx xor rcx, rcx
xor rbx, rbx xor rbx, rbx
movdqu [rsp], xmm0 movdqu [rsp], xmm0
L_AES_GCM_decrypt_cmp_tag_loop: L_AES_GCM_decrypt_aesni_cmp_tag_loop:
movzx r13d, BYTE PTR [rsp+rcx] movzx r13d, BYTE PTR [rsp+rcx]
xor r13b, BYTE PTR [r8+rcx] xor r13b, BYTE PTR [r8+rcx]
or bl, r13b or bl, r13b
inc ecx inc ecx
cmp ecx, r14d cmp ecx, r14d
jne L_AES_GCM_decrypt_cmp_tag_loop jne L_AES_GCM_decrypt_aesni_cmp_tag_loop
cmp rbx, 0 cmp rbx, 0
sete bl sete bl
add rsp, 16 add rsp, 16
xor rcx, rcx xor rcx, rcx
jmp L_AES_GCM_decrypt_cmp_tag_done jmp L_AES_GCM_decrypt_aesni_cmp_tag_done
L_AES_GCM_decrypt_cmp_tag_16: L_AES_GCM_decrypt_aesni_cmp_tag_16:
movdqu xmm1, [r8] movdqu xmm1, [r8]
pcmpeqb xmm0, xmm1 pcmpeqb xmm0, xmm1
pmovmskb rdx, xmm0 pmovmskb rdx, xmm0
@ -3365,7 +3365,7 @@ L_AES_GCM_decrypt_cmp_tag_16:
xor ebx, ebx xor ebx, ebx
cmp edx, 65535 cmp edx, 65535
sete bl sete bl
L_AES_GCM_decrypt_cmp_tag_done: L_AES_GCM_decrypt_aesni_cmp_tag_done:
mov DWORD PTR [rbp], ebx mov DWORD PTR [rbp], ebx
movdqu xmm6, [rsp+168] movdqu xmm6, [rsp+168]
movdqu xmm7, [rsp+184] movdqu xmm7, [rsp+184]
@ -3387,7 +3387,7 @@ L_AES_GCM_decrypt_cmp_tag_done:
pop rdi pop rdi
pop r13 pop r13
ret ret
AES_GCM_decrypt ENDP AES_GCM_decrypt_aesni ENDP
_text ENDS _text ENDS
_text SEGMENT READONLY PARA _text SEGMENT READONLY PARA
AES_GCM_init_aesni PROC AES_GCM_init_aesni PROC

140
wolfcrypt/src/aes_xts_asm.S
View File

@ -56,15 +56,15 @@ L_aes_xts_gc_xts:
.long 0x87,0x1,0x1,0x1 .long 0x87,0x1,0x1,0x1
#ifndef __APPLE__ #ifndef __APPLE__
.text .text
.globl AES_XTS_encrypt .globl AES_XTS_encrypt_aesni
.type AES_XTS_encrypt,@function .type AES_XTS_encrypt_aesni,@function
.align 16 .align 16
AES_XTS_encrypt: AES_XTS_encrypt_aesni:
#else #else
.section __TEXT,__text .section __TEXT,__text
.globl _AES_XTS_encrypt .globl _AES_XTS_encrypt_aesni
.p2align 4 .p2align 4
_AES_XTS_encrypt: _AES_XTS_encrypt_aesni:
#endif /* __APPLE__ */ #endif /* __APPLE__ */
pushq %r12 pushq %r12
pushq %r13 pushq %r13
@ -96,25 +96,25 @@ _AES_XTS_encrypt:
aesenc %xmm5, %xmm0 aesenc %xmm5, %xmm0
cmpl $11, %r10d cmpl $11, %r10d
movdqu 160(%r9), %xmm5 movdqu 160(%r9), %xmm5
jl L_AES_XTS_encrypt_tweak_aes_enc_block_last jl L_AES_XTS_encrypt_aesni_tweak_aes_enc_block_last
aesenc %xmm5, %xmm0 aesenc %xmm5, %xmm0
movdqu 176(%r9), %xmm6 movdqu 176(%r9), %xmm6
aesenc %xmm6, %xmm0 aesenc %xmm6, %xmm0
cmpl $13, %r10d cmpl $13, %r10d
movdqu 192(%r9), %xmm5 movdqu 192(%r9), %xmm5
jl L_AES_XTS_encrypt_tweak_aes_enc_block_last jl L_AES_XTS_encrypt_aesni_tweak_aes_enc_block_last
aesenc %xmm5, %xmm0 aesenc %xmm5, %xmm0
movdqu 208(%r9), %xmm6 movdqu 208(%r9), %xmm6
aesenc %xmm6, %xmm0 aesenc %xmm6, %xmm0
movdqu 224(%r9), %xmm5 movdqu 224(%r9), %xmm5
L_AES_XTS_encrypt_tweak_aes_enc_block_last: L_AES_XTS_encrypt_aesni_tweak_aes_enc_block_last:
aesenclast %xmm5, %xmm0 aesenclast %xmm5, %xmm0
xorl %r13d, %r13d xorl %r13d, %r13d
cmpl $0x40, %eax cmpl $0x40, %eax
movl %eax, %r11d movl %eax, %r11d
jl L_AES_XTS_encrypt_done_64 jl L_AES_XTS_encrypt_aesni_done_64
andl $0xffffffc0, %r11d andl $0xffffffc0, %r11d
L_AES_XTS_encrypt_enc_64: L_AES_XTS_encrypt_aesni_enc_64:
# 64 bytes of input # 64 bytes of input
# aes_enc_64 # aes_enc_64
leaq (%rdi,%r13,1), %rcx leaq (%rdi,%r13,1), %rcx
@ -201,7 +201,7 @@ L_AES_XTS_encrypt_enc_64:
aesenc %xmm4, %xmm11 aesenc %xmm4, %xmm11
cmpl $11, %r10d cmpl $11, %r10d
movdqu 160(%r8), %xmm4 movdqu 160(%r8), %xmm4
jl L_AES_XTS_encrypt_aes_enc_64_aes_enc_block_last jl L_AES_XTS_encrypt_aesni_aes_enc_64_aes_enc_block_last
aesenc %xmm4, %xmm8 aesenc %xmm4, %xmm8
aesenc %xmm4, %xmm9 aesenc %xmm4, %xmm9
aesenc %xmm4, %xmm10 aesenc %xmm4, %xmm10
@ -213,7 +213,7 @@ L_AES_XTS_encrypt_enc_64:
aesenc %xmm4, %xmm11 aesenc %xmm4, %xmm11
cmpl $13, %r10d cmpl $13, %r10d
movdqu 192(%r8), %xmm4 movdqu 192(%r8), %xmm4
jl L_AES_XTS_encrypt_aes_enc_64_aes_enc_block_last jl L_AES_XTS_encrypt_aesni_aes_enc_64_aes_enc_block_last
aesenc %xmm4, %xmm8 aesenc %xmm4, %xmm8
aesenc %xmm4, %xmm9 aesenc %xmm4, %xmm9
aesenc %xmm4, %xmm10 aesenc %xmm4, %xmm10
@ -224,7 +224,7 @@ L_AES_XTS_encrypt_enc_64:
aesenc %xmm4, %xmm10 aesenc %xmm4, %xmm10
aesenc %xmm4, %xmm11 aesenc %xmm4, %xmm11
movdqu 224(%r8), %xmm4 movdqu 224(%r8), %xmm4
L_AES_XTS_encrypt_aes_enc_64_aes_enc_block_last: L_AES_XTS_encrypt_aesni_aes_enc_64_aes_enc_block_last:
aesenclast %xmm4, %xmm8 aesenclast %xmm4, %xmm8
aesenclast %xmm4, %xmm9 aesenclast %xmm4, %xmm9
aesenclast %xmm4, %xmm10 aesenclast %xmm4, %xmm10
@ -246,18 +246,18 @@ L_AES_XTS_encrypt_aes_enc_64_aes_enc_block_last:
pxor %xmm4, %xmm0 pxor %xmm4, %xmm0
addl $0x40, %r13d addl $0x40, %r13d
cmpl %r11d, %r13d cmpl %r11d, %r13d
jl L_AES_XTS_encrypt_enc_64 jl L_AES_XTS_encrypt_aesni_enc_64
L_AES_XTS_encrypt_done_64: L_AES_XTS_encrypt_aesni_done_64:
cmpl %eax, %r13d cmpl %eax, %r13d
movl %eax, %r11d movl %eax, %r11d
je L_AES_XTS_encrypt_done_enc je L_AES_XTS_encrypt_aesni_done_enc
subl %r13d, %r11d subl %r13d, %r11d
cmpl $16, %r11d cmpl $16, %r11d
movl %eax, %r11d movl %eax, %r11d
jl L_AES_XTS_encrypt_last_15 jl L_AES_XTS_encrypt_aesni_last_15
andl $0xfffffff0, %r11d andl $0xfffffff0, %r11d
# 16 bytes of input # 16 bytes of input
L_AES_XTS_encrypt_enc_16: L_AES_XTS_encrypt_aesni_enc_16:
leaq (%rdi,%r13,1), %rcx leaq (%rdi,%r13,1), %rcx
movdqu (%rcx), %xmm8 movdqu (%rcx), %xmm8
pxor %xmm0, %xmm8 pxor %xmm0, %xmm8
@ -283,18 +283,18 @@ L_AES_XTS_encrypt_enc_16:
aesenc %xmm5, %xmm8 aesenc %xmm5, %xmm8
cmpl $11, %r10d cmpl $11, %r10d
movdqu 160(%r8), %xmm5 movdqu 160(%r8), %xmm5
jl L_AES_XTS_encrypt_aes_enc_block_last jl L_AES_XTS_encrypt_aesni_aes_enc_block_last
aesenc %xmm5, %xmm8 aesenc %xmm5, %xmm8
movdqu 176(%r8), %xmm6 movdqu 176(%r8), %xmm6
aesenc %xmm6, %xmm8 aesenc %xmm6, %xmm8
cmpl $13, %r10d cmpl $13, %r10d
movdqu 192(%r8), %xmm5 movdqu 192(%r8), %xmm5
jl L_AES_XTS_encrypt_aes_enc_block_last jl L_AES_XTS_encrypt_aesni_aes_enc_block_last
aesenc %xmm5, %xmm8 aesenc %xmm5, %xmm8
movdqu 208(%r8), %xmm6 movdqu 208(%r8), %xmm6
aesenc %xmm6, %xmm8 aesenc %xmm6, %xmm8
movdqu 224(%r8), %xmm5 movdqu 224(%r8), %xmm5
L_AES_XTS_encrypt_aes_enc_block_last: L_AES_XTS_encrypt_aesni_aes_enc_block_last:
aesenclast %xmm5, %xmm8 aesenclast %xmm5, %xmm8
pxor %xmm0, %xmm8 pxor %xmm0, %xmm8
leaq (%rsi,%r13,1), %rcx leaq (%rsi,%r13,1), %rcx
@ -307,17 +307,17 @@ L_AES_XTS_encrypt_aes_enc_block_last:
pxor %xmm4, %xmm0 pxor %xmm4, %xmm0
addl $16, %r13d addl $16, %r13d
cmpl %r11d, %r13d cmpl %r11d, %r13d
jl L_AES_XTS_encrypt_enc_16 jl L_AES_XTS_encrypt_aesni_enc_16
cmpl %eax, %r13d cmpl %eax, %r13d
je L_AES_XTS_encrypt_done_enc je L_AES_XTS_encrypt_aesni_done_enc
L_AES_XTS_encrypt_last_15: L_AES_XTS_encrypt_aesni_last_15:
subq $16, %r13 subq $16, %r13
leaq (%rsi,%r13,1), %rcx leaq (%rsi,%r13,1), %rcx
movdqu (%rcx), %xmm8 movdqu (%rcx), %xmm8
addq $16, %r13 addq $16, %r13
movdqu %xmm8, (%rsp) movdqu %xmm8, (%rsp)
xorq %rdx, %rdx xorq %rdx, %rdx
L_AES_XTS_encrypt_last_15_byte_loop: L_AES_XTS_encrypt_aesni_last_15_byte_loop:
movb (%rsp,%rdx,1), %r11b movb (%rsp,%rdx,1), %r11b
movb (%rdi,%r13,1), %cl movb (%rdi,%r13,1), %cl
movb %r11b, (%rsi,%r13,1) movb %r11b, (%rsi,%r13,1)
@ -325,7 +325,7 @@ L_AES_XTS_encrypt_last_15_byte_loop:
incl %r13d incl %r13d
incl %edx incl %edx
cmpl %eax, %r13d cmpl %eax, %r13d
jl L_AES_XTS_encrypt_last_15_byte_loop jl L_AES_XTS_encrypt_aesni_last_15_byte_loop
subq %rdx, %r13 subq %rdx, %r13
movdqu (%rsp), %xmm8 movdqu (%rsp), %xmm8
subq $16, %r13 subq $16, %r13
@ -352,41 +352,41 @@ L_AES_XTS_encrypt_last_15_byte_loop:
aesenc %xmm5, %xmm8 aesenc %xmm5, %xmm8
cmpl $11, %r10d cmpl $11, %r10d
movdqu 160(%r8), %xmm5 movdqu 160(%r8), %xmm5
jl L_AES_XTS_encrypt_last_15_aes_enc_block_last jl L_AES_XTS_encrypt_aesni_last_15_aes_enc_block_last
aesenc %xmm5, %xmm8 aesenc %xmm5, %xmm8
movdqu 176(%r8), %xmm6 movdqu 176(%r8), %xmm6
aesenc %xmm6, %xmm8 aesenc %xmm6, %xmm8
cmpl $13, %r10d cmpl $13, %r10d
movdqu 192(%r8), %xmm5 movdqu 192(%r8), %xmm5
jl L_AES_XTS_encrypt_last_15_aes_enc_block_last jl L_AES_XTS_encrypt_aesni_last_15_aes_enc_block_last
aesenc %xmm5, %xmm8 aesenc %xmm5, %xmm8
movdqu 208(%r8), %xmm6 movdqu 208(%r8), %xmm6
aesenc %xmm6, %xmm8 aesenc %xmm6, %xmm8
movdqu 224(%r8), %xmm5 movdqu 224(%r8), %xmm5
L_AES_XTS_encrypt_last_15_aes_enc_block_last: L_AES_XTS_encrypt_aesni_last_15_aes_enc_block_last:
aesenclast %xmm5, %xmm8 aesenclast %xmm5, %xmm8
pxor %xmm0, %xmm8 pxor %xmm0, %xmm8
leaq (%rsi,%r13,1), %rcx leaq (%rsi,%r13,1), %rcx
movdqu %xmm8, (%rcx) movdqu %xmm8, (%rcx)
L_AES_XTS_encrypt_done_enc: L_AES_XTS_encrypt_aesni_done_enc:
addq $0x40, %rsp addq $0x40, %rsp
popq %r13 popq %r13
popq %r12 popq %r12
repz retq repz retq
#ifndef __APPLE__ #ifndef __APPLE__
.size AES_XTS_encrypt,.-AES_XTS_encrypt .size AES_XTS_encrypt_aesni,.-AES_XTS_encrypt_aesni
#endif /* __APPLE__ */ #endif /* __APPLE__ */
#ifndef __APPLE__ #ifndef __APPLE__
.text .text
.globl AES_XTS_decrypt .globl AES_XTS_decrypt_aesni
.type AES_XTS_decrypt,@function .type AES_XTS_decrypt_aesni,@function
.align 16 .align 16
AES_XTS_decrypt: AES_XTS_decrypt_aesni:
#else #else
.section __TEXT,__text .section __TEXT,__text
.globl _AES_XTS_decrypt .globl _AES_XTS_decrypt_aesni
.p2align 4 .p2align 4
_AES_XTS_decrypt: _AES_XTS_decrypt_aesni:
#endif /* __APPLE__ */ #endif /* __APPLE__ */
pushq %r12 pushq %r12
pushq %r13 pushq %r13
@ -418,32 +418,32 @@ _AES_XTS_decrypt:
aesenc %xmm5, %xmm0 aesenc %xmm5, %xmm0
cmpl $11, %r10d cmpl $11, %r10d
movdqu 160(%r9), %xmm5 movdqu 160(%r9), %xmm5
jl L_AES_XTS_decrypt_tweak_aes_enc_block_last jl L_AES_XTS_decrypt_aesni_tweak_aes_enc_block_last
aesenc %xmm5, %xmm0 aesenc %xmm5, %xmm0
movdqu 176(%r9), %xmm6 movdqu 176(%r9), %xmm6
aesenc %xmm6, %xmm0 aesenc %xmm6, %xmm0
cmpl $13, %r10d cmpl $13, %r10d
movdqu 192(%r9), %xmm5 movdqu 192(%r9), %xmm5
jl L_AES_XTS_decrypt_tweak_aes_enc_block_last jl L_AES_XTS_decrypt_aesni_tweak_aes_enc_block_last
aesenc %xmm5, %xmm0 aesenc %xmm5, %xmm0
movdqu 208(%r9), %xmm6 movdqu 208(%r9), %xmm6
aesenc %xmm6, %xmm0 aesenc %xmm6, %xmm0
movdqu 224(%r9), %xmm5 movdqu 224(%r9), %xmm5
L_AES_XTS_decrypt_tweak_aes_enc_block_last: L_AES_XTS_decrypt_aesni_tweak_aes_enc_block_last:
aesenclast %xmm5, %xmm0 aesenclast %xmm5, %xmm0
xorl %r13d, %r13d xorl %r13d, %r13d
movl %eax, %r11d movl %eax, %r11d
andl $0xfffffff0, %r11d andl $0xfffffff0, %r11d
cmpl %eax, %r11d cmpl %eax, %r11d
je L_AES_XTS_decrypt_mul16_64 je L_AES_XTS_decrypt_aesni_mul16_64
subl $16, %r11d subl $16, %r11d
cmpl $16, %r11d cmpl $16, %r11d
jl L_AES_XTS_decrypt_last_31_start jl L_AES_XTS_decrypt_aesni_last_31_start
L_AES_XTS_decrypt_mul16_64: L_AES_XTS_decrypt_aesni_mul16_64:
cmpl $0x40, %r11d cmpl $0x40, %r11d
jl L_AES_XTS_decrypt_done_64 jl L_AES_XTS_decrypt_aesni_done_64
andl $0xffffffc0, %r11d andl $0xffffffc0, %r11d
L_AES_XTS_decrypt_dec_64: L_AES_XTS_decrypt_aesni_dec_64:
# 64 bytes of input # 64 bytes of input
# aes_dec_64 # aes_dec_64
leaq (%rdi,%r13,1), %rcx leaq (%rdi,%r13,1), %rcx
@ -530,7 +530,7 @@ L_AES_XTS_decrypt_dec_64:
aesdec %xmm4, %xmm11 aesdec %xmm4, %xmm11
cmpl $11, %r10d cmpl $11, %r10d
movdqu 160(%r8), %xmm4 movdqu 160(%r8), %xmm4
jl L_AES_XTS_decrypt_aes_dec_64_aes_dec_block_last jl L_AES_XTS_decrypt_aesni_aes_dec_64_aes_dec_block_last
aesdec %xmm4, %xmm8 aesdec %xmm4, %xmm8
aesdec %xmm4, %xmm9 aesdec %xmm4, %xmm9
aesdec %xmm4, %xmm10 aesdec %xmm4, %xmm10
@ -542,7 +542,7 @@ L_AES_XTS_decrypt_dec_64:
aesdec %xmm4, %xmm11 aesdec %xmm4, %xmm11
cmpl $13, %r10d cmpl $13, %r10d
movdqu 192(%r8), %xmm4 movdqu 192(%r8), %xmm4
jl L_AES_XTS_decrypt_aes_dec_64_aes_dec_block_last jl L_AES_XTS_decrypt_aesni_aes_dec_64_aes_dec_block_last
aesdec %xmm4, %xmm8 aesdec %xmm4, %xmm8
aesdec %xmm4, %xmm9 aesdec %xmm4, %xmm9
aesdec %xmm4, %xmm10 aesdec %xmm4, %xmm10
@ -553,7 +553,7 @@ L_AES_XTS_decrypt_dec_64:
aesdec %xmm4, %xmm10 aesdec %xmm4, %xmm10
aesdec %xmm4, %xmm11 aesdec %xmm4, %xmm11
movdqu 224(%r8), %xmm4 movdqu 224(%r8), %xmm4
L_AES_XTS_decrypt_aes_dec_64_aes_dec_block_last: L_AES_XTS_decrypt_aesni_aes_dec_64_aes_dec_block_last:
aesdeclast %xmm4, %xmm8 aesdeclast %xmm4, %xmm8
aesdeclast %xmm4, %xmm9 aesdeclast %xmm4, %xmm9
aesdeclast %xmm4, %xmm10 aesdeclast %xmm4, %xmm10
@ -575,21 +575,21 @@ L_AES_XTS_decrypt_aes_dec_64_aes_dec_block_last:
pxor %xmm4, %xmm0 pxor %xmm4, %xmm0
addl $0x40, %r13d addl $0x40, %r13d
cmpl %r11d, %r13d cmpl %r11d, %r13d
jl L_AES_XTS_decrypt_dec_64 jl L_AES_XTS_decrypt_aesni_dec_64
L_AES_XTS_decrypt_done_64: L_AES_XTS_decrypt_aesni_done_64:
cmpl %eax, %r13d cmpl %eax, %r13d
movl %eax, %r11d movl %eax, %r11d
je L_AES_XTS_decrypt_done_dec je L_AES_XTS_decrypt_aesni_done_dec
andl $0xfffffff0, %r11d andl $0xfffffff0, %r11d
cmpl %eax, %r11d cmpl %eax, %r11d
je L_AES_XTS_decrypt_mul16 je L_AES_XTS_decrypt_aesni_mul16
subl $16, %r11d subl $16, %r11d
subl %r13d, %r11d subl %r13d, %r11d
cmpl $16, %r11d cmpl $16, %r11d
jl L_AES_XTS_decrypt_last_31_start jl L_AES_XTS_decrypt_aesni_last_31_start
addl %r13d, %r11d addl %r13d, %r11d
L_AES_XTS_decrypt_mul16: L_AES_XTS_decrypt_aesni_mul16:
L_AES_XTS_decrypt_dec_16: L_AES_XTS_decrypt_aesni_dec_16:
# 16 bytes of input # 16 bytes of input
leaq (%rdi,%r13,1), %rcx leaq (%rdi,%r13,1), %rcx
movdqu (%rcx), %xmm8 movdqu (%rcx), %xmm8
@ -616,18 +616,18 @@ L_AES_XTS_decrypt_dec_16:
aesdec %xmm5, %xmm8 aesdec %xmm5, %xmm8
cmpl $11, %r10d cmpl $11, %r10d
movdqu 160(%r8), %xmm5 movdqu 160(%r8), %xmm5
jl L_AES_XTS_decrypt_aes_dec_block_last jl L_AES_XTS_decrypt_aesni_aes_dec_block_last
aesdec %xmm5, %xmm8 aesdec %xmm5, %xmm8
movdqu 176(%r8), %xmm6 movdqu 176(%r8), %xmm6
aesdec %xmm6, %xmm8 aesdec %xmm6, %xmm8
cmpl $13, %r10d cmpl $13, %r10d
movdqu 192(%r8), %xmm5 movdqu 192(%r8), %xmm5
jl L_AES_XTS_decrypt_aes_dec_block_last jl L_AES_XTS_decrypt_aesni_aes_dec_block_last
aesdec %xmm5, %xmm8 aesdec %xmm5, %xmm8
movdqu 208(%r8), %xmm6 movdqu 208(%r8), %xmm6
aesdec %xmm6, %xmm8 aesdec %xmm6, %xmm8
movdqu 224(%r8), %xmm5 movdqu 224(%r8), %xmm5
L_AES_XTS_decrypt_aes_dec_block_last: L_AES_XTS_decrypt_aesni_aes_dec_block_last:
aesdeclast %xmm5, %xmm8 aesdeclast %xmm5, %xmm8
pxor %xmm0, %xmm8 pxor %xmm0, %xmm8
leaq (%rsi,%r13,1), %rcx leaq (%rsi,%r13,1), %rcx
@ -640,10 +640,10 @@ L_AES_XTS_decrypt_aes_dec_block_last:
pxor %xmm4, %xmm0 pxor %xmm4, %xmm0
addl $16, %r13d addl $16, %r13d
cmpl %r11d, %r13d cmpl %r11d, %r13d
jl L_AES_XTS_decrypt_dec_16 jl L_AES_XTS_decrypt_aesni_dec_16
cmpl %eax, %r13d cmpl %eax, %r13d
je L_AES_XTS_decrypt_done_dec je L_AES_XTS_decrypt_aesni_done_dec
L_AES_XTS_decrypt_last_31_start: L_AES_XTS_decrypt_aesni_last_31_start:
movdqa %xmm0, %xmm4 movdqa %xmm0, %xmm4
movdqa %xmm0, %xmm7 movdqa %xmm0, %xmm7
psrad $31, %xmm4 psrad $31, %xmm4
@ -676,24 +676,24 @@ L_AES_XTS_decrypt_last_31_start:
aesdec %xmm5, %xmm8 aesdec %xmm5, %xmm8
cmpl $11, %r10d cmpl $11, %r10d
movdqu 160(%r8), %xmm5 movdqu 160(%r8), %xmm5
jl L_AES_XTS_decrypt_last_31_aes_dec_block_last jl L_AES_XTS_decrypt_aesni_last_31_aes_dec_block_last
aesdec %xmm5, %xmm8 aesdec %xmm5, %xmm8
movdqu 176(%r8), %xmm6 movdqu 176(%r8), %xmm6
aesdec %xmm6, %xmm8 aesdec %xmm6, %xmm8
cmpl $13, %r10d cmpl $13, %r10d
movdqu 192(%r8), %xmm5 movdqu 192(%r8), %xmm5
jl L_AES_XTS_decrypt_last_31_aes_dec_block_last jl L_AES_XTS_decrypt_aesni_last_31_aes_dec_block_last
aesdec %xmm5, %xmm8 aesdec %xmm5, %xmm8
movdqu 208(%r8), %xmm6 movdqu 208(%r8), %xmm6
aesdec %xmm6, %xmm8 aesdec %xmm6, %xmm8
movdqu 224(%r8), %xmm5 movdqu 224(%r8), %xmm5
L_AES_XTS_decrypt_last_31_aes_dec_block_last: L_AES_XTS_decrypt_aesni_last_31_aes_dec_block_last:
aesdeclast %xmm5, %xmm8 aesdeclast %xmm5, %xmm8
pxor %xmm7, %xmm8 pxor %xmm7, %xmm8
movdqu %xmm8, (%rsp) movdqu %xmm8, (%rsp)
addq $16, %r13 addq $16, %r13
xorq %rdx, %rdx xorq %rdx, %rdx
L_AES_XTS_decrypt_last_31_byte_loop: L_AES_XTS_decrypt_aesni_last_31_byte_loop:
movb (%rsp,%rdx,1), %r11b movb (%rsp,%rdx,1), %r11b
movb (%rdi,%r13,1), %cl movb (%rdi,%r13,1), %cl
movb %r11b, (%rsi,%r13,1) movb %r11b, (%rsi,%r13,1)
@ -701,7 +701,7 @@ L_AES_XTS_decrypt_last_31_byte_loop:
incl %r13d incl %r13d
incl %edx incl %edx
cmpl %eax, %r13d cmpl %eax, %r13d
jl L_AES_XTS_decrypt_last_31_byte_loop jl L_AES_XTS_decrypt_aesni_last_31_byte_loop
subq %rdx, %r13 subq %rdx, %r13
movdqu (%rsp), %xmm8 movdqu (%rsp), %xmm8
pxor %xmm0, %xmm8 pxor %xmm0, %xmm8
@ -727,30 +727,30 @@ L_AES_XTS_decrypt_last_31_byte_loop:
aesdec %xmm5, %xmm8 aesdec %xmm5, %xmm8
cmpl $11, %r10d cmpl $11, %r10d
movdqu 160(%r8), %xmm5 movdqu 160(%r8), %xmm5
jl L_AES_XTS_decrypt_last_31_2_aes_dec_block_last jl L_AES_XTS_decrypt_aesni_last_31_2_aes_dec_block_last
aesdec %xmm5, %xmm8 aesdec %xmm5, %xmm8
movdqu 176(%r8), %xmm6 movdqu 176(%r8), %xmm6
aesdec %xmm6, %xmm8 aesdec %xmm6, %xmm8
cmpl $13, %r10d cmpl $13, %r10d
movdqu 192(%r8), %xmm5 movdqu 192(%r8), %xmm5
jl L_AES_XTS_decrypt_last_31_2_aes_dec_block_last jl L_AES_XTS_decrypt_aesni_last_31_2_aes_dec_block_last
aesdec %xmm5, %xmm8 aesdec %xmm5, %xmm8
movdqu 208(%r8), %xmm6 movdqu 208(%r8), %xmm6
aesdec %xmm6, %xmm8 aesdec %xmm6, %xmm8
movdqu 224(%r8), %xmm5 movdqu 224(%r8), %xmm5
L_AES_XTS_decrypt_last_31_2_aes_dec_block_last: L_AES_XTS_decrypt_aesni_last_31_2_aes_dec_block_last:
aesdeclast %xmm5, %xmm8 aesdeclast %xmm5, %xmm8
pxor %xmm0, %xmm8 pxor %xmm0, %xmm8
subq $16, %r13 subq $16, %r13
leaq (%rsi,%r13,1), %rcx leaq (%rsi,%r13,1), %rcx
movdqu %xmm8, (%rcx) movdqu %xmm8, (%rcx)
L_AES_XTS_decrypt_done_dec: L_AES_XTS_decrypt_aesni_done_dec:
addq $16, %rsp addq $16, %rsp
popq %r13 popq %r13
popq %r12 popq %r12
repz retq repz retq
#ifndef __APPLE__ #ifndef __APPLE__
.size AES_XTS_decrypt,.-AES_XTS_decrypt .size AES_XTS_decrypt_aesni,.-AES_XTS_decrypt_aesni
#endif /* __APPLE__ */ #endif /* __APPLE__ */
#ifdef HAVE_INTEL_AVX1 #ifdef HAVE_INTEL_AVX1
#ifndef __APPLE__ #ifndef __APPLE__

5
wolfcrypt/src/asn.c
View File

@ -30979,6 +30979,9 @@ int wc_SetSubjectKeyId(Cert *cert, const char* file)
#ifdef HAVE_ECC #ifdef HAVE_ECC
wc_ecc_free(eckey); wc_ecc_free(eckey);
XFREE(eckey, cert->heap, DYNAMIC_TYPE_ECC); XFREE(eckey, cert->heap, DYNAMIC_TYPE_ECC);
#endif
#if defined(NO_RSA) && !defined(HAVE_ECC)
(void)idx;
#endif #endif
return ret; return ret;
} }
@ -32191,7 +32194,7 @@ int DecodeECC_DSA_Sig_Ex(const byte* sig, word32 sigLen, mp_int* r, mp_int* s,
#ifdef WOLFSSL_ASN_TEMPLATE #ifdef WOLFSSL_ASN_TEMPLATE
#ifdef WOLFSSL_CUSTOM_CURVES #if defined(HAVE_ECC) && defined(WOLFSSL_CUSTOM_CURVES)
/* Convert data to hex string. /* Convert data to hex string.
* *
* Big-endian byte array is converted to big-endian hexadecimal string. * Big-endian byte array is converted to big-endian hexadecimal string.

4
wolfcrypt/src/evp.c
View File

@ -4672,7 +4672,7 @@ int wolfSSL_EVP_read_pw_string(char* buf, int bufSz, const char* banner, int v)
} }
#endif /* WOLFSSL_APACHE_HTTPD */ #endif /* WOLFSSL_APACHE_HTTPD */
#if !defined(NO_PWDBASED) && !defined(NO_SHA) #if !defined(NO_PWDBASED) && !defined(NO_SHA) && !defined(NO_HMAC)
int wolfSSL_PKCS5_PBKDF2_HMAC_SHA1(const char *pass, int passlen, int wolfSSL_PKCS5_PBKDF2_HMAC_SHA1(const char *pass, int passlen,
const unsigned char *salt, const unsigned char *salt,
int saltlen, int iter, int saltlen, int iter,
@ -4698,7 +4698,7 @@ int wolfSSL_PKCS5_PBKDF2_HMAC_SHA1(const char *pass, int passlen,
} }
#endif /* !NO_PWDBASED !NO_SHA*/ #endif /* !NO_PWDBASED !NO_SHA*/
#if !defined(NO_PWDBASED) #if !defined(NO_PWDBASED) && !defined(NO_HMAC)
int wolfSSL_PKCS5_PBKDF2_HMAC(const char *pass, int passlen, int wolfSSL_PKCS5_PBKDF2_HMAC(const char *pass, int passlen,
const unsigned char *salt, const unsigned char *salt,
int saltlen, int iter, int saltlen, int iter,

8
wolfcrypt/src/logging.c
View File

@ -136,13 +136,6 @@ static struct log mynewt_log;
#endif /* DEBUG_WOLFSSL */ #endif /* DEBUG_WOLFSSL */
#ifdef DEBUG_VECTOR_REGISTER_ACCESS
THREAD_LS_T int wc_svr_count = 0;
THREAD_LS_T const char *wc_svr_last_file = NULL;
THREAD_LS_T int wc_svr_last_line = -1;
#endif
/* allow this to be set to NULL, so logs can be redirected to default output */ /* allow this to be set to NULL, so logs can be redirected to default output */
int wolfSSL_SetLoggingCb(wolfSSL_Logging_cb f) int wolfSSL_SetLoggingCb(wolfSSL_Logging_cb f)
{ {
@ -1546,4 +1539,3 @@ void WOLFSSL_ERROR_MSG(const char* msg)
} }
#endif /* DEBUG_WOLFSSL || WOLFSSL_NGINX || WOLFSSL_HAPROXY */ #endif /* DEBUG_WOLFSSL || WOLFSSL_NGINX || WOLFSSL_HAPROXY */

39
wolfcrypt/src/memory.c
View File

@ -31,7 +31,7 @@
#define WOLFSSL_NEED_LINUX_CURRENT #define WOLFSSL_NEED_LINUX_CURRENT
#endif #endif
#include <wolfssl/wolfcrypt/settings.h> #include <wolfssl/wolfcrypt/types.h>
/* /*
Possible memory options: Possible memory options:
@ -1231,7 +1231,6 @@ void* wolfSSL_Realloc(void *ptr, size_t size, void* heap, int type)
/* Example for user io pool, shared build may need definitions in lib proper */ /* Example for user io pool, shared build may need definitions in lib proper */
#include <wolfssl/wolfcrypt/types.h>
#include <stdlib.h> #include <stdlib.h>
#ifndef HAVE_THREAD_LS #ifndef HAVE_THREAD_LS
@ -1439,6 +1438,42 @@ void __attribute__((no_instrument_function))
} }
#endif #endif
#ifdef DEBUG_VECTOR_REGISTER_ACCESS
THREAD_LS_T int wc_svr_count = 0;
THREAD_LS_T const char *wc_svr_last_file = NULL;
THREAD_LS_T int wc_svr_last_line = -1;
THREAD_LS_T int wc_debug_vector_registers_retval =
WC_DEBUG_VECTOR_REGISTERS_RETVAL_INITVAL;
#ifdef DEBUG_VECTOR_REGISTER_ACCESS_FUZZING
WOLFSSL_LOCAL int SAVE_VECTOR_REGISTERS2_fuzzer(void) {
static THREAD_LS_T struct drand48_data wc_svr_fuzzing_state;
static THREAD_LS_T int wc_svr_fuzzing_seeded = 0;
long result;
if (wc_debug_vector_registers_retval)
return wc_debug_vector_registers_retval;
if (wc_svr_fuzzing_seeded == 0) {
long seed = WC_DEBUG_VECTOR_REGISTERS_FUZZING_SEED;
char *seed_envstr = getenv("WC_DEBUG_VECTOR_REGISTERS_FUZZING_SEED");
if (seed_envstr)
seed = strtol(seed_envstr, NULL, 0);
(void)srand48_r(seed, &wc_svr_fuzzing_state);
wc_svr_fuzzing_seeded = 1;
}
(void)lrand48_r(&wc_svr_fuzzing_state, &result);
if (result & 1)
return IO_FAILED_E;
else
return 0;
}
#endif
#endif
#ifdef WOLFSSL_LINUXKM #ifdef WOLFSSL_LINUXKM
#include "../../linuxkm/linuxkm_memory.c" #include "../../linuxkm/linuxkm_memory.c"
#endif #endif

4
wolfcrypt/src/pwdbased.c
View File

@ -559,6 +559,10 @@ int wc_PKCS12_PBKDF_ex(byte* output, const byte* passwd, int passLen,
#endif /* HAVE_PKCS12 */ #endif /* HAVE_PKCS12 */
#ifdef HAVE_SCRYPT #ifdef HAVE_SCRYPT
#ifdef NO_HMAC
#error scrypt requires HMAC
#endif
/* Rotate the 32-bit value a by b bits to the left. /* Rotate the 32-bit value a by b bits to the left.
* *
* a 32-bit value. * a 32-bit value.

478
wolfcrypt/test/test.c
View File

@ -574,7 +574,9 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t openssl_evpSig_test(void);
WOLFSSL_TEST_SUBROUTINE wc_test_ret_t pbkdf1_test(void); WOLFSSL_TEST_SUBROUTINE wc_test_ret_t pbkdf1_test(void);
WOLFSSL_TEST_SUBROUTINE wc_test_ret_t pkcs12_test(void); WOLFSSL_TEST_SUBROUTINE wc_test_ret_t pkcs12_test(void);
#if defined(HAVE_PBKDF2) && !defined(NO_SHA256) && !defined(NO_HMAC)
WOLFSSL_TEST_SUBROUTINE wc_test_ret_t pbkdf2_test(void); WOLFSSL_TEST_SUBROUTINE wc_test_ret_t pbkdf2_test(void);
#endif
WOLFSSL_TEST_SUBROUTINE wc_test_ret_t scrypt_test(void); WOLFSSL_TEST_SUBROUTINE wc_test_ret_t scrypt_test(void);
#ifdef HAVE_ECC #ifdef HAVE_ECC
WOLFSSL_TEST_SUBROUTINE wc_test_ret_t ecc_test(void); WOLFSSL_TEST_SUBROUTINE wc_test_ret_t ecc_test(void);
@ -9466,6 +9468,19 @@ static wc_test_ret_t aes_xts_128_test(void)
if (XMEMCMP(c2, buf, sizeof(c2))) if (XMEMCMP(c2, buf, sizeof(c2)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out); ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesXtsEncrypt(aes, buf, p2, sizeof(p2), i2, sizeof(i2));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &aes->aes.asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(c2, buf, sizeof(c2)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
XMEMSET(buf, 0, sizeof(buf)); XMEMSET(buf, 0, sizeof(buf));
wc_AesXtsFree(aes); wc_AesXtsFree(aes);
@ -9482,6 +9497,19 @@ static wc_test_ret_t aes_xts_128_test(void)
if (XMEMCMP(c1, buf, AES_BLOCK_SIZE)) if (XMEMCMP(c1, buf, AES_BLOCK_SIZE))
ERROR_OUT(WC_TEST_RET_ENC_NC, out); ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesXtsEncrypt(aes, buf, p1, sizeof(p1), i1, sizeof(i1));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &aes->aes.asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(c1, buf, AES_BLOCK_SIZE))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
/* partial block encryption test */ /* partial block encryption test */
XMEMSET(cipher, 0, sizeof(cipher)); XMEMSET(cipher, 0, sizeof(cipher));
ret = wc_AesXtsEncrypt(aes, cipher, pp, sizeof(pp), i1, sizeof(i1)); ret = wc_AesXtsEncrypt(aes, cipher, pp, sizeof(pp), i1, sizeof(i1));
@ -9492,6 +9520,21 @@ static wc_test_ret_t aes_xts_128_test(void)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out); ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(cp2, cipher, sizeof(cp2))) if (XMEMCMP(cp2, cipher, sizeof(cp2)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out); ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
XMEMSET(cipher, 0, sizeof(cipher));
ret = wc_AesXtsEncrypt(aes, cipher, pp, sizeof(pp), i1, sizeof(i1));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &aes->aes.asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(cp2, cipher, sizeof(cp2)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
wc_AesXtsFree(aes); wc_AesXtsFree(aes);
/* partial block decrypt test */ /* partial block decrypt test */
@ -9509,6 +9552,20 @@ static wc_test_ret_t aes_xts_128_test(void)
if (XMEMCMP(pp, buf, sizeof(pp))) if (XMEMCMP(pp, buf, sizeof(pp)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out); ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
XMEMSET(buf, 0, sizeof(buf));
ret = wc_AesXtsDecrypt(aes, buf, cipher, sizeof(pp), i1, sizeof(i1));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &aes->aes.asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(pp, buf, sizeof(pp)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
/* NIST decrypt test vector */ /* NIST decrypt test vector */
XMEMSET(buf, 0, sizeof(buf)); XMEMSET(buf, 0, sizeof(buf));
ret = wc_AesXtsDecrypt(aes, buf, c1, sizeof(c1), i1, sizeof(i1)); ret = wc_AesXtsDecrypt(aes, buf, c1, sizeof(c1), i1, sizeof(i1));
@ -9520,6 +9577,20 @@ static wc_test_ret_t aes_xts_128_test(void)
if (XMEMCMP(p1, buf, AES_BLOCK_SIZE)) if (XMEMCMP(p1, buf, AES_BLOCK_SIZE))
ERROR_OUT(WC_TEST_RET_ENC_NC, out); ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
XMEMSET(buf, 0, sizeof(buf));
ret = wc_AesXtsDecrypt(aes, buf, c1, sizeof(c1), i1, sizeof(i1));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &aes->aes.asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(p1, buf, AES_BLOCK_SIZE))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
/* fail case with decrypting using wrong key */ /* fail case with decrypting using wrong key */
XMEMSET(buf, 0, sizeof(buf)); XMEMSET(buf, 0, sizeof(buf));
ret = wc_AesXtsDecrypt(aes, buf, c2, sizeof(c2), i2, sizeof(i2)); ret = wc_AesXtsDecrypt(aes, buf, c2, sizeof(c2), i2, sizeof(i2));
@ -10368,17 +10439,83 @@ static wc_test_ret_t aesecb_test(void)
if (XMEMCMP(cipher, niCipher, AES_BLOCK_SIZE) != 0) if (XMEMCMP(cipher, niCipher, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out); ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
XMEMSET(cipher, 0, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesSetKey(enc, niKey, sizeof(niKey), cipher, AES_ENCRYPTION);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesEcbEncrypt(enc, cipher, niPlain, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
if (XMEMCMP(cipher, niCipher, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
XMEMSET(cipher, 0, AES_BLOCK_SIZE);
ret = wc_AesEcbEncrypt(enc, cipher, niPlain, AES_BLOCK_SIZE);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
if (XMEMCMP(cipher, niCipher, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
XMEMSET(cipher, 0, AES_BLOCK_SIZE);
ret = wc_AesSetKey(enc, niKey, sizeof(niKey), cipher, AES_ENCRYPTION);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesEcbEncrypt(enc, cipher, niPlain, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
if (XMEMCMP(cipher, niCipher, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
XMEMSET(plain, 0, AES_BLOCK_SIZE); XMEMSET(plain, 0, AES_BLOCK_SIZE);
ret = wc_AesSetKey(dec, niKey, sizeof(niKey), plain, AES_DECRYPTION); ret = wc_AesSetKey(dec, niKey, sizeof(niKey), plain, AES_DECRYPTION);
if (ret != 0) if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out); ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (wc_AesEcbDecrypt(dec, plain, niCipher, AES_BLOCK_SIZE) != 0) if (wc_AesEcbDecrypt(dec, plain, niCipher, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out); ERROR_OUT(WC_TEST_RET_ENC_NC, out);
wc_AesEcbDecrypt(dec, plain, niCipher, AES_BLOCK_SIZE);
if (XMEMCMP(plain, niPlain, AES_BLOCK_SIZE) != 0) if (XMEMCMP(plain, niPlain, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out); ERROR_OUT(WC_TEST_RET_ENC_NC, out);
}
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
XMEMSET(plain, 0, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesSetKey(dec, niKey, sizeof(niKey), plain, AES_DECRYPTION);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesEcbDecrypt(dec, plain, niCipher, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
if (XMEMCMP(plain, niPlain, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
XMEMSET(plain, 0, AES_BLOCK_SIZE);
ret = wc_AesEcbDecrypt(dec, plain, niCipher, AES_BLOCK_SIZE);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
if (XMEMCMP(plain, niPlain, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
XMEMSET(plain, 0, AES_BLOCK_SIZE);
ret = wc_AesSetKey(dec, niKey, sizeof(niKey), plain, AES_DECRYPTION);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesEcbDecrypt(dec, plain, niCipher, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
if (XMEMCMP(plain, niPlain, AES_BLOCK_SIZE) != 0)
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
}
out: out:
#if defined(WOLFSSL_SMALL_STACK) && !defined(WOLFSSL_NO_MALLOC) #if defined(WOLFSSL_SMALL_STACK) && !defined(WOLFSSL_NO_MALLOC)
@ -11009,6 +11146,86 @@ static wc_test_ret_t aesctr_test(Aes* enc, Aes* dec, byte* cipher, byte* plain)
#endif #endif
} }
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
for (i = 0; i < AES_CTR_TEST_LEN; i++) {
if (testVec[i].key != NULL) {
ret = wc_AesSetKeyDirect(enc, testVec[i].key, testVec[i].keySz,
testVec[i].iv, AES_ENCRYPTION);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
/* Ctr only uses encrypt, even on key setup */
ret = wc_AesSetKeyDirect(dec, testVec[i].key, testVec[i].keySz,
testVec[i].iv, AES_ENCRYPTION);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
}
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesCtrEncrypt(enc, cipher, testVec[i].plain, testVec[i].len);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesCtrEncrypt(dec, plain, cipher, testVec[i].len);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
if (XMEMCMP(plain, ctrPlain, testVec[i].len)) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
#if !(FIPS_VERSION_EQ(2,0) && defined(WOLFSSL_ARMASM))
if (XMEMCMP(cipher, testVec[i].cipher, testVec[i].len)) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
#endif
}
for (i = 0; i < AES_CTR_TEST_LEN; i++) {
if (testVec[i].key != NULL) {
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesSetKeyDirect(enc, testVec[i].key, testVec[i].keySz,
testVec[i].iv, AES_ENCRYPTION);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
/* Ctr only uses encrypt, even on key setup */
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesSetKeyDirect(dec, testVec[i].key, testVec[i].keySz,
testVec[i].iv, AES_ENCRYPTION);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
}
ret = wc_AesCtrEncrypt(enc, cipher, testVec[i].plain, testVec[i].len);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
ret = wc_AesCtrEncrypt(dec, plain, cipher, testVec[i].len);
if (ret != 0) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
if (XMEMCMP(plain, ctrPlain, testVec[i].len)) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
#if !(FIPS_VERSION_EQ(2,0) && defined(WOLFSSL_ARMASM))
if (XMEMCMP(cipher, testVec[i].cipher, testVec[i].len)) {
ERROR_OUT(WC_TEST_RET_ENC_I(i), out);
}
#endif
}
#endif /* DEBUG_VECTOR_REGISTER_ACCESS && WC_AES_C_DYNAMIC_FALLBACK */
out: out:
return ret; return ret;
} }
@ -11262,6 +11479,57 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t aes_test(void)
break; break;
} }
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
/* Iterate from one AES_BLOCK_SIZE of bigMsg through the whole
* message by AES_BLOCK_SIZE for each size of AES key. */
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
for (keySz = 16; keySz <= 32; keySz += 8) {
for (msgSz = AES_BLOCK_SIZE;
msgSz <= sizeof(bigMsg);
msgSz += AES_BLOCK_SIZE) {
XMEMSET(bigCipher, 0, sizeof(bigMsg));
XMEMSET(bigPlain, 0, sizeof(bigMsg));
ret = wc_AesSetKey(enc, bigKey, keySz, iv, AES_ENCRYPTION);
if (ret != 0) {
ret = WC_TEST_RET_ENC_EC(ret);
break;
}
ret = wc_AesSetKey(dec, bigKey, keySz, iv, AES_DECRYPTION);
if (ret != 0) {
ret = WC_TEST_RET_ENC_EC(ret);
break;
}
ret = wc_AesCbcEncrypt(enc, bigCipher, bigMsg, msgSz);
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &enc->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
if (ret != 0) {
ret = WC_TEST_RET_ENC_EC(ret);
break;
}
ret = wc_AesCbcDecrypt(dec, bigPlain, bigCipher, msgSz);
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &dec->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
if (ret != 0) {
ret = WC_TEST_RET_ENC_EC(ret);
break;
}
if (XMEMCMP(bigPlain, bigMsg, msgSz)) {
ret = WC_TEST_RET_ENC_NC;
break;
}
}
if (ret != 0)
break;
}
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
#endif /* DEBUG_VECTOR_REGISTER_ACCESS && WC_AES_C_DYNAMIC_FALLBACK */
#if defined(WOLFSSL_SMALL_STACK) && !defined(WOLFSSL_NO_MALLOC) #if defined(WOLFSSL_SMALL_STACK) && !defined(WOLFSSL_NO_MALLOC)
XFREE(bigCipher, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER); XFREE(bigCipher, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
XFREE(bigPlain, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER); XFREE(bigPlain, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
@ -11591,6 +11859,9 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t aes192_test(void)
#endif #endif
if (ret != 0) if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out); ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(cipher, verify, (int) sizeof(cipher)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#ifdef HAVE_AES_DECRYPT #ifdef HAVE_AES_DECRYPT
XMEMSET(plain, 0, AES_BLOCK_SIZE); XMEMSET(plain, 0, AES_BLOCK_SIZE);
ret = wc_AesCbcDecrypt(dec, plain, cipher, (int) sizeof(cipher)); ret = wc_AesCbcDecrypt(dec, plain, cipher, (int) sizeof(cipher));
@ -11604,9 +11875,6 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t aes192_test(void)
} }
#endif #endif
if (XMEMCMP(cipher, verify, (int) sizeof(cipher)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
wc_AesFree(enc); wc_AesFree(enc);
#ifdef HAVE_AES_DECRYPT #ifdef HAVE_AES_DECRYPT
wc_AesFree(dec); wc_AesFree(dec);
@ -11732,6 +12000,93 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t aes256_test(void)
wc_AesFree(enc); wc_AesFree(enc);
#ifdef HAVE_AES_DECRYPT #ifdef HAVE_AES_DECRYPT
wc_AesFree(dec); wc_AesFree(dec);
#endif
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
ret = wc_AesSetKey(enc, key, keySz, iv, AES_ENCRYPTION);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
#ifdef HAVE_AES_DECRYPT
ret = wc_AesSetKey(dec, key, keySz, iv, AES_DECRYPTION);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
#endif
XMEMSET(cipher, 0, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesCbcEncrypt(enc, cipher, msg, (int) sizeof(msg));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &enc->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
#ifdef HAVE_AES_DECRYPT
XMEMSET(plain, 0, AES_BLOCK_SIZE);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesCbcDecrypt(dec, plain, cipher, (int) sizeof(cipher));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &dec->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(plain, msg, (int) sizeof(plain))) {
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
}
#endif
#ifndef HAVE_RENESAS_SYNC
if (XMEMCMP(cipher, verify, (int) sizeof(cipher)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
wc_AesFree(enc);
#ifdef HAVE_AES_DECRYPT
wc_AesFree(dec);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesSetKey(enc, key, keySz, iv, AES_ENCRYPTION);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
#ifdef HAVE_AES_DECRYPT
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesSetKey(dec, key, keySz, iv, AES_DECRYPTION);
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
#endif
XMEMSET(cipher, 0, AES_BLOCK_SIZE);
ret = wc_AesCbcEncrypt(enc, cipher, msg, (int) sizeof(msg));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &enc->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
#ifdef HAVE_AES_DECRYPT
XMEMSET(plain, 0, AES_BLOCK_SIZE);
ret = wc_AesCbcDecrypt(dec, plain, cipher, (int) sizeof(cipher));
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &dec->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (XMEMCMP(plain, msg, (int) sizeof(plain))) {
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
}
#endif
#ifndef HAVE_RENESAS_SYNC
if (XMEMCMP(cipher, verify, (int) sizeof(cipher)))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
wc_AesFree(enc);
#ifdef HAVE_AES_DECRYPT
wc_AesFree(dec);
#endif
#endif #endif
out: out:
@ -11800,7 +12155,6 @@ static wc_test_ret_t aesgcm_default_test_helper(byte* key, int keySz, byte* iv,
/* AES-GCM encrypt and decrypt both use AES encrypt internally */ /* AES-GCM encrypt and decrypt both use AES encrypt internally */
ret = wc_AesGcmEncrypt(enc, resultC, plain, plainSz, iv, ivSz, ret = wc_AesGcmEncrypt(enc, resultC, plain, plainSz, iv, ivSz,
resultT, tagSz, aad, aadSz); resultT, tagSz, aad, aadSz);
#if defined(WOLFSSL_ASYNC_CRYPT) #if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &enc->asyncDev, WC_ASYNC_FLAG_NONE); ret = wc_AsyncWait(ret, &enc->asyncDev, WC_ASYNC_FLAG_NONE);
#endif #endif
@ -11813,6 +12167,23 @@ static wc_test_ret_t aesgcm_default_test_helper(byte* key, int keySz, byte* iv,
if (XMEMCMP(tag, resultT, tagSz)) if (XMEMCMP(tag, resultT, tagSz))
ERROR_OUT(WC_TEST_RET_ENC_NC, out); ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesGcmEncrypt(enc, resultC, plain, plainSz, iv, ivSz,
resultT, tagSz, aad, aadSz);
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &enc->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (cipher != NULL) {
if (XMEMCMP(cipher, resultC, cipherSz))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
}
if (XMEMCMP(tag, resultT, tagSz))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
#endif
#ifdef HAVE_AES_DECRYPT #ifdef HAVE_AES_DECRYPT
ret = wc_AesGcmSetKey(dec, key, keySz); ret = wc_AesGcmSetKey(dec, key, keySz);
@ -11831,6 +12202,22 @@ static wc_test_ret_t aesgcm_default_test_helper(byte* key, int keySz, byte* iv,
ERROR_OUT(WC_TEST_RET_ENC_NC, out); ERROR_OUT(WC_TEST_RET_ENC_NC, out);
} }
#if defined(DEBUG_VECTOR_REGISTER_ACCESS) && defined(WC_AES_C_DYNAMIC_FALLBACK)
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(SYSLIB_FAILED_E);
ret = wc_AesGcmDecrypt(dec, resultP, resultC, cipherSz,
iv, ivSz, resultT, tagSz, aad, aadSz);
#if defined(WOLFSSL_ASYNC_CRYPT)
ret = wc_AsyncWait(ret, &dec->asyncDev, WC_ASYNC_FLAG_NONE);
#endif
WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(0);
if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
if (plain != NULL) {
if (XMEMCMP(plain, resultP, plainSz))
ERROR_OUT(WC_TEST_RET_ENC_NC, out);
}
#endif
#endif /* HAVE_AES_DECRYPT */ #endif /* HAVE_AES_DECRYPT */
ret = 0; ret = 0;
@ -20202,6 +20589,7 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
word32 bytes; word32 bytes;
word32 idx = 0; word32 idx = 0;
WC_RNG rng; WC_RNG rng;
int rng_inited = 0;
wc_Sha sha; wc_Sha sha;
byte hash[WC_SHA_DIGEST_SIZE]; byte hash[WC_SHA_DIGEST_SIZE];
byte signature[40]; byte signature[40];
@ -20209,6 +20597,11 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
byte* der = 0; byte* der = 0;
#endif #endif
#define DSA_TEST_TMP_SIZE 1024 #define DSA_TEST_TMP_SIZE 1024
int key_inited = 0;
int derIn_inited = 0;
#ifdef WOLFSSL_KEY_GEN
int genKey_inited = 0;
#endif
#if defined(WOLFSSL_SMALL_STACK) && !defined(WOLFSSL_NO_MALLOC) #if defined(WOLFSSL_SMALL_STACK) && !defined(WOLFSSL_NO_MALLOC)
byte *tmp = (byte *)XMALLOC(DSA_TEST_TMP_SIZE, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER); byte *tmp = (byte *)XMALLOC(DSA_TEST_TMP_SIZE, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
DsaKey *key = (DsaKey *)XMALLOC(sizeof *key, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER); DsaKey *key = (DsaKey *)XMALLOC(sizeof *key, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
@ -20216,6 +20609,7 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
DsaKey *derIn = (DsaKey *)XMALLOC(sizeof *derIn, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER); DsaKey *derIn = (DsaKey *)XMALLOC(sizeof *derIn, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
DsaKey *genKey = (DsaKey *)XMALLOC(sizeof *genKey, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER); DsaKey *genKey = (DsaKey *)XMALLOC(sizeof *genKey, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
#endif #endif
if ((tmp == NULL) || if ((tmp == NULL) ||
(key == NULL) (key == NULL)
#ifdef WOLFSSL_KEY_GEN #ifdef WOLFSSL_KEY_GEN
@ -20264,6 +20658,7 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
ret = wc_InitDsaKey(key); ret = wc_InitDsaKey(key);
if (ret != 0) if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out); ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
key_inited = 1;
ret = wc_DsaPrivateKeyDecode(tmp, &idx, key, bytes); ret = wc_DsaPrivateKeyDecode(tmp, &idx, key, bytes);
if (ret != 0) if (ret != 0)
@ -20276,6 +20671,7 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
#endif #endif
if (ret != 0) if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out); ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
rng_inited = 1;
ret = wc_DsaSign(hash, signature, key, &rng); ret = wc_DsaSign(hash, signature, key, &rng);
if (ret != 0) if (ret != 0)
@ -20287,8 +20683,6 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
if (answer != 1) if (answer != 1)
ERROR_OUT(WC_TEST_RET_ENC_NC, out); ERROR_OUT(WC_TEST_RET_ENC_NC, out);
wc_FreeDsaKey(key);
#ifdef WOLFSSL_KEY_GEN #ifdef WOLFSSL_KEY_GEN
{ {
int derSz = 0; int derSz = 0;
@ -20296,56 +20690,39 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
ret = wc_InitDsaKey(genKey); ret = wc_InitDsaKey(genKey);
if (ret != 0) if (ret != 0)
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out); ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
genKey_inited = 1;
ret = wc_MakeDsaParameters(&rng, 1024, genKey); ret = wc_MakeDsaParameters(&rng, 1024, genKey);
if (ret != 0) { if (ret != 0)
wc_FreeDsaKey(genKey);
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out); ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
}
ret = wc_MakeDsaKey(&rng, genKey); ret = wc_MakeDsaKey(&rng, genKey);
if (ret != 0) { if (ret != 0)
wc_FreeDsaKey(genKey);
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out); ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
}
der = (byte*)XMALLOC(FOURK_BUF, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER); der = (byte*)XMALLOC(FOURK_BUF, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
if (der == NULL) { if (der == NULL)
wc_FreeDsaKey(genKey);
ERROR_OUT(WC_TEST_RET_ENC_NC, out); ERROR_OUT(WC_TEST_RET_ENC_NC, out);
}
derSz = wc_DsaKeyToDer(genKey, der, FOURK_BUF); derSz = wc_DsaKeyToDer(genKey, der, FOURK_BUF);
if (derSz < 0) { if (derSz < 0)
XFREE(der, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
ERROR_OUT(WC_TEST_RET_ENC_EC(derSz), out); ERROR_OUT(WC_TEST_RET_ENC_EC(derSz), out);
}
ret = SaveDerAndPem(der, derSz, keyDerFile, keyPemFile, ret = SaveDerAndPem(der, derSz, keyDerFile, keyPemFile,
DSA_PRIVATEKEY_TYPE); DSA_PRIVATEKEY_TYPE);
if (ret != 0) { if (ret != 0)
XFREE(der, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
wc_FreeDsaKey(genKey);
goto out; goto out;
}
ret = wc_InitDsaKey(derIn); ret = wc_InitDsaKey(derIn);
if (ret != 0) { if (ret != 0)
XFREE(der, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
wc_FreeDsaKey(genKey);
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out); ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
} derIn_inited = 1;
idx = 0; idx = 0;
ret = wc_DsaPrivateKeyDecode(der, &idx, derIn, derSz); ret = wc_DsaPrivateKeyDecode(der, &idx, derIn, derSz);
if (ret != 0) { if (ret != 0)
XFREE(der, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
wc_FreeDsaKey(derIn);
wc_FreeDsaKey(genKey);
ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out); ERROR_OUT(WC_TEST_RET_ENC_EC(ret), out);
} }
}
#endif /* WOLFSSL_KEY_GEN */ #endif /* WOLFSSL_KEY_GEN */
out: out:
@ -20368,15 +20745,20 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
#if defined(WOLFSSL_SMALL_STACK) && !defined(WOLFSSL_NO_MALLOC) #if defined(WOLFSSL_SMALL_STACK) && !defined(WOLFSSL_NO_MALLOC)
if (tmp) if (tmp)
XFREE(tmp, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER); XFREE(tmp, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
if (key) if (key) {
if (key_inited)
wc_FreeDsaKey(key);
XFREE(key, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER); XFREE(key, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
}
#ifdef WOLFSSL_KEY_GEN #ifdef WOLFSSL_KEY_GEN
if (derIn) { if (derIn) {
wc_FreeDsaKey(derIn); if (derIn_inited)
wc_FreeDsaKey(derIn);
XFREE(derIn, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER); XFREE(derIn, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
} }
if (genKey) { if (genKey) {
wc_FreeDsaKey(genKey); if (genKey_inited)
wc_FreeDsaKey(genKey);
XFREE(genKey, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER); XFREE(genKey, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
} }
#endif #endif
@ -20384,13 +20766,18 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t dsa_test(void)
#else /* !WOLFSSL_SMALL_STACK || WOLFSSL_NO_MALLOC */ #else /* !WOLFSSL_SMALL_STACK || WOLFSSL_NO_MALLOC */
#ifdef WOLFSSL_KEY_GEN #ifdef WOLFSSL_KEY_GEN
wc_FreeDsaKey(derIn); if (key_inited)
wc_FreeDsaKey(genKey); wc_FreeDsaKey(key);
if (derIn_inited)
wc_FreeDsaKey(derIn);
if (genKey_inited)
wc_FreeDsaKey(genKey);
#endif #endif
#endif #endif
wc_FreeRng(&rng); if (rng_inited)
wc_FreeRng(&rng);
return ret; return ret;
} }
@ -41352,7 +41739,7 @@ static wc_test_ret_t pkcs7authenveloped_run_vectors(byte* rsaCert, word32 rsaCer
#endif #endif
#if !defined(NO_AES) && defined(WOLFSSL_AES_256) && defined(HAVE_ECC) && \ #if !defined(NO_AES) && defined(WOLFSSL_AES_256) && defined(HAVE_ECC) && \
defined(WOLFSSL_SHA512) defined(WOLFSSL_SHA512) && defined(HAVE_AESGCM)
WOLFSSL_SMALL_STACK_STATIC const byte optionalUkm[] = { WOLFSSL_SMALL_STACK_STATIC const byte optionalUkm[] = {
0x00,0x01,0x02,0x03,0x04,0x05,0x06,0x07 0x00,0x01,0x02,0x03,0x04,0x05,0x06,0x07
}; };
@ -41455,7 +41842,12 @@ static wc_test_ret_t pkcs7authenveloped_run_vectors(byte* rsaCert, word32 rsaCer
NULL, 0, 0, NULL, 0, NULL, 0, 0, 0, 0, 0, 0, 0, 0, NULL, 0, 0, NULL, 0, NULL, 0, 0, 0, 0, 0, 0, 0, 0,
"pkcs7authEnvelopedDataAES256GCM_IANDS.der"); "pkcs7authEnvelopedDataAES256GCM_IANDS.der");
#endif #endif
#endif /* NO_AES */ #else /* NO_AES || !HAVE_AESGCM */
(void)rsaCert;
(void)rsaCertSz;
(void)rsaPrivKey;
(void)rsaPrivKeySz;
#endif /* NO_AES || !HAVE_AESGCM */
#endif #endif
/* key agreement key encryption technique*/ /* key agreement key encryption technique*/
@ -48972,7 +49364,7 @@ WOLFSSL_TEST_SUBROUTINE wc_test_ret_t cryptocb_test(void)
#endif #endif
#endif #endif
#ifndef NO_PWDBASED #ifndef NO_PWDBASED
#if defined(HAVE_PBKDF2) && !defined(NO_SHA256) #if defined(HAVE_PBKDF2) && !defined(NO_SHA256) && !defined(NO_HMAC)
if (ret == 0) if (ret == 0)
ret = pbkdf2_test(); ret = pbkdf2_test();
#endif #endif

4
wolfssl/wolfcrypt/aes.h
View File

@ -243,13 +243,15 @@ enum {
#endif #endif
struct Aes { struct Aes {
/* AESNI needs key first, rounds 2nd, not sure why yet */
ALIGN16 word32 key[60]; ALIGN16 word32 key[60];
#ifdef WC_AES_BITSLICED #ifdef WC_AES_BITSLICED
/* Extra key schedule space required for bit-slicing technique. */ /* Extra key schedule space required for bit-slicing technique. */
ALIGN16 bs_word bs_key[15 * AES_BLOCK_SIZE * BS_WORD_SIZE]; ALIGN16 bs_word bs_key[15 * AES_BLOCK_SIZE * BS_WORD_SIZE];
#endif #endif
word32 rounds; word32 rounds;
#ifdef WC_AES_C_DYNAMIC_FALLBACK
word32 key_C_fallback[60];
#endif
int keylen; int keylen;
ALIGN16 word32 reg[AES_BLOCK_SIZE / sizeof(word32)]; /* for CBC mode */ ALIGN16 word32 reg[AES_BLOCK_SIZE / sizeof(word32)]; /* for CBC mode */

2
wolfssl/wolfcrypt/cryptocb.h
View File

@ -89,7 +89,6 @@ typedef struct wc_CryptoInfo {
#if HAVE_ANONYMOUS_INLINE_AGGREGATES #if HAVE_ANONYMOUS_INLINE_AGGREGATES
union { union {
#endif #endif
#if !defined(NO_RSA) || defined(HAVE_ECC)
struct { struct {
int type; /* enum wc_PkType */ int type; /* enum wc_PkType */
#if HAVE_ANONYMOUS_INLINE_AGGREGATES #if HAVE_ANONYMOUS_INLINE_AGGREGATES
@ -206,7 +205,6 @@ typedef struct wc_CryptoInfo {
}; };
#endif #endif
} pk; } pk;
#endif /* !NO_RSA || HAVE_ECC */
#if !defined(NO_AES) || !defined(NO_DES3) #if !defined(NO_AES) || !defined(NO_DES3)
struct { struct {
int type; /* enum wc_CipherType */ int type; /* enum wc_CipherType */

14
wolfssl/wolfcrypt/error-crypt.h
View File

@ -238,19 +238,19 @@ enum {
BAD_LENGTH_E = -279, /* Value of length parameter is invalid. */ BAD_LENGTH_E = -279, /* Value of length parameter is invalid. */
ECDSA_KAT_FIPS_E = -280, /* ECDSA KAT failure */ ECDSA_KAT_FIPS_E = -280, /* ECDSA KAT failure */
RSA_PAT_FIPS_E = -281, /* RSA Pairwise failure */ RSA_PAT_FIPS_E = -281, /* RSA Pairwise failure */
KDF_TLS12_KAT_FIPS_E = -282, /* TLS12 KDF KAT failure */ KDF_TLS12_KAT_FIPS_E = -282, /* TLS12 KDF KAT failure */
KDF_TLS13_KAT_FIPS_E = -283, /* TLS13 KDF KAT failure */ KDF_TLS13_KAT_FIPS_E = -283, /* TLS13 KDF KAT failure */
KDF_SSH_KAT_FIPS_E = -284, /* SSH KDF KAT failure */ KDF_SSH_KAT_FIPS_E = -284, /* SSH KDF KAT failure */
DHE_PCT_E = -285, /* DHE Pairwise Consistency Test failure */ DHE_PCT_E = -285, /* DHE Pairwise Consistency Test failure */
ECC_PCT_E = -286, /* ECDHE Pairwise Consistency Test failure */ ECC_PCT_E = -286, /* ECDHE Pairwise Consistency Test failure */
FIPS_PRIVATE_KEY_LOCKED_E = -287, /* Cannot export private key. */ FIPS_PRIVATE_KEY_LOCKED_E = -287, /* Cannot export private key. */
PROTOCOLCB_UNAVAILABLE = -288, /* Protocol callback unavailable */ PROTOCOLCB_UNAVAILABLE = -288, /* Protocol callback unavailable */
AES_SIV_AUTH_E = -289, /* AES-SIV authentication failed */ AES_SIV_AUTH_E = -289, /* AES-SIV authentication failed */
NO_VALID_DEVID = -290, /* no valid device ID */ NO_VALID_DEVID = -290, /* no valid device ID */
IO_FAILED_E = -291, /* Input/output failure */ IO_FAILED_E = -291, /* Input/output failure */
SYSLIB_FAILED_E = -292, /* System/library call failed */ SYSLIB_FAILED_E = -292, /* System/library call failed */
USE_HW_PSK = -293, /* Callback return to indicate HW has PSK */ USE_HW_PSK = -293, /* Callback return to indicate HW has PSK */
ENTROPY_RT_E = -294, /* Entropy Repetition Test failed */ ENTROPY_RT_E = -294, /* Entropy Repetition Test failed */
ENTROPY_APT_E = -295, /* Entropy Adaptive Proportion Test failed */ ENTROPY_APT_E = -295, /* Entropy Adaptive Proportion Test failed */

166
wolfssl/wolfcrypt/memory.h
View File

@ -251,9 +251,173 @@ WOLFSSL_LOCAL void wc_MemZero_Add(const char* name, const void* addr,
WOLFSSL_LOCAL void wc_MemZero_Check(void* addr, size_t len); WOLFSSL_LOCAL void wc_MemZero_Check(void* addr, size_t len);
#endif #endif
#ifdef DEBUG_VECTOR_REGISTER_ACCESS
WOLFSSL_API extern THREAD_LS_T int wc_svr_count;
WOLFSSL_API extern THREAD_LS_T const char *wc_svr_last_file;
WOLFSSL_API extern THREAD_LS_T int wc_svr_last_line;
#ifdef DEBUG_VECTOR_REGISTERS_ABORT_ON_FAIL
#define DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE abort();
#elif defined(DEBUG_VECTOR_REGISTERS_EXIT_ON_FAIL)
#define DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE exit(1);
#else
#define DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE
#endif
#define SAVE_VECTOR_REGISTERS(fail_clause) { \
int _svr_ret = wc_debug_vector_registers_retval; \
if (_svr_ret != 0) { fail_clause } \
++wc_svr_count; \
if (wc_svr_count > 5) { \
fprintf(stderr, \
("%s @ L%d : incr : " \
"wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
wc_svr_last_file = __FILE__; \
wc_svr_last_line = __LINE__; \
}
WOLFSSL_API extern THREAD_LS_T int wc_debug_vector_registers_retval;
#ifndef WC_DEBUG_VECTOR_REGISTERS_RETVAL_INITVAL
#define WC_DEBUG_VECTOR_REGISTERS_RETVAL_INITVAL 0
#endif
#define WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(x) do { \
if (((x) != 0) && (wc_svr_count > 0)) { \
fprintf(stderr, \
("%s @ L%d : incr : " \
"wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
wc_debug_vector_registers_retval = (x); \
} while (0)
#ifdef DEBUG_VECTOR_REGISTER_ACCESS_FUZZING
#ifndef WC_DEBUG_VECTOR_REGISTERS_FUZZING_SEED
#define WC_DEBUG_VECTOR_REGISTERS_FUZZING_SEED 0
#endif
WOLFSSL_LOCAL int SAVE_VECTOR_REGISTERS2_fuzzer(void);
#define SAVE_VECTOR_REGISTERS2(...) ({ \
int _svr2_val = SAVE_VECTOR_REGISTERS2_fuzzer(); \
if (_svr2_val == 0) { \
++wc_svr_count; \
if (wc_svr_count > 5) { \
fprintf(stderr, \
("%s @ L%d : incr : " \
"wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
wc_svr_last_file = __FILE__; \
wc_svr_last_line = __LINE__; \
_svr2_val = 0; \
} \
_svr2_val; \
})
#else
#define SAVE_VECTOR_REGISTERS2(...) ({ \
int _svr2_val; \
if (wc_debug_vector_registers_retval != 0) { \
if (wc_svr_count > 0) { \
fprintf(stderr, \
("%s @ L%d : incr : " \
"wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
_svr2_val = wc_debug_vector_registers_retval; \
} else { \
++wc_svr_count; \
if (wc_svr_count > 5) { \
fprintf(stderr, \
("%s @ L%d : incr : " \
"wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
wc_svr_last_file = __FILE__; \
wc_svr_last_line = __LINE__; \
_svr2_val = 0; \
} \
_svr2_val; \
})
#endif
#define ASSERT_SAVED_VECTOR_REGISTERS(fail_clause) do { \
if (wc_svr_count <= 0) { \
fprintf(stderr, \
("ASSERT_SAVED_VECTOR_REGISTERS : %s @ L%d : " \
"wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
{ fail_clause } \
} \
} while (0)
#define ASSERT_RESTORED_VECTOR_REGISTERS(fail_clause) do { \
if (wc_svr_count != 0) { \
fprintf(stderr, \
("ASSERT_RESTORED_VECTOR_REGISTERS : %s @ L%d" \
" : wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
{ fail_clause } \
} \
} while (0)
#define RESTORE_VECTOR_REGISTERS(...) do { \
--wc_svr_count; \
if ((wc_svr_count > 4) || (wc_svr_count < 0)) { \
fprintf(stderr, \
("%s @ L%d : decr : " \
"wc_svr_count %d (last op %s L%d)\n"), \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
wc_svr_last_file = __FILE__; \
wc_svr_last_line = __LINE__; \
} while(0)
#endif
#ifdef __cplusplus #ifdef __cplusplus
} /* extern "C" */ } /* extern "C" */
#endif #endif
#endif /* WOLFSSL_MEMORY_H */ #endif /* WOLFSSL_MEMORY_H */

100
wolfssl/wolfcrypt/types.h
View File

@ -1566,90 +1566,24 @@ typedef struct w64wrapper {
#define PRAGMA_DIAG_POP /* null expansion */ #define PRAGMA_DIAG_POP /* null expansion */
#endif #endif
#ifdef DEBUG_VECTOR_REGISTER_ACCESS #ifndef SAVE_VECTOR_REGISTERS
WOLFSSL_API extern THREAD_LS_T int wc_svr_count; #define SAVE_VECTOR_REGISTERS(...) WC_DO_NOTHING
WOLFSSL_API extern THREAD_LS_T const char *wc_svr_last_file; #endif
WOLFSSL_API extern THREAD_LS_T int wc_svr_last_line; #ifndef SAVE_VECTOR_REGISTERS2
#define SAVE_VECTOR_REGISTERS2() 0
#ifdef DEBUG_VECTOR_REGISTERS_ABORT_ON_FAIL #endif
#define DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE abort(); #ifndef WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL
#elif defined(DEBUG_VECTOR_REGISTERS_EXIT_ON_FAIL) #define WC_DEBUG_SET_VECTOR_REGISTERS_RETVAL(x) WC_DO_NOTHING
#define DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE exit(1); #endif
#else #ifndef ASSERT_SAVED_VECTOR_REGISTERS
#define DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE #define ASSERT_SAVED_VECTOR_REGISTERS(...) WC_DO_NOTHING
#endif #endif
#ifndef ASSERT_RESTORED_VECTOR_REGISTERS
#define SAVE_VECTOR_REGISTERS(...) { \ #define ASSERT_RESTORED_VECTOR_REGISTERS(...) WC_DO_NOTHING
++wc_svr_count; \ #endif
if (wc_svr_count > 5) { \ #ifndef RESTORE_VECTOR_REGISTERS
fprintf(stderr, \ #define RESTORE_VECTOR_REGISTERS() WC_DO_NOTHING
"%s @ L%d : incr : wc_svr_count %d (last op %s L%d)\n", \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
wc_svr_last_file = __FILE__; \
wc_svr_last_line = __LINE__; \
}
#define ASSERT_SAVED_VECTOR_REGISTERS(fail_clause) { \
if (wc_svr_count <= 0) { \
fprintf(stderr, \
"ASSERT_SAVED_VECTOR_REGISTERS : %s @ L%d : wc_svr_count %d (last op %s L%d)\n", \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
{ fail_clause } \
} \
}
#define ASSERT_RESTORED_VECTOR_REGISTERS(fail_clause) { \
if (wc_svr_count != 0) { \
fprintf(stderr, \
"ASSERT_RESTORED_VECTOR_REGISTERS : %s @ L%d : wc_svr_count %d (last op %s L%d)\n", \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
{ fail_clause } \
} \
}
#define RESTORE_VECTOR_REGISTERS(...) { \
--wc_svr_count; \
if ((wc_svr_count > 4) || (wc_svr_count < 0)) { \
fprintf(stderr, \
"%s @ L%d : decr : wc_svr_count %d (last op %s L%d)\n", \
__FILE__, \
__LINE__, \
wc_svr_count, \
wc_svr_last_file, \
wc_svr_last_line); \
DEBUG_VECTOR_REGISTERS_EXTRA_FAIL_CLAUSE \
} \
wc_svr_last_file = __FILE__; \
wc_svr_last_line = __LINE__; \
}
#else
#ifndef SAVE_VECTOR_REGISTERS
#define SAVE_VECTOR_REGISTERS(...) WC_DO_NOTHING
#endif
#ifndef ASSERT_SAVED_VECTOR_REGISTERS
#define ASSERT_SAVED_VECTOR_REGISTERS(...) WC_DO_NOTHING
#endif
#ifndef ASSERT_RESTORED_VECTOR_REGISTERS
#define ASSERT_RESTORED_VECTOR_REGISTERS(...) WC_DO_NOTHING
#endif
#ifndef RESTORE_VECTOR_REGISTERS
#define RESTORE_VECTOR_REGISTERS() WC_DO_NOTHING
#endif
#endif #endif
#if FIPS_VERSION_GE(5,1) #if FIPS_VERSION_GE(5,1)
#define WC_SPKRE_F(x,y) wolfCrypt_SetPrivateKeyReadEnable_fips((x),(y)) #define WC_SPKRE_F(x,y) wolfCrypt_SetPrivateKeyReadEnable_fips((x),(y))