ASN: improve handling of ASN.1 parsing/encoding

ToTraditionalInline_ex2 original ASN code:
  - Now return 0 when no OCTECT_STRING data found.
  - Change callers to accept 0 as a valid returnb value.

SizeASN_Items:
  - Change encoded size to word32 as won't be negative.
- Change callers to supply a pointer to a word32 instead of integer.
Fix casting due to change of parameter type.

ASN_LEN_ENC_LEN: Function to calculate the length of the encoded ASN.1
length.

GetLength_ex:
  - Change minLen to word32
- Change length to word32 and change negative check appropriately for
different type.

GetASNHeader_ex:
  - If not checking lengths in GetLength_ex, check it here.
DecodeObjectId:
  - Ensure no overflow in calculation.

_RsaPrivateKeyDecode (original)
  - Clear RSA integers on failure (will be done in free anyway).

wc_CreatePKCS8Key (original):
  - safe check of overflow.

DecryptContent (templare):
- Parse will fail if OID not recognized, and recognized OIDs are 9/10
bytes long - but check idx is 9/10 anyway so we know we can read 2 end
bytes of data.

wc_RsaPublicKeyDecode_ex (original):
- Fix calculation of seqEndIdx and use it to bound modulus and
exponent.

DecodePolicyOID
  - enusre inSz is not too long.
  - Ensure no overflow in calculation.

SetOidValue (orginal):
  - Safe check of inSz and oidSz.

SetAltNames (original):
  - Improve length checks

FlattenAltNames:
  - Check for overflow.
  - Better length check.

ParseCRL_CertList (original):
  - overflow check

src/pk_ec.c

@@ -3581,7 +3581,7 @@ int wolfSSL_EC_KEY_LoadDer_ex(WOLFSSL_EC_KEY* key, const unsigned char* derBuf,
          * have a PKCS8 header then do not error out.
          */
         if ((ret = ToTraditionalInline_ex((const byte*)derBuf, &idx,
-                (word32)derSz, &algId)) > 0) {
+                (word32)derSz, &algId)) >= 0) {
             WOLFSSL_MSG("Found PKCS8 header");
             key->pkcs8HeaderSz = (word16)idx;
             res = 1;

src/pk_rsa.c

@@ -924,7 +924,7 @@ int wolfSSL_RSA_LoadDer_ex(WOLFSSL_RSA* rsa, const unsigned char* derBuf,
          * have a PKCS8 header then do not error out. */
         res = ToTraditionalInline_ex((const byte*)derBuf, &idx, (word32)derSz,
             &algId);
-        if (res > 0) {
+        if (res >= 0) {
             /* Store size of PKCS#8 header for encoding. */
             WOLFSSL_MSG("Found PKCS8 header");
             rsa->pkcs8HeaderSz = (word16)idx;

wolfcrypt/src/evp_pk.c

@@ -901,7 +901,7 @@ static WOLFSSL_EVP_PKEY* d2i_evp_pkey(int type, WOLFSSL_EVP_PKEY** out,
         /* Check if input buffer has PKCS8 header. In the case that it does not
          * have a PKCS8 header then do not error out. */
         if ((ret = ToTraditionalInline_ex((const byte*)(*in), &idx,
-                (word32)inSz, &algId)) > 0) {
+                (word32)inSz, &algId)) >= 0) {
             WOLFSSL_MSG("Found PKCS8 header");
             pkcs8HeaderSz = (word16)idx;
 

wolfssl/wolfcrypt/asn.h

@@ -352,7 +352,7 @@ typedef struct ASNGetData {
 } ASNGetData;
 
 WOLFSSL_LOCAL int SizeASN_Items(const ASNItem* asn, ASNSetData *data,
-    int count, int* encSz);
+    int count, word32* encSz);
 WOLFSSL_LOCAL int SetASN_Items(const ASNItem* asn, ASNSetData *data, int count,
     byte* output);
 WOLFSSL_LOCAL int GetASN_Items(const ASNItem* asn, ASNGetData *data, int count,