OK some more info :

To be safe I've tried with the same private key to generate another x509 (as per the comment on the example code).

See my gist to find out how to create x509 certificates for
Google HSM-hosted keys: https://gist.github.com/Arbitrage0/de4e0defb20bc539d6db27e4334e0e67

The result is exactly the same with a SIG_CRYPTO_FAILURE, except I also now have "The result of the 'X.509 Certificate Validation' building block is not conclusive" error which is expected with this manually created and untrusted certificate.

BUT reading the code from verifier.py I saw something that made me think that the "signature_algorithm" was maybe the culprit...I created a new HSM key but this time with the algo "2048 bit RSA key PKCS#1 v1.5 padding - SHA256 Digest" (instead of "2048 bit RSA key PSS Padding - SHA256 Digest" for the first key).

I created another x509 manually and...The resulting PDF does not have SIG_CRYPTO_FAILURE !

The problem is that now I have issued the certificate for 3 years with the RSA PSS key so I would very much like to use it 😢
Yeah I should have tried all that first, I did not think PSS would be a problem 😞

I can see that the "pss" param in cms is forced to False. And if I try to force it to True, I now have a (logical) Exception :

File "/.../lib/python3.8/site-packages/endesive/pdf/cms.py", line 680, in sign
contents = signer.sign(
File "/.../lib/python3.8/site-packages/endesive/signer.py", line 160, in sign
salt_length = padding.calculate_max_pss_salt_length(key, hashes.SHA512)
File "/.../lib/python3.8/site-packages/cryptography/hazmat/primitives/asymmetric/padding.py", line 97, in calculate_max_pss_salt_length
raise TypeError("key must be an RSA public or private key")
TypeError: key must be an RSA public or private key

Because the key==None when you use an HSM.

Does anyone have an idea on how to modify the salt_length / calculate_max_pss_salt_length code to adapt to this use-case ?

Thanks !

I tried some more things :

1. a) Force pss = True in cms.py:689 and cms.py:774
   b) Force salt_length = 256 (as per https://cloud.google.com/kms/docs/algorithms "For Probabilistic Signature Scheme (PSS), the salt length used is equal to the length of the digest algorithm. For example, RSA_SIGN_PSS_2048_SHA256 will use PSS with a salt length of 256 bits.") and key algo to "sha256" in signer.sign()

    if not pss:
        signer["signature_algorithm"] = algos.SignedDigestAlgorithm(
            {"algorithm": "rsassa_pkcs1v15"}
        )
    else:
        salt_length = 256
        key_algorithm = "sha256"
        signer["signature_algorithm"] = algos.SignedDigestAlgorithm(
            {
                "algorithm": "rsassa_pss",
                "parameters": algos.RSASSAPSSParams(
                    {
                        "hash_algorithm": algos.DigestAlgorithm(
                            {"algorithm": key_algorithm}
                        ),
                        "mask_gen_algorithm": algos.MaskGenAlgorithm(
                            {
                                "algorithm": algos.MaskGenAlgorithmId("mgf1"),
                                "parameters": {
                                    "algorithm": algos.DigestAlgorithmId(key_algorithm),
                                },
                            }
                        ),
                        "salt_length": algos.Integer(salt_length),
                        "trailer_field": algos.TrailerField(1),
                    }
                ),
            }
        )

But to no avail. I then tried

2. a) Same as 1.a
   b) Put only the algo in signature_algorithm like for rsassa_pkcs1v15

    if not pss:
        signer["signature_algorithm"] = algos.SignedDigestAlgorithm(
            {"algorithm": "rsassa_pkcs1v15"}
        )
    else:
        signer["signature_algorithm"] = algos.SignedDigestAlgorithm(
            {"algorithm": "rsassa_pss"}
        )

Not working either.

Anyone knows the right config of what to put there ?

Or maybe the problem is later, in the hsm.sign() call or code ?

I made it work !!! 😍

With pss = True and this code for the signer["signature_algorithm"] :

    if not pss:
        signer["signature_algorithm"] = algos.SignedDigestAlgorithm(
            {"algorithm": "rsassa_pkcs1v15"}
        )
    else:
        salt_length = hashes.SHA256.digest_size
        key_algorithm = "sha256"
        signer["signature_algorithm"] = algos.SignedDigestAlgorithm(
            {
                "algorithm": "rsassa_pss",
                "parameters": algos.RSASSAPSSParams(
                    {
                        "hash_algorithm": algos.DigestAlgorithm(
                            {"algorithm": key_algorithm}
                        ),
                        "mask_gen_algorithm": algos.MaskGenAlgorithm(
                            {
                                "algorithm": algos.MaskGenAlgorithmId("mgf1"),
                                "parameters": {
                                    "algorithm": algos.DigestAlgorithmId(key_algorithm),
                                },
                            }
                        ),
                        "salt_length": algos.Integer(salt_length),
                        "trailer_field": algos.TrailerField(1),
                    }
                ),
            }
        )

So in my case "salt_length" must be == 32 even though Google says

For Probabilistic Signature Scheme (PSS), the salt length used is equal to the length of the digest algorithm. For example, RSA_SIGN_PSS_2048_SHA256 will use PSS with a salt length of 256 bits.

The first part is correct but the second is not...I think you really need to read "use the digest size of the hash algo for the salt length", so for SHA256 it's 32 not 256 !

The rest of the endesive code is OK, except the two hardcoded "sha512" in the PSS parameters (I don't know why that is ?).

@m32 could I create a PR to use the hashalgo param instead of "sha512" (and the hashes.SHA512() further down for hsm == None and pss == True) in signer.py ?

I added a test: test_pdf_pss and the pss parameter in udct.
In the pss handling code I added code for selecting the appropriate hash function based on hashalgo.

Since I don't have the Google key, I can't check how it works, please check the Git versions with these corrections.

Hello @m32 !

Thank you very much for your edit ! Unfortunately it does not work as the code in signer.py:159 tries to calculate the salt length with a non-existing key. The private key is inaccessible in the HSM so the pdf.cms.sign() is made with Key==None :

salt_length = padding.calculate_max_pss_salt_length(key, md)

If I force the salt_length to md.digest_size like in my PR, everything else works as is !

        md = getattr(hashes, hashalgo.upper())
        if hsm is not None:
            salt_length = md.digest_size
        elif isinstance(key, keys.PrivateKeyInfo):
            salt_length = key.byte_size - md.digest_size - 2
            salt_length = md.digest_size
        else:
            salt_length = padding.calculate_max_pss_salt_length(key, md)
        signer["signature_algorithm"] = ...

Would you be able to release a new version with this change ?

Thank you again very much for your response and quick action.

I need to have an example and a test, an identical test for the case when Key!=None (current behavior), otherwise I don't know what repercussions it will have on the existing code. Please prepare an appropriate correction as (new) PR and we will know after testing.

Do you mean the case when hsm is None or key is None ?
Otherwise the key is known and the size of the salt can be calculated.

    md = getattr(hashes, hashalgo.upper())
    if key is None: # <-- this line is important
        salt_length = md.digest_size
    elif isinstance(key, keys.PrivateKeyInfo):
        salt_length = key.byte_size - md.digest_size - 2
        salt_length = md.digest_size
    else:
        salt_length = padding.calculate_max_pss_salt_length(key, md)
    signer["signature_algorithm"] = ...

I mean when hsm != None so key == None.

I'm sorry but I don't see theses lines

if key is None: # <-- this line is important
    salt_length = md.digest_size

anywhere in the repo.

They would indeed make my case work, they are similar to my if hsm is not None:

Can you point me to where this code lives ?

signer.py line 337:

    if hsm is not None:
        tosign is signed by hsm
   elif isinstance(key, keys.PrivateKeyInfo):
       tosign is signed by ec key
   else:
       if pss:
           tosign is signed by key and salt_length must be known
           in this case (yours ?) hsm must be None
           and key must be not not None
           and not instance of PrivateKeyInfo
           salt_length is calculated in line 155
       else:
           tosign is signed by key in pkcs1v15 mode

change added and commited, next version is ready to publish, please confirm

Yes that last version commited (634fff1) indeed works great !

Just to clarify : My case is (like in pdf-sign-cms-hsm-google.py) :

datas = pdf.cms.sign(
        datau=datau,
        udct=dct,
        key=None,
        cert=None,
        othercerts=[],
        algomd='sha256',
        hsm=Ghsm,
    )

And the problem was (as you found out) earlier in the code of signer.py (lines 154-162) than lines 337-

Thank you very much in advance for publishing the new version 🙏

published

2.17.2 tested and approved !

For future reference if anyone find this thread, Endesive works with both these Google Cloud KSM HSM key algos :
2048 bit RSA key PSS Padding - SHA256 Digest (with udct["pss"]=True)
and
2048 bit RSA key PKCS#1 v1.5 padding - SHA256 Digest (with udct["pss"]=False, default)

Just follow the example in /examples/pdf-sign-cms-hsm-google.py for the details.

Thanks again for everything @m32, keep up the good work with this great lib 😸

Cheers.