Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tests fail after 2027-06-23 #6978

Open
bmwiedemann opened this issue Jan 29, 2023 · 11 comments
Open

tests fail after 2027-06-23 #6978

bmwiedemann opened this issue Jan 29, 2023 · 11 comments
Assignees
@gilles-peskine-arm
Labels
component-x509 enhancement size-m Estimated task size: medium (~1w)

Comments

@bmwiedemann
Copy link

bmwiedemann commented Jan 29, 2023 •
edited
Loading

Summary

While working on reproducible builds for openSUSE, I found that in our mbedtls-2.28.2 package,
similar to #2357 , SSL tests start to fail after 2023-08-07.

System information

Mbed TLS version (number or commit id): 2.28.2
Operating system and version: openSUSE-Tumbleweed-20230126
Configuration (if not default, please attach mbedtls_config.h):
Compiler and options (if you used a pre-built binary, please indicate how you obtained it): https://code.opensuse.org/package/mbedtls/blob/master/f/mbedtls.spec#_97
Additional environment information:

Expected behavior

Tests should continue to pass at least 16 years into the future.

Actual behavior

98% tests passed, 2 tests failed out of 103
 The following tests FAILED:
         98 - ssl-suite (Failed)
        101 - x509parse-suite (Failed)

Steps to reproduce

On Debian or openSUSE do:

osc co openSUSE:Factory/mbedtls && cd $_
osc build --vm-type=kvm --noservice --build-opt=--vm-custom-opt="-rtc base=2023-08-08T12:00:00" standard

or set the system clock or use libfaketime to run the tests.

Additional information

tests/data_files/test-ca2_cat-present-future.crt has expiry of Aug 7 09:17:03 2023 GMT and many more expire in September.

Background:
As part of my work on reproducible builds for openSUSE, I check that software still gives identical build results in the future.
The usual offset is +16 years, because that is how long I expect some software will be used in some places.
This showed up failing tests in our package build.
See https://reproducible-builds.org/ for why this matters.
@gilles-peskine-arm gilles-peskine-arm self-assigned this Jan 30, 2023
@gilles-peskine-arm gilles-peskine-arm mentioned this issue Feb 1, 2023
Open
@gilles-peskine-arm
Copy link
Contributor

gilles-peskine-arm commented Feb 1, 2023 •
edited
Loading

Thank you for reporting this!

The build of the library and the test programs should be reproducible. (Do please let us know if it isn"t — we are not currently testing this in our CI.) However, the tests are unavoidably not fully reproducible because there are dates in the test data. We cannot generate the test data based on the current date (it"s far too complex), and we cannot always force a date when testing (it"s not possible in all environments).

Goal of this issue: fix the 2023 expiry. We"ll test manually. I"ve filed a separate issue to address the requirement of having tests pass 16 years in the future.

@gilles-peskine-arm
Copy link
Contributor

gilles-peskine-arm commented Feb 1, 2023 •
edited
Loading

The list of test cases that will fail on 07 August 2023 in the full config as of fd13a0f:

From test_suite_pkcs7:

PKCS7 Signed Data Verification Pass SHA256 #9 ..................... FAILED
PKCS7 Signed Data Verification Pass SHA256 #9.1 ................... FAILED
PKCS7 Signed Data Verification Pass SHA1 #10 ...................... FAILED
PKCS7 Signed Data Verification Pass SHA512 #11 .................... FAILED
PKCS7 Signed Data Verification Fail because of different certifica  FAILED
PKCS7 Signed Data Verification Fail because of different data hash  FAILED
PKCS7 Signed Data Verify with multiple signers #16 ................ FAILED
PKCS7 Signed Data Hash Verify with multiple signers #17 ........... FAILED

From test_suite_ssl:

Handshake, ECDHE-ECDSA-WITH-AES-256-CCM ........................... FAILED
Handshake, ECDH-ECDSA-WITH-CAMELLIA-256-CBC-SHA384 ................ FAILED
DTLS Handshake, ECDHE-ECDSA-WITH-AES-256-CCM ...................... FAILED
DTLS Handshake, ECDH-ECDSA-WITH-CAMELLIA-256-CBC-SHA384 ........... FAILED
Handshake, select ECDHE-ECDSA-WITH-AES-256-CCM, non-opaque ........ FAILED
Handshake, select ECDHE-ECDSA-WITH-AES-256-CCM, opaque, PSA_ALG_AN  FAILED
Handshake, select ECDHE-ECDSA-WITH-AES-256-CCM, opaque, PSA_ALG_SH  FAILED
Handshake, select ECDH-RSA-WITH-AES-256-CBC-SHA384, non-opaque .... FAILED
Handshake, select ECDH-RSA-WITH-AES-256-CBC-SHA384, opaque ........ FAILED
Handshake, select ECDH-ECDSA-WITH-CAMELLIA-256-CBC-SHA384, non-opa  FAILED
Handshake, select ECDH-ECDSA-WITH-CAMELLIA-256-CBC-SHA384, opaque,  FAILED
Handshake, select ECDH-ECDSA-WITH-CAMELLIA-256-CBC-SHA384, opaque,  FAILED
Raw key agreement: nominal ........................................ FAILED
Raw key agreement: bad server key ................................. FAILED

From test_suite_x509parse:

X509 Time Future #2 ............................................... FAILED
X509 CRT verification #1a (Revoked Cert, Future CRL, no CN) ....... FAILED
X509 CRT verification #2a (Revoked Cert, Future CRL) .............. FAILED
X509 CRT verification #3a (Revoked Cert, Expired CRL, CN Mismatch)  FAILED
X509 CRT verification #4a (Revoked Cert, Future CRL) .............. FAILED
X509 CRT verification #8 (Valid Cert) ............................. FAILED
X509 CRT verification #8a (Expired Cert) .......................... FAILED
X509 CRT verification #8b (Future Cert) ........................... FAILED
X509 CRT verification #8c (Expired Cert, longer chain) ............ FAILED
X509 CRT verification #8d (Future Cert, longer chain) ............. FAILED
X509 CRT verification #30 (domain matching multi certificate witho  FAILED
X509 CRT verification #31 (domain not matching multi certificate w  FAILED
X509 CRT verification #32 (Valid, EC cert, RSA CA) ................ FAILED
X509 CRT verification #33 (Valid, RSA cert, EC CA) ................ FAILED
X509 CRT verification #34 (Valid, EC cert, EC CA) ................. FAILED
X509 CRT verification #35 (Revoked, EC CA) ........................ FAILED
X509 CRT verification #36 (Valid, EC CA, SHA1 Digest) ............. FAILED
X509 CRT verification #37 (Valid, EC CA, SHA224 Digest) ........... FAILED
X509 CRT verification #38 (Valid, EC CA, SHA384 Digest) ........... FAILED
X509 CRT verification #39 (Valid, EC CA, SHA512 Digest) ........... FAILED
X509 CRT verification #43 (Depth 0, not CA, EC) ................... FAILED
X509 CRT verification #44 (Corrupted signature, EC) ............... FAILED
X509 CRT verification #45b (Corrupted signature, intermediate CA) . FAILED
X509 CRT verification #46 (Valid, depth 2, EC-RSA-EC) ............. FAILED
X509 CRT verification #47 (Untrusted, depth 2, EC-RSA-EC) ......... FAILED
X509 CRT verification #48 (Missing intermediate CA, EC-RSA-EC) .... FAILED
X509 CRT verification #49 (Valid, depth 2, RSA-EC-RSA) ............ FAILED
X509 CRT verification #52 (CA keyUsage valid) ..................... FAILED
X509 CRT verification #53 (CA keyUsage missing cRLSign) ........... FAILED
X509 CRT verification #54 (CA keyUsage missing cRLSign, no CRL) ... FAILED
X509 CRT verification #55 (CA keyUsage missing keyCertSign) ....... FAILED
X509 CRT verification #56 (CA keyUsage plain wrong) ............... FAILED
X509 CRT verification #57 (Valid, RSASSA-PSS, SHA-1) .............. FAILED
X509 CRT verification #58 (Valid, RSASSA-PSS, SHA-224) ............ FAILED
X509 CRT verification #59 (Valid, RSASSA-PSS, SHA-256) ............ FAILED
X509 CRT verification #60 (Valid, RSASSA-PSS, SHA-384) ............ FAILED
X509 CRT verification #61 (Valid, RSASSA-PSS, SHA-512) ............ FAILED
X509 CRT verification #62 (Revoked, RSASSA-PSS, SHA-1) ............ FAILED
X509 CRT verification #63 (Revoked, RSASSA-PSS, SHA-1, CRL badsign  FAILED
X509 CRT verification #64 (Valid, RSASSA-PSS, SHA-1, not top) ..... FAILED
X509 CRT verification #65 (RSASSA-PSS, SHA1, bad cert signature) .. FAILED
X509 CRT verification #66 (RSASSA-PSS, SHA1, no RSA CA) ........... FAILED
X509 CRT verification #67 (Valid, RSASSA-PSS, all defaults) ....... FAILED
X509 CRT verification #68 (RSASSA-PSS, wrong salt_len, USE_PSA) ... FAILED
X509 CRT verification #69 (RSASSA-PSS, wrong mgf_hash) ............ FAILED
X509 CRT verification #70 (v1 trusted CA) ......................... FAILED
X509 CRT verification #71 (v1 trusted CA, other) .................. FAILED
X509 CRT verification #72 (v1 chain) .............................. FAILED
X509 CRT verification #73 (selfsigned trusted without CA bit) ..... FAILED
X509 CRT verification #74 (signed by selfsigned trusted without CA  FAILED
X509 CRT verification #76 (multiple CRLs, not revoked) ............ FAILED
X509 CRT verification #77 (multiple CRLs, revoked) ................ FAILED
X509 CRT verification #78 (multiple CRLs, revoked by second) ...... FAILED
X509 CRT verification #79 (multiple CRLs, revoked by future) ...... FAILED
X509 CRT verification #82 (Not yet valid CA and valid CA) ......... FAILED
X509 CRT verification #83 (valid CA and Not yet valid CA) ......... FAILED
X509 CRT verification #84 (valid CA and Not yet valid CA) ......... FAILED
X509 CRT verification #85 (Not yet valid CA and valid CA) ......... FAILED
X509 CRT verification #86 (Not yet valid CA and invalid CA) ....... FAILED
X509 CRT verification #87 (Expired CA and invalid CA) ............. FAILED
X509 CRT verification #88 (Spurious cert in the chain) ............ FAILED
X509 CRT verification #89 (Spurious cert later in the chain) ...... FAILED
X509 CRT verification #90 (EE with same name as trusted root) ..... FAILED
X509 CRT verification #93 (Suite B invalid, EC cert, RSA CA) ...... FAILED
X509 CRT verification #94 (Suite B invalid, RSA cert, EC CA) ...... FAILED
X509 CRT verification #95 (Suite B Valid, EC cert, EC CA) ......... FAILED
X509 CRT verification callback: bad name .......................... FAILED
X509 CRT verification callback: trusted EE cert ................... FAILED
X509 CRT verification callback: simple, root expired .............. FAILED
X509 CRT verification callback: intermediate ca ................... FAILED
X509 CRT verification callback: intermediate ca, root included .... FAILED
X509 CRT verification callback: intermediate ca trusted ........... FAILED
X509 CRT verification callback: intermediate ca, EE expired ....... FAILED
X509 CRT verification callback: intermediate ca, int expired ...... FAILED
X509 CRT verification callback: intermediate ca, root expired ..... FAILED
X509 CRT verification callback: two intermediates ................. FAILED
X509 CRT verification callback: two intermediates, root included .. FAILED
X509 CRT verification callback: two intermediates, top int trusted  FAILED
X509 CRT verification callback: two intermediates, low int trusted  FAILED
X509 CRT verification callback: no intermediate, bad signature .... FAILED
X509 CRT verification callback: one intermediate, bad signature ... FAILED
X509 CRT verify long chain (max intermediate CA, trusted) ......... FAILED
X509 CRT verify long chain (max intermediate CA, untrusted) ....... FAILED
X509 CRT verify chain #12 (suiteb profile, RSA root) .............. FAILED
X509 CRT verify chain #13 (RSA only profile, EC root) ............. FAILED
X509 CRT verify chain #13 (RSA only profile, EC trusted EE) ....... FAILED
X509 CRT verify chain #15 (suiteb profile, rsa intermediate) ...... FAILED
X509 CRT verify chain #16 (RSA-only profile, EC intermediate) ..... FAILED
X509 CRT verify chain #17 (SHA-512 profile) ....................... FAILED
X509 CRT verify restart: trusted EE, max_ops=0 (disabled) ......... FAILED
X509 CRT verify restart: trusted EE, max_ops=1 .................... FAILED
X509 CRT verify restart: no intermediate, max_ops=0 (disabled) .... FAILED
X509 CRT verify restart: no intermediate, max_ops=1 ............... FAILED
X509 CRT verify restart: no intermediate, max_ops=40000 ........... FAILED
X509 CRT verify restart: no intermediate, max_ops=500 ............. FAILED
X509 CRT verify restart: no intermediate, badsign, max_ops=0 (disa  FAILED
X509 CRT verify restart: no intermediate, badsign, max_ops=1 ...... FAILED
X509 CRT verify restart: no intermediate, badsign, max_ops=40000 .. FAILED
X509 CRT verify restart: no intermediate, badsign, max_ops=500 .... FAILED
X509 CRT verify restart: one int, max_ops=0 (disabled) ............ FAILED
X509 CRT verify restart: one int, max_ops=1 ....................... FAILED
X509 CRT verify restart: one int, max_ops=30000 ................... FAILED
X509 CRT verify restart: one int, max_ops=500 ..................... FAILED
X509 CRT verify restart: one int, EE badsign, max_ops=0 (disabled)  FAILED
X509 CRT verify restart: one int, EE badsign, max_ops=1 ........... FAILED
X509 CRT verify restart: one int, EE badsign, max_ops=30000 ....... FAILED
X509 CRT verify restart: one int, EE badsign, max_ops=500 ......... FAILED
X509 CRT verify restart: one int, int badsign, max_ops=0 (disabled  FAILED
X509 CRT verify restart: one int, int badsign, max_ops=1 .......... FAILED
X509 CRT verify restart: one int, int badsign, max_ops=30000 ...... FAILED
X509 CRT verify restart: one int, int badsign, max_ops=500 ........ FAILED

Many of them are copies of the same few certificates (one RSA and one ECC?) used in different context. But quite a few are manually modified to exercise error detection in parsers.

@gilles-peskine-arm gilles-peskine-arm mentioned this issue Feb 1, 2023
Open
@gilles-peskine-arm gilles-peskine-arm added the size-m Estimated task size: medium (~1w) label Feb 1, 2023
@bmwiedemann
Copy link
Author

For other test-suites we created certs that expire in 2999, because these are not meant to expire (unlike regular production-grade certification). Would that work for you?

gnutls" certtool -u can update certs to an expiry of 2049-12-31 (but not beyond yet because of some format change there)

@gilles-peskine-arm
Copy link
Contributor

Changing to make our certs expire in 2999 would be great if it works. I don"t know if it"s possible: I don"t know if everything we"re using supports dates that far in the future. (We don"t even have tests for Y2038!) This includes interop tests with old versions of OpenSSL and GnuTLS.

The difficulty for us with updating certificates is that since we"re an X.509 library, we have a lot of negative test cases, many of them crafted by manually tweaking some nominal data. Updating the files containing nominal data is only the tip of the iceberg.

@bmwiedemann
Copy link
Author

bmwiedemann commented Feb 19, 2023 •
edited
Loading

An alternative solution could be to provide a date to the validation functions that is used instead of the current date.
Thus such a test would be about "a cert that expires in 2023 is valid in 2022" independent of the system clock, so it keeps passing even after the current year moves to 2024. This has the advantage that you do not need to recreate hand-crafted certs.

A date-override could even have advantages in production builds. E.g. when you run as non-root on a system with incorrect system time and still want SSL validation.

@gilles-peskine-arm gilles-peskine-arm mentioned this issue Mar 14, 2023
Closed
@bmwiedemann
Copy link
Author

Hi. What is the status of the fix for this issue? Only one week left until 2023-08-07

@gilles-peskine-arm
Copy link
Contributor

We"ve regenerated all the test certificates that expired in 2023. We"re keeping this issue open because we"re still working on an alert mechanism and on regenerating certificates that will expire in the next few years.

@gilles-peskine-arm
Copy link
Contributor

gilles-peskine-arm commented Aug 3, 2023 •
edited
Loading

For the sake of continuing to have passing tests in third-party packages that ship Mbed TLS such as Linux distributions and BSP, we"re going to make official releases today or tomorrow (right now we"re making some final preparations):

  • A 3.4.1 release which is 3.4.0 plus updated certificates in tests — no other bug fixes (except a compilation error that was accidentally merged into master instead of development). There will be no security fixes.
  • A 2.28 LTS release with the current state of the mbedtls-2.28 branch. This means updated certificates in tests and in the certs module, plus all the bug fixes that are already in that branch. There will be no security fixes.

@gilles-peskine-arm
Copy link
Contributor

The releases are out: 3.4.1 and 2.28.4. Their tests should be good until January 2024, and in a month or two we"re planning a new release with tests that are good until at least mid-2027. Incremental progress!

@bmwiedemann
Copy link
Author

Thanks a lot.
I still have hope, that some day, tests will continue to pass forever.

@gilles-peskine-arm gilles-peskine-arm changed the title tests fail after 2023-08-07 tests fail after 2027-06-23 Oct 26, 2023
@gilles-peskine-arm
Copy link
Contributor

As of 2.28.5 and 3.5.0, the next certificate expiry is on 2027-06-23 (dir-maxpath/*.crt).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gilles-peskine-arm gilles-peskine-arm

Labels
component-x509 enhancement size-m Estimated task size: medium (~1w)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bmwiedemann @gilles-peskine-arm