Fix ECDSA issuer digest initialization and improve algorithm handling

[adamboutcher]

New UKCA certs are signed with ECDSA.

I used Claude to write this fix so it might not be safe; I don't write C so take it with a pinch of salt/review.

[adamboutcher]

Currently in draft as I'm iteratively working through the issue and I haven't yet confirmed that a new certificate will work. I will pick up the rest of this on Monday.

[maarten-litmaath]

Hi Adam,
thanks for looking into this matter, but mind it may well be the tip of an iceberg!
Have a look at page 3 of this presentation:

https://indico.cern.ch/event/1635405/#26-certificate-support-and-gri

There are lots of Java-based storage services (dCache, StoRM) that would need to be upgraded
to releases containing a fixed version of canl-java --> not going to happen any time soon...

It would be best if the UKCA continued supporting RSA for the foreseeable future,
to avoid that concerned services may end up being largely unusable by their communities...

Hopefully I'm overly pessimistic here!

[adamboutcher]

It would be best if the UKCA continued supporting RSA for the foreseeable future, to avoid that concerned services may end up being largely unusable by their communities...

I'm not really in a position to enforce this; it would have been nice for it to be the cause though.

Would adding the fix here break things upstream? I believe this should fallback to its original methods so it should work nicely with incompatible upstream packages as they will just fail over if something with ECDSA touches them?

I'm waiting a test result with the grid-proxy-init and myproxy-init tools as these are required as part of a submission chain. Testing worked!

[jnc74743]

Hi Adam,
I'm happy to chat and figure out what's changed. The UKCA only supports RSA and this will not change anytime soon. I'm unsure where you're seeing the signing with ECDSA, could you provide some more detail please?

[adamboutcher]

Hi Adam, I'm happy to chat and figure out what's changed. The UKCA only supports RSA and this will not change anytime soon. I'm unsure where you're seeing the signing with ECDSA, could you provide some more detail please?

One of our users with a UKCA signed certificate. I've uploaded a redacted output from the UKCA RAOP Portal.

[jjensenral]

The new PKI hierarchy ("3") is indeed EC based. End entity certs (serials >100000) default to RSA but the intermediate and the root in IGTF are all EC. These have been published through IGTF since (looks it up) 1.141 in May.
The old hierarchy ("2") is still alive and of course will be supported until October 2027 (when CA certificates expire) but end entity certs are no longer issued on 2B since they need a validity of 13 months.
Extending the lifetime of the "2" CA certificates is not really an option for various technical reasons

[maarten-litmaath]

@jjensenral Could something still be done about this matter in some way?

Going to ECDSA was a rather significant change that ought to have been checked with experts from WLCG and other relying parties beforehand, I would think?

[msalle]

Out of curiosity, does the C-based voms-proxy-init work? If so we could port part of the code over.
EPEL: dnf install voms-clients-cpp
Debian: apt-get install voms-clients
If not, it might also be fixed in the latest 2.1.4-rc1. I would be easy to build an RPM if needed.

[adamboutcher]

Could we not, you know, just fix the tooling using the incorrect digest to check for the signing algorithm? Feels like if its being done wrong in the first place then fixing the underlying issue is the actual solution and not (yet again) trying to bodge in old stuff.

[jjensenral]

@jjensenral Could something still be done about that in some way?

DaveK forwarded the issue, we will discuss & get back to you

[maarten-litmaath]

Could we not, you know, just fix the tooling using the incorrect digest to check for the signing algorithm? Feels like if its being done wrong in the first place then fixing the underlying issue is the actual solution and not (yet again) trying to bodge in old stuff.

Sure, but this matter creates an unexpected, big exercise that came totally out of nothing!

[adamboutcher]

Could we not, you know, just fix the tooling using the incorrect digest to check for the signing algorithm? Feels like if its being done wrong in the first place then fixing the underlying issue is the actual solution and not (yet again) trying to bodge in old stuff.

Sure, but this matter creates an unexpected, big exercise that came totally out of nothing!

Not really out of nothing, ECDSA has been supported in OpenSSL since ~2010. And (if Claude is to be believed) the code base is using the wrong checks to detect the signing algorithm. Sure a bug is a bug and to be somewhat expected however people relying on a bug is less sensible. As previously mentioned this PR should fall back to its current behaviour so any tooling expecting the wrong thing should still work as expected or simply fail with ECDSA certificates which they would anyway just in a different part of the codebase.

[maarten-litmaath]

Could we not, you know, just fix the tooling using the incorrect digest to check for the signing algorithm? Feels like if its being done wrong in the first place then fixing the underlying issue is the actual solution and not (yet again) trying to bodge in old stuff.

Sure, but this matter creates an unexpected, big exercise that came totally out of nothing!

Not really out of nothing, ECDSA has been supported in OpenSSL since ~2010. And (if Claude is to be believed) the code base is using the wrong checks to detect the signing algorithm. Sure a bug is a bug and to be somewhat expected however people relying on a bug is less sensible. As previously mentioned this PR should fall back to its current behaviour so any tooling expecting the wrong thing should still work as expected or simply fail with ECDSA certificates which they would anyway just in a different part of the codebase.

To make those UK certificates usable in WLCG, we will need a really big deployment exercise to get storage services upgraded at the majority of our sites, to versions that do not even exist yet. So, a decision taken in the UK is leading to a lot of work for many parties around the world, including the UK...

[jjensenral]

There is a GridPP collaboration meeting right now (= this week) where I'm sure it's being discussed

[adamboutcher]

Breaking upstream software isn't the scope of this repository though, the maintainers of said software can fix them as and when. Especially as this fix doesn't change any fundamental functions just corrects parts that should work.

Also the move to ECDSA may be only a UK decision however pushing back on fixing it merely because "it's big" and not seeing the forwards progress is incredibly short sighted. Who's to say another NGI/CA doesn't move to ECDSA once it's actually supported?

[maarten-litmaath]

@adamboutcher The work you are doing in this repository is much appreciated and independent of what's happening elsewhere.

We will look further into other concerned stacks and see what realistically can be done for those by when. At the very least, UKCA v2 should continue issuing certificates well into next year (with ever decreasing lifetimes), probably at least until summer, to allow us to have some chance to get all the concerned services upgraded...

Mind, it's not so much the user certificates we need to worry about in WLCG proper (*), but rather the host certificates, in particular of the affected storage services.

(*) LHC experiment users can get CERN certificates, while users from other communities may not be able to.

[msalle]

Hi all,
I briefly discussed also with Francesco Giacomini and did some testing:
I've tested C-based voms-proxy-init and voms-proxy-info and they both work fine with CA and EEC signed using ecdsa-with-SHA256. The resulting proxy is signed using sha1WithRSAEncryption (ie. mixed chain) but we don't see a reason that would be problematic.

The java based voms-proxy-init complains though (as does indeed the grid-proxy-init) which might very well mean that the VOMS AA also will not accept it since it's relying on the same voms-api-java, canl-java and bouncycastle:

Error creating proxy certificate: can't identify EC private key.
org.italiangrid.voms.VOMSError: Error creating proxy certificate: can't identify EC private key.
        at org.italiangrid.voms.clients.impl.DefaultVOMSProxyInitBehaviour.createProxy(DefaultVOMSProxyInitBehaviour.java:475)
        at org.italiangrid.voms.clients.impl.DefaultVOMSProxyInitBehaviour.initProxy(DefaultVOMSProxyInitBehaviour.java:209)
        at org.italiangrid.voms.clients.VomsProxyInit.execute(VomsProxyInit.java:284)
        at org.italiangrid.voms.clients.VomsProxyInit.<init>(VomsProxyInit.java:60)
        at org.italiangrid.voms.clients.VomsProxyInit.main(VomsProxyInit.java:44)
Caused by: java.security.InvalidKeyException: can't identify EC private key.
        at org.bouncycastle.jcajce.provider.asymmetric.util.ECUtil.generatePrivateKeyParameter(Unknown Source)
        at org.bouncycastle.jcajce.provider.asymmetric.ec.ECUtils.generatePrivateKeyParameter(Unknown Source)
        at org.bouncycastle.jcajce.provider.asymmetric.ec.SignatureSpi.engineInitSign(Unknown Source)
        at java.base/java.security.Signature$Delegate.tryOperation(Signature.java:1283)
        at java.base/java.security.Signature$Delegate.chooseProvider(Signature.java:1237)
        at java.base/java.security.Signature$Delegate.engineInitSign(Signature.java:1321)
        at java.base/java.security.Signature.initSign(Signature.java:607)
        at eu.emi.security.authn.x509.helpers.proxy.X509v3CertificateBuilder.calculateSignature(X509v3CertificateBuilder.java:184)
        at eu.emi.security.authn.x509.helpers.proxy.X509v3CertificateBuilder.sign(X509v3CertificateBuilder.java:147)
        at eu.emi.security.authn.x509.helpers.proxy.X509v3CertificateBuilder.build(X509v3CertificateBuilder.java:137)
        at eu.emi.security.authn.x509.helpers.proxy.ProxyGeneratorHelper.buildCertificate(ProxyGeneratorHelper.java:339)
        at eu.emi.security.authn.x509.helpers.proxy.ProxyGeneratorHelper.generateCommon(ProxyGeneratorHelper.java:119)
        at eu.emi.security.authn.x509.helpers.proxy.ProxyGeneratorHelper.generate(ProxyGeneratorHelper.java:83)
        at eu.emi.security.authn.x509.proxy.ProxyGenerator.generate(ProxyGenerator.java:50)
        at org.italiangrid.voms.clients.impl.DefaultVOMSProxyInitBehaviour.createProxy(DefaultVOMSProxyInitBehaviour.java:467)
        ... 4 more

[maarten-litmaath]

@msalle Why is sha1WithRSAEncryption not an issue? Today it presumably works because we tell sites to re-enable SHA-1 on EL9, but we will not do that on EL10, which will be coming as of the start of 2027!

[msalle]

@msalle Why is sha1WithRSAEncryption not an issue? Today it presumably works because we tell sites to re-enable SHA-1 on EL9, but we will not do that on EL10, which will be coming as of the start of 2027!

my point was that mixed is not a problem. I do agree SHA-1 will become an issue, but that we should open as an issue against VOMS.
And it does do some magic: voms-proxy-init uses the SHA-1 when the EEC is signed with ecdsa-with-SHA256, but when the EEC is signed with sha256WithRSAEncryption it also will use that for the proxy.

[maarten-litmaath]

OK, issues opened for voms-proxy-init v2 (C++) and v3 (Java).

The fix for the latter will likely have to be in a common dependency also used by dCache and StoRM...