Service Bus for Windows and expired certificate

In our infrastructure we are using Service Bus for Windows. It fits good into our stack of MS technologies.

During initial setup we configured our farm with wildcard certificate used for every service that requires communicating over SSL. Occasionally the certificate has expired and farm stopped working. As it always happens no one has requested it in advance. Fix seemed quite easy and straightforward:

1. Request certificate
2. Install certificate
3. Set it for farm
4. Profit

But it turned out not so easy. Simple PowerShell command hadn't helped me:

$thumb = '<new cert. thimbprint>'
Set-SBCertificate -FarmCertificateThumbprint $thumb -EncryptionCertificateThumbprint $thumb

System.InvalidOperationException: Certificate requested with thumbprint '<expired cert. thumbprint>' not found in the certificate store LocalMachine\My.
   at Microsoft.ServiceBus.Commands.Common.DBEncryptionHelper.FindCert(String thumbprint)
...

Okay. We can't do almost anything now with that service bus farm, because of expired certificate. Without thinking twice I set date back the day before, when expired cert was still valid. And tried again:

$thumb = '<new cert. thimbprint>'
Set-SBCertificate -FarmCertificateThumbprint $thumb -EncryptionCertificateThumbprint $thumb

Set-SBCertificate : Certificate requested with thumbprint <new cert. thumbprint> not found in the certificate store LocalMachine\My.

How could that be? I checked permissions, reinstalled, set permissions again... It was very confusing so I took ILSpy and checked what was actually happening inside of Service Bus cmdleds:

[ValidateClusterCertificate, Parameter(Mandatory = false, HelpMessageBaseName = "Microsoft.ServiceBus.Commands", HelpMessageResourceId = "ClusterCertificateThumbprint", ParameterSetName = "FarmAndEncryptionCert")]
    public string FarmCertificateThumbprint
    {
        get;
        set;
    }

That attribute ValidateClusterCertificate looked like the one who can create a problem.

[AttributeUsage(AttributeTargets.Property, AllowMultiple = false)]
internal sealed class ValidateClusterCertificateAttribute : ValidateArgumentsAttribute
{
    public static void Validate(string thumbprint)
    {
        if (string.IsNullOrWhiteSpace(thumbprint))
        {
            throw new ValidationMetadataException(SR.ParameterNullOrWhiteSpace);
        }
        X509Store x509Store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
        x509Store.Open(OpenFlags.ReadOnly);
        X509Certificate2 x509Certificate;
        try
        {
            x509Certificate = x509Store.Certificates.Find(X509FindType.FindByThumbprint, thumbprint, true).OfType<X509Certificate2>().SingleOrDefault<X509Certificate2>();
            if (x509Certificate == null)
            {
                throw new ValidationMetadataException(string.Format(CultureInfo.CurrentCulture, SR.ThumbprintNotFound, new object[]
                {
                    thumbprint
                }));
            }
        }
        finally
        {
            x509Store.Close();
        }

So it just tries to find only valid certificates in private store. Check this signature of Find method

public X509Certificate2Collection Find(
	X509FindType findType,
	Object findValue,
	bool validOnly
)

Okay. Maybe I could find the date when both certificates were valid? No way.
Expired certificate was not valid after May X, ‎2015 12:38:00 PM.
New one was not valid before May ‎X, ‎2015 4:00:00 PM.
Very funny :)

Since it was staging environment I just reinstalled the farm with the new certificate and configured namespaces with pair of commands. Note that to remove Service Bus host and farm properly I still had to rollback date the day before.

Summary:

• renew certificates in advance
• make a plan of switching certificates

P.S. After finishing configuring I realized that in case of high importance of data inside of Service Bus it can be saved by using intermediate self-signed certificate. The flow is: rollback date -> install valid self-signed cert -> set actual date and time -> install proper renewed certificate.