IdentityServer 7 - Token validation fails after scaling out (multi-instance) - "Key was not found in the key ring"

[macupryk]

We're using Duende IdentityServer (or IdentityServer7) hosted in Azure App Service with scale-out to 2+ instances.

✅ Current Setup:

We're using StackExchange.Redis to persist Data Protection keys:

services.AddDataProtection()
    .PersistKeysToStackExchangeRedis(redis, "DataProtection-Keys")
    .SetApplicationName("IdentityServer")
    .SetDefaultKeyLifetime(TimeSpan.FromDays(90));

Redis connection is shared and confirmed to be reachable by all instances.

We're not using Azure Blob or file system persistence.

IdentityServer is configured with:

services.AddIdentityServer()
    .AddSigningCredential(...)
    .AddAspNetIdentity<ApplicationUser>()
    // other config...

🧪 Problem:

After deployment or during scaling events, we sometimes get the following error:

IDX10503: Signature validation failed. Token does not have a kid.
The key {guid} was not found in the key ring.

Logs show exceptions like:

System.Security.Cryptography.CryptographicException: The key {guid} was not found in the key ring.
at Duende.IdentityServer.Services.KeyManagement.DataProtectionKeyProtector.Unprotect(...)

🔍 What We've Tried:

Both Redis and a SQL database,

but still seeing key-related issues like:

❌ CryptographicException: The key {guid} was not found in the key ring

Verified Redis key presence (DataProtection-Keys*) via Redis CLI — keys exist.

Ensured same Redis connection string across all nodes.

Added logging to preload keys at startup using ISigningCredentialStore.

No fallback to blob storage is configured.

❓ What We Want to Know:

1. Why would a properly configured Redis key store still result in "key not found" errors across instances?

2. Are we missing a configuration to ensure all nodes load the key ring before token issuance/validation?

3. Is there a race condition with Duende’s in-memory caching of keys per instance?

4. What’s the best pattern to warm up key loading in a multi-instance Redis setup?

5. Can we configure IdentityServer or Data Protection to fail-fast if keys aren't loaded?

[wcabus]

The error you're seeing (The key {guid} was not found in the key ring.) is - as you suspect - caused by ASP.NET Core Data Protection when it can't find the specified key to protect or unprotect data. You're seeing this error when IdentityServer tries to sign or validate a token because its signing key material is protected using Data Protection.

I'll attempt to answer each of your questions:

1. Why would a properly configured Redis key store still result in "key not found" errors across instances?

   In theory, the Redis store could've been unavailable when the read attempt was made, or Redis didn't persist its data to survive a service restart (see this GitHub issue and Azure Cache for Redis - Premium persistence)|

   But the "key not found" can also be raised when the existing key material couldn't be decrypted.
   In your code sample showing Data Protection, did you omit the line ProtectKeysWith***, or is this the exact configuration you have in your codebase? In a production environment, especially when running multiple instances, we recommend using a shared certificate or Azure KeyVault to protect Data Protection's key material.
   Can you try adding the following code in your Program.cs to see what type of XmlEncryptor is being used?

   var options = app.Services.GetRequiredService<IOptions<KeyManagementOptions>>().Value;
   var logger = app.Services.GetRequiredService<ILogger<Program>>();
   logger.LogInformation("Data protection encryptor type: {EncryptorType}", options.XmlEncryptor is null ? "None" : options.XmlEncryptor.GetType().FullName);

2. Are we missing a configuration to ensure all nodes load the key ring before token issuance/validation?

   No, since Data Protection adds a hosted service which will attempt to load the key ring when the ASP.NET Core application starts, and will soft-fail if loading fails for whatever reason. The first attempt to protect or unprotect data will perform another key ring load.

3. Is there a race condition with Duende's in-memory caching of keys per instance

   This shouldn't be related. When IdentityServer loads the signing keys, they're loaded from their store (database typically), unprotected (using Data Protection, where the root cause is situated most likely) and then cached locally.

4. What's the best pattern to warm up key loading in a multi-instance Redis setup?

   If you're referring to Data Protection keys, then the answer is: these are already being loaded by the hosted service mentioned earlier.
   If you're referring to the signing keys: you can add your own hosted service to try and load the signing keys by injecting the IKeyManager and calling GetCurrentKeysAsync

5. Can we configure IdentityServer or Data Protection to fail-fast if keys aren't loaded?

   You can achieve this by looking at the Data Protection hosted service's code and perform similar logic after registering your services, creating the HTTP pipeline after builder.Build() and before app.Run() in your Program.cs file. To fail fast when IdentityServer can't load its signing keys, you can resolve an IKeyManager service instance and call GetCurrentSigningKeysAsync before app.Run().

[wcabus]

Another thing to verify: please check if your IdentityServer solution is using the same version of System.IdentityModel.* and Microsoft.IdentityModel.* NuGet packages. Different versions of the "Wilson" packages are known to cause issues when validating tokens.

[macupryk]

[wcabus]

While these versions are old (and deprecated), they are the same.

If you're certain that your Redis service's storage is being properly persisted across service restarts, then the issue may have been a fluke?
PersistKeysToStackExchangeRedis doesn't really do a lot: it replaces the XML storage type with a Redis-backed one, pushing new data protection keys to a Redis list object and enumerating over said list when retrieving the data protection keys again.

In your case, the enumeration may have timed out or was cancelled because of a temporary network issue. Unfortunately, there's nothing really we can do about that when it happens.

[macupryk]

I add this

private static void LogXmlEncryptorType(IHost host)
{
    using var scope = host.Services.CreateScope();
    var options = scope.ServiceProvider.GetRequiredService<IOptions<KeyManagementOptions>>().Value;
    var logger = scope.ServiceProvider.GetRequiredService<ILogger<Program>>();

    logger.LogInformation("Data protection encryptor type: {EncryptorType}", 
    options.XmlEncryptor is null ? "None" : options.XmlEncryptor.GetType().FullName);
}

Data protection encryptor type: None

[wcabus]

This means that your data protection keys are stored in plain format: everyone who can access your Redis cache will be able to decrypt cookies or IdentityServer's signature keys.

Even if access to the data store is limited, we highly recommend using a certificate, Azure KeyVault or equivalent services to protect the Data Protection keys. See https://learn.microsoft.com/en-us/aspnet/core/security/data-protection/configuration/overview?view=aspnetcore-9.0 for more details.

[macupryk]

I will definitely look into using cert.

The other thing I notice is that the keys are being left over (where I have to manually do delete from keys table).
This is also causing confusion.

So, I created a service to do the above clean.
Do you have any suggestions (best practices0, I would like to remove this service for now.

Much appreciated for sharing knowledge.

` public class KeyCleanupService : BackgroundService
{
private readonly IServiceProvider _serviceProvider;
private readonly ILogger _logger;

 public KeyCleanupService(IServiceProvider serviceProvider, ILogger<KeyCleanupService> logger)
 {
     _serviceProvider = serviceProvider;
     _logger = logger;
 }

 protected override async Task ExecuteAsync(CancellationToken stoppingToken)
 {
     while (!stoppingToken.IsCancellationRequested)
     {
         try
         {
             using var scope = _serviceProvider.CreateScope();
             var keyManager = scope.ServiceProvider.GetRequiredService<IKeyManager>();

             // Adjusted logic to handle missing Id property
             var allKeys = keyManager.GetAllKeys();
             var expiredKeys = allKeys.Where(k => k.IsExpired()).ToList(); // Assuming IsExpired() is a custom method

             foreach (var key in expiredKeys)
             {
                 _logger.LogInformation($"Deleting expired signing key.");
                 keyManager.RevokeKey(key.GetKeyIdentifier()); // Use a method to retrieve the key identifier
             }
         }
         catch (Exception ex)
         {
             _logger.LogError(ex, "Error occurred during signing key cleanup.");
         }

         // Wait for the next cleanup interval (e.g., 1 hour)
         await Task.Delay(TimeSpan.FromHours(1), stoppingToken);
     }
 }

}

// Extension method to check if a key is expired
public static class KeyExtensions
{
public static bool IsExpired(this IKey key)
{
// Replace with actual expiration logic if available
return false; // Placeholder logic
}

 public static Guid GetKeyIdentifier(this IKey key)
 {
     // Replace with logic to retrieve a unique identifier for the key
     throw new NotImplementedException("Provide logic to retrieve the key identifier.");
 }

}`