ServiceDiscovery: each ServiceEndpointWatcher lifecycle permanently leaks a change-token registration on IConfiguration

[ismailbennani]

Description

ServiceEndpointResolver evicts a service's ServiceEndpointWatcher when the name has not been resolved recently, using a cleanup timer with a 10 second period. The next resolution of that name creates a fresh watcher, which performs an initial refresh. Every one of those refreshes permanently adds a callback registration to the application's root IConfiguration reload token, and the registration is never released, neither when the watcher is disposed nor when it is evicted.

An application resolving a service name less often than roughly every 10 seconds therefore leaks one CompositeChangeToken, one CancellationTokenSource, two CancellationTokenSource+CallbackNodes and their supporting lists and arrays on every resolution, all rooted from the configuration reload token for the lifetime of the process. Resolving the same name more frequently avoids the leak, because the entry stays marked as recently used and its watcher is never evicted.

ServiceEndpointWatcher disposes its previous change-token registration on each refresh and again in DisposeAsync. Disposing a registration on a CompositeChangeToken releases the consumer's callback but leaves the composite's proxy registrations on its inner tokens in place; CompositeChangeToken releases those only when it fires, and a configuration reload token in a typical deployment never fires. ConfigurationServiceEndpointProvider contributes that inner token on every populate, on both of its branches.

Reproduction Steps

dotnet new console
dotnet add package Microsoft.Extensions.ServiceDiscovery --version 10.6.0
dotnet add package Microsoft.Extensions.Hosting

using System.Reflection;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.ServiceDiscovery;

static long Registrations(IConfigurationRoot root)
{
    var token = root.GetReloadToken();
    var cts = (CancellationTokenSource?)token.GetType()
        .GetField("_cts", BindingFlags.NonPublic | BindingFlags.Instance)!.GetValue(token);
    var regs = typeof(CancellationTokenSource)
        .GetField("_registrations", BindingFlags.NonPublic | BindingFlags.Instance)!.GetValue(cts!);
    if (regs is null) return 0;
    return (long)regs.GetType()
        .GetField("NextAvailableId", BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance)!
        .GetValue(regs)!;
}

var builder = Host.CreateApplicationBuilder();
builder.Services.AddServiceDiscovery();

using var host = builder.Build();
var root = (IConfigurationRoot)builder.Configuration;
var resolver = host.Services.GetRequiredService<ServiceEndpointResolver>();

for (var i = 1; i <= 8; i++)
{
    await resolver.GetEndpointsAsync("http://some-service", CancellationToken.None);
    Console.WriteLine($"resolution {i}: {Registrations(root)} registrations");
    await Task.Delay(TimeSpan.FromSeconds(25));   // longer than the 10 second cleanup period
}

Output, one registration added per resolution and never released:

resolution 1: 5
resolution 2: 6
resolution 3: 7
resolution 4: 8
resolution 5: 9
resolution 6: 10
resolution 7: 11
resolution 8: 12

Expected behavior

Registrations on IConfiguration's reload token stay bounded. Creating and discarding N watchers for a service name leaves O(1) registrations, not O(N).

Actual behavior

Registrations grow by one per watcher lifecycle and are never released.

In a .NET 10 service resolving 9 service names from a background loop on a 30 second timer, so that every name is evicted between uses and re-created on each pass, this ran at ~21 leaked registrations per minute, with generation 2 climbing ~15.7 MiB/day, zero drawdown across 22 gen2 collections and flat fragmentation.

A dump after 63 hours of uptime showed about 41 MB, roughly half the managed heap, in 77,182 CompositeChangeToken, 152,782 CancellationTokenSource+CallbackNode, 76,463 CancellationTokenSource, 76,424 CancellationTokenSource+Registrations, and ~76,350 each of IChangeToken[], IDisposable[], List<IChangeToken> and List<IDisposable>.

All of it hangs off a single CancellationTokenSource, ConfigurationManager._changeToken._cts, whose Registrations.NextAvailableId read 76,339 with the whole callback list still linked. Each leaked composite's own CTS reads Callbacks = null, NextAvailableId = 2: the watcher registered once and disposed correctly, and what is retained is the composite's proxy registration on its inner token.

Allocation stack, from dotnet-trace collect --providers Microsoft-Windows-DotNETRuntime:0x1:5:

System.Collections.Generic.List`1[System.__Canon].AddWithResize(!0)
Microsoft.Extensions.ServiceDiscovery.Configuration.ConfigurationServiceEndpointProvider.PopulateAsync(IServiceEndpointBuilder, CancellationToken)
Microsoft.Extensions.ServiceDiscovery.ServiceEndpointWatcher+<RefreshAsyncInternal>d__23.MoveNext()
System.Threading.ThreadPoolWorkQueue.Dispatch()
System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

Regression?

No response

Known Workarounds

No response

Configuration

• Microsoft.Extensions.ServiceDiscovery 10.6.0, base package only, so the registered providers are Configuration and PassThrough.
• .NET 10.0.10. Observed in a Linux container (mcr.microsoft.com/dotnet/aspnet:10.0, x64) and reproduced on Windows with the console app above.

Other information

The mechanism, with the source involved.

1. The resolver evicts idle watchers every 10 seconds, src/Libraries/Microsoft.Extensions.ServiceDiscovery/ServiceEndpointResolver.cs:

private static readonly TimerCallback s_cleanupCallback = s => ((ServiceEndpointResolver)s!).CleanupResolvers();
private static readonly TimeSpan s_cleanupPeriod = TimeSpan.FromSeconds(10);
...
if (resolver.CanExpire() && _resolvers.TryRemove(name, out var _))

A name resolved less often than that gets a new ServiceEndpointWatcher, and therefore a fresh initial refresh, on its next resolution.

2. The configuration provider contributes the configuration root's reload token on every populate, on both branches, Configuration/ConfigurationServiceEndpointProvider.cs:

var section = _configuration.GetSection(_options.Value.SectionName).GetSection(_serviceName);
if (!section.Exists())
{
    endpoints.AddChangeToken(_configuration.GetReloadToken());
    Log.ServiceConfigurationNotFound(_logger, _serviceName, $"{_options.Value.SectionName}:{_serviceName}");
    return default;
}

endpoints.AddChangeToken(section.GetReloadToken());

Both are the same object, since ConfigurationSection.GetReloadToken() delegates to the root (dotnet/runtime, Microsoft.Extensions.Configuration/src/ConfigurationSection.cs):

public IChangeToken GetReloadToken() => _root.GetReloadToken();

3. The builder wraps the tokens in a composite unconditionally, ServiceEndpointBuilder.cs:

public ServiceEndpointSource Build()
{
    return new ServiceEndpointSource(_endpoints, new CompositeChangeToken(_changeTokens), _features);
}

4. The watcher registers on that composite, ServiceEndpointWatcher.RefreshAsyncInternal:

else if (endpoints.ChangeToken.ActiveChangeCallbacks)
{
    _changeTokenRegistration = endpoints.ChangeToken.RegisterChangeCallback(
        static state => _ = ((ServiceEndpointWatcher)state!).RefreshAsync(force: false), this);
}

ConfigurationReloadToken.ActiveChangeCallbacks is always true, so this branch is always taken and the polling timer is always disabled.

5. CompositeChangeToken releases its inner registrations only when it fires, dotnet/runtime, Microsoft.Extensions.Primitives/src/CompositeChangeToken.cs. RegisterChangeCallback calls EnsureCallbacksInitialized():

IDisposable disposable = ChangeTokens[i].RegisterChangeCallback(_onChangeDelegate, this);
_disposables.Add(disposable);

_disposables is disposed only inside the static OnChange handler, and the type has no public Dispose.

The watcher is created, registers on composite C, and C registers a proxy on the configuration reload token. The watcher is later evicted; DisposeAsync disposes the watcher's registration on C, which leaves C's proxy registration on the reload token in place. C, its CTS, its callback nodes and its lists stay rooted from the application's configuration, and the cycle repeats on the next resolution.