System.IO.FileNotFoundException thrown from Orleans.Serialization.Internal.ReferencedAssemblyProvider.<GetApplicationPartAssemblies>g__ExpandAssembly|

[nkosi23]

I have a basic Orleans application configured as follow:

Silo Project:

 builder
        .UseOrleans(fun builder ->
            builder.UseLocalhostClustering() |> ignore
            ())
        .UseConsoleLifetime()
    |> ignore

And client project (ie the Web Api project)

builder.UseOrleansClient(fun client ->

        client.UseLocalhostClustering() |> ignore
        ())
    |> ignore

When starting my application, i get the following exception:

Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'MyAssembly, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.

File name: 'MyAssembly, Culture=neutral, PublicKeyToken=null'
   at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, RuntimeAssembly requestingAssembly, Boolean throwOnFileNotFound)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Orleans.Serialization.Internal.ReferencedAssemblyProvider.<GetApplicationPartAssemblies>g__ExpandAssembly|5_3(HashSet`1 assemblies, Assembly assembly) in /_/src/Orleans.Serialization/Hosting/ReferencedAssemblyProvider.cs:line 204
   at Orleans.Serialization.Internal.ReferencedAssemblyProvider.<GetApplicationPartAssemblies>g__ExpandApplicationParts|5_1(IEnumerable`1 assemblies) in /_/src/Orleans.Serialization/Hosting/ReferencedAssemblyProvider.cs:line 188
   at Orleans.Serialization.Internal.ReferencedAssemblyProvider.GetApplicationPartAssemblies(Assembly assembly) in /_/src/Orleans.Serialization/Hosting/ReferencedAssemblyProvider.cs:line 172
   at Orleans.Serialization.Internal.ReferencedAssemblyProvider.AddAssembly(HashSet`1 parts, Assembly assembly) in /_/src/Orleans.Serialization/Hosting/ReferencedAssemblyProvider.cs:line 54
   at Orleans.Serialization.Internal.ReferencedAssemblyProvider.AddFromAssemblyLoadContext(HashSet`1 parts, Assembly assembly) in /_/src/Orleans.Serialization/Hosting/ReferencedAssemblyProvider.cs:line 87
   at Orleans.Serialization.Internal.ReferencedAssemblyProvider.GetRelevantAssemblies() in /_/src/Orleans.Serialization/Hosting/ReferencedAssemblyProvider.cs:line 23
   at Orleans.Serialization.ServiceCollectionExtensions.AddSerializer(IServiceCollection services, Action`1 configure) in /_/src/Orleans.Serialization/Hosting/ServiceCollectionExtensions.cs:line 40
   at Orleans.DefaultClientServices.AddDefaultServices(IClientBuilder builder) in /_/src/Orleans.Core/Core/DefaultClientServices.cs:line 129
   at Orleans.Hosting.ClientBuilder..ctor(IServiceCollection services, IConfiguration configuration) in /_/src/Orleans.Core/Core/ClientBuilder.cs:line 21
   at Microsoft.Extensions.Hosting.OrleansClientGenericHostExtensions.AddOrleansClient(IServiceCollection services, IConfiguration configuration) in /_/src/Orleans.Core/Hosting/OrleansClientGenericHostExtensions.cs:line 204
   at Microsoft.Extensions.Hosting.OrleansClientGenericHostExtensions.AddOrleansClient(IServiceCollection services, IConfiguration configuration, Action`1 configureDelegate) in /_/src/Orleans.Core/Hosting/OrleansClientGenericHostExtensions.cs:line 183
   at Microsoft.Extensions.Hosting.OrleansClientGenericHostExtensions.UseOrleansClient(IHostApplicationBuilder hostAppBuilder, Action`1 configureDelegate) in /_/src/Orleans.Core/Hosting/OrleansClientGenericHostExtensions.cs:line 86

Apparently, Orleans tries to load an assembly and complains that the file does not exist while it does exist.

What puzzles me even more is that this assembly does not have any ApplicationPart attribute defined in it, and does not even reference Orleans, however the following section of the internal Orleans method:

private static IEnumerable<Assembly> GetApplicationPartAssemblies(Assembly assembly)
        {
            if (!assembly.IsDefined(typeof(ApplicationPartAttribute)))
            {
                return Array.Empty<Assembly>();
            }

            return ExpandApplicationParts(
                new[] { assembly }.Concat(assembly.GetCustomAttributes<ApplicationPartAttribute>()
                    .Select(name => Assembly.Load(new AssemblyName(name.AssemblyName)))));

Does not enter the if branch to return an empty array as i'd have expected it to do. It instead enters the ExpandApplicationParts.

And what is even more strange, is that the assembly parameter of the GetApplicationPartAssemblies(Assembly assembly) method is set to precisely the dll that supposedly does not exist.

The dll in question contains pre-generated code for an unrelated middleware framework sitting on top of ASP.NET Core. As such, the project of the middleware has a reference to the Web Api project. To avoid a circular dependency, the Web Api project does not have any reference to the middleware project, it just manually copies the middleware dll to its output directory using an msbuild <Content CopyToPublishDirectory="Always" CopyToOutputDirectory="Always"> directive (and not a ProjectReference directive). Because in the production environment, the pre-built types are used for performance reason.

None of this should be relevant to Orleans, but I mention it just in case as I find suspicious that this problem only occurs with the middleware assembly. I am scratching my head on this one and it is blocking as it crashes the application startup.

[nkosi23]

Apparently deleting the dll from the output directory of the Web Api project fixes the issue.
However this is actually scary, since the dll is guaranteed to be present in production builds...

There is something going on with the way Orleans discovers assemblies. I tried looking for an attribute that would explicitly instruct Orleans to ignore the Middleware assembly (even though it shouldn't include it in the first place) but without success.