Service Fabric fails to find resource file

Question

I am having a problem deploying a microservice application to a remote Service Fabric cluster in Azure.

The application consists of 5 services, 4 of which manage to start and their state is displayed as Ready. However one fails with following error (state varies between InBuild and Error on all nodes):

'System.RA' reported Warning for property 'ReplicaOpenStatus'.
Replica had multiple failures during open on _nt1bm_5. API call: 
IStatelessServiceInstance.Open(); Error = 
System.Resources.MissingManifestResourceException (-2146233038)
Could not find any resources appropriate for the specified culture or the 
neutral culture.  Make sure 
"Microsoft.ServiceFabric.Services.Communication.AspNetCore.SR.resources" was 
correctly embedded or linked into assembly 
"Microsoft.ServiceFabric.AspNetCore" at compile time, or that all the 
satellite assemblies required are loadable and fully signed.
at System.Resources.ManifestBasedResourceGroveler.HandleResourceStreamMissing(String fileName)
   at System.Resources.ManifestBasedResourceGroveler.GrovelForResourceSet(CultureInfo culture, Dictionary`2 localResourceSets, Boolean tryParents, Boolean createIfNotExists, StackCrawlMark& stackMark)
   at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo requestedCulture, Boolean createIfNotExists, Boolean tryParents, StackCrawlMark& stackMark)
   at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo culture, Boolean createIfNotExists, Boolean tryParents)
   at System.Resources.ResourceManager.GetString(String name, CultureInfo culture)
   at Microsoft.ServiceFabric.Services.Communication.AspNetCore.AspNetCoreCommunicationListener.GetEndpointResourceDescription(String endpointName)
   at Microsoft.ServiceFabric.Services.Communication.AspNetCore.KestrelCommunicationListener.GetListenerUrl()
   at Microsoft.ServiceFabric.Services.Communication.AspNetCore.AspNetCoreCommunicationListener.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.ServiceFabric.Services.Runtime.StatelessServiceInstanceAdapter.OpenCommunicationListenersAsync(CancellationToken cancellationToken)
   at Microsoft.ServiceFabric.Services.Runtime.StatelessServiceInstanceAdapter.System.Fabric.IStatelessServiceInstance.OpenAsync(IStatelessServicePartition partition, CancellationToken cancellationToken)
For more information see: https://aka.ms/sfhealth

A few notes:

All the services use almost exactly the same host initialization (ports vary)
The application is configured to use HTTPS only
The application in written using .NET Core 2.2
The current Service Fabric version is 6.4.637.9590
The host's operating system is Windows Server (probably 2012)
On the local cluster, it works just fine
Using .NET Core self-hosting, everything works

What might be the cause of it?

How to debug such errors?

Is the source code of Service Fabric available somewhere to see code that fails?

If you need any additional information just ask :)

The SF source is available at github, but it's C++, and cross-platform, and incredibly hard to slow through. I doubt that will lead you anywhere. Try this maybe? https://github.com/Microsoft/aspnet-api-versioning/issues/331 — Svend, Feb 08 '19 at 10:01
Try updating the packages to the version `3.3.638`, there were some [issues](https://github.com/MicrosoftDocs/azure-docs/issues/24021) with packages from the most recent release. — Diego Mendes, Feb 08 '19 at 15:14

score 0 · Answer 1 · answered Feb 11 '19 at 10:58

I found what was the reason for this error!

Namely this error:

Could not find any resources appropriate for the specified culture or the 
neutral culture.

In the solution we had peculiar project structure:

src:
- self-hosted project (.NET Core app, all the controllers were here, as was Startup etc.)
- Service Fabric Host (it used controllers and startup from self-hosted project, had its separate Program.cs and web host builder - suitable for Service Fabric)

Both the projects had different appsettings.json - but the names were the same (!!!)

When building and packing locally with Visual Studio everything was fine, but when building on Azure DevOps (regardless of used build type: dotnet build, Visual Studio build) strange error would occur.

In the package created for deployment (build .sfproj with -t:Package) even though we were referencing ServicFabric project appsettings.json were sometimes taken from self-host project. (which was different than appsettings.json from ServiceFabric project) The error came from trying to read non-existant entry in appsettings.json

TLDR;

The error came from trying to read non-existant entry in appsettings.json

Our current solution is to use different names for appsettings.json from different projects. If we find any better solution I will update the answer. Also any help on this matter would be appreciated.

Service Fabric fails to find resource file

1 Answers1