I am once again trying to debug my Blazor Server App which uses Entity Framework to communicate with an SQL Server DB.
Update 8/21/23
I created a Minimal Repro consisting of:
- Program.cs with minimal configuration code required for Blazor, EF and Controllers.
- A DB Context with 1 table with two columns and a single record in the DB.
- A Controller page that takes 1 argument as an Account ID.
- A Service that uses the DB Context to get the record associated with the ID and return the value in the Email column.
This implementation will allow roughly 250 or so page hits before it crashes, again with no debugging information of any kind anywhere. Not in the Event Log, not in any of the IIS logs, no exception is thrown, nothing beyond a cryptic message stating that the Application Pool is unhealthy, and a response of 'service unavailable' in the browser.
My issue is that there is no debugging data to base an analysis on, thus I cannot determine what the cause is. Now that I have as barebones a minimal repro as possible it is clear that it is not my code, it must be a flaw in the underlying system. I say this because the definition of a 'minimal repro' is that I cannot remove a single line of code anywhere in the project and have it work.
I have these files if anyone is interested, but it is super easy to reproduce. Build a DB Table with two columns, one for ID, one for content, then build a Blazor/RF Project with one controller and one service to query the DB, then hit this repeatedly for around 250 hits.
My direct question is: How do I obtain debugging information from which to derive a solution?
Update 5/7/23:
I added some output from a log file I was able to generate. It records the death of the App Pool. Maybe there is a clue in there?
End Update
Update: I just removed all static methods and references and the behavior still persists. Running a page that interacts with the database will kill the App Pool after a few runs. I have now complied with all the suggestions given and it still crashes.
What now?
End update**
[2023-05-07T16:06:47.031Z, PID: 13784] [aspnetcorev2_inprocess.dll] Event Log: 'Application 'C:\BookTrakkerWeb' started successfully.'
End Event Log Message.
[2023-05-07T16:07:19.672Z, PID: 13784] [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::OnGlobalStopListening
[2023-05-07T16:07:19.673Z, PID: 13784] [aspnetcorev2.dll] Stopping application '/LM/W3SVC/1/ROOT'
[2023-05-07T16:07:19.674Z, PID: 13784] [aspnetcorev2_inprocess.dll] Stopping file watching.
[2023-05-07T16:07:19.674Z, PID: 13784] [aspnetcorev2_inprocess.dll] Stopping file watcher thread
[2023-05-07T16:07:19.675Z, PID: 13784] [aspnetcorev2_inprocess.dll] Stopping CLR
[2023-05-07T16:07:19.676Z, PID: 13784] [aspnetcorev2_inprocess.dll] Starting shutdown sequence 0
[2023-05-07T16:07:29.690Z, PID: 13784] [aspnetcorev2_inprocess.dll] Clr thread wait ended: clrThreadExited: 0
[2023-05-07T16:07:29.701Z, PID: 13784] [aspnetcorev2_inprocess.dll] Event Log: 'Failed to gracefully shutdown application 'MACHINE/WEBROOT/APPHOST/TST.BOOKTRAKKER.NET'.'
End Event Log Message.
[2023-05-07T16:07:29.702Z, PID: 13784] [aspnetcorev2_inprocess.dll] Canceling standard stream pipe reader
[2023-05-07T16:07:29.705Z, PID: 13784] [aspnetcorev2_inprocess.dll] Stopping in-process worker thread
[2023-05-07T16:07:30.708Z, PID: 13784] [aspnetcorev2.dll] ASPNET_CORE_GLOBAL_MODULE::Terminate
I have gone through all the previous resolutions which required me to pass a DBCOntextFactory into classes rather than an actual DBContext, so all DBContexts are short lived and perform a single Unit of Work.
I have checked in the Windows Event Log for these events and find none of them:
Evidence of repeated w3wp.exe crashes and Rapid-Fail Protection may be found in Windows Events, in the System log with Source=WAS.
Evidence of what causes the w3wp.exe to crash may be found in Windows Events, in the Application log: second-chance crashing exceptions with w3wp.exe.
The code runs fine in the local VS Webserver. I can hit the page in question as many times as I wish and it will remain functional. However, if I publish the app to my Test Server, it will display 'service unavailable' after one or two accesses. This is likely due to the App Pool failing somehow, but the mechanism is not clear because no information is reported.
I am not seeing any relevant entries in the Application or System sections of the Event Log.
No errors are being thrown or reported. Connection Strings:
"ConnectionStrings": { "production": "server=tcp:B68BBA8;Data Source=B68BBA8\SQLBT;Initial Catalog=booktrak_booktrakker;Persist Security Info=True; User ID=;Password=; TrustServerCertificate=True; MultipleActiveResultSets=True",
"development": "server=tcp:Geckoserver;Data Source=GECKOSERVER\SQLEXPRESSBT;Initial Catalog=booktrak_booktrakker; Integrated Security=True; Persist Security Info=True;User ID=;Password=;Encrypt=False;TrustServerCertificate=True; MultipleActiveResultSets=True" //Integrated Security=True; },
Example of a Static Method useage. The internal method may call the static method in order to avoid code duplication. I am open to other ways to solve the problem that I use this to solve. Note: I converted to non-static and it still dies, so it does not seem to be related to this pattern below.
public Account? GetAccount(Guid? userID)
{
try
{
return GetAccount(userID, _dbContextFactory, errReport);
}
catch (Exception ex)
{
errReport.LogErr(ex);
return null;
}
}
public static Account? GetAccount(Guid? userID, IDbContextFactory<BtDbContext> dbContextFactory, ErrorReporterService errReport)
{
try
{
using BtDbContext context = dbContextFactory.CreateDbContext();
Account? account = context.Accounts.SingleOrDefault(c => c.UserId.Equals(userID));
return account;
}
catch (Exception ex)
{
errReport.LogErr(ex);
return null;
}
}
Update:
I just observed a strange behavior in the VS Dev environment where this was thrown from the Output:
[2023-05-02T13:56:43.835Z] Error: Connection disconnected with error 'Error: WebSocket closed with status code: 1006 (no reason given).'.
[2023-05-02T13:56:46.836Z] Information: Normalizing '_blazor' to 'https://localhost:7008/_blazor'.
This begs the question - what is going on here? Why did the websocket close?
Interestingly, it survived and produced the data it loads from the database.
I can reload the page indefinitely in the VS Test Server, but only two or three times on the IIS Web Server.
End Update
I have reviewed the code many times now, and covered all the bases that were mentioned as possible causes:
1.) I use the Blazor/EF DBContext Factory to generate my DBContext local to where it is used and always meet the UoW principle. 2.) I apply 'using' to every instance of the DBContext I generate. 3.) I have tracked disposal of the dbcontext and it is being disposed.
My next step is to look at using the .NET Exception Handling classes to see if there are exceptions being thrown that I am not seeing. That is the major problem, I have no visibility into anything that resembles a cause. No errors are visible anywhere I look.
Here is some background from my earlier experiences. I had it working well for some time, then two days ago it started crashing again, and I have reviewed my changes and not seen anything - possibly because I had not run through the process more than once, so whatever is going on seems to involve multiple hits to the page.
Here is what I see in the F12 console (both files are present and accounted for, it seems like the Blazor server just dies):
What other options might I try here? How can I see what is causing the failures?
History of previous issues:
Program.cs
using BTOnlineBlazor.Areas.Identity;
using BTOnlineBlazor.Data;
using BTOnlineBlazor.App_Code;
//using BTOnlineBlazor.Shared;
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.Components.Authorization;
using Microsoft.AspNetCore.Components.Web;
using Microsoft.AspNetCore.Identity;
using Microsoft.AspNetCore.Identity.UI;
using Microsoft.Data.SqlClient;
using Microsoft.EntityFrameworkCore;
using System.Diagnostics;
using Microsoft.AspNetCore.StaticFiles;
using Microsoft.AspNetCore.Rewrite;
using Microsoft.AspNetCore.Http.Connections;
var builder = WebApplication.CreateBuilder(args);
// Add services to the container.
//var connectionString = builder.Configuration.GetConnectionString("DefaultConnection") ?? throw new InvalidOperationException("Connection string 'DefaultConnection' not found.");
var connectionString = builder.Configuration.GetConnectionString("production");
if (Debugger.IsAttached || Environment.MachineName.ToUpper().Equals("GECKOSERVER"))
connectionString = builder.Configuration.GetConnectionString("development");
ErrorReporterFactory.Instance.CreateErrorReporter().LogEventMessage("Connection String Assigned: {0}", connectionString ?? "not assigned");
builder.Services.AddDbContext<BtDbContext>(options =>
options.UseSqlServer(connectionString,
providerOptions => providerOptions.EnableRetryOnFailure()),
optionsLifetime: ServiceLifetime.Transient);
//builder.Services.AddDbContextPool<BtDbContext>(
// options => options.UseSqlServer(connectionString,
// providerOptions => providerOptions.EnableRetryOnFailure()));
//builder.Services.AddDbContext<AppManagerContext>(options =>
// options.UseSqlServer(connectionString), optionsLifetime: ServiceLifetime.Singleton);
builder.Services.AddDbContextFactory<BtDbContext>(options => options.UseSqlServer(connectionString));
ErrorReporterFactory.Instance.CreateErrorReporter().LogEventMessage("DBContextFactory Connection String: {0}", connectionString ?? "not assigned");
builder.Services.AddDatabaseDeveloperPageExceptionFilter();
builder.Services.AddDefaultIdentity<IdentityUser>(options => options.SignIn.RequireConfirmedAccount = true)
.AddEntityFrameworkStores<BtDbContext>();
builder.Services.AddRazorPages();
builder.Services.AddServerSideBlazor();
builder.Services.AddScoped<AuthenticationStateProvider, RevalidatingIdentityAuthenticationStateProvider<IdentityUser>>();
//builder.Services.AddSingleton<WeatherForecastService>();
builder.Services.AddHttpClient();
builder.Services.RegisterApplicationServices();
builder.Services.AddMvc().AddJsonOptions(options =>
{
options.JsonSerializerOptions.MaxDepth = 256;
options.JsonSerializerOptions.PropertyNameCaseInsensitive = true;
options.JsonSerializerOptions.DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull;
options.JsonSerializerOptions.DefaultBufferSize = 200000000;
});
builder.Services.AddServerSideBlazor().AddHubOptions(options =>
{
// maximum message size of 2MB
options.MaximumReceiveMessageSize = 2000000;
});
builder.Services.AddHttpContextAccessor();
var app = builder.Build();
// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
app.UseMigrationsEndPoint();
}
else
{
app.UseExceptionHandler("/Error");
// The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
app.UseHsts();
}
//var provider = new FileExtensionContentTypeProvider();
//provider.Mappings[".ashx"] = "text/html";
//app.UseStaticFiles(new StaticFileOptions { ContentTypeProvider = provider });
RewriteOptions urlOptions = new RewriteOptions().AddRewrite(@"^(.*).ashx$", "api/$1", true);
urlOptions.AddRewrite(@"^(.*).inf$", "api/ComputerInfo", true);
urlOptions.AddRewrite(@"AmazonLAPconsent.aspx", "AmazonLAPconsent", false);
urlOptions.AddRewrite(@"AccountReview.aspx", "AccountReview", false);
urlOptions.AddRewrite(@"IpnHandler.aspx", "api/IpnHandler", false);
urlOptions.AddRewrite(@"^(.*).aspx$", "api/$1", true);
app.UseRewriter(urlOptions);
//app.UseHandlerTrapper();
app.UseStaticFiles();
app.UseHttpsRedirection();
app.UseRouting();
if (Debugger.IsAttached || Environment.MachineName.ToUpper().Equals("GECKOSERVER"))
{
app.Use(next => context =>
{
ErrorReporterFactory.Instance.CreateErrorReporter().LogToConsole($"Found: {context.GetEndpoint()?.DisplayName}");
return next(context);
});
}
else
{
ErrorReporterFactory.Instance.CreateErrorReporter().LogEventMessage("Connection String: {0}", connectionString ?? "not assigned");
}
app.UseAuthorization();
app.MapControllers();
//app.MapBlazorHub();
app.MapBlazorHub(configureOptions: options =>
{
options.Transports = HttpTransportType.WebSockets | HttpTransportType.LongPolling;
});
app.MapFallbackToPage("/_Host");
// setup app's root folders
AppDomain.CurrentDomain.SetData("ContentRootPath", app.Environment.ContentRootPath);
AppDomain.CurrentDomain.SetData("WebRootPath", app.Environment.WebRootPath);
//app.WaitForShutdown();
app.Run();
ServiceInitializer.cs:
using BlazorDownloadFile;
using BTOnlineBlazor.Handlers;
using BTOnlineBlazor.Services;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.JSInterop;
using BlazorDownloadFile;
namespace BTOnlineBlazor.App_Code
{
public static partial class ServiceInitializer
{
public static IServiceCollection RegisterApplicationServices(this IServiceCollection services)
{
services.AddTransient<AccountManagerService>();
services.AddTransient<GetAmzKeysHandler>();
services.AddTransient<GetAmzRefreshTokenHandler>();
services.AddTransient<ErrorReporterService>();
services.AddTransient<ComputerManagerService>();
services.AddTransient<EventLogManagerService>();
services.AddTransient<AppManagerService>();
services.AddTransient<ListingSiteManagerService>();
services.AddTransient<SyncManagerService>();
services.AddTransient<ConnectionManagerService>();
services.AddTransient<ImageItManagerService>();
services.AddTransient<GetImageFolderHandler>();
services.AddTransient<ReRegisterComputerHandler>();
services.AddTransient<IEmailManagerService, EmailManagerService>();
services.AddTransient<AmazonPaymentsService>();
services.AddTransient<RateCodeManagerService>();
services.AddTransient<PaymentProcessorService>();
services.AddTransient<ConsoleDataFactory>();
services.AddTransient<AccountUtilitiesService>();
services.AddTransient<ComputerUtilitiesService>();
services.AddBlazorDownloadFile();
return services;
}
}
}
