SignalR exception kills the service/app - System.Threading.Tasks.TaskExceptionHolder.Finalize()

Question

I've been battling this issue for sometimes. A windows service targeting .NET 4.0 creates a .NET signalr client but gets killed after sometimes for System.Threading.Tasks.TaskExceptionHolder.Finalize() due to aggregate exception thrown within the task. Here is the gist of the setup - other details are omitted.

Windows service:

onStart()
{
   try {
         var task1 = new Task(SignalrEnv.WireUp(),TaskCreationOption.LongRunning);
         task1.Wait();
   }
   catch(Exception ex)
   {
      //log the exception
   }
}

SingalrEnv class:

public void WireUp(){
   //create hubconnection
   //Create hubproxy

   try 
   {
         var task = Hubconnection.Start();
         task.Wait();
   }
   catch(AggregateException agex)
   {
       //observe the inner exceptions so the service won't crash
   }
   catch(Exception ex)
   {
       //Log the exception
   }
}

However, when Signalr throws exceptions inside the signalr client framework like OnError-System.TimeoutException or any network exceptions, the windows service ends up getting killed. I even tried handling the Taskscheduler.UnObservedTaskException but no avail. So it seems the exceptions are thrown in another thread and aren't handled there. With my setup, I should be capturing the exceptions and observing - preventing the crash. What am I missing in handling exceptions in TPL and signalr?

Note: On systems with .net 4.5, the service has no problem - no crash - since Microsoft made it a default behavior. One would have to enable the crashing behavior explicitly in web.config.