2

I have a little pickle here that I cannot seem to find a solution for;

As of a few days ago, my backups (VeeamB&R-9.5) started failing with the following error:

Veeam output;

Error: Cannot authenticate user. Soap fault. Cannot complete login due to an incorrect user name or password.Detail: 'InvalidLoginFault xmlns="urn:internalvim25" xsi:type="InvalidLogin"/InvalidLoginFault', endpoint: '' SOAP connection is not available

And also;

Task failed. Error: Cannot complete login due to an incorrect user name or password.

Now, nothing has changed (credential wise or config) prior to these errors. When I try logging into vCenter, it does not accept my credentials. I need to reboot the vCenter server in order to be able to login. Once the vCenter server has rebooted, backups run without issues - until it hits certain jobs, and it is not always the same jobs. It then gets stuck in this stale error state again.

So, I then drilled into Event Viewer of both vCenter and Veeam server and found this;

From the vCenter server:

The description for Event ID 4227 from source Tcpip cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

The handle is invalid

And;

The description for Event ID 113 from source MSiSCSI cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

0xefff0003 *172.20.20.5 0003260 ROOT\ISCSIPRT\0000_0

The handle is invalid

Also;

The description for Event ID 70 from source iScsiPrt cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

\Device\RaidPort1

The handle is invalid

on the Veeam server, I get;

The network interface "vmxnet3 Ethernet Adapter" has begun resetting. There will be a momentary disruption in network connectivity while the hardware resets.

Reason: The network driver detected that its hardware has stopped responding to commands. This network interface has reset 1 time(s) since it was last initialized.

I have tried updating VMTOOLS on the Veeam server - no effect.

I have drilled into the physical hosts diagnostics for the Veeam server - all good.

Also drilled into the SAN in question - all is green on that side as well.

The only thing that seems to fix the issue is a reboot of the vCenter server - but the issue just comes back after certain jobs are run.

Any pointers would be greatly appreciated!!

Thanks in advance.

d0nde
  • 31
  • 5

1 Answers1

1

So after manually triggering backup jobs and watching event viewer closely, all signs pointed to the Veeam server running W2016. I have 2 Veeam servers one which runs on 2012R2 and one that runs on 2016. The 2012 server was generating no events in the logs.

I was about to remove that security update that was previously installed before I was prompted to download KB4345418 which was released yesterday - https://support.microsoft.com/en-ca/help/4345418/windows-10-update-kb4345418

Applied it and sure enough, no backup jobs failed last night and event viewer stopped spitting flags about tcpip and such!

Hope this thread helps others out there.

Thanks for the heart attack Microsoft!

d0nde
  • 31
  • 5