I know its too late to solve your initial problem but I have found a solution and want to leave it here.
Sun/Oracle servers like x4140 x4240 x4440 x4150 x4250 x4450 x4170 (and possibly some other) have ILOM 2 or ILOM 3 based on ASPEED AST2000 video chip.
AST2000 is combined video and SP card providing both video and ILOM capabilities to the server. At some point (after flash upgrade or just a power cycle) the server may have video garbled and BMC not responsive.
The problem is caused by a memory used with AST2000. Most of the servers seem to have two F generation MT46V32M16P -5B chips, you can find it right near AST2000 chip. These memory chips are known as self-destructing through time. To know more about this you can search for "Cisco Big Memory Problem", many Cisco network devices were suffering from that problem too.
Bad news are if you have obtained one of those servers they are very likely to fail in future. Good news are memory replacement is quite simple. I just have replaced them with hynix HY5DU121622CTP-J pair and got both good video and SP back.
Few words about replacement HY5DU121622CTP-J is 166 MHz DDR1, but MT46V32M16P-5B is 200 MHz. I found that ASPEED AST2000 is 166 MHz so I think it was a good replacement. Nevertheless next time I would use original MT46V32M16P-5B but you have to find J generation of them, not faulty F one. Also direct replacement can be HY5DU121622CTP-D43, but I did not tried them yet. You also have to double check if this information suit your motherboard.
You also have "soft" option of fixing the server - you can install video card in PCI-E x16 slot either temporary or permanent. This way you can enter BIOS setup and enable redirection to serial port. Your OS should also redirect console to serial port. This can be used if you do not need SP functionality.