As well as Tobi's suggestion about openssl versions, you may not be installing the libraries into the correct locations. If you just run "./configure; make; make install" it will install into /usr/local/lib by default, for example. Even if you set the prefix properly, you might still be installing into /usr/lib/ rather than /usr/lib64, and ldconfig might not be set up properly
However, your distribution almost certainly has updated libraries available that do not involve compilation. Hand-compiling openssl now more or less commits you to hand-compiling it from now on, so this is really a last-resort method. No offence intended here, but if you're having problems getting this to work, it's not the right approach, at least not on a server you care about.
If you are running RedHat, CentOS, Fedora or any other distro that uses .rpm packages, try running 'sudo yum update'. If you are running debian, ubuntu, mint or any other distro that uses .deb packages, try running 'sudo apt-get update; sudo apt-get dist-upgrade'. If you need more help here, try googling for how to update your distro - there are heaps of people posting articles on how to upgrade various linux distributions for GHOST or Heartbleed or poodle, so one of those will have instructions you can use.