You are getting it all wrong buddy...IPSEC is required for a secure communication between two machines.
Like you want to send a packet to other machine but you want that no one could possibly even determine what protocol you are using (tcp/udp.. etc) then you use this IPSEC. and it is not all over there is so much to explore about IPSEC.
openssl is you can say just a encrytion/authentication functions library.
A clear difference could be understood wh a little example.
Suppose you want to secure traffic between two machines so you create secure encrypted packet , send it to other machine there it needs to be decrypted based on security associations.All this is part of IPSEC Protocol.
While when encrypting the packet on your sending machine you may have used some C/Linux functions to encrypt the packet.This is where openssl comes in place.
Similarly on the other end when you will capture the packet and extract the required part then you can decrypt it using openssl function used on your machine.
I tried explaining it with my best ... hope it helped !!! If still you have any doubt do clear !!!