There certainly is an improvement in TLS negotiation by negotiating it closer to the user. TLS negotiation is still a bunch of packets that happens before the first byte of actual data is sent, so getting it closer to the user makes the session establishment faster. This is just basic speed of light stuff - the less physical distance the packets have to travel, all else being equal, the faster it will be.
The question is: Does this really matter? Should you care? In the case of Stack Overflow we establish thousands of user TLS connections a minute, so when you add up millions of TLS sessions over 24 hours, every millisecond counts.
However if you're just getting a few hits an hour, there's not a huge amount of cumulative gains for session estalishment by moving the SSL to your CDN.
In terms of connections from the CDN back to the origin, CloudFlare should use the same few sessions (from each PoP) so it won't be re-establishing on the clients behalf. CloudFlare used to (maybe they still do?) have an WAN accellerator called "Railgun", which makes terminating TLS on the CDN even more adventageous.
However I would think twice about cutting off your Lets Encrypt on your origin. You probably still want to encrypt between CloudFlare and your local web server. It won't make any performance difference, and this way you're still stopping snooping from Cloudflare to your origin.