AFAIK, you can't do this at layer 3 as an ELB will allow access from anywhere (0.0.0.0/0).
If you're running Apache and can find a specific header that cloudfront uses/sets then you could do this at layer 7 using mod_headers.
According to http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html cloudfront will set the Header Via
to 1.1 alphanumeric-string.cloudfront.net, so you could match this in your virtualhost by doing something like:
SetEnvIf Via "^1\.1\ [a-z0-9]+\.cloudfront\.net$ VIA_CLOUDFRONT
<LocationMatch /origin/>
Options -Indexes
Order deny,allow
Deny from all
# allow from cloudfront only
Allow from env=VIA_CLOUDFRONT
</LocationMatch>