Cycle 2's caption support is very simple, as powerful and flexible as it is as a slider, like with one or two other use cases as soon as you need a little more than the absolute basic example given, you'll need to dig in a little deeper.
The way this can be solved, and create very powerful full featured "caption" support is to bypass the built in captions all together and rather use Cycles custom events to trigger your own functions. So step one, remove any caption specific cycle stuff.
Have a look at the Events API to see whats available (you can do a quite a lot of cool stuff if you get to grips with these). Cycle2 Events API
Secondly, you can continue to use the alt tag on your image, or use a hidden element that you can grab any html content from. This would be my preffered markup,
<div class="cycle-slideshow"
data-cycle-fx=scrollHorz
data-cycle-slides="> .slide"
>
<div class="slide">
<a href="/whatever"><img src="whatever/jpg"></a>
<p class="slide-caption" style="display:none;">
Hi, I'm a super fancy caption,
<a href="somelink">with anything</a>
you want in it :)
</p>
</div>
</div>
<div class="fancy-caption-box">
<div class="inner"></div>
<!-- You'll insert your caption here with jQuery, put the slider and this box in a wrapper element if you need to overlay your caption elements -->
</div>
Next you're going to want to hook into two events, (you could get fancier, and use three but lets keep it simple to start with). You'll need to update your caption firstly when the slider loads, and secondly every time the slider updates the slide. I assume you're using jQuery and will know where to put this.
<script>
<!-- when the slideshow has been fully initialized -->
$('.cycle-slideshow').on('cycle-initialized', function(event, optionHash) {
var capBox = $('.fancy-caption-box');
var firstCaption = $('.cycle-slideshow').children('.slide').eq(0).find('.slide-caption').html();
capBox.children('.inner').fadeOut().html(firstCaption).fadeIn();
});
<!-- when each slide changes -->
$('.cycle-slideshow').on('cycle-before', function(event, optionHash, outgoingSlideEl, incomingSlideEl, forwardFlag) {
var capBox = $('.fancy-caption-box');
var caption = $(incomingSlideEl).find('.slide-caption').html();
capBox.children('.inner').fadeOut().html(caption).fadeIn();
});
</script>
This is just a very quick and rough sample that approximates what I've done in my project and hasn't been tested, if you can't find your way through it I'll set up a jsfiddle with something you can see working. You'll of course also need to style this to your tastes.
The nice thing about this approach is you can use absolutely anything you want in your caption block, and secondly use any effects you like on the captions, as I mentioned, rolling in more events and callbacks will allow you to do even more with this.