I figured out how to do this, so here's my solution.
In your HTML, you can have these to create the controllers (this should be inside an a-scene
element):
<a-entity laser-controls="hand: left" raycaster="showLine: true; objects: .clickable;"></a-entity>
<a-entity laser-controls="hand: right" raycaster="showLine: true; objects: .clickable;"></a-entity>
These should also render with the actual controller models in the scene, and each have a laser pointer.
This is what it looks like with the Oculus Touch controllers (ignore the other stuff in the view):

As new types of headsets come out and are supported by A-Frame (e.g. the Valve Index controllers aren't supported yet), the laser-controls component should automatically be updated to support them.
See the docs for a bit more information on how to use controllers in your A-Frame scene.
I still haven't figured out exactly how to make it possible to click on buttons or objects in the environment using the laser, I'll need to figure that out next.