It would be interesting to see someone pull this off with Flash and Javascript. It entirely depends how frequently you would have to do the calculations.
If you only need to perform one iteration of calculations (essentially, one "render" pass or one draw call), you may be able to setup your VertexShader3D and FragmentShader3D like you would normally for any Stage3D project in Flash, and then return the results from the BitmapData of that render:
//You probably want this to be the same dimensions as your context3D dimensions:
_context3D.drawToBitmapData(yourBitmapDataDestination);
Then you could pass the information to Javascript. I'm not sure if BitmapData can be passed directly, but I'm quite positive you will have to serialize the data first (as an Array, or String, etc.). You'll have to look at how the ExternalInterface
class is used to communivate between Flash and Javascript.
Once you got your crunched numbers in JavaScript, do whatever you have to do with those values!
This exchange of information between Flash, the GPU, and JavaScript might not be optimal if you're seeking a solution for a process that needs to be executed "per frame" (like a game-loop, updating every n-frames).
Note: Sorry for the incomplete answer, but I mostly just wanted to guide you into a possible solution if you still were considering Flash as your point of access to the GPU.