There are two things to consider in this situation:
and
The performance of a GPU is strongly influenced by the pattern in which data is addressed in accessed from memory. A ray caster casts its ray from front to back of the view (or in the opposite direction, depending on the implementation and internal blending mode) so depending on from which side you look at a 3D texture you get completely different access patterns. And that can have a very large influence.
3D textures consume very large amounts of memory. OpenGL uses an abstract memory model, where there's no explicit limit on the size of objects like textures. Either loading them works or not. And if a driver can manage it, you can load textures larger than what fits into GPU memory. Effectively the GPU memory is a cache for OpenGL data. And if your 3D texture happens to be larger than the available GPU memory it might be, that the OpenGL implementation swaps out texture data while its being accessed at rendering. If your access pattern is lucky, this swapping processed nicely fits "into the gaps" and can sort of stream the texture data into the cache (by which I mean GPU memory) right before its required, without stalling the rendering process. However a different access pattern (other view) will not work well with on demand swapping of data, trashing performance.
I suggest you reduce the size of your 3D texture.