Gluster can help you implement some very cheap storage solutions in which every node is both a compute node and a storage node.
With Nexenta you are looking more at a NAS type architecture, which means an additional server which becomes a single point of failure unless you build a storage cluster.
With Gluster you can build a huge distributed file system very cheaply. It's great for HA.
ZFS/Nexenta has a lot of great features that you don't get with Gluster. Data-deduplication and features that detect "silent" data corruption come to mind.
If you want to go cheap go Gluster. If you want something that is cheap compared to Netapp go Nexenta.