Check out my HOW-TO Install EMS 8 on CentOS 6
(CentOS is a compatible free alternative to RedHat 6).
But to answer more directly :
You seem to be trying to install on linux with the Windows package.
TIBCO provides versions for Linux (and other OSes) in separate files.
When downloading your files from TIBCO, you can take the occasion to read the README file for the product.
In it, you will find the list of supported distributions.
As of now (EMS 8.2, Nov 2014), supported OS are:
- Mac OS X 10.9.x
- HP HP-UX 11i
- IBM AIX 6.1, 7.1
- Microsoft Windows 7 SP1, 8, Vista SP2
- Microsoft Windows Server 2003, 2008 SP2, 2012, 2012 R2
- Novell SUSE Linux Enterprise Server 11.3
- Oracle Solaris 11
- Red Hat Enterprise Linux Server 5,6 & 7
- Sun Solaris 10
NOTE : The command for running the installer in Linux is usually : TIBCOUniversalInstaller-lnx-x86.bin