1

I am trying to install Sybase ASE 12.5.4 on a Sun OS Solaris 11 machine but I am getting the following messages when launching the setup file :

-bash-4.1$ ./setup
InstallShield Wizard

Initializing InstallShield Wizard...

Searching for Java(tm) Virtual Machine...
Bad System Call (core dumped)

Would you have any idea of what could cause this? I want to run a trace on this to see what's missing exactly but I cannot find how to.

Thank you for your help. Cheers

mentinet
  • 744
  • 3
  • 9
  • 23
  • Consider asking your question at http://unix.stackexchange.com/ or http://serverfault.com/ . You also have a coredump file (usually written to working directory of failed app, otherwise check `coreadm`), check it by running `mdb core` and use `$c` command in debugger -- it will give you a clue, what system call you need. – myaut May 07 '15 at 10:40
  • I would suggest to check the Sybase compatibility matrix because Solaris 11 was released after ASE 12.5.4. They might be incompatible. – Vince May 07 '15 at 13:05
  • Thank you for your reply myaut. `mdb core` `finished with the following lines: ffbfee08 access+8(baeb9, 89bb4, ba4b0, ba4b0, 0, 9) and ffbfef00 __1cMPlatformImplWcreateTempSubDirectory6F_nHSString__+0x30(baeb9, 1, 0, b8800, 0, b99a8). Any idea of what this could mean? Thank you – mentinet May 07 '15 at 13:08
  • [Source that the access syscall was removed in Oracle Solaris 11](https://docs.oracle.com/cd/E26502_01/html/E28556/gkzlf.html#gkzip). – Tim Oct 06 '16 at 00:21

2 Answers2

3

Do not use this hacks on production systems!

Many system calls were deprecated in Solaris 11, access seem to be deleted:

$ grep access /usr/include/sys/syscall.h 
#define SYS_faccessat   45

Some of them are still accessible from Solaris 10 brand zones, some seem to be gone forever. Instead you have to use "at"-functions with AT_FDCWD param, in your case it is faccessat. That change was hidden by libc, but if Sybase brings their own libc things go bad, if they statically link with it, it is even worse.

If access is a shared library function in their own libc, you can still workaround that by using LD_PRELOAD hack. Write the following source:

#include <unistd.h>
#include <sys/fcntl.h>

int access(const char *fname, int amode) {
        return (faccessat(AT_FDCWD, fname, amode, 0));
}

Compile it with GCC or Solaris Studio (depends on what Sybase used, probably Solaris Studio) into shared library:

$ gcc -shared -fPIC xaccess.c -o libaccess.so

Export LD_PRELOAD variable: 

$ export LD_PRELOAD=./libaccess.so

And pray for luck because it may fail at another system calls, or you used wrong compiler, etc. Note that Sybase binaries may not run correctly too!

Have I warned about hacks on production systems? Instead, use brand Solaris 10 zones or Solaris 10 LDoms/VMs

myaut
  • 11,174
  • 2
  • 30
  • 62
  • It's C code - GCC or Solaris Studio won't matter. What will matter is 32- and 64-bit libraries. Set `LD_PRELOAD` with a 32-bit library on Solaris, and 64-bit binaries won't run. Set it with a 64-bit library, and 32-bit binaries won't run. Use either `LD_PRELOAD_64` or `LD_PRELOAD_32`. – Andrew Henle May 08 '15 at 00:23
  • Thank you for the reply. I am quite surprised though, my client confirms that they have this Sybase version on a Solaris 11 system. I am running Solaris 11.2 (sparcv9), maybe it is deprecated since 11.1 or 11.2? I will try the workaround and try with other Solaris versions as well – mentinet May 11 '15 at 08:22
  • I tried the workaround now and it won't work. I created the C file and compiled it with gcc like you suggested to do. I exported the variable and ran the setup again, I am getting the same message. I tried to export it in `LD_PRELOAD_64` as well but in vain. And I also tried running `LD_PRELOAD_64=/tmp/libaccess.so ./setup` but it won't solve the issue. – mentinet May 11 '15 at 08:52
  • @mentinet: Why don't you use `LD_PRELOAD` like I said, at least we will see if library needs to be 32-bit. Also, like I said, there are many factors affecting my method, it is quite generic. – myaut May 11 '15 at 08:55
  • I did run `export LD_PRELOAD=/tmp/libaccess.so` and then the setup but it doesn't work, it does just the same. Although, when I first assign the preload in my root environnement, then did `ls` I had the following message `ld.so.1: ls: fatal: /tmp/libaccess.so: wrong ELF class: ELFCLASS32 Killed` (I don't have it in my sybase environnement). I unfortunately have no idea what I am doing, so thanks a lot for your help. Any suggestion on what this means and how I could adapt your code to hopefully make it work in my situation? Thanks – mentinet May 11 '15 at 09:09
  • Add `-m32` to compilation options when compiling `libaccess.so` – myaut May 11 '15 at 09:11
  • Doesn't seem to change anything. I recomplied it with this option (tried from sybase user as well) and I get the same message when launching the setup. I realised when I do `export LD_PRELOAD=/tmp/libaccess.so` instead of `export LD_PRELOAD_64=/tmp/libaccess.so` that I still have this error : `ld.so.1: mdb: fatal: /tmp/libaccess.so: wrong ELF class: ELFCLASS32 Killed` (when with sybase user) but only when I do `mdb core` aften attempting to run the setup. – mentinet May 11 '15 at 10:10
  • Than probably `access()` is not came from shared library, and my method not helping Sybase :( Try Solaris 11.0 or 11.1 – myaut May 11 '15 at 10:23
  • 1
    The `access()` **library** call is part of the POSIX standard. (http://pubs.opengroup.org/onlinepubs/9699919799/functions/access.html) It's certainly provided by `libc.so` on Solaris 11.2. If Sybase isn't using the library call but is making the system call directly, that's a Sybase bug and all the LD_PRELOAD gyrations aren't going to help, even were you to get them correct. – Andrew Henle May 11 '15 at 11:02
2

Thank you all for your replies.

myaut was right when saying:

Many system calls were deprecated in Solaris 11, access seem to be deleted

My problem therefore seemed related to a incompatibility between Sybase ASE 12.5.4 and Solaris 11 as this link shows: http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.dc35889_1254/html/installsol/X22779.htm

Sybase ASE 12.5.4 is compatible with Solaris 2.8 (therefore Solaris 8) and I got from the SAP forum that it is 100% compatible with Solaris 9 and most likely with Solaris 10 too but clearly, it doesn't seem to be with Solaris 11.

I have managed to launch the setup from a Solaris 10 machine and it seems to work perfectly fine.

Thank you for your help. Regards,

mentinet
  • 744
  • 3
  • 9
  • 23