libreadline.so.7: undefined symbol: UP

Question

I'm having trouble building cURL and Git from sources when using GNU's Readline 7. When configuring libraries like cURL the result is:

$ ./configure ...
...
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
gawk: symbol lookup error: /usr/local/lib64/libreadline.so.7: undefined symbol: UP
config.status: error: could not create Makefile
Failed to configure cURL

Sure enough:

$ nm -D /usr/local/lib64/libreadline.so | egrep 'UP|DOWN|LEFT|RIGHT'
    U UP

And:

$ nm /usr/local/lib64/libreadline.a | egrep 'UP|DOWN|LEFT|RIGHT'
    U UP

Now, the weird thing is, the messages gawk: symbol lookup error: /usr/local/lib64/libreadline.so.7: undefined symbol: UP and undefined symbol: UP do not show up in cURL's config.log. Its almost as if an Autotools command is producing it, and not the configure test scripts.

Readline's CHANGE log and README do not discuss the change or the missing symbol. I found several posts where other users experience the same problem, but the fix usually degenerates to, downgrade to readline 6.3.

At this point I am not sure if this is a Readline problem; or a problem in a program or library like cURL or Git. Because the symbol is undefined in Readline, I think its closer to a Readline problem (or a library Readline depends on).

My first question is, where should I begin looking for the problem?

My second question is, how should we fix it?

Here's the relevant portion of cURL's config.log. There is no tail and no test because cURL does not seem to test for it or it was not logged. Even the error strings are missing from the log.

$ cat curl-7.56.0/config.log
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

  $ ./configure --prefix=/usr/local --libdir=/usr/local/lib64 --enable-shared --
enable-static --enable-optimize --enable-symbol-hiding --enable-http --enable-ft
p --enable-file --enable-ldap --enable-ldaps --enable-rtsp --enable-proxy --enab
le-dict --enable-telnet --enable-tftp --enable-pop3 --enable-imap --enable-smb -
-enable-smtp --enable-gopher --enable-cookies --enable-ipv6 --with-zlib=/usr/loc
al --with-ssl=/usr/local --without-gnutls --without-polarssl --without-mbedtls -
-without-cyassl --without-nss --without-libssh2 --with-libidn2=/usr/local --with
-nghttp2 --with-ca-path=/etc/ssl/certs/ --with-ca-bundle=/etc/ssl/certs/ca-bundl
e.crt

...

For completeness, I did not build awk, gawk or mawk from sources and install them somewhere. gawk is Fedora 26's standard one, and it does not use libraries in /usr/local/lib64:

Build-Scripts$ command -v gawk
/bin/gawk

Build-Scripts$ ldd /bin/gawk
    linux-vdso.so.1 (0x00007ffc33d2d000)
    libsigsegv.so.2 => /lib64/libsigsegv.so.2 (0x00007f3da9135000)
    libreadline.so.7 => /lib64/libreadline.so.7 (0x00007f3da8ee9000)
    libmpfr.so.4 => /lib64/libmpfr.so.4 (0x00007f3da8c87000)
    libgmp.so.10 => /lib64/libgmp.so.10 (0x00007f3da8a10000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f3da880c000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f3da84f6000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f3da8125000)
    libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007f3da7ef9000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f3da95e0000)

@AnttiHaapala - `$ gawk --version` returns `GNU Awk 4.1.4, API: 1.1 (GNU MPFR 3.1.5, GNU MP 6.1.2)`. It does not appear broken. — jww, Oct 23 '17 at 04:08
@AnttiHaapala - Could a screen colorizer be doing this? I'm working over SSH. — jww, Oct 23 '17 at 04:09
Well it could be, I am not sure. Btw `UP` seems to come from `libtinfo.so.5` on my Ubuntu — Antti Haapala -- Слава Україні, Oct 23 '17 at 04:13
Thanks @AnttiHaapala. `tinfo` solves one mystery. Here's a pastebin of how Readline references `tinfo`: [Readline and tinfo](https://pastebin.com/XkcBHkkS). I guess I need to explicitly create a readline dependency on tinfo when linking. — jww, Oct 23 '17 at 04:29
On Fedora `yum whatprovides libtinfo.so.6` returns `ncurses-libs-6.0-8.20170212.fc26.i686 : Ncurses libraries`. The odd thing is, I have an updated copy of nurses built and available; and there is no readline configuration option to specify ncurses. — jww, Oct 23 '17 at 04:44
`configure` does not create your makefiles directly. Instead, after performing all its checks, it creates a separate script, `config.status`, that fills in the templates to create your makefiles. It appears to be this second script that is failing for you, and that's why you see no details in the log. Failures of `config.status` are very unusual; I'm not sure what to make of it, but the first thing I would do myself is to try to find the offending command in `config.status`. — John Bollinger, Oct 23 '17 at 04:51
Note in particular that your error message seems to indicate that gawk *does* use libraries in `/usr/local/lib64` when `config.status` runs it. That's a bit surprising, but plausible. You could certainly persuade it to do so yourself, by use of `LD_LIBRARY_PATH`, for instance, and trying to reproduce the error that way might be a useful thing to do. — John Bollinger, Oct 23 '17 at 04:58
By the way, if you already have a perfectly good copy of libreadline provided by the OS, then why did you building your own, separate copy? — John Bollinger, Oct 23 '17 at 04:59
@JohnBollinger - I use modern OSes, like Fedora 26 and Ubuntu 17, to develop the scripts. The scripts are intended to run on older machines, like CentOS 5 (no Git available), Solaris 11 (no updates unless purchase support contract), OS X 10.5 and 10.8 (abandoned by OEM), and friends. I've got almost all of them working. Git and GnuTLS are the problems. — jww, Oct 23 '17 at 05:13

Brian Fitzgerald · Answer 1 · 2020-08-25T15:12:24.463

2

What's UP?

Symbol UP is defined in lib_termcap.c and lib_termcap.o in libraries libncurses.so and libncurses.a in package ncurses

Package readline requires curses. ncurses is "New curses".

$ nm -D lib/libncurses.so | grep -w UP
000000000025f6c8 B UP

You must use --with-shared to compile the shared object libraries.

./configure --prefix=/u01/sw --with-shared
make

edited Aug 25 '20 at 15:12

answered Aug 18 '20 at 19:16

Brian Fitzgerald

634
7
14

You perhaps overlooked that the OP already answered this question themselves -- and nearly three years ago at that. Moreover, their characterization of the problem differs from your apparent conclusion. And theirs is far more plausible to boot, since no matter what symbols ncurses defines, libreadline does not depend on that library in practice, nor does it seem likely that it would do. – John Bollinger Aug 25 '20 at 15:59
The OP already pointed out that in libreadline.so, symbol UP is undefined. libncurses defines symbol UP, and installing libncurses fixed rlwrap and python3 compile issues for me. I therefore concluded that libreadline depends on libncurses. – Brian Fitzgerald Aug 29 '20 at 03:52
What is the --prefix=/u01/sw option? – mcmuffin6o Aug 22 '23 at 22:45

score 2 · Answer 2 · answered Jul 28 '21 at 07:43

It appears that, in recent ncurses releases, the UP (and related) symbols were moved to libtinfo, which ncurses does not build by default. If you pass the --with-termlib flag to ncurses (./configure --with-shared --with-termlib) it will build this library and the symbols should resolve. Notably, it is important to not follow the suggestion from the previous answer from jww, since that patch will explicitly avoid linking against libtinfo.

score 1 · Answer 3 · answered Oct 24 '17 at 05:08

It appears there were two problems. First, the dynamic linker was selecting the wrong library. That is, the runtime linker incorrectly guessed /usr/bin/gawk was authorized to use /usr/local/lib64/libreadline.so. In my case, /usr/local is full of updates and it is somewhat of a playground. /usr programs should only use libraries from /lib64 and /usr/lib64; and /usr/local programs can use libraries from both /lib64, /usr/lib64 and /usr/local/lib64. They should never mix but we have no way to express the policy to the linker.

The dynamic linker vulnerability has festered for years, and Linux is the only major OS I am aware that still has not solved it. OS X has install names, Windows has manifest, the BSD's have rpath's set at build time ...

Second, the makefiles needed patching. @AnttiHaapala and @JohnBollinger made the observations in the comments but I was struggling with a "it should work" mindset since Red Hat and Fedora are major distros. It appears things are simply untested and the results are a break.

To clear the issue with Fedora this should be done after unpacking the tarball and configuring the library:

for mfile in $(find "$PWD" -name 'Makefile'); do
    sed -i 's|SHLIB_LIBS =|SHLIB_LIBS = -ltinfo|g' "$mfile"
done

After things started working as expected it occurred to me Fedora must be doing the same thing but I did not see it in Fedora's readline.spec. However it was done in readline-7.0-shlib.patch, which is the second patch the spec file applies.

Note that Linux does support rpath, just like other ELF-based systems do. It is a matter of explicit Fedora policy to avoid its use, however, on the basis that it thwarts the dynamic linker's general configuration options. You have evoked unexpected behavior by installing shared libraries in the dynamic linker's search path that conflict with the standard system binaries. I would characterize this as a system administration error (which, to be sure, Fedora may be more open to than some other systems). You could have avoided it by putting your built libs somewhere else. — John Bollinger, Oct 24 '17 at 20:02

libreadline.so.7: undefined symbol: UP

3 Answers3