There are some really good answers above. A further note would be "what difference does it really make". Already mentioned is the cost of maintenance (e.g. problems when someone installs a fresh operating system, which doesn't have Lib3, so the user has to go find lib3 somewhere and install it, and because lib3 also needs lib17 which you also isn't installed, it adds more work for the user).
But also, when you load the binary, if you have linked against shared libraries that aren't actually used, the system will still go look for those libraries, and refuse to load if they are not present - this adds time, and install nightmare.
Once the code is loaded, it should have no additional runtime penalty.
Having said that, there are sometimes arguments for linking against unused libraries. Say your code has an option USE_FOO, where the FOO feature is only included based on some arbitrary choice when building (e.g. "is this on Linux kernel > 3.0" or "Does the system have a fancy graphics card"), and FOO uses Lib1 to do it's business, it can make the build system (makefile or similar) a little simpler to always link against lib1, even if you don't actually need it when USE_FOO is not set.
But in general, don't link against libraries not needed. It causes more dependencies, and that's never a good thing.