0

I need to deploy a software project (packaged as an rpm) from a developer machine into a server. I'm using Fedora 23, along with the dnf package manager. I have to collect all dependencies of my rpm before I deploy to the server. The server can't be connected to the internet due to internal regulation (But I can ssh onto it). Running repository mirrors, etc. is not an option. I'm afraid I just have to collect all dependencies on the developer machine, scp (or ansible) them to the server and install them on the server.

I hoped that --installroot option in dnf could be of much help, as I could retrieve all rpms that would get installed into what dnf thinks is an empty system. This however doesn't work.

mkdir foo && sudo dnf install --installroot=$PWD/foo golang

gives an error:

Failed to synchronize cache for repo 'fedora'

Why does this fail? What are my options?

I'd like to see an elegant and robust solution. I'd prefer not to install anything on the server (I'd be most happy to do a single scp followed by one or two commands over ssh). A combination of rpm + yum/dnf magic would be great, but other solutions, including apt + deb are also of interest. I'd prefer not to use docker, and I'm strongly against running any additional infrastructure (docker registry, rpm mirror, etc.)

Adam Kurkiewicz
  • 1,526
  • 1
  • 15
  • 34
  • Yup, seen these restrictions before and I've concluded that the necessary work-arounds make this the least secure way to install software.... Would be a lot better to enable signed package repositories, one could even control the repository contents, if worried about the packages being installed. In the absence of such, I've been forced to copy all necessary RPMs up and just install them one at a time. Lastly you could also consider the use of Docker containers, which could be built offline, signed and then pushed into a deployment registry. Likely to be far to radical an idea. Hope that helps – Mark O'Connor Jun 21 '16 at 20:08
  • @MarkO'Connor any idea why the install-root switch is not working? This sounds like the cleanest way to collect all packages. – Adam Kurkiewicz Jun 22 '16 at 08:09
  • You want a chroot installed with all prerequisites of a package called "golang"? Poky-Yocto uses RPM5 with a "solvedb" (which is nothing more than an rpmdb of "everything"). Given a solvedb (configured with a macro), then "rpm -Uvh --root=$PWD/foo +golang" will install golang and all its prereq's into a chroot in $PWD/foo. I can add further details, but be forewarned: the operation is only as good as the metadata contained in packages. – Jeff Johnson Jun 23 '16 at 20:28
  • @JeffJohnson this sounds fantastic! Could you please give me more details? Do I add this poky-yocto's solvedb the way I would add any other repository (in such way it will appear in dnf's repolist?) or do I run it locally? I'm anxious to learn more! – Adam Kurkiewicz Jun 23 '16 at 20:42
  • Creating a solvedb is a single "rpm -Uvh --justdb --dbpath /tmp/solvedb *.rpm" command. The hard part is in the "*.rpm" download of "everything". The solvedb contains all the package metadata, including the path to the package. The path to the solvedb is configured in RPM5 using the macro %_solvedb_dbpath. When configured, all unresolved dependencies are looked up in the solvedb to find RPMTAG_PACKAGEORIGIN (i.e. the path to the *.rpm) which is then added to the install transaction, which is then continued, finding all prereqs recursively. A complete transaction is then attempted. Make sense? – Jeff Johnson Jun 23 '16 at 20:50
  • Not quite the same as a yum repository, which uses markup for a subset of the metadata contained in *.rpm headers. The advantage of a solved is that _ALL_ the metadata in *.rpm files is available, and rpm dependency checking is used to populate the transaction for the chroot. Again, lots depends on the quality of the metadata in packages: this is a data, not an implementation, issue. – Jeff Johnson Jun 23 '16 at 20:53
  • One last note: you want to add --noscripts and --notriggers (and possibly other disablers like --ignorearch) while creating a solvedb. The reasons should be obvious ;-) – Jeff Johnson Jun 23 '16 at 20:56
  • Thank you @JeffJohnson this is a fantastic insight into quite non-standard options of rpm. To sum up: the "solvedb" step downloads metadata about all packages from all rpm repositories I'm subscribed to (fedora, updates, rpmfusion-free, copr, google-chrome, etc.), then `rpm -Uvh --root=$PWD/foo +golang` solves dependencies and downloads appropriate packages? The point about noscripts and notriggers granted. – Adam Kurkiewicz Jun 23 '16 at 21:04
  • @JeffJohnson I've been trying to do this just now, doesn't the "solvedb" step require the rpms to be sitting on the disk? The only way I was able to get this to work is by pointing it to an rpm like this: `rpm -Uvh --justdb --dbpath /tmp/solvedbX /var/cache/PackageKit/metadata/updates/packages/python3-3.4.3-6.fc23.x86_64.rpm` – Adam Kurkiewicz Jun 23 '16 at 21:35
  • Yes: creating the solvedb adds PACKAGEORIGIN (which can be a URL, but lets keep things simple) in order to discover the packages that need to be added to a transaction. – Jeff Johnson Jun 23 '16 at 22:37
  • Note that most versions of rpm also have the ability to install from a manifest. A manifest is just a list of paths to packages, just use the manifest file instead of listing all the packages. The harder task there is doing the depsolving to generate the manifest. – Jeff Johnson Jun 23 '16 at 22:39
  • @JeffJohnson this looks a bit like a chicken and egg problem, in order to resolve rpms you have to resolve rpms. – Adam Kurkiewicz Jun 24 '16 at 12:51

1 Answers1

2

Here is a (ad hoc, lightly tested) script (assuming you have an already installed rpm system) to generate the list of all the rpm package names needed to install a given package (the script assumes goal="bash", edit to taste).

Feed the output names to dnf/yum to install.

#!/bin/sh
goal=bash
deps=$(rpm -q --qf '[%{REQUIRENAME}\n]'  $goal | egrep -v '^(rpmlib|rtld|config|/)')
goals=
while true; do
  subs=$(rpm -q --qf '%{NAME}\n' --whatprovides $deps | sort -u | tr '\n' ' ')
  if [ ."$subs" = ."$goals" ]; then
    echo "--- packages needed"
    echo "$goals" | tr ' ' '\n'
    exit 0
  fi
  goals=$(echo $goals $subs | tr ' ' '\n' | sort -u | tr '\n' ' ')
  for sub in $subs; do
    subdeps=$(rpm -q --qf '[%{REQUIRENAME}\n]' $sub | egrep -v '^(rpmlib|rtld|config|/)')
    deps=$(echo $deps $subdeps | sort -u)
  done
done
Jeff Johnson
  • 2,310
  • 13
  • 23
  • Interesting. I'll give it a try this evening. – Adam Kurkiewicz Jun 24 '16 at 13:43
  • Thanks @JeffJohnson, this seems to work. I'm a little surprised that a bare-bones "hello world" go website has dependencies weighting 150 MB, but I understand why that is: some dependencies one just never thinks of, such as ca-certificates or libc, and some dependencies are clearly redundant, such as sed, awk or grep. In any case, I've changed the code a slight bit and put it on [github](https://github.com/picrin/rpmDownloader) (with attribution). I still need to check if the downloaded RPMs will provide enough functionality for the website to run. – Adam Kurkiewicz Jun 24 '16 at 21:25
  • You likely do not need both i686 & x86_64 packages, but your manifest seems about right. – Jeff Johnson Jun 25 '16 at 16:10