1

How efficient is puppet with handling large files? To give you a concrete example:

Let's assume we're dealing with configuration data (stored in files) in the order of gigabytes. Puppet needs to ensure that the files are up-to-date with every agent run.

Question: Is puppet performing some file digest type of operation beforehand, or just dummy-copying every config file during agent runs?

Liviu Chircu
  • 1,000
  • 3
  • 11
  • 24

2 Answers2

5

When using file { 'name': source => <URL> }, the file content is not sent through the network unless there is a checksum mismatch between master and agent. The default checksum type is md5.

Beware of the content property for file. Its value is part of the catalog. Don't assign it with contents of large files via the file() or template() functions.

So yes, you can technically manage files of arbitrary size through Puppet. In practice, I try to avoid it, because all of Puppet's files should be part of a git repo or similar. Don't push your tarballs inside there. Puppet can deploy them by other means (packages, HTTP, ...).

Felix Frank
  • 8,125
  • 1
  • 23
  • 30
  • The big files are actually configuration files and are to be loaded into memory upon app startup. – Liviu Chircu Jul 28 '15 at 11:25
  • 1
    @LiviuChircu Out of curiosity: what kind of software uses configuration files of that magnitude? – Felix Frank Jul 28 '15 at 12:04
  • Ok, maybe "configuration" is bad wording. Let's just call them "DB files". It's not expensive at all to store each DB file on each node and have it cached at runtime, rather than reading the data across network on each app restart. Alternatives such as SQLite or packaging data into RPMs are way too much of an overkill. – Liviu Chircu Jul 28 '15 at 13:09
1

Im not entirely certain now puppets file server works in the latest update but in previous versions Puppet read the file into memory and thats why it was not recommended using the file server to transfer files larger than 1gb. I suggest you go through these answers and see if it makes sense https://serverfault.com/a/398133

Community
  • 1
  • 1
letsc
  • 2,515
  • 5
  • 35
  • 54