The configuration of even a not-so-big logging infrastructure (eg.: tens of servers generating, together, several hundreds MBs of LOGs per day) is quite a challenging task.
In your question, you "touch" several critical points. I'm going to address them, starting from the explicit questions.
- "is it possible to have rsyslog write locally (on the client) and to the remote central log server"
Yes. For sure. Assuming you have a running rsyslog-server already logging locally, to ALSO send those log to a remote machine is as simple as adding, at the end of your current configuration, something like:
*.* @10.0.49.251
that means: "send via UDP/514 to 10.0.49.251 the LOG messages matching ALL facilities and ALL priorities". Unfortunately, despite this is an answer to your question, there area several other things to consider, as you seems to be interested in a "reliable forwarding" of such log messages. This article should be a good starting point for further analysis.
- "I don't want to lose the ability to have some log data to diagnose other systems."
I've read that very requirement (not loosing LOG messages) several times. Every SysAdmin want it. But things are, unfortunately, very complex (local filesystems might get full and it's possible that there's no space for storing log messages locally; log-forwarding implies network activities with plenty of "get-lost" factors; etc.) and even if it can be technically achieveble, it comes with a very high price. I'm not going to discuss it in detail, as there is this very interesting post that clearly explain the main figure.
In addition to above, let me add something more, related to what you wrote in your question:
- "it would be great to setup a central log server that can capture incoming rsyslog packets"
Common syslog-forwarding from one Linux box to another Linux box is really simple to configure: one line in rsyslog.conf is enough. Anyway, based on my experience, you might reach a point where having a simple line of destructurated text is not enough for your analisys needs. We surely can "filter" based on syslog-facility, syslog-priority, syslog-tag, source-host and message. But those could be limiting. Rsyslog provides several properties that can be used when "assembling" the messages to store/forward and/or when parsing the message received. Also, some of those properties can further represent other sub-structures. This is inline with what specified with RFC 5424. I'm not saying that you should work with JSON (having local rsyslog prepare JSON-structured messages and having that relayed to your central log-server) but, at least, you should carefully evaluate such an approach, expecially if you plan to "index" your LOGs for analytics purposes.
- "Each hostname would have its own folder and set of logs."
With rsyslog is really simple to setup an environment where log-messages are automatically stored in files based on various properties (like YEAR, MONTH, DAY and HOSTNAME). This is as easy as:
template(name="DYNmail" type="string" string="/var/log/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME:::lowercase%-mail.log")
[...]
if ($syslogfacility-text == 'mail') then -?DYNmail;RSYSLOG_FORMAT
if ($syslogfacility-text == 'mail') then stop
so with three rows of configurations, you'll have all of mail.* messages coming from whatever remote server, being stored under a YEAR/MONTH/DAY hierarchy in files named HOSTNAME-mail. Automatic-log-rotation included. Nice, isn't it?
- "my hosts run Debian 7 and Debian 8."
regardless of your debian version, I suggest you to update to latest stable rsyslog-release. There are well-mantained repositories for main distributions, debian included. Also, should you plan to invest some time with rsyslog-configuration/administration, I invite you to subscribe the related mailing-list, that is both very active and valuable.
