0

Due to local data retention laws, companies operating in my country are obligated to store & sign log files digitally every day so they can be verified later using the key to check the file integrity (to make sure that files aren't tampered after they were signed) in case of a court order.

I keep daily server logs using Morgan, it creates logs each day with YYYYMMDD format, like so : access-20151123.log

I followed the tutorial below and created keys.

Note that I'm testing this on my Mac OSX right now.

https://github.com/coolaj86/node-ssl-root-cas/wiki/Painless%20Self-Signed%20Certificates%20in%20node.js

Now I have 7 files under three different folders.

all

  • my-private-root-ca.key.pem
  • my-private-root.ca.srl
  • my-server.csr.pem

client

  • my-private-root-ca.crt.pem

server

  • my-private-root-ca.crt.pem
  • my-server.crt.pem
  • my-server.key.pem

I am planning to use node-rsa package to sign (with a timestamp) and verify my log files. I'm kinda confused on whether I can do all the job using only node-rsa package (without the tutorial above) or not.

Now, how can I sign a specific log file located in /logs folder and test its integrity (verify) after signing it? That's my question. (For instance, if I edit the signed log file and test it using a key to verify its integrity, it should throw an error)

I'm not looking for a "step by step" guide specifically, so any kind of constructive contribution is welcomed.

EDIT #2

The part below is written in Turkish and I've followed those tutorials while I was working with PHP. Right now I stopped using PHP and moved to Node stack, so I want to handle the same thing with Node.

1-) https://www.syslogs.org/openssl-ile-5651-sayili-kanun-geregi-log-imzalamak/

2-) https://www.syslogs.org/openssl-1-0-x-tsa-ozelligi-ve-5651a-uygun-log-imzalamak/

And this is my bash script, but I'll handle this part with Node.js

3-) https://www.syslogs.org/openssl-ve-tsa-ile-otomatik-log-imzalayici-shell-script/

salep
  • 1,332
  • 9
  • 44
  • 93

2 Answers2

3

IMPORTANT EDIT

The original answer was more of a generic "how to sign data". The question has since become more clearly defined and requires time stamping authority signing. A revised answer follows, with the original below.

OpenSSL >= 0.9.9 Required

Time Stamping (ts) was not an available OpenSSL command until 0.9.9. Mac users will need to use Homebrew or similar to download a suitable version, and may need to explicitly specify the path for each call.

Structure and Certificates

We'll need to first generate certificates to be used throughout the process. This involves a slight config modification, creating a basic directory structure, and a few calls to openssl.

For the purposes of this demonstration, we shall assume the base directory to be ~/.openssl. Create this directory if it does not already exist

cd ~/.openssl
export TSA_DIR=$(pwd)
mkdir ca 
mkdir ca/private 
mkdir ca/newcerts 
touch ca/index.txt 
echo "0000000000000001" > ca/serial 
echo "0000000000000001" > ca/tsaserial 
export OPENSSL_CONF=$PWD/openssl.cnf 

Copy the configuration file from your current openssl directory to this directory. In my case, I found it at /usr/local/etc/openssl/openssl.cnf

cp /usr/local/etc/openssl/openssl.cnf .

You'll need to modify this file slightly to uncomment the line beginning with extendedKeyUsage. You can do this manually, or using sed

sed -i -e 's/\# extendedKeyUsage/extendedKeyUsage/' openssl.cnf

Next, we'll set up the keys and certificates. We'll need a CA key, CA cert, TSA key, and TSA cert.

openssl genrsa 4096 > ca/private/cakey.pem
openssl req -new -x509 -days 3650 -key ca/private/cakey.pem > ca/newcerts/cacert.pem

Answer the questions that follow however you like but make sure you remember your answers must agree on both certificates. Afterward, we'll make a copy of the cert.

cp ca/newcerts/cacert.pem ca

Now we'll make TSA key and TSA certificate signing request

openssl genrsa 4096 > ca/private/tsakey.pem
openssl req -new -key ca/private/tsakey.pem > tsacert.csr

Another round of questions. Answers must agree. After, we convert the CSR into a certificate (you'll have to answer Y to two questions)

openssl ca -in tsacert.csr > ca/newcerts/tsacert.pem
cp ca/newcerts/tsacert.pem ca

Putting It To Work

So, where's the Node? We can wrap the commands to create the ts query and reply files, and another to verify the whole process. Let's dive in.

The following script assumes it's being run from the $TSA_DIR we exported above. If it is not, adjust the definitions to fit.

'use strict';


const ssl = '/usr/local/bin/openssl';

const path = require('path');
const exec = require('child_process').exec;


const key = path.resolve(__dirname, 'ca/private/tsakey.pem');
const cert = path.resolve(__dirname, 'ca/tsacert.pem');
const ca = path.resolve(__dirname, 'ca/cacert.pem');
const config = path.resolve(__dirname, 'openssl.cnf');



function generateQuery(logfile, callback) {
  const dirname = path.dirname(logfile);
  const basename = path.basename(logfile, path.extname(logfile));
  const query = path.resolve(dirname, `${basename}.tsq`);

  const cmd = `${ssl} ts -query -data ${logfile} -policy tsa_policy1 > ${query}`;
  const child = exec(cmd, (err, stdout, stderr) => {
    if (err) return callback(err);

    // no stdout

    const cmd = `${ssl} ts -query -in ${query} -text`;
    const child2 = exec(cmd, (err, stdout, stderr) => {
      if (err) return callback(err);

      // successful stdout:
      // Hash Algorithm: sha1
      // Message data:
      //     0000 - 3d 2b 54 8f 5f da 76 08-09 d8 0f b4 e3 98 7e 87   =+T._.v.......~.
      //     0010 - a6 03 b4 bd                                       ....
      // Policy OID: tsa_policy1
      // Nonce: 0x0B3B80EDD01053D5
      // Certificate required: no
      // Extensions:



      callback(null, query);
    })
  })

}


function generateReply(query, callback) {
  const dirname = path.dirname(query);
  const basename = path.basename(query, path.extname(query));
  const reply = path.resolve(dirname, `${basename}.tsr`);

  const cmd = `${ssl} ts -config ${config} -reply -queryfile ${query} -inkey ${key} -signer ${cert} > ${reply}`;
  const child = exec(cmd, (err, stdout, stderr) => {
    if (err) return callback(err);

    const cmd = `${ssl} ts -reply -in ${reply} -text`;
    const child2 = exec(cmd, (err, stdout, stderr) => {
      if (err) return callback(err);

      // Successful stdout:
      // Status info:
      // Status: Granted.
      // Status description: unspecified
      // Failure info: unspecified
      //
      // TST info:
      // Version: 1
      // Policy OID: tsa_policy1
      // Hash Algorithm: sha1
      // Message data:
      //     0000 - 3d 2b 54 8f 5f da 76 08-09 d8 0f b4 e3 98 7e 87   =+T._.v.......~.
      //     0010 - a6 03 b4 bd                                       ....
      // Serial number: 0x05
      // Time stamp: Dec  1 18:02:07 2015 GMT
      // Accuracy: 0x01 seconds, 0x01F4 millis, 0x64 micros
      // Ordering: yes
      // Nonce: 0x95DFDBA4DCE33E7B
      // TSA: DirName:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/OU=section/CN=example.com/emailAddress=rw@xandocs.com
      // Extensions:

      callback(null, reply);
    })
  })

}



function validateToken(logfile, query, reply, callback) {
  const cmd = `${ssl} ts -verify -queryfile ${query} -in ${reply} -CAfile ${ca} -untrusted ${cert}`;
  const child = exec(cmd, (err, stdout, stderr) => {
    if (err) return callback(err);

    // Successful stdout:
    // Verification: OK

    const cmd = `${ssl} ts -verify -data ${logfile} -in ${reply} -CAfile ${ca} -untrusted ${cert}`;
    const child2 = exec(cmd, (err, stdout, stderr) => {
      if (err) return callback(err);

      // Successful stdout:
      // Verification: OK

      callback(null, stdout);
    })
  })

}



// sample use for all steps
const logfile = './foo.txt';

generateQuery(logfile, (err, query) => {
  if (err) console.log(err);

  generateReply(query, (err, reply) => {
    if (err) console.log(err);

    validateToken(logfile, query, reply, (err, result) => {
      if (err) console.log(err);

      console.log(result);
    })
  })
})

And there you have it - TSA query, reply, and verification from within your Node.js application.

One more edit

I borrowed heavily from this thread. Credit where it's due.


Original Answer Follows

If I understand the question correctly, you have a series of log files that you must digitally sign in order to verify each file's integrity. Correct?

You mention node-rsa, so I'll stick to RSA here, which you'll probably find quite simple after running through it a few times. I'll also avoid use of any module, but after learning how this process actually works you should have no problem working with one if you feel so inclined. We'll also start from scratch on the keys.

When we digitally sign a file, what we are actually doing is applying a cryptographic hash function to the file's contents to produce a digest--essentially a one-way mathematical reduction to a manageable value difficult to guess from the original content and impossible to work back from to the original content. Among other properties, of course. We then encrypt this digest, possibly along with additional content such as the date, using a private key. The result is a signature that can be decrypted by anybody with access to your public key for verification. So let's get started.

Generate Keys

Generating a private key is as easy as:

openssl genrsa 4096

But you'll want to protect this with a passcode and probably output it to a file, so just adding a couple of flags yields:

openssl genrsa -aes256 -out private_key.pem

Now we generate a public key from the private key as follows

openssl rsa -in private_key.pem -outform PEM -pubout -out public_key.pem

And just like magic--er, math--your key pair is ready to use. Keep the private key as secure as you would any other confidential information but the public key can be distributed freely. Let's put these to use.

Computing The Digest

Next, we'll use the Node.js core crypto module to compute the digest and sign. You could also very easily write a module to generate keys if you like; bonus credits, I suppose. For this example, I'll pretend the log files are in a subdirectory of the script's directory; not likely the case but I'll leave you the directory mapping to you unless you need help with it.

In your signing module, create an instance of the crypto.Hash and read the desired log file's contents into this instance (using ES6 features for extra credit):

'use strict';
const crypto = require('crypto');
const fs = require('fs');
const path = require('path');

const hasher = crypto.createHash('sha256');

const pathname = path.resolve(__dirname, 'logs', 'access-20151201.log');
const rs = fs.createReadStream(pathname);

rs.on('data', data => hasher.update(data))

rs.on('end', () => {

  const hash = hasher.digest('hex');

  // we'll be back to this spot

}

That's the digest. Now we need to encrypt it using your private key.

Signing The Digest

Node.js makes this part almost embarrassingly easy. We'll simply load the private key into memory and use the crypto module to sign it. Let's jump right back in to the comment we left above:

const digest = hasher.digest('hex');

const privateKey = fs.readFileSync('private_key.pem');

const signer = crypto.createSign('RSA-SHA256');

signer.update(digest)

const signature = signer.sign(privateKey, 'base64')

It's that easy!

Verifying The Signature

You can verify any signature by decrypting signature and comparing the result to the digest of the file. Let's run through one of these.

const publicKey = fs.readFileSync('public_key.pem');

const verifier = crypto.createVerify('RSA-SHA256');

const testSignature = verifier.verify(publicKey, signature, 'base64');

const verified = testSignature === digest;

And there you have it. I would recommend you read as much as you can find on exactly how this is working. This will help you to overcome any unexpected difficulties and also encryption is a wonder of mathematics and fascinating in its own right.

Best of luck - hope this helped. Glad to hear if I've presented anything incorrectly here.

Fissure King
  • 1,250
  • 7
  • 17
  • Thanks, it means a lot to me. I have one more question. I hope I'm not asking too much, but could you please check my edit #2? Using that solution (edit #2) right now I have original log file, .der and .tsq file as a result of a signing process (one of these files is "timestamp authority" and other one is the signing request file) and this format (3 files for each day) is required legally. Can I do the same thing with Node? If yes, can you show me how? – salep Dec 01 '15 at 15:59
  • 1
    Happy to take a look. Would you mind providing the country or the governing statute? – Fissure King Dec 01 '15 at 16:03
  • Yeah, of course. Turkey. TIB (it's a part of BTK, Information and Communication Technologies Authority) wants website owners (commercial or personal use, you need to keep logs if you allow other people to contribute on your website or you can be held responsible for content and pay fines) to keep logs and sign them digitally. I think this should be the datacenter's or ISP's job, not mine, but here we go. – salep Dec 01 '15 at 16:08
  • Ok, I'll need to update this just a little to use DER in place of PEM and incorporate the CA/CSR requirement. Check back in an hour or so? And wow, thanks for the bounty - that's a first :) – Fissure King Dec 01 '15 at 16:14
  • A late edit. I said that I had three files, but I need to edit something . I said "original log file", I was wrong. It's the signed version of the log, so if you edit it you won't be able to verify it using its certificate. – salep Dec 01 '15 at 16:15
  • Yep, that's the idea: after you sign it, anybody can attempt to verify the signature but it will only verify if the contents are identical. – Fissure King Dec 01 '15 at 16:18
  • I said that I had three files for each day, but I need to edit something . I said "original log file", I was wrong. It's the signed version of the log file, so if you edit it you won't be able to verify it using its certificate. Right now I'm using the bash script for a CentOS given the third link (a friend of mine edited it for me so logs are signed every day at 23:58 and signed log file, .der and .tsq are getting zipped in a folder.) I think we can achieve the same thing using the node-cron package. If I manage to have a working example, cron part is relatively easy :) Thanks again. – salep Dec 01 '15 at 16:21
  • I wanted to make it clear as much as possible, thanks again. I'll check in a few hours. Your welcome, by the way. – salep Dec 01 '15 at 16:22
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/96701/discussion-between-fissure-king-and-salep). – Fissure King Dec 01 '15 at 16:27
  • 1
    just wondering if the revised answer worked for you. All is good? – Fissure King Dec 10 '15 at 05:52
  • Oh, I totally forget to update here. It's all good, I had to tweak a little bit, but not tested fully. Yet. I think it'll help many people in the future regarding to this topic since there's pretty much no tutorial out there, so I'll update the question with a complete solution that includes a few additional settings. – salep Dec 10 '15 at 09:51
  • Note: Based on discussions on [another question](https://stackoverflow.com/q/56062691/238704), at least 3 of us believe this answer is wrong in at least the "Verifying the signature" part at the end. – President James K. Polk May 09 '19 at 21:06
1

Add a cronjob to run every day to find and sign the log files using openssl:

find /logs -name \*.log -mtime +1 -exec openssl dgst -sha1 -sign my-server.key.pem -out {}.sig {} \;

You can verify the signatures like this:

openssl dgst -sha1 -verify my-server.crt.pem -signature access-2015-11-25.log.sig access-2015-11-25.log
Thomas Cort
  • 331
  • 2
  • 5
  • This works (I tested by tampering the original log file after signing it and it returned verification failure, so I presume that it supposed to work that way), but there's only one small problem. It says `"unable to load key file"` unless I run this command `openssl x509 -pubkey -noout -in my-server.crt.pem > pubkey5.pem` and change `my-server.crt.pem` to `pubkey5.pem` while using your command. As far as I understand, `pubkey5.pem` is a "public" key. Is that still "valid" signing or do I need to change something here? I found that method here : http://stackoverflow.com/a/2387569/3210431 – salep Nov 26 '15 at 14:28