2

I have written a very simple webserver in Scala (based on Actors). The purpose of it so to log events from our frontend server (such as if a user clicks a button or a page is loaded). The file will need to be rotated every 64-100mb or so and it will be send to s3 for later analysis with Hadoop. the amount of traffic will be about 50-100 calls/s

Some questions that pops into my mind:

  1. How do I make sure that all actors can write to one file in a thread safe way?
  2. What is the best way to rotate the file after X amount of mb. Should I do this in my code or from the filesystem (if I do it from the filesystem, how do I then verify that the file isn't in the middle of a write or that the buffer is flushed)
Tobias
  • 1,062
  • 1
  • 9
  • 15

2 Answers2

7

One simple method would be to have a single file writer actor that serialized all writes to the disk. You could then have multiple request handler actors that fed it updates as they processed logging events from the frontend server. You'd get concurrency in request handling while still serializing writes to your log file. Having more than a single actor would open the possibility of concurrent writes, which would at best corrupt your log file. Basically, if you want something to be thread-safe in the actor model, it should be executed on a single actor. Unfortunately, your task is inherently serial at the point you write to disk. You could do something more involved like merge log files coming from multiple actors at rotation time but that seems like overkill. Unless you're generating that 64-100MB in a second or two, I'd be surprised if the extra threads doing I/O bought you anything.

Assuming a single writing actor, it's pretty trivial to calculate the amount that has been written since the last rotation and I don't think tracking in the actor's internal state versus polling the filesystem would make a difference one way or the other.

dml
  • 469
  • 2
  • 3
  • Sounds like a good idea. It should also be easy, if I need it, to add multiple writer actors that, as you said, writes to multiple files. I will mark your answer as ok after I have tried this out – Tobias Mar 07 '11 at 02:55
0

U can use Only One Actor to write every requests from different threads, since all of the requests go through this actor, there will be no concurrency problems.

As per file write rolling, if your write requests can be logged in line by line, then you can resort to log4j or logback's FileRollingAppender things. Otherwise, you can write your own which will be easy as long as remembering to lock the file before performing any delete or update operations.

The rolling usually means you rename the older files and current file to other names and then create a new file with current file name, at last, u can always write to the file with current file name.

DarrenWang
  • 478
  • 2
  • 3