is it possible to add a option for
players to host a game on their own
root server for better connection?
Why do you think this would not be possible?
I'm sure they'd have to install some server to make it happen, I assume; whether something you provided or something like GraniteDS or Red5 (or Flash Media Server or LiveCycle) or some other application server software. I'm sure there are ones out there designed specifically for games.
I doubt most "normal" game users have the chops to setup servers [or even linux], but it depends what your target market for the game is.
or is it impossible to host a flash
server without a gui on a linux root?
I don't see why you couldn't do this. All versions of Apache Iv'e used are, esentially, GUI-less and they run fine on Linux. Conceptually it isn't much different with any other server. Of course, I probably wouldn't throw such responsibilities to the 'layman' gamer.