Mental block with os.walk - want to process a file not the filename string

Question

Trying to implement a little script to move the older log files out of apache (actually using a simple bash script to do this in 'real life' - this is just an exercise to practice using Python). I'm getting the filename as a string as the variable f, but I want this to actually be a file when i pass it to self.processFile(root, f, age, inString).

I tried opening the actual file a few different ways, but I'm missing the target, and end up getting an error, a path that doesn't always seem to be correct, or just a string. I'll blame it on the late night, but I'm blanking on the best way to open f as a file right before passing to self.processFile (where it will be gzipped). Usually its something very simple that i'm missing, so I have to assume that's the case here. I'd appreciate any constructive advice/direction.

 """recursive walk through /usr/local/apache2.2/logs"""
    for root, dirs, files in os.walk(basedir):
            for f in files:
                m=self.fileFormatRegex.match(f)
                if m:
                    if (('access_log.' in f) or
                        ('error.' in f) or
                        ('access.' in f) or
                        ('error_log.' in f) or
                        ('mod_jk.log.' in f)):
                        #This is where i'd like to open the file using the filename f
                        self.processFile(root, f, age, inString)

chown · Accepted Answer · 2011-09-29T05:56:47.010

1

Use os.path.abspath:

self.processFile(root, open(os.path.abspath(f)), age, inString)

Like so:

import os
for root, dirs, files in os.walk(basedir):
    for f in files:
        m=self.fileFormatRegex.match(f)
        if m:
            if (set('access_log.', 'error.', 'access.', 'error_log.','mod_jk.log.').intersection(set(f))):
                self.processFile(root, open(os.path.abspath(f)), age, inString)

Or os.path.join:

import os
for root, dirs, files in os.walk(basedir):
    for f in files:
        m=self.fileFormatRegex.match(f)
        if m:
            if (set('access_log.', 'error.', 'access.', 'error_log.','mod_jk.log.').intersection(set(f))):
             self.processFile(root, open(os.path.join(r"/", root, f)), age, inString)
             # Sometimes the leading / isnt necessary, like this:
             # self.processFile(root, open(os.path.join(root, f)), age, inString)

More about os.path

Yet another way using file() instead of open() (does almost the same thing as open):

self.processFile(root, file(os.path.join(root, f), "r"), age, inString)
self.processFile(root, file(os.path.abspath(f), "r+"), age, inString)

edited Sep 29 '11 at 05:56

answered Sep 29 '11 at 02:50

chown

51,908
16
134
170

Thanks chown. I tried this and it worked. I thought I'd tried this earlier and had it fail, but i may have fat-fingered something during the earlier trial. – Sep 29 '11 at 03:10
Glad you got it fixed up @jonny! Happy to help. – chown Sep 29 '11 at 03:11
if f.startswith(('access_log.', 'error.', 'acess.', 'error_log.', 'mod_jk.log/')) – plaes Sep 29 '11 at 05:19
sadly i had this snippet working in another function a few lines down from the problem area. I just realized this and i am feeling very much like Homer Simpson right now. – Sep 29 '11 at 14:24
`file` and `open` do *exactly* the same thing, as they are two different names for the same command. – Ethan Furman Sep 29 '11 at 15:03
Almost, `file()` is a `file` type constructor, and `open` is a function that constructs and returns a `file` object. They achieve the same end, but not technically the exact same. – chown Sep 29 '11 at 15:07
And `file` goes away in Python 3, so `open` or `io.open` are preferred. – tzot Oct 17 '11 at 13:14

SingleNegationElimination · Answer 2 · 2011-09-29T03:28:52.220

1

base = "/some/path"
for root, dirs, files in os.walk(base):
    for f in files:
        thefile = file(os.path.join(root, f))

You'll have to join the root argument to each of the files argument to get a path to the actual file.

edited Sep 29 '11 at 03:28

answered Sep 29 '11 at 02:52

SingleNegationElimination

151,563
33
264
304

there is no `os.join`, did you mean `os.path.join`? – chown Sep 29 '11 at 02:58
@TokenMacGuy - Thanks for taking the time to answer. This would have also worked, and in fact i have a very similar snippet a few lines down from the area i was having a problem with. As usual it was me that was the issue, and it was a simple solution I was blanking on for some reason. – Sep 29 '11 at 14:26

Mental block with os.walk - want to process a file not the filename string

2 Answers2