How to parse warc data for robots.txt info

Asked Mar 12 '19 at 21:00

Active Mar 12 '19 at 21:00

Viewed 65 times

I have the following code that I am writing to get values from a warc file. My goal is to find sites that have:

User-Agent: * 
Disallow: /

I would like it to only print URLs that have the above robots.txt rules ^

My Python code that currently only prints one line that is the URL:

file = 'robots.warc'
num_lines = sum(1 for line in open(file, errors='ignore'))
print('file has', num_lines , 'lines')

with open(file, errors='ignore') as lines:
    for line in lines:
        if line.startswith("WARC-Target-URI:"):
            print(line)

Here is an example warc file

Thanks for your help!

asked Mar 12 '19 at 21:00

Trey Copeland

3,387
7
29
46

1

Python has a built-in [robotparser](https://docs.python.org/3/library/urllib.robotparser.html) library that may be helpful (I've never used it). – t.m.adam Mar 12 '19 at 21:06
@t.m.adam thanks. that looks more like it calls the url. my file already has the robots.txt data, I just need to parse through it – Trey Copeland Mar 12 '19 at 21:15
Regular expressions then? – t.m.adam Mar 12 '19 at 21:19

How to parse warc data for robots.txt info

0 Answers0