8

I'm trying to extract each page of a PDF as a string:

import pyPdf

pages = []
pdf = pyPdf.PdfFileReader(file('g-reg-101.pdf', 'rb'))
for i in range(0, pdf.getNumPages()):
    this_page = pdf.getPage(i).extractText() + "\n"
    this_page = " ".join(this_page.replace(u"\xa0", " ").strip().split())
    pages.append(this_page.encode("ascii", "xmlcharrefreplace"))
for page in pages:
    print '*' * 80
    print page

But this script ignore newline characters, leaving me with messy strings like information concerning an individual which, because of name, identifyingnumber, mark or description (i.e, this should read identifying number, not identifyingumber).

Here's an example of the type of PDF I'm trying to parse.

Joe Mornin
  • 8,766
  • 18
  • 57
  • 82

3 Answers3

11

I don't know much about PDF encoding, but I think you can solve your particular problem by modifying pdf.py. In the PageObject.extractText method, you see what's going on:

def extractText(self):
    [...]
    for operands,operator in content.operations:
        if operator == "Tj":
            _text = operands[0]
            if isinstance(_text, TextStringObject):
                text += _text
        elif operator == "T*":
            text += "\n"
        elif operator == "'":
            text += "\n"
            _text = operands[0]
            if isinstance(_text, TextStringObject):
                text += operands[0]
        elif operator == '"':
            _text = operands[2]
            if isinstance(_text, TextStringObject):
                text += "\n"
                text += _text
        elif operator == "TJ":
            for i in operands[0]:
                if isinstance(i, TextStringObject):
                    text += i

If the operator is Tj or TJ (it's Tj in your example PDF) then the text is simply appended and no newline is added. Now you wouldn't necessarily want to add a newline, at least if I'm reading the PDF reference right: Tj/TJ are simply the single and multiple show-string operators, and the existence of a separator of some kind isn't mandatory.

Anyway, if you modify this code to be something like

def extractText(self, Tj_sep="", TJ_sep=""):

[...]

        if operator == "Tj":
            _text = operands[0]
            if isinstance(_text, TextStringObject):
                text += Tj_sep
                text += _text

[...]

        elif operator == "TJ":
            for i in operands[0]:
                if isinstance(i, TextStringObject):
                    text += TJ_sep
                    text += i

then the default behaviour should be the same:

In [1]: pdf.getPage(1).extractText()[1120:1250]
Out[1]: u'ing an individual which, because of name, identifyingnumber, mark or description can be readily associated with a particular indiv'

but you can change it when you want to:

In [2]: pdf.getPage(1).extractText(Tj_sep=" ")[1120:1250]
Out[2]: u'ta" means any information concerning an individual which, because of name, identifying number, mark or description can be readily '

or

In [3]: pdf.getPage(1).extractText(Tj_sep="\n")[1120:1250]
Out[3]: u'ta" means any information concerning an individual which, because of name, identifying\nnumber, mark or description can be readily '

Alternatively, you could simply add the separators yourself by modifying the operands themselves in-place, but that could break something else (methods like get_original_bytes make me nervous).

Finally, you don't have to edit pdf.py itself if you don't want to: you could simply pull out this method into a function.

DSM
  • 342,061
  • 65
  • 592
  • 494
0

pyPdf is not really made for this kind of text extraction, try pdfminer (or use pdftotext or something like that if you don't mind creating another process)

Steven
  • 28,002
  • 5
  • 61
  • 51
0

Extending on DSM's answer. Following is how you would implement it by extending few classes

import PyPDF2
import pandas as pd
from PyPDF2.generic import TextStringObject
from PyPDF2.pdf import ContentStream, IndirectObject, NameObject
from PyPDF2.utils import b_, u_

class PageObject2(PyPDF2.pdf.PageObject):
    def extractText(self, Tj_sep="", TJ_sep=""):
        """
        Locate all text drawing commands, in the order they are provided in the
        content stream, and extract the text.  This works well for some PDF
        files, but poorly for others, depending on the generator used.  This will
        be refined in the future.  Do not rely on the order of text coming out of
        this function, as it will change if this function is made more
        sophisticated.

        :return: a unicode string object.
        """
        text = u_("")
        content = self["/Contents"].getObject()
        if not isinstance(content, ContentStream):
            content = ContentStream(content, self.pdf)
        # Note: we check all strings are TextStringObjects.  ByteStringObjects
        # are strings where the byte->string encoding was unknown, so adding
        # them to the text here would be gibberish.
        for operands, operator in content.operations:
            if operator == b_("Tj"):
                _text = operands[0]
                if isinstance(_text, TextStringObject):
                    text += Tj_sep
                    text += _text
            elif operator == b_("T*"):
                text += "\n"
            elif operator == b_("'"):
                text += "\n"
                _text = operands[0]
                if isinstance(_text, TextStringObject):
                    text += operands[0]
            elif operator == b_('"'):
                _text = operands[2]
                if isinstance(_text, TextStringObject):
                    text += "\n"
                    text += _text
            elif operator == b_("TJ"):
                for i in operands[0]:
                    if isinstance(i, TextStringObject):
                        text += TJ_sep
                        text += i
                text += "\n"
        return text


class PdfFileReader2(PyPDF2.PdfFileReader):
    def _flatten(self, pages=None, inherit=None, indirectRef=None):
        inheritablePageAttributes = (
            NameObject("/Resources"), NameObject(
                "/MediaBox"),
            NameObject("/CropBox"), NameObject("/Rotate")
        )
        if inherit == None:
            inherit = dict()
        if pages == None:
            self.flattenedPages = []
            catalog = self.trailer["/Root"].getObject()
            pages = catalog["/Pages"].getObject()

        t = "/Pages"
        if "/Type" in pages:
            t = pages["/Type"]

        if t == "/Pages":
            for attr in inheritablePageAttributes:
                if attr in pages:
                    inherit[attr] = pages[attr]
            for page in pages["/Kids"]:
                addt = {}
                if isinstance(page, IndirectObject):
                    addt["indirectRef"] = page
                self._flatten(page.getObject(), inherit, **addt)
        elif t == "/Page":
            for attr, value in list(inherit.items()):
                # if the page has it's own value, it does not inherit the
                # parent's value:
                if attr not in pages:
                    pages[attr] = value
            pageObj = PageObject2(self, indirectRef)
            pageObj.update(pages)
            self.flattenedPages.append(pageObj)


# creating an object
file = open('travelers.pdf', 'rb')

# creating a pdf reader object
fileReader = PdfFileReader2(file)

# print the number of pages in pdf file
no_of_pages = fileReader.numPages

pageObj = fileReader.getPage(page_no)
page = pageObj.extractText(Tj_sep='\n')
Aseem
  • 5,848
  • 7
  • 45
  • 69