I am currently working with PDFMiner.six to extract text from multiple PDFs. Looking at my output I can see that I get some weird conversions of special characters like brakets:
Opening and closing brackets:
Finally, I delete all paragraphs 共defined as two lines containing text with a blank line before and after兲 with more than 50 percent
Other brackets:
具TEXT典
Plus:
Words+Tables
WORDS⫹TABLES
Minus:
(-0.141)
共⫺1.41兲
Test of (SML * COMPLEX-LRG * COMPLEX) < 0
Test of 共SML ⴱ COMPLEX⫺LRG ⴱCOMPLEX兲 ⬍ 0
I am using the following code:
from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter
from pdfminer.converter import TextConverter
from pdfminer.layout import LAParams
from pdfminer.pdfpage import PDFPage
from io import StringIO
import os
import re
# from datetime import datetime
# a = datetime.now()
i = 0
path = r"C:\Users\1_T_Python"
save_to = r"C:\Users\1_T\txt files"
for filename in os.listdir(path):
if filename.endswith(".pdf"):
rsrcmgr = PDFResourceManager()
retstr = StringIO()
codec = 'utf-8'
laparams = LAParams()
device = TextConverter(rsrcmgr, retstr, codec=codec, laparams=laparams)
fp = open(path+"\\"+filename,'rb')
interpreter = PDFPageInterpreter(rsrcmgr, device)
password = ""
caching = True
pagenos=set()
for page in PDFPage.get_pages(fp, pagenos, password=password,caching=caching, check_extractable=True):
interpreter.process_page(page)
fp.close()
device.close()
string = retstr.getvalue()
retstr.close()
#print(string)
with open(save_to+"\\"+filename+".txt", "w", encoding="utf-8") as text_file:
text_file.write(string)
i = i+1
print(i)
I think this is an encode/decode issue, however could not find any solution on SO so far. Using utf-8 as encoding, I thought this should handle the problem, but it did not....
Any help appreciated!