0

I built many spiders to get news articles from different websites and i have an api to convert the text to audio clips, but i need a framework or python tools to refine the articles' text such as:

removing anything related to the source. removing any dates formats. removing urls. change acronyms such as CEO to chief excution officer for example. removing special characters and typos.

making sure that the sentence is written correctly after all the edits. use the previously edited articles as a reference for the new articles.

I am using python, nltk and re, but it's exhausting and each time i think i covered all the cases, i find new cases to add and i think i am stuck in an infinite loop.

Any suggestions?

Hat hout
  • 471
  • 1
  • 9
  • 18

1 Answers1

0

First of all, expanding acronyms to their full form is non-trivial and should probably not be considered part of scraping but rather part of a second step of processing (cf. IBM's The Art of Tokenization).

Cleaning scraped data is tedious, unfortunately: There is no magical solution because everyone is interested in scaping something different than what you are — some might be interested only in URLs, for example. Nevertheless, have you not tried using BeautifulSoup? — it's a Python library which offers a very nice API for handling many common scraping-related tasks.

errantlinguist
  • 3,658
  • 4
  • 18
  • 41