3

I'm trying to make requests with headless chrome using pyppeteer. But I keep getting "OSError: [Errno 24] Too many open files" after a certain amount of requests. I checked the open resources of the python process with losf and found out that with every new request there's a new line like the following

python3 14840 root   11r  FIFO        0,8      0t0 64208510 pipe

Can someone tell me what resources aren't being closed? The code that's producing this error is below

def search(self, search_path):
    async def main(url):
        browser = await launch(args=['--no-sandbox'], handleSIGINT=False, handleSIGTERM=False, handleSIGHUP=False)
        page = await browser.newPage()
        await page.setJavaScriptEnabled(False)
        try:
            response = await page.goto(url, options={"timeout": 50000})
        except pyppeteer.errors.TimeoutError:
            pass

        src = await page.content()
        await browser.close()
        return src

    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    url = "https://www.example.com" + search_path
    val = asyncio.get_event_loop().run_until_complete(main(url))
    loop.close()

EDIT

I managed to close the open pipes by calling

browser.process.communicate() 
Serwj
  • 43
  • 1
  • 8

0 Answers0