1

I want to scrape the x and y axis of a highcharts graph. As shown here:

https://www.highcharts.com/demo/line-basic

I´m using html_requests which uses pypeteer to send JavaScript.

chart = r.get("a.com")
script= """return Highcharts.charts[0].series[0].data.map(d=>d.y);"""
    
chart.html.render(script=script, reload=False)

Now this code results in the following error:

pyppeteer.errors.ElementHandleError: Evaluation failed: SyntaxError: Unexpected token return

I tried another variation of the code:

 script='''values = [];
 Highcharts.charts[0].series[0].data.forEach((d) => values.push(d.y));
 return values;'''

which results in:

pyppeteer.errors.ElementHandleError: Evaluation failed: SyntaxError: Unexpected token return ;

Can someone explain what is happening? Is the response the problem or the JS code itself?

ragnarswanson
  • 315
  • 3
  • 10
MaxFrost
  • 91
  • 6

1 Answers1

1

Working example of what you want to do. I wrote and tested.

from requests_html import HTMLSession

session = HTMLSession()

script = ('() => {'
          '    return Highcharts.charts[0].series[0].data.map(d=>d.y);'
          '}')

resp = session.get('https://www.highcharts.com/demo/line-basic')
chartdata = resp.html.render(script=script, reload=False)
print(chartdata)

A method html.render takes an argument script in this format:

() => {
    return document.title;
}

And it returns a result of execution of the script.

Updated

The text below is related to pyppeteer and its work. I check my first examples using exactly pyppeteer, but requests_html working by a litle different principle. For instance, it is correct for first library and not correct for second:

js_code = 'Highcharts.charts[0].series[0].data.map(d=>d.y);'
data = await page.evaluate(js_code)

I have experimented with variety of js code. I guess in python implementation of puppeteer that works like this. An argument is a piece of js code is passed to python function. There it is wrapped by an anonymous javascript function and last statement of the user-defined code is joined with return operator. After this process the formed anonymous function is performed. Attention! I did not look at code of the library. It is only conjecture based on executing various statements using pyppeteer. Sorry for my English if you have questions because you do not understand what I have written. Please, ask me.

Арман Айрапетов
  • 28
  • 5
  • Thank you so much for taking the time to answer! Unfortunately removing 'return' made no difference. I get this error: pyppeteer.errors.ElementHandleError: Evaluation failed: SyntaxError: Unexpected token ; – MaxFrost Oct 03 '22 at 12:08
  • I just tried this: "values = [];Highcharts.charts[0].series[0].data.forEach(function(d){ values.push(d.y) });console.log(values);" The code executes for some reason but how do I retrieve the output to the console now? – MaxFrost Oct 03 '22 at 12:31
  • @MaxFrost check the updated answer, please. I fix my mistake. – Арман Айрапетов Oct 03 '22 at 15:21