0

Would like to get calendar data from page like http://www.dukascopy.com/swiss/english/marketwatch/calendars/eccalendar/ The data is dynamic and loads with js-applet - "DukascopyApplet" I tried to extract data this way:

app = QApplication(sys.argv)
web = QWebView()
web.page().mainFrame().loadFinished.connect(print_content)
web.load(QUrl("http://www.dukascopy.com/swiss/english/marketwatch/calendars/eccalendar/"))
web.show()

frame = web.page().mainFrame() 

print(frame.evaluateJavaScript("DukascopyApplet"))

sys.exit(app.exec_())

But it shows "none". I am using win7, Py3.3, pyside.

So the question is how to get the result of js code and/or applet, which exists on the loaded page?

user3003873
  • 543
  • 1
  • 4
  • 21

2 Answers2

1

Deciding by PySide documentation for QWebFrame.evaluateJavaScript, it...

Evaluates the JavaScript defined by scriptSource using this frame as context and returns the result of the last executed statement.

Se definitely it should be returning something from the JS.

Therefore, the only way to get an idea why you're getting None out of that method invocation, is to know what is the actual value of the "last executed statement" on the JS side. (As a side note, it's possible that the JS side is actually returning a null or undefined value, which would be correctly mapped to Python as None and printed as "None".)

I propose you first try to execute a very very simple snippet of Javascript that returns a constant hardcoded value known to you, and see if that works out. If the problem persists, I would first suggest solving it on that very very simple snippet, and only once that's working, moving on to your real Javascript code.

The lesson of the story is that debugging should start by reducing the problem incrementally so that at some point the problem disappears, and then working out the exact moment that it appears again. Otherwise you're just fighting something you haven't even found/seen.

Erik Kaplun
  • 37,128
  • 15
  • 99
  • 111
  • Ok lets suppose that scripts run correctly and print "none" is right result. But one question I need to be answered. When I select for example in firefox calendar data and click "inspect element", it shows me all the data and even loaded, so I suppose that there is buffer where the loaded data is written. Is it possible to access to such buffer in pyside. My script loads all data and SHOWS it in the GUI, but where rendered data stored to get it? – user3003873 Nov 18 '13 at 12:43
  • 1
    I'm sorry but this has absolutely nothing to do with what you asked—I have no idea how the JS script you're using behaves and works internally; you will have to read its documentation or its source code, and play around with it to learn how to extract data from it. It is not possible to answer that based on the information you've given. – Erik Kaplun Nov 18 '13 at 12:46
  • You might end up having to modify your 3rd party provided JS code in order for it to give you the data you need as a Javascript object as opposed to putting it in the DOM and throwing away the so called "buffer". – Erik Kaplun Nov 18 '13 at 12:48
0

The QWebFrame.evaluateJavaScript will do it, but it doesn't work that well. It doesn't always return the right type, and I've found that it always returns None on function calls. One way of getting the result is to set the returned item to a variable you have access to then call evaluateJavaScript again on that variable.

frame.evaluateJavaScript("myVariable = getResult()")
result = frame.evaluateJavaScript("myVariable")

I just looked at the "DukascopyApplet", and its not a typical data type. Only standard data types like strings, ints, float, bools ... can be transferred or returned to python. You will probably have to go through the "DukascopyApplet" and find the specific data you want. Another thing that may help is to attach a Python object to the JavaScript. You can then call that python object's slot methods inside of your JavaScript.

class MyCLass(object):
    @QtCore.Slot(str)
    def doSomething(self, info):
        # do something with the string info here

frame.addToJavaScriptWindowObject("varName", MyClass)
frame.evaluateJavaScript("varName.doSomething(DukascopyApplet.params.height)")

web.settings().setAttribute(QtWebKit.QWebSettings.WebAttribute.DeveloperExtrasEn‌​abled, True)
inspector = QtWebKit.QWebInspector()
inspector.setPage(web.page())

After looking through the DukascopyApplet it doesn't look like there is any useful stored information there. Finding that data may be difficult.

justengel
  • 6,132
  • 4
  • 26
  • 42
  • Thank you! But I am very novice in js and trying your code by copy-paste, the same result. Also trying 'frame.evaluateJavaScript("myVariable = DukascopyApplet.getResult()") result = frame.evaluateJavaScript("myVariable")' Also - none. – user3003873 Nov 23 '13 at 18:49
  • One thing that might help you is to look at the JavaScript console. you can do this with web.settings().setAttribute(QtWebKit.QWebSettings.WebAttribute.DeveloperExtrasEnabled, True) and then you can set an inspector inspector = QtWebKit.QWebInspector(). inspector.setPage(web.page()). After Than you can right click on the browser and look in the console for your variables and methods. – justengel Nov 26 '13 at 13:29