1

I am trying scrape the live data at the to of this page: https://www.wallstreet-online.de/devisen/euro-us-dollar-eur-usd-kurs/realtime

My current method:

import time
import re
import bs4 from bs4 import BeautifulSoup as soup
import requests
while (1==1):
   con =  requests.request('get','https://www.wallstreet- 
   online.de/devisen/euro-us-dollar-eur-usd-kurs/realtime', stream = True)
   page = con.text
   kursSoup = soup(page, "html.parser")
   kursDiv = kursSoup.find("div", {"class":"pull-left quoteValue"})
   print(kursDiv.span)
   del con
   del page
   del kursSoup
   del kursDiv
   #time.sleep(2)
print("end")

works but is not in sync with the data on the website. I dont really get why because i delete all the variables at the end of the loop so the result should change when the data on the website changes but seems to stay the same for a fixed amount of times. Does anyone know why or has a better way of doing this (Im a bloody beginner and have no idea how the site even works thats why im parsing the html).

Cœur
  • 37,241
  • 25
  • 195
  • 267
Golgorie Haus
  • 55
  • 9
  • I've rollback your edit: Please do not include a solution in the question itself; instead, please find it in the [revision history](https://stackoverflow.com/posts/51313195/revisions) and post it as an answer of its own. – Cœur Dec 01 '18 at 10:40

1 Answers1

2

It looks like that web page may be using JavaScript to populate and update that number. I'm not familiar with BeautifulSoup but I don't think it will run the JavaScript on the page to update that number.

You may want to use something like Chrome Developer Tools to keep an eye on the network tab. I looked and it looks like there is a websocket connection to wss://push.wallstreet-online.de/lightstreamer going on behind the scenes. You may want to use a websocket client Python library to read from this socket and either find some API docs or reverse engineer the data that comes from the socket. Good luck!

Aidan Rosswood
  • 1,212
  • 1
  • 10
  • 21
  • thank you very much can you suggest a particular websocket client library and or a tutorial or something like that on how it works (you are right bs does not run the javascript but i thought by reconnecting to the website everytime i should get the updateted data everytime) – Golgorie Haus Jul 12 '18 at 20:48
  • I've used this one before it gets the job done. https://github.com/websocket-client/websocket-client. I don't have any tutorials on hand sorry bud. – Aidan Rosswood Jul 12 '18 at 20:57