How to extract css data from already opened webpage in python

Question

There is a game using the hmtl canvas that places objects on the screen dynamically. The data is stored in divs and has an associated css id that store its position. You can observe this data using the dev tools and inspecting each element. I want to be able to get those x y positions in python but It has to be from an already opened and running tab (chrome or firefox). I tried beautifulsoup but it seems to open a new tab everytime. I need the data from the currently opened tab. Note that the data gets updated every so often. So, how do I go about doing that?

Alternatively can it be done via javascript but, how since there is already some javascript running on that tab?

Pleas try to make it more clear. What is your goal? what is it you are trying to do with the element position? — Santi, Dec 23 '19 at 21:27
Once I have the x and y coordinate I can then have the program automatically click on that position. That is all. Since items appear dynamically the positions are always changing. — Geo, Dec 23 '19 at 21:34
BeautifulSoup is definitely not what you need for this. In any case, this seems to be too broad/vague, and possibly off-topic. — AMC, Dec 23 '19 at 21:54
How about this. In a webpage there is a div with id junk
Find the x y position of that div and use that position to click on it. Once you click on it, that div is removed and a new div is genarated
Find its position and click on it. So on and so forth. — Geo, Dec 23 '19 at 22:00

score 0 · Answer 1 · answered Dec 27 '19 at 21:35

0

After reading your most recent comment, you should only need selenium for that. You don't need to find the x/y since you can have selenium click on specific divs with specific classes.

driver.findElementByClassName("junk1").click();

Then you can make it repeat on a certain interval with sleep and a for loop if you want. Here are some good resources: 1, 2, 3. Selenium documentation

answered Dec 27 '19 at 21:35

Joseph Rajchwald

487
5
13

Can selenium access the data I need asynchronously? – Geo Jan 20 '20 at 02:40

score 0 · Answer 2 · answered Jan 23 '20 at 00:58

I found a solution or at least part of the solution. The project is for fun but its turning out to be more complicated than originally expected and this is just the basic part.

What I did was to: 1)install greasemonkey addon for firefox. There is an equivalent for chrome 2)install flask

Then I wrote a script in greasemonkey. Again, probably not the greates code but, I'm sure it will help someone somewhere.

// ==UserScript==
// @name     js send data experiment
// @version  1
// @require       http://ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js
// @grant       GM.xmlHttpRequest
// ==/UserScript==

let data = [
  {"who":"greasemonkey", "what":"Hi there4"}
 ];

GM.xmlHttpRequest({
    method: "POST",
    url: "http://localhost:5000/worker2",
    data: JSON.stringify(data),
  headers: {
    "Content-Type": "application/x-www-form-urlencoded"
    }
});

I'm not sure about needing the ajax but you definitely need the @grant GM.xmlHttpRequest

Then I wrote this python script, ran it in the command line and it all worked.

#!flask/bin/python

import sys

from flask import Flask, render_template, request, redirect, Response
import random, json

app = Flask(__name__)

@app.route('/worker2', methods = ['POST'])
def worker2():
    # read json + reply
    data = request.get_json(force=True)
    print("**************** ")
    print(data)
    print("**************** ")
    return "success!!!"


if __name__ == '__main__':
    # run!
    app.run(debug=True)

How to extract css data from already opened webpage in python

2 Answers2