Flutter: How to get Response from a url after the webpage is finished loading?

Question

I wanted to read through the HTML of a website from a dio get request in my flutter code. The website takes in an input, for my case in the URL and loads for some time before showing all the received data from the backend. Because it takes time to parse my request and fetch and display, my dio request only shows me half of the website, and the other half is not displayed because it is not yet fetched. I need help for making it wait or on how I can get the other half.

Here is my code:

 BaseOptions options = new BaseOptions(
      connectTimeout: 10000,
      receiveTimeout: 10000,
    );
    Dio dio = new Dio(options);
    String url = 'https://x.com/?url=https://x.x.x/x/';

    try {
      final response = await dio.get(url);
      print(response.data);
      final document = parse(response.data);
      final downloadPage = document.getElementsByClassName('results-list');

      for (Element resultsAll in downloadPage) {
        final downloadLinks =
            resultsAll.getElementsByClassName('download-link');
        print(resultsAll.classes.toString());
      }
    } catch (e) {
      print(e);
    }

triple7 · Answer 1 · 2021-02-12T10:40:27.887

I think the problem is not with your code, but with the site you are trying to load. The package you are using is waiting for the server to return an HTTP response code (200 on success, in most cases) and then it is closing the future and returning the response from the server.

I guess what's happening in your case is that the page you are trying to load is using JavaScript to load additional content after the initial page is loaded. Similar to how Facebook, Instagram and the others have an infinite feed to scroll through.

To counter this you would will not be able to use a request-based package to pull the data, but something more in-line with a Web View package. Such a package would load the site as if it was loaded in a browser, and you would then be able to read the data from the view.

You can have a look here, for an example: https://medium.com/@janithsg/flutter-extract-web-page-html-content-from-webview-1c39e43e675d

Flutter: How to get Response from a url after the webpage is finished loading?

1 Answers1