0
Hi I'm facing a problem , I'm calling a website using jQuery AJAX to Scrape data from it , so there are 300 links which I want to scrape but the site shows only 50 at a time so we have to scroll down to load all those links can Someone Please Help me ?

var baseURL = "https://hr.myu.umn.edu/psc/hrprd/EMPLOYEE/HRMS/c/HRS_HRAM_FL.HRS_CG_SEARCH_FL.GBL?Page=HRS_APP_JBPST_FL&Action=U&SiteId=1&FOCUS=Applicant&JobOpeningId=";

var appendURL = "&PostingSeq=1";


var result = {};

jQuery.ajax({

async: false,

type: 'GET',

url: 'https://hr.myu.umn.edu/psc/hrprd/EMPLOYEE/HRMS/c/HRS_HRAM_FL.HRS_CG_SEARCH_FL.GBL?Page=HRS_APP_SCHJOB_FL&ACTION=U&FOCUS=Applicant&SiteId=1&',

success: function(resp) {

console.log('ajax success');

var content = $(".ps_box-grid");

   
var objDiv = document.getElementById("win0divHRS_AGNT_RSLT_I$grid$0");    objDiv.scrollTop = objDiv.scrollHeight;

for (var i = 0; i < jQuery('[class="ps_box-edit psc_disabled psc_has_value psc_num"] [class="ps_box-value"]', resp).length; i++) {

result[baseURL + jQuery('[class="ps_box-edit psc_disabled psc_has_value psc_num"] [class="ps_box-

value"]', resp)[i].innerText + appendURL] = {}

}

},

});

result;


Here's a Link related to a similar problem answered on stack overflow: 

https://stackoverflow.com/questions/34239525/scroll-to-bottom-of-div-using-jquery-if-ajax-returns-success/65727077?noredirect=1#comment116210292_65727077
Kashan Akhun
  • 3
  • 3

1 Answers1

0

I'm going to start off by saying I'm not 100% sure about how to do this or if this solution that I'm going to say is even going to work.

My suggestion (in a nutshell) is using your browser's Developer's Tools to record all the incoming and outgoing network requests sent between the clients website and the servers. Then search the requests for anything that seems like an update call for more info (not exactly sure what that looks like but I think you will know better than me).

Here are some points to get you going in the right direction (I use firefox Developer Edition so these will be firefox DE based. This also assumes you are a complete novice; manley for future people who may read this):

  • how to use firefox developer tools, Network tab: Look Here

  • How to record network traffic traffic: Look Here

  • if you click on a network traffic item you should see thing's like headers and cookies, you should also see request which is the server asking for certain info. you will need the requests tab.

Here are some steps that I would do (to help get the idea):

  1. Open network tab and start recording

  2. Refresh the page are scraping and delete old traffic if it has not auto removed any past traffic after refreshing the page.

  3. when recording scroll down and load the new content then stop recording.

  4. then search through the list of network traffic and for items with request parameters in them (most will be empty) the one you are looking for most likely comes from the same domain.

  5. find things that seem like it happens every time you scroll down then try to implement that request (with the parameters) into your code. and if it work then your done and if it did not then IDK.

Edit: hit Tab then enter and caused it to post before I was done.

Edit 2: Since the content is most likely loaded in from requests live in your browser. I just wanna say I have no idea how you would implement this into code using jquery except for the fact that you will most likely need to implement POST requests using AJAX. I am someone that when needing to scprape something I usually just use puppeteer.

SleepyCatGuy
  • 105
  • 2
  • 6
  • I got what you are saying , all this can be Done easily Using Puppeteer ,Axios , request , cheerio but the thing is I'm bound to jQuery AJAX , when opening that site and load dev tools I written Code which Scrolls down to bottom and Scrapes all the Data I want ,But Same thing I want calling the Code from remote place or a System , so when I get Response it get the Initial data which includes the starting 50 divs/li which it's showing . But I can't Get above that ] – Kashan Akhun Jan 15 '21 at 06:18
  • I'm going to assume you are trying to make a browser extension? There is really only way that I can think of and that is to create a separate Node.js server running on your machine (To scrape the website. e.g. puppeteer, Axios, JSDOM) and to have that talk to your client via e.g. axios, request. and AJAX. then just send data between them. have the node.js server scrape and your client just needs to talk to the server. – SleepyCatGuy Jan 15 '21 at 20:23