Get complete source code from a website generated on clicking a button

Question

The webpage for which I want the source code contains some products.

When I simply load it, it gives me only the source code of the first few products.

What I need is the complete source code after clicking the Show 140-250 items which is generated several times and get the code when there is no something like Show x - y items anymore. The id of the button being generated is see-more-products.

I have tried urllib, requests modules, which don't do the job as expected.

Link to that particular page is in comments.

Any help is appreciated. Thanks :)

Why downvoting without explanation? The question seems legitimate to me — galinette, Mar 06 '16 at 18:33
Why so many downvotes? I didn't find any solution anywhere.. It would be better if the downvoters could come up with a reason. — Prashanth, Mar 06 '16 at 18:34
You'll need to use `selenium-webdriver` to fetch such data. These sites are generated by a lot of DOM manipulation and hence you need to emulate a browser. — JRodDynamite, Mar 06 '16 at 18:34
Generally you just open your browser tools and look for requets patterns the page is generating when scrolling. Then program them with your python — Lol4t0, Mar 06 '16 at 18:35
Yes, I have tried even `selenium`. I use proxy with authentication to connect to the internet. `Selenium` emulates browser with a fresh profile where the browser isn't ready to run with a proxy. So, I can't use `selenium`. — Prashanth, Mar 06 '16 at 18:38
@galinette Didn't downvote, but: 1) There is no code in the question 2) we have no idea which site he's talking about and what exactly doesn't load or why. 3) The question is phrased in a confusing way. — Martin Tournoij, Mar 06 '16 at 18:39
I had provided the link in the question. It has been edited. — Prashanth, Mar 06 '16 at 18:41
If you look at the page in developer tools you can see how the data is requested — Padraic Cunningham, Mar 06 '16 at 18:49

Padraic Cunningham · Accepted Answer · 2016-03-06T19:50:25.477

There are a few different ways, to get you started

import requests
from bs4 import BeautifulSoup
for i in range(0, 3339, 48):
    r = requests.get("http://www.snapdeal.com/acors/json/product/get/search/175/{}/48".format(i))
    print([a["href"] for a in BeautifulSoup(r.content).select("div.product-desc-rating.title-section-expand a")])

Which will gives you all the phone links, a sample of which are:

['http://www.snapdeal.com/product/micromax-canvas-spark-2-4/658765871888', 'http://www.snapdeal.com/product/alpha-cod/639184667095', 'http://www.snapdeal.com/product/apple-iphone-5s-16-gb/1204769399', 'http://www.snapdeal.com/product/micromax-canvas-a1-aq4502-8/630310793485', 'http://www.snapdeal.com/product/infocus-m370-8gb/685769734109', 'http://www.snapdeal.com/product/lenovo-a1000-8gb-black/634810874852', 'http://www.snapdeal.com/product/infocus-m350-16gb/667944684160', 'http://www.snapdeal.com/product/motorola-moto-x-force-32gb/677683041597', 'http://www.snapdeal.com/product/infocus-m812i-16gb-silver/672166899922', 'http://www.snapdeal.com/product/xolo-black-1x-32gb/633488491545', 'http://www.snapdeal.com/product/meizu-m2-m578h-4g-16gb/632749055332', 'http://www.snapdeal.com/product/iphone-6s-16gb/663413326062', 'http://www.snapdeal.com/product/apple-iphone-6-16-gb/1270529654', 'http://www.snapdeal.com/product/apple-iphone-5s-16-gb/347830397', 'http://www.snapdeal.com/product/micromax-s301-4gb-black/668491526095', 'http://www.snapdeal.com/product/infocus-m260-zl2-8gb/619518605458', 'http://www.snapdeal.com/product/micromax-juice-2-aq5001-dual/625046032091', 'http://www.snapdeal.com/product/swipe-konnect-51-8gb-sand/626439596420', 'http://www.snapdeal.com/product/lenovo-a6000-black/670500124351', 'http://www.snapdeal.com/product/redmi-2-prime-16-gb/683138068892', 'http://www.snapdeal.com/product/asus-zenfone-go-8gb/646618265597', 'http://www.snapdeal.com/product/karbonn-aura-8gb-3g/627156357326', 'http://www.snapdeal.com/product/panasonic-eluga-turbo-32gb-marine/644981636428', 'http://www.snapdeal.com/product/asus-zenfone-2-laser-ze550kl/633082651487', 'http://www.snapdeal.com/product/redmi-2-8-gb-grey/628016512272', 'http://www.snapdeal.com/product/infocus-m680-16gb-4g/649492687203', 'http://www.snapdeal.com/product/lava-iris-atom-8gb-white/665532304471', 'http://www.snapdeal.com/product/micromax-x073-black/632400291797', 'http://www.snapdeal.com/product/intex-aqua-3g-pro-4gb/661217697971', 'http://www.snapdeal.com/product/micromax-x605-black/663717815370', 'http://www.snapdeal.com/product/micromax-x088-red/550645299', 'http://www.snapdeal.com/product/micromax-unite-3-q372/639750556852', 'http://www.snapdeal.com/product/micromax-canvas-selfie-2-q340/673685061115', 'http://www.snapdeal.com/product/samsung-j7-16gb-espresso-brown/661359071561', 'http://www.snapdeal.com/product/hpl-a50-white-dual-core/752819001', 'http://www.snapdeal.com/product/intex-intex-aqua-power-hd/677955090549', 'http://www.snapdeal.com/product/samsung-galaxy-core-2-4gb/95934411', 'http://www.snapdeal.com/product/samsung-galaxy-j2-8gb/637947695307', 'http://www.snapdeal.com/product/micromax-bolt-x088/1252626810', 'http://www.snapdeal.com/product/lenovo-a7000-turbo-16gb-matte/676482356343', 'http://www.snapdeal.com/product/nokia-105-dual-sim-black/625877378012', 'http://www.snapdeal.com/product/lenovo-vibe-x2ap/619606468183', 'http://www.snapdeal.com/product/xolo-hive-8x1000i-16-gb/620631303689', 'http://www.snapdeal.com/product/panasonic-p41-black-android-dual/1202820682', 'http://www.snapdeal.com/product/asus-zenfone-2-laser-ze500kl/686932521097', 'http://www.snapdeal.com/product/samsung-galaxy-grand-max/647437791381', 'http://www.snapdeal.com/product/panasonic-p55-novo-8-gb/630374183875', 'http://www.snapdeal.com/product/samsung-tizen-z1/214936653']
['http://www.snapdeal.com/product/micromax-canvas-juice-3-q392/675338203115', 'http://www.snapdeal.com/product/micromax-x853-256-mb-gray/678836590088', 'http://www.snapdeal.com/product/iball-prince-2-black/683878915532', 'http://www.snapdeal.com/product/intex-cloud-swift-4g-16gb/651487709065', 'http://www.snapdeal.com/product/htc-desire-820-white/2096877771', 'http://www.snapdeal.com/product/celkon-2gb-star-16gb/668257971604', 'http://www.snapdeal.com/product/lava-iris-atom-8gb-black/671938280548', 'http://www.snapdeal.com/product/asus-zenfone-max-zc550kl-16gb/659567821492', 'http://www.snapdeal.com/product/micromax-x084/1189906548', 'http://www.snapdeal.com/product/infocus-bingo-21-m430-8gb/685620250128', 'http://www.snapdeal.com/product/xolo-black-4g-16gb-with/670501820959', 'http://www.snapdeal.com/product/nokia-130-dual-sim-black/1238158129', 'http://www.snapdeal.com/product/phicomm-e670-energy-2-16gb/642536836073', 'http://www.snapdeal.com/product/lenovo-a7000-8-gbwhite/660909822842', 'http://www.snapdeal.com/product/infocus-m260-8gb-yellow-3g/658001307243', 'http://www.snapdeal.com/product/gionee-m5l-32gb-golden-4g/664586124833', 'http://www.snapdeal.com/product/spice-gaming-mobile-x2-red/673912591179', 'http://www.snapdeal.com/product/lenovo-lenovo-a1000-8gb-white/656199391476', 'http://www.snapdeal.com/product/xolo-cube-50-8gb-black/681508479062', 'http://www.snapdeal.com/product/microsoft-lumia-540-8-gb/641216010217', 'http://www.snapdeal.com/product/infocus-m810-16gb-gold-4g/683493537817', 'http://www.snapdeal.com/product/samsung-guru-music-2-sm/672045180352', 'http://www.snapdeal.com/product/samsung-guru-music-2-duos/197236301', 'http://www.snapdeal.com/product/intex-cloud-cube-8gb-gray/678043979304', 'http://www.snapdeal.com/product/samsung-tizen-z3-8gb/638555381978', 'http://www.snapdeal.com/product/micromax-canvas-spark/634407752835', 'http://www.snapdeal.com/product/moto-e-2nd-gen-3g/627496057276', 'http://www.snapdeal.com/product/intex-cloud-breeze-8gb-grey/671476050074', 'http://www.snapdeal.com/product/samsung-j5-8gb-espresso-brown/659785514599', 'http://www.snapdeal.com/product/lg-google-nexus-5-16/848745269', 'http://www.snapdeal.com/product/mi4i-16gb/654856488809', 'http://www.snapdeal.com/product/micromax-joy-1850-35-kb/673275470330', 'http://www.snapdeal.com/product/phicomm-energy-653-4g-8gb/621455892516', 'http://www.snapdeal.com/product/panasonic-eluga-icon-16gb/650547748150', 'http://www.snapdeal.com/product/lenovo-vibe-x2ap-32gb-white/660083187849', 'http://www.snapdeal.com/product/xolo-prime/683763827810', 'http://www.snapdeal.com/product/meizu-mx5-silver-black/618643364985', 'http://www.snapdeal.com/product/karbonn-titanium-s205-2gb-16gb/662465421207', 'http://www.snapdeal.com/product/micromax-canvas-nitro-4g-e455/667663492017', 'http://www.snapdeal.com/product/i-kall-k88-gray/659450396588', 'http://www.snapdeal.com/product/karbonn-mach-two-titanium-s360/653888615715', 'http://www.snapdeal.com/product/asus-zenfone-2-laser-55/681651791446', 'http://www.snapdeal.com/product/i-kall-k55-fruity-orange/621445557898', 'http://www.snapdeal.com/product/yu-yunique-8gb/621238835100', 'http://www.snapdeal.com/product/htc-desire-326g-dual-sim/673825142381', 'http://www.snapdeal.com/product/micromax-h375-below-256-mb/626813860848', 'http://www.snapdeal.com/product/intex-intex-aqua-star-2/677150759819', 'http://www.snapdeal.com/product/micromax-canvas-nitro-2/674813358269']
['http://www.snapdeal.com/product/karbonn-s-15-plus-8gb/678547994280', 'http://www.snapdeal.com/product/huawei-honor-bee-8gb-3g/623753457685', 'http://www.snapdeal.com/product/intex-cloud-m6-16-gb/652438065304', 'http://www.snapdeal.com/product/karbonn-s310/2046781351', 'http://www.snapdeal.com/product/htc-desire-826/673132800168', 'http://www.snapdeal.com/product/oneplus-one-64gb/683926070447', 'http://www.snapdeal.com/product/samsung-galaxy-grand-prime-4g/682950239423', 'http://www.snapdeal.com/product/panasonic-panasonic-eluga-switch-32gb/651502528631', 'http://www.snapdeal.com/product/infocus-f120-dual-sim-gsm/647132622326', 'http://www.snapdeal.com/product/samsung-j5-8gb-white/625632131578', 'http://www.snapdeal.com/product/micromax-canvas-juice-3-q394/647451126738', 'http://www.snapdeal.com/product/micromax-x610-black/677067716681', 'http://www.snapdeal.com/product/videocon-v1393-dual-sim-mobile/519151652', 'http://www.snapdeal.com/product/nokia-105/1397084', 'http://www.snapdeal.com/product/micromax-canvas-blaze-4g-q400/635578549162', 'http://www.snapdeal.com/product/lenovo-a-6000-shot-16gb/652479804547', 'http://www.snapdeal.com/product/micromax-x597-black/666857291079', 'http://www.snapdeal.com/product/lenovo-a2010-16gb-black/641561931090', 'http://www.snapdeal.com/product/micromax-x597-blue/635774497484', 'http://www.snapdeal.com/product/lenovo-a1000-8gb-white-3g/682465160925', 'http://www.snapdeal.com/product/oorie-gsm-4gb-gray/644950735714', 'http://www.snapdeal.com/product/videocon-v1429w-black/659876102140', 'http://www.snapdeal.com/product/wham-q4-8gb-black/683618702751', 'http://www.snapdeal.com/product/lenovo-s60-8gb-white/682858759745', 'http://www.snapdeal.com/product/lenovo-a6000-white-8gb-white/687038221568', 'http://www.snapdeal.com/product/k2-air-8gb-black/662606076269', 'http://www.snapdeal.com/product/infocus-m680-16gb-4g/623852692656', 'http://www.snapdeal.com/product/micromax-x707-gray/675166544662', 'http://www.snapdeal.com/product/micromax-bolt-a-24-champagne/318667695', 'http://www.snapdeal.com/product/htc-desire-620-g/2140463474', 'http://www.snapdeal.com/product/whitecherry-k9-gray/634412911071', 'http://www.snapdeal.com/product/intex-neo-v-plus-black/49904988', 'http://www.snapdeal.com/product/intex-aqua-power-hd-mobile/639191044460', 'http://www.snapdeal.com/product/i-kall-k88-black/624856889024', 'http://www.snapdeal.com/product/apple-iphone-6-64-gb/131798351', 'http://www.snapdeal.com/product/infocus-f135-dual-sim-gsm/662122536540', 'http://www.snapdeal.com/product/apple-iphone-5s-32-gb/936530455', 'http://www.snapdeal.com/product/micromax-bolt-q338-8gb-black/633509849227', 'http://www.snapdeal.com/product/micromax-x084-black/676316493911', 'http://www.snapdeal.com/product/infocus-f110-dual-sim-gsm/668147567472', 'http://www.snapdeal.com/product/kenxinda-watch-mobile-dual-sim/149800122', 'http://www.snapdeal.com/product/panasonic-eluga-s/1270286256', 'http://www.snapdeal.com/product/swipe-virtue-16gb-white-3g/633490528209', 'http://www.snapdeal.com/product/micromax-canvas-play-q355-dual/663710099854', 'http://www.snapdeal.com/product/samsung-tizen-z1-4gb-gold/683743955020', 'http://www.snapdeal.com/product/asus-zenfone-go-45-zc451tg/632893288226', 'http://www.snapdeal.com/product/karbonn-a6-turbo/677649317861', 'http://www.snapdeal.com/product/karbonn-titanium-delight-s22-blackgrey/679556902984']

You might have to go tweak this a little bit but this should get close to getting all the links you want:

import requests

from bs4 import BeautifulSoup
def get_all_links():
    r = requests.get("http://www.snapdeal.com/acors/json/product/get/search/175/0/48")
    total = int((BeautifulSoup(r.content).select("div.jsNumberFound.hidden")[0].text))
    yield from (a["href"] for a in BeautifulSoup(r.content).select("div.product-desc-rating.title-section-expand a"))
    print("There are {} total items".format(total))
    for pge in range(50, total+1, 48):
        r = requests.get("http://www.snapdeal.com/acors/json/product/get/search/175/{}/48".format(pge))
        yield  from (a["href"] for a in BeautifulSoup(r.content).select("div.product-desc-rating.title-section-expand a"))


There are 3340 total items
http://www.snapdeal.com/product/micromax-canvas-spark/634407752835
http://www.snapdeal.com/product/iphone-6s-16gb/663413326062
http://www.snapdeal.com/product/apple-iphone-5s-16-gb/1204769399
http://www.snapdeal.com/product/moto-e-2nd-gen-3g/627496057276
http://www.snapdeal.com/product/redmi-2-prime-16-gb/683138068892
http://www.snapdeal.com/product/micromax-juice-2-aq5001-dual/625046032091
http://www.snapdeal.com/product/alpha-cod/639184667095
http://www.snapdeal.com/product/intex-cloud-breeze-8gb-grey/671476050074
http://www.snapdeal.com/product/micromax-canvas-spark-2-4/658765871888
http://www.snapdeal.com/product/infocus-m370-8gb/685769734109
http://www.snapdeal.com/product/infocus-m350-16gb/667944684160
http://www.snapdeal.com/product/lg-google-nexus-5-16/848745269
http://www.snapdeal.com/product/lenovo-a6000-black/670500124351
http://www.snapdeal.com/product/apple-iphone-6-16-gb/1270529654
http://www.snapdeal.com/product/meizu-m2-m578h-4g-16gb/632749055332
http://www.snapdeal.com/product/micromax-canvas-a1-aq4502-8/630310793485
http://www.snapdeal.com/product/apple-iphone-5s-16-gb/347830397
http://www.snapdeal.com/product/infocus-bingo-21-m430-8gb/685620250128
http://www.snapdeal.com/product/samsung-j7-16gb-espresso-brown/661359071561
http://www.snapdeal.com/product/apple-iphone-6-64-gb/131798351
http://www.snapdeal.com/product/panasonic-p55-novo-8-gb/630374183875
http://www.snapdeal.com/product/lenovo-a1000-8gb-black/634810874852
http://www.snapdeal.com/product/infocus-m260-zl2-8gb/619518605458
http://www.snapdeal.com/product/asus-zenfone-2-laser-ze500kl/686932521097
http://www.snapdeal.com/product/meizu-mx5-silver-black/618643364985
http://www.snapdeal.com/product/moto-e-2nd-gen-4g/682444504031
http://www.snapdeal.com/product/xolo-black-1x-32gb/633488491545
http://www.snapdeal.com/product/panasonic-eluga-turbo-32gb-marine/644981636428
http://www.snapdeal.com/product/mi4i-16gb/654856488809
http://www.snapdeal.com/product/samsung-j5-8gb-espresso-brown/659785514599
http://www.snapdeal.com/product/intex-cloud-swift-4g-16gb/651487709065
http://www.snapdeal.com/product/lenovo-vibe-x2ap-32gb-white/660083187849
http://www.snapdeal.com/product/panasonic-eluga-icon-16gb/650547748150
http://www.snapdeal.com/product/micromax-s301-4gb-black/668491526095
http://www.snapdeal.com/product/apple-iphone-5s-32-gb/936530455
http://www.snapdeal.com/product/lenovo-a7000-turbo-16gb-matte/676482356343
http://www.snapdeal.com/product/samsung-tizen-z3-8gb/638555381978
http://www.snapdeal.com/product/oneplus-one-64gb/683926070447
http://www.snapdeal.com/product/asus-zenfone-2-laser-55/681651791446
http://www.snapdeal.com/product/htc-desire-826/673132800168
http://www.snapdeal.com/product/gionee-m5l-32gb-golden-4g/664586124833
http://www.snapdeal.com/product/intex-intex-aqua-power-hd/677955090549
http://www.snapdeal.com/product/samsung-a7-2016-16gb-4g/632787468738
http://www.snapdeal.com/product/micromax-unite-3-q372/639750556852

That page was an example. The number products may change according to the product I choose. Anyways thanks for your response :) — Prashanth, Mar 06 '16 at 19:11
So, I should change the number manually for every page, if I'm correct? — Prashanth, Mar 06 '16 at 19:12
@Prashanth, the output above is for 3 pages it goes until you hit the max amount which you will have to tweak yourself, that information is on the first page if you look. You will have 3000+ links by the end — Padraic Cunningham, Mar 06 '16 at 19:15
Yes, that's a great idea sir! I can get the value from the first page itself. Thanks a lot :) — Prashanth, Mar 06 '16 at 19:17
Yep, if you look at the developer console you can see the urls are the params that are used, also check out http://www.snapdeal.com/acors/json/product/get/search/175/100/47 and you will see how it all adds up — Padraic Cunningham, Mar 06 '16 at 19:20
No worries, I added a function that should be close to getting all the links, the max items seems to be 48 after an initial 50 for the first query — Padraic Cunningham, Mar 06 '16 at 19:51

Get complete source code from a website generated on clicking a button

1 Answers1