What I am doing is
- Google for some linkedin specific links
- Login to linkedin.com (successful)
- Revisit home page (it fails here)
- Extract some the the desired info from links I googled in first step
My scrapy bot fails at step 3. So my questions is, Is there any way to bypass this content security policy? If its please share your thoughts in detail thanks.
One of the related issue google points me to are CSP header and Eval Issue, put that is not helpful as it does not provide any details about proxy setup.
2017-12-29 10:59:47 [scrapy_splash.middleware] WARNING: Bad request to Splash: {u'info': {u'source': u'[string "..."]', u'message': u'Lua error: [string "..."]:32: JavaScript error: EvalError: Refused to evaluate a string as JavaScript because \'unsafe-eval\' is not an allowed source of script in the following Content Security Policy directive: "script-src \'report-sample\' \'sha256-[redacted...]=\' \'unsafe-inline\' static.licdn.com s.c.lnkd.licdn.com static-fstl.licdn.com static-src.linkedin.com https://www.linkedin.com/voyager/service-worker-push.js https://platform.linkedin.com/js/analytics.js static-exp1.licdn.com static-exp2.licdn.com s.c.exp1.licdn.com s.c.exp2.licdn.com static-lcdn.licdn.com s.c.lcdn.licdn.com https://www.linkedin.com/sc/ https://www.linkedin.com/scds/ https://qprod.www.linkedin.com/sc/ https://www.linkedin.com/sw.js https://www.linkedin.com/voyager/abp-detection.js".\n', u'type': u'LUA_ERROR', u'line_number': 32, u'error': u'JavaScript error: EvalError: Refused to evaluate a string as JavaScript because \'unsafe-eval\' is not an allowed source of script in the following Content Security Policy directive: "script-src \'report-sample\' \'sha256-[redacted...]=\' \'unsafe-inline\' static.licdn.com s.c.lnkd.licdn.com static-fstl.licdn.com static-src.linkedin.com https://www.linkedin.com/voyager/service-worker-push.js https://platform.linkedin.com/js/analytics.js static-exp1.licdn.com static-exp2.licdn.com s.c.exp1.licdn.com s.c.exp2.licdn.com static-lcdn.licdn.com s.c.lcdn.licdn.com https://www.linkedin.com/sc/ https://www.linkedin.com/scds/ https://qprod.www.linkedin.com/sc/ https://www.linkedin.com/sw.js https://www.linkedin.com/voyager/abp-detection.js".'}, u'type': u'ScriptError', u'description': u'Error happened while executing Lua script', u'error': 400} 2017-12-29 10:59:47 [scrapy.core.engine] DEBUG: Crawled (400) https://www.linkedin.com/feed/ via http://localhost:8050/execute> (referer: None) 2017-12-29 10:59:47 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <400 https://www.linkedin.com/feed/>: HTTP status code is not handled or not allowed
