Saving Image from URL using Python Requests - URL type error

Question

Using the following code:

    with open('newim','wb') as f:
        f.write(requests.get(repr(url)))

where the url is:

    url = 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAFoAAAArCAYAAAD41p9mAAAAzUlEQVR42u3awQ4DIQhFUf7/p9tNt20nHQGl5yUuh4c36BglgoiIiIiIiGiVHq+RGfvdiGG+lxKonGiWd4vvKZNd5V/u2zXRO953c2jx3bGiMrewLt+PgbJA/xJ3RS5dvl9PEdXLduK3baeOrKrc1bcF9MnLP7WqgR4GOjtOl28L6AlHtLSqBhpooIEGGmiggQYaaKCBBhpodx3H3XW4vQN6HugILyztoL0Zhlfw9G4tfR0FfR0VnTw6lQoT0XtXmMxfdJPuALr0x5Pp+wT35KKWb6NaVgAAAABJRU5ErkJggg=='

I get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python33\lib\site-packages\requests\api.py", line 69, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Python33\lib\site-packages\requests\api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "C:\Python33\lib\site-packages\requests\sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python33\lib\site-packages\requests\sessions.py", line 567, in send
    adapter = self.get_adapter(url=request.url)
  File "C:\Python33\lib\site-packages\requests\sessions.py", line 641, in get_adapter
    raise InvalidSchema("No connection adapters were found for '%s'" % url)

I have seen other posts with what, at first glance, appears to be a similar problem but I haven't had any luck just adding 'https://' or anything like that...I seriously want to avoid having to do this in webdriver+Autoit or something because I have to do a similar exercise for thousands of images.

Seems this is your problem http://stackoverflow.com/questions/15115328/python-requests-no-connection-adapters — Laraconda, Oct 09 '15 at 23:54

sokin · Answer 1 · 2015-10-10T06:08:09.787

There seems to be a problem with your understanding of the concept of embedded images. The url you have posted is, actually, what your browser returns when you select 'View Image' or 'Copy Image Location' (or something similar, depending on the browser) from the context menu, and formally is called a data URI.

It is not an http url pointing to an image, and you can not use it to retrieve actual images from any server: this is exactly what requests points out in the error message.

So, how do we get these images? The following script will handle this task:

import requests
from lxml import html
import binascii as ba

i = 0
url="<Page URL goes here>" #Ex: http://server/dir/images.html
page = requests.get(url)
struct = html.fromstring(page.text)
images = struct.xpath('//img/@src')

for img in images:
    i += 1
    ext = img.partition('data:image/')[2].split(';')[0]
    with open('newim'+str(i)+'.'+ext,'wb') as f:
        f.write(ba.a2b_base64(img.partition('base64,')[2]))

print("Done")

To run it you will need to install, along with requests, the lxml library which can be found here.

Here follows a short description of how the script functions:

First it requests the url from the server and, after it gets the server's response, it stores it in a Response object (page).

Then it utilizes html.fromstring() from lxml to transform the "textified" content of page into a tree-structure which can be processed by commands utilizing XPath syntax, like this one: images = struct.xpath('//img/@src').

The result is a list containing the contents of the src attribute of every image in the page. In this case (embedded images) these are the data URIs.

Then, for every image in the list, it first gets the image type (which will be used as the newim's extension), using partition() and split() and stores it in ext. Then it converts the base64 encoded data to binary (using a2b_base64() from binascii module) and writes the output to the file.

As a small demo, save this html code (as, eg, images.html) somewhere in your server

<h1>Images</h1>
<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEcAAAAmCAYAAACWGIbgAAACKElEQVR4nO2aPWsUQRiAnznSKYI/IbBJoTkCEhDuuhQnMY2VhdjYbAoLO9N4VfwBlyKFFlYKYqNN9iCFnQeCCHrJFdkFf4IgKcO9FnLJJtzOfs3OGJgHtpi5nZl3H+abUyIieObSch3A/4yXo8HL0eDlaPByNHg5GrwcDV6OhnJyPt0bK6XYGhqMYLiFUir1dNlNDNafpmz8kkskU1gHZPaEUX6pfGKZ9nkOSPe9xJfz6AxmmTWpHn+2nDe8S1doVk5KAqFcqC4Kz9uq05CB+OfL0VRsRM7H3s9MAfFAWnCtVluG4p8/5zyRR/JPHBIPaMH1gqO0AAny/eBt5s/BMqdwd5Z8/XKX0lOQofjtr1bJPgs77BV+f/SB/aYm6BzsyzmMxlM4KV5gxCRuLhwd9uX8PhiXLXL0p/zIMoFlOQnyix/pnO76Ru6Hf/k8DJqLKZursUM+PHbSdSzLiWGHb3bbrM7V6DmOsCxnCfqslS62soyLSceynAC1yGrZUkUm7SawP6xu9trpZJGV6PYNJx3HgZyV++1y2/kOt5aaC0eHfTnBJqd9nhZ+v/OQTSf9xslqFaDu9B6fJS/vYZJjFuDrLBm+eOZmTFFRTu3t/IO99rTPNgCjCReOTvGEs7NXGPFqo1ZLcykcf6W7ESO3dDk3gWauG2vFX+myK/2cf1hF0jd/INCRQV3zhuJXIv5fFln444MGL0eDl6PBy9Hg5WjwcjR4ORr+Aq7+02kTcdF1AAAAAElFTkSuQmCC" />  
<br />
<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAFoAAAArCAYAAAD41p9mAAAAzUlEQVR42u3awQ4DIQhFUf7/p9tNt20nHQGl5yUuh4c36BglgoiIiIiIiGiVHq+RGfvdiGG+lxKonGiWd4vvKZNd5V/u2zXRO953c2jx3bGiMrewLt+PgbJA/xJ3RS5dvl9PEdXLduK3baeOrKrc1bcF9MnLP7WqgR4GOjtOl28L6AlHtLSqBhpooIEGGmiggQYaaKCBBhpodx3H3XW4vQN6HugILyztoL0Zhlfw9G4tfR0FfR0VnTw6lQoT0XtXmMxfdJPuALr0x5Pp+wT35KKWb6NaVgAAAABJRU5ErkJggg=="></img>
<br />
<img src="data:image/jpg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQH/2wBDAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQH/wAARCAAhADYDASIAAhEBAxEB/8QAHAABAQACAgMAAAAAAAAAAAAAAAoICQILBAYH/8QAKxAAAQQDAAICAQIGAwAAAAAABQMEBgcBAggACQoWFRQhERMXNjh2d7W2/8QAGQEAAgMBAAAAAAAAAAAAAAAAAAQBAgMF/8QAIxEAAgMAAgMBAQEAAwAAAAAAAwQBAgUSEwAGERQjIRUiJP/aAAwDAQACEQMRAD8Av48eYVd0Vd2Ba1RphuOuxQHFUyEvnUgktmP+aIn0vIC4IWPWXSi0cBT+dxiFRjUg5xtkwbKRyYEVG2rdENgEumu7dRu/DrvK7+hOs/YbYPQVx2helhf0dokEpOrcncosOV7hxU3sbI8TodlxUyUQEMN3LrYcJSd6jx+HC2jRBLVTbGYzZnR1W8uIle6aDejchJi0mXXzHnxlXoObVkZmESoTByrsCINhiVrgqrZ2X/iOYLRme6pn87PrQccajM9ohStQ9ycb1IEJqOUgAWAmpagbMANJahv38eSO/Kc9k9vcyRrk/irn2yZTU8z68myq9s2BX5JeO2GApUNIY5GtgcQlzTfUhE304PnnWjo0H/llPxcUICP1KQ0wRbPcQfe1LYt6C+lvWH0j67K/ifOsbk2LLg3RlMVIFZV/W/TFc1G5rDcQPuaMxpMeDn0xahLJnbQHYsoFlZuFMGUZC3PbEhzHdCiZhMyiYxKK5+l7I16sm/e0WH/yGejVh9lqs9cL5irjCmbdihGGO+mmSydAoAtoXMEtSsJrUs1qK+vz7KZCvEVvwXIzZYAy3njZ9rPzdbREAlBAkIs2n6pvp/8Akug8eaUfZnzb2d7Oqtq3n3lTpDbjPmC268Xsi+elAbNeWWVNwsgGsvodBwmFxycwM59XlDEgRO3AaezGKCyMe+vRRi5lwsvNourCrUlOdvfHg95vPfLkCujewgV2WdQgwkyiah2MQLpOjLbsQjWoxvP65fviY0VNguzyX6BkHD+Xb15L08FozJizVfLsiwgIrO2rhuDJmsaWkbHy5MO5CMaICHWKNpekQTPDVpZiKSfk20mAmkik0kRMjazRhDxj7KV66IUs8Oq/UX0VQIHqA47rmvE1dLZNgJJqCv5gOmDlutqO1con2rHjx48z8189dl/9pSj/AF03/wBY6869r4R/+Q/sP/40pr/3FgeXb9NPOpk4KsL5VragLCl58fIhBJboK8rFpeNxXDwOq3CG2Olc8736UnOUyi2NysbX2r3G7BDOGkp0cOf4tZZvRr6SPal6eLwuKwTK/Ad7wq+YtFojLwYro/oWv5BF8x2WbGm8rAvHnFMvZyBZiMKSJrrEXv11Aw7djt1pYHTZqbrxg/x9o0WzfxVL64xliOWOAyPHz/YBjDWZ/wBinZoJUszaKqUsaYuev52ug2f6euAVF/VmPYMvSsEf/a9Ekn0CHLaI/wA51GA96L1mzJKi/mG0lXg2APyzo0fjvtk9WlqmlVUq9KRuAx8c4/HqbItTdf8AR2ZBMt8O0ElF3i34abxNXZjphRRvonooglnd5vjb6F85HGdgfrX11xnO2S/VmMYxjOc5zlnz3jGMYx++c5z+2MY/fOfKgPb36oa69rtBxWvTM4IVDcdPTRCy6CuYYI0kf0iYJaIokB8gjKj8T9jiElaNmiZYe2LiSDQmMBG2T/fIlUWTw9v71M9Cez3pzje4fZk351gdO8Tj3Z4PRHNlpWdeDfom1ZK/jxOTF7FlVo0ZQGlaVsi8gMKwlAAsds03JhLiShCk9EaKs3+ySaNTI4uKe8rCzPfd72Bt21azScXeVh/9C44mvc0q+NnLslW9mC3ui5NaqsMFVeI9ZXXd9goOGpb9IWwBIhmKGrq5yj2CARL2jjRZpTVU05cmvWIaWsG/JgaYnNuPIo7FHcK8zC7SkI0FrUnKVOs7ClEjJIig4NODVLHkpSZOFyrhNsNFidBj50QIkXSSDRo2VculUk099tdFvG3Ku/sL9lkv983RAfMN5yraGYrr1swudMHkYMG6ph7U+ht2BPhp39FqBisv+wzmaU21PIIvXcflw2fumolnHYSZkGQvvi4X9ovsXqCPcscS2rzFSPPkgQSK36ZtKxrdi1lWY+YEN1AdbMR9f0fYAgVV4/DVhITzjMl/MTM1swCuhgWPR17pNtC8f+PP8jqTGo9Fb79uQ+waFMmgge4a8e96d5TEdMapckWjadRNaGyun28UkqBuK7ExeI9IVm4ErhxqwKLIMVllNOzV0uv7I/skrTNYY0XoyTtTa6uYbWIyDR2jCHxKe4Vmio514kP5kzabtRGdayWsjjVSHletoY8XK+IWcjTXElA4PoJ5gglXxRdswD48wuBt6liRS5AKZpr/AJb6YD3wMH7AqwZFBb1oSGEmjZ+OIsHKLxg/YPEdHDR6ydt91G7po6bqJrtnKCiiK6KmiqW+2m2u2XnNo0asGjViybpNGTJui0aNW+mqSDZq2T1Rbt0UtMY0TSRS00TT01xjXTTXXXXGMYx48XJw7L9XPq526+yYknD7PDnNYis34/OU1iI+/fkfPGBdnWPu4d3Cnb1TaR9nGOzrm8RaacvvCbRFuPz7H3755Hjx48p5fx48ePDw8ePHjw8PHjx48PDz/9k="/>

and point to it in the script: requests.get("http://yourserver/somedir/images.html").

When you run the script you will get the following 3 images: , , , respectively named newim1.png, newim2.png and newim3.jpg.

As a reminder, do note that this script (in its current form) will only handle embedded images. If you want to process also ordinary linked images, then you have to modify it accordingly (but this is not difficult).

I seem to be getting rerouted to a login with the requests method...this is a bit mystifying because I should already be logged in on the webdriver object...the site i'm working on is rbauction.com — SciPyInTheHole, Oct 12 '15 at 18:05
My answer covers the general way to handle _data uris_ with `requests`. The problem you mention is related to the fact that you have not made the session info -which is known to the `webdriver` since you are logged in- available for the `requests`. [Here](http://stackoverflow.com/a/11435762/4711309) is a possible way to do this. — sokin, Oct 12 '15 at 18:24

score 0 · Accepted Answer · edited May 23 '17 at 12:06

0

This is an image encoded in base64. Quoting the URL below: "base64 equals to text (string) representation of the image itself".

Read this for a detailed explanation: http://www.stoimen.com/blog/2009/04/23/when-you-should-use-base64-for-images/

In order to use them you'll have to implement a base64 decoder. Luckily SO already provides you with the answer on how to do it:

Python base64 data decode

edited May 23 '17 at 12:06

Community

1
1

answered Oct 10 '15 at 01:42

Ricardo Cid

199
1
5

This is the right answer...the correct function is base64.b64decode(datauri) just write the decoded string to an image file and voile...you have your image. – SciPyInTheHole Oct 12 '15 at 18:14

Saving Image from URL using Python Requests - URL type error

2 Answers2