-3

I need to use 'requests' and a function from urllib3. In the code
code importing requests and printing dir(urllib3)
you can see the requests library is being imported, but at the same time it is a module/package inside urllib3, which has already been imported too.

Doing some research I've found that Python comes with the urllib package, that comes with the request module. On the other hand, requests is a module inside urllib3, but it is a library on its own.

urllib and urllib2 are standard Python librares, but urllib3 is a completely separated library with a confusing name. A portion of it has been included in the standard library and requests depends on it, but it is not a newer version of urllib/urllib2; the library that actually wants to improve is httplib (ref: Github).

"Under the hood, requests uses urllib3 to do most of the http heavy lifting. When used properly, it should be mostly the same unless you need more advanced configuration"
(ref: Stackexchange):

I got to these conclusions but I'm still confused: if I have already imported urllib, do I still need to import requests? What if I had imported urllib3?

Also, should requests be imported separately, as in the depicted code, or should it be import imported from one of the mentioned libraries?

Cœur
  • 37,241
  • 25
  • 195
  • 267
Martin
  • 414
  • 7
  • 21
  • It depends on which library you are actually using. – Daniel Nov 04 '18 at 14:11
  • 2
    `urllib3` is an independent project; no portions of it have been included in the standard library. `requests` is an independent project that builds on top of `urllib3`. If you are using `requests`, you need to import `requests`. How it implements its functionality should not really be of significance to you, the user of the `requests` API. – Martijn Pieters Nov 04 '18 at 14:14
  • Well, I'm actually referring to either urllib or urllib3, and the pic shows also other libraries... I don't understand the comment @Daniel, am I missing something? – Martin Nov 04 '18 at 14:14
  • You need to clarify what you are asking. Do you need to use `requests`? If not, don't worry about it. if yes, then you should import it explicitly. The fact that it is contained in another library doesn't help you directly, except to imply that many of its functions might exist in the outer library. – Josh Friedlander Nov 04 '18 at 14:17
  • @MartijnPieters, so it is not possible to use `request` from the imported `urllib3` (as it is a module included in it)? Is it still necessary to import it even though it is supposed to have been already installed with `urllib3` (and maybe with `urllib`)? – Martin Nov 04 '18 at 14:19
  • 1
    @Martec: It is possible to use it, but *you don't need to or need to worry about it*. It's a different, more advanced and so more complicated API to handle HTTP requests. The `requests` project *wraps* `urllib3` to give you a simpler API. – Martijn Pieters Nov 04 '18 at 14:29
  • 1
    @Martec you have it backwards. `requests` uses `urllib3`: not the other way around. So you can't use `requests` from `urllib3`: and you'd not want to do that even if it *did* use `requests`. Python is not like C/C++ where there is some translation unit and includes are more or less global, it's just not how the import statement works in Python. If you want to use *anything* in `requests`, then import `requests`. Otherwise don't. You can import it many times in multiple modules, or once in a single module, or not at all: it doesn't matter. – Matt Messersmith Nov 04 '18 at 14:29
  • @JoshFriedlander Yes, I need to use `requests` (edited the question in case it wasn't clear). I had understood (but my confusion might lie here) that when you imported a library, you imported all the modules in contained, but from your question I understand that it is not the case, that even though `requests` is listed after `dir(urllib3)`, this doesn't mean that `requests` is included?? – Martin Nov 04 '18 at 14:30
  • 2
    @Martec `requests` is *not* listed from `dir(urllib3)`. Some name called `request` is, though (no s on the end). Read Martjin's answer, it's enlightening in this regard – Matt Messersmith Nov 04 '18 at 14:39
  • @MattMessersmith yes, it was enlightening, as it was yours. Don't know why you don't post them as answers, but at least I could voted them up. Maybe I should read more on what the `dir` function does as I understood it listed the functions/methods/modules from an Agora video on request, I recently saw... – Martin Nov 04 '18 at 14:46
  • 1
    @Martec yep, you've got it right, that's what the `dir` function does. I think you're confusion might be stemming around how importing works, not `dir`. – Matt Messersmith Nov 04 '18 at 14:49
  • Thank (also to @JoshFriedlander ): I'll search a little bit more about how import works... Regarding the downvotes are they because I didn't know enough about the topic to make a question? I'd like to know it to not making more questions about related issues until I'm learned on requests and libraries... (no problem at all, just to know) – Martin Nov 04 '18 at 16:07

1 Answers1

3

If you are using the requests module, then that's the only thing you need to import. The rest is taken care of for you by Python. That urllib3 is used by requests is an implementation detail, unless you need to access specific objects defined by the urllib3 library you don't need to import that into your codebase.

requests is not a module inside urllib3. The urllib3 module defines a urllib3.request (singular!) module, which is something completely different from the requests package:

>>> import urllib3
>>> import requests
>>> urllib3.request
<module 'urllib3.request' from '/Users/mj/Development/venvs/stackoverflow-latest/lib/python3.7/site-packages/urllib3/request.py'>
>>> requests
<module 'requests' from '/Users/mj/Development/venvs/stackoverflow-latest/lib/python3.7/site-packages/requests/__init__.py'>
>>> requests is urllib3.request
False

Note that urllib3.request is a module inside the urllib3 package namespace. import request (no s) would not import it, as that requests a top-level name. Neither would import somethingelse.request, as that would be a different namespace from urllib3. And just because the names requests and request are similar does not mean that they are the same thing. The HTTP standard deals with the abstract concept of a request sent to a server and the server giving back a response, so names in different projects dealing with HTTP will often reflect the concepts that they represent in code.

The same applies to the names urllib and urllib2. Python, once upon a time, started with a urllib package that was limited, and later on a urllib2 package was added to that extended the first and added more functionality. Python 3 consolidated the two packages and reorganised the functionality into a series of new modules under the urllib namespace. The urllib3 package is an independent project to build a better library for the same functionality. No parts of that project are incorporated into the Python standard library.

requests is an project that gives you a clean, straightforward, easy to use API to make HTTP requests and handle the resulting responses. Use it for that API, don't worry about the implementation details.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Awesome, thanks. I actually didn't realize it was singular in `urllib3`. Also, more than worrying about implementation it was that I also had to import the urlllib3 for a `urllib3.connection_from_url(url)` instruction that now I'm finding it is probably unneeded, and thought my code would duplicate instructions or sth... – Martin Nov 04 '18 at 16:04