2

Or, Saltstack + docker-py AttributeError: 'RecentlyUsedContainer' object has no attribute 'lock'

I have been digging into this issue to no avail. I'm trying to use SaltStack to manage my docker images/containers but ran into this problem.

Initially I was using the salt state docker.running but that presented as the command does not exist. When I changed the state to docker.running, I got the traceback I posted over at that GitHub issue:

      ID: scheduler
Function: docker.pulled
  Result: False
 Comment: An exception occurred in this state: Traceback (most recent call last):
            File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1563, in call
              **cdata['kwargs'])
            File "/usr/lib/python2.7/dist-packages/salt/states/dockerio.py", line 271, in pulled
              returned = pull(name, tag=tag, insecure_registry=insecure_registry)
            File "/usr/lib/python2.7/dist-packages/salt/modules/dockerio.py", line 1599, in pull
              client = _get_client()
            File "/usr/lib/python2.7/dist-packages/salt/modules/dockerio.py", line 277, in _get_client
              client._version = client.version()['ApiVersion']
            File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 837, in version
              return self._result(self._get(url), json=True)
            File "/usr/local/lib/python2.7/dist-packages/docker/clientbase.py", line 86, in _get
              return self.get(url, **self._set_request_timeout(kwargs))
            File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 310, in get
              #: Stream response content default.
            File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 279, in request
            File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 374, in send
              url=request.url,
            File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 155, in send
              **proxy_kwargs)
            File "/usr/local/lib/python2.7/dist-packages/docker/unixconn/unixconn.py", line 74, in get_connection
              with self.pools.lock:
          AttributeError: 'RecentlyUsedContainer' object has no attribute 'lock'
 Started: 09:33:42.873628
Duration: 22.115 ms

After searching Google a bit more and coming up with nothing, I went ahead and started reading the source.

After reading unixconn.py and realizing that RecentlyUsedContainer was coming from urllib3, I went and tracked down the source for that and discovered that there was a _lock attribute that was changed to lock a while ago. That seemed strange.

I looked closer at the imports and realized that unixconn.py was attempting to use requests' built-in urllib3 and then falling back to the stand alone urllib3. So I checked out the requests urllib3 and found that it did, indeed have the _lock -> lock change. But it was newer than my version of requests. So I upgraded requests and tried again. Still no dice - same AttributeError.

Now things start to get weird.

In order to get information back to my salt master, I started mucking with the docker-py and urllib3 code on my salt minion. At first I raised exceptions with urllib3.__file__ to make sure I was using the right file. But occasionally the file name that it would return was in a file and a folder that did not exist. Usually it was displaying /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/_collections.pyc, but when I would delete that file thinking that maybe the .pyc being cached was causing a problem it would still say that was the __file__, even though it didn't exist.

Then I discovered inspect.getfile. And I got the same bizarre behavior - I could delete the .pyc file and yet inspect.getfile(self.pools) would return the non-existent file.

To make life even better, I've added

raise Exception('Pining for the Fjords')

to

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/_collections.py

At the end of the RecentlyUsedContainer.__init__. Yet that exception does not raise.

And I have just confirmed that something is in fact lying to me, because despite changing unixconn.py

 def get_connection(self, url, proxies=None):                                
     import inspect                                                          
     r = RecentlyUsedContainer(10)                                           
     raise Exception(inspect.getfile(r.__class__) + '\n' + r.__doc__) 

which returns /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/_collections.pyc, when I go edit that .pyc and modify the RecentlyUsedContainer's docstring I get the original docstring.

And finally, when I edit /usr/lib/python2.7/dist-packages/urllib3/_collections.pyc and change it's docstring, (or the same path but _collections.py instead)...

I still get the same docstring!

Why is the wrong code getting executed here, and how can I find out where it is so I can fix the problem?

Wayne Werner
  • 49,299
  • 29
  • 200
  • 290

1 Answers1

1

So I finally figured out the problem:

It did have something to do with salt. For some reason the way the salt minion imported the docker-py library it did some sort of... partial hold on the imports. I suspect that what was happening was that salt was re-importing just the docker-py library specifically so when I would make changes to those files the changes would show up.

However, since the Python import mechanism will search for pre-imported modules first the urllib3 code was never re-imported.

Ultimately all that is required is to restart the salt minion:

salt 'my-minion' cmd.run "nohup /bin/sh -c 'sleep 10 && salt-call --local service.restart salt-minion'"
Wayne Werner
  • 49,299
  • 29
  • 200
  • 290