Python slow for-loop in async func

Question

I am trying to and Song-Objects to my Queue-Object, to get the actual songs I loop over a Playlist-Object from the libary Pafy.

playlist = pafy.get_playlist("playlist url here")
counter = 0
for s in playlist["items"]:
    counter += 1
    s = s["pafy"]
    song = Song.Song(s.watchv_url, msg.author.name, s.title, s.description, s.author, s.published, s.duration,
                     s.likes, s.dislikes,
                     s.viewcount, s.thumb)
    if queue.add_song(song=song):
        print("Song " + str(counter) + " finished")
    else:
        print("Song " + str(counter) + " not added")

class Song:
    def __init__(self, url, requested_by, title, description, uploader, upload_date, duration, likes, dislikes, views, thumbnail):
        self.url = url
        self.requested_by = requested_by
        self.title = title
        self.description = description
        self.uploader = uploader
        self.upload_date = upload_date
        self.duration = duration
        self.likes = likes
        self.dislikes = dislikes
        self.views = views
        self.thumbnail = thumbnail

class PlayerQueue:
    list_of_songs = []
    length = 0
    current_song = []
    replaying = False
    random = False

    def add_song(self, song):
        try:
            self.length += 1
            self.list_of_songs.append(song)
            return True
        except Exception:
            return False

    def remove_song(self, song):
        if song in self.list_of_songs:
            self.length -= 1
            self.list_of_songs.remove(song)
            return True
        else:
            return False

    def replay(self):
        self.replaying = True

    def randomize(self):
        self.random = True

    def clear(self):
        self.list_of_songs = []
        self.length = 0
        self.replaying = False
        self.random = False

Should I make the add_song-Method async too?

it takes approximately 1-2 sec for looping over this code once. This leads me into a problem with asyncio, because it will throw an TimeoutError. This error occurs after 30 seconds and for 70 Songs the loop takes more than one minute. Is this loop so slow, because it's run in a async def function? Can I even make it faster?

Here is the error:

    ERROR:asyncio:Task exception was never retrieved
    future: <Task finished coro=<VoiceClient.poll_voice_ws() done, defined 
    at C:\Users\user\AppData\Local\Programs\Python\Python35-32\lib\site-packages\discord\voice_client.py:269> exception=TimeoutError()>
    Traceback (most recent call last):
      File "C:\Users\user\AppData\Local\Programs\Python\Python35-32\lib\asyncio\tasks.py", line 239, in _step
        result = coro.send(None)
      File "C:\Users\user\AppData\Local\Programs\Python\Python35-32\lib\site-packages\discord\voice_client.py", line 276, in poll_voice_ws
        yield from self.ws.poll_event()
      File "C:\Users\user\AppData\Local\Programs\Python\Python35-32\lib\site-packages\discord\gateway.py", line 676, in poll_event
        msg = yield from asyncio.wait_for(self.recv(), timeout=30.0, loop=self.loop)
      File "C:\Users\user\AppData\Local\Programs\Python\Python35-32\lib\asyncio\tasks.py", line 396, in wait_for
        raise futures.TimeoutError()
    concurrent.futures._base.TimeoutError

Can I make a new task for my for-loop in a existing asyncio.event.loop so I don't run into this TimeoutError? Should I just try to catch it and then continue?

Full code here

Currently running on:

Intel I5 Series CPU

64 GB DDR4 Ram

Python 3.x

How exactly do you create the asyncio task? Where do the `Song` class and the `add_song` queue method come from? — user4815162342, Apr 14 '18 at 08:42
I am creating the task after getting a command input and the calling it normally in a `async def get_playlist_ready()`. [myBot](https://github.com/Stegosawr/Dino-Bot) — stego, Apr 14 '18 at 09:04
Could it be that `get_playlist` itself takes a long time to complete? How large is the `playlist["items"]` list? — user4815162342, Apr 14 '18 at 10:37
In my test case I used a playlist that has 76 songs in it.When I debugged to me it looked like it doesn't take that long. So `playlist["items"]` has 76 objects in it. — stego, Apr 14 '18 at 12:18
*for 70 Songs the loop takes more than one minute* Have you tried to measure where this minute is spent? Looking at the loop, it's hard to imagine it taking a minute to execute. — user4815162342, Apr 14 '18 at 12:36
while debugging the `song = Song.Song(attributes)` takes a long time, very long time — stego, Apr 14 '18 at 12:47
I mean it might be the `debugger`, but everytime I ran over the step of creating the `Song-object`, it took very long rill it continued. It might be that all the object in the `playlist-object` aren't loaded yet. — stego, Apr 14 '18 at 19:32
Ok, it's slow because Pafy objects retrieve their information lazily. You didn't indicate who is raising the timeout error, but judging from the traceback it is some sort of voice client for `discord`. Maybe you should ensure that the information you need is already available once this is activated? (The 30-seconds timeout for a "voice client" appears reasonable.) As written the question requires a design/debugging session, which is outside the scope of StackOverflow. Please edit it to clearly state the problem and include a **minimal** and **reproducible** example that demonstrates it. — user4815162342, Apr 15 '18 at 05:33
I mean if it's lazily there is not much I can do about it.I would get the data myself with the gdata libary, but the installer of that has an error currently so. — stego, Apr 15 '18 at 15:46

score 0 · Accepted Answer · answered Apr 20 '18 at 08:48

0

I am now using another, library to download the playlist. I am using the thread library to run it sepretly so, before joining a vc I start the download in a separate thread, so I don't get a timeout error.

answered Apr 20 '18 at 08:48

stego

193
2
11

Python slow for-loop in async func

1 Answers1