0

I'm trying to do a lot of API Request to load Data. The Requests itself are not very heavy on Data:

struct myFollower {
    public var followed_at: String = ""
    public var from_id: String = ""
    public var from_login: String = ""
    public var from_name: String = ""
    public var to_id: String = ""
    public var to_login: String = ""
    public var to_name: String = ""
    public var pagination: String = ""
}

The TwitchAPI gives 100 entitites per Request. So as soon as you fetch more than 100, you have to use the pagination System.

func getAllFollowers(token: String, from_id: String, limitRequests:Int = GlobalConstants.LOADING_LIMIT, completion: @escaping ([myFollower]?, Int?, Error?) -> Void) {
        var allFollowers = [myFollower]()
        var cursor: String? = nil
        var totalCount: Int? = nil

        let group = DispatchGroup()

        group.enter()

        func fetchNextBatch() {
            group.enter()
            let endpoint = "https://api.twitch.tv/helix/users/follows?to_id=\(from_id)&first=100\(cursor.flatMap { "&after=\($0)" } ?? "")"
            var request = URLRequest(url: URL(string: endpoint)!)
            request.addValue("Bearer \(token)", forHTTPHeaderField: "Authorization")
            request.addValue(GlobalConstants.CLIENT_ID, forHTTPHeaderField: "Client-ID")

            let task = URLSession.shared.dataTask(with: request) { (data, response, error) in
                defer { group.leave() }

                // ... code for request error handling

                do {
                    let followerData = try JSONDecoder().decode(myFollowers.self, from: data)
                    let nonNilFollowers = followerData.data.compactMap { $0 }
                    allFollowers.append(contentsOf: nonNilFollowers)

                    if cursor == nil {
                        totalCount = followerData.total
                    }

                    cursor = followerData.pagination?.cursor
                    if cursor != nil {
                        fetchNextBatch()
                    } else {
                        group.leave()
                    }
                } catch {
                    completion(nil, nil, error)
                }
            }
            task.resume()
        }

        fetchNextBatch()

        group.notify(queue: DispatchQueue.main) {
            completion(allFollowers, totalCount, nil)
        }
    }

This code works very well for smaller amounts of Requests. As soon as the "from_id" user has a lot of Followers or I'm using it on more than one id, the code starts to litter the memory like there is no tomorrow. The memory reaches easily 5 GB and the app crashes.

I guess the problem lays within the way my app handles the many URLRequests. It is kind of random how the memory clears itself again. Sometimes it is no problem as the memory clears itself before reaching 5 GB, sometimes it doesnt to that and it reaches 5 GB and crashes. I tried solving it with the "DispatchQueue.main" which had no effect at all. I'm somehow not able to get this way of executing the code:

Loop through 10 users
  User 1
     Load first 100 Events
     Wait until Loaded
     Load next 100 Events
     ...
  User 2
     Load first 100 Events
     Wait until Loaded
     Load next 100 Events
     ...

Instead it seems like it has this way:

Loop through 10 user
   User 1
   User 2
   ...
      Load first 100 Events of User 1
      Load next 100 Events of User 1
      ...
      Load first 100 Events of User 2
      ...
      crash.

Thank you for your help in advance.

EDIT 1: As lorem ipsum proposed I tried it with async await:

func getAllFollowers(token: String, from_id: String, limitRequests: Int = GlobalConstants.LOADING_LIMIT) async throws -> ([myFollower], Int) {
    var allFollowers = [myFollower]()
    var cursor: String? = ""
    var totalCount: Int? = nil
    while allFollowers.count < limitRequests, let cursorValue = cursor {
        let endpoint = "https://api.twitch.tv/helix/users/follows?to_id=\(from_id)&first=100&after=\(cursorValue)"
        var request = URLRequest(url: URL(string: endpoint)!)
        request.addValue("Bearer \(token)", forHTTPHeaderField: "Authorization")
        request.addValue(GlobalConstants.CLIENT_ID, forHTTPHeaderField: "Client-ID")
        
        do {
            let (data, response) = try await URLSession.shared.data(for: request)
            
            guard let httpResponse = response as? HTTPURLResponse else {
                throw NSError(domain: "Error", code: 0, userInfo: [NSLocalizedDescriptionKey: "Invalid response"])
            }
            
            if httpResponse.statusCode != 200 {
                throw NSError(domain: "Error", code: httpResponse.statusCode, userInfo: [NSLocalizedDescriptionKey: "Invalid response"])
            }
            
            let followerData = try JSONDecoder().decode(myFollowers.self, from: data)
            let nonNilFollowers = followerData.data.compactMap { $0 }
            allFollowers.append(contentsOf: nonNilFollowers)
            
            if cursor == nil {
                totalCount = followerData.total
            }
            
            cursor = followerData.pagination?.cursor
        } catch {
            throw error
        }
    }
    
    return (allFollowers, totalCount ?? 0)
}

This code works pretty much the same as the old one (while being way cleaner - thank you for that :) ). Nevertheless, the main problem remains. The memory gets loaded and it crashes. The Memory starts filling up until it crashes at 5 GB

Zeddi
  • 13
  • 3
  • Try switching to async await – lorem ipsum Feb 25 '23 at 11:25
  • @lorem ipsum I tried your change. Thank you for your suggestion, but it still has the same problem. I attached the new code and a screenshot of the memory. – Zeddi Feb 27 '23 at 16:57
  • Hard to know I would need working code, what happens if the user doesn’t have “limit” of followers? Are you calling this from a “safe” async await context? If you have “Task{}” your likely operating in an unsafe context. – lorem ipsum Feb 27 '23 at 17:35
  • https://stackoverflow.com/questions/75101759/data-race-occurring-in-swift-actor/75102705#75102705 – lorem ipsum Feb 27 '23 at 17:44
  • Notice in the link above how the OP has Task inside the loop, I would put it outside or else there is no “awaiting” you just let loose a ton of tasks with no control. – lorem ipsum Feb 27 '23 at 17:46

0 Answers0