I'm trying to do a lot of API Request to load Data. The Requests itself are not very heavy on Data:
struct myFollower {
public var followed_at: String = ""
public var from_id: String = ""
public var from_login: String = ""
public var from_name: String = ""
public var to_id: String = ""
public var to_login: String = ""
public var to_name: String = ""
public var pagination: String = ""
}
The TwitchAPI gives 100 entitites per Request. So as soon as you fetch more than 100, you have to use the pagination System.
func getAllFollowers(token: String, from_id: String, limitRequests:Int = GlobalConstants.LOADING_LIMIT, completion: @escaping ([myFollower]?, Int?, Error?) -> Void) {
var allFollowers = [myFollower]()
var cursor: String? = nil
var totalCount: Int? = nil
let group = DispatchGroup()
group.enter()
func fetchNextBatch() {
group.enter()
let endpoint = "https://api.twitch.tv/helix/users/follows?to_id=\(from_id)&first=100\(cursor.flatMap { "&after=\($0)" } ?? "")"
var request = URLRequest(url: URL(string: endpoint)!)
request.addValue("Bearer \(token)", forHTTPHeaderField: "Authorization")
request.addValue(GlobalConstants.CLIENT_ID, forHTTPHeaderField: "Client-ID")
let task = URLSession.shared.dataTask(with: request) { (data, response, error) in
defer { group.leave() }
// ... code for request error handling
do {
let followerData = try JSONDecoder().decode(myFollowers.self, from: data)
let nonNilFollowers = followerData.data.compactMap { $0 }
allFollowers.append(contentsOf: nonNilFollowers)
if cursor == nil {
totalCount = followerData.total
}
cursor = followerData.pagination?.cursor
if cursor != nil {
fetchNextBatch()
} else {
group.leave()
}
} catch {
completion(nil, nil, error)
}
}
task.resume()
}
fetchNextBatch()
group.notify(queue: DispatchQueue.main) {
completion(allFollowers, totalCount, nil)
}
}
This code works very well for smaller amounts of Requests. As soon as the "from_id" user has a lot of Followers or I'm using it on more than one id, the code starts to litter the memory like there is no tomorrow. The memory reaches easily 5 GB and the app crashes.
I guess the problem lays within the way my app handles the many URLRequests. It is kind of random how the memory clears itself again. Sometimes it is no problem as the memory clears itself before reaching 5 GB, sometimes it doesnt to that and it reaches 5 GB and crashes. I tried solving it with the "DispatchQueue.main" which had no effect at all. I'm somehow not able to get this way of executing the code:
Loop through 10 users
User 1
Load first 100 Events
Wait until Loaded
Load next 100 Events
...
User 2
Load first 100 Events
Wait until Loaded
Load next 100 Events
...
Instead it seems like it has this way:
Loop through 10 user
User 1
User 2
...
Load first 100 Events of User 1
Load next 100 Events of User 1
...
Load first 100 Events of User 2
...
crash.
Thank you for your help in advance.
EDIT 1: As lorem ipsum proposed I tried it with async await:
func getAllFollowers(token: String, from_id: String, limitRequests: Int = GlobalConstants.LOADING_LIMIT) async throws -> ([myFollower], Int) {
var allFollowers = [myFollower]()
var cursor: String? = ""
var totalCount: Int? = nil
while allFollowers.count < limitRequests, let cursorValue = cursor {
let endpoint = "https://api.twitch.tv/helix/users/follows?to_id=\(from_id)&first=100&after=\(cursorValue)"
var request = URLRequest(url: URL(string: endpoint)!)
request.addValue("Bearer \(token)", forHTTPHeaderField: "Authorization")
request.addValue(GlobalConstants.CLIENT_ID, forHTTPHeaderField: "Client-ID")
do {
let (data, response) = try await URLSession.shared.data(for: request)
guard let httpResponse = response as? HTTPURLResponse else {
throw NSError(domain: "Error", code: 0, userInfo: [NSLocalizedDescriptionKey: "Invalid response"])
}
if httpResponse.statusCode != 200 {
throw NSError(domain: "Error", code: httpResponse.statusCode, userInfo: [NSLocalizedDescriptionKey: "Invalid response"])
}
let followerData = try JSONDecoder().decode(myFollowers.self, from: data)
let nonNilFollowers = followerData.data.compactMap { $0 }
allFollowers.append(contentsOf: nonNilFollowers)
if cursor == nil {
totalCount = followerData.total
}
cursor = followerData.pagination?.cursor
} catch {
throw error
}
}
return (allFollowers, totalCount ?? 0)
}
This code works pretty much the same as the old one (while being way cleaner - thank you for that :) ). Nevertheless, the main problem remains. The memory gets loaded and it crashes. The Memory starts filling up until it crashes at 5 GB