1

I am working on an application, using Python and Django for the backend and React for the frontend. I ran into a time complexity issue with a backend post route. I am allowing users to pass in credentials and making API requests to pull a massive amount of data. My post route creates the model in the backend, but it takes about 45 seconds. My app is deployed on Heroku, and they have a policy that times out after a process takes longer than 30 seconds.

My solution was to break my backend post route into shorter faster process. I got it to work on the back end, by making one POST request, that sets up the database architecture, and then running PUT requests for each year that I need dats from.

I am however running into an issue trying to set this up in the front end.

I am using a promise chain with .thens

The POST route works, but at the end of my handleSubmit function, only the last year's data is saved to the database. The console.logs also show this happening as they log each year independently.

My assumption is that the current years data isn't being set to my global data variable properly, so every-time the year loop happens, the data is being set back to its original state.

My Code:

function handleSubmit(e){
    setDisplaySpinner("true")
    e.preventDefault();
    let data = {}
    //console.log(`newLeagueHost: ${newLeague.host}`)'
    //sets up the database architecture
    axios.post(`${base_url}/newwel/`, {
        user: user.user_id,
        host: newHost,
        year_started: startYear,
        League_Id: newLeagueId,
        S2: new_S2,
        Swid: new_Swid
      })
      .then((res) =>{
          //get the just created data.
          axios.get(`${base_url}/newwel/${newLeagueId}/`)
          .then(res1 => {
              console.log(res1.data.bigdata)
              //save the response to my global data variable.
              data = res.data;
              //loop through each year.
              for(let i = startYear; i < 2022; i++){
                console.log(data.bigdata)
                // send a put requerst to back end with data, and new year.
                axios.put(`${base_url}/newwel/${newLeagueId}/`, {
                  user: data.user,
                  host: data.host,
                  year_started: data.year_started,
                  year: i,
                  League_Id: data.League_Id,
                  S2: data.S2,
                  Swid: data.Swid,
                  owners: data.owners,
                  standings: data.standings,
                  bigdata: data.bigdata
                })
                .then(res2 => {
                  console.log(res2.data.bigdata)
                  console.log(`ran put for year: ${i}`)
                  //then after the put route runs, get the new data.
                  axios.get(`${base_url}/newwel/${newLeagueId}/`)
                  .then(res3 => {
                    // console.log(res3.data.bigdata)
                    // set response to global data variable, before loop runs again with new year. 
                    data = res.data;
                    console.log(data.bigdata)
                  })
                  .catch(err => {})
                })
                .catch(err => {})
            }})
          .catch(err => {})
      })
      .catch((err) =>{})
  }

res1.bigdata output:

champ: 0
count: 0
finals: 0
legacypoints: 0
madePlayoffs: 0
name: "Person Name"
points_against: 0
points_for: 0
yearsPlayed: 0

res2.bigdata output, there is one for each year:

2014: (16) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
champ: 0
count: 0
finals: 0
legacypoints: 0
madePlayoffs: 0
name: "Person"
points_against: 0
points_for: 0
yearsPlayed: 0
[[Prototype]]: Object

res3.bigdata output is interesting. There are 12 responses, for the 12 years of the loop, but only 5 unique responses. And even though 2014 was logged in res2, 2014 doesnt show up in res3, which i think is a timing issue?:

2017: (16) [0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 3, 3, 4, 4, 5]
champ: 0
count: 5
finals: 0
legacypoints: 0
madePlayoffs: 0
name: "Person"
points_against: 1609.24
points_for: 1235.86
yearsPlayed: 1

My backend Put requests are pretty long and complicated, as there is a lot of data manipulation that happens, but they are working. The data is saved properly, but only for the last year in the loop.

The output ends up looking like this:

this is just one item in my bigdata dictionary

'Person's Name': {'name': 'Person's Name', 'champ': 0, 'count': 0, 'finals': 0, 'points_for': 0, 'yearsPlayed': 0, 'legacypoints': 0.0, 'madePlayoffs': 0, 'points_against': 0, 2021: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]},

The Output Should look like, if the startyear is 2017:

'Person's Name': {'name': 'Person's Name', 'champ': 0, 'count': 0, 'finals': 0, 'points_for': 0, 'yearsPlayed': 0, 'legacypoints': 0.0, 'madePlayoffs': 0, 'points_against': 0, 2017: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 2018: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 2019: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 2020: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 2021: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]},

So it isn't saving each year it loops through, just the last year.

Am I missing something obvious? Thanks y'all!

user3840170
  • 26,597
  • 4
  • 30
  • 62
JF12345
  • 11
  • 3
  • Each time `data = res.data;` is executed, the previous reference that `data` had, is overwritten. There is nothing *collected* here, only *overwritten*. – trincot Aug 14 '22 at 15:20
  • But without knowing what actually the response is for each request, it is difficult to say. If the server makes sure to take the bigdata from the client and *extend* it, it should work fine. So probably it doesn't, and the client should do it. But this question needs more information about the exact response. It would be good to have a concrete example of all the request and response data -- literally. – trincot Aug 14 '22 at 15:33
  • I'd definitely refactor this to async/await, this would make reasoning about the correctness much easier. – Wiktor Zychla Aug 14 '22 at 15:58
  • @trincot thanks for the reply! I just added the responses for each request. When you say extend in terms of the back end what do you mean. Atm my put route, takes the data, and hits an api call with the new year. Gets more data and uses .update to update a variable that is storing the data. It then passes it through a serializer to store it in the database. When I manually go year by year in the Django REST Framework, it works properly. – JF12345 Aug 14 '22 at 15:59
  • So you get different values for like `points_against`. Where in your desired result do you expect to get multiple values for that? (just an example). – trincot Aug 14 '22 at 15:59
  • @Wiktor, cool I was wondering if I may need to do that. .then's just make much more sense to me lol. But I will give that a go! – JF12345 Aug 14 '22 at 16:00
  • @trincot points_against will increment over the years. So if there were 100 points aginast in 2010, and then 150 in 2011, the final output after those two years would be 250. that logic happens in back end during the put request, which I am happy to post its just ALOT lol – JF12345 Aug 14 '22 at 16:01
  • Yea it deff is not going in order, I figured I would deal with that issue, once I got the data at least updating and saving properly. Kinda a fun problem yea? – JF12345 Aug 14 '22 at 16:04
  • Well, it is a very common problem. You need to make sure that you flatten your code structure, and make your API slim. If you only are interested in the sum of certain attributes, then create an API endpoint for that, and avoid sending it back with each request, ...Etc. – trincot Aug 14 '22 at 16:05
  • The issue with this question is that the missing years are not your only problem. There is much more going wrong (like not summing up some values). I only realised this after I posted an answer, and then had to delete it. I hope someone else will have time to address all points for you, but I am going to pass. – trincot Aug 14 '22 at 16:14

0 Answers0