5

I have a function that does some basic web harvesting. This function is called after a successful login. (website has been masked xxxxxx)

Search Function:

search <-function(HorseList){
  url <- "http://tnetwork.xxxxxx.com/tnet/HorseSearch.aspx"
  s <- GET(url)
  xxxxxx <- tibble(
    horse_name = character(),
    race_date = character(),
    race_nbr = character(),
    trk = character(),
    peak = character(),
    dist_run = character()
  )
  for (row in 1:nrow(HorseList))
  {
    
    url <-paste(c('http://tnetwork.xxxxxx.com/tnet/HorseSearchAPI.aspx?HorseName=',toString(HorseList[[row, 1]])),collapse='')
    #print(url)
    h <- GET(url)
    temp<-content(h, "text")
    doc <- htmlParse(temp)
    horse_name <- HorseList[[row,1]]
    horse_ID <-xpathSApply(doc,"//*[@id=\"resultsDiv\"]/p[1]/a/@href")
    horse_ID <-substr(horse_ID,27,40)
    h_list <- list()
    c <- nchar(horse_ID)
    if (length(c)>0)
    {
      h_list[1] <- horse_ID
    }
    
    
    id_count <- length(h_list)
    
    for (k in 1:id_count)
    {
      url <-paste(c('http://tnetwork.xxxxxx.com/tnet/t_PastPerf.aspx?HorseID=',toString(h_list[k])),collapse='')
      t <- GET(url)
      temp <- content(t, "text")
      pastperf <- htmlParse(temp)
      row_count<-length(xpathSApply(pastperf,"//*[@id=\"pastPerfTable\"]/tr"))
      
      for(j in 2:row_count)
      {
        j<- toString(j)
        race_data <- xpathSApply(pastperf,paste("//*[@id=\"pastPerfTable\"]/tr[",j,"]/td[1][1]"),xmlValue)
        race_date <- substr(race_data,1,10)
        race_number <-trimws(substr(race_data,12,100))
        horse_name <- URLdecode(toString(horse_name))
        race_nbr = str_match(race_number,'(Race\\s\\d+)(.*)')[,2]
        trk = str_match(race_number,'(Race\\s\\d+)(.*)')[,3]
        peak <-xpathSApply(pastperf,paste("//*[@id=\"pastPerfTable\"]/tr[",j,"]/td[13]"),xmlValue)
        cum_distance <-xpathSApply(pastperf,paste("//*[@id=\"pastPerfTable\"]/tr[",j,"]/td[14]"),xmlValue)
        newrow <- paste(horse_name,',',race_date,',',race_nbr,',',trk, ',',peak,',',cum_distance)
        xxxxxx <- add_row(trakus, horse_name = horse_name, race_date = race_date, race_nbr = race_nbr, trk=trk, peak = peak, dist_run = cum_distance)
      }
    }
  }
  
  return(xxxxxx)
}

The function has worked successfully in the past, but today it is throwing the following error:

Error: Internal error in `vec_assign()`: `value` should have been recycled to fit `x`.

I ran the rlang::last_error() and last_trace() commands to gain some additional insight, but I'm still not sure what's going on.

> rlang::last_error()
<error/rlang_error>
Internal error in `vec_assign()`: `value` should have been recycled to fit `x`.
Backtrace:
 1. base::source("~/TimeForm/Scripts/past_perf.R", echo = TRUE)
 6. global::search(horse_list) ~/TimeForm/Scripts/past_perf.R:627:2
 7. tibble::add_row(...) ~/TimeForm/Scripts/past_perf.R:85:8
 8. tibble:::rbind_at(.data, df, pos)
 9. vctrs::vec_rbind(old, new)
Run `rlang::last_trace()` to see the full context.

> rlang::last_trace()
<error/rlang_error>
Internal error in `vec_assign()`: `value` should have been recycled to fit `x`.
Backtrace:
     x
  1. +-base::source("~/TimeForm/Scripts/past_perf.R", echo = TRUE)
  2. +-base::source("~/TimeForm/Scripts/past_perf.R", echo = TRUE)
  3. | +-base::withVisible(eval(ei, envir))
  4. | \-base::eval(ei, envir)
  5. |   \-base::eval(ei, envir)
  6. \-global::search(horse_list) ~/TimeForm/Scripts/past_perf.R:627:2
  7.   \-tibble::add_row(...) ~/TimeForm/Scripts/past_perf.R:85:8
  8.     \-tibble:::rbind_at(.data, df, pos)
  9.       \-vctrs::vec_rbind(old, new)
 10.         \-(function () ...

It appears the add_row() line in my code may be the culprit, but I'm not sure what the error is telling or how to fix it. Does anyone have any insights they could share?

Mutuelinvestor
  • 3,384
  • 10
  • 44
  • 75

1 Answers1

2

I found that the problem occurs in the handling of empty character fields, null values or the use of NA. The problem was corrected with a map_if replacement of null values.

    map_depth(.depth = 1, map_if, is_empty, ~paste0(""))

Of course, you'll need to adjust the depth command to make the correction at the appropriate level for your list construct.

Ideally, bind_rows() would handle vectors containing NA or NULL values more robustly.

GGAnderson
  • 1,993
  • 1
  • 14
  • 25