Shiny and Leaflet integration is really slow - how can I speed it up?

Question

Right now i'm almost certain that my current use of shiny and leaflet is sub-optimal.

At a high level my current approach looks like this:

1. Generate a leaflet.
2. Create a reactive dataframe on user input.
3. Create a reactive dataframe of lat lon coordinates on user selection of their area of interest.
4. Merge a spatial dataframe (containing postcode polygon boundaries) with the reactive dataframe from step 2, then draw the map with the joined dataframe. This keeps all the data necessary for drawing polygons, adding colorBins and fillColor and labels inside the same final dataframe.

In more detail, the steps are executed as follows:

1. Generate a map like this:

   output$leaflet_map <- renderLeaflet({
   leaflet() %>%
       addTiles()
       })
   

2. Produce a reactive dataframe of marketing data to be joined onto an sf spatial dataframe containing postcode polygons via sp::merge() (the join happens a little later, i'll get to that):

   reactive_map_data1 <- reactive({
   df %>%
       filter(BrandAccount_Brand %in% input$selectBrandRecruitment1) %>%
       group_by(POA_CODE, ordertype) %>%
       summarise("Number of Orders type and postcode" = n(), "AOV" = round(mean(TotalDiscount), 2)) %>%
       left_join(seifa, by = "POA_CODE") %>%
       left_join(over25bypostcode, by = "POA_CODE") %>%
       mutate(`Proportion of Population Over 25` = round(n() / `25_and_over` * 100, 4))
       })
   

3. Create a reactive dataframe containing the lat and lon coordinates of the State selected by the user to be fed into the call to render the map:

   reactive_state_recruitment1 <- reactive({
   australian_states %>%
       filter(States == input$selectState_recruitment1)
       })
   

4. Render the final map - profvis determines that this is infact the slow part:

   observeEvent(
   input$gobutton_recruitment1, {
   
   ## First I load the spatial data with each call to render the 
   ## map - this is almost certainly sub-optimal however I can't 
   ## think of another way to do this as each time the data are 
   ## joined I have no other way of re-setting the gdal.postcodes2 
   ## spatial dataframe to its original state which is why I reload 
   ## it from .rds each time:
   
       gdal.postcodes_recruitment1 <- readRDS("gdal.postcodes2.rds")
   
   ## I then merge the marketing `reactive_map_data1()` dataframe 
   ## created in Step 2 with the freshly loaded `gdal.postcodes2` 
   ## spatial dataframe - `profvis` says this is pretty slow but 
   ## not as slow as the rendering of the map  
   
       gdal.postcodes_recruitment1@data <- sp::merge(gdal.postcodes_recruitment1@data, reactive_map_data1(), by.x = "POA_CODE", all.x = TRUE)
   
   ## Next I generate the domain of `colorBin` with the `Number of 
   ## Orders type and postcode` variable that only exists after the 
   ## merge and is subject to change from user input - it resides 
   ## within the `reactive_map_data1()` dataframe that gets merged 
   ## onto the `gdal.postcodes2()` spatial dataframe.               
   
   pal <- colorBin("YlOrRd", domain = 
   gdal.postcodes_recruitment1$`Number of Orders type and 
   postcode`, bins = bins_counts)
   
   ## Lastly I update the leaflet with `leafletProxy()` to draw the 
   ## map with polygons and fill colour based on the 
   ## `reactive_map_data1()` values            
   
   leafletProxy("leaflet_map_recruitment1", data = gdal.postcodes_recruitment1) %>%
           addPolygons(data = gdal.postcodes_recruitment1, 
                       fillColor = ~pal(gdal.postcodes_recruitment1$`Number of Orders type and postcode`), 
                       weight = 1,
                       opacity = 1,
                       color = "white",
                       dashArray = "2",
                       fillOpacity = .32,
                       highlight = highlightOptions(
                           weight = 3.5,
                           color = "white",
                           dashArray = "4",
                           fillOpacity = 0.35,
                           bringToFront = TRUE),
                       layerId = gdal.postcodes_recruitment1@data$POA_CODE,
                       label = sprintf(
                           "<strong>%s<br/>%s</strong><br/>%s<br/>%s<br/>%s<br/>%s",
                           paste("Postcode: ", gdal.postcodes_recruitment1$POA_CODE, sep = ""),
                           paste("% of Population Over 25: ", gdal.postcodes_recruitment1$`Proportion of Population Over 25`, "%"),
                           paste("Number of Orders: ", gdal.postcodes_recruitment1$`Number of Orders type and postcode`, sep = ""),
                           paste("Ave Order Value: $", gdal.postcodes_recruitment1$`AOV`, sep = ""),
                           paste("Advantage & Disadvantage: ", gdal.postcodes_recruitment1$`Relative Socio-Economic Advantage and Disadvantage Decile`, sep = ""),
                           paste("Education and Occupation: ", gdal.postcodes_recruitment1$`Education and Occupation Decile`, sep = "")
                       ) %>% 
                           lapply(htmltools::HTML),
                       labelOptions = labelOptions(
                           style = list("font-weight" = "normal", padding = "3px 8px"),
                           textsize = "15px",
                           direction = "auto")) %>%
           addLegend("bottomright", pal = pal, values = ~bins_counts,
                     title = "# of Recruits (All Time)",
                     labFormat = labelFormat(suffix = ""),
                     opacity = 1
           ) %>%
           setView(lng = reactive_state_recruitment1()$Lon, lat = reactive_state_recruitment1()$Lat, zoom = reactive_state_recruitment1()$States_Zoom)
   })
   

All up the map takes between 7 and 20 seconds to render as the data are quite large.

Some things to note:

• The polygons have already been simplified to death, they are currently only displaying at 10% of the detail that was originally provided to define postcode boundaries by the Australian Bureau of Statistics. Simplifying the polygons further is not an option.

• sp::merge() is not the fastest of join functions I have come across, but it is necessary in order to merge a spatial dataframe with a non-spatial dataframe (other joins such as those offered by dplyr will not accomplish this task - a look at the sp::merge() documentation reveals that this has something to do with S3 and S4 datatypes, in any case this part is not the slow part according to profvis).

• According to profvis the actual rendering of the map in step 4 (drawing polygons) is the slow part. Ideally a solution to speed this whole process up would involve drawing the polygons on the original leaflet, and only updating the fillColor and labels applied to each polygon upon input of the 'Go' actionButton. I have not figured out a way to do this.

Can anyone think of a way to re-structure this whole procedure to optimise efficiency?

Any input is greatly appreciated.