0

I created a shinyapp that displays output data from a 10,000-trial Monte Carlo simulation. The simulation is of an entire football season with 238 teams, tracking seven variables for each team in each trial (16.66+ million data points). The data from the simulation is stored in an excel file.

When I try to run the app from shinyapps.io, the ui.R loads just fine, including a small portion of the excel sheet (team names in a selectInput() command). The app always crashes before any of the server.R loads (all of my read_excel() commands are at the very beginning of my server.R. This, along with some other research I've done into other example of apps crashing unexpectedly, leads me to believe my excel file is too large for shiny to handle. I have each variable stored on a separate sheet in excel, with a different read_excel() command for each sheet. I was able to get a version of the app, storing only 2 of the 7 different tracking variables, to work fine when accessing the data from separate tabs. When I tried accessing all of the data from a single tab, the app crashes like before.

I have tried the advice from the shiny help pages in regards to changing the app advanced settings, instances, workers, etc. Nothing changed. I have checked my app logs: 

2016-07-19T23:48:17.796234+00:00 shinyapps[115363]: 
2016-07-20T00:04:10.853831+00:00 shinyapps[115363]: Server version: 0.4.4.1986
2016-07-20T00:04:10.853884+00:00 shinyapps[115363]: R version: 3.3.1
2016-07-20T00:04:10.853895+00:00 shinyapps[115363]: shiny version: 0.13.2
2016-07-20T00:04:10.853906+00:00 shinyapps[115363]: rmarkdown version: NA
2016-07-20T00:04:10.853918+00:00 shinyapps[115363]: knitr version: NA
2016-07-20T00:04:10.853948+00:00 shinyapps[115363]: jsonlite version: 1.0
2016-07-20T00:04:10.853992+00:00 shinyapps[115363]: RJSONIO version: NA
2016-07-20T00:04:10.854026+00:00 shinyapps[115363]: htmltools version: 0.3.5
2016-07-20T00:04:11.030246+00:00 shinyapps[115363]: Using jsonlite for JSON processing
2016-07-20T00:04:11.034832+00:00 shinyapps[115363]: 
2016-07-20T00:04:11.034838+00:00 shinyapps[115363]: Starting R with process ID: '50'
2016-07-20T00:04:11.052472+00:00 shinyapps[115363]: 
2016-07-20T00:04:11.052477+00:00 shinyapps[115363]: Listening on http://0.0.0.0:50671
2016-07-20T00:04:11.119620+00:00 shinyapps[115363]: Loading required package: ggplot2
2016-07-20T00:04:11.341254+00:00 shinyapps[115363]: 
2016-07-20T00:04:11.341260+00:00 shinyapps[115363]: Attaching package: ‘plotly’
2016-07-20T00:04:11.341262+00:00 shinyapps[115363]: 
2016-07-20T00:04:11.341981+00:00 shinyapps[115363]: The following object is masked from ‘package:ggplot2’:
2016-07-20T00:04:11.341984+00:00 shinyapps[115363]: 
2016-07-20T00:04:11.341985+00:00 shinyapps[115363]:     last_plot
2016-07-20T00:04:11.341986+00:00 shinyapps[115363]: 
2016-07-20T00:04:11.342754+00:00 shinyapps[115363]: The following object is masked from ‘package:graphics’:
2016-07-20T00:04:11.342755+00:00 shinyapps[115363]: 
2016-07-20T00:04:11.342758+00:00 shinyapps[115363]: 
2016-07-20T00:04:11.342757+00:00 shinyapps[115363]:     layout
2016-07-20T03:20:09.766501+00:00 shinyapps[115363]: Server version: 0.4.4.1986
2016-07-20T03:20:09.766562+00:00 shinyapps[115363]: R version: 3.3.1
2016-07-20T03:20:09.766569+00:00 shinyapps[115363]: shiny version: 0.13.2
2016-07-20T03:20:09.766574+00:00 shinyapps[115363]: rmarkdown version: NA
2016-07-20T03:20:09.766575+00:00 shinyapps[115363]: knitr version: NA
2016-07-20T03:20:09.766575+00:00 shinyapps[115363]: jsonlite version: 1.0
2016-07-20T03:20:09.766603+00:00 shinyapps[115363]: RJSONIO version: NA
2016-07-20T03:20:09.766604+00:00 shinyapps[115363]: htmltools version: 0.3.5
2016-07-20T03:20:09.935430+00:00 shinyapps[115363]: Using jsonlite for JSON processing
2016-07-20T03:20:09.939423+00:00 shinyapps[115363]: 
2016-07-20T03:20:09.939429+00:00 shinyapps[115363]: Starting R with process ID: '17'
2016-07-20T03:20:09.955391+00:00 shinyapps[115363]: 
2016-07-20T03:20:09.955393+00:00 shinyapps[115363]: Listening on http://0.0.0.0:59412
2016-07-20T03:20:10.045463+00:00 shinyapps[115363]: Loading required package: ggplot2
2016-07-20T03:20:10.260363+00:00 shinyapps[115363]: 
2016-07-20T03:20:10.260369+00:00 shinyapps[115363]: Attaching package: ‘plotly’
2016-07-20T03:20:10.260372+00:00 shinyapps[115363]: 
2016-07-20T03:20:10.261097+00:00 shinyapps[115363]: The following object is masked from ‘package:ggplot2’:
2016-07-20T03:20:10.261100+00:00 shinyapps[115363]: 
2016-07-20T03:20:10.261101+00:00 shinyapps[115363]:     last_plot
2016-07-20T03:20:10.261834+00:00 shinyapps[115363]: The following object is masked from ‘package:graphics’:
2016-07-20T03:20:10.261836+00:00 shinyapps[115363]: 
2016-07-20T03:20:10.261839+00:00 shinyapps[115363]: 
2016-07-20T03:20:10.261838+00:00 shinyapps[115363]:     layout
2016-07-20T03:20:10.261102+00:00 shinyapps[115363]:

I have changed my excel file so that all formulas and formatting is removed, and only values are stored, but the app is still crashing before running any of the server.R code. I have tried saving the excel file as .csv files, but the file size is pretty much the same, and it would require a drastic change to my code to fully function, so I didn't pursue it much further.

I have seen some talk of using Google Sheet to write data from an app. Would reading my data from Google Sheets be a better route to work around this issue? Does anyone have any other solutions? Is reading my excel data even my problem?

A link to my app: https://loghan.shinyapps.io/Season_Projections/

L. Hansen
  • 33
  • 5
  • Update: Google sheets is unlikely to work. The number of unique values for each tracking variable (238 teams * 10,000 trials = 2.38 million values) exceeds the cell limit for a single google spreadsheet (2 million). Didn't know google sheets had a cell limit. – L. Hansen Jul 21 '16 at 01:44
  • Version of the app that is working, but with only two variables loaded into the app: [link](https://loghan.shinyapps.io/2016_Projections/). The data is stored in two separate sheets. Trying to put two variables-worth of data into a single sheet had the same problem. Definitely a problem relating to too much data in excel. – L. Hansen Jul 21 '16 at 01:59

2 Answers2

0

You can try openxlsx package in R. You may find the package useful. As it reads data much much faster. It uses read.xlsx to read the excel file.

Bhushan Pant
  • 1,445
  • 2
  • 13
  • 29
0

You can use .csv files instead of .xls files. Transitions between the two formats are easy in excel and R likes much more .csv files. To read those .csv files use fread in data.table package.

avidalvi
  • 128
  • 1
  • 6