I am trying to run many linear regressions and diagnostics over them and to speed things up I am using the doParallel package in the R programming language.
I have come across though an interesting issue. Although I have seen performance improvement -as expected- the usage of the CPU is not consistent.
For example, if I do a run of my code, CPU utilization may be 30-40% for all cores.
If I run again my code, it is possible that CPU utilization goes up to 90% without me changing anything in the meantime.
In both cases, I am not running anything else on the same time.
Is there an explanation why the cores would be used 30% one time and 90% another time without me changing anything?
I am running Windows XP, 4GB or RAM, and have an Inter(R) Xeon(R) CPU X5650 @ 3.67GHz.
My code looks something like:
results <- foreach(i=seq(1:regressions), .combine='rbind', .export=c('lmResults','lmCSig', 'lmCVar')) %dopar%
{model <- lm(as.formula(as.character(dat[i])), data=df) lmResults(model) }