0

I am running a large loop in parallel to do regression. The loop was working fine and everything was well but I am suddenly observing very strange a behavior. Some iterations do not reach the step of regression itself. Here's a simple snippet of what I am trying:

cl<-makeCluster(14,outfile="")
registerDoParallel(cl)
sink("MajorSink.log")
NG_PSKU=c(1:234)
NG<-foreach(i=NG_PSKU,.multicombine = T,.errorhandling = "pass",.verbose = T) %dopar%
              {
                print(i)
                conn <- file( sprintf(paste0("output_%d.txt") , Sys.getpid()) , open = "a+" )
                sink(conn,append = T)
                cat("NG_PSKU:",i,'\n')
                #regression on NG_PSKU
               }

What I observed in the output_%d log files is that the print(i) command was successfully run but cat("NG_PSKU:",i,'\n') was never reached beyond 162nd iteration.

[1] 162
NG_PSKU:    162
[1] 163
[1] 164
[1] 165
[1] 166
[1] 167
[1] 168
[1] 169
[1] 170
[1] 171
[1] 172
[1] 173
[1] 174
[1] 175
[1] 176
[1] 177
[1] 178
[1] 179
[1] 180
[1] 181
[1] 182
[1] 183
[1] 184
[1] 185
[1] 186
[1] 187
[1] 188
[1] 189
[1] 190
[1] 191
[1] 192
[1] 193
[1] 194
[1] 195
[1] 196
[1] 197
[1] 198
[1] 199
[1] 200
[1] 201
[1] 202
[1] 203
[1] 204
[1] 205
[1] 206
[1] 207
[1] 208
[1] 209
[1] 210
[1] 211
[1] 212
[1] 213
[1] 214
[1] 215
[1] 216
[1] 217
[1] 218
[1] 219
[1] 220
[1] 221
[1] 222
[1] 223
[1] 224
[1] 225
[1] 226
[1] 227
[1] 228
[1] 229
[1] 230
[1] 231
[1] 232
[1] 233
[1] 234

To further investigate I ran the code again with a shorter NG_PSKU (102- 199). To my amazement the code ran successfully. I am left wondering what is happening here. Has anyone else faced a similar situation before? Any help is highly appreciated.

user3825354
  • 13
  • 6
  • Have you tried to combine afterwards? – F. Privé Apr 03 '18 at 10:21
  • @F.Privé by afterwards you mean? I am sorry I don't think I get the full context here.. – user3825354 Apr 03 '18 at 10:34
  • I meant: don't use the `.combine` argument (you will get results as a list) and them combine the results based on this list, only once at the end. – F. Privé Apr 03 '18 at 11:11
  • @F.Privé okay but combine was not the only (rather major) problem here. I am concerned on the behavior of the loop execution. Why did the loop not proceed to cat statement (or regression after that).. You have any idea what could be happening there? – user3825354 Apr 03 '18 at 12:34
  • Difficult to say without a fully reproducible example. It could be the combining function so that you try without first and see if it solves the problem. – F. Privé Apr 03 '18 at 18:01
  • @F.Privé I tried running the code without combine function and nothing changed. editing my question to remove combine from the scenario.. – user3825354 Apr 04 '18 at 03:06

0 Answers0