2

I have a fortran code which needs to read a series of ascii data files (which all together are about 25 Gb). Basically the code opens a given ascii file, reads the information and use it to do some operations, and then close it. Then opens another file, reads the information, do some operations, and close it again. And so on with the rest of ascii files.

Overall each complete run takes about 10h. I usually need to run several independent calculations with different parameters, and the way I do is to run each independent calculation sequentially, so that at the end if I have 10 independent calculations, the total CPU time is 100h.

A more rapid way would be to run the 10 independent calculations at the same time using different processors on a cluster machine, but the problem is that if a given calculation needs to open and read data from a given ascii file which has been already opened and it's being used by another calculation, then the code gives obviously an error.

I wonder whether there is a way to verify if a given ascii file is already being used by another calculation, and if so to ask the code to wait until the ascii file is finally closed.

Any help would be of great help. Many thanks in advance.

Obamakoak.

obamakoak
  • 23
  • 5

2 Answers2

1

Two processes should be able to read the same file. Perhaps action="read" on the open statement might help. Must the files be human readable? The I/O would very likely be much faster with unformatted (sometimes call binary) files.

P.S. If your OS doesn't support multiple-read access, you might have to create your own lock system. Create a master file that a process opens to check which files are in use or not, and to update said list. Immediately closing after a check or update. To handle collisions on this read/write file, use iostat on the open statement and retry after a delay if there is an error.

M. S. B.
  • 28,968
  • 2
  • 46
  • 73
1

I know this is an old thread but I've been struggling with the same issue for my own code.

My first attempt was creating a variable on a certain process (e.g. the master) and accessing this variable exclusively using one-sided passive MPI. This is fancy and works well, but only with newer versions of MPI.

Also, my code seemed happy to open (with READWRITE status) files that were also open in other processes.

Therefore, the easiest workaround, if your program has file access, is to make use of an external lock file, as described here. In your case, the code might look something like this:

  • A process checks whether the lock file exists using the NEW statement, which fails if a file already exists. It will look something like:

    file_exists = .true.
do while (file_exists)
    open(STATUS='NEW',unit=11,file=lock_file_name,iostat=open_stat)
    if (open_stat.eq.0) then
        file_exists = .false.
        open(STATUS='OLD',ACTION=READWRITE',unit=12,file=data_file_name,iostat=ierr)
        if (ierr.ne.0) stop
    else
        call sleep(1)
    end if
end do

  • The file is now opened exclusively by the current process. Do the operations you need to do, such as reading, writing.

  • When you are done, close the data file and finally the lock file

    close(12,iostat=ierr)
if (ierr.ne.0) stop
close(11,status='DELETE',iostat=ierr)
if (ierr.ne.0) stop

  • The data file is now again unlocked for the other processes.

I hope this may be useful for other people who have the same problem.

Toon
  • 187
  • 11