GNU FORTRAN OpenACC Accessing an allocatable member of a derived type on device

Question

I need to access an allocatable member of a derived type on device. I found a good solution here. However, it uses a PGI compiler. I could not figure out how to install that compiler with all the licensing hurdles. So I am looking for a solution with GNU Fortran (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0.

The sample program below runs fine without the -fopenacc flag. However, once the flag is enabled, I get the below error:

libgomp: cuStreamSynchronize error: an illegal memory access was encountered

Here is how I run the code (saved in a file named: stackOverflow.f95) without OpenACC:

$ gfortran -c stackOverflow.f95 
$ gfortran stackOverflow.o -o stackOverflow
$ ./stackOverflow 

Here is how I run the code with OpenACC:

$ gfortran -fopenacc -c stackOverflow.f95 
$ gfortran -fopenacc stackOverflow.o -o stackOverflow
$ ./stackOverflow 

Please see if there is a solution? Here is the sample code:

module modTest2

use openacc
implicit none
 
type :: Container
  sequence
  integer :: n
  integer, allocatable :: arr(:)
end type Container

interface Container
  procedure :: new_Container
end interface Container
   
contains

  type(Container) function new_Container(n)
    integer, intent(in) :: n
    
    allocate(new_Container%arr(n))
  end function new_Container
end module modTest2




program test2

  use modTest2
  implicit none
  

  integer :: n, i
  type(Container) :: c


  print *, "Enter array size: "
  read *, n  
  print *, "Allocating..."
  c = Container(n)
  print *, "Allocation complete!"

  
  print *, "Initializing..."
  !$acc data copyin(c)
  !$acc parallel loop present(c)
  do i = 1, n
    c%arr(i) = i*20
  end do
  !$acc end data
  print *, "Initialization complete..."

  do i = 1, n
    print *, i, c%arr(i)
  end do

end program test2

Answer 1

The problem is that you need to do a deep copy of "c%arr". By default, the copy clauses perform a shallow copy. Hence with "copy(c)", it's copying the host address of "arr" which is not accessible on the device.

Since I work for NVIDIA and on the NVHPC Compiler team (aka PGI), I don't have insights into gfortran, but assume the following code will work for them as well. I prefer using unstructured data regions so included both structured and unstructured versions. Note that order is important since "c" should be created on the device prior to "c%arr" so the compiler can implicitly "attach" it to "c". "attach" basically fills in the device pointer to "arr" in the device copy of "c". Reversing the order would require the code to explicitly attach via an "attach" clause or call to the "acc_attach" API.

Example:

% cat test2.f90
module modTest2

use openacc
implicit none

type :: Container
  sequence
  integer :: n
  integer, allocatable :: arr(:)
end type Container

interface Container
  procedure :: new_Container
end interface Container

contains

  type(Container) function new_Container(n)
    integer, intent(in) :: n

    allocate(new_Container%arr(n))
  end function new_Container
end module modTest2

program test2

  use modTest2
  implicit none


  integer :: n, i
  type(Container) :: c


  print *, "Enter array size: "
  read *, n
  print *, "Allocating..."
  c = Container(n)
  print *, "Allocation complete!"


  print *, "Initializing..."
  !acc enter data copyin(c) create(c%arr)
  !$acc data copyin(c) copyout(c%arr)
  !$acc parallel loop present(c)
  do i = 1, n
    c%arr(i) = i*20
  end do
  !$acc end data
  !acc exit data copyout(c%arr) delete(c)
  print *, "Initialization complete..."

  do i = 1, n
    print *, i, c%arr(i)
  end do

end program test2
% nvfortran test2.f90 -acc; a.out
 Enter array size:
1024
 Allocating...
 Allocation complete!
 Initializing...
 Initialization complete...
            1           20
            2           40
            3           60
            4           80
            5          100
            6          120
            7          140
... cut ...
         1020        20400
         1021        20420
         1022        20440
         1023        20460
         1024        20480

Note that the NVHPC SDK, including the compilers, is available at no cost. Please see: https://developer.nvidia.com/hpc-sdk

Answer 2

Instead of Syntax error in OpenMP variable list, current GCC diagnoses mixed component and non-component accesses for !$acc data copyin(c) copyout(c%arr) as well as !$acc enter data copyin(c) create(c%arr) and !$acc exit data copyout(c%arr) delete(c). It's not clear if that is meant to be supported by OpenACC -- I'm aware there are unresolved issues with respect to clause processing ordering etc., that the OpenACC Technical Committee needs to look into. (I'm a member of that committee, too.) So what PGI/NVHPC are implementing may not actually be standard OpenACC.

That said, I'm able to make the program work with GCC/OpenACC as follows: Split the !$acc data copyin(c) copyout(c%arr) into two nested data regions:

!$acc data copyin(c) 
!$acc data copyout(c%arr)
[...]
!$acc end data
!$acc end data

Or alternatively, instead of !$acc enter data copyin(c) create(c%arr) and !$acc exit data copyout(c%arr) delete(c) split these up into:

!$acc enter data copyin(c)
!$acc enter data create(c%arr)
[...]
!$acc exit data copyout(c%arr)
!$acc exit data delete(c)

With both these variants, I'm then getting the same results as what Mat posted above.