0

I have a file write in big-endian format. I wrote a library to access this file. Now I try to use this library to access this file inside a python script. For this I wrote some routines and compile then using f2py. The problem is that in gfortran (a compiler that I use) the options "-fconvert=big-endian" has an effect only when used in the main program, not inside a library. So I can't access correctly this file in python.

Bellow I put a little example to reproduce this issue:

The file "fileTest.bin" was created by this program:

program ttt
   implicit none
   real(kind=4) :: m(2,2)

   m=10.0

   open(unit=100,file='fileTeste.bin', form='unformatted')
   write(100)m

   close(100)
end program

to compile:

gfortran -o ttt.x -fconvert=big-endian ttt.f90

and execute :

./ttt.x

I wrote a test module to read this big-endian file inside python:

module test
   implicit none
   contains
   subroutine openFile(fileName)
      character(len=*), intent(in) :: fileName
      real(kind=4), allocatable :: array(:,:)

      print*,"open File:", trim(fileName)
      open(unit=100,file=trim(fileName),form='unformatted')

      allocate(array(2,2))
      read(100)array
      print*,array
   end subroutine openFile
end module test

compile using f2py:

f2py -c -m test --f90flags='-fconvert=big-endian' test.f90

load test module inside python:

from test import test as t

a=t.open(teste.bin)

that result in:

open File:fileTeste.bin
   1.15705214E-41   1.15705214E-41   1.15705214E-41   1.15705214E-41

So, how can I read a big-endian file using f2py?

thanks !

João Gerd Zell de Mattos
  • 11
  • 4
  • I never liked these convert options. IMHO it is much better to convert the byte order yourself. We have a few Fortran questions about that. You can also do it in Python https://stackoverflow.com/questions/27506474/how-to-byte-swap-a-32-bit-integer-in-python (first search hit, I did not read the answers). – Vladimir F Героям слава Nov 11 '21 at 21:17
  • @VladimirF, yes, python way: `struct.unpack("f", 1.15705214E-41 ))[0]` return `10.0`, and vice versa. – Serge3leo Nov 12 '21 at 00:13
  • @VladimirF, I don't like it either, but I use a atmosheric model and its files are in spectral form write in big-endian format. So, my library read this files, convert from spectral to grid space then "send" to python. Maybe I'll write a code to convert byte order, I think that is the best way ... Thanks for your suggestions! – João Gerd Zell de Mattos Nov 12 '21 at 15:11
  • @Serge3leo the problem with this option is that I need use the matrix inside fortran to do some mathematical operations before send to python. Thanks for your suggestions too! – João Gerd Zell de Mattos Nov 12 '21 at 15:11
  • As I wrote, there are already examples on this site, like https://stackoverflow.com/questions/30061362/fortran-subroutine-that-changes-byte-order or the link to my module I gave at https://stackoverflow.com/questions/68931014/byte-swap-real-array?noredirect=1&lq=1 – Vladimir F Героям слава Nov 12 '21 at 15:46
  • @VladimirF, I saw your post about that. I will pick up your code (https://bitbucket.org/LadaF/elmm/src/master/src/endianness.f90) and will do a test in my code. How is the performance? I have a lot of 3D matrices to do this (each matrix has 900x450 points. Some others are bigger than that) – João Gerd Zell de Mattos Nov 13 '21 at 16:48
  • I do not think you can do much faster in Fortran. It is quite a long time since I did a performance test, but I use it for very large arrays. – Vladimir F Героям слава Nov 13 '21 at 17:50

1 Answers1

0

You can use CONVERT specifier of OPEN statement:

open(unit=100,file=trim(fileName),form='unformatted',convert='big_endian')

See https://gcc.gnu.org/onlinedocs/gcc-11.2.0/gfortran/CONVERT-specifier.html

Or set environment variable GFORTRAN_CONVERT_UNIT, see https://gcc.gnu.org/onlinedocs/gcc-11.2.0/gfortran/GFORTRAN_005fCONVERT_005fUNIT.html

Serge3leo
  • 411
  • 2
  • 4
  • 1
    It is good to note that it is not standard Fortran and will not compile with many compilers. – Vladimir F Героям слава Nov 12 '21 at 06:57
  • @VladimirF, Yes, `open(... convert='big_endian'...)` non-standard specifier. But many compilers: GNU/NAG/Intel/IBM XL/Absoft/... fortran, support it. – Serge3leo Nov 12 '21 at 19:48
  • I agree with @VladimirF, again. That's why I didn't do it like this in my library. I use gfortran, but my work colleagues use Cray Fortran that does not support CONVERT because it's not Fortran standard. The library used in python is the same used in another program written in Fortran. I'm trying to write an interface to access my data inside python too, so is more easy use this already written library. – João Gerd Zell de Mattos Nov 13 '21 at 16:56