Reading the Fortran real data type into Matlab from a hex dump

Question

As the title says I am writing a large amount of real arrays from Fortran into an unformatted file, and then trying to read that binary file into Matlab. I have successfully made my script work for strings and integers, but It does not correctly read my real numbers from the hex dump.

As a test case I was using the number 5.49. Interesting side note, according to an online converter that is 40 af ae 14, yet when I check my hexfile that portion of my code is reading 14 ae af 40. I have tried reading it in as a float32 and double, and I have changed fro neutral to little endian to big endian. Any ideas?

Here is a simple example of my code:

First the Fortran write statements

REAL :: floating = 5.49
open(unit = 2, file = "anxietySource", form = "unformatted", status = "new", action = "readwrite")
write(unit = 2 ) floating

Now the Matlab read statement

fid = fopen('anxietySource', 'rb');
h1 = fread(fid, 1, 'int32'); %this is just reading off the starter bits that tell me how long my write statement is
floating = fread(fid,1,'float32');
display(floating);
fclose(fid);

My guess is that there is something funky with the Fortran REAL type. Maybe it's not quite a floating point?

not really addressing the central question, but with modern fortran you can specify `access='stream'` on open and so not need to deal with the header data. Try that and verify the resulting file is precisely four bytes. — agentp, Nov 17 '16 at 16:07
for comparison on my little endian system the bytes are `14 ae af 40`and mathematica reads it as a "Real32" just fine. I assume you realize `5.49` is not exactly representable and the actual value you have is `5.48998..` or so. (not that familiar with matlab but I suppose `floating` is 64 bit so you will see something like that. ) — agentp, Nov 17 '16 at 16:34

Holmz · Answer 1 · 2016-11-23T12:34:14.950

Good job on the bit level work getting this far, you are almost there. "Big Endien versus little..."

https://gcc.gnu.org/onlinedocs/gfortran/CONVERT-specifier.html

http://www.lahey.com/docs/lfenthelp/F95ARINQUIREStmt.htm

https://software.intel.com/en-us/forums/intel-fortran-compiler-for-linux-and-mac-os-x/topic/270026

There is also manually shifting it https://www.nsc.liu.se/~boein/f77to90/a5.html#section10

MODULE Shifty
PRIVATE

INTERFACE Shift
  MODULE PROCEDURE Shift4 !,Shift8
END INTERFACE

CONTAINS
REAL FUNCTION Shift4(MyFloat)
IMPLICIT NONE
REAL, INTENT(IN) :: MyFloat
INTEGER*4        :: Float1
INTEGER*4        :: Float2
INTEGER*4        :: Float3
INTEGER*4        :: Float4

INTEGER*4        :: MyInt
EQUIVALENCE(MyInt, MyFloat)
INTEGER*4        :: MyIntOut
EQUIVALENCE(MyIntOut, Shift4)

WRITE(*,20) MyInt
20 FORMAT('Incoming Real=',1PE12.5,' ==",Z8.8,'"')

Float1 = IBITS(MyInt, 0, 8)
Float2 = IBITS(MyInt, 8, 8)
Float3 = IBITS(MyInt,16, 8)
Float4 = IBITS(MyInt,24, 8)

WRITE(*,30) 1, Float1      !Check
WRITE(*,30) 2, Float2
WRITE(*,30) 3, Float3
WRITE(*,30) 4, Float4
30 FORMAT('Float',I1,'="',Z2.2,'"')

Float1 = ISHFT(Float1,  24)
Float2 = ISHFT(Float2,  16)
Float3 = ISHFT(Float3,   8)
Float4 = ISHFT(Float4,   0)

MyIntOut = IOR(Float1  , Float2)
MyIntOut = IOR(MyIntOut, Float3)
MyIntOut = IOR(MyIntOut, Float4)

WRITE(*,20) MyInt, MyIntOut
20 FORMAT('Incoming Real="',Z8.8,' Outgoing="',Z8.8,'"')

RETURN
END FUNCTION Shift4

END MODULE Shifty

PROGRAM MAT2F90
USE Shifty
IMPLICIT NONE

TYPE MATLAB_HEADER
  CHARACTER(LEN=120) :: Descriptor  !should be 116 and INT*8 !!
  INTEGER*4          :: Offset
  INTEGER*2          :: Version
  CHARACTER(LEN=2)   :: Endien
END TYPE MATLAB_HEADER

TYPE MATLAB_SUBHEADER
  INTEGER*4          :: Type
  INTEGER*4          :: Bytes
END TYPE MATLAB_SUB_HEADER

TYPE(MATLAB_HEADER)                   :: Head
TYPE(MATLAB_SUB_HEADER),DIMENSION(20) :: Tag
CHARACTER(LEN=12), DIMENSION(18) :: Matlab_Version =  RESHAPE(&
['miINT8      ','miUINT8     ','miINT16     ','miUINT16    ', &
 'miINT32     ','miUINT32    ','miSINGLE    ','Reserved    ', &
 'miDOUBLE    ','Reserved    ','Reserved    ','miINT64     ', &
 'miUINT64    ','miMATRIX    ','miCOMPRESSED','miUTF8      ', &
 'miUTF16     ','miUTF32     '],[18])
LOGICAL :: Swap
...
OPEN(UNIT=22,FILE='<somename>',ACCESS='STREAM',FORM='UNFORMATTED',IOSTAT=Status)
IF(Status =/0 ) ...Do something

READ(20,IOSTAT=Status) Head
IF(Status =/0 ) ...Do something

WRITE(*,*)'Head.Descriptor="',Head.Descriptor,'"'
WRITE(*,*)'Head.Offset    = ',Head.Offset
WRITE(*,*)'Head.Version   = ',Head.Version
WRITE(*,*)'Head.Endien    ="',Head.Endian,'"'
IF(Head.Endian == 'IM') THEN
  SWAP = .FALSE.
ELSEIF(Head.Endian == 'MI') THEN
  SWAP = .TRUE.
ELSE
  WRITE(*,*)'Unknown  Endien="',Head.Endian,'"'
  STOP
ENDIF

READ(20,IOSTAT=Status) Tag(1)
IF(Status =/0 ) ...Do something
WRITE(*,*)'Tag(1).Type = ',Tag(1).Type,' == "',Matlab_Version(Tag(1).Type),'"'
WRITE(*,*)'Tag(1).Bytes= ',Tag(1).Bytes
!read and swap if need be...
!There is padding to an 8type boundary
!Read the next tag and data... etc

Section 1-5 to 1-9 https://data.cresis.ku.edu/data/mat_reader/matfile_format.pdf . You may notice like I did that type15 is 'miCOMPRESSED', so at that point one needs the uncompressor to make sense of it.

You'll need to test it like you did before, because it is easy for me to get the order wrong, and I am doing this by memory.(Updated as I was doing this today, but in a subroutine, so it should work as a function??)

I made it 4 in case you need an '8' version... And then you just call Shifty() and you can get the one that matches your data.

Thanks! I have tried reading it in as both big and little endian, but I haven't tried to write it as a specific type yet. Maybe if a I explicitly write it in as one or the other, it will fix my problem. I will let you know ! — Chair, Nov 17 '16 at 12:26
If it is an endian issue It may be preferable to fix it on the matlab side (For one thing the fortran fixes are not standard ). `fread` takes a simple argument to do the byte swap. — agentp, Nov 17 '16 at 17:59
I was doing this today, and poked in what I recall. The basic gist of the problem is that with type15 it is 'miCOMPRESSED', so whether the endien matters or not, one needs to run through an unCompressor to make sense of it. The fact I got a 15 for the type and that data was 864 Bytes for a length=100 vector means that the integers were coming in fine, so I may not need to be concerned with Endien='MI'... https://data.cresis.ku.edu/data/mat_reader/matfile_format.pdf — Holmz, Nov 23 '16 at 12:38

Reading the Fortran real data type into Matlab from a hex dump

1 Answers1