1

I have a file of bytes 1.5GB in size {filebyte}. I want to read the entire file in one instance instance similar to Delphi's

bytedata:=filebyte.readallbytes(filename);

The result being that in one instance you will have a bytearray with the number of elements being high(bytedata)-low(bytedata)+1. Is there equivalent code in Cache. Can a file of 1.5G in size be held in memory in cache. I do not want to read the file in blocks as the operation to analyse the data requires that the whole file be in memory at one time. Thanks

VC.One
  • 14,790
  • 4
  • 25
  • 57
Jimmy Dean
  • 153
  • 2
  • 7

1 Answers1

1

You can read from the stream as many data as you need. The problem is here, how much you can store in a local variable.

set fs=##class(%Stream.FileCharacter).%New()
set fs.Filename="c:\test.txt"
set length=fs.Size
set data=fs.Read(length) \\ if size no more than 3.5Mb

Local variable size limited by 3,641,144 bytes or 32,767 bytes of long strings diabled. And up to 2012.1 memory per process was limited by 48mbytes. And in 2012.2 it was changed and it is possible to set up to 2 terabytes per process, and in real time programmatically just for a current process with special variable $zstorage.

DAiMor
  • 3,185
  • 16
  • 24
  • In your example does the variable data hold the entire file? If so how would access the sub-elements? – Jimmy Dean Jan 31 '17 at 15:01
  • Yes, it will have the entire content, but what do you mean by sub elements? – DAiMor Jan 31 '17 at 16:35
  • If the file consists of 1000 bytes of values between (1-4) exclusive will data(1)=2,data(2)=4....data(999)=1,data(1000)=4 or will data=2421123.....14 where 14 are the 999th and 1000th byte. If the latter then the individual bytes cannot be easily manipulated and there is no advantage in reading the entire file over reading one value at a time. In my delphi example bytedata:=filebyte.readallbytes(filename) bytedata is an an array with values from 0 to length-1 and bytedata[0]=2,bytedata[2]=4, bytedata[998]=1 and bytedata[999]=4. Is there a similar operation in cache? – Jimmy Dean Jan 31 '17 at 17:07
  • data, it is just a string, and you can get any symbol or subscript, with $extract function, for details look at the documentation - http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=RCOS_fextract – DAiMor Jan 31 '17 at 19:55
  • a string of length 1.5GB will exceed max string length so $extract is not appropriate. – Jimmy Dean Jan 31 '17 at 20:18
  • I don't understand, why this way is not appropriate for you. You want to load entire all file and you don't want to use $extract, which can be used in both ways, to get particular parts, or change it. Maybe I don't understand your task, so, you should a bit more details. You asked about reading all file to the memory. I've answered you, is not it. – DAiMor Jan 31 '17 at 21:44
  • I should have been clearer. What I was after was to read the entire file into memory contained automatically in a byte array with no itermediate steps. see the delphi example. It seems to me that this cannot be done directly in cache and your solution is the one that is most likely to succeed i.e. s data=fs.read(length) f i=1:1:length s byte[i-1]=$e(data,i) increasing the storage and memory capacity will I think allow me to do this for my 1.5GB file. I thank you for your insight and solution that has helped me alot. – Jimmy Dean Feb 01 '17 at 02:35