1

Can the feather package in R support 64-bit integers?

When the dataset is passed to feather::write_feather(), the column is converted to a 64-bit float, and loses precision. I'd like to avoid converting it to a character.

Here's a simplified example. In the real project, a database table (retrieved with the odbc package) has columns that are legit 64-bit integers (as specified in the bit64 package). 

requireNamespace("bit64")

path <- base::tempfile(fileext = ".feather")

ds <-
  tibble::tibble(
    patient_id   = bit64::as.integer64(1:6)
  )
ds

# # A tibble: 6 x 1
#   patient_id
#   <int64>   
# 1 1         
# 2 2         
# 3 3         
# 4 4         
# 5 5         
# 6 6 

feather::write_feather(x = ds, path = path)

ds_read <- feather::read_feather(path)
# # A tibble: 6 x 1
#    patient_id
#         <dbl>
# 1 Inf.Nae-324
# 2 Inf.Nae-324
# 3   1.50e-323
# 4   2.00e-323
# 5   2.50e-323
# 6   3.00e-323


as.integer(ds_read$patient_id)
# Returns: [1] 0 0 0 0 0 0

unlink(path_out)

Note: I don't want to store them as floats, as suggested here.

wibeasley
  • 5,000
  • 3
  • 34
  • 62

2 Answers2

1

It is actually "complicated". As you probably know, R itself has only two types: 32-bit integer and 64-bit double.

So to represent 64-bit integers, Jens did quite some work in his bit64 package to use double as a "carrier" for the 64-bit payload and redefining all accessor functionality to treat it as as 64-bit (signed) integer. That works.

Several packages support it natively, for example data.table. I took advantage of this when I created nanotime -- which uses 64-bit integers for nanoseconds since the epoch. This also works: we never convert to double in between and get faithful integer64 representation.

I have also been following reticulate over the years, and it has very similar conversion issues from 64-bit integers (as those are native in Python) which are by now generally addressed.

So long story short: your question is more of a feature request for feather. And as those involved focus now on arrow which appears to have 64-bit integer support, you most likely will just be asked to move to arrow. Or you could use data.table.

Dirk Eddelbuettel
  • 360,940
  • 56
  • 644
  • 725
  • 3
    That context helps, thanks. On the downside [installing arrow](https://arrow.apache.org/install/) required some more work to install the dependencies. But once installed, the code modification was as simple as replacing `feather::write_feather()` with `arrow::write_feather()`. – wibeasley Oct 19 '19 at 02:46
  • I don't think `data.table` will work, as I am trading datasets with a Python user (which I didn't specify in my question), but maybe I'm misunderstanding your last sentence. – wibeasley Oct 19 '19 at 02:46
  • 1
    Yes. Once you have your data in R you could use `data.table` to store. It won't help you on the _initial conversion from Python_. For that `reticulate` may help. – Dirk Eddelbuettel Oct 19 '19 at 10:31
0

I have the similar problems with saving "big" numbers (integer64) in the feather format. Using "arrow::write/read" was looking good but in production it gives me unexpected problems and shows unstable behaviour.

So, at the end I returned back to the original "feather::write/read" approach and added transformation integer64 into "as.double(x)" format.

Hope this can be helpful!

Andrii
  • 2,843
  • 27
  • 33