1

So I have a collection on json files located on my local machine that I am reading in currently using the command

file <- tbl_df(ndjson::stream_in("path/to/file.json")

I have copied these files to a linux server (using WinSCP) and I want to stream them in to my R session just like I did in the above code with ndjson. When searching for ways to do this I came across one method using RCurl that looked like this

file <- scp(host = "hostname", "path/to/file.json", "pass", "user")

but that returned an error 

Error in function (type, msg, asError = TRUE)  : Authentication failure

but either way I want to avoid copying my passphrase into my Rscript as other will see this script. I also came across a method suggesting this

d <- read.table(pipe('ssh -l user host "cat path/to/file.json"'))

however this command returned the error

no lines available in input

and I believe read.table would cause me issues anyways. Does anyone know I way I could read new line delimited json files from a remote server into an R session? Thank you in advance! Let me know if I can make my question more clear.

grapestory
  • 183
  • 11
  • Using `scp` and other ssh-related programs without needing to type in a password is well-documented in the use of a key pair and some variant of `ssh-agent` (depending on your OS). If you don't want to type your password within the R script (very smart, btw), then no combination of `scp` or `ssh` or `pipe(...)` or something else is going to fix that, you need to address it outside of R. – r2evans Apr 06 '20 at 21:44
  • (I don't know that `RCurl`'s functions use ssh-agent functionality, so it might be moot.) – r2evans Apr 06 '20 at 21:53
  • Are you hoping to automate it so that it works for whomever is running the script (with your account), or do they have their own accounts to be able to use their own passwords? – r2evans Apr 06 '20 at 21:53
  • Okay good to know that no combination of `scp` or `ssh` etc will solve my problem! As for your question, I would like to automate it so that it works for whomever (this a previously established end-goal), but other users do have their own accounts and can use their own passwords. I am currently more interested in simplicity rather than automation! – grapestory Apr 06 '20 at 23:20
  • I've never had success (and recently not tried hard) to do file transfer from within R. I have no doubt that it can be done somehow, but I have better efficiencies using `scp` and `rsync` (or WinSCP for some users) with my ssh-agent managing keys for me. – r2evans Apr 06 '20 at 23:24
  • Well my issue is more related to being able to "stream in" the files rather than transfer them. I actually used WinSCP to move them *onto* this remote server. I want to be able to free my local machine's memory and store (but be able to read/stream in) these files on the remote machine. Am I making sense? – grapestory Apr 08 '20 at 16:28
  • I understand. And if I could get my R to see my ssh-agent, I would be able to test some things. But I can't and have had no impetus to figure out why it isn't working (b/c my ssh-agent stuff works just fine in the terminal/shell). If you can get an `ssh` command to work, then perhaps `jsonlite::stream_in(pipe("cat ~/StackOverflow/file.json"))` (which works) can be replaced with `jsonlite::stream_in(pipe("ssh cat remotehost:/path/to/file.json"))` and it should just work. (No `read.table`, btw.) – r2evans Apr 08 '20 at 16:39

0 Answers0