0

I'm trying to setup a sftp server over GCS.

An example gcsfuse mount in the container is

gcsfuse -o nonempty --only-dir user1 $BUCKET /home/user1/ftp

When I upload two files using sftp:

sftp> put 1024K.txt
Uploading 1024K.txt to /ftp/1024K.txt
1024K.txt                                             100% 1025KB 426.0KB/s   00:02    
sftp> put 8K.txt
Uploading 8K.txt to /ftp/8K.txt
8K.txt                                                100% 8200   117.6KB/s   00:00    
Couldn't close file: Failure

but both files are transferred ok:

sftp> ls -l
-rw-r--r--    1 1003     1003      1049600 Mar 15 13:33 1024K.txt
-rw-r--r--    1 1003     1003         8200 Mar 15 13:33 8K.txt
  • I am running on a standard debian-stretch instance.
  • The instance has storage R/W permission.
  • I have tried some gcsfuse options with no success: --type-cache-ttl 0 --stat-cache-ttl 0 --implicit-dirs
  • I tried ssh debug logging on the client and server - no new information.

As the large file upload is successful, I don't think it relates to permissions or sftp setup. What could cause the small file upload to fail?

(Edited to remove the reference to K8 and GKE as the behaviour is reproducible without)

Jasper Humphrey
  • 1
  • 4

1 Answers1

0

It seems to be caused by writes not being flushed to disk after every transfer. See this example:

sftp> put 8K.txt
Uploading 8K.txt to /home/jasper_humphrey/user1/8K.txt
8K.txt                                        100% 8200   111.5KB/s   00:00    
Couldn't close file: Failure
sftp> ls
8K.txt  
sftp> put 8K.txt
Uploading 8K.txt to /home/jasper_humphrey/user1/8K.txt
8K.txt                                        100% 8200   125.9KB/s   00:00    
sftp> ls
8K.txt  
sftp> put 8K.txt
Uploading 8K.txt to /home/jasper_humphrey/user1/8K.txt
remote open("/home/jasper_humphrey/user1/8K.txt"): Failure

If I use the -f argument on the sftp client, then this triggers an fsync after every transfer, and the error does not occur. From the sftp man page: 

 -f      Requests that files be flushed to disk immediately after transfer.  When
         uploading files, this feature is only enabled if the server implements the
         "fsync@openssh.com" extension.
Jasper Humphrey
  • 1
  • 4
  • It's also possible to fix this server side by setting an argument "-o sync" to the gcsfuse command that will force synchronous IO. – Jasper Humphrey Apr 11 '19 at 16:10