CONTEXT
School project : to host several services on a kubernetes node minikube including a ftps server on port 21 using a load balancer. School ask us to make our own images based on Alpine.
What I did so far
I setup metallb as loadbalancer.
I build an image in which I setup vsftpd and generate certificate and keys :
openssl req -x509 -subj "/C=FR/ST=PARIS/L=Paris/O=42_Corp/OU=myname/CN=mywebsite" -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/vsftpd.key -out /etc/ssl/certs/vsftpd.crt
side note : I use supervisor in order to run telegraf in the same container to get a dashboard.
I deploy this image with this yml:
metadata:
name: ftps
spec:
serviceName: "ftps"
selector:
matchLabels:
app: ftps
replicas: 2 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: ftps
spec:
containers:
- name: ftps
image: ft_service/ftps
imagePullPolicy: Never
volumeMounts:
- name: ftps-data
mountPath: /mnt/ftps_data
volumes:
- name: ftps-data
---
apiVersion: v1
kind: Service
metadata:
labels:
app: ftps
name: ftps
namespace: default
spec:
type: LoadBalancer
externalTrafficPolicy: Cluster
ports:
- name: port-20
port: 20
protocol: TCP
targetPort: 20
- name: port-21
port: 21
protocol: TCP
targetPort: 21
selector:
app: ftps
Then I have a conf file for vsftp which contain this :
local_enable=YES
write_enable=YES
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
chroot_local_user=YES
allow_writeable_chroot=YES
listen=YES
rsa_cert_file=/etc/ssl/certs/vsftpd.crt
rsa_private_key_file=/etc/ssl/private/vsftpd.key
ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
require_ssl_reuse=NO
ssl_ciphers=HIGH
seccomp_sandbox=NO
(the last "seccomp_sandbox=NO" seemed to resolve some issues I had a the begining)
Which gives me...
Once I deploy everything, I get my pods :
default pod/ftps-1 1/1 Running 0 7m55s
I get my service:
default service/ftps LoadBalancer 10.104.132.127 192.168.1.241 20:31268/TCP,21:31581/TCP 8m43s
Then when I run minikube service ftps --url,
I get
http://172.17.0.2:31268
http://172.17.0.2:31581
WHERE I HAVE AN ISSUE
I use lftp (I tried few other with pretty much the same result, so I ruled out the client issue... but I pretty new to the whole ftp thing so... I let you judge)
1. Using lftp with ftps://
lftp -u user,password ftps://172.17.0.2 -p 31581
then ls
ls: Fatal error: gnutls_handshake: An unexpected TLS packet was received
Here is the very long version using debug (I truncated many "GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65" lines because of post limitations) :
lftp ftp42@172.17.0.2:~> debug 99999999
lftp ftp42@172.17.0.2:~> ls
FileCopy(0x563eb3eed780) enters state INITIAL
FileCopy(0x563eb3eed780) enters state DO_COPY
---- dns cache hit
---- attempt number 1 (max_retries=1000)
---- Connecting to 172.17.0.2 (172.17.0.2) port 31581
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
[...]
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: common.c[x509_read_value]:698
GNUTLS: ASSERT: common.c[x509_read_value]:698
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: REC[0x563eb42050a0]: Allocating epoch #0
GNUTLS: ASSERT: constate.c[_gnutls_epoch_get]:600
GNUTLS: REC[0x563eb42050a0]: Allocating epoch #1
GNUTLS: HSK[0x563eb42050a0]: Adv. version: 3.3
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_256_GCM_SHA384 (C0.2C)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_CAMELLIA_256_GCM_SHA384 (C0.87)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_CHACHA20_POLY1305 (CC.A9)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_256_CCM (C0.AD)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_256_CBC_SHA1 (C0.0A)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_128_GCM_SHA256 (C0.2B)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_CAMELLIA_128_GCM_SHA256 (C0.86)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_128_CCM (C0.AC)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_128_CBC_SHA1 (C0.09)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_3DES_EDE_CBC_SHA1 (C0.08)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_AES_256_GCM_SHA384 (C0.30)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_CAMELLIA_256_GCM_SHA384 (C0.8B)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_CHACHA20_POLY1305 (CC.A8)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_AES_256_CBC_SHA1 (C0.14)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_AES_128_GCM_SHA256 (C0.2F)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_CAMELLIA_128_GCM_SHA256 (C0.8A)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_AES_128_CBC_SHA1 (C0.13)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_3DES_EDE_CBC_SHA1 (C0.12)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_256_GCM_SHA384 (00.9D)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_CAMELLIA_256_GCM_SHA384 (C0.7B)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_256_CCM (C0.9D)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_256_CBC_SHA1 (00.35)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_CAMELLIA_256_CBC_SHA1 (00.84)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_128_GCM_SHA256 (00.9C)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_CAMELLIA_128_GCM_SHA256 (C0.7A)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_128_CCM (C0.9C)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_128_CBC_SHA1 (00.2F)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_CAMELLIA_128_CBC_SHA1 (00.41)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_3DES_EDE_CBC_SHA1 (00.0A)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_256_GCM_SHA384 (00.9F)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CAMELLIA_256_GCM_SHA384 (C0.7D)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CHACHA20_POLY1305 (CC.AA)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_256_CCM (C0.9F)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_256_CBC_SHA1 (00.39)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CAMELLIA_256_CBC_SHA1 (00.88)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_128_GCM_SHA256 (00.9E)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CAMELLIA_128_GCM_SHA256 (C0.7C)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_128_CCM (C0.9E)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_128_CBC_SHA1 (00.33)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CAMELLIA_128_CBC_SHA1 (00.45)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_3DES_EDE_CBC_SHA1 (00.16)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Extended Master Secret (0 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Encrypt-then-MAC (0 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension OCSP Status Request (5 bytes)
GNUTLS: HSK[0x563eb42050a0]: sent server name: '172.17.0.2'
GNUTLS: EXT[0x563eb42050a0]: Sending extension Server Name Indication (15 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Safe Renegotiation (1 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Session Ticket (0 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Supported curves (12 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Supported ECC Point Formats (2 bytes)
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (4.1) RSA-SHA256
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (4.3) ECDSA-SHA256
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (5.1) RSA-SHA384
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (5.3) ECDSA-SHA384
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (6.1) RSA-SHA512
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (6.3) ECDSA-SHA512
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (3.1) RSA-SHA224
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (3.3) ECDSA-SHA224
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (2.1) RSA-SHA1
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (2.3) ECDSA-SHA1
GNUTLS: EXT[0x563eb42050a0]: Sending extension Signature Algorithms (22 bytes)
GNUTLS: HSK[0x563eb42050a0]: CLIENT HELLO was queued [220 bytes]
GNUTLS: REC[0x563eb42050a0]: Preparing Packet Handshake(22) with length: 220 and min pad: 0
GNUTLS: ENC[0x563eb42050a0]: cipher: NULL, MAC: MAC-NULL, Epoch: 0
GNUTLS: REC[0x563eb42050a0]: Sent Packet[1] Handshake(22) in epoch 0 and length: 225
GNUTLS: ASSERT: buffers.c[get_last_packet]:1160
GNUTLS: ASSERT: buffers.c[_gnutls_io_read_buffered]:587
GNUTLS: ASSERT: buffers.c[get_last_packet]:1160
GNUTLS: ASSERT: buffers.c[_gnutls_io_read_buffered]:587
GNUTLS: ASSERT: buffers.c[get_last_packet]:1160
GNUTLS: REC[0x563eb42050a0]: SSL 48.48 Unknown Packet packet received. Epoch 0, length: 8271
GNUTLS: ASSERT: record.c[check_recv_type]:580
GNUTLS: Received record packet of unknown type 53
GNUTLS: ASSERT: record.c[recv_headers]:1093
GNUTLS: ASSERT: record.c[_gnutls_recv_in_buffers]:1175
GNUTLS: ASSERT: buffers.c[_gnutls_handshake_io_recv_int]:1415
GNUTLS: ASSERT: handshake.c[_gnutls_recv_handshake]:1474
GNUTLS: ASSERT: handshake.c[handshake_client]:2841
**** gnutls_handshake: An unexpected TLS packet was received.
GNUTLS: REC[0x563eb42050a0]: Start of epoch cleanup
GNUTLS: REC[0x563eb42050a0]: End of epoch cleanup
GNUTLS: REC[0x563eb42050a0]: Epoch #0 freed
GNUTLS: REC[0x563eb42050a0]: Epoch #1 freed
---- Closing control socket
ls: Fatal error: gnutls_handshake: An unexpected TLS packet was received.
lftp ftp42@172.17.0.2:~> exit
2. Using lftp with ftp://
If I remove the s from ftp in the lftp command above, I get :
➜ repo git:(master) ✗ lftp -u user,password ftp://172.17.0.2 -p 31581
lftp ftp42@172.17.0.2:~> set ssl:verify-certificate false
lftp ftp42@172.17.0.2:~> ls
Interrupt
lftp ftp42@172.17.0.2:/> debug 9
lftp ftp42@172.17.0.2:/> ls
---- Connecting to 172.17.0.2 (172.17.0.2) port 31581
<--- 220 (vsFTPd 3.0.3)
---> FEAT
<--- 211-Features:
<--- AUTH TLS
<--- EPRT
<--- EPSV
<--- MDTM
<--- PASV
<--- PBSZ
<--- PROT
<--- REST STREAM
<--- SIZE
<--- TVFS
<--- UTF8
<--- 211 End
---> AUTH TLS
<--- 234 Proceed with negotiation.
---> OPTS UTF8 ON
Certificate: C=FR,ST=PARIS,L=Paris,O=42_Corp,OU=user,CN=email
Issued by: C=FR,ST=PARIS,L=Paris,O=42_Corp,OU=user,CN=email
WARNING: Certificate verification: Not trusted (28:51:50:4B:83:40:96:E7:2C:9A:1E:9B:38:C5:25:46:B4:CC:4C:24)
WARNING: Certificate verification: certificate common name doesn't match requested host name ‘172.17.0.2’ (28:51:50:4B:83:40:96:E7:2C:9A:1E:9B:38:C5:25:46:B4:CC:4C:24)
<--- 200 Always in UTF8 mode.
---> USER ftp42
<--- 331 Please specify the password.
---> PASS 42
<--- 230 Login successful.
---> PBSZ 0
<--- 200 PBSZ set to 0.
---> PROT P
<--- 200 PROT now Private.
---> PASV
<--- 227 Entering Passive Mode (172,18,0,6,194,222).
---- Connecting data socket to (172.18.0.6) port 49886
Interrupt
---> LIST
---> ABOR
---- Closing aborted data socket
lftp ftp42@172.17.0.2:/>
<--- 425 Failed to establish connection.
<--- 225 No transfer to ABOR.
lftp ftp42@172.17.0.2:/>
TO SUM UP
I guess my issue is in the ssl configuration, but where ?... What am I missing ? Thanks for reading this hidiously long post :)
