1

Has anyone created a script for connection testing sftp for multi users in separate threads?

I have an issue with JMeter and I'm trying to make an alternative to SSH sampler. I'm going to create a simple script to connect the SFTP and execute a "ls" command. Currently I try to use GNU parallel(Thanks @Barmar). This command works fine:

parallel < list.txt

I created a list.txt with a list of commands:

sftp -i /rsa_key user@sftp_url.com <<< ls
sftp -i /rsa_key2 user2@sftp_url.com <<< ls
...
sftp -i /rsa_keyX userX@sftp_url.com <<< ls

But still I'm limited by the number of my physical cores. Is it possible to make 100+ requests at the same time by different threads?

3oJIoTou
  • 23
  • 4
  • Bash doesn't have threads. You can run commands in the background with `&`, you can use GNU `parallel`. – Barmar Apr 28 '23 at 17:46
  • Thank you for your response. Not clear does it support SFTP. parallel -S '/rsa_key 'user@sftp_url.com echo custom ::: ssh doesn't work – 3oJIoTou Apr 28 '23 at 18:24
  • Parallel doesn't "support" anything in particular, it's a way to run multiple any command in multiple processes. – Barmar Apr 28 '23 at 19:14
  • Can I ask you to show an example with sftp connection? I have a list of users with the same ssh key. I used a JMeter before, but looks like it has issue with proxy. I'm new to linux. – 3oJIoTou Apr 28 '23 at 19:30
  • See examples here: https://www.gnu.org/software/parallel/parallel_examples.html – Barmar Apr 28 '23 at 19:32
  • I used the example above from there. A bit confusing and unclear, especially when new to Linux. Thank you for your help! – 3oJIoTou Apr 28 '23 at 19:41
  • If you have trouble getting it working, post what you tried and we'll help you fix it. We won't write it for you. – Barmar Apr 28 '23 at 19:42
  • @Barmar sorry for misunderstanding I got success to use: seq 10 | parallel -n0 sftp -i /rsa_key user@sftp_url.com Can I use a list of commands in the text file? Something like this: parallel < commands.txt and provide: sftp -i /rsa_key user@sftp_url.com & sftp -i /rsa_key user2@sftp_url.com & ... – 3oJIoTou Apr 28 '23 at 19:58
  • I don't use parallel myself, I can't provide detailed answers like that. – Barmar Apr 28 '23 at 20:00
  • But if the commands are all the same except for parameters, I don't see why you would do it that way. It already provides a way to substitute parameters into the command. – Barmar Apr 28 '23 at 20:01
  • 1
    Please click [edit] and update your question so it properly reflects what you are asking rather than adding a bunch of comments. You can't expect folk to wade through a whole story in the comments - Stack Overflow works best when questions are clear, concise and complete. If you have a list/array of things, show a representative array. If you want to run multiple command in parallel, show the first 3-4 so we can deduce the pattern. Thank you. – Mark Setchell Apr 28 '23 at 21:18
  • Mmm... it's still not very clear what you are trying to do. – Mark Setchell Apr 28 '23 at 21:52
  • What does `sftp user1@sftp.com` achieve? – Mark Setchell Apr 28 '23 at 21:59
  • @MarkSetchell sftp -i /rsa_key user@sftp_url.com to connect to sftp using ssh rsa-key and user name – 3oJIoTou Apr 28 '23 at 22:04
  • Ok, you connect and then what? – Mark Setchell Apr 28 '23 at 22:13
  • I just want to create multi connections in separate threads and use a simple command ls – 3oJIoTou Apr 28 '23 at 22:14
  • So please click [edit] and show the actual commands you want to run. – Mark Setchell Apr 28 '23 at 22:17

1 Answers1

0

Any reason you cannot do:

parallel -j100 < list.txt

or:

parallel --colsep ' ' -j100 'sftp -i {1} {2}@sftp_url.com <<< ls' < list.txt

with list.txt containing:

/rsa_key user
/rsa_keyX userX
Ole Tange
  • 31,768
  • 5
  • 86
  • 104
  • Thank you for your response! Looks like GNU parallel can't run more threads than physical cores in the system, at least with sftp. – 3oJIoTou May 19 '23 at 19:04
  • @3oJIoTou That sounds like a limitation in your system. It is not limited by GNU Parallel, and I can run 1500 simultaneous `sftp`s with no problem. – Ole Tange May 20 '23 at 00:35