2

Background:

I am trying to run the Jepsen (https://github.com/jepsen-io/jepsen) on Ubuntu 16.04. I have created 6 virtual machines (on my windows 10 pc) and replaced the host name by n1, n2, .... n6. The user name and password is same for all VMs.

Now, from any VM, I can directly ssh to any other VM. ( For example, from n6, I can just execute this command "ssh n1" and it works without any problem). I am trying to run the node jepsen tool from node 6 ( hostname n6 ) so that it can create the db in node n1,n2,n3,n4 and n5.

So far I have not made any changes in the jepsen code. I have compiled it using "lein install" command and now trying to run it.

Problem:

To simply run the test, I followed the instruction provided in the jepsen's webpage ( "cd aerospike"; "lein test" ) But it is giving me the following error:

user@Ubuntu6:~/Desktop/jepsen_cassandra/aerospike$ lein test

lein test aerospike.core-test
jepsen.generator$concat$reify__7285@577dac16>

lein test :only aerospike.core-test/counter

ERROR in (counter) (Session.java:781)
Uncaught exception, not in assertion.
expected: nil
actual: com.jcraft.jsch.JSchException: reject HostKey: n1 at com.jcraft.jsch.Session.checkHost (Session.java:781)
com.jcraft.jsch.Session.connect (Session.java:342)
com.jcraft.jsch.Session.connect (Session.java:183)
clj_ssh.ssh$eval6796$fn__6803.invoke (ssh.clj:118)
clj_ssh.ssh.protocols$eval6722$fn__6745$G__6713__6754.invoke (protocols.clj:4)
clj_ssh.ssh$connect.invoke (ssh.clj:401)
jepsen.control$session.invoke (control.clj:197)
clojure.lang.AFn.applyToHelper (AFn.java:154)
clojure.lang.AFn.applyTo (AFn.java:144)
clojure.core$apply.invoke (core.clj:624)
clojure.core$with_bindings_STAR_.doInvoke (core.clj:1862)
clojure.lang.RestFn.applyTo (RestFn.java:142)
clojure.core$apply.invoke (core.clj:628)
clojure.core$bound_fn_STAR_$fn__4140.doInvoke (core.clj:1884)
clojure.lang.RestFn.applyTo (RestFn.java:137)
clojure.core$apply.invoke (core.clj:624)
jepsen.core$fcatch$wrapper__8355.doInvoke (core.clj:55)
clojure.lang.RestFn.invoke (RestFn.java:408)
clojure.core$pmap$fn__6328$fn__6329.invoke (core.clj:6466)
clojure.core$binding_conveyor_fn$fn__4145.invoke (core.clj:1910)
clojure.lang.AFn.call (AFn.java:18)
java.util.concurrent.FutureTask.run (FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
java.lang.Thread.run (Thread.java:748)
jepsen.generator$concat$reify__7285@254210b1>

lein test :only aerospike.core-test/cas-register

ERROR in (cas-register) (Session.java:781)
Uncaught exception, not in assertion.
expected: nil
actual: com.jcraft.jsch.JSchException: reject HostKey: n1
at com.jcraft.jsch.Session.checkHost (Session.java:781)
com.jcraft.jsch.Session.connect (Session.java:342)
com.jcraft.jsch.Session.connect (Session.java:183)
clj_ssh.ssh$eval6796$fn__6803.invoke (ssh.clj:118)
clj_ssh.ssh.protocols$eval6722$fn__6745$G__6713__6754.invoke (protocols.clj:4)
clj_ssh.ssh$connect.invoke (ssh.clj:401)
jepsen.control$session.invoke (control.clj:197)
clojure.lang.AFn.applyToHelper (AFn.java:154)
clojure.lang.AFn.applyTo (AFn.java:144)
clojure.core$apply.invoke (core.clj:624)
clojure.core$with_bindings_STAR_.doInvoke (core.clj:1862)
clojure.lang.RestFn.applyTo (RestFn.java:142)
clojure.core$apply.invoke (core.clj:628)
clojure.core$bound_fn_STAR_$fn__4140.doInvoke (core.clj:1884)
clojure.lang.RestFn.applyTo (RestFn.java:137)
clojure.core$apply.invoke (core.clj:624)
jepsen.core$fcatch$wrapper__8355.doInvoke (core.clj:55)
clojure.lang.RestFn.invoke (RestFn.java:408)
clojure.core$pmap$fn__6328$fn__6329.invoke (core.clj:6466)
clojure.core$binding_conveyor_fn$fn__4145.invoke (core.clj:1910)
clojure.lang.AFn.call (AFn.java:18)
java.util.concurrent.FutureTask.run (FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
java.lang.Thread.run (Thread.java:748)

Ran 2 tests containing 2 assertions.
0 failures, 2 errors.
Tests failed.

First I tried to run the code as it is. But later I have added the username and password of the 5 VMs ( that is required to ssh ) in Line 21 and 22 of ( https://github.com/jepsen-io/jepsen/blob/master/jepsen/src/jepsen/control.clj ). But it did not solve the problem.

I will highly appreciate any solution/guideline to solve this problem.

Community
  • 1
  • 1
Pallab
  • 23
  • 3
  • Welcome to SO, and thanks for the well formed clear first question. I'd like to edit the question to remove the Ubuntu specific title since there is not much ubuntu specific about the question – Arthur Ulfeldt Jun 28 '17 at 23:18
  • 1
    Hi @ArthurUlfeldt, thanks for the suggestion. As per your suggestion, I have modified the title. – Pallab Jun 29 '17 at 17:09

1 Answers1

1

this line

actual: com.jcraft.jsch.JSchException: reject HostKey: n1 at com.jcraft.jsch.Session.checkHost

indicates that the system initiating the ssh connection is unwilling to trust the identity of the computer it's connecting to. So it's not a problem with passwords/keys not allowing you in, rather it's the opposite, the system starting the connection is not sure of the identity of the system it's connecting to.

The most common cause is that you have connected to a system in the past at the same IP or hostname which has since been replaced with this system. SSH guarantees that if a different computer responds to the same name, it will set off the alarms and not connect rather than risk a man-in-the-middle attack. This results in false positives when a system really has been replaced.

Try sshing from this system to the target host and make sure it work. if not try removing that system's IP from ~/.ssh/known_hosts if it's there. also consider if anything in your ~/.ssh/config file would be affecting this host, perhaps by requiring it to have a specific host key for instance

since it doesn't seem to be a host key conflict the next thing to check is what kind of host key the hosts are using. Jsch is it's own implementation of ssh that supports a different set of keys than openSSH. you can determine the system's host key type by running

ssh -vvv hostname

...
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256

then google for jsch and that host key algorithm and see if there are any open issues on it.

Arthur Ulfeldt
  • 90,827
  • 27
  • 201
  • 284
  • Hi Arthur, thanks for the answer. I have already tried sshing from the system to the target host ( all of the 5 hosts, n1, n2... n5) and they are working without any issue. But when the Jepsen program is trying to establish a connection, it is throwing this error. As per your suggestion I have removed system's "known_hosts" file. But that did not help. Surprisingly, I haven't found any "config" file under "~/.ssh" folder. The only files available are : 1) id_rsa, 2) id_rsa.pub and 3) known_hosts. – Pallab Jun 29 '17 at 17:12
  • the config file is not present if everything is default. – Arthur Ulfeldt Jun 29 '17 at 17:39
  • @birdspider :: Hi, thanks for pointing it out. It resolves the ssh related issue. Somehow I missed the most important part :( – Pallab Jun 29 '17 at 17:41
  • ssh-keyscan -t rsa your_target_host1 >> ~/.ssh/known_hosts – Roman Ivanov Nov 28 '17 at 18:45
  • "ssh-keyscan -t rsa your_target_host1 >> ~/.ssh/known_hosts" helped to resolve problem. Without rsa key such command in ~/.ssh/known_hosts it did not work. Even I could connect to server by ssh, as in knonw_host was "ecdsa-sha2-nistp256" key for such host. So workaround is to remove non rsa key and generate rsa key. – Roman Ivanov Nov 28 '17 at 18:52