6

I am having trouble connecting to eventstore cluster (with 3 nodes). So I have a 3 nodes each running on Azure VM and below is the code to connect and append streams

var settings = EventStoreClientSettings.Create("esdb://admin:changeit@{node1IP}:2113,{node2IP}:2113,{node3IP}:2113?tls=true&tlsVerifyCert=false&keepAliveTimeout=10000&keepAliveInterval=10000");

var client = new EventStoreClient(settings);
                
var evt = new 
{
    EntityId = Guid.NewGuid().ToString("N"),
    mportantData = "I wrote my first event!"
};
                
var eventData = new EventData(
    uid.NewUuid(),
    "TestEvent",
    JsonSerializer.SerializeToUtf8Bytes(evt)
);
                
await client.AppendToStreamAsync(
    "some-stream",
    StreamRevision.None, 
    new[] { eventData });
                
var result = client.ReadStreamAsync(
    Direction.Forwards,
    "some-stream",
    StreamPosition.Start);
                
var events = await result.ToListAsync();
foreach (var eventIn in events)
{
    Console.WriteLine($"event: {eventIn.ToString()}\n");
}

Error I am getting is DiscoveryException: Failed to discover candidate in 10 attempts.

I can telnet the address from my local machine and seems to the problem is only with gRPC connection.

---
# Paths
Db: /var/lib/eventstore
Index: /var/lib/eventstore/index
Log: /var/log/eventstore
LogLevel: Information
# LogFileInterval: 62
LogFileRetentionCount: 100

SkipDbVerify: True
SkipIndexVerify: True
ChunksCacheSize: 1073741824
CachedChunks: 4

# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/eventstore/certs/ca

# Network configuration
## TCP/IP Settings
IntIp: 10.3.0.4
ExtIp: 10.3.0.4
IntTcpPort: 1112
ExtTcpPort: 1113
IntTcpHeartbeatInterval: 5000
IntTcpHeartbeatTimeout: 2000
ExtTcpHeartbeatInterval: 5000
ExtTcpHeartbeatTimeout: 2000

## HTTP Settings
HttpPort: 2113
HttpPortAdvertiseAs: 2113

# Cluster gossip
ClusterSize: 3
DiscoverViaDns: false
GossipSeed: 10.3.0.5:2113,10.3.0.6:2113

# Projections configuration
RunProjections: All
StartStandardProjections: True

# Misc Settings
EnableExternalTcp: True
EnableAtomPubOverHTTP: True
---

server config for one of the node, other 2 looks pretty much same except for the ip changes.

enter image description here

Grpc.Core.RpcException: Status(StatusCode="DeadlineExceeded", Detail="")
   at EventStore.Client.Interceptors.TypedExceptionInterceptor.<AsyncUnaryCall>b__5_0[TRequest,TResponse](Task`1 t)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at EventStore.Client.GrpcServerCapabilitiesClient.GetAsync(CallInvoker callInvoker, CancellationToken cancellationToken)
   at EventStore.Client.EventStoreClientBase.GetChannelInfoExpensive(ReconnectionRequired reconnectionRequired, Action`1 onReconnectionRequired, IChannelSelector channelSelector, CancellationToken cancellationToken)
   at EventStore.Client.SharingProvider`2.FillBoxAsync(TaskCompletionSource`1 box, TInput input)
   at EventStore.Client.TaskExtensions.WithCancellation[T](Task`1 task, CancellationToken cancellationToken)
   at EventStore.Client.EventStoreClientBase.GetChannelInfo(CancellationToken cancellationToken)
   at EventStore.Client.EventStoreClient.<CreateStreamAppender>g__GetCall|14_0()
   at EventStore.Client.EventStoreClient.StreamAppender.IsUsable()
   at EventStore.Client.EventStoreClient.AppendToStreamAsync(String streamName, StreamRevision expectedRevision, IEnumerable`1 eventData, Action`1 configureOperationOptions, Nullable`1 deadline, UserCredentials userCredentials, CancellationToken cancellationToken)
   at TestClusterConnection.Program.Main() in /Users/krishna/Desktop/TestClusterConnection/TestClusterConnection/Program.cs:line 97
Unhandled exception. Grpc.Core.RpcException: Status(StatusCode="DeadlineExceeded", Detail="")
   at EventStore.Client.Interceptors.TypedExceptionInterceptor.<AsyncUnaryCall>b__5_0[TRequest,TResponse](Task`1 t)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at EventStore.Client.GrpcServerCapabilitiesClient.GetAsync(CallInvoker callInvoker, CancellationToken cancellationToken)
   at EventStore.Client.EventStoreClientBase.GetChannelInfoExpensive(ReconnectionRequired reconnectionRequired, Action`1 onReconnectionRequired, IChannelSelector channelSelector, CancellationToken cancellationToken)
   at EventStore.Client.SharingProvider`2.FillBoxAsync(TaskCompletionSource`1 box, TInput input)
   at EventStore.Client.TaskExtensions.WithCancellation[T](Task`1 task, CancellationToken cancellationToken)
   at EventStore.Client.EventStoreClientBase.GetChannelInfo(CancellationToken cancellationToken)
   at EventStore.Client.EventStoreClient.<CreateStreamAppender>g__GetCall|14_0()
   at EventStore.Client.EventStoreClient.StreamAppender.IsUsable()
   at EventStore.Client.EventStoreClient.AppendToStreamAsync(String streamName, StreamRevision expectedRevision, IEnumerable`1 eventData, Action`1 configureOperationOptions, Nullable`1 deadline, UserCredentials userCredentials, CancellationToken cancellationToken)
   at TestClusterConnection.Program.Main() in /Users/krishna/Desktop/TestClusterConnection/TestClusterConnection/Program.cs:line 97
   at TestClusterConnection.Program.<Main>()

Error logs I can see in the console...

Krishna
  • 61
  • 3
  • You are connecting to a secure cluster, what's your server config (one node is enough) and certificate CN and SANs? – Alexey Zimarev Aug 14 '22 at 15:00
  • @AlexeyZimarev added the config. – Krishna Aug 14 '22 at 19:09
  • Where does you run your code from ? Is it from a machine in the same ip range ? (I.e same or other network) can you perform a curl from that machine where that program is run from & give the output ? curl -i -k -u admin:changeit https://10.3.0.4:2113/web/index.html curl -i -k -u admin:changeit https://10.3.0.5:2113/web/index.html curl -i -k -u admin:changeit https://10.3.0.6:2113/web/index.html – ylorph Aug 14 '22 at 19:29
  • @ylorph Tried to run from local machine and the error is same as above and also tried to run from a VM in the same azure network and still the same error. Tried the curl you provided and the result is `curl: (28) Failed to connect to 10.3.0.4 port 2113 after 75007 ms: Operation timed out` (because the IP is azure private ip). Tried the curl with external ip as well and the result is `curl: (52) Empty reply from server` – Krishna Aug 14 '22 at 19:36
  • * "Local Machine" what do you mean ? * "WM in same azure network" : that's odd. * Are the nodes able to see each others ? any errors in the logs `/var/log/eventstore` ? – ylorph Aug 14 '22 at 19:55
  • @ylorph Local machine means my personal computer, eventstore is running on the azure vms. Nodes able to gossip each other. See the newly attached picutre above. – Krishna Aug 14 '22 at 20:04
  • I am confused as the logs show that you connect via TCP, but the code you show uses the gRPC client. Even the logger context name is `EventStoreConnection` which corresponds with the TCP client. Could you please clarify that? – Alexey Zimarev Aug 15 '22 at 08:50
  • Just to make 100% sure: the following line: `var settings = EventStoreClientSettings.Create("esdb://admin:changeit@{node1IP}:2113,{node2IP}:2113,{node3IP}:2113?tls=true&tlsVerifyCert=false&keepAliveTimeout=10000&keepAliveInterval=10000");` you did repleace {node1IP}, {node2IP}, {node3IP} with the corresponding IP addresses ? – ylorph Aug 15 '22 at 11:07
  • @AlexeyZimarev sorry for the confusion, its my bad to put the wrong console logs, was trying different kinds of connections and none of them are working. Updated the logs now. – Krishna Aug 15 '22 at 13:21
  • @ylorph Updated the logs now – Krishna Aug 15 '22 at 13:21
  • @AlexeyZimarev FYI I have added the server cert (ca.crt) to the keychain and set that to `Trust Always`. I have a string feeling that it must be wrong with either client or with the network. – Krishna Aug 15 '22 at 13:26
  • It seems it's your local machine that can't connect to the vnet in azure. Where did you take the screenshot from ? – ylorph Aug 16 '22 at 11:03

0 Answers0