Service Fabric local development cluster with 5 nodes runs less instances and less partitions than expected

Question

I am running a Service Fabric application on a local development cluster with 5 nodes "simulated" on my PC.

The application has a public API stateless service with instance count set to -1.

I expect to see 5 instances of the stateless service in Service Fabric Explorer but I only see 1.

The application also has an actor service with partition count set to 10 (auto generated configuration by Visual Studio).

When the application is deployed to the development cluster on my PC only one partition can be seen in the Service Fabric Explorer. After I simulate a "big" load and the CPU and the Memory usage of my PC are around and above 90% there is still only one partition of the actor service. I made a stateful service with partition count set to 5 to check if there is something wrong with my environment, but it runs as expected.

Is this normal for stateless services or there is something wrong with my configuration. Is this behavior specific for development cluster, set to avoid things like port conflict.

What about the actor service. According to the docs Dynamic partition scaling is possible, but the number of partitions for the actor service does not increase even during high load. In addition there is nothing mentioned for dynamic partition scaling in the Actor docs.

EDIT: After some tests with different configurations I got it working.

Original configuration in ApplicaitonManifest.xml:

<Parameters>
   ...
    <Parameter Name="HttpAPI_InstanceCount" DefaultValue="-1" />

    <Parameter Name="SystemStatusConsumerActorService_PartitionCount" 
               DefaultValue="10" />
   ...
</Parameters>

<DefaultServices>
    <Service Name="HttpAPI">
      <StatelessService ServiceTypeName="HttpAPIType" 
                        InstanceCount="[HttpAPI_InstanceCount]">
        <SingletonPartition />
      </StatelessService>
    </Service>

    <Service Name="SystemStatusConsumerActorService" 
             GeneratedIdRef="faad4d24-04db-4e06-8a1d-22bc6255c7fe|Persisted">

      <StatefulService ServiceTypeName="SystemStatusConsumerActorServiceType" TargetReplicaSetSize="SystemStatusConsumerActorService_TargetReplicaSetSize]" MinReplicaSetSize="[SystemStatusConsumerActorService_MinReplicaSetSize]">

        <UniformInt64Partition 
          PartitionCount="[SystemStatusConsumerActorService_PartitionCount]" 
          LowKey="-9223372036854775808" 
          HighKey="9223372036854775807" />
      </StatefulService>
    </Service>
</DefaultServices>

The configuration that works:

<Parameters>
   ...
    <Parameter Name="HttpAPIInstanceCount" DefaultValue="-1" />

    <Parameter Name="SystemStatusConsumerActorServicePartitionCount" 
               DefaultValue="10" />
   ...
</Parameters>

<DefaultServices>
    <Service Name="HttpAPI">
      <StatelessService ServiceTypeName="HttpAPIType" 
                        InstanceCount="[HttpAPIInstanceCount]">
        <SingletonPartition />
      </StatelessService>
    </Service>

    <Service Name="SystemStatusConsumerActorService" 
             GeneratedIdRef="faad4d24-04db-4e06-8a1d-22bc6255c7fe|Persisted">

      <StatefulService ServiceTypeName="SystemStatusConsumerActorServiceType" TargetReplicaSetSize="SystemStatusConsumerActorService_TargetReplicaSetSize]" MinReplicaSetSize="[SystemStatusConsumerActorService_MinReplicaSetSize]">

        <UniformInt64Partition 
          PartitionCount="[SystemStatusConsumerActorServicePartitionCount]" 
          LowKey="-9223372036854775808" 
          HighKey="9223372036854775807" />
      </StatefulService>
    </Service>
</DefaultServices>

Notice that the only differences are the parameter names:

HttpAPI_InstanceCount

changed to

HttpAPIInstanceCount

SystemStatusConsumerActorService_PartitionCount

changed to

SystemStatusConsumerActorServicePartitionCount

This is normal. Stateless and actor service instances on local dev clusters will always be one instance. For stateless, this happens because of port conflicts. You only define 1 port for stateless and fabric cannot open 5 instances with a single port number. For actors, fabric internally has only 1 app domain[exe] for multiple partitions. This is to enable fast activation of actors. You can however, resolve individual actor partitions with "new ActorId()". — teeboy, Nov 27 '18 at 09:13
@teeboy There is no strict requirement for stateless service to have endpoint port defined. You can remove the port value and Service Fabric will assign ports automatically - so you would have multiple instances of stateless service. — Oleg Karasik, Nov 27 '18 at 12:14

score 2 · Answer 1 · answered Nov 27 '18 at 09:30

2

After trying a lot of different configurations there was still a problem. I found it after checking the git diff of my project.

My problem was that the ApplicationManifest parameters are overridden in the ApplicatonParameters\Local.5Node.xml (because I use 5 node local cluster) file in the service fabric application folder.

The tricky part was that even if I delete or comment an override of SystemStatusConsumerActorService_PartitionCount the studio adds one every time I build the application. The only solution to this was changing the name of the parameter in the ApplicationManifest.xml .

After changing the configuration according to the new facts the stateless service and the actor service start with the desired amount of instances and partitions respectively.

Of course 4 of the five stateless instances break but this is totally logical, considering the cluster "runs" on one machine.

answered Nov 27 '18 at 09:30

Zapo

41
6

You can remove the port specification from the service declaration in `ApplicationManifest.xml` to allow service fabric to allocate port for you (so you would be able to have multiple running instances). – Oleg Karasik Nov 27 '18 at 12:12
One more question - have you tried to change values of parameters in `Local.5Node.xml`? – Oleg Karasik Nov 27 '18 at 12:15
Yes I changed the values in the Local.5Node.xml file and it also works. Also I removed the ports from the service configuration file. Now I reach the service through the reverse proxy, which also acts like load balancer or at least redirects the requests to a random instance of a stateless service. – Zapo Nov 27 '18 at 13:45
Yes this is correct. For stateless service reverse proxy will redirect requests to a random instance in case of stateful service by default reverse proxy will redirect all requests to primary replica. – Oleg Karasik Nov 27 '18 at 13:46

Service Fabric local development cluster with 5 nodes runs less instances and less partitions than expected

1 Answers1