Just want to know if I can configure multiple nodes from different data centers on the same physical server. Example - Want to have 2 data centers with 3 nodes each. 1 node from each data center will be on each server.
Total of 2 data centers, 6 nodes on 3 physical servers.
You can technically configure it as you describe; however, DataCenter is typically thought of as a location, so having nodes in two locations but configured as a datacenter is confusing (especially for anyone who would have to troubleshoot the environment later).
A best practice would be to have the topology of 3 nodes in each data center (actually be physically located in each data center). Then you could configure the cluster to have your data in both data centers for availability and also have appropriate latency within a single data center for all reads, writes, etc...
For example, using RF: 3 in each data center and then Using a consistency of LOCAL_QUORUM would balance data availability while reducing latency of your request. This example configuration would ensure the read/write occurs in a single data center (lower latency than across datacenters) but ensures the data is saved across both data centers (eventually consistent design).
Yes it is possible to follow the topology you have listed but think about the following scenario
With two nodes from different DC on single machine, there is high chance that you will have the unit of data replicated on a single machine in two different data center nodes. If the single machine fails you would loose two copies of a piece of data.
Assuming you have RF of DC1:2 DC2:2 and using a CF of Quorum, you would need 3 nodes to respond to read requests. With one physical server being down a unit of data will be loosing 2 replicas and your reads will fail and indeed the writes with same CF will also fail.
