For example, I have 3 NICs (1GB) and make a bonded NIC. Do I get a 3GB NIC?
Should I connect these 3 NICs into 1 switch? If one NIC dies, do I still have the network between server and switch alive?
Or should I connect each NIC to 3 different switches? So if one NIC or one switch dies, I still have the network alive?
Which mode should I use?
For example, I have 3 NICs(1GB) and make a bond NIC, Do I get a 3GB NIC?
No. You don't. The specific details depend on the hash mode, but typical hash modes distribute individual connections to each member. Even with a round-robin mode only possible between two directly connected Linux systems you will not get the full 3GB, probably more like 2GB.
Should I connect
No idea. We can't make a decision for you. You need to decide if you need fault tolerance or better performance. You need to figure out if your switches allow you to have a link aggregation group that spans switches, this is a pretty high end feature, that usually requires a stack setup.
If you have no idea what you should be doing, perhaps start with a switch-independent mode for fault tolerance only and connect it to multiple switches. This is a really simple setup and would require no config on your switches, but it wouldn't really give you much of a performance boost.
Past that, just try some modes and run some performance tests that reflect your typical load and usage of this system in various configurations. Pick the best one based on your results.
Yes, if a link fails, you will still have a connection.
As for the bandwidth question, when you LAG these connections together, remember that each individual flow of data will only traverse the link that the flow starts on. That one flow isn't shared among the other links.
It's also worth pointing out that you would actually be better off having either 2 links or 4, due to the hashing of the bits.
Explanation:
If the number of physical ports is not equal to power of 2 (not 2, 4 or 8), you will have unequal load-balance as well. For example, pick a case,
Under port-channel X, we have 3 interfaces being bundled gi8/8, gi9/8 and gi9/12. The load balance algorithm will need to use two bits from the hash.
Two bits will give 4 different patterns (00, 01, 10, and 11) and the patterns are assigned to the physical port. Therefore, the overall traffic is not 33% on each channel. It will be 50% on one and 25% on the other two links.
Say for a moment you have a file server with three 1Gbps NICs that is easily capable of saturating a 3Gbps link with any file, and a computer that also has three 1Gbps NICs and is capable of receiving and processing 3Gbps worth of traffic. You set up the three 1Gbps links on each end into a single 3Gbps trunk directly between the file server and your computer, and then try to transfer a file.
My understanding is you will be limited to 1Gbps for this file transfer, because even though the link can do 3Gpbs, it's still three separate 1Gbps links bonded. But now say you try to download 3 files. You'll find that you can get each file at 1Gbps, for 3Gbps total throughput.
Of course, this is a contrived scenario. What's much more common is you have network switches in multiple locations. These switches support many users that all have 1Gbps network ports in their computers, and a few servers that may have multiple nics, or even 10Gbps or higher network connections. What you may find is that 1Gbps is not enough throughput between switches. Individual users can't take more than the 1Gbps link between switches, but in aggregate a number of users can easily saturate that link. And it might be very costly to get faster single links between switches.
So what do you do? It turns out it's very common that the link between these switches is fiber, rather than copper (though copper can work here, too), and that multiple pairs of fiber (in a single cable) were pulled between the switch closets for redundancy; some of those pairs are currently sitting dark. So you get a few extra SFPs, pop them in some free ports on each switch, set up an LACP trunk, and in this way get higher throughput between the switches.
Now it doesn't matter that individual sessions (and endpoints) are still limited to 1Gbps, because you don't really need one person to use more than 1Gbps. This is about supporting a whole set of users at the higher speeds.
Additionally, you need to be careful how you set up the link. If you do things wrong (ie, you've never turned on spanning tree), it's easy to create a routing loop when you plug in the new link, and that can give you a real bad day in a hurry.
A final note here is that LACP links are resilient to one member of the link going down, so you can also use this a way to help protect against outages. Typically, the wiring is using the same cable path, and an event that breaches one set of wires will breach the other as well. But sometimes you get lucky.
You can also connect to different switches, to have redundant cable paths. This costs more (you need cable pulls between each of the switches), and in this case it's even more important to be sure you understand spanning tree, as well as the specific spanning tree types supported by your switches. Not every type of spanning tree allows you take much advantage of the redundant links in terms of increased performance, especially in older or cheaper equipment, and will instead simply disable the link until an outage in a different link changes the tree topology. But the right setup here on the right equipment can use these links not only for redundancy but also for increased throughput.
The best opgion for you is to get a switch that supports link aggregation : the switch will handle the 3 NICs as only one.
Depending on your switch, you'll propably have to setup the failover, or you'll loose connection if one cable is unplugged. It'd probably reconnect as a normal 1 GB NIC (2nd one won't be used.
It is hard to list the behavior of the network, because it depends on both hardware and software settings.
