Simply put, it tells the OS which part of the IP address refers to the network, and which part refers to a host on that network. If the host needs to send data to an IP address where the network portion of the address differs, then it routes that data through whatever the default gateway/router address happens to be.
So if your subnet mask is 255.255.255.0, then the first three bytes of your IP address are the network, and the fourth byte refers to hosts within that network. If any of those first three bytes are different, the host will send the data through its default gateway (or fail if one isn't configured).