Chances are, you didn't get the whole thing out. But you probably got enough of it out so that when/if it does re-grow, it will be a much smaller plant.
The only way to make sure the entire thing is dead is to spray it with something that will get absorbed into the plant and kill it. But that's not nearly as satisfying as ripping them out.
Edit: One suggestion is to spray a bit of weed killer or vinegar into the hole created by most weed removal tools, which will kill the entire plant without affecting the surrounding grass.
If you do leave them in, the leaves will eventually kill grass under them.