In noob speak, Dispose() is about cleaning up after you're done using an unmanaged resource.
What's an unmanaged resource? It's all the stuff that CLR doesn't manage for you. They're thing like file handles, database connections, network sockets, GDI+ pens, etc. You get access to those things through a typical .NET object, but it will implement IDisposable, to allow you to clean up properly.
Why clean up? Until you've cleaned up after yourself, that resource isn't available by other parts of the program for use. In that respect, you're breaking things, because you're hogging a resource.
Why do this yourself? You should do this yourself as soon as you stop needing the resource, rather than rely on the auto-magic of the garbage collector, because it could take a long (well, unspecified) amount of time before the garbage collector gets to it. Until an object has been disposed of properly, you can't reuse the underlying resource, so you program will not function reliably.