I tend to split the controllers up based on their responsibility to a specific section of a site/application. This makes maintaining the code so much easier. Furthermore, I group controllers (and views, models) within modules (folders). Here's an example from a current project I'm working on:
- Blog
- Posts
- Comments
- Categories
- Settings
The more complex a site, the more modules I use. Although most of my modules only contain one 'Index' controller, I do like the organization they provide.
Then I use a router (front controller) that maps a REST style URI to the proper module / controller / action. Ex: mysite.com/blog/posts/view/7 would call Controller_Posts::view(7) from the "blog" module. An added benefit of using modules is I can have more specific URIs than if I didn't have modules. Although I suppose that could be remedied by using a router that supports defining custom routes, but I'm not too fond of that.
As many other things, it boils down to what you're comfortable with as a developer, but we can probably agree that more organization you have, the better off you are, so long as you don't over complicate things.
As a quick aside, I would recommend you look into using a framework. I understand if you don't want to use one of the ones out there already, as I avoided those too. I ended up writing my own which for the past year has served me very well. It was a great learning experience and it only contains what I want / need. That being said, you might want to look into Kohana and CakePHP - they're not overly bloated IMO and they will definitely save you time should you decide not to write your own.