I have a similar problem with cats in my yard. I've read a number of things that can help, but first I'll explain what does help for me (I'm not sure about for raccoons, though):
- Grow thick plants that are at least a couple feet tall, wherein the cats cannot step anywhere without stepping on 2'+ tall plants. I'm not sure exactly why this deters cats, since they could just walk all over and through the plants like a jungle creature, but it prevents cats from entering the main body of our garden in the backyard. They don't look any more interested in walking through them than your average person might. Some example plants include such as sprawling tomatoes, melons and watermelon. They do need to be growing thick, though (no dirt visible).
A more impenetrable, taller hedge that may even deter raccoons (and will survive somewhat colder temperatures) is Morelle De Balbis. They have thorns and they grow nice little white flowers (and edible red fruits). However, how many plants you grow per spot matters a lot. You can sprout a bunch of plants in one container, and transplant them all in one spot (one spot per container of plants); then you'll get a much thicker hedge. Plant each container of plants 1 to 2 feet apart. I tried that, this year (not as a hedge, though), and they grow and fruit just fine like that. They look like they'd make a great hedge that way. With only one plant per spot, they don't look at all like a hedge in my yard. I have lots of seeds if you need some.
I haven't looked into perennial plants suitable for colder zones that might work all year round. Also, it will definitely make the yard more inaccessible to humans, too. For me, I still walk through the plants (but no one else does without encouragement, as far as I know). I've seen a cat just staring longingly at the plot of plants as if she really wanted to go in, but she wouldn't go in, even if she saw me do it.
Getting the ground wet can sometimes deter cats for a while.
I've tried putting Cayenne pepper in the soil, and it seemed to work, for that year, but I really don't know if it was the reason the cats stopped there.
Snow. Cats don't like to poop in snow. They'll still poop at the sides of your house that are snow-free.
Okay, I've heard that putting plastic forks or something like that in the ground or tin foil on the ground can deter some cats. I'm not sure about raccoons. I've heard cats don't like citrus.
You might like trying laying down some astroturf, black plastic or such. Then they won't be able to poop through that, and I don't recall ever seeing cats poop on black plastic in our yard.
You might try mulching. I haven't seen cats poop in our bark mulch.
Growing any kind of plants (even if they're short or soft enough for cats to walk on) should reduce the amount of poop you get there. Cats like to dig, and it's hard to dig through plants.
Cats also don't seem to like pooping in compact, clay soil (they can't dig very easily in it). You do need to make sure that it is compacted, however.