I think the total shade is marginally ok (rhodies do prefer some sun, though). Type of shade matters, too - tree shade is marginally brighter than wall shade, which seems to be what you have.
What's certain is that it's planted way too close to the fence. PJMs can get four feet wide, so it should be at least 2.5 feet from each side of the fence to account for mature width. This would also give it more (indirect?) light, since it would be further from the fence.
Rhodies require acidic soil as well, and planting it near the concrete pavers/edging that you have by the fence is not good for the plant (they'll cause the soil to become basic over time).
Have you done a soil test? If not, you should test the soil for pH where the plant will be located. If it's not low enough, I recommend applying soil sulfur (which is just granulated elemental sulfur) twice a year. Because this takes time to decompose, you might want to also apply ammonium sulfate this year and next year.
Yes, move the rhodie - at least get it away from the fence and concrete. And make sure you give it an acidic home.