What is the height or rise in elevation from your fence to that wall? What is the horizontal distance from your fence to the wall? Rise/Run = slope 6' of rise over 24' of run gives you a 25% slope. Making your slope less rounded and a straight shot will increase the speed of water flowing down your slope.
If that wall is not engineered or built properly you might have problems. Your 'surcharge' (the water in the soil behind and above the retaining wall) might be less but the erosion going over your wall will be more.
Did you build that retaining wall? What material? Is there a drain pipe behind and at the footing of this wall? Is there a foundation of sorts beneath your wall? Or is your wall a 'talus slope' of rock debris? Boulders? Please send a picture of your wall.
How will you be grading this slope theoretically? A big excavator? Or by hand? What is the square footage of this area (width X length or a side times another adjoining side)? Make sure that the elevation starts 2" minimum below your fence. There should be no soil or mulch or vegetation touching the bottom of your fence. If this is a rip rap retaining wall system, cutting that curve off your slope might overwhelm your wall with erosion and speed of water.
Grins. I don't know if I am looking at a slope of 100 feet or 5' in width. Where is your home? Above slope or down slope? That is very important! Please send pictures. This drainage stuff is kind of my forte. I taught this for college students. Changing drainage will always affect your neighbors unless the water is contained within your site only. There are laws and regulations about this we might need you to look into when you are able to send more information and pictures.
I'll eliminate all of these questions to answer and tell you what you need to know. Please fire questions to me back if you don't understand what I am asking.
I am sorry for all the questions within a non - answer. This is not an answer, just easier to ask all these questions.
Another question is WHERE exactly do you want to move the soil? Behind the fence or below the rocks? Uphill or downhill? Are these loose rocks in a gabion or just a 'talus' slope or are these boulders (larger than 18 to 24 inches or 2 man rock...takes 2 men to life said rock versus a rock that one man can lift or 3 man rock that takes 3 men to life, this is real)!
Where is it that you live? What kind of soil do you have? Are you planning to replant this slope? What is planted on it now? Have you considered terracing? Making steps out of the slope?