I reread your question and you say you put 'grit' beneath the plant when you planted? I take back that that might not be the problem. If you've got silt loam soil (small pores) and you used 'grit' that has large pores you might have made a perched water table which actually inhibits drainage. How much did you use beneath these lavenders? What size is this grit is it sand or gravel? How thick of a layer? The deal with adding rock, grit, gravel below the soil means the smaller pores in the soil above have to become saturated before any water will start to drain into the...more porous layer. This is more prevalent in pots because some old lady started this thing with the rock layer beneath the soil to enhance drainage. It does the opposite. The soil has to be completely saturated before the water is able to slowly move down and into the rock/grit. Pretty interesting physics.
But this wouldn't be true for the lavender on the other side of the yard.
Pull up the dead plants, lets not worry about that grit right now. These plants are pretty tough and this is in the garden not a pot type situation. The plants look healthy otherwise and what I am seeing looks fairly normal for lavender. The reason this plant isn't used for hedges is exactly because parts or whole plants die for no discernible reason. What you do need to learn is how to prune these guys. After flowering at some point you need to use hedging shears and lop of all the flowers and tips of the leaves. Imagine a salad bowl upside down. Keep the shears tangent on the surface of this bowl. This should be done at least once per year. The bowl should be a wide gently sloping UPSIDE DOWN bowl. The bottom needs to be the widest point of this bowl. I like a flat bottom for the top of the shrub...ha ha ha does that make sense? Doing this will thicken and re energize lavenders. You should be able to buy one gallon plants and plant them offset, a zigzag pattern. Take the top picture, pull the dead one out and plant a new plant about 8" closer to the wall and a couple of inches away from the existing one. The next plant going to the left would be closer to the rocks in front, not in line with the original plant on the right.
This way there are no straight lines and when one dies it will not be noticeable. This is called a 'soft' hedge. Try to keep the distances consistent but always one forward, one back zigzag. Think of salad bowls touching and pruning so that you have a soft mounding mass of lavender sweeping the length. Also plant a bit of a lavender hedge of at least three elsewhere in your yard to pull this plant into your landscape. I would also incorporate an evergreen shrub to be part of a 'skeleton' planting.
This is just an idea and I hope this is somewhat clear using just words to describe...I'd use Ilex 'Sky Pencil' in a line behind the lavender. Placed very regularly in a line behind the lavender. Perhaps 3 feet apart, the line exactly 1 1/2 feet from your fence. The lavender will become this gray purple cloud at and around the base of these shiny, vertical, architectural looking dark green plants. I can't see what you've got going, wish I knew how to DRAW on this internet...if you could send a picture of your entire back yard and also a clear picture of this hedge and brick wall I could try to explain better if necessary.