Old question, but I just had the same problem:
AdornerLayers are shared by multiple UIElements. To access the AdornerLayer of any ui-element (of any Visual, to be more specific), you can call static AdornerLayer.GetAdornerLayer(visual). This method walks up the VisualTree, finds the first AdornerDecorator (a container to hold an AdornerLayer) and returns the AdornerLayer associated with the UIElement. Then you can access the Adorners related ot the UIElement, inside that AdornerLayer.
Here's the code to remove all Adorners, related to your StackPanel, from the AdornerLayer that is used by the StackPanel:
var adornerLayer = AdornerLayer.GetAdornerLayer(yourStackPanel);
var adornersOfStackPanel = adornerLayer.GetAdorners(yourStackPanel);
foreach (var adorner in adornersOfStackPanel)
adornerLayer.Remove(adorner);