Unwanted extra whitespace around polygon graphic fills

Question

I am using Geoserver to make a map using SLD. I have a problem with trying to portray symbols with graphic fills: extra whitespace is added around the symbols, which makes my map unreadable. Here is an example:

I have this tiling using both QGIS, and the OpenLayers map preview from Geoserver. This effect is less present at really high zoom levels, which are of little interest to me:

The code I used for the SLD is the following:

<FeatureTypeStyle>
  <Rule>
    <PolygonSymbolizer uom="http://www.opengeospatial.org/se/units/metre">
      <Fill>
        <GraphicFill>
          <Graphic>
            <ExternalGraphic>
              <OnlineResource xmlns:xlink="http://www.w3.org/1999/xlink" xlink:type="simple" xlink:href="SVG/mySVG.svg"/>
              <Format>image/svg+xml</Format>
            </ExternalGraphic>
            <Size>500</Size>
          </Graphic>
        </GraphicFill>
        <VendorOption name="graphic-margin">-20</VendorOption>
      </Fill>
    </PolygonSymbolizer>
  </Rule>
</FeatureTypeStyle>

The code is quite simple, and quite similar to the code from Geoserver documentation. Note that I tried to use the "graphic-margin" vendor option to reduce the tiling, to no avail. Whatever value I put into this option, nothing changes.

How can I get rid of the whitespace?