Well after few days of struggle the answer is here:
For IM command line version step by step (a.png is input, tiled.png is source of background)
convert a.png -thumbnail 100x100 -background none -gravity center -extent 100x100 resized.png
convert tiled.png -crop 100x100+0+0 +repage cropped.png
convert cropped.png out.png -composite result.png
For IM command line combined code
convert tiled.png -crop 1000x1000+0+0 +repage a.png -thumbnail 1000x1000 -background none -gravity center -extent 1000x1000 -composite result.png
For use in image magick copy the default Thumbnail processor and make changes to the make method (or just create your own processor)
def make
src = @file
dst = Tempfile.new([@basename, @format ? ".#{@format}" : ''])
dst.binmode
begin
tgeom = @target_geometry.width.to_s + 'x' + @target_geometry.height.to_s
tiled_path = Rails.root.join('public','system','tiled.png')
parameters = []
parameters << tiled_path
parameters << '-crop'
parameters << tgeom + '+0+0 +repage'
parameters << ':source'
parameters << '-thumbnail ' + tgeom
parameters << '-background none'
parameters << '-gravity center'
parameters << '-extent ' + tgeom
parameters << '-composite'
parameters << ':dest'
parameters = parameters.flatten.compact.join(' ').strip.squeeze(' ')
success = convert(parameters, :source => "#{File.expand_path(src.path)}#{'[0]' unless animated?}", :dest => File.expand_path(dst.path))
rescue Cocaine::ExitStatusError => e
raise Paperclip::Error, "There was an error processing the thumbnail for #{@basename}" if @whiny
rescue Cocaine::CommandNotFoundError => e
raise Paperclip::Errors::CommandNotFoundError.new("Could not run the `convert` command. Please install ImageMagick.")
end
dst
end