I am using linux and when I run your code, I get a button with a thick red border, so it looks like that the default Windows theme does not support highlightthickness
while the default linux theme does.

If you want to change the border color, it is possible with some ttk themes like 'clam':
from Tkinter import *
import ttk
root = Tk()
style = ttk.Style(root)
style.theme_use('clam')
style.configure('my.TButton', bordercolor="red")
ttk_button = ttk.Button(root, text='Go UP', style='my.TButton')
ttk_button.pack()
root.mainloop()

However, changing the borderwidth, with style.configure('my.TButton', borderwidth=10)
does not increase the width of the red border as expected.