Set smarttab
as well. I use set expandtab shiftwidth=4 softtabstop=4 smarttab
From vim's :help smarttab
When on, a Tab in front of a line inserts blanks according to
'shiftwidth'. 'tabstop' or 'softtabstop' is used in other places. A
backspace will delete a 'shiftwidth' worth of space at the start of the
line.
When off, a Tab always inserts blanks according to 'tabstop'
or 'softtabstop'. 'shiftwidth' is only used for shifting text left or
right shift-left-right.
What gets inserted (a Tab or spaces) depends
on the 'expandtab' option. Also see ins-expandtab. When 'expandtab'
is not set, the number of spaces is minimized by using Tabs. NOTE:
This option is reset when 'compatible' is set.