how can i do authorities access controll in spring security oauth2?

Question

I`m trying to use spring security oauth2 to do authority access controll with annotations like @PreAuthorize("hasAuthority('perm2')") or with security config in WebSecurityConfigurerAdapter.But I always get 403 forbiden when I try to access an api annotated by @PreAuthorize("hasAuthority('perm2')") after I login by a user that has 'perm2' authority .I set breakpoint in ClientDetailsUserDetailsService but the loaded UserDetails did not contain any authority.What is more,I found that I could extract authorities contain 'perm2' from AuthoritiesExtractor.
One way to do this is rewriting the UserDetailService so that I could set authorities to UserDetails by AuthoritiesExtractor. But my question is:

Is it a best practice? or am I just missing something else?

My tried code is below,followed by baeldung tutorial

my authserver code is here

@SpringBootApplication
@EnableResourceServer
@RestController
public class App extends SpringBootServletInitializer {


    @GetMapping("/user/me")
    public Principal user(Principal principal) {
        return principal;
    }


    public static void main(String[] args) {
        SpringApplication.run(App.class, args);
    }


    @Configuration
    @EnableAuthorizationServer
    public static class AuthServerConfig extends AuthorizationServerConfigurerAdapter {
        @Autowired
        private BCryptPasswordEncoder passwordEncoder;

        @Override
        public void configure(
                AuthorizationServerSecurityConfigurer oauthServer) throws Exception {
            oauthServer.tokenKeyAccess("permitAll()")
                    .checkTokenAccess("isAuthenticated()");
        }

        @Override
        public void configure(ClientDetailsServiceConfigurer clients) throws Exception {
            clients.inMemory()
                    .withClient("SampleClientId")
                    .secret(passwordEncoder.encode("secret"))
                    .authorizedGrantTypes("authorization_code")
                    .scopes("user_info")
                    .autoApprove(true)
                    .redirectUris("http://localhost:8082/ui/login", "http://localhost:8083/ui2/login");
        }
    }

    @Configuration
    @Order(1)
    public static class SecurityConfig extends WebSecurityConfigurerAdapter {

        @Override
        protected void configure(HttpSecurity http) throws Exception {
            http.requestMatchers()
                    .antMatchers("/login", "/oauth/authorize")
                    .and()
                    .authorizeRequests()
                    .anyRequest().authenticated()
                    .and()
                    .formLogin().permitAll();
        }

        @Override
        protected void configure(AuthenticationManagerBuilder auth) throws Exception {
            auth.inMemoryAuthentication()
                    .withUser("john")
                    .password(passwordEncoder().encode("123"))
                    .roles("USER").authorities("perm1").and()
                    .withUser("tom")
                    .password(passwordEncoder().encode("123"))
                    .roles("USER")
                    .authorities("perm1", "perm2");
        }

        @Bean
        public BCryptPasswordEncoder passwordEncoder() {
            return new BCryptPasswordEncoder();
        }
    }

my client configuration is here:

@Configuration
@EnableOAuth2Sso
@SpringBootApplication
@EnableGlobalMethodSecurity(securedEnabled = true,prePostEnabled =true)
public class App extends WebSecurityConfigurerAdapter {

    @Bean
    AuthoritiesExtractor authoritiesExtractor(){
        return new FixedAuthoritiesExtractor();
    }

    @Override
    public void configure(HttpSecurity http) throws Exception {
        http.antMatcher("/**")
                .authorizeRequests()
                .antMatchers("/", "/login**")
                .permitAll()
                .anyRequest()
                .hasAuthority("perm1").anyRequest()
                .authenticated();
    }

    public static void main(String[] args) {
        SpringApplication.run(App.class, args);
    }
}

and this is my controller:

@Controller
public class MyController {

    @GetMapping("/securedPage")
    @PreAuthorize("hasAuthority('perm2')")
    public String securedPage() {
        return  "securedPage";
    }

    @GetMapping("/testPerm")
    @PreAuthorize("hasAuthority('perm3')")
    public @ResponseBody
    String testPerm() {
        return  "This could not be happened!";
    }
}

What I expect is :
when I login with john,I got forbidden when I access securedPage.
But when I login with tom,I could acess securedPage.

The fact is whoever I login,I get forbidden page below

Whitelabel Error Page
This application has no explicit mapping for /error, so you are seeing this as a fallback.

Fri May 24 11:29:09 CST 2019
There was an unexpected error (type=Forbidden, status=403).
Forbidden