ItemTouchHelper startSwipe not working

Question

I tried to use startSwipe within adapter, it seems not working. Strangely, when I tried swipe it with finger manually, it worked. Just that startSwipe does not work when i clicked on of the button inside holder. Need help on this~

This is where i called the startSwipe (in onCreateViewHolder method).

final ViewHolder holder = new ViewHolder(rootView);
holder.rightItemLayout.setOnClickListener(new View.OnClickListener() {
    @Override
    public void onClick(View view) {
        mItemTouchHelper.startSwipe(holder);
    }
});

This is how I setup the ItemTouchHelper:

mItemTouchHelper = new ItemTouchHelper(new ItemTouchHelper.SimpleCallback(0, ItemTouchHelper.LEFT) {
    @Override
    public boolean onMove(RecyclerView recyclerView, RecyclerView.ViewHolder viewHolder, RecyclerView.ViewHolder target) {
        return true;
    }

    @Override
    public void onSwiped(RecyclerView.ViewHolder viewHolder, int direction) {

    }
});

How ItemTouchHelper attached to recyclerView:

RecyclerView.LayoutManager mLayoutManager = new LinearLayoutManager(getApplicationContext());
recyclerView.setLayoutManager(mLayoutManager);
recyclerView.setItemAnimator(new DefaultItemAnimator());

CategoryAdapter adapter = new CategoryAdapter(this, items);
recyclerView.setAdapter(adapter);
adapter.getItemTouchHelper().attachToRecyclerView(recyclerView);