Edittext call afterTextChanged only if user modified it and not the program

Question

I am new to android and having some trouble with this method and wondering if someone can assist with this.

Question: I have an array adapter with EditText box that the user can modify in the ListView. But, I only want user modifications and not the program's in my fragment. How do I do this?(read on for full context) The issue here is that when the user modifies this EditText in the listView, the adapter will make a call to the fragment and the fragment will manipulate the numbers. Then it will modify the other EditText in the list view.

The problem for me is, when the user modifies or clicks on the EditText in the list view, it's value using afterTextChanged() will be passed to the fragment, but since i do some modification and modify the same EditText in different positions in the list view programmatically, the afterTextChanged events gets called for other EditText views in the list view as well. I just want user changed.

To tackle this, I tried using this onTouchListener(). Just want to know if this is correct.

Class myArrayAdapter extends ArrayAdapter<Obj>{

    //private defines and static viewHolder class

    public myArrayAdapter (Context c, ArrayList<Obj> obj, MyFragment myFrag
    {
        //this....
    }

    @Override
    public View get View (int position, View convertView, ViewGroup parent){
    obj posObj = getItem(position);
    if (convertView == NULL){

        //inflate (ObjlistView)
        viewHolder.userInputValue = (EditText) covertView.findViewById(R.id.view_userInput);

       //user onTouch
                 final int pos= position;
        viewHolder.userInputValue.setOnTouchListener(new View.OnTouchListener(){
            @Override
            public boolean onTouch(View view, MotionEvent event){
                EditText UserInput= (EditText)view;
                //this is a custom Text watcher for passing editText Value to Fragment. SO I want this to be only called when user modifies it. Not when the Fragment modifies it and calling notifyDataSetChanged
                UserInput.addTextChangedListener(new CustomTextWatcher(pos, myFrag));
                return false;
            }
        });

        convertView.setTag(viewHolder)
     }else {//...}

     viewHolder.userInputValue.setText(posObj.getValue());

    } 

}

XML for the ListVie

<EditText
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:inputType="number"     // another annoying issue when i user clicks the editText it will show the number pad then will go to the keyboard and every time there is a delete or a new entry this keyboard change will happen. But that is not for this question. 
    android:id="@+id/view_userInput"

In the fragment this is how I am modifying the edittext again

Obj myTempUserobj ;
for (int i = 0; i< userObj.size(); i++){
    myTempUserobj = userObj.get(i);
    //modify the field in the object
    userObj.set(i,myTempUserobj) 
    myAdapter.notifyDataSetChanged();
}

Thanks for your Help!