ListView items in fragment turns blank when touching another view

Question

I have a somewhat complex layout, involving multiple fragments, and a ListView in a fragment on top of a GLView. When i end a touch(MotionEvent.ACTION_UP) anywhere on screen other than the list view the textviews in the list view rows turn blank, calling adapter.notifyDataSetChanged() will refresh them, but then disappear again on the next touch.

The relevant bits of the layout:
SlidingFragmentActivity
    A SherlockFragment that fills the whole screen
         RelativeLayout
               A SherlockFragment containing LinearLayout, layout_width=WrapContent, layout_height=wrap_content, orientation=vertical
                   ImageButton, layout_width=WrapContent, layout_height=wrap_content
                   ListView(The problem view), layout_width=200, layout_height=wrap_content, background=transparent
                         Each row is a RelativeLayout containing several TextViews, with a semi-transparent background
               TextView
               ImageView
               View that Overrides onDraw(Canvas) to do custom drawing
          GLView - a Nutiteq MapView

Calling adapter.notifyDatasetChanged() any time a MotionEvent.ACTION_UP happens makes the problem less obvious, but that seems like an inelegant solution that wastes cpu time.

A couple screen shots that illustrate the problem(the list view in the lower right)