In similar situations I have used the brute force approach of specializing the class template for all special types. I realize that these are quite a few and that it can become quickly rather painful. However, factoring the interesting members into a base class shared by all specializations works reasonably well. In the cases where I used this approach I typically only had static members in the class, i.e., the fact that constructors are not inherited wasn't a problem (C++ 2011 support a syntax to also inherit constructors but the while point is to keep the notation down):
template <typename T> class Reader { ... };
template <typename T> class IntReader { /* special version for integers */ };
// to bool or not bool?
template <> class Reader<wchar_t>: public IntReader<wchar_t> {};
template <> class Reader<char>: public IntReader<char> {};
template <> class Reader<char16_t>: public IntReader<char16_t> {};
template <> class Reader<char32_t>: public IntReader<char32_t> {};
template <> class Reader<signed char>: public IntReader<signed char> {};
template <> class Reader<signed short>: public IntReader<signed short> {};
template <> class Reader<signed int>: public IntReader<signed int> {};
template <> class Reader<signed long>: public IntReader<signed long> {};
template <> class Reader<signed long long>: public IntReader<signed long long> {};
template <> class Reader<unsigned char>: public IntReader<unsigned char> {};
template <> class Reader<unsigned short>: public IntReader<unsigned short> {};
template <> class Reader<unsigned int>: public IntReader<unsigned int> {};
template <> class Reader<unsigned long>: public IntReader<unsigned long> {};
template <> class Reader<unsigned long long>: public IntReader<unsigned long long> {};
Admittedly, we have too many integer types! Having six different character types is kind of ridiculous.