In tesseract you can set TessBaseAPI.VAR_CHAR_WHITELIST
and TessBaseAPI.VAR_CHAR_BLACKLIST
in order to ignore some special characters.
Following would make tesseract only recognize A-Z and digits
String whiteList = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
tessBaseApi.setVariable(TessBaseAPI.VAR_CHAR_WHITELIST,whiteList);
Next snippet would allow you to recognize everything except for ~ and fl
String blackList = "~fl";
tessBaseApi.setVariable(TessBaseAPI.VAR_CHAR_BLACKLIST,blackList );
Also please note that as mentioned in tesseract github issue you can't black or whitelist characters with tesseract 4.0 Alpha LSTM, instead you should train LSTM with characters you expect on your image.
Of course if you want - you can still use 3.* versions of tesseract, its tessdata is located here