Most likely you are trying to remove an object that doesn't come from your DbContext. For example if your object comes from a web form in ASP .Net MVC it has noting to do with the database until you attach it to the database.
There are 2 things you can do to be able to remove your item:
db.Set<T>.Attach(item);
// now you should be able to remove it
or
item = db.Set<T>.Single(x => x.Id == itme.Id);
// now you should be able to remove it
EDIT
You should also make sure that you're not trying to attach an object that already is attached. Take a look at the EntitState:
db.Entry(item).State