EF6 Removing child items from one to many relation, from parent rather than from dbContext

Question

I have this POCO class for User which has and Id and a collection of RoleUser objects among other properties

public class SystemUser : IEntity
{
    public int Id { get; set; }

    [MaxLength(50)]
    [Index("ExternalUserIdIndex",IsUnique = true)]
    public string ExternalUserId { get; set; }
    public virtual ICollection<RoleUser> UserRoles { get; set; }
}

My RoleUser class is this

public class RoleUser : IEntity
{
    public int Id { get; set; }
    public int SystemUserId { get; set; }
    public virtual Role Role { get; set; }
}

The Role class is not relevant for this question. I have some seeding method for adding default users with some roles, before adding roles I delete existing roles. I was trying to do the following:

var user = context.SystemUsers.FirstOrDefault(u => u.ExternalUserId == "extId");

As you can see I get this instance as tracked by my dbContext. Then first I tried:

user.UserRoles.Clear();
//add roles

But I got this when running my seed method:

The relationship could not be changed because one or more of the foreign-key properties is non-nullable. When a change is made to a relationship, the related foreign-key property is set to a null value. If the foreign-key does not support null values, a new relationship must be defined, the foreign-key property must be assigned another non-null value, or the unrelated object must be deleted.

Then I tried

user.UserRoles.ToList().ForEach(r => developerPaco.UserRoles.Remove(r));

An got same error. Finally I did:

user.UserRoles.ToList().ForEach(r=>context.RoleUsers.Remove(r)); 

And it worked. So I want to understand why it did not work the other way.

Note: I got it working with my first two attempts by changing my RoleUser class like the following:

public class RoleUser : IEntity
{
    public int Id { get; set; }
    public int? SystemUserId { get; set; }
    public virtual Role Role { get; set; }
}

Making nullable SystemUserId, but then I just get data with null values and not deleted as I wanted.