django auth_group geting selected when auth_user is selected

Question

I have created a custom User model

class User(AbstractUser):
    phone_no = models.CharField(blank=True, null=True, max_length=20, default="")

    class Meta:
        db_table = 'auth_user'

When I am accessing an object of User, auth_group and auth_permission get selected automatically.

from django import db
from example.models import User
import simplejson as json

In : print json.dumps(db.connection.queries)
Out: []

In : user = User.objects.get(id=1)

In : print json.dumps(db.connection.queries)
Out: 
[
    {
        "time": "0.001",
        "sql": "SELECT `auth_user`.`id`, `auth_user`.`password`, `auth_user`.`last_login`, `auth_user`.`is_superuser`, `auth_user`.`username`, `auth_user`.`first_name`, `auth_user`.`last_name`, `auth_user`.`email`, `auth_user`.`is_staff`, `auth_user`.`is_active`, `auth_user`.`date_joined`, `auth_user`.`phone_no` FROM `auth_user` WHERE `auth_user`.`id` = 1 "
    },
    {
        "time": "0.000",
        "sql": "SELECT `auth_group`.`id` FROM `auth_group` INNER JOIN `auth_user_groups` ON ( `auth_group`.`id` = `auth_user_groups`.`group_id` ) WHERE `auth_user_groups`.`user_id` = 1 "
    },
    {
        "time": "0.001",
        "sql": "SELECT `auth_permission`.`id` FROM `auth_permission` INNER JOIN `auth_user_user_permissions` ON ( `auth_permission`.`id` = `auth_user_user_permissions`.`permission_id` ) INNER JOIN `django_content_type` ON ( `auth_permission`.`content_type_id` = `django_content_type`.`id` ) WHERE `auth_user_user_permissions`.`user_id` = 1ORDER BY `django_content_type`.`app_label` ASC, `django_content_type`.`model` ASC, `auth_permission`.`codename` ASC"
    }
]

How do I stop this behavior because when I use select_related where User is foreign key and iterate over the result for user.first_name there it becomes costly. BTW I do not use auth_group and auth_permission so I don't mind droping it