AngularJS directive does not work inside Angular

Question

I have a simple Angular 7 component, that I'm using to normalize a page markup.

page-title.component.ts

import { Component } from '@angular/core';
import { Input } from '@angular/core';

@Component({
  selector: 'my-page-title',
  templateUrl: './page-title.component.html',
  styleUrls: ['./page-title.component.css']
})
export class PageTitleComponent {
      @Input() title?: string;
}

page-title.component.html

<div class="panel panel-default">
  <div class="panel-heading" *ngIf="title">
    <h3 class="panel-title">{{ title }}</h3>
  </div>
  <div class="panel-body">
    <ng-content></ng-content>
  </div>
</div>

I did downgrade this component with ngUpgrade library and it works fine, but there is one problem. In my AngularJS app I have the following directive

(function () {
  function IsGrantedDirective(AuthManager) {
    return {
      restrict: 'A',
      link(scope, elem, attr) {
        if (AuthManager.isGranted(attr.isGranted)) {
          return;
        }
        console.log("I'm called");
        elem.remove();
      },
    };
  }

  angular
    .module('my.common')
    .directive('isGranted', IsGrantedDirective);
}());

The directive is pretty simple, but when I'm using it somewhere inside Angular my-page-title component:

<my-page-title>
    <div is-granted="role">Whatever</div>
</my-page-title>

It does not work:

1. console.log("I'm called") works
2. there are no errors in console
3. the div is being displayed anyway

It seems that elem.remove() does not take any affect inside Angular component, any ideas, please?

P.S. it seems to be working if a put another bare div around an element with the attribute

 <my-page-title>
     <div> // <----
         <div is-granted="role">Whatever</div>
     </div>
 </my-page-title>