Eloquent relationship only returns result for first result in array

Question

I have the following model with these relationsships:

class Site extends Model
{
    use HasFactory;

    public function tests() {
        return $this->hasMany(Test::class,'requestedUrl','frontpage');
    }

    public function latestTests() {
        return $this->tests()->latest()->take(1);
    }   
}

I have the following eloquent statement:

$sites = Site::
    ->with('latestTest')
    ->orderBy($sort, $direction)
    ->paginate($size);

But in the result it is only the first returned record that contains latestTest. I assume it's because of take(1) but if I remove that I will choke the server since it seems to be fetching all the tests (100k+ rows) before servering the latest (at least that is what I can see in my queries log).

So how do I get the latest test "the right way"?