How to filter the results from a composite aggregation?

Question

I want to filter the results of the composite aggregation which inside has a top_hits aggregation. So I first group my data with a top_hits, then I use this as a subaggregation inside my composite aggregation that has a single source based on an Id and I don't know how to filter those gruped results.

I've tried using the filters aggregation but I'm not sure since composite aggregation must be the father of all aggregations. Tried different combinations of these aggregation but none of these show me the results as I want.

{
  "size": 0,
  "aggs": {
    "grouped_data": {
      "composite": {
        "sources": [
          {
            "artifact": {
              "terms": {
                "field": "artifactId.keyword"
              }
            }
          }
        ],
        "size": 20
      },
      "aggs": {
        "top_artifacts_hits": {
          "top_hits": {
            "size": 1,
            "sort": [{
              "initialDate": {
                "order": "desc"
              }
            }]
          }
        }
      }
    }
  }
}

I tried using the query API for filtering but that is not a good option for me since the filters I want to apply are meant for the grouped results. Using some query before the main aggregation makes ElasticSearch query first and then group. I need it to be backwards. I'm using ES 6.3 under AWS.

So my documents look something like this:

{
  "artifactId": "foo",
  "clientId": "bar",
  "artifactState": "foozz",
  "initialDate": 1559745246
}

What I need to do is to get the last artifactState based on the initialDate for each different artifactId so this is why I'm using top_hits + composite.