AWS CloudWatch Evidently experiment does not record events for custom metrics

Question

I have configured an Evidently project, with one feature and prepared an experiment, where I defined one custom metric. The problem is quite simple: I just can't get Evidently to record any events. I just can't figure out, what's wrong.

By this moment I have simplified the troubleshooting and I am using command line to send event to Evidently:

aws evidently put-project-events \
    --project "myprojectname"\
    --events '[{"data" : "{\"value\":\"1\", \"userIP\":\"12.345.678.90\"}", "timestamp":"1693156893", "type":"aws.evidently.custom" }]'

the result seems to be successful:

{
    "eventResults": [
        {
            "eventId": "fddbaf08-ad49-40d1-bcaa-aa49f44ea5e1"
        }
    ],
    "failedEventCount": 0
}

But still, my experiment can't seem to register any events (event count is still zero for both variations of my feature and is not increasing with each script run; also, not increasing after a while (https://i.stack.imgur.com/WFoJP.png)).

In the experiment configuration the metric rule is looking like this:

{
  "entityIdKey": "userIP",
  "valueKey": "value",
  "eventPattern": {
    "userIP": [
      {
        "exists": true
      }
    ],
    "value": [
      {
        "exists": true
      }
    ]
  }
}

I can see that the metrics are coming in to "CloudWatch" -> "Metrics" -> "All metrics", but they still do not show up in Evidently's experiment.

My code in the React app looks like this:

    import { EvidentlyClient, EvaluateFeatureCommand, PutProjectEventsCommand } from "@aws-sdk/client-evidently";
    
    const eviclient = new EvidentlyClient({ 
        region : 'eu-north-1',
        endpoint: "https://evidently.eu-north-1.amazonaws.com",
        credentials : {
            accessKeyId     : 'AKIA2NxxxxxxxxW6P3UL', 
            secretAccessKey : 'fo5hMrufII7eHZxxxxxxxxS/5ZHnXqIFXNXGHt2e'
        }
    });
    
    async function evaluate() {
        const command1 = new EvaluateFeatureCommand({ 
            project: "myprojectname", feature: "SolutionInterest",  entityId: "12.345.678.90"
        });
        const command2 = new PutProjectEventsCommand({
            project: "myprojectname", events: [
              { timestamp: new Date(), type: "aws.evidently.custom", data: { "userIP": "12.345.678.90", "value": "1" } }
            ]
        });
        let response1 =  await eviclient.send(command1);
        console.log('response1: ' + JSON.stringify(response1));
        let response2 =  await eviclient.send(command2);
        console.log('response2: ' + JSON.stringify(response2));
    }
    
    function App() {
        evaluate();
        return null;
    }

What I get in the console is the following:

[Log] response1: {"$metadata":{"httpStatusCode":200,"requestId":"f9f35837-4796-4456-925f-97be4afaa240","attempts":1,"totalRetryDelay":0},"details":"{\"experiment\":\"Test4\",\"treatment\":\"Variation2\"}","reason":"EXPERIMENT_RULE_MATCH","value":{"boolValue":true},"variation":"Variation2"}

[Log] response2: {"$metadata":{"httpStatusCode":200,"requestId":"fb29f316-85bc-4416-9dcf-920ba98fae57","attempts":1,"totalRetryDelay":0},"eventResults":[{"errorCode":null,"errorMessage":null,"eventId":"434115a0-0e18-4cae-a2fe-c53f8dca117d"}],"failedEventCount":0}

So far I tried the following:

to create a project and an experiment in another region - still same issue, so it is not region specific.
to create an experiment with the same user identity, that is being used with client configuration - still same problem.
to create a metric with and without rule pattern - no difference.

Does anyone have a clue, what could be wrong here or if there are any pre-requisite configurations that I have overlooked?

Really appreciate your time and advise.

AWS CloudWatch Evidently experiment does not record events for custom metrics

0 Answers0