How to measure a model after A/B testing?

Question

I am working on a system that has worked for years and now they decided to add a recommendation engine. After doing the A/B testing, we will decide on a model. Now, is there a way to measure how good this model is performing?

Answer 1

Your target metric is a matter of experiment design and generally noone will be able to answer this question better than you. Typically, you need a way to measure engagement. Some of this will be immediate but some of the effects may take a longer term to be registered, e.g. when a user remembered your recommendation but returned to it after some period of time. The immediate impact, like a click on a recommended item, is easy to record because it happens in the timeframe of the current user experience. The long term impact is harder to record because you'll need a way to correlate a latent action to recommendations that happened some time ago.