0

I want to compare the error between y and yhat. y is generated using known values which are the coefficients of a Moving Average model. yhat is generated using estimates of the coefficients. What are the statistics which show how close the output are? In machine learning papers I have seen standard deviation and mean square error as the performance metric. But I cannot understand how I can apply these in this example. Any guidance will be very helpful. Thankyou.

N = 100;
a1=0.2;
b1=0.5;
h = [1 a1 b1]; %channel coefficients
h_hat  = [1 0.23 0.45];
data = rand(1,N);
y = filter(h,1,data); %transmitted signal through MA channel
yhat = filter(h_hat,1,data);
SKM
  • 959
  • 2
  • 19
  • 45

1 Answers1

1

How to calculate MSE:

MES= mean((y - yhat).^2)

And here the standard error of the mean:

err=y - yhat;
SE = std(err)/sqrt(length(err));

However, the metric you are using should address your research question/ hypothesis. It might be that SE or MSE are not the right choices. Without knowing what you are investigating it is difficult to give any suggestions.

Irreducible
  • 864
  • 11
  • 24
  • Why divide by `sqrt(N)`? `std` already computes the standard deviation. – mikkola Oct 26 '17 at 06:41
  • [SE in Matlab](http://www.investopedia.com/ask/answers/061715/how-do-i-calculate-standard-error-using-matlab.asp): To calculate the standard error of the mean in a sample, the user needs to run a one-line command in Matlab "stderror = std( data ) / sqrt( length( data ))" – Irreducible Oct 26 '17 at 06:54
  • In that case you should update your answer to say "standard error of the mean" rather than "standard deviation of the error". – mikkola Oct 26 '17 at 06:57
  • Thank you for answering. I have some doubts, could you please clarify?(1) if `y` is noisy with an SNR of 10 dB, say `y=filter(h,1,data);' noisy_y = awgn(y,10,'measured)` and `h_hat` is obtained using estimation algorithm based on `noisy_y`. Then will MSE = `noisy_y - y_hat`? Also, lesser the standard error of the mean, the better is the performance? – SKM Oct 26 '17 at 16:13
  • @Irreducible: I have put up some doubts under the comments. Can you please clarify? thanks – SKM Oct 26 '17 at 16:55
  • In general I would not make any statement regarding the performance without seeing the distribution of `y-yhat` and comparing them with the appropriate statistical test. Your sentence "Then will MSE..." is not clear to me – Irreducible Oct 27 '17 at 05:30
  • @Irreducible: I was asking about the formula of the MSE for comparing. Also about standard deviation of the error...is lower the standard deviation better is the estimation? I mean lower value is preferred? – SKM Oct 27 '17 at 05:42
  • @Irreducible: sorry to sound noisy, but just a request if you could kindly clarify if lower or higher SE value is preferred or not. I have also posted a new question which is based on how to use these metrics here https://stackoverflow.com/questions/46969316/matlab-how-to-perform-comparison-of-simulated-model-after-estimation-and-what Could you please take a glance? Your help is much appreciated. – SKM Oct 28 '17 at 16:17