3

I am walking through the intra prediction part of HEVC reference software. I can see inter prediction and intra prediction in xCompressCU method in TEncCu.cpp file. But I can't locate the place of planar and DC modes are calculated. Where are these modes in the reference software? Can someone help me with this?

harsh
  • 2,399
  • 9
  • 29
  • 47

1 Answers1

1

For the DC and all the angular modes, go find the function xPredIntraAng in the TComPrediction.cpp file. These intra modes are performed there.

For the planar, there is another function probably called xPredIntraPlanar. I'm not sure about the name, though. I don't have access to code right now. But if you find the first function, this one must be next to it in the TComPrediction.cpp file.

Good luck.

Mosen
  • 403
  • 2
  • 7
  • 1
    Thanks a lot for the answer. Yes you are correct about the planar method name it is xPredIntraPlanar. :-) But I cannot locate where the planar mode cost calculation. I mean the RD cost for planar mode. Sorry if I was not clear in the question, I want the calculation of RD cost so that I can turn off all other directional modes and get cost for that only. Can you make this clear for me please? Thanks a lot for help. – harsh Dec 16 '17 at 11:51
  • 1
    The RDCost is basically calculated somewhere else. As you might saw, the functions that I mentioned are responsible for the prédiction step. However, for the full RDCost calculation you need to get the residual quantized and transformed. Then calculate the rate and the distortion. This things are done two or three levels before the prediction functions.  Again, since I don't have access to the code, everything I say is not fully accurate. -> next comment – Mosen Dec 16 '17 at 12:10
  • 1
    I suggest that you find all occurrences of the function `xGetIntraBitsQtLuma` (or something very similar) in the code. This function gives you the number of bits for the current CU and normally is a few line away from the distortion calculation, and then full RDCost calculation.  As far as I remember, the RDCost calculation function is called `CalcRdCost`. Note that I emphasized on the full RDCost so that you don't confuse it with the Hadamard-based rough cost estimation. – Mosen Dec 16 '17 at 12:11
  • Thanks a lot @Mohsen. Hadamard-based rough cost estimation is done in the loop that goes on for all intra modes, right? I located where the RD costs get calculated but it happens after all modes are done to calculate `CandNum`. Can't I do only planar or DC RD calculations without going to this loop? Also what is this `CandNum`? Please give me answer for these. Thanks a lot. – harsh Dec 16 '17 at 19:58
  • 1
    @harsh if you are dealing with the `CandNum`, you are probably still in the hadamard-based cost estimation part. Find a `for` loop after these ones, where the iterator loops over `NumFullRDModes`. – Mosen Dec 16 '17 at 23:45
  • 1
    Okay I will look at that. Thanks a lot. – harsh Dec 17 '17 at 00:07
  • 1
    That is numModesForFullRD. Thanks a lot. – harsh Dec 17 '17 at 00:14