This will involve customizing the multifield js and creating custom xtypes by extending composite field. Most of the part of nesting custom multifields is same as creating a single custom multifield.
The inner multifield will return a comma separated String on getValue, this will have to be concatenated with other fields of outer multifield separated by a delimiter. The set value of inner multifield will expect a string array, for this you will have to modify the multifield js of inner multifield and override the set value method to take the comma separated string and split into an array.
The outer multifield will also have to be updated on change of inner multifield content.This can be done by calling the update method right after updating inner multifield(reference to the outer multifield can be obtained through findParentBy method)
The end result will look like this
String array
[0] : a-outer-field1<#->a-outer-field2<#->a-1-inner-field1<#-#>a-1-inner-field2<#-#>,a-2-innerfield1<#-#>a-2-innerfield2<#-#><#-*>
[1] : b-outer-field1<#->b-outer-field2<#->b-1-inner-field1<#-#>b-1-innerfield2<#-#>,b-2-innerfield1<#-#>b-2-innerfield2<#-#><#-*>
Check out this link http://cq5tutorials.blogspot.com/2014/04/cq5-multifield-in-multifield.html