I have posted my package on CRAN and recived Valgrind's check results revealing some memory leaks (link). Unfortunately, I can't reproduce these errors. Therefore, I have some hypothesis why these errors occur but can't test them. So I need an advice whether I am going in a right direction and how it is possible to fix corresponding errors.
There are lots of problem's indicated by Valgrind but some of them are similar in a sense that they refer to the same or similar lines of code. Therefore below I have described just some problems (error message) which are representative to others. Other messages could be found via above mentioned link.
Problem 1
When running the following example code (R call to function which uses Rcpp code i.e. C++ functions in some places):
# ... some preceding code with data preparation
pgn <- mvoprobit(z ~ w1 + w2,
data = data,
marginal = list("PGN" = 3))
Valgrind produces the error message:
==2563572== Invalid read of size 8
==2563572== at 0x52B841: STRING_ELT (svn/R-devel/src/main/memory.c:3923)
==2563572== by 0x175F1952: begin (R-devel/site-library/Rcpp/include/Rcpp/vector/string_proxy.h:167)
==2563572== by 0x175F1952: operator!= (R-devel/site-library/Rcpp/include/Rcpp/vector/string_proxy.h:194)
==2563572== by 0x175F1952: pmnorm(Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Matrix<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, int, Rcpp::String, Rcpp::String, bool, bool, bool, bool, bool, bool, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, int, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, bool, bool) (/tmp/RtmpvZ7OOy/R.INSTALL331e2639ef54f9/mnorm/src/pmnorm.cpp:384)
==2563572== by 0x175C2584: _mnorm_pmnorm_try(SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*) (/tmp/RtmpvZ7OOy/R.INSTALL331e2639ef54f9/mnorm/src/RcppExports.cpp:232)
==2563572== by 0x1767C9B3: mnorm::pmnorm(Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Matrix<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, int, Rcpp::String, Rcpp::String, bool, bool, bool, bool, bool, bool, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, int, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, bool, bool) (R-devel/site-library/mnorm/include/mnorm_RcppExports.h:138)
==2563572== by 0x176867DE: lnL_mvoprobit(arma::Col<double>, Rcpp::Vector<19, Rcpp::PreserveStorage>, Rcpp::String, int, int) (packages/tests-vg/switchSelection/src/mvoprobit.cpp:399)
==2563572== by 0x17663D34: _switchSelection_lnL_mvoprobit (packages/tests-vg/switchSelection/src/RcppExports.cpp:75)
==2563572== by 0x4A268D: R_doDotCall (svn/R-devel/src/main/dotcode.c:880)
==2563572== by 0x4E126C: bcEval (svn/R-devel/src/main/eval.c:8002)
==2563572== by 0x4F43CF: Rf_eval (svn/R-devel/src/main/eval.c:1013)
==2563572== by 0x4F620E: R_execClosure (svn/R-devel/src/main/eval.c:2187)
==2563572== by 0x4F6F42: Rf_applyClosure (svn/R-devel/src/main/eval.c:2113)
==2563572== by 0x4E45FD: bcEval (svn/R-devel/src/main/eval.c:7414)
After reading some tutorials on interpreting the results of Valgrind's check I have come to the conclusion that it is reasonable to read the error message from the down up. First, I see that the problem starts from the line 399 of the code in mvoprobit.cpp
file which contains function lnL_mvoprobit
. This line of code uses the other function pmnorm
. Therefore I conclude that the problem occurs on line 384 of pmnorm.cpp
which has the following line of code (Valgrind's mention of operator!=
reinforce the hypothesis that it mentions exactly this line of code):
if (marginal_names[i] != "normal")
where marginal_names
is a CharacterVector
. I thought that Invalid read of size 8
message indicates that I am trying to read marginal_names[i]
while it is empty (reinforce this hypothesis with begin
mention by Valgrand) . However intensive check (including the check on other examples including other example code and various appropriate alternative input specifications) have revealed that when the example is running i
inside a cycle where marginal_names[i]
occures is always 0
(and never other value) while marginal_names
is a character vector of a single (one) element. Therefore I can't understand the source of the problem.
Problem 2
The next Valgrind's message going straight after the first one is as follows:
==2563572== Address 0x1920c570 is 7,840 bytes inside a block of size 7,960 alloc'd
==2563572== at 0x484086F: malloc (/builddir/build/BUILD/valgrind-3.18.1/coregrind/m_replacemalloc/vg_replace_malloc.c:381)
==2563572== by 0x5318D0: GetNewPage (svn/R-devel/src/main/memory.c:972)
==2563572== by 0x533503: Rf_allocVector3 (svn/R-devel/src/main/memory.c:2812)
==2563572== by 0x175C7ADA: SEXPREC* Rcpp::internal::primitive_range_wrap__impl__nocast<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, int>(__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, std::random_access_iterator_tag) [clone .isra.0] (R-devel/site-library/Rcpp/include/Rcpp/internal/wrap.h:116)
==2563572== by 0x175CBFB2: primitive_range_wrap__impl<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, int> (R-devel/site-library/Rcpp/include/Rcpp/internal/wrap.h:167)
==2563572== by 0x175CBFB2: range_wrap_dispatch___impl<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, int> (R-devel/site-library/Rcpp/include/Rcpp/internal/wrap.h:179)
==2563572== by 0x175CBFB2: range_wrap_dispatch<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, int> (R-devel/site-library/Rcpp/include/Rcpp/internal/wrap.h:419)
==2563572== by 0x175CBFB2: range_wrap<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > > > (R-devel/site-library/Rcpp/include/Rcpp/internal/wrap.h:429)
==2563572== by 0x175CBFB2: wrap<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > > > (R-devel/site-library/Rcpp/include/Rcpp/internal/wrap.h:941)
==2563572== by 0x175CBFB2: operator SEXP (R-devel/site-library/Rcpp/include/Rcpp/api/meat/Dimension.h:26)
==2563572== by 0x175CBFB2: wrap_dispatch_unknown<Rcpp::Dimension> (R-devel/site-library/Rcpp/include/Rcpp/internal/wrap.h:503)
==2563572== by 0x175CBFB2: wrap_dispatch_eigen<Rcpp::Dimension> (R-devel/site-library/Rcpp/include/Rcpp/internal/wrap.h:770)
==2563572== by 0x175CBFB2: wrap_dispatch_unknown_importable<Rcpp::Dimension> (R-devel/site-library/Rcpp/include/Rcpp/internal/wrap.h:787)
==2563572== by 0x175CBFB2: wrap_dispatch<Rcpp::Dimension> (R-devel/site-library/Rcpp/include/Rcpp/internal/wrap.h:807)
==2563572== by 0x175CBFB2: wrap<Rcpp::Dimension> (R-devel/site-library/Rcpp/include/Rcpp/internal/wrap_end.h:30)
==2563572== by 0x175CBFB2: operator=<Rcpp::Dimension> (R-devel/site-library/Rcpp/include/Rcpp/api/meat/proxy.h:33)
==2563572== by 0x175CBFB2: arma_wrap<arma::Mat<double> > (R-devel/site-library/RcppArmadillo/include/RcppArmadillo/interface/RcppArmadilloWrap.h:33)
==2563572== by 0x175CBFB2: SEXPREC* Rcpp::wrap<double>(arma::Mat<double> const&) (R-devel/site-library/RcppArmadillo/include/RcppArmadillo/interface/RcppArmadilloWrap.h:72)
==2563572== by 0x175C7E23: get<arma::Mat<double> > (R-devel/site-library/Rcpp/include/Rcpp/vector/converter.h:80)
==2563572== by 0x175C7E23: replace_element__dispatch__isArgument<Rcpp::traits::named_object<arma::Mat<double> > > (R-devel/site-library/Rcpp/include/Rcpp/vector/Vector.h:532)
==2563572== by 0x175C7E23: replace_element__dispatch<Rcpp::traits::named_object<arma::Mat<double> > > (R-devel/site-library/Rcpp/include/Rcpp/vector/Vector.h:525)
==2563572== by 0x175C7E23: replace_element<Rcpp::traits::named_object<arma::Mat<double> > > (R-devel/site-library/Rcpp/include/Rcpp/vector/Vector.h:514)
==2563572== by 0x175C7E23: Rcpp::Vector<19, Rcpp::PreserveStorage> Rcpp::Vector<19, Rcpp::PreserveStorage>::create__dispatch<Rcpp::traits::named_object<Rcpp::Matrix<14, Rcpp::PreserveStorage> >, Rcpp::traits::named_object<Rcpp::Matrix<14, Rcpp::PreserveStorage> >, Rcpp::traits::named_object<arma::Mat<double> >, Rcpp::traits::named_object<arma::Mat<double> >, Rcpp::traits::named_object<arma::Mat<double> >, Rcpp::traits::named_object<arma::Mat<double> >, Rcpp::traits::named_object<arma::Mat<double> > >(Rcpp::traits::integral_constant<bool, true>, Rcpp::traits::named_object<Rcpp::Matrix<14, Rcpp::PreserveStorage> > const&, Rcpp::traits::named_object<Rcpp::Matrix<14, Rcpp::PreserveStorage> > const&, Rcpp::traits::named_object<arma::Mat<double> > const&, Rcpp::traits::named_object<arma::Mat<double> > const&, Rcpp::traits::named_object<arma::Mat<double> > const&, Rcpp::traits::named_object<arma::Mat<double> > const&, Rcpp::traits::named_object<arma::Mat<double> > const&) [clone .isra.0] (R-devel/site-library/Rcpp/include/Rcpp/generated/Vector__create.h:378)
==2563572== by 0x175CAA52: create<Rcpp::traits::named_object<Rcpp::Matrix<14> >, Rcpp::traits::named_object<Rcpp::Matrix<14> >, Rcpp::traits::named_object<arma::Mat<double> >, Rcpp::traits::named_object<arma::Mat<double> >, Rcpp::traits::named_object<arma::Mat<double> >, Rcpp::traits::named_object<arma::Mat<double> >, Rcpp::traits::named_object<arma::Mat<double> > > (R-devel/site-library/Rcpp/include/Rcpp/generated/Vector__create.h:343)
==2563572== by 0x175CAA52: cmnorm(Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Matrix<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, bool, bool, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, int) (/tmp/RtmpvZ7OOy/R.INSTALL331e2639ef54f9/mnorm/src/cmnorm.cpp:261)
==2563572== by 0x175F38F7: pmnorm(Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Matrix<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, int, Rcpp::String, Rcpp::String, bool, bool, bool, bool, bool, bool, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, int, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, bool, bool) (/tmp/RtmpvZ7OOy/R.INSTALL331e2639ef54f9/mnorm/src/pmnorm.cpp:638)
==2563572== by 0x175FBAA4: pmnorm(Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Matrix<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, int, Rcpp::String, Rcpp::String, bool, bool, bool, bool, bool, bool, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, int, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, bool, bool) (/tmp/RtmpvZ7OOy/R.INSTALL331e2639ef54f9/mnorm/src/pmnorm.cpp:1343)
==2563572== by 0x175C2584: _mnorm_pmnorm_try(SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*) (/tmp/RtmpvZ7OOy/R.INSTALL331e2639ef54f9/mnorm/src/RcppExports.cpp:232)
==2563572== by 0x1767C9B3: mnorm::pmnorm(Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Matrix<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, int, Rcpp::String, Rcpp::String, bool, bool, bool, bool, bool, bool, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, int, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, bool, bool) (R-devel/site-library/mnorm/include/mnorm_RcppExports.h:138)
==2563572== by 0x17670748: lnL_mnprobit(arma::Col<double>, Rcpp::Vector<19, Rcpp::PreserveStorage>, Rcpp::String, int, int) (packages/tests-vg/switchSelection/src/mnprobit.cpp:287)
Since it mentiones lnL_mnprobit
function that does not related to mvoprobit
function from the abovementioned example code (instead lnL_mnprobit
associated with mnprobit
function) I conclude that despite the fact where the error message located it is associated with another part of example code (that is going before the example code I have posted above):
# some data preparation code
model <- mnprobit(z ~ w1 + w2,
data = data)
summary(model)
Using the same logic as before I think that the problem associated with line 261 of cmnorm.cpp file that is as follows:
List return_list = List::create(Named("mean") = mean_cond_numeric,
Named("sigma") = sigma_cond_numeric,
Named("s12s22") = s12s22,
Named("sigma_d") = sigma_d,
Named("sigma_g") = sigma_g,
Named("sigma_dg") = sigma_dg,
Named("sigma_g_inv") = sigma_g_inv);
where mean_cond_numeric
and sigma_cond_numeric
are Rcpp::NumericMatrix
while s12s22
, sigma_d
, sigma_dg
and sigma_g_inv
are arma::mat
. By running the example code I have checked that these matrices are always non-empty and contain correct values. My only hypothesis that the problem is that Rcpp::List
contains both Rcpp::NumericMatrix
and arma::mat
. Could it be the source of memory problems?
Problem 3
The error message is as follows:
==2563572== 352 bytes in 1 blocks are possibly lost in loss record 94 of 1,822
==2563572== at 0x4845464: calloc (/builddir/build/BUILD/valgrind-3.18.1/coregrind/m_replacemalloc/vg_replace_malloc.c:1328)
==2563572== by 0x401364B: UnknownInlinedFun (/usr/src/debug/glibc-2.33-21.fc34.x86_64/elf/../include/rtld-malloc.h:44)
==2563572== by 0x401364B: allocate_dtv (/usr/src/debug/glibc-2.33-21.fc34.x86_64/elf/../elf/dl-tls.c:366)
==2563572== by 0x401364B: _dl_allocate_tls (/usr/src/debug/glibc-2.33-21.fc34.x86_64/elf/../elf/dl-tls.c:612)
==2563572== by 0x53C8008: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.33.so)
==2563572== by 0x5395283: ??? (in /usr/lib64/libgomp.so.1.0.0)
==2563572== by 0x538C2D0: GOMP_parallel (in /usr/lib64/libgomp.so.1.0.0)
==2563572== by 0x17607F70: normcdf_helper<arma::Mat<double>, arma::Gen<arma::Mat<double>, arma::gen_zeros>, arma::Gen<arma::Mat<double>, arma::gen_ones> > (R-devel/site-library/RcppArmadillo/include/armadillo_bits/fn_normcdf.hpp:67)
==2563572== by 0x17607F70: arma::enable_if2<arma::is_real<arma::Mat<double>::elem_type>::value, arma::Mat<arma::Mat<double>::elem_type> >::result arma::normcdf<arma::Mat<double> >(arma::Base<arma::Mat<double>::elem_type, arma::Mat<double> > const&) (R-devel/site-library/RcppArmadillo/include/armadillo_bits/fn_normcdf.hpp:153)
==2563572== by 0x175F4C5B: pmnorm(Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Matrix<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, int, Rcpp::String, Rcpp::String, bool, bool, bool, bool, bool, bool, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, int, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, bool, bool) (/tmp/RtmpvZ7OOy/R.INSTALL331e2639ef54f9/mnorm/src/pmnorm.cpp:725)
==2563572== by 0x175C2584: _mnorm_pmnorm_try(SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*, SEXPREC*) (/tmp/RtmpvZ7OOy/R.INSTALL331e2639ef54f9/mnorm/src/RcppExports.cpp:232)
==2563572== by 0x1767C9B3: mnorm::pmnorm(Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Matrix<14, Rcpp::PreserveStorage>, Rcpp::Vector<14, Rcpp::PreserveStorage>, int, Rcpp::String, Rcpp::String, bool, bool, bool, bool, bool, bool, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, int, Rcpp::Nullable<Rcpp::Vector<19, Rcpp::PreserveStorage> >, bool, bool) (R-devel/site-library/mnorm/include/mnorm_RcppExports.h:138)
==2563572== by 0x17670748: lnL_mnprobit(arma::Col<double>, Rcpp::Vector<19, Rcpp::PreserveStorage>, Rcpp::String, int, int) (packages/tests-vg/switchSelection/src/mnprobit.cpp:287)
==2563572== by 0x17662A54: _switchSelection_lnL_mnprobit (packages/tests-vg/switchSelection/src/RcppExports.cpp:47)
==2563572== by 0x4A268D: R_doDotCall (svn/R-devel/src/main/dotcode.c:880)
I think that the problem associated with 725 line of code from pmnorm.cpp
:
arma::vec prob0 = (arma::normcdf(upr1) - arma::normcdf(lwr1)) %
(arma::normcdf(upr2) - arma::normcdf(lwr2));
Unfortunatelly I have no hypothesis why the error message occurs since arguments of arma::normcdf
are always well defined.
I will be extremelly greatful for any help in understanding the source of abovementioned problems and comments on how to fix them!