Avoiding virtual function calls in a computational graph

Question

I'm using a DAG (directed acyclic graph) to represent and evaluate expressions; each node represents an operation (+, -, /, *, accumulate, etc...) and evaluation of the entire expression is achieved by sequentially evaluating each node in topologically sorted order. Each node inherits for a base class RefNode and implements a virtual function, evaluate, according to the operator it represents. The Node class is templatized on a functor that represents the operator. The node evaluation order is maintained in a vector<RefNode*> with ->evaluate() calls made to each element.

Some quick profiling shows that a virtual evaluate slows down an addition node by a factor of 2x [1], either from the overhead or trashing the branch prediction.

As a first step encoded the type information as an integer an used static_cast accordingly. This did help, but its clunky and I'd rather not jump around in the hot portion of my code.

struct RefNode {
    double output;
    inline virtual void evaluate(){}
};

template<class T>
struct Node : RefNode {
    double* inputs[NODE_INPUT_BUFFER_LENGTH];
    T evaluator;
    inline void evaluate(){ evaluator(inputs, output); }
};

struct Add {
    inline void operator()(double** inputs, double &output)
    {
        output=*inputs[0]+*inputs[1];
    }
};

An evaluation may look like:

Node<Add>* node_1 = ...
Node<Add>* node_2 = ...
std::vector<RefNode*> eval_vector;

eval_vector.push_back(node_1);
eval_vector.push_back(node_2);

for (auto&& n : eval_vector) {
    n->evaluate();
}

I have the following questions, bearing in mind performance is critical:

1. How can I avoid virtual functions in this situation?
2. If not, how can I change the way I represent an expression graph to support multiple operations, some of which must hold state, and avoid virtual function calls.
3. How do other frameworks such as Tensorflow/Theano represent computational graphs?

[1] A single addition operation on my system takes ~ 2.3ns with virtual functions and 1.1ns without. While this is small, the entire computational graph is mostly addition nodes and hence there is a good portion of time to be saved.

Answer 1

As mentioned in the comments, you will need to know the graph at compile-time to remove the virtual dispatch. To do that, you only need to use a std::tuple:

auto eval_vector = std::make_tuple(
    Node<Add>{ ... },
    Node<Add>{ ... },
    ...
);

Then, you only need to remove the virtual and override keywords, and removing the empty function in the base class.

You will find that range based for loop don't support tuples yet. To iterate on it, you will need that function:

template<typename T, typename F, std::size_t... S>
void for_tuple(std::index_sequence<S...>, T&& tuple, F&& function) {
    int unpack[] = {(static_cast<void>(
        function(std::get<S>(std::forward<T>(tuple))
    ), 0)..., 0};
    static_cast<void>(unpack);
}

template<typename T, typename F>
void for_tuple(T&& tuple, F&& function) {
    constexpr std::size_t N = std::tuple_size<std::remove_reference_t<T>>::value;
    for_tuple(std::make_index_sequence<N>{}, std::forward<T>(tuple), std::forward<F>(function));
}

You can then iterate on your tuple like that:

for_tuple(eval_vector, [](auto&& node){
    node.evaluate();
});