0

I am setting up a self made tracer in PyTorch source code. When I try to build a previous version of the code, I encounter the following error :

(edited the output, it is with make VERBOSE=1)

[ 47%] Linking CXX shared library ../lib/libcaffe2.so
cd /opt/pytorch/build/caffe2 && /usr/bin/cmake -E cmake_link_script CMakeFiles/caffe2.dir/link.txt --verbose=1
/usr/bin/c++  -fPIC  -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -O2 -fPIC -Wno-narrowing -fPIC -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic 

[...]

,--whole-archive,/opt/pytorch/build/lib/libonnx.a -Wl,--no-whole-archive ../lib/libonnx_proto.a ../lib/libprotobuf.a -pthread -Wl,-rpath,/opt/pytorch/build/lib:/opt/conda/envs/build/lib:/usr/local/cuda/lib64:
/usr/bin/ld: CMakeFiles/caffe2.dir/__/aten/src/ATen/native/cpu/Activation.cpp.AVX2.cpp.o: relocation R_X86_64_PC32 against undefined symbol `_ZN2at6vec25612_GLOBAL__N_118SOURCE_CODE_TRACERE' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
caffe2/CMakeFiles/caffe2.dir/build.make:17251: recipe for target 'lib/libcaffe2.so' failed
make[3]: *** [lib/libcaffe2.so] Error 1
make[3]: Leaving directory '/opt/pytorch/build'
CMakeFiles/Makefile2:3900: recipe for target 'caffe2/CMakeFiles/caffe2.dir/all' failed
make[2]: *** [caffe2/CMakeFiles/caffe2.dir/all] Error 2
make[2]: Leaving directory '/opt/pytorch/build'
Makefile:138: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/opt/pytorch/build'
Makefile:4: recipe for target 'all' failed
make: *** [all] Error 2

I am not sure if I am missing a step for the tracer class import, or maybe the problem is linked to a shared library (libcaffe2.so ?) that is needed (like here : Relocation R_X86_64_PC32 against undefined symbol can not be used when making a shared object; recompile with -fPIC ) ?

To use the tracer, I am doing this :

#include "ATen/cpu/vec256/SourceCodeTracer.h"

namespace at {
namespace vec256 {
namespace {

#ifdef __AVX2__

struct Vec256i {
protected:
  __m256i values;

  static inline __m256i invert(const __m256i& v) {
    // some code
    extern SourceCodeTracer SOURCE_CODE_TRACER;
    SOURCE_CODE_TRACER.trace("vec256_int.h __m256i invert()");
    return something;
  }

The class cpp file :

#include "SourceCodeTracer.h"

SourceCodeTracer* SourceCodeTracer::_singleton = NULL;
extern SourceCodeTracer SOURCE_CODE_TRACER = SourceCodeTracer::getInstance();

SourceCodeTracer::~SourceCodeTracer() {
        // TODO: Write into
        std::cout << "Destructing object" << std::endl;
        ofstream traceOutputFile;
        try
        {
                std::cout << "Writing to " << _filePath << std::endl;
                traceOutputFile.open(_filePath);
                traceOutputFile << "Run at TODO PUT TIME" << std::endl;
                for (auto it = _map.begin(); it != _map.end(); ++it) {
                        traceOutputFile << it->first << " => " << it->second << '\n';
                }
        }
        catch (const std::exception& e)
        {
                std::cout << "Got error while trying to write trace file:" << e.what() << std::endl;
        }
        traceOutputFile.close();
}

SourceCodeTracer & SourceCodeTracer::getInstance() {
        if (SourceCodeTracer::_singleton == nullptr) {
                std::cout << "Constructing object" << std::endl;
                SourceCodeTracer::_singleton = new SourceCodeTracer();
        }
        return *SourceCodeTracer::_singleton;
}

void SourceCodeTracer::trace(const string & tr) {
        _map[tr] = _map[tr] + 1;
}

SourceCodeTracer::SourceCodeTracer() {
        _filePath = "trace.txt"; //get_env_var("TRACER_OUTPUT_FILE_PATH");
        std::cout << "Got environment value for TRACER_OUTPUT_FILE_PATH of " << _filePath << std::endl;
        if (_filePath == "") {
                // Warn that no environment variable was set
                std::cout << "No value for TRACER_OUTPUT_FILE_PATH set !" << std::endl;
                exit(1);
        }
        // Assert path is writable
}

// https://stackoverflow.com/q/631664/9876427
std::string SourceCodeTracer::get_env_var(std::string const & key) const {
        char * val;
        val = std::getenv(key.c_str());
        std::string retval = "";
        if (val != NULL) {
                retval = val;
        }
        return retval;
}

header file :

#include <iostream>
#include <string>
#include <fstream>
#include <list>
#include <map>
#include <memory>
#include <iomanip>
#include <ios>
#include <cstddef>
#include <cstdlib>


#ifndef __SOURCE_CODE_TRACER_H__
#define __SOURCE_CODE_TRACER_H__

using namespace std;
class SourceCodeTracer {
        public:
        ~SourceCodeTracer();

        static SourceCodeTracer& getInstance();
        void trace(const string& tr);
private:
        SourceCodeTracer();
        // https://stackoverflow.com/q/631664/9876427
        std::string get_env_var(std::string const & key) const;
        static SourceCodeTracer* _singleton;
        // Traceur() = delete;
        std::map<string, long> _map;
        string _filePath;
};

#endif // __SOURCE_CODE_TRACER_H__
KaHinCostner
  • 167
  • 3
  • 17
  • Are you compiling with -fPIC as the error message suggests? – Ted Lyngmo Jul 23 '19 at 21:01
  • @Yunnosch I removed the C tag. Indeed, it's a C++ question. – KaHinCostner Jul 23 '19 at 22:12
  • @TedLyngmo I tried the command on the class cpp : `g++ -std=c++11 -fPIC -shared SourceCodeTracer.cpp -o SourceCodeTracer.so` . Is it what you meant ? – KaHinCostner Jul 23 '19 at 22:15
  • 1
    The object files (for example `Activation.cpp.AVX2.cpp.o`) you put in the shared lib should have been compiled into object files using `-fPIC`. – Ted Lyngmo Jul 23 '19 at 22:29
  • 1
    Build with `make VERBOSE=1` and put that output in the question instead. – Ted Lyngmo Jul 24 '19 at 06:39
  • @TedLyngmo Alright, I updated the output, do you see anything wrong ? Ok, should I try compiling with -fPIC as you suggest ? Is it with the Makefile or with g++ ? (I am asking this to know if I need to compile the entire project or just the concerned directory) – KaHinCostner Jul 24 '19 at 15:43
  • 1
    I don't see anything wrong but you need to include the line where `Activation.cpp.AVX2.cpp.o` is being built too. Yes, you should compile the source files that becomes object files that are put in the shared library with `-fPIC`. You make the change in the `CMakeLists.txt` file. Not sure how though. `cmake` isn't my strong side. – Ted Lyngmo Jul 25 '19 at 11:35

0 Answers0