Compiling error for deepmd-gpu

[LuisStudio]

Hi, it occurs an error when I compile the deepmd-gpu version.
The environment as follows:
gcc version: 5.2.0
bazel version: 0.10.0
tensorflow version: 1.8
Python version: 3.6.5
cuda version: 9.0
cudnn version: 7.0.5

My command:
cmake -DTENSORFLOW_ROOT=/home/liuxy/version3/env/tensorflow -DCMAKE_INSTALL_PREFIX=/home/liuxy/version3/env/deepmd .. & make

The error as below:
[ 97%] Building CXX object ipi/CMakeFiles/dp_ipi.dir/driver.cc.o
[100%] Linking CXX executable dp_ipi
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::assign(char const*)@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_replace_aux(unsigned long, unsigned long, unsigned long, char)@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_framework.so: undefined reference to std::basic_istream<char, std::char_traits<char> >& std::operator>><char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_stringstream<char, std::char_traits, std::allocator >::~basic_stringstream()@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_framework.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::rfind(char, unsigned long) const@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::basic_string(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, unsigned long, unsigned long)@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long)@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to vtable for std::__cxx11::basic_stringstream<char, std::char_traits, std::allocator >@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find(char, unsigned long) const@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::substr(unsigned long, unsigned long) const@GLIBCXX_3.4.21'
../op/libdeepmd_op.so: undefined reference to tensorflow::internal::CheckOpMessageBuilder::NewString()' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_replace(unsigned long, unsigned long, char const*, unsigned long)@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::_M_sync(char*, unsigned long, unsigned long)@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_framework.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::compare(unsigned long, unsigned long, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to VTT for std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__throw_out_of_range_fmt(char const*, ...)@GLIBCXX_3.4.20'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find_last_not_of(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::append(char const*)@GLIBCXX_3.4.21'
../lib/libdeepmd.so: undefined reference to tensorflow::Status::ToString() const' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to __cxa_throw_bad_array_new_length@CXXABI_1.3.8'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::find(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_framework.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_erase(unsigned long, unsigned long)@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::rfind(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_append(char const*, unsigned long)@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to typeinfo for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21' ../lib/libdeepmd.so: undefined reference to tensorflow::ReadBinaryProto(tensorflow::Env*, std::string const&, google::protobuf::MessageLite*)'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(unsigned long, unsigned long, char const*) const@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to VTT for std::__cxx11::basic_istringstream<char, std::char_traits, std::allocator >@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::resize(unsigned long, char)@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::find_last_of(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long)@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to vtable for std::__cxx11::basic_istringstream<char, std::char_traits, std::allocator >@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_framework.so: undefined reference to std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>, void (*)())@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_framework.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::find_first_of(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_ostringstream()@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_framework.so: undefined reference to std::basic_istream<char, std::char_traits >& std::getline<char, std::char_traits, std::allocator >(std::basic_istream<char, std::char_traits >&, std::__cxx11::basic_string<char, std::char_traits, std::allocator >&, char)@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::str() const@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::swap(std::__cxx11::basic_string<char, std::char_traits, std::allocator >&)@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find_first_not_of(char const*, unsigned long, unsigned long) const@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_stringstream<char, std::char_traits, std::allocator >::basic_stringstream(std::_Ios_Openmode)@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::compare(char const*) const@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to vtable for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_construct(unsigned long, char)@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::random_device::_M_init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)@GLIBCXX_3.4.21' ../op/libdeepmd_op.so: undefined reference to tensorflow::strings::StrCat(tensorflow::strings::AlphaNum const&)'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to VTT for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >@GLIBCXX_3.4.21' /home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_create(unsigned long&, unsigned long)@GLIBCXX_3.4.21'
/home/liuxy/version3/env/tensorflow/lib/libtensorflow_cc.so: undefined reference to `vtable for std::__cxx11::basic_stringbuf<char, std::char_traits, std::allocator >@GLIBCXX_3.4.21'
collect2: error: ld returned 1 exit status

Do you know what's wrong with it and how to deal with it?
Thank you so much.

[amcadmus]

Did you use the same gcc to compile the C++ interface of tensorflow?

Could you please show me the file CMakeFiles/CMakeOutput.log?

[LuisStudio]

Did you use the same gcc to compile the C++ interface of tensorflow?

Could you please show me the file CMakeFiles/CMakeOutput.log?

Thank you for your quick reply.
I used the same gcc to compile the C++ interface of tensorflow.
Which part of CMakeOutput.log do you need? The full file contains 820 lines.

[amcadmus]

@LuisStudio you can attach files when you reply

[LuisStudio]

@LuisStudio you can attach files when you reply

CMakeOutput.log

[amcadmus]

@LuisStudio According to line 59 of the log, the compiler you are using is gcc 4.8.5
You may want to cmake in this way:
CC=/full/path/to/gcc CXX=/full/path/to/g++ cmake ..

You may also consult https://github.com/deepmodeling/deepmd-kit#troubleshooting for compiling with gcc4.8.5

[LuisStudio]

@LuisStudio According to line 59 of the log, the compiler you are using is gcc 4.8.5
You may want to cmake in this way:
CC=/full/path/to/gcc CXX=/full/path/to/g++ cmake ..

You may also consult https://github.com/deepmodeling/deepmd-kit#troubleshooting for compiling with gcc4.8.5

Thanks, your suggestion did work. But when I run the command to train the model, an error occurred.

Traceback (most recent call last):
File "/home/liuxy/version3/env/deepmd/bin/dp_train", line 16, in
from deepmd.Model import NNPModel
File "/home/liuxy/version3/env/deepmd/bin/../lib/deepmd/Model.py", line 21, in
op_module = tf.load_op_library(module_path + "libop_abi.so")
File "/home/liuxy/version3/env/tensorflow_venv/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 56, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/liuxy/version3/env/deepmd/lib/deepmd/libop_abi.so: undefined symbol: _ZN10tensorflow8internal21CheckOpMessageBuilder9NewStringB5cxx11Ev

Do you know how to fix it? Thank you so much.

[amcadmus]

@LuisStudio According to line 59 of the log, the compiler you are using is gcc 4.8.5
You may want to cmake in this way:
CC=/full/path/to/gcc CXX=/full/path/to/g++ cmake ..
You may also consult https://github.com/deepmodeling/deepmd-kit#troubleshooting for compiling with gcc4.8.5

Thanks, your suggestion did work. But when I run the command to train the model, an error occurred.

Traceback (most recent call last):
File "/home/liuxy/version3/env/deepmd/bin/dp_train", line 16, in
from deepmd.Model import NNPModel
File "/home/liuxy/version3/env/deepmd/bin/../lib/deepmd/Model.py", line 21, in
op_module = tf.load_op_library(module_path + "libop_abi.so")
File "/home/liuxy/version3/env/tensorflow_venv/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 56, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/liuxy/version3/env/deepmd/lib/deepmd/libop_abi.so: undefined symbol: _ZN10tensorflow8internal21CheckOpMessageBuilder9NewStringB5cxx11Ev

Do you know how to fix it? Thank you so much.

Please read this
https://github.com/deepmodeling/deepmd-kit#training-tensorflow-abi-binary-cannot-be-found-when-doing-training

[LuisStudio]

@LuisStudio According to line 59 of the log, the compiler you are using is gcc 4.8.5
You may want to cmake in this way:
CC=/full/path/to/gcc CXX=/full/path/to/g++ cmake ..
You may also consult https://github.com/deepmodeling/deepmd-kit#troubleshooting for compiling with gcc4.8.5

Thanks, your suggestion did work. But when I run the command to train the model, an error occurred.
Traceback (most recent call last):
File "/home/liuxy/version3/env/deepmd/bin/dp_train", line 16, in
from deepmd.Model import NNPModel
File "/home/liuxy/version3/env/deepmd/bin/../lib/deepmd/Model.py", line 21, in
op_module = tf.load_op_library(module_path + "libop_abi.so")
File "/home/liuxy/version3/env/tensorflow_venv/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 56, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/liuxy/version3/env/deepmd/lib/deepmd/libop_abi.so: undefined symbol: _ZN10tensorflow8internal21CheckOpMessageBuilder9NewStringB5cxx11Ev
Do you know how to fix it? Thank you so much.

Please read this
https://github.com/deepmodeling/deepmd-kit#training-tensorflow-abi-binary-cannot-be-found-when-doing-training

Thanks for your answer. I set -DTF_GOOGLE_BIN=true and now it occurs another error.

@LuisStudio According to line 59 of the log, the compiler you are using is gcc 4.8.5
You may want to cmake in this way:
CC=/full/path/to/gcc CXX=/full/path/to/g++ cmake ..
You may also consult https://github.com/deepmodeling/deepmd-kit#troubleshooting for compiling with gcc4.8.5

Thanks, your suggestion did work. But when I run the command to train the model, an error occurred.
Traceback (most recent call last):
File "/home/liuxy/version3/env/deepmd/bin/dp_train", line 16, in
from deepmd.Model import NNPModel
File "/home/liuxy/version3/env/deepmd/bin/../lib/deepmd/Model.py", line 21, in
op_module = tf.load_op_library(module_path + "libop_abi.so")
File "/home/liuxy/version3/env/tensorflow_venv/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 56, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/liuxy/version3/env/deepmd/lib/deepmd/libop_abi.so: undefined symbol: _ZN10tensorflow8internal21CheckOpMessageBuilder9NewStringB5cxx11Ev
Do you know how to fix it? Thank you so much.

Please read this
https://github.com/deepmodeling/deepmd-kit#training-tensorflow-abi-binary-cannot-be-found-when-doing-training

Thank you so much and I compiled it successfully.
But when I tried to train the example, an error occurred.
libgomp: Thread creation failed: Resource temporarily unavailable

Should I need to adjust the num_inter_threads or num_intra_threads?

[amcadmus]

@LuisStudio Please let us know if using fewer threads solves the problem.

[LuisStudio]

@LuisStudio Please let us know if using fewer threads solves the problem.

The problem is when we run the training program, we didn't see the process using nvidia-smi and the %CPU of the process is more than 700% and it caused the error "libgomp: Thread creation failed: Resource temporarily unavailable". And we still don't know how to fix it.

[amcadmus]

Please make sure that the GPU support of tensorflow is installed,
for python interface it is
pip install --upgrade tensorflow_gpu==1.8.0
for c++ interface please read
https://github.com/deepmodeling/deepmd-kit#build-deepmd-kit-with-gpu-support

[LuisStudio]

Please make sure that the GPU support of tensorflow is installed,
for python interface it is
pip install --upgrade tensorflow_gpu==1.8.0
for c++ interface please read
https://github.com/deepmodeling/deepmd-kit#build-deepmd-kit-with-gpu-support

Thank you for your help and I found the reason is I run the command pip install --upgrade tensorflow==1.8.0 instead of pip install --upgrade tensorflow-gpu==1.8.0 and now I can run the program via gpu.

I tested the example without modifying anything and when I used the water.json, the program runs ok but when I used the water_smth.json, it still occurs the error libgomp: Thread creation failed: Resource temporarily unavailable. Do you know the possible reason for this?