This is a tensorflow implementation of Deformable Convolutional Network in Faster R-CNN fashion. This project is largely built on TFFRCNN, the original implementation in mxnet and many other upstream projects. This repository is only in test phase right now, any contributions helping with bugs and compatibility issues are welcomed.

• TF_Deformable_Net
• TODO
• Requirements: software
• Requirements: Hardware
• Installation (sufficient for the demo)
• Demo
  • Download list
• Training
• Testing
• FAQ

• Faster R-CNN

• Trained on ResNet-50

• More Networks

• Potential bugs

• R-FCN

Python 3 (Insufficient compatibility guaranteed for Python 2, though TFFRCNN is built on python 2, I use ilovin's refactored version as base and add some __future__ imports, so any report on compatibility issues welcomed)

Tensorflow==1.4 (Build from source recommended, or else you might need to check ./lib/cuda_config.h to fill in some options.)

matplotlib

python-opencv

easydict

scikit-image

cython

g++ 4.9(For gcc5 users, you should check make.sh and modify it as told. But there are still chances you might failed with some undefined symbol message, I haven't figured it out totally. Therefore g++ 4.9 should be the safest.)

Cuda 8.0

Cudnn 6.0

For cuda and cudnn of a different version, please modify ./lib/cuda_config.h accordingly, but I don't give the guarantee it will work.

Any NVIDIA GPUs with at least 4GB memory should be OK.(Only single gpu mode supported, if you encounter any memory issue on a multi-gpu machine, try export $CUDA_VISIBLE_DEVICE=$(the gpu id you want to use)).

1. Clone this repository

   git clone https://github.com/Zardinality/TF_Deformable_Net.git
   cd TF_Deformable_Net

2. Before setup all the Cython module and GPU kernels, you might want to set some parameter specific to your GPU model by hand. For example, if you install Tensorflow in binary way, the CUDA_CAPABILITIES of your GPU model should be filled in in the following file:

   • ./lib/cuda_config.h
   • ./lib/make.sh

   Cuda and Cudnn version, plus the path to Cuda toolkit should also be filled in ./lib/cuda_config.h.

   Note that the build of roi_pooling and psroi_pooling in make.sh are not necessarily needed, because deformable_psroi_pooling can work as them in many ways, also, tensorflow already has tf.image.crop_and_resize as a faster version for roi_pooling).

3. setup all the Cython module and GPU kernels.

   cd ./lib
   make

After successfully completing basic installation, you'll be ready to run the demo.

To run the demo

cd $TF_Deformable_Net
python ./faster_rcnn/demo.py --model model_path
# e.g. python faster_rcnn/demo.py --model ~/tf_deformable_net/restore_output/Resnet50_iter_145000.ckpt

Also, for many people work on a remote machine, an ipython notebook version demo and test are provided in case visually debug is needed. Except for that all arguments are made in an easydict object, the ipython notebook version demo and test are the same as the script version ones.

The demo performs detection using a ResNet50 network trained for detection on PASCAL VOC 2007. , where the model can be download below. Note that since TF 0.12 the checkpoint must contain more than one file, so you need to unzip the downloaded model to a folder whose path is model_path. Also, when you restore, be sure to set the checkpoint name to be the name you save them(no matter what suffix the checkpoint files now have).

1. VGG16 trained on ImageNet
2. Resnet50 trained on ImageNet
3. Resnet50 Model(map@0.5 66%)

1. Download the training, validation, test data and VOCdevkit

   wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar
   wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar
   wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCdevkit_08-Jun-2007.tar

2. Extract all of these tars into one directory named VOCdevkit

   tar xvf VOCtrainval_06-Nov-2007.tar
   tar xvf VOCtest_06-Nov-2007.tar
   tar xvf VOCdevkit_08-Jun-2007.tar

3. It should have this basic structure

   $VOCdevkit/                           # development kit
   $VOCdevkit/VOCcode/                   # VOC utility code
   $VOCdevkit/VOC2007                    # image sets, annotations, etc.
   # ... and several other directories ...

4. Create symlinks for the PASCAL VOC dataset

   cd $TF_Deformable_Net/data
   ln -s $VOCdevkit VOCdevkit2007

5. Download pre-trained model VGG16 or Resnet50 and put it in the path ./data/pretrain_model

6. Run training scripts

   cd $TF_Deformable_Net
   # for resnet-50
   python ./faster_rcnn/train_net.py --gpu 0 --weights ./data/pretrain_model/Resnet50.npy --imdb voc_2007_trainval --iters 70000 --cfg  ./experiments/cfgs/faster_rcnn_end2end_resnet.yml --network Resnet50_train --set EXP_DIR exp_dir
   # for vggnet
   python ./faster_rcnn/train_net.py --gpu 0 --weights ./data/pretrain_model/VGG_imagenet.npy --imdb voc_2007_trainval --iters 70000 --cfg  ./experiments/cfgs/faster_rcnn_end2end.yml --network VGGnet_train --set EXP_DIR exp_dir

   Or equivalently, edit scripts in ./experiments/scripts, and start training by running shell scripts. For example:

   # for resnet-50
   ./experiments/scripts/faster_rcnn_voc_resnet.sh
   # for vggnet
   ./experiments/scripts/faster_rcnn_vggnet.sh

7. Run a profiling

   cd $TF_Deformable_Net
   # install a visualization tool
   sudo apt-get install graphviz  
   ./experiments/profiling/run_profiling.sh 
   # generate an image ./experiments/profiling/profile.png

After training, you could run scripts in ./experiments/evalto evaluate on VOC2007. Or by running ./faster_rcnn/test_net.py directly.

# for resnet-50
./experiments/eval/voc2007_test_res.sh
# for vggnet
./experiments/scripts/voc2007_test_vgg.sh

1. cudaCheckError() failed : invalid device function.

   Check ./lib/make.sh and change the -arch flag accordingly. (Credit to here)

2. undefined symbol: _ZN10tensorflow8internal21CheckOpMessageBuilder9NewStringB5cxx11Ev

   If you use gcc5 to build, modify make.sh to gcc5 version(simply adding a -D_GLIBCXX_USE_CXX11_ABI=0 flag as pointed out in this issue). If it doesn't work, try reinstall gcc4.9 as following:

   • installing gcc-4.9 and g++-4.9

   • changing all nvcc related command in make.sh to this form:

     nvcc -std=c++11 -ccbin=/usr/bin/g++-4.9 -c -o deform_conv.cu.o deform_conv.cu.cc -I $TF_INC -D GOOGLE_CUDA=1 -x cu -Xcompiler -fPIC -L /usr/local/cuda-8.0/lib64/ --expt-relaxed-constexpr

   • and changing all g++ related command in make.sh to this form:

     g++-4.9 -std=c++11 -shared -o deform_conv.so deform_conv.cc deform_conv.cu.o -I TF_INC -fPIC -lcudart -L CUDA_HOME/lib64 -D GOOGLE_CUDA=1 -Wfatal-errors -I $CUDA_HOME/include -D_GLIBCXX_USE_CXX11_ABI=0

credit to @cotrane in this issue