Environment: Ubuntu 16.04, Python 2.7, GTX1060, opencv 3.0.0, CUDA 8.0, cuDNN 5.1

1. Install Dependencies and Numpy

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential cmake git pkg-config libjpeg8-dev \
libjasper-dev libpng12-dev libgtk2.0-dev \
libavcodec-dev libavformat-dev libswscale-dev libv4l-dev gfortran
sudo apt-get install libtiff5-dev
sudo apt-get install libatlas-base-dev
sudo apt-get install python2.7-dev

Choose the numpy version 1.11.0, because the py-faster-rcnn was based on that version, the later version will caused error.

sudo pip install numpy==1.11.0

2. Install opencv 3.0

Use git to checkout the 3.0.0 version.

cd ~
git clone https://github.com/Itseez/opencv.git
cd opencv
git checkout 3.0.0
cd ~
git clone https://github.com/Itseez/opencv_contrib.git
cd opencv_contrib
git checkout 3.0.0
cd ~/opencv
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
    -D CMAKE_INSTALL_PREFIX=/usr/local \
    -D INSTALL_C_EXAMPLES=ON \
    -D INSTALL_PYTHON_EXAMPLES=ON \
    -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
    -D BUILD_EXAMPLES=ON ..

make
sudo make install
sudo ldconfig

Check:

python
>>> import cv2
>>> cv2.__version__
'3.0.0'

2. Install CUDA, cuDNN

Download and install CUDA, choose version 8.0:

https://developer.nvidia.com/cuda-80-ga2-download-archive.

Download cuDNN, choose version 5.1

https://developer.nvidia.com/cudnn

After the download process completes, extract the downloaded file:

cd ~/Downloads
tar -xvf cudnn-7.0-linux-x64-v4.0-prod.tgz

Copy the two extracted folders to where CUDA was installed:

sudo cp lib64/* /usr/local/cuda/lib64
sudo cp include/* /usr/local/cuda/include

3. Install py-faster-rcnn

Install dependencies:

sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libboost-all-dev libhdf5-serial-dev \
libgflags-dev libgoogle-glog-dev liblmdb-dev protobuf-compiler
sudo apt-get install -y --no-install-recommends libboost-all-dev

Follow the github readme:

https://github.com/rbgirshick/py-faster-rcnn

# Make sure to clone with --recursive
git clone --recursive https://github.com/rbgirshick/py-faster-rcnn.git

Build the Cython modules

cd $FRCN_ROOT/lib
make

Install cython, python-opencv, easydict.

For the support of cuDNN v5.1, we need to merge caffe master branch into caffe-fast-rcnn.

cd caffe-fast-rcnn  
git remote add caffe https://github.com/BVLC/caffe.git  
git fetch caffe  
git merge -X theirs caffe/master  

Remove self_.attr("phase") = static_cast<int>(this->phase_); from include/caffe/layers/python_layer.hpp after merging.

After merging, create the Makefile.config file from the example file and modify it.

# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1

# Uncomment if you're using OpenCV 3
OPENCV_VERSION := 3

# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /usr/include/python2.7 \
        /usr/local/lib/python2.7/dist-packages/numpy/core/include

# Uncomment to support layers written in Python (will link against    Python libs)
WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial/

Build Caffe and pycaffe:

mkdir build
cd build
cmake ..
make all -j8
make pycaffe -j8

4. Test py-faster-rcnn

Download pre-computed Faster R-CNN detectors:

cd $FRCN_ROOT
./data/scripts/fetch_faster_rcnn_models.sh

Run the demo:

cd $FRCN_ROOT
./tools/demo.py

Reference:

[1] https://chunml.github.io/ChunML.github.io/project/Installing-Caffe-Ubuntu/

[2] https://github.com/rbgirshick/py-faster-rcnn

[3] http://caffe.berkeleyvision.org/installation.html

[4] https://github.com/rbgirshick/py-faster-rcnn/issues/237

[5] https://github.com/rbgirshick/py-faster-rcnn/issues/198

[6] http://blog.csdn.net/nicky_lyu/article/details/53181434