inna2.0 for Deep Learning

Introduction

inna2.0 is developed by AI & HPC Application R&D team of Inspur. It is a parallel version for multi-node GPU cluster, which is designed based on the NVIDIA/Caffe forked from the BVLC/caffe ( https://github.com/NVIDIA/caffe, more details please visit http://caffe.berkeleyvision.org).

Features

(1) The design basics

The inna2.0 is designed for high density GPU clusters; The new version supports InfiniBand (IB) high speed network connection and shared storage system that can be equipped by distributed file system, like NFS and GlusterFS. The training dataset is read in parallel for each process. The hierarchical communication mechanisms were developed to minimize the bandwidth requirements between computing nodes.

Updates of the inna2.0

Support NCCL 2.0

Both inter and intra node GPU communication are managed by NCCL with GPU direct RDMA.

(2) High performance and high scalability

The AlexNet, GoogLeNet and ResNet model have been tested with inna2.0 on a GPU cluster, which includes 4 nodes, and each of which has 4 P40 GPUs. The dataset is ImageNet. The speedup is 14.65X, 14.25X, 15.34X, for AlexNet (batchsize=1024), GoogLeNet (batchsize=128) and ResNet (batchsize=32) respectively on 4 nodes with 16 GPUs.

(3) Good inheritance and easy-using

inna2.0 retains all the features of the original Caffe architecture, namely the pure C++/CUDA architecture, support of the command line, Python interfaces, and various programming methods. As a result, the cluster version of the Caffe framework is user-friendly, fast, modularized and open, and gives users the optimal application experience.

Try your first inna2.0

This program can run 1 processes at least.

cifar10

1. Run data/cifar10/get_cifar10.sh to get cifar10 data.
2. Run examples/cifar10/create_cifar10.sh to conver raw data to leveldb format.
3. Run train_quick.sh to train the net.
4. Example of train_quick.sh script. run -host node1,node2 -mca btl_openib_want_cuda_gdr 1 --mca io ompio -np 2 -npernode 1 ./build/tools/caffe train --solver=examples/cifar10/cifar10_quick_solver.prototxt --gpu=0,1,2,3

Reference

• More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server

• Deep Image: Scaling up Image Recognition

Ask Questions

• For reporting bugs, please use the inna2.0/issues page or send email to us.

• Email address: wush@inspur.com

Author

• Shaohua Wu.