1.Used Libraries, Datasets, and Models

1.1 Libraries

1. TensorFlow (from Google): https://www.tensorflow.org/
2. Theano (from U Montreal): https://github.com/Theano/Theano
3. Caffe (from Berkeley): http://caffe.berkeleyvision.org/
4. Torch (from Facebook): http://torch.ch/
5. Nervana Graph (from Intel):
   1. https://www.nervanasys.com/intel-nervana-graph-preview-release/
   2. https://github.com/NervanaSystems/ngraph
6. MXNet (from Amazon): http://mxnet.io/
7. Deeplearning4j (from Skymind): https://deeplearning4j.org/
8. CNTK (from Microsoft): https://github.com/Microsoft/CNTK/wiki
9. Keras (high-level API for both TensorFlow and Theano): https://keras.io/
10. TensorFlow-Slim (high-level API for TensorFlow)
    1. https://research.googleblog.com/2016/08/tf-slim-high-level-library-to-define.html
    2. https://github.com/tensorflow/models/tree/master/slim
11. Various other high-level APIs for Tensorflow, like TFLearn, TensorLayer
12. Spark + Caffe and/or TensorFlow:
    1. https://databricks.com/blog/2016/01/25/deep-learning-with-apache-spark-and-tensorflow.html
    2. https://github.com/yahoo/CaffeOnSpark
13. SparkNet with Caffe as backend for each worker (from Berkeley):
    1. https://arxiv.org/pdf/1511.06051v4.pdf
    2. https://github.com/amplab/SparkNet

1.2 Datasets

1. ImageNet (large scale image-classification; pre-training)
2. MS COCO (large scale object bounding box detection, image captioning, visual question answering)
3. Cifar10 (small scale image classification)
4. MNIST(small scale digits classification)
5. Visual Genome https://visualgenome.org/ (large scale multi-task image understanding)

1.3 Models Structure

1. Inception
2. ResNet
3. VGG
4. AlexNet
5. MobileNet
6. SqueezeNet

1.3 Available pre-trained model

1. tensorflow / slim 
   1) https://github.com/tensorflow/models/tree/master/slim (inceptionv1-v4; resnet50, 101, 152; vgg16,19; inception-resnet-v2)
2. caffe 
   1) https://github.com/BVLC/caffe/wiki/Model-Zoo (model zoo)
   2) https://github.com/beniz/deepdetect/issues/89 (resnet, inception-v1, vgg)
3. keras
   1) https://keras.io/applications/ (xception, vgg16, vgg19, resnet50, inceptionv3, mobilenet)
4. mxnet
   1) model galaxy https://github.com/dmlc/mxnet-model-gallery

2. TensorFlow Tips

2.1 Using GPUs

By default, TensorFlow eats all your GPUs' memory (Yes, every single bit of them!), even when you explicitly specify which gpu(s) to use. To limit this "intended" behavior, one needs to set GPUOptions to your session, and we recommend the following. One can refer to here for the rationale.

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0., allow_growth=True)

config=tf.ConfigProto(gpu_options=gpu_options, <your other configuraions>)

sess = tf.Session(config=config, <your other session settings>)

with sess.as_default():

    <rest of session code>

The above has been tested under TF version r0.12 on DL Workstation 1, 2, 3. The testing code is here (tested with both single-gpu training and multi-gpu training). Note that the code is different from the example provided in TF examples, and it is modified by us to be compatible with r0.12.

As of 2017/01/11, experimenting such GPU setting with Keras using TF backend is inconclusive. Further investigation is needed.

2.1.1 To limit the GPU to use

Use CUDA_VISIBLE_DEVICES, see http://stackoverflow.com/questions/34775522/tensorflow-mutiple-sessions-with-mutiple-gpus

$ CUDA_VISIBLE_DEVICES=0 python my_script.py  # Uses GPU 0.

$ CUDA_VISIBLE_DEVICES=1 python my_script.py  # Uses GPU 1.

$ CUDA_VISIBLE_DEVICES=2,3 python my_script.py  # Uses GPUs 2 and 3.

2.2 Upgrade to TF1.0

https://www.tensorflow.org/install/migration#how_to_upgrade