python - Tensorflow Multiple processes with less GPUs Model inference

Question

I have 2 GPUs and 3 deep learning models that I'd like to be able to perform inference on, and I'd really like to optimize the speed. Would it be the most optimal if I perform inference on 2 models from GPU 0 and then on the 3rd model from GPU 1?

If that's the case, I'm confused on how the code would look like for that. Some sample code below:

for model_idx in range(len(model_names)):
        # create thread for performing inference
        p = multiprocessing.Process(target=perform_inference_processed, args=(sess, 
                                                                      curr_graph,
                                                                      meta_graph_def,
                                                                      input_list,
                                                                      per_model_output))
        process_list.append(p)

As far as I know, each GPU should only have a single process running on it. If I structure my code as above, I would have 2 processes on GPU 0 (which I don't think is possible?). Is there any other way I can structure it instead? Can I control which process gets executed twice?

Thank you.

标签： pythonmultithreadingtensorflowmachine-learninggpu