tensorflow - amazon sagemaker 上的 base64 编码图像推断

Question

SageMaker 菜鸟在这里。
问这个问题是因为我无法直接找到任何可行的示例，并且信息非常分散在各个站点中。这是我的任务描述：我正在尝试在 Amazon Sagemaker 中部署模型。该模型是使用 keras(tf<2.0) 构建和训练的。到目前为止，我遵循了上传和部署模型的标准教程并让模型正常工作。现在这是我的推理代码：

import os
import numpy as np 
from tensorflow.keras.preprocessing.image import load_img, img_to_array
import cv2
from sagemaker.tensorflow import TensorFlowPredictor
import multiprocessing as mp
def get_result(payload,original_shape):
    predictor = TensorFlowPredictor(endpoint_name = "SOME_SAGEMAKER_ENDPOINT")
    op = np.asarray(predictor.predict(pixels)["predictions"][0])
    pix2pix_version=cv2.normalize(op,None,0,1,cv2.NORM_MINMAX)*255
    outImg = pix2pix_version.astype(np.uint8)
    outImg = cv2.cvtColor(outImg,cv2.COLOR_RGB2BGR)
    outImg = cv2.resize(outImg,(original_shape[1],original_shape[0]))
    cv2.imwrite("SOME_OUTPUT_IMAGE_PATH",outImg)
pixels = load_img("SOME_IMAGE_PATH")
pixels = img_to_array(pixels)
original_shape = pixels.shape
pixels = cv2.resize(pixels,(512,512))
pixels=(pixels/255)*2-1 #Required for preprocessing
pixels = np.resize(pixels,(1,512,512,3))
print("starting")
get_result(pixels.tolist(),original_shape)

这目前工作正常。
但是这种方法不适用于较大的图像，因为 sagemaker 的输入数据限制为 5MB（我猜是 json 编码数据）。因此，经过一些研究，我发现我们可以使用application/x-image内容标题直接提供图像。然后我们可以根据需要使用自定义的inference.py来解码图像。经过更多的挖掘，我发现我实际上可以通过修改输入层并添加一些预处理来将一些 base64 编码的字符串提供给我的模型。所以我用这段代码做到了：

import tensorflow as tf
tf.compat.v1.disable_eager_execution()
sess = tf.compat.v1.Session() # get the tensorflow session to reuse it in keras

from tensorflow.compat.v1.keras import backend as K
from tensorflow.compat.v1.keras.models import load_model, Model

K.set_session(sess)
K.set_learning_phase(0) # make sure we disable dropout and other training specific layers

string_inp = tf.compat.v1.placeholder(tf.string, shape=(None,)) #string input for the base64 encoded image
imgs_map = tf.map_fn(
    tf.image.decode_image,
    string_inp,
    dtype=tf.uint8
) # decode jpeg
imgs_map.set_shape((None, None, None, 3))
imgs = tf.compat.v1.image.resize_images(imgs_map, [512, 512]) # resize images
imgs = tf.compat.v1.reshape(imgs, (-1, 512, 512, 3)) # reshape them 
img_float = (tf.cast(imgs, dtype=tf.float32) / 255)*2 - 1.0 # and make them to floats

model = load_model('MODEL_PATH', compile=False) # load the model
w = model.get_weights()
output = model(img_float) # use the image tensor as input for keras
# ...(save to savedModel format and load in tensorflow serve)
builder = tf.compat.v1.saved_model.builder.SavedModelBuilder('cnn')

tensor_info_input = tf.compat.v1.saved_model.utils.build_tensor_info(string_inp)
tensor_info_output = tf.compat.v1.saved_model.utils.build_tensor_info(output)

# we need to init all missing placeholders
sess.run(tf.compat.v1.local_variables_initializer()) 
sess.run(tf.compat.v1.global_variables_initializer())

#set the weights to make sure they are not somehow changed by the init we did before
#single_gpu.set_weights(w)

# define the signature
signature = tf.compat.v1.saved_model.signature_def_utils.predict_signature_def(                                                                        
    inputs={'b64': string_inp}, outputs={'predictions': output})

#finally save the model
builder.add_meta_graph_and_variables(                                                                                                        
    sess=K.get_session(),                                                                                                                    
    tags=[tf.compat.v1.saved_model.tag_constants.SERVING],                                                                                             
    signature_def_map={                                                                                                                      
        tf.compat.v1.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:                                                                
            signature                                                                                                                        
    })                                                                                                                                       
builder.save()

这将 keras 模型转换为 SavedModel 格式，并将其放在 cnn/ 目录中。之后，我将 cnn 目录上传到我的 SageMaker 笔记本环境并运行!saved_model_cli show --all --dir {model_path} Here Model path is export20/Servo/1。这是我放置模型的地方。
saved_model_cli 的输出

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['b64'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: Placeholder:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['predictions'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 512, 512, 3)
        name: model_2/activation_10/Tanh:0
  Method name is: tensorflow/serving/predict

此外，对于预处理，我需要一个 inference.py，我将它放在代码目录中。
推理.py

import base64
import io
import json
import requests

def input_handler(data, context):
    """ Pre-process request input before it is sent to TensorFlow Serving REST API
    Args:
        data (obj): the request data stream
        context (Context): an object containing request and configuration details
    Returns:
        (dict): a JSON-serializable dict that contains request body and headers
    """
    if context.request_content_type == 'application/x-image':
        payload = data.read()
        encoded_image = base64.b64encode(payload).decode('utf-8')
        instance = [{"b64": encoded_image}]
        return json.dumps({"instances": instance})
    else:
        _return_error(415, 'Unsupported content type "{}"'.format(context.request_content_type or 'Unknown'))


def output_handler(response, context):
    """Post-process TensorFlow Serving output before it is returned to the client.
    Args:
        response (obj): the TensorFlow serving response
        context (Context): an object containing request and configuration details
    Returns:
        (bytes, string): data to return to client, response content type
    """
    if response.status_code != 200:
        _return_error(response.status_code, response.content.decode('utf-8'))
    response_content_type = context.accept_header
    prediction = response.content
    return prediction, response_content_type


def _return_error(code, message):
    raise ValueError('Error: {}, {}'.format(str(code), message))

然后我对代码和 export20 目录进行了 tar 处理。
tar 目录结构

--code/->inference.py
--export20/->Servo/->1/->Saved_model.pb
                       ->Variables
 

然后我部署了模型。该模型的部署没有问题。
部署代码：

model_data = sess.upload_data(path=model_archive, key_prefix='model')
tf_framework_version = tf.__version__
sm_model = Model(model_data=model_data, framework_version=tf_framework_version,role=role)
uncompiled_predictor = sm_model.deploy(initial_instance_count=1,  instance_type=instance_type)

现在我的推理代码是

import boto3
runtime = boto3.Session().client(service_name='runtime.sagemaker')
with open("SOME_IMAGE.jpg","rb") as f:
    data= f.read()
response = runtime.invoke_endpoint(EndpointName=SOME_ENDPOINT,
                                   ContentType='application/x-image',
                                   Body=bytearray(data))

但我收到一个错误

An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from model with message "{"error": "Error: 400, { \"error\": \"Specified a list with shape [1] from a tensor with shape []\\n\\t [[{{node model_5/lambda_8/map/TensorArrayUnstack/TensorListFromTensor}}]]\" }"}".

所以我的问题是，我在哪里犯错了？如果有人可以提供一些指导，我将不胜感激。

TIA

标签： tensorflowkerasdeep-learningamazon-sagemaker