Running TensorFlow Serving #1

January 15, 2022•689 words

Steps

Keras モデルを TensorFlow Serving が扱えるフォーマットに変換する
TensorFlow Serving をローカルで実行する
画像を前処理、TensorFlow Serving と通信するサービスを構築する
これらのモデルを Kubernetes 上にデプロイする

次に Kubeflow を使用し、より簡単にデプロイできるようにする。

KFServing を使用したモデルのサービング
画像の前処理のための transformer と予測のための後処理

`saved_model` フォーマットへの変換

Keras でトレーニングしたモデルは h5 という形式で保存されているので、 saved_model というフォーマットに変換する。

# model_converter.py

import tensorflow as tf
from tensorflow import keras

model = keras.models.load_model('/path/to/model.h5')
tf.saved_model.save(model, 'clothing-model')

➜ pip install tensorflow

➜ python model_converter.py

clothing-model ディレクトリができて、その中に saved_model 形式のモデルが生成される。

➜ tree clothing-model
clothing-model
├── assets
├── saved_model.pb
└── variables
    ├── variables.data-00000-of-00001
    └── variables.index

TF Serving を利用する際、以下の情報が必要になるので、saved_model_cli で調べる

Model signature
入力層の名前
出力層の名前

➜ saved_model_cli show --dir clothing-model --all

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['__saved_model_init_op']:
  The given SavedModel SignatureDef contains the following input(s):
  The given SavedModel SignatureDef contains the following output(s):
    outputs['__saved_model_init_op'] tensor_info:
        dtype: DT_INVALID
        shape: unknown_rank
        name: NoOp
  Method name is: 

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input_8'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 299, 299, 3)
        name: serving_default_input_8:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['dense_7'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 10)
        name: StatefulPartitionedCall:0
  Method name is: tensorflow/serving/predict

Defined Functions:
  Function Name: '__call__'
    Option #1
      Callable with:
        Argument #1
          input_8: TensorSpec(shape=(None, 299, 299, 3), dtype=tf.float32, name='input_8')
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None
    Option #2
      Callable with:
        Argument #1
          inputs: TensorSpec(shape=(None, 299, 299, 3), dtype=tf.float32, name='inputs')
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None
    Option #3
      Callable with:
        Argument #1
          input_8: TensorSpec(shape=(None, 299, 299, 3), dtype=tf.float32, name='input_8')
        Argument #2
          DType: bool
          Value: True
        Argument #3
          DType: NoneType
          Value: None
    Option #4
      Callable with:
        Argument #1
          inputs: TensorSpec(shape=(None, 299, 299, 3), dtype=tf.float32, name='inputs')
        Argument #2
          DType: bool
          Value: True
        Argument #3
          DType: NoneType
          Value: None

  Function Name: '_default_save_signature'
    Option #1
      Callable with:
        Argument #1
          input_8: TensorSpec(shape=(None, 299, 299, 3), dtype=tf.float32, name='input_8')

  Function Name: 'call_and_return_all_conditional_losses'
    Option #1
      Callable with:
        Argument #1
          input_8: TensorSpec(shape=(None, 299, 299, 3), dtype=tf.float32, name='input_8')
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None
    Option #2
      Callable with:
        Argument #1
          input_8: TensorSpec(shape=(None, 299, 299, 3), dtype=tf.float32, name='input_8')
        Argument #2
          DType: bool
          Value: True
        Argument #3
          DType: NoneType
          Value: None
    Option #3
      Callable with:
        Argument #1
          inputs: TensorSpec(shape=(None, 299, 299, 3), dtype=tf.float32, name='inputs')
        Argument #2
          DType: bool
          Value: True
        Argument #3
          DType: NoneType
          Value: None
    Option #4
      Callable with:
        Argument #1
          inputs: TensorSpec(shape=(None, 299, 299, 3), dtype=tf.float32, name='inputs')
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None

先頭の

signature_def['serving_default']: # signature
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input_8'] tensor_info: # input layer
        dtype: DT_FLOAT
        shape: (-1, 299, 299, 3)
        name: serving_default_input_8:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['dense_7'] tensor_info: # output layer
        dtype: DT_FLOAT
        shape: (-1, 10)
        name: StatefulPartitionedCall:0
  Method name is: tensorflow/serving/predict

から

signatureの定義は serving_default
入力層は input_8 ()
出力層は dense_7 ()

というのがわかる。

Running TFServing locally

Docker で動かしてみる。

https://www.tensorflow.org/tfx/serving/docker

➜ docker run -it --rm \
    -p 8500:8500 \
    -v "$(pwd)/clothing-model:/models/clothing-model/1" \
    -e MODEL_NAME=clothing-model \
    tensorflow/serving:2.3.0

...
2022-01-15 02:24:12.894817: I tensorflow_serving/model_servers/server.cc:387] Exporting HTTP/REST API at:localhost:8501 ...
[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...

Entering the event loop ... が表示されたら正常に起動してリクエストを受け付けている状態になる。

Invoking the TF Serving model

➜ pip install grpcio tensorflow-serving-api

import grpc
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

from tensorflow_serving.apis import predict_pb2, prediction_service_pb2_grpc

HOST = 'localhost:8500'
channel = grpc.insecure_channel(HOST)
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)

def preprocess(name, url):
    img_url = tf.keras.utils.get_file(name, origin=url)
    img = tf.keras.utils.load_img(img_url, target_size=(299,299))
    plt.imshow(img)

    x = tf.keras.utils.img_to_array(img)
    batch = np.expand_dims(x, axis=0)
    batch /= 127.5
    batch -= 1.0
    return batch

def numpy_to_protobuf(data):
    """convert numpy array to protobuf"""
    return tf.make_tensor_proto(data, shape=data.shape)

URL = 'https://dl.dropboxusercontent.com/s/02q2qbwu4yweajm/17169926dg_29_f.jpg'
X = preprocess('margiela-boots-2', URL)

print(X.shape)

> (1, 299, 299, 3)

pb_req = predict_pb2.PredictRequest()

pb_req.model_spec.name = 'clothing-model'
pb_req.model_spec.signature_name = 'serving_default'
pb_req.inputs['input_8'].CopyFrom(numpy_to_protobuf(X))

result = stub.Predict(pb_req, timeout=20.0)
pred = result.outputs['dense_7'].float_val

labels = [
    'dress',
    'hat',
    'longsleeve',
    'outwear',
    'pants',
    'shirt',
    'shoes',
    'shorts',
    'skirt',
    't-shirt'
]

{c: p for c, p in zip(labels, pred)}

{'dress': -0.268576055765152,
 'hat': -2.0750882625579834,
 'longsleeve': -3.4710824489593506,
 'outwear': -1.9989770650863647,
 'pants': -0.18006758391857147,
 'shirt': -1.1924179792404175,
 'shoes': 5.7457098960876465,
 'shorts': -1.4281518459320068,
 'skirt': -1.4905409812927246,
 't-shirt': -0.20355109870433807}

Running TensorFlow Serving #1

Steps

`saved_model` フォーマットへの変換

Running TFServing locally

Invoking the TF Serving model

More from daigo3
All posts

Jupyter に Kernel を追加する

Running TensorFlow Serving #1

Steps

saved_model フォーマットへの変換

Running TFServing locally

Invoking the TF Serving model

More from daigo3All posts

Jupyter に Kernel を追加する

`saved_model` フォーマットへの変換

More from daigo3
All posts