mlflow - 自定义python模型：加载成功但预测/服务失败

问题描述

我有一个自定义的 python 模型，它基本上设置了 scikit-learn 估计器的几个扰动。我确实成功地使用mlflow run project_directoryCLI 运行了项目，并使用save_model()语句保存了模型。它显示在仪表板上，带有mlflow ui。我什至可以在我的main.py脚本中加载保存的模型并在 pandas.DataFrame 上进行预测而没有任何问题。

当我尝试使用 of 时，我的问题就来mlflow models serve -m project/models/run_id了mlflow models predict -m project/models/run_id -i data.json。我收到以下错误：

ModuleNotFoundError: No module named 'multi_model'

在 MLflow 文档中，没有提供自定义模型的示例，因此我无法弄清楚如何解决此依赖问题。这是我的项目树：

project/
├── MLproject
├── __init__.py
├── conda.yaml
├── loader.py
├── main.py
├── models
│   └── 0ef267b0c9784a118290fa1ff579adbe
│       ├── MLmodel
│       ├── conda.yaml
│       └── python_model.pkl
├── multi_model.py

multi_model.py：

import numpy as np
from mlflow.pyfunc import PythonModel
from sklearn.base import clone

class MultiModel(PythonModel):

    def __init__(self, estimator=None, n=10):
        self.n = n
        self.estimator = estimator

    def fit(self, X, y=None):
        self.estimators = []
        for i in range(self.n):
            e = clone(self.estimator)
            e.set_params(random_state=i)
            X_bootstrap = X.sample(frac=1, replace=True, random_state=i)
            y_bootstrap = y.sample(frac=1, replace=True, random_state=i)
            e.fit(X_bootstrap, y_bootstrap)
            self.estimators.append(e)
        return self

    def predict(self, context, X):
        return np.stack([
            np.maximum(0, self.estimators[i].predict(X))
            for i in range(self.n)], axis=1
        )

main.py：

import os
import click
from sklearn.ensemble import RandomForestRegressor
import mlflow.pyfunc
import multi_model

@click(...) # define the click options according to MLproject file
def run(next_week, window_size, nfold):
    train = loader.load(start_week, current_week)
    x_train, y_train = train.drop(columns=['target']), train['target']

    model = multi_model.MultiModel(RandomForestRegressor())

    with mlflow.start_run() as run:
        model.fit(x_train, y_train)
        model_path = os.path.join('models', run.info.run_id)
        mlflow.pyfunc.save_model(
            path=model_path, 
            python_model=model,
        )

if __name__ == '__main__':
    run()

标签： mlflow

mlflow.pyfunc.save_model( path=model_path, python_model=model, code_path=['multi_model.py'], conda_env={ 'channels': ['defaults', 'conda-forge'], 'dependencies': [ 'mlflow=1.2.0', 'numpy=1.16.5', 'python=3.6.9', 'scikit-learn=0.21.3', 'cloudpickle==1.2.2' ], 'name': 'mlflow-env' } )

mlflow - 自定义python模型：加载成功但预测/服务失败

问题描述

解决方案

推荐阅读