python - 如何在 pyspark 管道阶段处理字符串索引器和 onehot 编码器

Question

面对此代码的此错误：

stage_string = [StringIndexer(inputCol=c, outputCol=c + "_string_encoded")  for c in categorical_columns]
stage_one_hot = [OneHotEncoder(inputCol=c + "_string_encoded", outputCol=c + "_one_hot") for c in categorical_columns]
assembler = VectorAssembler(inputCols=feature_list, outputCol="features")
rf = RandomForestClassifier(labelCol="output", featuresCol="features")
pipeline = Pipeline(stages=[stage_string,stage_one_hot,assembler, rf]) 
pipeline.fit(df)

Cannot recognize a pipeline stage of type <class 'list'>.
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/base.py", line 132, in fit
    return self._fit(dataset)
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 97, in _fit
    "Cannot recognize a pipeline stage of type %s." % type(stage))
TypeError: Cannot recognize a pipeline stage of type <class 'list'>.

标签： pythonmachine-learningpysparkpipeline

Answer 1

问题在于这个pipeline = Pipeline(stages=[stage_string,stage_one_hot,assembler, rf])语句stage_string，并且是和rfstage_one_hot的列表是单独的管道阶段。PipelineStageassembler

修改您的声明如下 -

stages = stage_string + stage_one_hot + [assembler, rf]
pipeline = Pipeline(stages=stages)