r - sparklyr下的值匹配

Question

我尝试在 sparklyr 下使用值匹配：

spark_parquet %>% filter(customer_id %in% spark_unique_customer_ids)

但是，我收到以下错误：

Error in UseMethod("escape") : no applicable method for 'escape' applied to an object of class "c('tbl_spark', 'tbl_sql', 'tbl_lazy', 'tbl')"

任何建议如何解决这个问题？

标签： rapache-sparkdplyrmatchingsparklyr

Answer 1

最好使用semi_join：

spark_parquet <- copy_to(
  sc, 
  tibble(customer_id = c(1, 2, 3), value = c(-1, 0, 1))
)
spark_unique_customer_ids <- copy_to(sc, tibble(customer_id = c(1, 3)))

spark_parquet %>% semi_join(spark_unique_customer_ids, by = "customer_id")

# Source: spark<?> [?? x 2]
  customer_id value
*       <dbl> <dbl>
1           1    -1
2           3     1

虽然 SQL API 也应该可以工作：

spark_parquet %>%  sdf_register("spark_parquet")
spark_unique_customer_ids %>% sdf_register("spark_unique_customer_ids")

sc %>% spark_session() %>% 
  invoke(
    "sql", 
    "SELECT * FROM spark_parquet 
     WHERE customer_id IN (
       SELECT customer_id FROM spark_unique_customer_ids)") %>% 
   sdf_register()

# Source: spark<?> [?? x 2]
  customer_id value
*       <dbl> <dbl>
1           1    -1
2           3     1