r - 2个文档术语矩阵之间的余弦相似度

Question

这是对此处先前提出的问题的后续问题。这个帖子比我原来的要清楚一点。

我创建了两个文档术语矩阵（一个用于 2000，另一个用于 2001）。

          Terms
Docs       activity badminton court football per played racquet rugby shuttlecock side
  ID1_2000        0         0     0       12   0      0       0     3           0    0
  ID2_2000        1         2     2        0   2      3       1     0           1    2
  ID3_2000        1         2     2        0   2      4       2     0           2    2

和

          Terms
Docs       ball football game player players rugby side teams the two
  ID1_2001    1       10    0      0       0     3    0     0   0   0
  ID2_2001    3        0    2      1       2     0    3     2   2   4
  ID3_2001    3        0    1      3       1     0    0     1   2   2

我一直在关注这篇似乎和我有同样问题的帖子，但是我似乎无法让解决方案与我的数据一起工作。我收到以下错误：

cosine_sim <- tcrossprod_simple_triplet_matrix(dtm_t, dtm_Lt)/sqrt(row_sums(dtm_t^2) %*% t(row_sums(dtm_Lt^2))) .tcrossprod_simple_triplet_matrix(x, y) 中的错误：“x”和'y' 不符合

我要做的是计算两个文档术语矩阵之间的余弦相似度。这个例子有效，但我无法让它与我的数据一起工作。

library(slam)
library(tm)
data("acq")
data("crude")

dtm <- DocumentTermMatrix(c(acq, crude))

index <- sample(1:70, size = 10)

dtm1 <- dtm[index, ]
dtm2 <- dtm[-index, ]

cosine_sim <- tcrossprod_simple_triplet_matrix(dtm1, dtm2)/sqrt(row_sums(dtm1^2) %*% t(row_sums(dtm2^2)))
cosine_sim

数据/代码

text <- c("Football is a family of team sports that involve, to varying degrees, kicking a ball to score a goal. Unqualified, the word football is understood to refer to whichever form of football is the most popular in the regional context in which the word appears. Sports commonly called football in certain places include association football (known as soccer in some countries); gridiron football (specifically American football or Canadian football); Australian rules football; rugby football (either rugby league or rugby union); and Gaelic football.[1][2] These different variations of football are known as football codes.",
          "Football is a family of team sports that involve, to varying degrees, kicking a ball to score a goal. Unqualified, the word football is understood to refer to whichever form of football is the most popular in the regional context in which the word appears. Sports commonly called football in certain places include association football (known as soccer in some countries); gridiron football (specifically American football or Canadian football); Australian rules football; rugby football (either rugby league or rugby union); and Gaelic football.[1][2]",
          "Tennis is a racket sport that can be played individually against a single opponent (singles) or between two teams of two players each (doubles). Each player uses a tennis racket that is strung with cord to strike a hollow rubber ball covered with felt over or around a net and into the opponent's court. The object of the game is to maneuver the ball in such a way that the opponent is not able to play a valid return. The player who is unable to return the ball will not gain a point, while the opposite player will.",
          "Tennis is a racket sport that can be played individually against a single opponent (singles) or between two teams of two players each (doubles). Each player uses a tennis racket that is strung with cord to strike a hollow rubber ball covered with felt over or around a net and into the opponent's court. The object of the game is to maneuver the ball in such a way that the opponent is not able to play a valid return.",
          "Rugby refers to the team sports rugby league and rugby union. Legend claims that rugby football was started about 1845 in Rugby School, Rugby, Warwickshire, England, although forms of football in which the ball was carried and tossed date to medieval times. Rugby eventually split into two sports in 1895 when twenty-one clubs split from the original Rugby Football Union, to form the Northern Union (later to be named rugby league in 1922) in the George Hotel, Huddersfield, Northern England over the issue of payment to players, thus making rugby league the first code to turn professional and pay its players, rugby union turned fully professional in 1995. Both sports are run by their respective world governing bodies World Rugby (rugby union) and the Rugby League International Federation (rugby league). Rugby football was one of many versions of football played at English public schools in the 19th century.[1][2] Although rugby league initially used rugby union rules, they are now wholly separate sports. In addition to these two codes, both American and Canadian football evolved from rugby football.",
          "Badminton is. Although it may be played with larger teams, the most common forms of the game are singles (with one player per side) and doubles (with two players per side). Badminton is often played as a casual outdoor activity in a yard or on a beach; formal games are played on a rectangular indoor court. Points are scored by striking the shuttlecock with the racquet and landing it within the opposing side's half of the court.",
          "Cricket is a bat-and-ball game played between two teams of eleven players on a field at the centre of which is a 20-metre (22-yard) pitch with a wicket at each end, each comprising two bails balanced on three stumps. The batting side scores runs by striking the ball bowled at the wicket with the bat, while the bowling and fielding side tries to prevent this and dismiss each player (so they are out). Means of dismissal include being bowled, when the ball hits the stumps and dislodges the bails, and by the fielding side catching the ball after it is hit by the bat, but before it hits the ground. When ten players have been dismissed, the innings ends and the teams swap roles. The game is adjudicated by two umpires, aided by a third umpire and match referee in international matches. They communicate with two off-field scorers who record the match's statistical information.",
          "Basketball is a opposing one another on a rectangular court, compete with the primary objective of shooting a basketball (approximately 9.4 inches (24 cm) in diameter) through the defender's hoop (a basket 18 inches (46 cm) in diameter mounted 10 feet (3.048 m) high to a backboard at each end of the court) while preventing the opposing team from shooting through their own hoop. A field goal is worth two points, unless made from behind the three-point line, when it is worth three. After a foul, timed play stops and the player fouled or designated to shoot a technical foul is given one or more one-point free throws. The team with the most points at the end of the game wins, but if regulation play expires with the score tied, an additional period of play (overtime) is mandated.",
          "Football is a family of team sports that involve, to varying degrees, kicking a ball to score a goal. Unqualified, the word football is understood to refer to whichever form of football is the most popular in the regional context in which the word appears. Sports commonly called football in certain places include association football (known as soccer in some countries); gridiron football (specifically American football or Canadian football); Australian rules football; rugby football (either rugby league or rugby union); and Gaelic football.[1][2] These different variations of football are known as football codes.",
          "Badminton is a racquet sport played using racquets to hit a shuttlecock across a net. Although it may be played with larger teams, the most common forms of the game are singles (with one player per side) and doubles (with two players per side). Badminton is often played as a casual outdoor activity in a yard or on a beach; formal games are played on a rectangular indoor court. Points are scored by striking the shuttlecock with the racquet and landing it within the opposing side's half of the court.",
          "Tennis is a racket sport that can be played individually against a single opponent (singles) or between two teams of two players each (doubles). Each player uses a tennis racket that is strung with cord to strike a hollow rubber ball covered with felt over or around a net and into the opponent's court. The object of the game is to maneuver the ball in such a way that the opponent is not able to play a valid return. The player who is unable to return the ball will not gain a point, while the opposite player will.",
          "Rugby refers to the team sports rugby league and rugby union. Legend claims that rugby football was started about 1845 in Rugby School, Rugby, Warwickshire, England, although forms of football in which the ball was carried and tossed date to medieval times. Rugby eventually split into two sports in 1895 when twenty-one clubs split from the original Rugby Football Union, to form the Northern Union (later to be named rugby league in 1922) in the George Hotel, Huddersfield, Northern England over the issue of payment to players, thus making rugby league the first code to turn professional and pay its players, rugby union turned fully professional in 1995. Both sports are run by their respective world governing bodies World Rugby (rugby union) and the Rugby League International Federation (rugby league). Rugby football was one of many versions of football played at English public schools in the 19th century.[1][2] Although rugby league initially used rugby union rules, they are now wholly separate sports. In addition to these two codes, both American and Canadian football evolved from rugby football.",
          "Cricket is a bat-and-ball game played between two teams of eleven players on a field at the centre of which is a 20-metre (22-yard) pitch with a wicket at each end, each comprising two bails balanced on three stumps. The batting side scores runs by striking the ball bowled at the wicket with the bat, while the bowling and fielding side tries to prevent this and dismiss each player (so they are out). Means of dismissal include being bowled, when the ball hits the stumps and dislodges the bails, and by the fielding side catching the ball after it is hit by the bat, but before it hits the ground. When ten players have been dismissed, the innings ends and the teams swap roles. The game is adjudicated by two umpires, aided by a third umpire and match referee in international matches. They communicate with two off-field scorers who record the match's statistical information.")


ID <- c("ID1", "ID1", "ID1", "ID1", "ID1", "ID2", "ID2", "ID2", "ID2",  "ID3", "ID3", "ID3", "ID3")

years <- c("2000", "2001", "2002", "2003", "2004", "2000", "2001", "2002", "2003", "2000", "2001", "2002", "2003")

data <- data.frame(text, ID, years)




docs_df_Lt <- data %>%                              # Where Lt represents "lagged_t"
  filter(years %in% c("2000")) %>%
  mutate(doc_id = paste(ID, years, sep = "_")) %>%
  select(doc_id, text) %>%
  setNames(c("doc_id", "text")) %>%
  distinct(tolower(text), .keep_all = TRUE)

docs_df_t <- data %>%
  filter(years %in% c("2001")) %>%
  mutate(doc_id = paste(ID, years, sep = "_")) %>%
  select(doc_id, text) %>%
  setNames(c("doc_id", "text")) %>%
  distinct(tolower(text), .keep_all = TRUE)

docs_t = VCorpus(DataframeSource(docs_df_t))
docs_Lt = VCorpus(DataframeSource(docs_df_Lt))

# Soem document cleaning
docs_t <- tm_map(docs_t, removePunctuation)
docs_t <- tm_map(docs_t, removeWords, stopwords('english'))

docs_Lt <- tm_map(docs_Lt, removePunctuation)
docs_Lt <- tm_map(docs_Lt, removeWords, stopwords('english'))


dtm_t <- DocumentTermMatrix(docs_t)
dtm_Lt <- DocumentTermMatrix(docs_Lt)

dtm_t
dtm_Lt

inspect(dtm_t)
inspect(dtm_Lt)

编辑：

这让我更接近我想要实现的目标。

dtm <- DocumentTermMatrix(c(docs_t, docs_Lt))

m <- as.matrix(dtm)
dist.matrix = proxy::dist(m, method = "cosine") 
dist.matrix

输出：

           ID1_2001   ID2_2001   ID3_2001   ID1_2000   ID2_2000
ID2_2001 0.97192896                                            
ID3_2001 0.97288923 0.69527190                                 
ID1_2000 0.01505221 0.97565046 0.97648342                      
ID2_2000 1.00000000 0.75908178 0.77840308 1.00000000           
ID3_2000 1.00000000 0.77099402 0.76921180 1.00000000 0.05728332

这里的问题是我不需要/想要所有信息......即我对ID1_2001和ID2_2001（单元格1）的余弦不感兴趣。我只对以下内容感兴趣；

• ID1_2001并ID1_2000从第 1 列或 ( 0.01505221)。t那就是和t-1文档之间的相似性。

• ID2_2001并ID2_2000来自第 2 列，或 ( 0.75908178)。

• ID3_2001并ID3_2000来自第 3 列，或 ( 0.76921180)。

我确实计划在数据中使用更多年和更多 ID，因此当我对 90% 的文档不感兴趣时​​，计算所有文档的余弦似乎在计算上也很昂贵。

标签： r

Answer 1

忽略所有的tm东西，因为它似乎是无关紧要的，proxy::dist()有一个论点pairwise可以让你做你想做的事。

set.seed(1)
N <- 6*8
m <- matrix(sample(c(0, 1, 1), N, rep=TRUE)*rpois(N, 6), 6)
dimnames(m) <- list(c(paste0("ID", 1:3, "_2000"), paste0("ID", 1:3, "_2001")),
                     sample(LETTERS, ncol(m)))

library(proxy)
proxy::dist(m[1:3,], m[4:6,], pairwise=TRUE, method="cosine")
# 0.6160563 0.2746764 0.2038266

# Which is the same as
diag(proxy::dist(m[1:3,], m[4:6,], method="cosine"))
# 0.6160563 0.2746764 0.2038266