python - 如何在 Tensorflow 中扩充文本数据集？

Question

我正在尝试通过添加一些单词的随机交换来增加 imdb 电影评论数据集。与图像数据不同，我不认为这个函数最初是在 tensorflow 中的。例如对于图像，您可以执行类似的操作

def transform(image, label):
    image = tf.image.flip_left_right(image)
    return image, label

在哪里使用 tensorflow 的原生函数来翻转图像。但是对于扩充文本，我在 tf.string 中看不到任何可以做到这一点的东西。所以我正在使用来自 textaugment 的 Easy Data Augmentation 实现。https://github.com/dsfsi/textaugment

例如：

try:
  import textaugment
except ModuleNotFoundError:
  !pip install textaugment
  import textaugment
from textaugment import EDA
import nltk
nltk.download('stopwords')

t = EDA()
t.random_swap("John is going to town")

返回“约翰要去城里”

但是现在当我尝试使用这个 random_swap 命令来扩充整个 imdb 评论数据集时，它会遇到错误，因为它试图对张量起作用。

例子：

try:
  import textaugment
except ModuleNotFoundError:
  !pip install textaugment
  import textaugment

import pandas as pd

import tensorflow as tf
from tensorflow.keras.preprocessing import sequence
from tensorflow.keras.models import Sequential
from tensorflow.keras.datasets import imdb

# set parameters:
max_features = 5000
maxlen = 400
batch_size = 32
embedding_dims = 50
filters = 250
kernel_size = 3
hidden_dims = 250
epochs = 1
runs = 1

(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
print(len(x_train), 'train sequences')
print(len(x_test), 'test sequences')

print('Pad sequences (samples x time)')
x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
x_test = sequence.pad_sequences(x_test, maxlen=maxlen)
print('x_train shape:', x_train.shape)
print('x_test shape:', x_test.shape)

from textaugment import EDA
import nltk
nltk.download('stopwords')
t = EDA()
for text in x_train:
  text = t.random_swap(text)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-7fc9edb2f37b> in <module>()
      1 for text in x_train:
----> 2   text = t.random_swap(text)

1 frames
/usr/local/lib/python3.7/dist-packages/textaugment/eda.py in validate(**kwargs)
     72                 raise TypeError("p must be a fraction between 0 and 1")
     73         if 'sentence' in kwargs:
---> 74             if not isinstance(kwargs['sentence'].strip(), str) or len(kwargs['sentence'].strip()) == 0:
     75                 raise TypeError("sentence must be a valid sentence")
     76         if 'n' in kwargs:

AttributeError: 'numpy.ndarray' object has no attribute 'strip'

那么，当本地命令不存在并且您想要创建自定义函数来进行扩充时，您如何在 TensorFlow 中扩充数据？

标签： pythontensorflowtextnlpdata-augmentation

Answer 1

通过加载数据集，imdb.load_data()您不会以文本形式获得电影评论。它已经过预处理：评论（单词序列）已被转换为整数序列，其中每个整数代表字典中的特定单词。

因此，您不能申请t.random_swap(text)。您必须首先将这些评论解码回英文单词。

因此，您需要相应的word_index. 它是将单词映射到整数索引的字典。

在下一步中，您应该反转它，以获取将整数索引映射到单词的字典。请注意，索引偏移 3，因为 0、1 和 2 是保留索引，用于填充序列的开始，并且未知。您可以在此处找到更多详细信息。

word_index = imdb.get_word_index()
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])

您应该在申请之前对评论进行解码sequence.pad_sequences()。否则评论中会有很多用零表示的未知词。

因为print(x_train[0])你会得到：

[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 2, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 2, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 2, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 2, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 2, 19, 178, 32]

让我们解码这篇评论：

decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in x_train[0]])

你会得到：

print(decoded_review)
>>> "? this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert ? is an amazing actor and now the same being director ? father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for ? and would recommend it to everyone to watch and the fly ? was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also ? to the two little ? that played the ? of norman and paul they were just brilliant children are often left out of the ? list i think because the stars that play them all grown up are such a big ? for the whole film but these children are amazing and should be ? for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was ? with us all"

在评论被解码回文本后，您可以使用t.random_swap(decoded_review). word_index可以使用字典将增强数据编码回整数序列。