时间戳

首页 > 解决方案 > 在 Tensorflow 2 中分层批次

tensorflow - 在 Tensorflow 2 中分层批次

问题描述

我有从 sqlite 数据库中获得的小批量数据，其中包含整数和浮点类型的数据，x以及 0 和 1 中的二进制标签y。我正在寻找类似X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(y, x, test_size=0.1, random_state=1, stratify=True)scikit-learn 的东西，其中关键字可以对数据进行分层（即相同数量的 class-0 和 class-1 实例）。

在 Tensorflow 2 中，分层似乎并非一蹴而就。我非常复杂的解决方案对我有用，但由于所有的重塑和转置需要很多时间：

def stratify(x, y):
    # number of positive instances (the smaller class)
    pos = np.sum(y).item() # how many positive bonds there are
    x = np.transpose(x)

    # number of features 
    f = np.shape(x)[1] 

    # filter only class 1
    y = tf.transpose(y)
    x_pos = tf.boolean_mask(x, 
    y_pos = tf.boolean_mask(y, y)

    # filter only class 1
    x_neg = tf.boolean_mask(x, tf.bitwise.invert(y)-254)
    x_neg = tf.reshape(x_neg, [f,-1])
    y_neg = tf.boolean_mask(y, tf.bitwise.invert(y)-254)

    # just take randomy as many class-0 as there are class-1 
    x_neg = tf.transpose(tf.random.shuffle(tf.transpose(x_neg)))
    x_neg = x_neg[:,0:pos]
    y_neg = y_neg[0:pos]

    # concat the class-1 and class-0 together, then shuffle, and concat back together
    x = tf.concat([x_pos,tf.transpose(x_neg)],0)
    y = tf.concat([y_pos, tf.transpose(y_neg)],0)
    xy = tf.concat([tf.transpose(x), tf.cast(np.reshape(y,[1, -1]), tf.float64)],0)
    xy = tf.transpose((tf.random.shuffle(tf.transpose(xy)))) # because there is no axis arg in shuffle
    x = xy[0:f,:]
    x = tf.transpose(x)
    y = xy[f,:]

    return x, y

我很高兴看到对我自己的功能或新颖、更简单的想法的一些反馈/改进。

标签： tensorflowdeep-learning

解决方案

如果仅以原始格式或在将其转换为张量之前完成数据划分，则它是最好的。如果强烈要求只在 TensorFlow 中执行此操作，那么我建议您使用tf.data.Dataset类。我添加了演示代码以及解释步骤的相关注释。

import tensorflow as tf
import numpy as np

TEST_SIZE = 0.1
DATA_SIZE = 1000

# Create data
X_data = np.random.rand(DATA_SIZE, 28, 28, 1)
y_data = np.random.randint(0, 2, [DATA_SIZE])
samples1 = np.sum(y_data)
print('Percentage of 1 = ', samples1 / len(y_data))

# Create TensorFlow dataset
dataset = tf.data.Dataset.from_tensor_slices((X_data, y_data))

# Gather data with 0 and 1 labels separately
class0_dataset = dataset.filter(lambda x, y: y == 0)
class1_dataset = dataset.filter(lambda x, y: y == 1)

# Shuffle them
class0_dataset = class0_dataset.shuffle(DATA_SIZE)
class1_dataset = class1_dataset.shuffle(DATA_SIZE)

# Split them
class0_test_samples_len = int((DATA_SIZE - samples1) * TEST_SIZE)
class0_test = class0_dataset.take(class0_test_samples_len)
class0_train = class0_dataset.skip(class0_test_samples_len)

class1_test_samples_len = int(samples1 * TEST_SIZE)
class1_test = class1_dataset.take(class1_test_samples_len)
class1_train = class1_dataset.skip(class1_test_samples_len)

print('Train Class 0 = ', len(list(class0_train)), ' Class 1 = ', len(list(class1_train)))
print('Test Class 0 = ', len(list(class0_test)), ' Class 1 = ', len(list(class1_test)))

# Gather datasets
train_dataset = class0_train.concatenate(class1_train).shuffle(DATA_SIZE)
test_dataset = class0_test.concatenate(class1_test).shuffle(DATA_SIZE)

print('Train dataset size = ', len(list(train_dataset)))
print('Test dataset size = ', len(list(test_dataset)))

样本输出：

Percentage of 1 =  0.474
Train Class 0 =  474  Class 1 =  427
Test Class 0 =  52  Class 1 =  47
Train dataset size =  901
Test dataset size =  99

推荐阅读