pandas - 熊猫选择列不同的帧

Question

我有一堆 csv 文件。每个 csv 文件来自一台机器，每个 csv 文件的纪元时间大致相同。

我想完成几件事：

1）绘制每台机器的同一列。因此，我可以在该机器上的某些指标的机器之间进行比较。例如内存使用或吞吐量。

2）“求和”每台机器的列。例如，如果每台机器每秒执行 100 次上下文切换并且我有 10 台机器，我希望将这些列合并到每个“集群”的单个“1000”上下文切换中。通过这种方式，我可以轻松了解所有机器上“x”的总数。

3）当我运行多个基准测试时，无法比较数据的纪元时间。那么更改偏移量的最简单方法是什么？我是否应该使用索引列（例如减去每个列值的偏移量）。我对此并不太疯狂，因为如果我修改索引值，我就会丢失数据。

我已经启动并运行了第一个，但是标签有一些严重的问题（所以我看不到任何标签）。

通过从每一列中减去每个测试的起始值，我还启动并运行了最后一个。因此，如果它们是例如 [100,104, 120]，则在清理后它们将减少到 [0,4,20]。但是手动迭代 aa 列的内容并不像他们打算使用 Panda 的方式。

import pandas as pd
import matplotlib.pyplot as plt

fr1 = pd.read_csv(
    "/home/pveentjer/PycharmProjects/report-generator/20KB/A2_W1-54.161.173.44-javaclient/performance-IntByteMapTest.csv")
fr2 = pd.read_csv(
    "/home/pveentjer/PycharmProjects/report-generator/20KB/A3_W1-54.167.139.194-javaclient/performance-IntByteMapTest.csv")

fr1["operations/second"].plot(label="foo")
fr2["operations/second"].plot(label="bar")

plt.savefig("foo.png")

样本数据机1

epoch,timestamp,operations,operations-delta,operations/second
1519997386.971,02/03/2018 13:29:46,11596,11596,11584.416
1519997387.971,02/03/2018 13:29:47,17935,6339,6339
1519997388.971,02/03/2018 13:29:48,24145,6210,6210
1519997389.972,02/03/2018 13:29:49,30356,6211,6204.795
1519997390.972,02/03/2018 13:29:50,36566,6210,6210
1519997391.972,02/03/2018 13:29:51,42777,6211,6211
1519997392.972,02/03/2018 13:29:52,48987,6210,6210
1519997393.972,02/03/2018 13:29:53,55199,6212,6212
1519997394.972,02/03/2018 13:29:54,61409,6210,6210
1519997395.972,02/03/2018 13:29:55,67620,6211,6211
1519997396.973,02/03/2018 13:29:56,73789,6169,6162.837
1519997397.973,02/03/2018 13:29:57,80038,6249,6249
1519997398.973,02/03/2018 13:29:58,86248,6210,6210
1519997399.973,02/03/2018 13:29:59,92454,6206,6206
1519997400.973,02/03/2018 13:30:00,98663,6209,6209
1519997401.973,02/03/2018 13:30:01,104870,6207,6207
1519997402.974,02/03/2018 13:30:02,111081,6211,6204.795
1519997403.974,02/03/2018 13:30:03,117283,6202,6202
1519997404.974,02/03/2018 13:30:04,123498,6215,6215
1519997405.974,02/03/2018 13:30:05,129705,6207,6207
1519997406.974,02/03/2018 13:30:06,135913,6208,6208
1519997407.974,02/03/2018 13:30:07,142091,6178,6178
1519997408.974,02/03/2018 13:30:08,148327,6236,6236
1519997409.975,02/03/2018 13:30:09,154536,6209,6202.797
1519997410.975,02/03/2018 13:30:10,160746,6210,6210
1519997411.975,02/03/2018 13:30:11,166956,6210,6210
1519997412.975,02/03/2018 13:30:12,173167,6211,6211
1519997413.975,02/03/2018 13:30:13,179376,6209,6209
1519997414.975,02/03/2018 13:30:14,185586,6210,6210
1519997415.975,02/03/2018 13:30:15,191796,6210,6210
1519997416.976,02/03/2018 13:30:16,198007,6211,6204.795
1519997417.976,02/03/2018 13:30:17,204217,6210,6210
1519997418.976,02/03/2018 13:30:18,210429,6212,6212
1519997419.976,02/03/2018 13:30:19,216636,6207,6207
1519997420.976,02/03/2018 13:30:20,222846,6210,6210
1519997421.976,02/03/2018 13:30:21,229057,6211,6211
1519997422.976,02/03/2018 13:30:22,235266,6209,6209
1519997423.977,02/03/2018 13:30:23,241476,6210,6203.796
1519997424.977,02/03/2018 13:30:24,247685,6209,6209
1519997425.977,02/03/2018 13:30:25,253893,6208,6208
1519997426.977,02/03/2018 13:30:26,260104,6211,6211
1519997427.977,02/03/2018 13:30:27,266312,6208,6208
1519997428.977,02/03/2018 13:30:28,272522,6210,6210
1519997429.977,02/03/2018 13:30:29,278732,6210,6210
1519997430.978,02/03/2018 13:30:30,284943,6211,6204.795
1519997431.978,02/03/2018 13:30:31,291154,6211,6211
1519997432.978,02/03/2018 13:30:32,297363,6209,6209
1519997433.978,02/03/2018 13:30:33,303574,6211,6211
1519997434.978,02/03/2018 13:30:34,309785,6211,6211
1519997435.978,02/03/2018 13:30:35,315996,6211,6211
1519997436.978,02/03/2018 13:30:36,322205,6209,6209
1519997437.979,02/03/2018 13:30:37,328418,6213,6206.793
1519997438.979,02/03/2018 13:30:38,334628,6210,6210
1519997439.979,02/03/2018 13:30:39,340839,6211,6211
1519997440.979,02/03/2018 13:30:40,347049,6210,6210
1519997441.979,02/03/2018 13:30:41,353260,6211,6211
1519997442.979,02/03/2018 13:30:42,359470,6210,6210
1519997443.979,02/03/2018 13:30:43,365681,6211,6211
1519997444.98,02/03/2018 13:30:44,371889,6208,6201.798
1519997445.98,02/03/2018 13:30:45,378099,6210,6210
1519997446.98,02/03/2018 13:30:46,384310,6211,6211
1519997447.98,02/03/2018 13:30:47,390518,6208,6208
1519997448.98,02/03/2018 13:30:48,396729,6211,6211
1519997449.98,02/03/2018 13:30:49,402939,6210,6210
1519997450.98,02/03/2018 13:30:50,409149,6210,6210
1519997451.98,02/03/2018 13:30:51,415358,6209,6209
1519997452.981,02/03/2018 13:30:52,421568,6210,6203.796
1519997453.981,02/03/2018 13:30:53,427779,6211,6211
1519997454.981,02/03/2018 13:30:54,433983,6204,6204
1519997455.981,02/03/2018 13:30:55,440196,6213,6213
1519997456.981,02/03/2018 13:30:56,446406,6210,6210
1519997457.981,02/03/2018 13:30:57,452615,6209,6209
1519997458.981,02/03/2018 13:30:58,458825,6210,6210
1519997459.982,02/03/2018 13:30:59,465036,6211,6204.795
1519997460.982,02/03/2018 13:31:00,471247,6211,6211
1519997461.982,02/03/2018 13:31:01,477457,6210,6210
1519997462.982,02/03/2018 13:31:02,483668,6211,6211
1519997463.982,02/03/2018 13:31:03,489876,6208,6208
1519997464.982,02/03/2018 13:31:04,496085,6209,6209
1519997465.982,02/03/2018 13:31:05,502293,6208,6208
1519997466.983,02/03/2018 13:31:06,508503,6210,6203.796
1519997467.983,02/03/2018 13:31:07,514713,6210,6210
1519997468.983,02/03/2018 13:31:08,520925,6212,6212
1519997469.983,02/03/2018 13:31:09,527135,6210,6210
1519997470.983,02/03/2018 13:31:10,533344,6209,6209
1519997471.983,02/03/2018 13:31:11,539554,6210,6210
1519997472.983,02/03/2018 13:31:12,545764,6210,6210
1519997473.984,02/03/2018 13:31:13,551974,6210,6203.796
1519997474.984,02/03/2018 13:31:14,558184,6210,6210
1519997475.984,02/03/2018 13:31:15,564394,6210,6210
1519997476.984,02/03/2018 13:31:16,570604,6210,6210
1519997477.984,02/03/2018 13:31:17,576814,6210,6210
1519997478.984,02/03/2018 13:31:18,583024,6210,6210
1519997479.984,02/03/2018 13:31:19,589235,6211,6211
1519997480.985,02/03/2018 13:31:20,595445,6210,6203.796
1519997481.985,02/03/2018 13:31:21,601657,6212,6212
1519997482.985,02/03/2018 13:31:22,607869,6212,6212
1519997483.985,02/03/2018 13:31:23,614081,6212,6212
1519997484.985,02/03/2018 13:31:24,620293,6212,6212
1519997485.985,02/03/2018 13:31:25,626504,6211,6211
1519997486.985,02/03/2018 13:31:26,632715,6211,6211
1519997487.986,02/03/2018 13:31:27,638903,6188,6181.818
1519997488.986,02/03/2018 13:31:28,645133,6230,6230
1519997489.986,02/03/2018 13:31:29,651343,6210,6210
1519997490.986,02/03/2018 13:31:30,657553,6210,6210
1519997491.986,02/03/2018 13:31:31,663763,6210,6210
1519997492.986,02/03/2018 13:31:32,669973,6210,6210
1519997493.987,02/03/2018 13:31:33,676183,6210,6203.796
1519997494.987,02/03/2018 13:31:34,682394,6211,6211
1519997495.987,02/03/2018 13:31:35,688604,6210,6210
1519997496.987,02/03/2018 13:31:36,694815,6211,6211
1519997497.987,02/03/2018 13:31:37,701026,6211,6211
1519997498.987,02/03/2018 13:31:38,707224,6198,6198
1519997499.987,02/03/2018 13:31:39,713442,6218,6218
1519997500.988,02/03/2018 13:31:40,719624,6182,6175.824
1519997501.988,02/03/2018 13:31:41,725861,6237,6237
1519997502.988,02/03/2018 13:31:42,732068,6207,6207
1519997503.988,02/03/2018 13:31:43,738279,6211,6211
1519997504.988,02/03/2018 13:31:44,744490,6211,6211
1519997505.988,02/03/2018 13:31:45,750698,6208,6208

样本数据机2

epoch,timestamp,operations,operations-delta,operations/second
1519997387.035,02/03/2018 13:29:47,11600,11600,11576.846
1519997388.035,02/03/2018 13:29:48,17811,6211,6211
1519997389.035,02/03/2018 13:29:49,24023,6212,6212
1519997390.035,02/03/2018 13:29:50,30235,6212,6212
1519997391.035,02/03/2018 13:29:51,36448,6213,6213
1519997392.035,02/03/2018 13:29:52,42659,6211,6211
1519997393.036,02/03/2018 13:29:53,48873,6214,6207.792
1519997394.036,02/03/2018 13:29:54,55083,6210,6210
1519997395.036,02/03/2018 13:29:55,61295,6212,6212
1519997396.036,02/03/2018 13:29:56,67507,6212,6212
1519997397.036,02/03/2018 13:29:57,73716,6209,6209
1519997398.036,02/03/2018 13:29:58,79928,6212,6212
1519997399.037,02/03/2018 13:29:59,86140,6212,6205.794
1519997400.037,02/03/2018 13:30:00,92348,6208,6208
1519997401.037,02/03/2018 13:30:01,98560,6212,6212
1519997402.037,02/03/2018 13:30:02,104768,6208,6208
1519997403.037,02/03/2018 13:30:03,110917,6149,6149
1519997404.037,02/03/2018 13:30:04,117187,6270,6270
1519997405.037,02/03/2018 13:30:05,123397,6210,6210
1519997406.038,02/03/2018 13:30:06,129606,6209,6202.797
1519997407.038,02/03/2018 13:30:07,135817,6211,6211
1519997408.038,02/03/2018 13:30:08,142026,6209,6209
1519997409.038,02/03/2018 13:30:09,148226,6200,6200
1519997410.038,02/03/2018 13:30:10,154445,6219,6219
1519997411.038,02/03/2018 13:30:11,160656,6211,6211
1519997412.038,02/03/2018 13:30:12,166869,6213,6213
1519997413.039,02/03/2018 13:30:13,173083,6214,6207.792
1519997414.039,02/03/2018 13:30:14,179296,6213,6213
1519997415.039,02/03/2018 13:30:15,185509,6213,6213
1519997416.039,02/03/2018 13:30:16,191721,6212,6212
1519997417.039,02/03/2018 13:30:17,197935,6214,6214
1519997418.039,02/03/2018 13:30:18,204147,6212,6212
1519997419.039,02/03/2018 13:30:19,210359,6212,6212
1519997420.04,02/03/2018 13:30:20,216569,6210,6203.796
1519997421.04,02/03/2018 13:30:21,222780,6211,6211
1519997422.04,02/03/2018 13:30:22,228993,6213,6213
1519997423.04,02/03/2018 13:30:23,235204,6211,6211
1519997424.04,02/03/2018 13:30:24,241415,6211,6211
1519997425.04,02/03/2018 13:30:25,247628,6213,6213
1519997426.04,02/03/2018 13:30:26,253841,6213,6213
1519997427.041,02/03/2018 13:30:27,260053,6212,6205.794
1519997428.041,02/03/2018 13:30:28,266265,6212,6212
1519997429.041,02/03/2018 13:30:29,272475,6210,6210
1519997430.041,02/03/2018 13:30:30,278686,6211,6211
1519997431.041,02/03/2018 13:30:31,284900,6214,6214
1519997432.041,02/03/2018 13:30:32,291112,6212,6212
1519997433.041,02/03/2018 13:30:33,297324,6212,6212
1519997434.042,02/03/2018 13:30:34,303535,6211,6204.795
1519997435.042,02/03/2018 13:30:35,309749,6214,6214
1519997436.042,02/03/2018 13:30:36,315960,6211,6211
1519997437.042,02/03/2018 13:30:37,322172,6212,6212
1519997438.042,02/03/2018 13:30:38,328385,6213,6213
1519997439.042,02/03/2018 13:30:39,334597,6212,6212
1519997440.042,02/03/2018 13:30:40,340809,6212,6212
1519997441.043,02/03/2018 13:30:41,347002,6193,6186.813
1519997442.043,02/03/2018 13:30:42,353233,6231,6231
1519997443.043,02/03/2018 13:30:43,359445,6212,6212
1519997444.043,02/03/2018 13:30:44,365657,6212,6212
1519997445.043,02/03/2018 13:30:45,371869,6212,6212
1519997446.043,02/03/2018 13:30:46,378081,6212,6212
1519997447.043,02/03/2018 13:30:47,384294,6213,6213
1519997448.044,02/03/2018 13:30:48,390506,6212,6205.794
1519997449.044,02/03/2018 13:30:49,396719,6213,6213
1519997450.044,02/03/2018 13:30:50,402932,6213,6213
1519997451.044,02/03/2018 13:30:51,409144,6212,6212
1519997452.044,02/03/2018 13:30:52,415355,6211,6211
1519997453.044,02/03/2018 13:30:53,421573,6218,6218
1519997454.044,02/03/2018 13:30:54,427784,6211,6211
1519997455.045,02/03/2018 13:30:55,433996,6212,6205.794
1519997456.045,02/03/2018 13:30:56,440208,6212,6212
1519997457.045,02/03/2018 13:30:57,446417,6209,6209
1519997458.045,02/03/2018 13:30:58,452629,6212,6212
1519997459.045,02/03/2018 13:30:59,458840,6211,6211
1519997460.045,02/03/2018 13:31:00,465052,6212,6212
1519997461.045,02/03/2018 13:31:01,471262,6210,6210
1519997462.046,02/03/2018 13:31:02,477470,6208,6201.798
1519997463.046,02/03/2018 13:31:03,483656,6186,6186
1519997464.046,02/03/2018 13:31:04,489892,6236,6236
1519997465.046,02/03/2018 13:31:05,496106,6214,6214
1519997466.046,02/03/2018 13:31:06,502315,6209,6209
1519997467.046,02/03/2018 13:31:07,508528,6213,6213
1519997468.046,02/03/2018 13:31:08,514740,6212,6212
1519997469.047,02/03/2018 13:31:09,520951,6211,6204.795
1519997470.047,02/03/2018 13:31:10,527165,6214,6214
1519997471.047,02/03/2018 13:31:11,533376,6211,6211
1519997472.047,02/03/2018 13:31:12,539589,6213,6213
1519997473.047,02/03/2018 13:31:13,545801,6212,6212
1519997474.047,02/03/2018 13:31:14,552013,6212,6212
1519997475.048,02/03/2018 13:31:15,558225,6212,6205.794
1519997476.048,02/03/2018 13:31:16,564436,6211,6211
1519997477.048,02/03/2018 13:31:17,570650,6214,6214
1519997478.048,02/03/2018 13:31:18,576862,6212,6212
1519997479.048,02/03/2018 13:31:19,583073,6211,6211
1519997480.048,02/03/2018 13:31:20,589287,6214,6214
1519997481.048,02/03/2018 13:31:21,595499,6212,6212
1519997482.049,02/03/2018 13:31:22,601711,6212,6205.794
1519997483.049,02/03/2018 13:31:23,607923,6212,6212
1519997484.049,02/03/2018 13:31:24,614136,6213,6213
1519997485.049,02/03/2018 13:31:25,620350,6214,6214
1519997486.049,02/03/2018 13:31:26,626562,6212,6212
1519997487.049,02/03/2018 13:31:27,632775,6213,6213
1519997488.049,02/03/2018 13:31:28,638951,6176,6176
1519997489.05,02/03/2018 13:31:29,645195,6244,6237.762
1519997490.05,02/03/2018 13:31:30,651408,6213,6213
1519997491.05,02/03/2018 13:31:31,657619,6211,6211
1519997492.05,02/03/2018 13:31:32,663831,6212,6212
1519997493.05,02/03/2018 13:31:33,670044,6213,6213
1519997494.05,02/03/2018 13:31:34,676256,6212,6212
1519997495.05,02/03/2018 13:31:35,682469,6213,6213
1519997496.051,02/03/2018 13:31:36,688679,6210,6203.796
1519997497.051,02/03/2018 13:31:37,694892,6213,6213
1519997498.051,02/03/2018 13:31:38,701103,6211,6211
1519997499.051,02/03/2018 13:31:39,707317,6214,6214
1519997500.051,02/03/2018 13:31:40,713485,6168,6168
1519997501.051,02/03/2018 13:31:41,719733,6248,6248
1519997502.051,02/03/2018 13:31:42,725945,6212,6212
1519997503.052,02/03/2018 13:31:43,732152,6207,6200.799
1519997504.052,02/03/2018 13:31:44,738363,6211,6211
1519997505.052,02/03/2018 13:31:45,744574,6211,6211
1519997506.052,02/03/2018 13:31:46,750781,6207,6207

标签： pandas

Answer 1

通过合并位置，您的意思是在行位置？因为在这种情况下，您可以像这样连接它们：

df = pd.concat([fr1, fr2], axis=1)

不确定我是否正确理解了您的求和问题，但是因为您在一帧中拥有所有数据，您现在可以将列添加在一起：

df['sum_ops'] = df['operations'] + df['operations2']

（我重命名了第二个 csv 的标题，这样就没有同名的列）

对于绘图来说，拥有一个 可能会很好DatetimeIndex，即使并非所有时间都完美排列。只需选择最佳时间戳列，将其转换为日期时间并将其分配给索引：

df = df.set_index(pd.to_datetime(df['timestamp']))

接下来为了更好地绘图，我将删除所有epoch列timestamp：

del df['epoch']等等