python - MemoryError using dpkt Python 2.7
问题描述
I've captured data using wireshark. I am now trying to parse the data in Python 2.7 using dpkt
into a dictionary which I will then pickle.
I consistently keep getting MemoryError
. I am assuming it could be because I am dealing with thousands of MB and am simply running out of memory, but I am not sure.
storepcap.py
from socket import inet_ntoa
import traceback
import datetime
try:
import cPickle as pickle
except:
import pickle
myAddress = '10.0.0.3'
files = os.listdir(os.getcwd())
dataFileName = 'alldata.dat'
dataFile = open(dataFileName, 'wb')
try:
dpktData = pickle.load(dataFile)
except IOError:
dpktData = {}
def inet_to_str(inet):
return socket.inet_ntop(socket.AF_INET, inet)
badFrames = 0
data = ""
for file in files:
if file.endswith('pcap'):
print "Bad Frames " + str(badFrames)
print 'Parsing ' + str(file)
data = open(file, 'rb')
pcap = dpkt.pcap.Reader(data)
try:
for timestamp, buf in pcap:
try:
eth = dpkt.ethernet.Ethernet(buf)
except:
print "Error parsing frame"
continue
if not isinstance(eth.data, dpkt.ip.IP):
continue
ip = eth.data
if myAddress != inet_ntoa(ip.src) and myAddress != inet_ntoa(ip.dst):
continue
try:
dpktData[str(datetime.datetime.utcfromtimestamp(timestamp))] = ip
except:
badFrames += 1
continue
except Exception, e:
traceback.print_exc()
finally:
print "Closing file %s"%str(file)
data.close()
pickle.dump(dpktData, dataFile)
dataFile.close()
output:
Bad Frames 0
Parsing 2019-04-17-traffic.pcap
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Error parsing frame
Traceback (most recent call last):
File "storepcap.py", line 34, in <module>
for timestamp, buf in pcap:
File "C:\Python27\lib\site-packages\dpkt\pcap.py", line 315, in __iter__
MemoryError
Closing file 2019-04-17-traffic.pcap
Bad Frames 93907
Parsing traffic-2019-4-20-3.pcap
Error parsing frame
Traceback (most recent call last):
File "storepcap.py", line 34, in <module>
File "C:\Python27\lib\site-packages\dpkt\pcap.py", line 315, in __iter__
MemoryError
Closing file traffic-2019-4-20-3.pcap
Bad Frames 93915
Parsing traffic_00001_20190418010523.pcap
Error parsing frame
Traceback (most recent call last):
File "storepcap.py", line 34, in <module>
File "C:\Python27\lib\site-packages\dpkt\pcap.py", line 315, in __iter__
MemoryError
Closing file traffic_00001_20190418010523.pcap
Bad Frames 93915
Parsing traffic_00001_20190418010610.pcap
Error parsing frame
Traceback (most recent call last):
File "storepcap.py", line 34, in <module>
File "C:\Python27\lib\site-packages\dpkt\pcap.py", line 315, in __iter__
MemoryError
Closing file traffic_00001_20190418010610.pcap
Bad Frames 93915
Parsing traffic_00002_20190418070000.pcap
Error parsing frame
Traceback (most recent call last):
File "storepcap.py", line 34, in <module>
File "C:\Python27\lib\site-packages\dpkt\pcap.py", line 315, in __iter__
MemoryError
Closing file traffic_00002_20190418070000.pcap
Bad Frames 93915
Parsing traffic_00003_20190418130000.pcap
Error parsing frame
Traceback (most recent call last):
File "storepcap.py", line 34, in <module>
File "C:\Python27\lib\site-packages\dpkt\pcap.py", line 315, in __iter__
MemoryError
Closing file traffic_00003_20190418130000.pcap
Bad Frames 93915
Parsing traffic_00004_20190418190000.pcap
Traceback (most recent call last):
File "storepcap.py", line 31, in <module>
pcap = dpkt.pcap.Reader(data)
File "C:\Python27\lib\site-packages\dpkt\pcap.py", line 245, in __init__
self.__fh = FileHdr(buf)
MemoryError
G:\AIdata>```
解决方案
推荐阅读
- c# - 为什么我不能在 Visual Studio 的另一个项目中为同一个文件添加另一个链接?
- python - 为什么 Amadeus 传统航班搜索(航班优惠)API 缺少定期航班?例如 EasyJet、Wizz Air、TUI?
- javascript - 更改图表的宽度和高度(Echarts 库)
- arrays - 如何将新项目附加到数组并使用 F# 中的新项目更新数组?
- java - ToggleButton 阻止我的应用程序启动
- c# - 使用 C# 在多个线程中以块的形式从 FTP 服务器下载单个文件
- html - 轻松使用 css 动画
- fortran - 如何在 FORTRAN 77 的 FORMAT 中按列写入
- ios - 缩放后如何找到 UIImageView 的新中心?
- python - Python,openpyxl,循环使用基于值的单元格的随机 bg 颜色