python - 将数据从 Scrapy 传递到 Socket

Question

目前我有一个用于从网站抓取数据的scrapy。我想将当前数据发送到它使用套接字抓取的连接客户端。

clients = set()
clients_lock = threading.Lock()

host = socket.gethostbyname("") 
port = 10010

def listener(client, address, item):
    print ("Accepted connection from: ", address)
    with clients_lock:
        clients.add(client)
    try:    
        data = client.recv(1024)
        if data == ('0').encode():
            timestamp = datetime.datetime.now().strftime("%I:%M:%S %p")
            client.send(timestamp.encode())
    finally:
        with clients_lock:
            clients.remove(client)
            client.close()

def send_to_socket(item):
    s = socket.socket()
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind((host,port))
    s.listen(3)
    th = []

    client, address = s.accept()
    th.append(Thread(target=listener, args = (client,address, item)).start())
    s.close()

下面给出的代码是调用启动套接字的函数的scrapy代码，当前的问题是我无法从客户端代码实时检索数据。当前代码每次发送数据时都会关闭客户端连接。有没有办法在不关闭客户端连接的情况下连续发送数据。如果我删除 client.remove(client) 代码会卡住并且不会运行 _build_link_item。

_build_link_item 连续运行以从不同的 URL 生成数据，我想将它生成的数据发送到客户端。

def _build_link_item(response):
    """Builds a LinkItem object for the given response"""
    parser = HTMLParser(response)

    item = LinkItem(
        url=response.url,
        status=response.status,
        content_type=response.headers.get("Content-Type", b"").decode("utf-8") or "",
    )

    print(json.dumps(item.__dict__), flush=True)
    send_to_socket(json.dumps(item.__dict__))

标签： pythonsocketswebsocketscrapy