vb.net - 如何同时从数千个 URL 中获取网页源

Question

我试图将数千个 URL 加载到一个列表中，然后同时下载所有这些 URL 的网页源。我以为我对如何实现这一点有一个清晰的了解，但似乎这个过程是一个接一个地进行的（这非常缓慢）。

有没有办法一次启动所有这些 URL，或者一次可能超过 1 个？

Public Partial Class MainForm

Dim ImportList As New ListBox
Dim URLList As String
Dim X1 As Integer
Dim CurIndex As Integer

Public Sub New()
    Me.InitializeComponent()
End Sub

Sub MainFormLoad(sender As Object, e As EventArgs)
    Try
        Dim lines() As String = IO.File.ReadAllLines("C:\URLFile.txt")
        ImportList.Items.AddRange(lines)    
    Catch ex As Exception
        MessageBox.Show(ex.ToString)
    Finally
        label1.Text = "File Loaded" 
        X1 = ImportList.Items.Count
        timer1.Enabled = True
        If Not backgroundWorker1.IsBusy Then
            backgroundWorker1.RunWorkerAsync()
        End If
    End Try
End Sub

Sub BackgroundWorker1DoWork(sender As Object, e As System.ComponentModel.DoWorkEventArgs)
    URLList = ""
    For Each item As String In ImportList.Items
        CheckName(item)
        CurIndex = CurIndex + 1
    Next
End Sub

Sub BW1_Completed()
    timer1.Enabled = False
    label1.Text = "Done"
End Sub

Sub CheckName(ByVal CurUrl As String)
    Dim RawText As String
    Try
        RawText = New System.Net.WebClient().DownloadString(CurUrl)         
    Catch ex As Exception
        MessageBox.Show(ex.ToString)
    Finally
        If RawText.Contains("404") Then
            If URLList = "" Then
                URLList = CurUrl
            Else
                URLList = URLList & vbCrLf & CurUrl
            End If
        End If
    End Try
End Sub

Sub Timer1Tick(sender As Object, e As EventArgs)
    label1.Text = CurIndex.ToString & " of " & X1.ToString
    If Not URLList = "" Then
        textBox1.Text = URLList
    End If
End Sub

Sub Button1Click(sender As Object, e As EventArgs)
    Clipboard.Clear
    Clipboard.SetText(URLList)
End Sub

结束类

标签： vb.netwebclient