c# - C# - 如何使用 HttpClient 对象从 JSON API 中提取 CSV 文件？

Question

我的公司使用 C# .net 应用程序以编程方式从供应商网站以 CSV 文件的形式检索数据。但是，供应商最近将其 API 更改为 JSON API。我是一个新手程序员，对 JSON API 调用没有任何经验。当供应商注册我的新 API 用户 ID 时，他们提供了一个示例，说明如何使用curl下载必要的 csv 数据文件。这是示例：

echo 'Going to log in to sso'
data=$(curl -c cookies.txt -X POST https://sso.theice.com/api/authenticateTfa -H "Content-Type: application/json" \
--data '{"userId":"yourApiUserId","password":"yourpassword","appKey":"yourAppKey"}')

echo 'Going to download icecleared_oil_2020_06_26.dat file from Settlement_Reports_CSV/Oil'
curl -b cookies.txt -O ' https://downloads2.theice.com/Settlement_Reports_CSV/Oil/icecleared_oil_2020_06_26.dat

由于我没有使用 Curl 的经验，因此我使用https://curl.olsh.me/将上面的示例转换为 C#。这样做会生成以下代码：

//Authenticate userId (and get cookie?)
using (var httpClient = new HttpClient())
{
    using (var request = new HttpRequestMessage(new HttpMethod("POST"), "https://sso.theice.com/api/authenticateTfa"))
    {
        request.Content = new StringContent("{\"userId\":\"yourApiUserId\",\"password\":\"yourpassword\",\"appKey\":\"yourAppKey\"}");
        request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json"); 

        var response = await httpClient.SendAsync(request);
    }
}

//Using authentication cookie, from previous code block, retrieve dat file
var handler = new HttpClientHandler();
handler.UseCookies = false;

using (var httpClient = new HttpClient(handler))
{
    using (var request = new HttpRequestMessage(new HttpMethod("GET"), "https://downloads2.theice.com/Settlement_Reports_CSV/Oil/icecleared_oil_2020_06_26.dat"))
    {
        request.Headers.TryAddWithoutValidation("Cookie", "cookies.txt");

        var response = await httpClient.SendAsync(request);
    }
}

我将此代码集成到我的项目中，并且能够让它对我的用户 ID（POST 请求）进行身份验证，但我终生无法弄清楚如何从第二个代码块（GET 请求）中检索 DAT 文件。我观察到无论我是否使用 POST 请求成功登录，GET 请求的响应都是相同的。在这一点上，我假设 POST 请求中的验证 cookie 没有被 GET 请求获取，但即使它是我也不知道如何从 GET 请求中的响应对象中提取文件。

谁能建议我需要做什么？以下是已集成到我的项目中的代码：

public async Task DownloadAsync(List<string> iceReportFileNamesFullUrl)
{
    await AuthenticateUserCredentialsOnICE();
    await RetrieveIceRportFiles(iceReportFileNamesFullUrl);

}

private async Task AuthenticateUserCredentialsOnICE()
{
    using (HttpClient httpClient = new HttpClient())
    {
        using (HttpRequestMessage request = new HttpRequestMessage(new HttpMethod("POST"), "https://sso.theice.com/api/authenticateTfa"))
        {               
            request.Content = new StringContent("{\"userId\":\"xxxxxxxxx\",\"password\":\"xxxxxxxxxx\",\"appKey\":\"xxxxxxxxxxx\"}");
            request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json");

            HttpResponseMessage response = await httpClient.SendAsync(request);
            var value = response.Content.ReadAsStringAsync();
        }
    }
}
private static async Task RetrieveIceRportFiles(List<string> iceReportFileNamesFullUrl)
{
    foreach (string iceReportUrl in iceReportFileNamesFullUrl)
    {
        HttpClientHandler handler = new HttpClientHandler();
        handler.UseCookies = false;

        using (HttpClient httpClient = new HttpClient(handler))
        {
            using (HttpRequestMessage request = new HttpRequestMessage(new HttpMethod("GET"), iceReportUrl))
            {
                request.Headers.TryAddWithoutValidation("Cookie", "cookies.txt");

                HttpResponseMessage response = await httpClient.SendAsync(request);
                var value = response.Content.ReadAsStringAsync();

            }
        }
    }
}

-更新：2020 年 8 月 25 日-

谢谢大家的意见。您明确表示我没有从 POST 请求中捕获 cookie。经过一些研究（this、this和this），我更新了代码，以便它现在可以成功下载 CSV/DAT 文件。我已经在下面发布了我的内容，尽管它仍然需要错误处理、检查登录失败等。我也相信会有重要的反馈。

这里是：

private CookieContainer authenticationCookieContainer;
//main method for downloading file
public async Task DownloadAsync(string iceReportDownloadUrl)
{
    await AuthenticateUserCredentialsOnIceAsync();
    await SendDownloadReportRequestAsync(iceReportDownloadUrl);
}

通过 JSON API 对凭证进行身份验证并将身份验证 cookie 存储在 authenticationCookieContainter

private async Task AuthenticateUserCredentialsOnIceAsync()
{
    //This will contain the ICE authentication cookies
    //if the login is successful
    authenticationCookieContainer = new CookieContainer();

    //Create and assign handler for proxy config and cookie storage
    using (HttpClientHandler handler = new HttpClientHandler() { CookieContainer = authenticationCookieContainer })
    {
        using (HttpClient httpClient = new HttpClient(handler))
        {
            using (HttpRequestMessage request = new HttpRequestMessage(new HttpMethod("POST"), "https://sso.theice.com/api/authenticateTfa"))
            {
                request.Content = new StringContent("{\"userId\":\"xxxxxxxxxxx\",\"password\":\"xxxxxxxxxxxxxx\",\"appKey\":\"xxxxxxxxxxxxx\"}");
                request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json");
                HttpResponseMessage response = await httpClient.SendAsync(request);
            }
        }
    }
}

如果登录成功， authenticationCookieContainer 将包含登录 cookie 并允许您在指定的 url 请求数据

private async Task SendDownloadReportRequestAsync(string iceReportDownloadUrl)
{
    if (authenticationCookieContainer != null)
    {
        //Create and assign handler for proxy config and cookie storage
        using (HttpClientHandler handler = new HttpClientHandler() { CookieContainer = authenticationCookieContainer })
        {
            //Set to true to use the cookies obtained during authentication call
            handler.UseCookies = true;

            using (HttpClient httpClient = new HttpClient(handler))
            {
                Uri iceReportUri = new Uri(iceReportDownloadUrl);
                //Request the ICE data file using the url passed through
                using (HttpRequestMessage request = new HttpRequestMessage(new HttpMethod("GET"), iceReportUri.AbsoluteUri))
                {
                    HttpResponseMessage response = await httpClient.SendAsync(request);
                    using (Stream responseStream = await response.Content.ReadAsStreamAsync())
                    {
                        //Write the extracted file to the file path
                        byte[] content = StreamToByteArray(responseStream);
                        File.WriteAllBytes(@"C:\Users\my.name\Desktop\" + iceReportUri.Segments.Last(), content);
                    }
                }
            }
        } 
    }
}
private static byte[] StreamToByteArray(Stream stream)
{
    using (MemoryStream memoryStream = new MemoryStream())
    {
        stream.CopyTo(memoryStream);
        return memoryStream.ToArray();
    }
}

标签： c#dotnet-httpclient