c# - 如何在 C# 中进行并发 API 调用?
问题描述
我需要从抵押 API 中提取自定义字段。问题是总共有 11000 条记录,每个 API 请求需要 1 秒。我想找到一种异步并行发送请求的方法,以提高效率。
我尝试遍历所有请求,然后Task.WaitAll()
等待响应返回。我只收到两个响应,然后应用程序无限期地等待。
我首先为HttpClient
public static class ApiHelper
{
public static HttpClient ApiClient { get; set; }
public static void InitializeClient()
{
ApiClient = new HttpClient();
ApiClient.DefaultRequestHeaders.Add("ContentType", "application/json");
}
}
我收集我的抵押 ID 列表并循环通过 API Post Calls
static public DataTable GetCustomFields(DataTable dt, List<string> cf, string auth)
{
//set auth header
ApiHelper.ApiClient.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", auth);
//format body
string jsonBody = JArray.FromObject(cf).ToString();
var content = new StringContent(jsonBody, Encoding.UTF8, "application/json");
var responses = new List<Task<string>>();
foreach (DataRow dr in dt.Rows)
{
string guid = dr["GUID"].ToString().Replace("{", "").Replace("}", ""); //remove {} from string
responses.Add(GetData(guid, content));
}
Task.WaitAll(responses.ToArray());
//some code here to process through the responses and return a datatable
return updatedDT;
}
每个 API 调用都需要 URL 中的抵押 ID (GUID)
async static Task<string> GetData(string guid, StringContent json)
{
string url = "https://api.elliemae.com/encompass/v1/loans/" + guid + "/fieldReader";
Console.WriteLine("{0} has started .....", guid);
using (HttpResponseMessage response = await ApiHelper.ApiClient.PostAsync(url, json))
{
if (response.IsSuccessStatusCode)
{
Console.WriteLine("{0} has returned response....", guid);
return await response.Content.ReadAsStringAsync();
}
else
{
Console.WriteLine(response.ReasonPhrase);
throw new Exception(response.ReasonPhrase);
}
}
}
我现在只测试 10 条记录并发送所有 10 个请求。但我只收到两回。
结果就在这里。
你能告诉我发送并发 API 调用的正确方法吗?
解决方案
所有人GetData
Task
都使用相同的HttpClient
单例实例。HttpClient 不能同时服务多个调用。最佳实践是使用Pool
HttpClient 来确保没有 Task 同时访问同一个 HttpClient。
另外,小心抛出exception
任务,它会停止第WaitAll()
一次抛出的异常
解决方案我在这里发布了整个项目:https
://github.com/jonathanlarouche/stackoverflow_58137212 这个解决方案使用[3] 池
发送 25 个请求;max sized
基本上,ApiHelper
包含一个HttpClient
pool,使用 Generic 类ArrayPool<T>
。您可以使用任何其他池化库,我只是想发布一个独立的解决方案。
建议 ApiHelper Bellow,这个类现在包含一个池和一个Use
接收 的方法,池中的Action
一个 Item 将在 Action 期间“租用”,然后通过该ArrayPool.Use
函数返回到池中。Use
函数还接收 apiToken 以更改请求身份验证标头。
public static class ApiHelper
{
public static int PoolSize { get => apiClientPool.Size; }
private static ArrayPool<HttpClient> apiClientPool = new ArrayPool<HttpClient>(() => {
var apiClient = new HttpClient();
apiClient.DefaultRequestHeaders.Add("ContentType", "application/json");
return apiClient;
});
public static Task Use(string apiToken, Func<HttpClient, Task> action)
{
return apiClientPool.Use(client => {
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", apiToken);
return action(client);
});
}
}
获取数据函数。Get Data 将收到 apiToken 并等待该ApiHelper.Use
函数。对象的新实例StringContent()
需要在此函数中完成,因为它不能在不同的 Http Post 调用中重复使用。
async static Task<string> GetData(string apiToken, Guid guid, string jsonBody)
{
string url = "https://api.elliemae.com/encompass/v1/loans/" + guid + "/fieldReader";
Console.WriteLine("{0} has started .....", guid);
string output = null;
await ApiHelper.Use(apiToken, (client) =>
{
var json = new StringContent(jsonBody, Encoding.UTF8, "application/json");
return client.PostAsync(url, json).ContinueWith(postTaskResult =>
{
return postTaskResult.Result.Content.ReadAsStringAsync().ContinueWith(s => {
output = s.Result;
return s;
});
});
});
Console.WriteLine("{0} has finished .....", guid);
return output;
}
数组池
public class ArrayPool<T>
{
public int Size { get => pool.Count(); }
public int maxSize = 3;
public int circulingObjectCount = 0;
private Queue<T> pool = new Queue<T>();
private Func<T> constructorFunc;
public ArrayPool(Func<T> constructorFunc) {
this.constructorFunc = constructorFunc;
}
public Task Use(Func<T, Task> action)
{
T item = GetNextItem(); //DeQueue the item
var t = action(item);
t.ContinueWith(task => pool.Enqueue(item)); //Requeue the item
return t;
}
private T GetNextItem()
{
//Create new object if pool is empty and not reached maxSize
if (pool.Count == 0 && circulingObjectCount < maxSize)
{
T item = constructorFunc();
circulingObjectCount++;
Console.WriteLine("Pool empty, adding new item");
return item;
}
//Wait for Queue to have at least 1 item
WaitForReturns();
return pool.Dequeue();
}
private void WaitForReturns()
{
long timeouts = 60000;
while (pool.Count == 0 && timeouts > 0) { timeouts--; System.Threading.Thread.Sleep(1); }
if(timeouts == 0)
{
throw new Exception("Wait timed-out");
}
}
}
推荐阅读
- android - 在 android 构建中包含自定义文件
- php - 如何使用从 React / Laravel(+maatwebsite/excel) 中获取来下载 excel 文件?
- css - :hover 之前的 SCSS 目标类
- youtube-api - 如何在 youtube 上获取给定游戏的直播列表
- reactjs - React:如何将一条路线与 CSSTransition 分开?
- java - 求解释 Long -> Byte Array -> String -> Byte Array -> Long
- android - 半矩形形状可绘制形状android
- javascript - 如何在vue js中使用axios body raw json获取请求
- border - Chromium 中缺少表格右边框
- go - 如何重建 net/http 库