c# - Bulk Upload in ElasticSearch using Nest library in C#

Question

I a trying to bulk insert into ElasticSearch using BulkAll API from Nest library in C#. In order to assign an Id to each document, I am using BufferToBulk option to customize my individual documents. In general, the BulkAll finishes very fast. However, when I use BufferToBulk, it takes entire timeout time to finish. I can not figure out, what is wrong in the code below. Please note that I am using a list of dictionary objects to upload and the uploading works fine, but it is extremely slow.

        BulkAllObservable<Dictionary<string, object>> bulkAll = esClient.BulkAll(BatchedDocumets1, b => b
        .Index(indexName)
        .BufferToBulk((descriptor, list) =>
        {
            foreach (var item in list)
            {
                if (keyindex < ListOfKeys.Count)
                {
                    descriptor.Index<Dictionary<string, object>>(bi => bi
                      .Index(indexName)
                      .Id(ListOfKeys[keyindex])
                      .Document(item)
                    );
                    keyindex++;
                }
            }
        })
        //how many retries are attempted if a failure occurs. Can be configurable.
        .BackOffRetries(2)
        //how long to wait between retries. Can be configurable
        .BackOffTime("30s")
        //refresh the index once the bulk operation completes
        .RefreshOnCompleted(true)
        .MaxDegreeOfParallelism(this.maxDegreeOfParallelism)
        .Size(this.batchSize)
        .ContinueAfterDroppedDocuments(true)
        .DroppedDocumentCallback((item, Document) =>
        {
            Logger.LogMessage(MOD.Common.Logging.LogLevel.Error, " The document can not be indexed. Bulk all indexing will be halted.");
        })
        );

        bulkAll.Subscribe(new BulkAllObserver(
            onNext: (b) => { Logger.LogMessage(MOD.Common.Logging.LogLevel.Debug, " Next value: "); },
            onError: (e) => { throw e; },
            onCompleted: () => Logger.LogMessage(MOD.Common.Logging.LogLevel.Debug, "Bulk Upload Completed!")
        ));

        bulkAll.Wait(TimeSpan.FromSeconds(15), null); // We can configure this.

Any help is greatly appreciated.

标签： c#elasticsearchnest