node.js - Updating large amount of documents in mongodb

Question

I am working on solution to this problem:

I have got a large collection of entries, that regularly I have to update from the other database. I am using express.js with mongodb. The schedule of updating starts everyday at 1am. I fetch external data, compare to currently existing and update all entries in our databse.

The process looks like: 1) Fetch internal and external data 2) Combine them together (past entries from our database, future entries from the other database) 3) Delete all records in our database 4) InsertMany - records that the program just combined.

So as you can see it's a quite risky task. If any error would occur between deleting and inserting data in the collection, we lose all data.

And there are my questions:

1) Is there any effective way of reverting data that has been just deleted in mongo? Or keep them on hold and insert back on its place, if any error occurs?

2) Is there any other effective way of updating few hundreds/thousands of documents apart from deleteMany -> insertMany // updateOne on each document?

Any advice would be appreciated.

标签： node.jsmongodbexpress

Answer 1

An alternative to your set of operation could be:

1. fetch external data
2. add external data in to a new database/collection
3. merge the data together with an aggeration query using $merge (https://docs.mongodb.com/manual/reference/operator/aggregation/merge/)

That way you wouldn't have to delete all your data and could possibly end up with data loss.