elasticsearch - Will the Crawler reindex the records after deleted

Question

Working on Storm Crawler 1.12.1 and Elastic Search 6.5.2. I need to increase the efficiency of my search engine. I deleted some of the documents for security reasons after indexing documents into the elastic search. So my question is that the storm crawler will re grab the deleted urls and re-index again? I don't want to re-crawl the deleted records,How can I achieve this?

标签： elasticsearchweb-crawlerstormcrawler

Answer 1

I assume you deleted the documents from the content index. They are probably still in the status index and even if they are not, they might be rediscovered and added back.

The best thing to do would be to add new entries to whichever flavour of URLfilters you are using so that these URLs are covered, this way they won't be added back if rediscovered then delete them from the status index.