python - 有人在使用 Trafilatura 时遇到重复文本的问题吗?
问题描述
尝试使用trafilatura
抓取以下网页时遇到重复问题,即使我设置了deduplicate=True
. 有谁知道这是否是软件包的缺点,或者是否有任何参数我可以切换以摆脱这种行为?
https://www.federalreserve.gov/monetarypolicy/fomcminutes20200429.htm
import trafilatura
downloaded = trafilatura.fetch_url(url)
text = trafilatura.extract(downloaded, target_language='en', include_tables=True, deduplicate=True)
输出:
全文太长,无法在此处引用,但有几个重复的片段:
**Notation Vote**
By notation vote completed on May 19, 2020, the Committee unanimously approved the minutes of the Committee meeting held on April 28–29, 2020.
Notation Vote
By notation vote completed on May 19, 2020, the Committee unanimously approved the minutes of the Committee meeting held on April 28–29, 2020.
**Staff Economic Outlook**
The projection for the U.S. economy prepared by the staff for the June FOMC meeting was downgraded, on balance, as compared with the April meeting forecast in response to information on the spread of the coronavirus and changes in the measures undertaken to contain it both at home and abroad, along with incoming economic data. U.S. real GDP was forecast to show a historically large decline in the second quarter of this year, and the unemployment rate was expected to be sharply higher than in the first quarter. The substantial fiscal policy measures and appreciable support from monetary policy, along with the Federal Reserve's liquidity and lending facilities, were expected to help mitigate the deterioration in current economic conditions and to help boost the recovery.
Staff Economic Outlook
The projection for the U.S. economy prepared by the staff for the June FOMC meeting was downgraded, on balance, as compared with the April meeting forecast in response to information on the spread of the coronavirus and changes in the measures undertaken to contain it both at home and abroad, along with incoming economic data. U.S. real GDP was forecast to show a historically large decline in the second quarter of this year, and the unemployment rate was expected to be sharply higher than in the first quarter. The substantial fiscal policy measures and appreciable support from monetary policy, along with the Federal Reserve's liquidity and lending facilities, were expected to help mitigate the deterioration in current economic conditions and to help boost the recovery.
解决方案
推荐阅读
- android - 如何从 App 中的 Assets 文件夹更新数据库
- java - 注释 @RunAs 在启用 Java EE 安全 API (JSR 375) 的 WildFly 上不起作用
- python - 无法在 Spyder 变量资源管理器中读取来自 read_sql 的数据框
- python - 如何在 python 中添加到 txtfile 的顶部?
- reactjs - 如果我尝试使用 API 响应数据更新我的状态,为什么会开始无限循环?
- java - Micrometer TimedAspect 不会拦截对使用 @Timed 注释的方法的调用
- r - ggplot:按工作日排序
- android - 无法使用 glide 库从 firebase 存储中获取图像
- velocity - 在衡量分析中的速度时如何考虑 UserStory 的故事点?
- nestjs - Nestjs自定义配置服务错误注入其他模块服务