View Source SpiderMan.Pipeline.DuplicateFilter (spider_man v0.4.7)
filter msg while duplicate key
usage
Usage
settings = [
*_options: [
pipelines: [SpiderMan.Pipeline.DuplicateFilter | {SpiderMan.Pipeline.DuplicateFilter, scope}]
],
...
]
Support for all component: downloader
| spider
| item_processor
.
scope
Scope
:common
|[scope: :common]
: save key tocommon_pipeline_tid
.:pipeline
|[scope: :pipeline]
: save key topipeline_tid
.
common_pipeline_tid
use by all component, pipeline_tid
only use by one component.