Seen
A lightweight crawling/spider framework for everyone
Screenshots

Hunter's comment
"Seen is a lightweight web crawling framework for everyone."
This is used to crawl your webpages very fast.
No need of Robot.txt
Pros
Supports JavaScript
Crawls webpages Fast
Simple to use
Open Source
Note:Python 3.5+ is Needed to install
Syntax for Installation
pip install seen
Link
https://github.com/HuberTRoy/Seen
Contributors
Hunter: @alizamanjutt454

This is posted on Steemhunt - A place where you can dig products and earn STEEM.
View on Steemhunt.com