A Web Cawler built using Python, Selenium and BeautifulSoup4 Which crawl and scrapes Data from StackOverflow,Quora and Google.
It Provides user with a CSV file which contains link of the sites through which the crawler has gone through, and also have a feature to take full page screenshots of the visited pages.
It Can also give outputs in a nicely format WebPage if user wants, so that user can easily navigate through results.
Install the required dependencies run : pip install -r requirements.txt
For the Script to run you need Chrome driver, download the chrome Driver as per your Google Chrome Version from : http://chromedriver.chromium.org/downloads
In Windows OS
Put .exe file in
Python3/Script/directory
In Unix/Linux based OS
Put Chrome driver in
/usr/bin/directory
Python3 cicada.py "what you want to search".
Files Created:
