-
The provided code is a part of a web scraper project. The project seems to have two spiders, one for crawling the CoinGecko website and another one for CoinMarketCap.
-
The crawler/parsers directory contains three Python files, base.py, coinmarketcap.py, and parserCoinGecko.py. The base.py file contains a base class ParseBase that is inherited by two other classes, ContractParseBase and CryptoCurrencyParseBase. Similarly, the coinmarketcap.py and parserCoinGecko.py files contain classes for parsing cryptocurrency data from the CoinMarketCap and CoinGecko websites, respectively. These classes inherit the ContractParseBase and CryptoCurrencyParseBase classes to parse the required data.
-
The crawler/spiders directory contains two spider files, spidercoinmarketcap.py and spiserCoingecko.py. The spidercoinmarketcap.py spider scrapes data from the CoinMarketCap website by sending requests to its API. The spiserCoingecko.py spider crawls the CoinGecko website and scrapes data from its HTML pages.
-
The spidercoinmarketcap.py file defines a CoinMarketCapAllSpider class that sends a request to the CoinMarketCap API and retrieves a list of cryptocurrencies with some details. For each cryptocurrency, it sends another request to the API to get more detailed information and then uses the CryptoCurrencyCoinMarketCapParse class to parse the response and extract the required data. Finally, the parsed data is printed and yielded to a pipeline.
-
The spiserCoingecko.py file defines a MySpider class that starts with the first page of the CoinGecko website and uses the response.xpath method to extract links to cryptocurrency pages. For each cryptocurrency page, it sends a request and uses the CryptoCurrencyCoinGeckoParse class to parse the response and extract the required data. Finally, the parsed data is printed and yielded to a pipeline.
-
Both spider files have a custom_settings dictionary that defines the settings for the spider. The settings include a download delay of 1 second, log level of INFO, and an item pipeline that saves the scraped data to a Postgres database using the PostgresPipeline class from the crawler.cryptoParsers.pipelines module.
-
Notifications
You must be signed in to change notification settings - Fork 0
Goltzishpt/cryptoParsers
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
About
No description, website, or topics provided.
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published