Skip to content

Goltzishpt/cryptoParsers

Repository files navigation

IN DEVELOPMENT

CryptoParser

  • The provided code is a part of a web scraper project. The project seems to have two spiders, one for crawling the CoinGecko website and another one for CoinMarketCap.

  • The crawler/parsers directory contains three Python files, base.py, coinmarketcap.py, and parserCoinGecko.py. The base.py file contains a base class ParseBase that is inherited by two other classes, ContractParseBase and CryptoCurrencyParseBase. Similarly, the coinmarketcap.py and parserCoinGecko.py files contain classes for parsing cryptocurrency data from the CoinMarketCap and CoinGecko websites, respectively. These classes inherit the ContractParseBase and CryptoCurrencyParseBase classes to parse the required data.

  • The crawler/spiders directory contains two spider files, spidercoinmarketcap.py and spiserCoingecko.py. The spidercoinmarketcap.py spider scrapes data from the CoinMarketCap website by sending requests to its API. The spiserCoingecko.py spider crawls the CoinGecko website and scrapes data from its HTML pages.

  • The spidercoinmarketcap.py file defines a CoinMarketCapAllSpider class that sends a request to the CoinMarketCap API and retrieves a list of cryptocurrencies with some details. For each cryptocurrency, it sends another request to the API to get more detailed information and then uses the CryptoCurrencyCoinMarketCapParse class to parse the response and extract the required data. Finally, the parsed data is printed and yielded to a pipeline.

  • The spiserCoingecko.py file defines a MySpider class that starts with the first page of the CoinGecko website and uses the response.xpath method to extract links to cryptocurrency pages. For each cryptocurrency page, it sends a request and uses the CryptoCurrencyCoinGeckoParse class to parse the response and extract the required data. Finally, the parsed data is printed and yielded to a pipeline.

  • Both spider files have a custom_settings dictionary that defines the settings for the spider. The settings include a download delay of 1 second, log level of INFO, and an item pipeline that saves the scraped data to a Postgres database using the PostgresPipeline class from the crawler.cryptoParsers.pipelines module.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages