A Long-term Product Reliability Review Platform

6 Months Later a web application where various consumer goods are given reliability ratings over longer periods of time (6 months). Most reviews written these days only cover the product at launch and not how it holds up several months (or years) later.

More information available in our wiki

Folder Structure:

├── server The Remix web server application

├── docker Configuration files for running Postgres and RabbitMQ in docker

├── scraper The Python scraper application and it's shared files

│ ├── crawler.py The Python Crawler application. Starts a daemon that waits for messages in RabbitMQ

│ ├── scraper.py The Python Scraper application. Starts a daemon that waits for messages in RabbitMQ

│ ├── analyzer.py The Python Analysis application

Postgres is used by the web server to store data and RabbitMQ is used by all services to communicate with eachother.

These can be started with docker using docker-compose up

The web server is a remix application.

You first must install the dependencies.

cd server
npm install

Check to see if ./server/.husky exists with the expected contents.

If it's missing, you must also run npm run prepare in ./server/ to enable the pre-commit hooks.

Then you can start the server with npm run dev or npm run start in production.

The scraper is a Python application and is divided into three parts.

Firstly, run the setup.sh bash script to install the SUTime library used by the analyzer. The script may fail if Maven is not installed on your system: if mvn -v is not recognized as a command, please first install Maven. If the script succeeds, the folder analyzer/jars should now exist.

You must then install create a Python Virtual Environment and install the remaining dependencies.

cd scraper
python3 -m venv venv
source venv/bin/activate  # On Windows: .\venv\Scripts\activate

pip install -r requirements.txt
python -m textblob.download_corpora

cp .env.example .env # Create environment variables file based on the example

Then you can start the daemons with python crawler.py and python scraper.py.

These daemons will wait for messages in RabbitMQ, and execute accordingly.

For more information on running and testing the analyzer, see its README file

Automated tests are available for the parser and analyzer, and can be executed by running pytest.

Further information about how the scraper works can be found in it's dedicated documentation.