The project contains 3 parts. Each part is an extension of the previous one. The end goal is to code a Scrapy project that can crawl tens of thousands of apps from the Xiaomi AppStore, or any other app store with which you are familiar.
First stage: Create a Scrapy project to crawl the content in the Xiaomi Appstore homepage or any other Appstore homepage
Second stage: Save the crawled content in MongoDB. Install Python MongoDB driver and modify pipelines.py to insert crawled data into MongoDB.
Suggested Prerequisite Knowledge
Please upload your final code to your Github account
Please record a video explaining the design choices you made including: the structure of your code, how you chose efficiently collected and stored the gathered data, and how you dealt with gathering data from non-static sources. Please keep the video under 5 minutes
Splash & ScrapyJS https://github.com/scrapinghub/scrapy-splash