Part I: Scraping Stock data to file

Image for post
Image for post
Photo by Lukas Blazek on Unsplash

Introduction

Data is the new oil. The ability to “mine” this oil is elementary to any activity that may be performed on it, be it analysis, modelling, or deploying data products. Apart from getting data from data-brokers and the open source data, made available by non-profits or the government, scraping data directly from the web gives companies and individuals a tremendous level of freedom and flexibility when working on solving problems through data. Scrapy, BeautifulSoup and Selenium are a few of the popular web scraping python libraries used currently. In this project, we get hands on with Scrapy and explore how…


Part III: Pagination

Image for post
Image for post
Photo by Aaron Burden on Unsplash

Introduction:

Welcome to the third and final part of the yahoo finance/ scrapy web scraping tutorial. If you have not read the previous parts, I recommend that you do so by clicking here (part-I, part-II) as the following tutorial builds upon them.

So far, we have loaded the most active stocks page on yahoo finance(URL-https://finance.yahoo.com/most-active/ ) and scraped all the stocks which appear on page 1 in a .csv file. In this tutorial, we will cover how to navigate across pages(by clicking on the ‘Next’ button at the bottom of the most active stocks page) and scrape data across all…


Part II: Following links

Image for post
Image for post
Photo by Jaden Barton on Unsplash

Introduction:

This is a continuation of the Yahoo Finance/Scrapy Web Scraping Tutorial. If you have not read the previous part, I recommend that you do so by clicking here as the following tutorial builds upon it.

In the previous part, we were able to get data from the stock summary page into a .csv file. In this part we will get the stock data a list of all the stocks on a single page on the most active stocks page.

Overview: In part 2, we want our spider will do the following:

  1. Load the most active stocks page.
  2. Get a list…

Immanuel Ryan Augustine

Data Analytics Graduate student, aspiring full stack data scientist. LinkedIn: linkedin.com/in/immanuelryan

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store