Web Scraping For Profit Reddit

Praw is a Python wrapper for the Reddit API, which enables us to use the Reddit API with a clean Python interface. The API can be used for webscraping, creating a bot as well as many others. This article covered authentication, getting posts from a subreddit and getting comments. Web scraping is used to scrape the data from different websites and glean actionable intelligence from these sites in terms of equity research. Extracting Financial Statement Analysts need financial statements in order to determine the health of a company and advise their clients regarding whether to invest or not to invest in it. Reddit web app with Instagram's UI. Using express in the backend, Reactjs in the frontend, and Reddit API / scraping for data. jaesharma/redditfornormies.

I made a Python web scraping guide for beginners I've been web scraping professionally for a few years and decided to make a series of web scraping tutorials that I wish I had when I started. The series will follow a large project I'm building that analyzes political rhetoric in the news.

Thousands of new images are uploaded to Reddit every day.

Downloading every single image from your favorite subreddit could take hours of copy-pasting links and downloading files one by one.

A web scraper can easily help you scrape and download all images on a subreddit of your choice.

Web Scraping Images

To achieve our goal, we will use ParseHub, a free and powerful web scraper that can work with any website.

We will also use the free Tab Save Chrome browser extension. Make sure to get both tools set up before starting.

If you’re looking to scrape images from a different website, check out our guide on downloading images from any website.

Scraping Images from Reddit

Now, let’s get scraping.

  1. Open ParseHub and click on “New Project”. Enter the URL of the subreddit you will be scraping. The page will now be rendered inside the app. Make sure to use the old.reddit.com URL of the page for easier scraping.

NOTE: If you’re looking to scrape a private subreddit, check our guide on how to get past a login screen when web scraping. In this case, we will scrape images from the r/photographs subreddit.

  1. You can now make the first selection of your scraping job. Start by clicking on the title of the first post on the page. It will be highlighted in green to indicate that it has been selected. The rest of the posts will be highlighted in yellow.
  1. Click on the second post on the list to select them all. They will all now be highlighted in green. On the left sidebar, rename your selection to posts.
  1. ParseHub is now scraping information about each post on the page, including the thread link and title. In this case, we do not want this information. We only want direct links to the images. As a result, we will delete these extractions from our project. Do this by deleting both extract commands under your posts selection.
  1. Now, we will instruct ParseHub to click on each post and grab the URL of the image from each post. Start by clicking on the PLUS(+) sign next to your posts selection and choose the click command.
  1. A pop-up will appear asking you if this a “next page” button. Click on “no” and rename your new template to posts_template.
For
  1. Reddit will now open the first post on the list and let you select data to extract. In our case, our first post is a stickied post without an image. So we will open a new browser tab with a post that actually has an image in it.
  2. Now we will click on the image on the page in order to scrape its URL. This will create a new selection, rename it to image. Expand it using the icon next to its name and delete the “image” extraction, leaving only the “image_url” extraction.

Adding Pagination

ParseHub is now extracting the image URLs from each post on the first page of the subreddit. We will now make ParseHub scrape additional pages of posts.

  1. Using the tabs at the top and the side of ParseHub return to the subreddit page and your main_template.
  2. Click on the PLUS(+) sign next to your page selection and choose the“select: command.
  1. Scroll all the way down to the bottom of the page and click on the “next” link. Rename your selection to “next”.
  1. Expand your next selection and remove both extractions under it.
  2. Use the PLUS(+) sign next to your next selection and add a “click” command.
  3. A pop-up will appear asking you if this a “next page” link. Click on Yes and enter the number of times you’d like to repeat this process. In this case, we will scrape 4 more pages.

Running your Scrape

It is now time to run your scrape and download the list of image URLs from each post.

Start by clicking on the green Get Data button on the left sidebar.

Web Scraping For Profit Reddit Post-op

Here you will be able to test, run, or schedule your web scraping project. In this case, we will run it right away.

R Web Scraping

Once your scrape is done, you will be able to download it as a CSV or JSON file.

Web Scraping Reddit Python

Downloading Images from Reddit

Now it’s time to use your extracted list of URL to download all the images you’ve selected.

For this, we will use the Tab Save Chrome browser extension. Once you’ve added it to your browser, open it and use the edit button to enter the URLs you want to download (copy-paste them from your ParseHub export).

Web Scraping Reddit

Once you click on the download button, all images will be downloaded to your device. This might take a few minutes depending on how many images you’re downloading.

Closing Thoughts

Web Scraping For Profit Reddit Download

Service

You now know how to download images from Reddit directly to your device.

If you want to scrape more data, check out our guide on how to scrape more data from Reddit, including users, upvotes, links, comments and more.