Supercharging your Reddit API Access

This was originally posted to a former employer's blog. It has been mirrored here for posterity.

Here at HumanGeo we do all sorts of interesting things with sentiment analysis and entity resolution. Before you get to have fun with that, though, you need to bring data into the system. One data source we've recently started working with is reddit.

Compared to the walled gardens of Facebook and LinkedIn, reddit's API is as open as open can be; Everything is nice and RESTful, rate limits are sane, the developers are open to enhancement requests, and one can do quite a bit without needing to authenticate. The most common objects we collect from reddit are submissions (posts) and comments. A submission can either be a link, or a self post with a text body, and can have an arbitrary number of comments. Comments contain text, as well as references to parent nodes (if they're not root nodes in the comment tree). Pulling this data is as simple as GET (Protip: pretty much any view in reddit has a corresponding API endpoint that can be generated by appending .json to the URL.)

With little effort a developer could hack together a quick 'n dirty reddit scraper. However, as additional features appear and collection-breadth grows, the quick 'n dirty scraper becomes more dirty than quick, and you discover bugsfeatures that others utilizing the API have already encountered and possibly addressed. API wrappers help consolidate communal knowledge and best practices for the good of all. We considered several, and, being a Python shop, settled on PRAW (Python Reddit API Wrapper).