Supercharging your Reddit API Access
This was originally posted to a former employer's blog. It has been mirrored here for posterity.
Here at HumanGeo we do all sorts of interesting things with sentiment analysis and entity resolution. Before you get to have fun with that, though, you need to bring data into the system. One data source we've recently started working with is reddit.
Compared to the walled gardens of Facebook and LinkedIn, reddit's API is as
open as open can be; Everything is
nice and RESTful, rate limits are sane, the developers are open to enhancement
requests, and one can do quite a bit without needing to authenticate. The most
common objects we collect from reddit are submissions (posts) and comments. A
submission can either be a link, or a self post with a text body, and can have
an arbitrary number of comments. Comments contain text, as well as references
to parent nodes (if they're not root nodes in the comment tree). Pulling this
data is as simple as GET http://www.reddit.com/r/washingtondc/new.json.
(Protip: pretty much any view in reddit has a corresponding API endpoint that
can be generated by appending .json to the URL.)
With little effort a developer could hack together a quick 'n dirty reddit
scraper. However, as additional features appear and collection-breadth grows,
the quick 'n dirty scraper becomes more dirty than quick, and you discover
bugsfeatures that others utilizing the API have already
encountered and possibly addressed. API wrappers help consolidate communal
knowledge and best
practices for the good of all. We considered
several, and, being a
Python shop, settled on PRAW (Python Reddit API
Wrapper).