Hack 100 Using All Consuming's SOAP and REST Interfaces

You can retrieve a list of the most-mentioned books in the weblog community, as well as personal book lists and recommendations, through either of All Consuming's two Web Service APIs.

This hack could represent the future of web applications. It glues together pieces of several Web Service APIs, and then in turn offers an API to its features. If someone were to create a derivative application with this API, it would represent a third layer of abstraction from Amazon's service. Entire lightweight services may someday be built layer upon layer like this—with dozens of interconnected applications exchanging data freely behind the scenes.

All Consuming [Hack #47] is a fairly small application built on top of a mountain of information that has been made freely available through Web Services. Amazon's Web Services fuel the invaluable book information, Google's API allows me to get related web sites for book titles, and weblogs.com has an XML file that lets me know which web sites have been updated each hour. Combining these three services, I can create lists of books that are being talked about on the Web. It only makes sense for me to give back to this generous community by opening up SOAP and REST interfaces to All Consuming's information, to be used for free and in any way that can be invented.

100.1 The SOAP Code

Here's an example of how you can access All Consuming information on your own using SOAP and Perl [Hack #88]. Create a file called display_weekly_list_with_soap.cgi:

#!/usr/bin/perl -w
# display_weekly_list_with_soap.cgi

use strict; 
my ($hour,$day,$month,$year) = qw( 12 05 28 2003 );

use SOAP::Lite +autodispatch => 
    uri => 'http://www.allconsuming.net/AllConsumngAPI',
    proxy => 'http://www.allconsuming.net/soap.cgi';

my $AllConsumingObject = 
AllConsumingAPI->new(
                   $hour,  # optional
                   $day,   # optional
                   $month, # optional
                   $year   # optional
                  );

This creates a new object, $AllConsumingObject, which you can then use to retrieve the following types of data.

100.1.1 Most Mentioned Lists

Every hour, All Consuming crawls recently updated weblogs to see if any new books were mentioned for the first time on any given weblog. It combines this information with Amazon.com's Web Services, aggregates frequently mentioned books into hourly and weekly lists, and archives them all the way back to August 2002. GetHourlyList will send you the most recent hour's list information, GetWeeklyList will send you the most recent aggregation of all activity during the last week, and GetArchiveList will send you the hourly or weekly list that corresponds with the date that you specified when creating the object (the $hour, $day, $month, and $year variables). For example:

my $HourlyData = $AllConsumingObject->GetHourlyList;
my $WeeklyData = $AllConsumingObject->GetWeeklyList;
my $ArchivedData = $AllConsumingObject->GetArchiveList;

100.1.2 Personal Book Lists

People have created their own book lists directly through All Consuming, assigning them to categories like "Currently Reading" [Hack #48], "Favorite Books," and "Completed Books." Although some of these lists are available for use on other sites through other methods like JavaScript includes, if you wanted to completely integrate the "Favorite Books" list with your site, you'd have to use the SOAP or REST interfaces to do so.

my $CurrentlyReadingList = $AllConsumingObject->[RETURN]
GetCurrentlyReadingList('insert name');

my $FavoriteBooksList = [RETURN]
$AllConsumingObject->GetFavoriteBooksList('insert name');

my $PurchasedBooksList = [RETURN]
$AllConsumingObject->GetPurchasedBooksList('insert name');

my $CompletedBooksList = [RETURN]
$AllConsumingObject->GetCompletedBooksList('insert name');

100.1.3 Book Metadata and Weblog Mentions

Some users have added valuable metadata about books, like first lines and number of pages. This is mostly for fun, and allows me to have an hourly "first line trivia" question on my home page to see if people can guess the book that a given first line comes from. In any case, if you want to retrieve book metadata for a given book, you can do so with the following method:

My $MetadataForBook = $AllConsumingObject->GetMetadataForBook('insert [RETURN]
ASIN');

The argument passed in is the ISBN for the book you'd like to retrieve metadata from. For a list of metadata that's currently available for use, you can check out the metadata scorecard at All Consuming at http://www.allconsuming.net/scorecard.html.

Alternatively, if you'd like to receive a list of all the weblogs that have mentioned a particular book, you can do so using the following method:

My $WeblogMentionsForBook = $AllConsumingObject-> [RETURN]
  GetWeblogMentionsForBook('insert ASIN');

100.1.4 Friends and Recommendations

All Consuming also tracks "friend relationships" between people who have marked their favorite web sites so they can keep track of what they're reading, as well as book recommendations based on the network of all those relationships. You can get a list of web sites that you or someone else has marked as a favorite, or "friend," by including your weblog URL:

my $Friends = $AllConsumingObject->GetFriends('insert URL');

And to get a list of books that all of your friends are currently reading, sorted by those that are mentioned recently and the most times, you can do this:

my $Recommendations = $AllConsumingObject->GetRecommendations('insert URL');

To iterate through the results that these methods return, you can do something like this:

# The array here may differ depending on the type of data 
# being returned
print "Content-type: text/html\n\n";
if (ref($WeeklyData->{'asins'}) eq 'ARRAY') {
    foreach my $item (@{$WeeklyData->{'asins'}}) {
        print "<p>TITLE: $item->{'title'}<br />",
              "AUTHOR: $item->{'author'}</p>";
    }
}

Of course, in either of the above examples, you can change the URL passed through to any other URL. For a full list of methods that you can invoke on this object, visit the instructions (http://allconsuming.net/news/000012.html) and code samples (http://allconsuming.net/soap-code-example.txt).

100.2 The REST Code

For those who think using SOAP is overkill for simple applications like this, you can get the exact same information REST-style. Add this code to a file called display_weekly_list_with_rest.cgi:

#!/usr/bin/perl -w
# display_weekly_list_with_rest.cgi

use strict;
use LWP::Simple;
use XML::Simple;

# Any of the URLs mentioned below can replace this one
my $URLToGet = 'http://allconsuming.net/rest.cgi?weekly=1';

my $XML = get($URLToGet);
my $ParsedXML = XMLin($XML, suppressempty => 1);
print "Content-type: text/html\n\n";

# The array here may differ depending on the type of data 
# being returned
if (ref($ParsedXML->{'asins'}) eq 'ARRAY') {
  foreach my $item (@{$ParsedXML->{'asins'}}) {
     print "<p>TITLE: $item->{'title'}<br />", "AUTHOR: [RETURN]
$item->{'author'}</p>";
  }
}

Here's a list of the URL formats you can access via HTTP to return XML data directly.

100.2.1 Most-Mentioned Lists

Here's the REST interface for requesting the hourly and weekly most mentioned lists:

http://allconsuming.net/rest.cgi?hourly=1
http://allconsuming.net/rest.cgi?weekly=1

If you'd like to retrieve an archived list of most-mentioned books, you can specify the date like so:

http://allconsuming.net/rest.cgi?archive=1&hour=12&day=12&month=5&year=2003

100.2.2 Personal Book Lists

To retrieve any of your categorized books in an XML format, add your username to any of the following URLs (note the category names):

http://allconsuming.net/rest.cgi?currently_reading =1&username=insert name 
http://allconsuming.net/rest.cgi?favorite_books =1&username=insert name 
http://allconsuming.net/rest.cgi?purchased_books =1&username=insert name 
http://allconsuming.net/rest.cgi?completed_books =1&username=insert name 

100.2.3 Book Metadata and Weblog Mentions

To get XML data about specific items, include the ASIN in these URLs. You can get either the item's metadata or weblog mentions:

http://allconsuming.net/rest.cgi?metadata =1&isbn=insert ASIN 
http://allconsuming.net/rest.cgi?weblog_mentions_for_book =1&isbn=insert ASIN 

100.2.4 Friends and Recommendations

To find XML data that includes friends or recommendations for a given weblog, you can include the weblog's URL in the appropriate format:

http://allconsuming.net/rest.cgi?friends =1&url=insert weblog URL 
http://allconsuming.net/rest.cgi?recommendations =1&url=insert weblog URL 

100.3 Running the Hack

Both of these example CGI scripts should be uploaded to your web server and called from your browser. They should display a list of titles and authors.

The returned output of both the SOAP and REST interfaces will be XML that looks something like this:

<opt>
  <header 
    lastBuildDate="Sat May 28 13:30:02 2003" 
    title="All Consuming" 
    language="en-us" 
    description="Most recent books being talked about by webloggers." 
    link="http://allconsuming.net/" 
    number_updated="172" 
  />
  <asins 
    asin="0465045669" 
    title="Metamagical Themas" 
    author="Douglas R. Hofstadter" 
    url="http://www.erikbenson.com/"
    image="http://images.amazon.com/images/P/0465045669.01.THUMBZZZ.jpg" 
    excerpt="Douglas Hoftstadter's lesser-known book, Metamagical Themas, 
has a great chapter or two on self-referential sentences like 'This sentence 
was in the past tense.'." 
    amazon_url="http://amazon.com/exec/obidos/ASIN/0465045669/"
    allconsuming_url="http://allconsuming.net/item.cgi?id=0465045669"
  />
</opt>

If multiple items are returned, there will be multiple <asins /> elements.

100.4 Hacking the Hack

Although All Consuming currently surfaces only book trends, it also stores information about other types of items available at Amazon like CDs, DVDs, and electronics. You can't find this information anywhere on All Consuming's site, but if you use either of the APIs to retrieve weblog mentions for an ASIN that belongs to another product category, All Consuming will faithfully return any weblog data that it has for that item.

—Erik Benson