On Message with Ben Gross
Web-based Ad Hoc Tools
The current rate of evolution on the Web makes it a good time to trend watch and an even better time to build new tools, products, and services. In many ways, we are entering a world where an average information worker can be effective with a single desktop application-the browser. Due to recent technical improvements, Web-based applications are approaching the usability level of desktop applications. An interesting development is new options for assembling simple, yet effective applications, often called mashups, from ad hoc combinations of services and data from around the Web.
The collection of loosely coupled systems that comprise the Web are rapidly converging on a set of semi-standardized syndication formats, typically a variant of RSS or Atom. Most variants are closely related, so it is increasingly common for systems to be able to handle the majority of the formats without requiring users to understand the difference. This standardization of formats greatly increases the options available to programmatically access the Web. Systems are commonly able to both produce and consume these syndication formats fitting well with David Weinberger's description of the Web as "small pieces loosely joined."
APIs and Mashups-A Delicate Balance
Many popular services from major vendors, such as Google, Yahoo!, and eBay among others, offer open Application Program Interfaces (APIs) to access their services. APIs offer a programmatic way for external tools and services to directly interact with other services and data in order to build new applications and services.
Mashups are a popular new type of application often created from a mixture of data, services, and other functions from both local and remote services. Think of mashups as a extreme manipulation of a existing applications, data, and services. Mashups can be created server side or directly in the browser using a variety of languages and tools.
There are a few disadvantages to constructing and relying on services built in this manner. First, you don't have control of the infrastructure and there are often multiple points of failure. Second, the security and privacy of your data can be difficult to determine. Finally, debugging can be hard to impossible since there many be many far-flung parts with unknown failure states.
For better and worse, once the content exits the provider's servers it can be manipulated. Screen scraping is a typically unsupported technique to access functionality and data when there is no API provided-effectively synthesizing an API. Often, consumers of data have more control than they think, while providers have less control than they wish.
There is, however, a delicate balance. When service providers feel they are not deriving value from developers using their APIs, they may take steps to block access. On the other hand a number of companies have created their own APIs when they find people building services on top of their own that still drive traffic back to them and help increase adoption. For example, Google did not originally provide an API with the Google Maps service. However, many developers built services that utilized the Google Maps. Once Google recognized the value of providing mapping infrastructure for others to build upon, the company developed their open APIs for their mapping service.
Yahoo! Pipes
Yahoo! Pipes includes a browser-based graphical (drag and drop) integrated development environment, which is reasonably straightforward to use. The built-in visual editor is basically a graphical development environment that is usable by non-programmers. The service provides a number of modules to extract, transform, merge, or interconnect data from a variety of services. All output from these combinations can be delivered as RSS.
The end-result of a Pipe is that it might be used for yet another service. Take a minute to grasp the implications-it is quite powerful. One example should be illustrative. The Pipe "Apartment Near Something" asks the user for the desired area of the apartment, the maximum distance to search as wells as a "feature", such as a park. The Pipe then searches Craigslist housing listings, uses it's internal location extractor to extract the place names, restricts listings to those with the desired feature and area covered, and finally creates an RSS feed with the results.
The name Pipes is taken from the Unix pipe operator, which allows output of one program or operation to be the input to another. Pipes essentially provide a standard mechanism for a program to feed processed data to another program. A nice feature is that sharing is built in. You can view other people's Pipes, make a clone, and modify them for your own needs. Many of the features can have a difficulty level roughly comparable to creating Microsoft Excel formulas or making transitions in PowerPoint. Yahoo! released Pipes as a beta service in early February 2007 and has been steadily improving the functionality, stability, and documentation. The advantage of a service like Yahoo! Pipes is that people can create ad hoc, flexible, rapid prototypes of software, services, and communication tools. Many functions that previously would have required significant programming experience can now be accomplished with a visual interface. These tools require no installation or configuration for the enduser. Successful ad hoc creations can be optimized and scaled. Today, Web applications such as Pipes allow the distinction between consumers and producers to blur. BG/TMP