---

Contents

• Executive Summary
• Objective
• Technical Accomplishments
• Architecture
• Internet Resource Advertisements, Trader Federation, and Rebinding
• WebTrader Query Tool
• Gridifying WebTrader
• Technology Transition
• Next Steps

---

Executive Summary

We demonstrated evolving versions of WebTrader at CoABS Workshops at Las Vegas (January 1999), Northhampton (June 1999), Science Fair in Arlington (October 1999), Atlanta (February 2000), and Boston (August 2000).  Thereafter, we halted the project, per Jim Hendler's guidance, to provide added resources for eGents and MIATA, CoAX and JBI TIEs.

---

Objective

Specific scientific and engineering subgoals were:

• represent Internet resource advertisements for services, agents, data sources, search engines, traders.  (This opens the door to the ontology problem that is the focus of the DARPA DAML program.)
• locate these advertisements via WebTrader which itself piggybacks on one or more Internet search engines that have indexed the resource advertisements
• demonstrate trader federation and rebinding
• re-engineer client-server WebTrader so the client is an applet that can download quickly to most browsers
• explore how to locate and mine local domain search engines as a way to broaden and deepen Web searching
• demonstrate WebTrader as a CoABS grid agent
• demonstrate WebTrader in scenarios of interest to DoD

Technical Accomplishments

Architecture

Internet Resource Advertisements, Trader Federation and Rebinding

• XML-based resource advertisements
• service binding and rebinding
• trader federation

We also made progress on service binding and rebinding and trader federation.  In the Salt Lake City demo, trading ads exist for a variety of components (e.g., service agents, clients, and WebTraders). Metadata including color and cost is included into the ads.  A Blue client, for example, asks the USA WebTrader to locate a Blue agents implementing a particular interface and with zero cost, if possible.  One is found and bound, and the client makes use of the agent.  When the agent unexpectedly dies, the WebTrader is consulted again by the client, in case the “state of the world” has changed.  It gets back a new list of agents, sorts them by cost, and goes down the list trying each agent until it finds one that works.  If the WebTrader fails to respond, the client can fall back on its cached list of previously found agents. When the Yellow client asks the USA WebTrader for Yellow agents, the WebTrader’s initial search turns up none, as it only consults a domain that indexes ads of Red, White, or Blue agents.  However, it does find an ad for a WebTrader that knows about Yellow agents, and so passes on the original client query to the Euro WebTrader, which finds a Yellow agent, passes it back to the USA WebTrader, which passes it back to the client, which then uses it to connect to the agent.  At the right in the figure is shown a service advertisement in XML.  For more detail, see WebTrader Demo Script.

WebTrader Query Tool

In a demonstration from the CoABS Boston Workshop (August 2000), WebTrader Query Tool was coupled with a modified Java App Builder (a visual programming tool), extended with WebTrader drag-and-drop capability.  In the demo, the user creates a want ad called ColorButtonsSpec.xml, drags it to WebTrader Query Tool, runs a query which returns pages containing ads for colored buttons, then selected buttons are dragged to the WebTrader App Builder and connected together to build a simple application.
 
 

WebTrader Query Tool is a generalization, redesign, and reimplementation of previous work in numerous ways -- the more significant changes are listed below.  Almost all code was rewritten from the Salt Lake City implementation.

• advertisements - We redesigned the XML advertisements.  The new design included <!ELEMENT resource ANY> as described above.  We redesigned the WebTrader's JavaRMI DTD (now called JavaRMIObject), modified its CGI DTD, and modified all the example ad web pages to reflect these changes.  We also created a new ad type (via a DTD) called SiteSearch to advertise local search engines.  We located several local site search engines that indexed content having to do with agents and manually created ads for them.  Later, we developed heuristics for discovering unadvertised local search engines on webpages.  If found, we include these as a sub-result of that page and add it to the ad repository - this kind of discovery is a general approach to populating ad repositories.
• applet-servlet architecture - We separating searching from the display the results, and implemented an applet-servlet architecture.  We reimplementing the client-side GUI as an applet, around 30KB in Java 1.02 for widest portability, about half for the Tree GUI component.    The tree component is used to record query histories, multiple live searches, canceling and deleting queries, adding increments to a search (e.g., top 20, 50, or 100 search results.   The applet also supports a polling capability to handle the deep results as well as the top-level ones, scrollbars, and a concern for efficiency, to reduce the number of messages between the servlet and applet, so that web-sized scaling of the search service can be realistically achieved.
• inputs - WebTrader Query Tool can take either search terms or want ads as input.  We made WebTrader Java Web Start-able, ie. one can download and run DeepQ as a desktop application with one click.  When run on JVM 1.2 or later, such as via Java Web Start (JWS) or the Java Plugin, then dragging the WebTrader results over to the Web Trader App Builder (WTAB) is automatically enabled.
• target search engines - We developed a modular search engine results parser and specialized it to Webinator, Google and later to Open Directory. This kind of thing is an ongoing effort, and not just with Google, as SEs all seem to tweak their output syntax from time to time.  We redesigned and implemented the match information returned to clients.  Originally we expected to return W3C DOM parse trees of the XML ad resource to clients as part of this information and build convience routines for clients to pick out desired information from the resource (such as the URL of a Java RMI object to connect to to get the service advertised in the ad).  However, the DOM objects generated by Sun's XML parser are not (directly) serializable and thus not able to be sent through the RMI link the WebTrader uses between its client and server parts.  So, instead we return the raw XML of the resource and use SAX-based XML parsing tools on the client side to accomplish the same thing, a better design since we are sending less information and SAX is lighter weight than DOM.  Later, we developed an algorithm to accurately parse the results of foreign search engines encountered on the fly (or fail safely).
• matchmaking algorithm - We modified the WebTrader metadata matching algorithm to more successfully match metadata dynamically created from user inputted search keywords.  Also, we ideveloped a generic scoring model.
• federation - We developed a new WebTrader federation design.  One insight from the new design of the WebTrader federation scheme is that a WebTrader does not have to wait to determine that it cannot find any or enough matches to satisfy a client; at any point it comes across an ad for another WebTrader that looks promising it can propagate its query to it, and have all the results dynamically merged (sorted) on the client end.  In fact, there is probably not any other practical way - what WebTrader has time for secondary WebTraders (and so on) to build up a list of results - all results must be streamed dynamically to be timely.
• performance - We implemented a new result polling capability.  Now all search engines are equally controllable from the applet, instead of special consideration given to the top level engines in the old design.  Now, the GUI will not keep up constant connection to servlet, but will rather poll for results.
• security - We modified the applet GUI to better work as a JWS app for the situation where there is no web page that the applet is embedded in.

Gridifying WebTrader

For the Science Fair (October 1999), we gridified the WebTrader Agent (WTA).  Previously, in NEO TIE #2 (see description below), the WebTraderAgent (WTA) was used  as a WAN matchmaker to provide SRI OAA and USC/ISI Ariadne the ability to dynamically extend its information sources based on user queries it received.  For the Science Fair, we modified the WTA to query the Grid for any WebTraders registered there.  If none is found (or timeout), then the WTA automatically falls back to its local embedded WebTrader.  This essentially extends CoABS grid JINI lookup service to be a WAN lookup service.  The implementation involved creating the WebTrader Grid Service (WTGS), a Java program that registers a WebTrader on the CoABS Grid and logs its actions, along with a new version of WebTrader Agent (WTA) that lists WebTraders found on the Grid.  We worked with Hank Seebeck and Adam Wenchel (both GITI) to test WTA on the grid.  For the Boston CoABS Workshop (August 2000), we upgraded WebTrader Query Tool to use the grid in the same way as shown in the diagram below.

We also created a launch page for a CoABS 7x24 grid accessible version of DeepQ.  WTA was maintained on the CoABS 7x24 grid for approximately a year.

Technology Transition

In March 2000, we presented work on Agent Discovery Matchmaking to OMG, essentially, a sketch of a reference model for traders that subsumes WebTrader as well as conventional agent matchmakers and LAN traders like OMG's Trader.

Next Steps

It is worth noting that the basic design of WebTrader appears to be that of an open world repository, that is, ads can come from any pages that search engines index.  Interestingly, it is not a large change to convert WebTrader into a closed world search engine only containing ads that are inserted into its database (instead of SE index).  This might make more sense for some kinds of services, e.g. resume services or a company's web services, where the service provider might want control over who can advertise.

Although robust in several ways, the WebTrader implementation is still a prototype.  More work is needed in several areas:

• We would like to explore type specific matchmaking so different matchmakers could be plugged into WebTrader on-the-fly depending on the type of advertisement.  We ran into this requirement in the NEO TIE where it would have been useful to match the predicate, the arguments, and the variables in OAA solvables so only exact matches would be found
• We need more work on the algorithm for parsing results of search engines to extend the meta search capability.
• For many purposes, it makes more sense to index advertisements during the crawling phase rather than when pages containing ads are returned from search engines.
• We would like to remove the dependence on Thunderstone's Webinator which requires a live Internet connection (so it can check the online license at Thunderstone), does not always rank the results of OR queries correctly, does not support some of the query logic we need, and does not index XML pages.  Open Directory ISearch is a potential alternative.
• We would like to use toolkits flike WEBL and XWRAP Elite for web page content extraction as a way to locate SEs on web pages faster.

© Copyright 1998, 1999, 2000, 2001 Object Services and Consulting, Inc.  All rights reserved.  Permission is granted to copy this document provided this copyright statement is retained in all copies. Disclaimer: OBJS does not warrant the accuracy or completeness of the information in this survey.

Last revised:  October 2001.  Send comments to Craig Thompson.

Acknowledgements:   Paul Pazandak did most of the design and implementation with Craig Thompson providing brainstorming and review.  Steve Ford installed and demoed MBNLI at CoABS Workshops that Pazandak did not attend.  Thompson completed the CoAX TIE demo.