Over the last several years I’ve been in the midst of helping companies build intelligent content applications using taxonomy, and almost all of these had already spend a pile of cash on search and content management technology, only to see it fall short of their vision. Many had even implemented component content in DITA or another XML vocabulary. Taxonomy helps to bridge the gap in several important ways.

First of all, taxonomy helps companies organize and manage their source content. A well designed taxonomy can become the basis for a metadata and content type strategy for the CMS, and the source of the controlled vocabularies that content authors and publishers use to classify their content. Content classification is important for defining the “aboutness” of content as well as administering it. We all need well-defined, clear, unambiguous terms for administrative metadata, including our organization structures, customers, products, information types, and information security classifications, to name a few important categories. Leaving CMS users to enter this metadata by freely typing exposes us to human errors and inconsistency. At a minimum, we want to maintain an authoritative term list and expose it in drop-down lists for users to select values when they upload content to the CMS. Users are usually able to enter this metadata with low error rates when selecting from controlled term lists, especially if their job is content publishing. The list just makes it easy and consistent.

For “aboutness” metadata, we need to help users be comprehensive and consistent, so we often use the taxonomy to inform a classification engine that analyzes the document and provides suggested metadata to the user. Subject classification schemes are conceptually similar to the subject headings in library card catalog systems – they start with broad domains and categories that are broken into increasingly narrower topic spaces. The big difference is that each organization will need to develop and maintain subject classifications that are relevant to their business content. For example, I’m helping a high tech manufacturer classify technical documents; their taxonomy covers technologies, manufacturing process steps, customer needs / applications, as well as symptoms, fault codes, and root causes for troubleshooting and repairing their products. They had a rich set of terms for all of these topics, some in a corporate taxonomy and others in specific systems for service and quality management.  Putting them into a taxonomy helped us use that information for auto-classification of content. Metadata is proposed to content publishers when they upload it to the CMS, and they can add or remove terms form the proposed metadata using the same taxonomy the classifier used. The result is that documents are now more completely tagged with “aboutness” metadata in the CMS.

Intelligent content doesn’t end with the CMS, however. Search engines classify content using their algorithms to match end-user search queries (what you type in the search box) to content – whether it’s unstructured document content or structured data in an enterprise system, or both. A search engine doesn’t understand your business – it only looks at all of your content as a “bag of words” that it statistically determines to be “about” something looking at unusual combinations of terms,  or high-frequency terms. A taxonomy can tell the search indexing engine that certain terms are more meaningful to your business, and that there are relationships between terms that matter when it comes to relevance.  Even if the search engine is placing higher value on metadata values, those usually contain “preferred” terms – the official business labels. Users, on the other hand, are not so disciplined when they type in the search box – they may use “non-preferred” terms. A favorite example is searching a NASA site for “moon buggy” when NASA calls the item the “lunar excursion vehicle.” The taxonomy can relate those terms so the search engine returns relevant documents – even if they never contain the term “moon buggy” or that referred to the acronym for it.  

Finally, taxonomy can be used to driving the search user experience in major ways. It can become the basis for the facets in search refinement, allowing users to narrow their search along the dimensions of the taxonomy (show me only information about these document types, or these products, etc.). It can define the terms we show in tag clouds and other interface objects. The taxonomy can also help the search engine identify related searches – for instance, all of the astronauts who ever drove a LEV, or the Apollo missions that included a LEV.

I’ve actually seen a recent example of an intelligent content application that is entirely defined in a taxonomy – the UK National Health Service has build a Flash application that lets you navigate a hyperbolic tree of symptoms and diseases, all of which is directly managed in a taxonomy and flowed directly into the portal. Their taxonomy aids search and results relevancy as well by taking terms like “AIDS” and assuring that search engine stemming doesn’t return documents that contain the terms “aid” or “aiding” – imagine all the results for “first aid”, “band aid”, “hearing aid”, or “health aid”.  

We also tend not to think of intelligent content systems as being driven by rather “dumb” legacy content, instead they are all about XML and structured content. In fact, most of the intelligent content portals I’ve worked on in the last several years were being populated by legacy PDF content – which was made intelligent only through the use of auto-classification with a well-crafted taxonomy and exposed through faceted search. For lengthy documents, document preview technologies can help hone-in on relevant pages – also being guided by the taxonomy to map search queries to preferred terms.

My conclusion in working with search and enterprise content management technology is that taxonomy development and management is a key success factor in creating effective intelligent content systems. Taxonomy can inform content types and metadata schema, make for consistent tagging, harmonize disparate structured data, and drive dynamic search and navigation user experiences, even with not-so-intelligent content.

 I’ll be presenting more on this topic with extensive examples at Intelligent Content 2011 in Palm Springs, February 16-18. Hope to see you there!

Take the Survey

Depending upon context, “tagging” can mean one of three different things: tagging a document, tagging within a document, or tagging a content object.

Tagging documents.  These days most of us think of tagging as the keywords we put on our documents – like our photos and websites – so that others can find them when they search.  User tags are fine for finding photos in flickr, but for tagging to be effective in business we need to make it systematic, so that we avoid ambiguity and improve search recall and relevance.  So we’re increasingly “mature” in our approaches to tagging: We use taxonomy to organize our terms into classes and to manage the relationships between terms.  We develop thesauri and foreign language equivalents.  We integrate taxonomies and thesauri into search indexes for ECM and site search and SEO.

Tagging within a document.  I got interested in tagging in the early days of XML (back when we spelled it “S-G-M-L”), when we were tagging within documents.  By tagging unstructured content inside documents we could do really sophisticated things – not just multi-channel output.  For example, knowing that a paragraph in a document was a step in a service procedure or that a string of gibberish was a part number let us bring life to that content when we transformed it from markup into an interactive electronic technical manual.  Tagging let us turn books into diagnostic software.

Tagging reusable content objects. As content reuse matured with standards like DITA, organizations had more reusable components, with more people creating them in more departments.  Tagging reusable content objects became essential to actually reusing them – if you couldn’t find it, you’d never reuse it.  If you had a single service manual with 100 procedures, now you have at least 100 reusable content objects, so the search scope increased by two orders of magnitude.  At IBM, colleagues report having over a million DITA topics in more than six repositories, with over a dozen departments sharing content across thousands of publications.  Searching for content objects is like trying to find a needle in a haystack, except you’re trying to find the right needle, and you have more and smaller needles to search amongst, in more and increasingly bigger haystacks.

Measuring Metadata Maturity.  Each type of tagging can have measurable benefits on your business.  Five years ago, Earley & Associates and Taxonomy Strategies developed a survey to understand metadata maturity for various types of businesses.  Earley is conducting an updated survey to see how organizations have moved up the learning curve.  Since we have a baseline of responses from five years ago, we’ll be able to describe how metadata and taxonomy practices have matured over time.  Also, the original survey was focused on the impact of metadata best practices on knowledge management and e-commerce search.  We now recognize that metadata is also used by technical communicators – especially those that use XML and other technologies to create, manage, and multichannel publish reusable content.  We want to hear from you all for the first time.

The survey is pretty detailed, so you might want to grab your favorite caffeinated beverage before you dig in.  As compensation for your time (about 15 minutes) Earley & Associates is offering these nifty incentives:

• A free pass to any future Earley & Associates Community of Practice conference call (a $50 value).  These are monthly, and the next one is Wednesday June 2^nd on Taxonomy for Portals featuring Giovanni Piazza, Chief Knowledge Officer of Ernst & Young, and Ralph Poole of Earley & Associates.
• A $200 discount on registration to the Henry Stewart conference on digital asset management, June 1-2 in NYC.  Seth Earley will be there presenting preliminary results.
• Free participation in a webcast reviewing the results of the survey (date TBA).

Take the Survey

Google Squared is Google’s answer (or perhaps one of their answers) to semantic search and Linked Data.  ‘Squared gets its moniker from the matrix used for displaying results – each “square” in the matrix contains some fact derived from the content on the source site. 
Each row in the Google Squared matrix is a search result, but the interesting part is the columns.  ‘Squared relies on RDFa and microformats on the source sites to extract structure for the search category – if it’s available (I’m not entirely sure how ‘Squared derives its semantic structure in the absence of metadata, but clearly it does).  So a search on “rollercoasters” will generate columns for height, speed, construction, etc.  Essentially, ‘Squared is generating search facets on the fly using structure that is implied or explicit in the set results set. 

Because sites are inconsistent with the amount of structure they provide, ‘Squared can – and will – make errors in interpreting free text.  For example, in the video, we can see that “height” – while intended to describe the height of the rollercoaster – sometimes returns text about the minimum height requirements for riders.  Still, from the demo ‘Squared does look interesting.  Predictable categories like Restaurants (e.g. a search on “pizza”) have the dimensions you’d expect to see in columns – description, address, price range, ambiance, etc. 

Google Squared affords general-purpose faceted search, because the columns can be used to refine the search results.  The current alpha doesn’t let you sort on a column, but clearly this is where things are heading.

Google Squared will be made publicly available on the Google Labs site in the next week or so. 

Here is the video courtesy of TechCrunch (over six minutes and shaky-cam – but you’ll get the idea).

As a footnote, TechCrunch declares that ‘Squared “is how Google will crush Wolfram|Alpha”.  I’m sorry – they missed on that point by a country mile.  Alpha isn’t a search engine – it’s a user experience (and a cool one at that) built atop a “curated” database, designed to answer queries that are primarily computational in nature.  Google searches the web.  No comparison – the products aren’t in the same class, and don’t solve the same problem.  Google can’t plot the Julia Set for you – Alpha can.  That doesn’t mean Alpha “crushes” ‘Squared, either.  Geez.   Anyway, more on Alpha later.

Several things were important to me about how to do this:

1. I wanted to use this feature to notify about interesting content without a lot of authoring on my part.  Ideally posting an item to the digest would be automated with a semantic search bot pulling in the content.  Worst case I would have to drag and drop headlines and links in a Twitter-like fashion.
2. I didn’t want to add commentary – I wanted a simple re-blog capability.
3. I wanted attribution of the author to be very clear – this was not my content, just items of interest to share.

Inline RSS for a WordPress page

I concluded that what I really needed was a way to display RSS feeds in-line on a WordPress page to create my headlines or digest.  WordPress has loads of plug-ins that are great for integrating RSS feeds into sidebar widgets, but I found one plug-in – DGE inlineRSS – that provided all of the features I needed to put the RSS into the body of a WordPress posting or page. 

inlineRSS is pretty simple to use.  After install, there are three easy steps to getting a feed embedded into a page.  First, you need to configure inlineRSS to point to your feeds.  This is a simple matter of entering the RSS feed URL, and associating an XSLT file that will transform the feed into the HTML that displays on your site (inlineRSS provides a simple XSLT file that you can alter to meet your needs).  Next, you need to be sure that the configuration sets the path to the XSLT file on your site, and make any changes to the XSLT for your unique formatting.  Lastly, you need to enter the embed code for the inline RSS itself – this is simply:

!inlineRSS:myrssfeed

where myrssfeed is any of the feeds you configured in the inlineRSS options screen.  inlineRSS implements a WordPress filter to replace this code with the XSLT-formatted RSS feed.

Creating the Digest RSS Feed

The next step to getting a digest page up and running was to create the source of the RSS feed itself.  I considered three options:

• reFeed – a server for creating custom RSS feeds from items of interest
• Twitter or Ping.fm – generate a “tweetstream” of headlines and links (re-blogging vs. micro-blogging)
• TextWise – using a semantic search bot (specifically TextWise’s Gyzork demonstration app) to auto-generate a custom RSS feed that contains blog posts and news items that match specific semantic signatures

reFeed.  Last month, Mike Axelrod set up a reFeed server that he and I experimented with briefly.  This may hold some promise for the future for hand-selecting items of interest to reblog via an RSS feed.  However, Mike noted some technical issues, particularly with the quality of the RSS, so we tabled that project for the time being (he and I promise to blog more about reFeed in the near future).

Twitter and Ping.fm.  As Mike discusses in a related post, we’ve both been exploring Twitter and Ping.fm to stream reblog-type items to our websites.  Mike has the Ping.fm custom URL plug-in for WordPress working and has his “pingstream” going to headline items in his sidebar.  I’m doing the same, using Ping to drive Twitter, then putting the Twitter feed into my sidebar with the  Twitter Tools plug-in (look under Headlines in the sidebar to your right).  I’m still working on getting my Ping feed to display in-line with RSS or other techiques.

TextWise Gyzork.  You may be familiar with TextWise if only from my use of their technology here for finding related products, Wikipedia articles, and blogs for my blog posts.  TextWise uses semantic analysis to search the web for similar documents.  Each document (web page, blog post, etc.) gets a semantic signature that identifies the concepts in the document and their weight (i.e. relevance).  Gyzork lets me apply a semantic signature to any document, then create a saved semantic search that returns either blog posts or news items that match.  Gyzork also lets me create an RSS feed for that search.  If you check out my Headlines tab above, you’ll see a set of headlines (embedded using inlineRSS) that were generated by a Gyzork search seeded by one of my posts.  This search has been pretty good over the last six months at surfacing items I’m interested in.

Next Steps

I plan to continue to experiment with the automatic news digest.  Things I plan to try out:

• Propagating the news feed items to my status updates in LinkedIn and Twitter
• Getting my Ping.fm stream to display inline as a set of headlines
• Micro-blogging using Ping as the front end
• Working with TextWise technology to create more Gyzork feeds for the news page
• Cleaning up the XSLT to better format the news feed
• Experiment with other semantic technologies to create automatic feeds of interest

If you have any further ideas on this, please comment.

The Content Guy is pleased to have been invited by Earley & Associates to present in their Semantic Technologies Jumpstart series this fall.  This free series runs every Thursday, from October 30th – November 20th, between 11:30 am – 1:00 pm ET. 

Join us November 6, 2008:

Implementing Semantic Search
Presented by Paul Wlodarczyk and Amber Swope
Semantic search helps business people find answers to pressing questions by wading through oceans of information to find nuggets of meaningful information.  In this presentation we’ll discuss how semantic search and content analysis technologies are starting to appear in the marketplace today.  We’ll provide a recap of what semantic search is and what the key benefits are, then we’ll answer the following questions:
• Is semantic search a feature, an application, or enterprise system?
• How can I add semantic search to my existing work processes?
• Will I need to replace my existing content technologies?
• What will I need to do to prepare my content for semantic search?
• Is semantic search just for documents or can I search my data too?
• Can I use semantic search to find information on the internet and other public data sources?
• Are there standards to consider?

This conference call is for implementers and decision makers of all technical levels including those new to semantic technologies.  We will introduce technical concepts in terms everyone can understand.

For more information on the other presentations in the Semantic Technologies Jumpstart series or to register click here.

There were three separate projects in the integrations with the following goals:

1. To expand the number of queries for which Live Search shows Answers (using data from Freebase). 
2. To use Powerset’s semantic technology to generate improved captions for Wikipedia articles (shown below)
3. To use Powerset’s Factz extraction to generate a list of related searches for a set of queries

While the impact of Powerset’s semantic processing on Live Search is still limited, you can see the beginnings of it (that is, if you’re lucky enough to be in the random sample of searches in the experiment): semantically improved summaries in the captions for results that point to Wikipedia articles.

The integration with Live Search is two-way: Powerset.com now boasts related searches that are powered by Live Search.