In Developing a Custom Taxonomy Only Time Can Tell

OK Quick Monday Quiz: How Many Minutes Does It Take to Create a Category (aka term, node, leaf, etc)???

I suspect that anyone who has worked on developing a taxonomy has heard this question or a variation of it. It seems like we get it daily! Once a client decides they need or want a taxonomy – they need or want it immediately so figuring out when becomes the next question.

After almost 30 years of being involved in the development of controlled vocabularies, thesauri and taxonomies I should be able to say it takes X minutes per term but I’m still forced to tell clients that it will depend on a number of things that are usually covered in the Assessment Phase of any engagement like:

• What is the topic of the taxonomy?
• What is its intended purpose?
• What systems will you use to develop and maintain it?

Once we’ve answered all these questions, the next one is frequently whether they could just use a taxonomy that is already developed. No matter what approach is ultimately chosen to create a taxonomy – it still takes time and the ultimate answer is that it depends on what the client needs, how many terms there will be, how technical those terms are and the taxonomy development tool that is being used.

Building a taxonomy for an area that you are familiar with can be done fairly quickly while building one on scientific, technical or medical areas might be much slower. Adding to the issue of the topic is the issue of the tool where the taxonomy is being built. The more efficient the tool the faster the development once terms have been decided upon and research for the terms completed.

Experience in developing taxonomies has given me some general metrics that can be used for pricing a taxonomy but the reality is that the best answer is that it all depends on what is needed.

So – how long does it take?? – it takes as long as necessary!!


Taxonomies are a Commodity

For some reason or another (lots of travel, several hats at home and work) I've had trouble finalizing this post. Earlier today though, I read Paul Miller's latest post on ZDNet. There seems to be some discussion about whether or not data is a commodity. I think there IS most definitely data that are a commodity.

Taxonomies are a valuable raw material in the management of information. A file that can be bought and sold and used to improve services. They can be generated by humans, machines, or even better: humans working with machines. Monkey Chewed Coffee Beans 4Many taxonomies are a dime a dozen, with little to differentiate between versions of the same data. Some are like Kopi Luwak coffee - rare and extremely valuable. The word "taxonomy" is itself suffering from a kind of genericide. Classical definitions still apply: taxonomies have become commoditized.

The complexity of the controlled vocabulary will determine its value to a degree. A simple pick list should be easy and cheap to acquire - a list of countries, for example. Or colors, seasons, months - you get the idea. What is the value of a list of industries? Or companies? Maintenance is the primary cost factor - frequent changes require frequent updates, but an authority file in and of itself is not that complex. A broad and deep poly-hierarchical taxonomy I would expect to have more value. A poly-hierarchical taxonomy is one where a term in the taxonomy can have more than one parent term. Managing these relationships takes more time. An ontology - well, those aren't quite commodities yet, but they will get there. Why? Because they still require a great deal of thought and effort.

The source of the data will also help determine its value. Data from trusted sources - for whom integrity is paramount - should be valued higher. Is the data accurate? Is it maintained? Is it in a usable format? Does it have high availability? (Many quality vendors can be found at

The uniqueness of the taxonomy will drive its value. Like our coffee example above, a taxonomy as ubiquitous as Starbucks will not be as valuable as say a pharmaceutical research vocabulary. Given the, uh, processes needed to produce Kopi Luwak, it is rare and therefore fetches a higher price, as would our R&D taxonomy.

The information security concerns also impact value. Our pharmaceutical company, or a financial services provider, is not about to release it's vocabulary into the wild. It is a significant intellectual asset that merits a substantial IT effort to protect.

I actually like the fact that taxonomies have become commoditized. Why? Competition drives improvement - in quality, in focus, in security and in usability. These are areas that the semantic web community needs to focus on - in my experience, security and usability need attention NOW. Good fences make good neighbors, and when we've got good fences, we can make more links and learn to trust. Icing on the cake!

Flickr image by INeedCoffee

Synaptica Featured in New Report on Industry and Leading Vendors in the Semantic Web Space

The Synaptica team is in Denver this week doing strategic planning, or as I say, scheming ! ;) There are a great number of really interesting problems in information management, and it's fun and rewarding to brainstorm ways of solving them. We're not the only ones scheming and it's great to see the market itself growing.

David Provost this week published a Global Review of the Industry and Leading Vendors in the Semantic Web space titled On The Cusp: A Global Review of the Semantic Web Industry in which Synaptica from Dow Jones and Dow Jones Client Solutions were highlighted. Dow Jones is in a unique position as a software vendor, consulting services provider and deployer of semantic solutions, which made for a great conversation - I highly recommend you read the report. (Not just because I was involved!)

Paul Miller has posted on his ZDNet blog a review of the report New report places Semantic Web ‘On the Cusp’ of something big. Paul adds some great commentary to his summary of the report, should you not be able to get through David's entire document at once.

Happy Reading!

Congratulations to Gabe Rivera on being named to Business Weeks 25 Most Influential People on the Web

I am a huge fan of Techmeme and therefore of Gabe Rivera who just yesterday got named by Business Week as one of the 25 Most Influential People on the Web!!

So i recorded this message earlier today:

For the month of September, i managed to convince our wonderful marketing department to have our new Dow Jones Synaptica Central Blog as one of the sponsored posts on Techmeme and we have certainly had success in driving traffic and interest in the Taxonomy side of the Dow Jones Client Solutions business.

I was an early fan of Techmeme - for example here is a video of me and Robert Scoble in February 2007 showing Clare Hart, Dow Jones EVP, Techmeme and can you can hear how excited i get.

Well today is the last day of the month- and we decided to skip the month of October as a sponsored post - but we hope to be back soon!

Congratulations once again Gabe and thanks for letting us be a sponsored guest to get our Synaptica Central blog off the ground!

Centralized Taxonomy Management and Synaptica Case Study by ProQuest

At last week's conferences we had the wonderful opportunity to show the attendees how one of our customers utilizes Synaptica our Taxonomy and Metadata Management software.

Paula McCoy, Taxonomy Manager at ProQuest joined me on stage for two sessions during Enteprise Search Summit and Taxonomy Bootcamp. ProQuest provides global access to one of the largest online content repositories in the world and Paula is responsible for maintaining their controlled vocabularies that both editor's and end users use.

The case study addresses the challenges ProQuest faced in managing multilingual controlled vocabularies using multiple Word documents and authority files maintained in an Oracle database. During her presentations, Paula describes how implementing a thesaurus management tool helped ProQuest simplify and standardize its business semantic management to create a common language and connect disparate information assets as well as handling large and varied vocabularies and authority files, linking new and existing editorial systems and enabling hierarchical views, and automating thesaurus management tasks.

Note: Unfortunately the video does not focus on the slides during the taping like some of my other videos i posted from the conference so i have embedded the slides for you to follow along (maybe open a second window and click through).

The first session is 30 minutes and i go through what is and why customers use Centralized Taxonomy Management tools in the first 15 minutes and then Paula presents how she uses Synaptica daily to maintain the ProQuest taxonomies.

Centralized Taxonomy Management for Enterprise Information Systems

The second is a 45minute full Case Study (you only hear me introduce Paula and make some comments about how we need to make Taxonomy exciting!)

Finding a Common Language: Bringing Complex and Disparate Vocabularies Together

Thanks again to Paul McCoy who did an awesome Job!! Thanks!

Enterprise Search Summit: Terry Jansenn - “An Integrated Continuum of Search Technologies”

Terry Janssen of Lockheed Martin gave a very interesting presentation on Wednesday the 24th at the Enterprise Search Summit West in San Jose. During his talk he discussed some of the technologies that Lockheed employs as a systems integrator to improve search within their own and other enterprises.

A common theme was revealed during his talk that was repeated by in several other presentations and by vendors at the Summit and its sister conference, Taxonomy Bootcamp. Namely, that taxonomies are the key to improving and refining enterprise search.

The Eight Habits of Successful Taxonomists

I think next year i am going to do a Top Ten reasons you should your Taxonomist as a late afternoon skit to get Taxonomy Bootcamp attendees jamming after those heavy lunches! Back to back sessions on the subject of taxonomy sometimes needs a little excitement injected into the day so i am plotting already...

This year one of the sessions i enjoyed the most was Gary Carlson's session on 'The Eight Habits of Successful Taxonomists' that he kindly let me video tape so we can share with all of you who did not make it to Bootcamp this year.

Gary runs his own consulting company called Gary Carlson Consulting and has over 20 years of experience as a taxonomist, consultant, product manager, and information manager working for small to Fortune 100 companies. [Click full screen toggle from panel below for best quality]

So what are the eight habits of a successful taxonomist?

#1- Sets expectations
#2- Knows the Technology
#3- Pays Attention to Workflow
#4- Avoids Religious Wars "Leave dogma at the door!"
#5- Follows the $$ Money $$
#6- Is a Good Listener
#7- Does not use the word 'taxonomy' in polite company
#8- Is a good Juggler!

Some good practical advice that Gary shared with us in conclusion to his presentation:

1. Identify the business problem at the start of the project
2. Gathering requirements for a taxonomy is a huge process and can lead to many different areas of the organization
3. Align your projects with the business and preferably with generating revenue rather then efficiency
4. And most importantly? Have Fun!!!

You can download the complete slide deck here:

What do you think are other habits that a successful Taxonomist might have? Please leave them in the comments!

Juggler Image| Flicker |Marco Fedele1089

Taxonomy and Resource Location: Finding the Who and the What Using Controlled Vocabularies

While I was at Enterprise Search Summit and Taxonomy Boot Camp, I heard some really interesting presentations. On Wednesday I sat in on a presentation by Ahren Lehnert titled "Taxonomy and Resource Location: Finding the Who and the What" . This was a really good presentation because it addressed some of the ways we have been advocating using 'taxonomies' that might be new approaches for some and therefore it resonated with me because it’s something I have been speaking to my customers about.

Ahren’s presentation was about how it can be difficult for an organization to capture and retrieve knowledge and expertise held by resources, both internal and external. He discussed how combining taxonomy and search can help organizations with resource location.

He noted that one challenge is finding the right person with the right skills. In a large company this can be difficult because there are so many roles and such a variety in skill sets among employees. In small organizations the difficulty stems from having fewer roles, and therefore, fewer skill sets. He also made the point that job titles don’t always indicate what knowledge and skill sets are associated. An additional challenge can be the result of a merger or acquisition—each company could have unique titles for the same position, or the same title could be used to describe different positions. The information about these resources can be found in a variety of places—wikis, blogs, human resources applications, sales applications, project management applications, and content management systems. The key is to be able to surface content from all of these repositories through a single search.

Here is an Example:
Employee X creates a user profile on the wiki and lists her job skills and interests. She also co-authors a report for a shoe manufacturer. She then attended a seminar on SharePoint and Taxonomy and blogged about it. HR already has her resume on file, which lists her former positions.

If I am working on a project with that same shoe manufacturer, I might be interested in talking with others who have expertise in that area. Employee X could be a great resource, but it is possible I don’t know that she exists, much less that she has experience working with that particular client. If I could search across all the repositories of information, and a controlled vocabulary were leveraged in that search, I should be able to find Employee X and contact her about the project. The knowledge is there, it just needs to be organized with a taxonomy and retrieved using search combined with the taxonomy.

When we work on client engagements, we work directly with the client to assess the various repositories of information and develop taxonomy strategies that many times includes vast amount of information about their employees. Using our Process Model for Developing & Deploying Taxonomies we can also build custom taxonomies that can be leveraged in the client’s enterprise search solutions. Often however a client will also have an existing taxonomy in place so it just needs to be enhanced to meet the current needs as well as expand it out for expertise location.

In conclusion- controlled vocabularies can and should be used in various ways to assist corporate users in finding information- from finding the right report with a quick search to finding the right person to validate an opportunity with a quick call or email and Ahren's presentation at the Enterprise Search Summit gave us all some good examples of how and why this should be done.

Image| created with

Taxonomies for Human Vs Auto-Indexing by Heather Hedden

Day 1 at Taxonomy Bootcamp covered a lot of basic taxonomy principles such as planning and implanting taxonomy, choosing taxonomy software and indexing principles. The talk by Heather covered the perennial issue of human vs. auto-indexing and whether it was possible to ascertain whether one was better than the other or not. Ultimately, whichever method was selected, it depended on the purpose of the taxonomy and its use. It was emphasized that indexing was best used for search and retrieval.

Before the virtues and drawbacks of each indexing method were explored, Heather provided clarity on what indexing was and how it differed from tagging and categorisation. In a nutshell:

• Indexing is done by a trained indexer, preferably with subject matter knowledge and is largely used for browsing.
• Tagging can be done by anyone and is the applying of labels to documents. These tags can then be used by a database.
• Categorisation is the grouping and placing of information in buckets in a systematic manner.

The differences in human and auto-indexing were covered in 3 broad areas, namely contents/materials handled, methodology and technology. In terms of contents, human indexing would be at its best if the contents were in manageable numbers and included a variety of formats and subjects/topics. On the other hand, auto-indexing would work well for very large numbers of documents, textual documents (no images!) and single subject areas.

Technology-wise, indexers (humans) use fairly simple and straightforward indexing tools which were designed so that indexing could be carried out in quickly and accurately. There was also the flexibility for indexers to input new terms, when necessary. Training for indexers could be carried out with the use of indexing guideline (both for development and quality checking). Auto-indexing was a little more complex as in required an entity extractor and text had to be mined and analysed. Although auto-indexing is done by the machines, there still has to be human intervention in the form of rules building as well as to provision of sample documents to of the ‘train the automated indexing’.

Having covered the pros and cons of each, the next part of the talk focused on the differences in the terms. Terms indexed by human and machines can be differentiated through their granularity, types of relationships, descriptions/notes and types synonyms/variants. The main difference in the term relationship between human and auto indexing is that in human indexing, there are both hierarchical and associative relationships. In human indexing, there can also be more notes which are visible to the end user and indexer.

Heather also touched on the differences in synonyms/variants between humanly indexed terms and auto-indexed terms. For example, in human indexing, abbreviations are allowed for common terms whereas in auto-indexing, the machine will not be able to understand the abbreviations.

She concluded with a short description of the additional tasks that an indexer would have to do in both human and auto-indexing. Both would require human intervention, its just that the tasks and extent of work is different. For human indexing, terms have to be checked and amended/added in if terms are omitted or misused. In the case of auto-indexing, the work is more focused on the training documents and adjustments of the rules.

This was a very factual and descriptive presentation on both human and automated indexing. It was reiterated that no one method is better than the other and the choice of either one is simply dependent on the usage of the taxonomy. The use of the taxonomy should determine whether human or automated indexing should be done. Both will yield different results in terms of structure and terms created. Both will also require a different level of human intervention in rules building or policy development.

Heather’s website can be found at

Peter Morville on Connecting Knowledge Management and Discovery: Search 3.0

This morning I attended the Taxonomy Bootcamp and KMWorld Joint Keynote by Peter Morville on "Connecting Knowledge Management and Discovery: Search 3.0" . Peter delivered an engaging overview of many aspects that are key to successful Knowledge Management and Discovery. Some of the points that were covered included:

  • Good search and discovery being achieved through collaboration of people with different skills and an appreciation of Information Architecture focusing on business goals as well as user needs
  • For website design it is critically important from the point of view of findability to have multiple paths to information such as alphabetical indexes, search engines, topical schemes and site maps due to users looking for information for different reasons and having different approaches to finding that information
  • Information Architecture and website design is linked to a honeycomb of different qualities. A site needs to be useful, valuable, desirable, usable, findable, accessible and credible. These qualities are all interactive and interdependent
  • The relationship between search and Knowledge Management is very important. Good quality content will be used and found, which encourages maintenance of the quality of this content
  • When developing portals, Information Architects need to think about taxonomies and vocabularies. Content is more dynamic these days and we need to look at work done in both the collaboration and 2.0 space. A critical component of portals is Enterprise search. This needs federated search solutions that bridge the gaps between all repositories, including external websites and databases
  • Any architect (physical or digital) needs to have one foot in the past and one in the future. We need to learned lessons from the past, but at the same time we need to understand that systems will be used into years in the future and will become the legacy systems of the future.
  • One interesting concept that Peter talked about was that of the disciplines of way finding (finding our way in the physical world) and information retrieval are converging. Examples of this are Google World and GPS devices that help to converge mobile devices with location awareness. But just because we can do this, do we really want to?
  • People are becoming findable objects as well as other things. It will probably be about 30 years before the Internet of objects is fully realised via technologies such as RFID. This technology can help in many aspects, the example given was that of Cisco allowing the tagging and locating of high value objects such as wheelchairs left in rooms in hospitals. These technologies will help with costs and customer service

A balance needs to be found with the web 2.0 movement, but we shouldn’t throw away ideas of Information Architecture and vocabulary development. In 10 years time we are still going to be using a search box. This means we will still need taxonomies to provide options for browsing navigation and filtering. Search and browsing will continue to work hand in hand.

The process of search is iterative and interactive and over the course of a search a query can evolve. Search is also one of the most important ways in which we learn. We need to recognise it is a complex adaptive system. It is not just about the interface or the user. We need to know how to get systems to work together, remove outdated content and design interfaces to help users for when they get stuck. Narrow down results etc.

Three key questions when redeveloping a site are:

  • Can users find our website?
  • Can users find their way around our website?
  • Can they find information and their way around the site DESPITE the website?

Design Patterns used in website creation:

  • Best Bets – Opportunity to query disambiguation
  • Federated Search – Searching across multiple database and locations. Users often don’t know which database to search in
  • Faceted Navigation - Bringing search and browsing together and leveraging taxonomies and vocabularies. Need to take a decision on whether to push navigation to users or show it in a more subtle way

Ultimately we need to expand what we think about as search. For example Google Books dramatically expands what we think about the searchable internet. Other examples are the searching of video and podcasts through sites such as Everyzing.

There are lots of possible futures for search. User experience design helps to identify future concepts. Search is a wicked problem. The only way to move forward is by sharing and working together.

Peter has made his slides available from the keynote and blogs over at