Overview

There are hundreds of bioinformatic software programs available that serve a variety of applications. For someone new to bioinformatics, there is a bewildering array of freeware, shareware, and commercial analytical programs, and it is difficult to make a decision between them. We are mainly influenced by what our peer group uses, but we still need a method to assess, or evaluate, the software to see if it fits our particular purpose. Established workers in bioinformatics also need some way to determine how best a particular software program integrates into the existing infrastructure. Software evaluation is an art, a science, and a business. It can be as casual as a "let's have a play" approach or as formal as a rigorous, structured and systematic investigation. A large or networked organization would usually require detailed and specific criteria for evaluating software and also effective project management of the evaluation process. What follows, however, is a general guide for evaluating "off the shelf" bioinformatics analysis software. Both an individual PC-user or a bioinformatics/IT manager, responsible for a suite of programs over a local or distributed network, may equally use these guidelines.

Why Evaluate?

Who Will Evaluate?

The end user is an obviously important evaluator. However, when organizational issues must be addressed, then the information technology, bioinformatics, and other appropriate departments need to be consulted for the evaluation process. A third party can also be consulted if necessary and this may be used to provide unbiased, independent advice, and thus help to avoid internal politics or vested interests.

Software can also be used to evaluate software - although you'll then have the "chicken and the egg" paradox of trying to evaluate the software which is evaluating the software!

How to Evaluate?

Evaluation can be approached from several different perspectives encompassing scientific, business, or procedural issues. Invariably, however, evaluation is based on a set of criteria, and the software is assessed based on its ability to meet that criteria. The following criteria can be used for scientific evaluation and is subdivided into the four general areas of needs analysis, presentation, technical, and performance sections.

Needs Analysis

Probably the most important first step is to decide what you want to do, and from this develop your criteria. What features do you want? What do you want to achieve? What benefits do you want from the software? What feel and aesthetics do you want? How does it fit within your existing infrastructure? How many users will there be and what is their competency? What do the users need to perform their duties? How do your requirements compare with the vendors' advertised solution?

The next step is to search for candidate software for your evaluation and to gather the necessary background information. Where do you find candidate programs to help you solve your particular needs? Well, you can check out software databases or lists (box 1), vendors (box 2), colleagues, journal articles and advertisements, trade shows, conferences, industry associations, user groups, Internet newsgroups, and Web sites such as BioMedNet. Some of the background information for the software will have to be provided by the vendor or manufacturer or programmer, provided that their inherent vested interest is recognized. Additional third party information can come from any literature reviews and testimonials from previous and existing users.

Finally, you need to have some way of scoring the evaluation. For example, a yes/no answer or a quantitative numbering scale for each of the selected criteria. This is to set a standard for evaluation, to enable fair comparison of proposed benefits and features of the program, and to help discriminate between competing products.

Presentation

In terms of presentation, is the software intuitive and user-friendly? Is the interface a command line, Web-based, X Windows, or other graphical user interface (GUI)? How effectively does it use the monitor screen space - is it uncluttered, easily read, and does it make appropriate use of color, graphics, and sound?

Technical

Technical detail can be further subdivided into method, features, and programming.

Method

What algorithm is used and is it appropriate - is it based on old or new knowledge; what are the assumptions inherent in the algorithm, how good is it; is it fast enough; does it do what you expect? Are there any better algorithms? What sort of testing has been done and is this supported in the literature? Is it a "black box" system using proprietary algorithms, or is it an auditable system open to inspection and assessment?

Program Features or Functions

The questions relating to a software's features and functions are numerous, but some general questions (in no particular order) include:

• What platform does it use: Unix, Windows NT, Windows, Macintosh, or other?
• What is the size of software and what is the minimum configuration required in terms of memory (hard disk and RAM) and central processing unit (CPU)? Is it CPU intensive, and does it tend to "hog" the CPU? Is a graphics card needed?
• Is the software networkable, and what kind of interface does it have for distributed systems?
• Does it support multiuser, multitasking, and background operations?
• What is the level of content and expertise within the program - is it for beginners, intermediate, or advanced users?
• Is training or a tutorial available for the software? If so, what form does it take - books, online, or personal? How long does it take to learn and to become reasonably competent?
• Is it robust and reliable - does it have an ability to handle inappropriate calls or directions without crashing? Can the program be applied under varying conditions (adjusting parameters) and continue to give reliable and trustworthy results without crashing?
• Is it powerful - is it quick and can it do what you want it to do?
• Is it scaleable - does it have the ability to handle different sizes of data sets? What is the maximum data size it can handle?
• What file formats are used? Proprietary or nonstandard file formats make it difficult to use elsewhere, and limit flexibility.
• Does it have the ability to export and import different file formats? This affects its ability to share data and information with other programs.
• Does it have the ability to interface with the Internet and download data?
• How does it report results - in PostScript files, text files, or proprietary files? Can you edit the results?
• What is its printing ability - can it handle PostScript, graphics, and specialized text?
• What is the cost (initial and ongoing)? Does the cost cover a single user, multiple users, site license, cost of upgrades, support, and maintenance?
• How old is the technology it uses, and has it been recently surpassed? Does it have a lineage and version history? Has it been recently upgraded?
• Have all the bugs been worked out and does it conflict with any other program?
• How much time and effort is required for installation and backup?
• Once installed, how long does it take to load and begin to work?
• Are there any benchmark performance data for the software - time required to open and read a particular file, draw a graphic, and export or import files? Is the benchmarking performance data independent and can it be validated by a third party?
• Is customization available, and can the user set preferences?
• Are there any security measures available - password protection and prevention of online access with proprietary sequences?
• Is help available and what form does this help take - for example, documentation, online, context sensitive, menus, and balloon help?
• How long has the vendor (if commercial), manufacturer, or programmer been in existence? What experience and qualifications do they have? How likely is it that they will be operating in the future? What support systems are in place - such as a personal representative, email, fax-back, toll-free number, support for a fee, bulletin board, online chat, and newsgroup?

Programming

Further questions relating to programming include:

• Is the source code provided and is it hackable? That is, is it possible to change and modify the program?
• Does it have the facility to accept plug-ins/modules/subroutines?
• Which programming language is used?
• What standards are incorporated into the software?
• For bundled programs (in which two or more programs are packaged together in one application) how good is the integration? Is it easy to move data from one program to the other, and what interim storage is necessary when moving between programs? How are the programs integrated - what language "glues" the components together?

Performance

The crucial test of any bioinformatic "off the shelf" software is if it makes biological sense. For example, a program may be designed to find patterns, but do the patterns have any biological significance? Does the software do the job you thought it could do? Is it fit for the purpose you intended? Does it give you error-free, accurate, and precise results? Can you trust the results - especially if it is a "black box" system? What sort of quality control has been performed, and does it adhere to your own company's quality control standards? Finally, is a test data set available to determine a software's performance, and is it validated by a third party or from the literature?