1. Conduct an extensive review of the literature on how to best evaluate electronic performance support systems.
2. Conduct an extensive review of the literature on how to best evIdentify metrics that align with evaluation guidelines promoted by Kirkpatrick (1996) to evaluate the impact of electronic performance support on an enterprise system.
3. Conduct an extensive review of the literature on how to best evBuild tools to streamline and conduct "Level 4" evaluations of electronic performance support systems (e.g., procedural networks to track performance; interactive modeling software such as STELLA, or a comprehensive database to create a simulation for predictive purposes).
4. Conduct a "Level 4" evaluation with an identified RWD client (time permitting and client willing).
5. Publish a white paper on methodologies, tools, and generalizable results.

The literature review focused on levels of evaluation, calculating the return on investment, and electronic performance support systems. The team hoped to learn about any existing standards against which to measure return on investment and how such standards may be translated for use with EPSSs.

One thread consistently ran through all of the literature; in order for companies to be successful, they must provide employees with contextualized training and support such as that found in an EPSS. The emerging pattern today is one in which training and support are evaluated in terms of performance outcomes and strategic alignment with business goals. In many businesses, excellence is "a moving target" versus a stable, consistent, easily identified end-point. This makes it especially difficult to establish metrics by which to evaluate interventions. There is no one-size-fits-all cookie cutter approach that can be used to calculate overall return on investment (ROI).

Another emerging trend is the link between knowledge, known commonly as "human capital," and outcomes. Human capital can lead to increased productivity (both individual and organizational). These intellectual assets may be worth three to five times the value of an organization's physical assets (Stolovitch and Maurice, 1998) and point to the importance and tangible value of effective training and support. Once again, concerted efforts to identify and track human capital—let alone relate it to ROI—are the exception rather than the rule and there are no set standards that one can follow. Knowledge has always meant power yet executives and financial experts have been reluctant or unwilling to acknowledge the importance of the return on investment from training. Why is this? Perhaps there are several reasons why calculating the return on investment of training and support has been overlooked (Stolovitch and Maurice, 1998):

• senior management doesn't require it
• training managers don't know how to do it
• training managers don't know what to measure
• the effort involved may be costly and risky.

A new era of self-examination is beginning to emerge which places increased emphasis on issues such as human capital and the strategic alignment of business goals. Several studies have recently been conducted to determine what, if any, measures are being conducted by businesses and how the data are being used. Stivers, Covin, Hall, and Smalt (1998), for example, published the results of a survey involving 253 U.S. Fortune 500 and Canadian Post 300 companies. The survey was part of a study designed to identify which non-financial factors firms perceive as important, whether firms are measuring these factors, and whether or not the data influence their planning processes. Results of the study indicate that customer service factors are rated most important. Ninety-two percent of the survey respondents rated customer satisfaction and delivery performance/customer service as highly important. Product/process quality was rated as highly important by 206 (81.4%) of the companies and service quality was rated highly important by 205 or 81%. Market share was rated highly important by 79.1% of the respondents, and "productivity" was rated highly important by 83.4%. Innovation was rated highly important by 44.3% and employee turnover was rated highly important by 48.2%.

According to Stivers, et al., a number of critical non-financial factors must also be included in comprehensive performance measurement including: market standing, innovation, productivity, customer service, and employee involvement. This study highlighted the fact that respondents rated "innovation" and "employee involvement"—key aspects of human capital—as less important than other factors. The authors suggest that as intellectual assets increasingly impact the bottom line, methods of managing and controlling knowledge will become more important. This list of reasons why calculating the return on investment of training and support has been overlooked provided by Stolovitch and Maurice (1998) may be exemplified in this study.

Stivers, et al. (1998) indicates the importance of measuring performance gaps. One gap is called the "importance-measurement." While many companies view non-financial performance factors as highly important, they are not actively capturing the data. For example, from the employee involvement category of the survey, "moral and corporate culture" was rated as highly important by 75.9% of the respondents; however only 37.5% of the companies that responded are collecting data on this factor. Innovation is the factor least likely to be measured (21.8%). Another measurement gap revealed in this study was called the "measurement-use." Although the factors mentioned above might be rated by respondents as highly important and as being measured by companies, the data are often not fully used in strategic planning. For example delivery performance/customer service was rated highly important by 92% of survey respondents and close to 85% reported actually measuring this factor. However, only 71.1% reported using the data for planning. The largest gap is in the "employee involvement" category, which includes items such as employee satisfaction, employee turnover, internal recognition, and morale and corporate culture. About 40% of the survey respondents reported not using such factors in corporate planning although the data are collected and available.

In January 1998, a survey of 150 information systems executives was conducted by InformationWeek magazine. The results were published in the April 27 issue (Violino, 1998). Eighty percent of the respondents claimed that their organizations require demonstrated potential revenue, payback or budget impact from information technology (IT) projects--an increase of 45% from the previous year. The number of businesses requiring ROI measures are growing, especially in businesses involved in enterprise resource planning. Businesses recognize the need to show returns, yet in this survey, only one-fifth of the respondents reported planning to adopt any formal ROI measures within the next 12 months.

A common industry phrase is "what gets measured, gets done." Some of the literature suggests that the financial climate within corporations and industry is changing. The unlimited flow of dollars into new projects of any sort is coming under more careful scrutiny. More than ever before, in order to get things get done, there needs to be justification. According to several of the articles, massive amounts of money are being poured into solving Year 2000 computer problems, and shortages of technology-savvy labor have driven salary costs to exorbitant heights. These factors are part of the reason that more IT budgets are coming under more careful scrutiny. At the same time, businesses are finding a competitive edge in decentralizing their decision-making functions and allowing decisions to be made quickly and at the point where the activity occurs. This change in business tactics supports Drucker's (1992) assertion that "knowledge is only productive when integrated into a task" (p. 96). A visible result of changing business tactics is the development of IT tools that integrate specialized knowledge and are readily accessible to those employees who most need them. One such tool is an electronic performance support system (EPSS).

Levels of Evaluation
Let's return to the reasons why ROI calculations have not been integrated into business practice. Two that stand out are: 1) training managers don't know how to do it and 2) training managers don't know what to measure. Selecting an appropriate ROI method requires some front-end analysis to determine which of the several available methods are most appropriate. Appropriateness is determined by strategic focus and, more importantly, the level at which the strategic focus takes place (Foshay, 1998). Some of the various approaches are discussed below.

The Kirkpatrick Four-Level Approach
Kirkpatrick (1996) first defined four levels of evaluation when writing his dissertation some 40 years ago on evaluating a supervisory training program. For years the framework has been the most popular and primary evaluation approach used by training professionals. Why? It's both simple and practical (Kirkpatrick, 1996). Table 1 below illustrates the concepts.

Table 1. The Kirkpatrick Four-Level Approach (Kirkpatrick, 1996).

Kaufman's Five Levels of Evaluation
Kaufman, Keller & Watkins (1996) promote an assessment strategy called the Organizational Elements Model (OEM) which involves four levels of analysis:

1. audit or cost-cost - based solely on inputs, looks at reductions of cost between old and new resources
2. products - the building blocks of a product or service within an organization that contribute to the overall product or service (e.g., automobile fenders)
3. outputs - products or services that are delivered to external clients
4. outcomes - the value of the outputs (the aggregated products or services) delivered to external clients and their clients and ultimately to society.

Since the introduction of Kaufman's four-level OEM model, many researchers have used it as a viable framework for evaluation. Others, though, have found it restrictive and have attempted to modify and/or add to it. Kaufman, et al. (1996), for example, later added levels of impact that go beyond the traditional four-level, training-focused approach which they felt did not adequately address substantive issues an organization faces. Such modification to the model resulted in the addition of a fifth level, which assesses how the performance improvement program contributes to the good of society in general as well as satisfying the client.

Table 2. Kaufman's Five Levels of Evaluation (Kaufman et al., 1996)

Level	Evaluation	Focus	Suggested Levels*
5	Societal outcomes	Societal and client responsiveness, consequences and payoffs.	Mega
4	Organizational output	Organizational contributions and payoffs.	Macro
3	Application	Individual and small group (products)utilization within the organization.	Micro
2	Acquisition	Individual and small group mastery and competency.	Micro
1b	Reaction	Methods', means', and processes' acceptability and efficiency.	Process
1a	Enabling	Availability and quality of human, financial, and physical resources input.	Input
*Based on Kaufman's Organizational Elements Model (1992, 1995)

The Phillips Five-Level ROI Framework
The evaluation framework favored by Phillips (1997) adds a fifth level to the evaluation model. Instead of examining whether the program enhanced society, as Kaufman et al. propose, he adds return on investment. Return on investment measurement allows an organization to compare the monetary benefits from the program with its costs. Using a cost/benefit ratio, evaluators can determine if the program is cost-effective.

Table 3. The Phillips Five-Level ROI Framework (Phillips, 1997)

Level	Questions
1. Reaction and planned action	What are participants' reactions to the program? What do they plan to do with what they learned?
2. Learning	What skills, knowledge, or attitudes have changed? By how much?
3. Applied learning on the job	Did participants apply what they learned on the job?
4. Business results	Did the on-the-job application produce measurable results?
5. Return on investment	Did the monetary value of the results exceed the cost of the program?

Most evaluation processes check for participant satisfaction (Level 1) and whether or not the knowledge and skills learned are useful (Level 2). But moving up the management ladder puts more importance on the outcomes of Levels 3-5. Immediate managers want their investment in training and development to yield measurable impacts on business performance (Levels 3 and 4). Top executives care only that the training is monetarily justified (Level 5). If measurements are not taken at each level, it is difficult to show that any improvement can be attributed to a program, process, or other intervention. Level 5 evaluation (ROI) is rapidly gaining acceptance by training professionals. It provides a solid framework for reporting the monetary benefits of a program.

Return on Investment
Several different methods for calculating return on investment were uncovered during the literature search. Five areas were recommended by Davidson (1998) in her recent article entitled, "Measure what you bring to the Bottom Line." She suggests measuring:

1. productivity—output per unit of input,
2. processes—systems, workflow,
3. human resource costs and benefits for a specific initiative,
4. employee retention, morale, commitment, and skills,
5. capabilities of the organization—cycle time, learning, shared mind set, accountability.