RWD Technologies, Inc. is a "thinking" company. According to Webster's Dictionary (7th Collegiate Edition Dictionary) (1977), to think is "to conceive, to surmise, to believe, to reason, to deliberate and to recollect." This definition of "thinking" underlies the ADDIE process (analyze, design, develop, implement, evaluate) of instructional development, and epitomizes RWD's philosophy. Both of the founders of RWD and many of the company's top-level employees have extensive engineering and educational technology backgrounds. They bring to the table a respect for systems thinking and the logical processes that flow from their respective disciplines. In order for any company to survive today, and especially a company involved in technology, there must be a constant process of systematic and systemic data gathering and evaluation. The current study is a piece of this process.

RWD has provided the evaluation team with five distinct challenges:

1. Conduct an extensive review of the literature on how to best evaluate electronic performance support systems.
2. Conduct an extensive review of the literature on how to best evIdentify metrics that align with evaluation guidelines promoted by Kirkpatrick (1996) to evaluate the impact of electronic performance support on an enterprise system.
3. Conduct an extensive review of the literature on how to best evBuild tools to streamline and conduct "Level 4" evaluations of electronic performance support systems (e.g., procedural networks to track performance; interactive modeling software such as STELLA, or a comprehensive database to create a simulation for predictive purposes).
4. Conduct a "Level 4" evaluation with an identified RWD client (time permitting and client willing).
5. Publish a white paper on methodologies, tools, and generalizable results.

There are numerous "flavors" within the realm of performance support systems (strategic decision support systems and data mining; online help reference programs; paper-based or electronic-based materials supporting manufacturing processes; enterprise knowledge management systems; etc.). In order to narrow the study's focus and to help ensure the study's success, the evaluation team will be focusing particularly on RWD's service areas, Information Technology (http://www.rwd.com/infotech/infotech.html). Ideally, this study may coincide with the implementation of an EPSS for a new client, which will allow the evaluation team to gather both pre- and post-assessment data.

This study is summative when looking purely at the impact of EPSS on the RWD client that uses it. However, an ancillary purpose of this study is to develop and then promote the use of generalizable metrics to gauge EPSS impact. This aspect of the study is formative in nature because it will be used by RWD to assist in design decisions for future development. We will measure the return on investment (ROI) as it pertains to changes in human performance, costs of implementation, and net return on the money spent to improve operations. Measurements may include the time saved performing work, dollars saved, materials saved, customer satisfaction, or other measures which are yet to be determined.

George Bradford leads RWD's West Coast Electronic Services Support for the Enterprise Resource Planning (ERP) Division and he is sponsoring this study. He also serves as the team's contact person. Vince Killian, Staff Analyst, and the rest of the Enterprise Resource Planning Support (ERPS) Division are our primary stakeholders. The ERPS Division's major function is to provide documentation, training, and support to its client base. The stakeholders in this group number approximately 250, and the group has already begun to use a special website set up by the evaluation team to expedite communication and information-sharing between interested parties as this study evolves (http://edweb.sdsu.edu/Courses/Ed791A/RWD). The website contains access to:

• Meeting minutes
• Evaluation plan
• Schedule
• Information Resources
• Data Collection
• Cutting-edge strategies

Robert Danna, West Coast Executive Director of RWD, is interested in the study and has been very helpful in defining its parameters. He sees as one of its outcomes the development of a marketable tool to streamline and conduct a "Level 4" evaluation of EPSSs. He predicts that in time other divisions may become interested in determining benchmarks and metrics that can be used to dynamically assess and compare individual and group performance. RWD clients, as well as other businesses that use or are considering EPSSs may also have an interest in the study's results and/or products because the potential exists to generate individualized information for each client regarding dollars saved. In addition, the possibilities for estimating expected savings versus actual savings is very real. All study participants recognize that savings may be direct (output, time, quality, and costs) as well as indirect (absenteeism, turnover rate, and quality).

The biggest issue that the evaluators face in conducting this evaluation is the lack of a prior body of knowledge and experience that we can tap into and emulate. The evaluators are doing something that, in many ways, has not been done before. This type of evaluation is as new as the system it intends to measure. Access to experience and well-proven strategies to conduct such highly complex evaluations exist on a limited basis. The evaluation may be at the "bleeding edge" and require merging existing types of evaluations and measurement techniques found during the review of literature into a hybrid solution. We will be feeling our way rather than following a clearly lit path.

In addition to the physical constraints of getting access to RWD online products and support materials at the client level, the evaluators are also grappling with which data to gather and how to rapidly conduct multiple levels of evaluation. Time is the enemy. Data collection would be difficult even under the best circumstances. Many businesses do not use accounting practices that generate the proper information needed to conduct the evaluation. In addition, internal data may not be accessible.

RWD personnel have been extremely supportive; but the size and compartmentalization of the company has inhibited our ability to access certain data such as information needed to create descriptors about clients, job tasks, and work environments in general. RWD has not yet determined the exact client that will be the object of the evaluation, which has further impeded the ability to scope in and generate specific evaluation questions.

These factors may ultimately compromise the design. The evaluation may have to focus on one single aspect of the EPSS and follow one specific unit of measurement (output, time, quality, costs, absenteeism, turnover rate, etc.) through the various 'levels' of an evaluation. Ideally, this process will continue until a body of data exists which can be placed into a database. RWD and their clients could make predictions of expected returns on investment before an EPSS is implemented and then compare it with real outcomes. In a perfect world, management and individuals could assess individual or group performance at any given time by using tools embedded into the EPSS. The evaluators' part is to lay the initial groundwork by conducting an evaluation to determine which variables are impacted by the EPSS and how data might be gathered in the future for further ongoing evaluation.