Evaluation Questions
The RWD evaluation is external and objectives-oriented. From the customer's perspective, this is a summative evaluation, assessing the impact of an electronic performance support system on a company in terms of return on investment (ROI) and user processes. For RWD, it is a formative evaluation, helping the company to improve their processes and model. Deb Linder and Linda Hyman are external consultants working together with RWD and one of their clients. As the evaluation plan is refined, attributes and language appropriate to the specific client implementation will be generated. For example, the use of terms such as Cash Disbursements, Invoices, or Personal Expense Reimbursement will vary based on the client and the functions being evaluated. In general, "Level 4" data will be gathered and metrics created that can then be used to calculate return on investment. Evaluation questions are based on the items from Table 4 (Hard and soft data). As stated earlier, the functions of an EPSS are process simplification, performance information, and decision support. Performance measurement is then calculated by determining and comparing time spent to perform tasks, error rates, results or output. The main questions are "What measurable changes are there in output, time, costs, and/or quality as a result of the implementation of the EPSS?" and "What is the cost of these changes over time?"

Table 10.
Metric Unit of Measure Sources Projected Data Collection
1. Time required to complete a transaction Hours per reporting period 1.1 Supervisors, New User, Line workers, Expert users
1.2 Interviews
1.3 Questionnaires
1.4 Observations
1.5 Reporting Software
  • Number of users performing transactions.
  • Number of transactions performed in defined period of time.
  • Average time required to complete a transaction.
  • Average hourly burdened pay rate ($/hr) of group.
2. Time spent answering transaction/job-related questions Hours per reporting period 2.1 Supervisors, New User, Line workers, Expert users, Help Desk
2.2 Interviews
2.3 Questionnaires
2.4 Reporting Software
  • Number answering transaction/job-related questions
  • Number of transactions performed in defined period of time
  • Percent of transactions requiring additional person/help desk action
  • Average time to answer questions per transaction.
  • Average hourly burdened pay rate ($/hr) of supervisory group
3. Time spent training new users, including OJT and class Hours per reporting period 3.1 Supervisor, Expert user, Help Desk, Training Manager
3.2 Interviews
3.3 Questionnaires
  • Number of supervisors performing new employee training
  • Average number of hours per month spent training new employees
  • Average hourly burdened pay rate ($/hr) of supervisory group
TD>
4. Time spent troubleshooting and managing system Hours per reporting period 4.1 Supervisor, Expert user
4.2 Interviews
4.3 Questionnaires
4.4 Reporting Software
  • Number performing system troubleshooting and management
  • Average number of hours per month spent performing system troubleshooting and management
  • Average hourly burdened pay rate ($/hr) of supervisory group
5. Time the help desk spent answering calls from users on system/transaction questions Hours per reporting period 5.1 Help desk supervisor
5.2 Help desk records
5.3 Interviews
5.4 Questionnaires
  • Number of help desk operators answering transaction questions
  • Number of transactions calls answered by the help desk in defined period of time
  • Percent of transactions requiring help desk action
  • Average time per transaction required to answer questions during a defined period of time
  • Average hourly burdened pay rate ($/hr) of help desk operators
6. Time spent referencing/ researching questions (hard-copy documentation Hours per reporting period 6.1 Supervisor, Expert user, Line worker
6.2 Interviews
6.3 Questionnaires
6.4 Observations of users
  • Number of users performing transactions in defined period of time
  • Percent of transactions requiring referencing of documentation
  • Average time per transaction to answer questions using hard copy documentation
  • Average hourly burdened pay rate ($/hr) of user group
7. Time spent correcting transaction input errors Hours per reporting period 7.1 Supervisor, Expert user, Line worker
7.2 Interviews
7.3 Questionnaires
7.4 Reporting Software
  • Number of users performing transactions
  • Percent of transactions requiring correction in defined period of time
  • Average time required to complete a correction
  • Average hourly burdened pay rate ($/hr) of group
8. Time spent training users in classroom Hours per reporting period 8.1 Training manager, Human Resources manager
8.2 Interviews
8.3 Questionnaires
  • Number of hours per month spent training users
  • Average burdened pay rate for instructional staff
9. Productivity impact (non- transaction) based on overall error rate (optional Hours per reporting period 9.1 Operations director, Finance
9.2 Extant data Productivity data)
9.3 Interviews
  • Productivity rate
  • Impact of company, plant, facility or operation productivity (this should not include hours included in other metrics)
10. Time required to recruit and hire new employees based on current turnover rate Hours per reporting period 10.1 Human Resources manager, Supervisor
10.2 Interviews
10.3 Questionnaires
  • Turnover rate
  • Number of employees recruited
  • Time spent to recruit (total, average)
11. Time required by new employees to reach level of competency to perform transactions Hours per reporting period 11.1 Human Resources manager, Supervisor
11.2 Interviews
11.3 Questionnaires
  • Number of new employees
  • Average time to reach competency
  • Average burdened pay rate of group
12. Time spent maintaining performance support system (including documentation, training materials, technical) Hours per reporting period 12.1 IT manager interviews
12.2 Training manager interviews
  • Number of IT staff performing performance system support
  • Average number of hours per reporting period spent performing system maintenance, upgrades
  • Average burdened pay rate of group

 

Management Plan

ID Task_Name Duration Start_Date Finish_Date Predecessors Resource_Names
1 SDSU - RWD EPSS ROI Evaluation Project 274 days 9/8/98 6/8/99    
2 Phase 1: Fall '98 100 days 9/8/98 12/16/98    
3 Project Plan 100 days 9/8/98 12/16/98    
4 Client Selection 8 days 9/8/98 9/17/98   RWD
5 Definition of Project Purpose 1 day 9/16/98 9/16/98 4 RWD
6 Definition of Objectives 1 day 9/16/98 9/16/98 5SS RWD
7 Initial Draft Project Plan 8 days 9/17/98 9/28/98 6 RWD
8 Project Schedule (Mgmt Plan) 2 days 9/29/98 9/30/98 7 Deb Linder
9 Budget 1 day 10/1/98 10/1/98 8 Deb Linder
10 Evaluation Plan: Initial Submission 3 days 9/25/98 9/29/98   Linda Hyman[67%],Deb Linder
11 RWD-SDSU Initial Alignment Meeting 1 day 10/16/98 10/16/98 7 RWD,Linda Hyman,Deb Linder
12 Project Communication 83 days 9/25/98 12/16/98    
13 Regular meeting schedule 71 days 9/25/98 12/4/98    
14 Regular meeting schedule 1 1 day 9/25/98 9/25/98   RWD[13%],Linda Hyman[13%],Deb Linder[13%]
15 Regular meeting schedule 2 1 day 10/9/98 10/9/98   RWD[13%],Linda Hyman[13%],Deb Linder[13%]
16 Regular meeting schedule 3 1 day 10/23/98 10/23/98   RWD[13%],Linda Hyman[13%],Deb Linder[13%]
17 Regular meeting schedule 4 1 day 11/6/98 11/6/98   RWD[13%],Linda Hyman[13%],Deb Linder[13%]
18 Regular meeting schedule 5 1 day 11/20/98 11/20/98   RWD[13%],Linda Hyman[13%],Deb Linder[13%]
19 Regular meeting schedule 6 1 day 12/4/98 12/4/98   RWD[13%],Linda Hyman[13%],Deb Linder[13%]
20 Web Site - SDSU 80 days 9/28/98 12/16/98 14  
21 Initial Setup 1 day 9/28/98 9/28/98   Linda Hyman
22 Update w/ Lit Review 1 day 10/22/98 10/22/98 31 Linda Hyman[12%]
23 Update w/Eval Plan-Draft 2 1 day 12/2/98 12/2/98 40 Linda Hyman[13%]
24 Update w/Question/Procedure Matrix 1 day 12/2/98 12/2/98 39 Linda Hyman[13%]
25 Update w/Analysis/Management Plan 1 day 12/9/98 12/9/98 42 Linda Hyman[13%]
26 Update w/Evaluation Plan-Final 1 day 12/16/98 12/16/98 45 Linda Hyman[13%]
27 Web Site - RWD 1 day 9/30/98 9/30/98 22 RWD
28 Draft 1: Evaluation Plan - Part 1 10 days 9/30/98 10/13/98 10 Linda Hyman[15%],Deb Linder[10%]
29 Literature Review 49 days 10/14/98 12/1/98 28  
30 Identify Search Strategy 1 day 10/14/98 10/14/98   Linda Hyman[50%],Deb Linder[50%]
31 Gather Source Material 5 days 10/15/98 10/21/98 30 Linda Hyman[50%],Deb Linder[50%]
32 Write Draft Lit Review 5 days 10/22/98 11/6/98 31 Linda Hyman,Deb Linder
33 Edit Final Lit Review 2 days 11/30/98 12/1/98 40FF Deb Linder[50%],Linda Hyman
34 Identification of Measurement Tools 2 days 11/9/98 11/10/98 32  
35 Determine form of originating data 1 day 11/9/98 11/9/98   Linda Hyman[50%],Deb Linder[50%]
36 Determine data collection methods 1 day 11/10/98 11/10/98 35 Linda Hyman,Deb Linder
37 Question/Procedure Matrix 21 days 11/11/98 12/1/98 34  
38 Write Draft Question/Procedure Matrix 8 days 11/11/98 11/20/98   Linda Hyman[19%],Deb Linder[19%]
39 Edit Final Question/Procedure Matrix 2 days 11/30/98 12/1/98 38,40FF Deb Linder[50%]
40 Draft 2: Evaluation Plan 7 days 11/23/98 12/1/98 38 Linda Hyman[29%],Deb Linder[57%]
41 Metrics Collection and Analysis Plan 14 days 12/2/98 12/15/98 40  
42 Draft Metrics Collection and Analysis Plan 5 days 12/2/98 12/8/98   Linda Hyman[20%],Deb Linder[20%]
43 Final Metrics Collection and Analysis Plan 2 days 12/14/98 12/15/98 45FF Linda Hyman[13%],Deb Linder[13%]
44 Project Plan - Full (Fall '98 - Spring '99) 5 days 9/8/98 9/14/98   RWD
45 Evaluation Plan - Final 5 days 12/9/98 12/15/98 42 Linda Hyman[10%],Deb Linder[10%]
46 Phase 2: Winter and Spring '99 145 days 1/15/99 6/8/99    
47 Client Alignment 15 days 1/15/99 1/29/99    
48 Client Alignment Meeting 1 day 1/15/99 1/15/99 45 RWD,Linda Hyman,Deb Linder
49 Finalize Data Gathering Strategy with Client 10 days 1/18/99 1/29/99 48 Linda Hyman[25%],Deb Linder[25%]
50 Analysis 82 days 2/1/99 4/23/99    
51 Data Gathering 30 days 2/1/99 3/12/99 49 Linda Hyman[17%],Deb Linder[17%]
52 Comprehensive Exams (Project Break) 10 days 3/1/99 3/10/99    
53 Analysis 15 days 3/11/99 3/31/99 52 Linda Hyman[33%],Deb Linder[33%]
54 Draft Report on Findings 10 days 4/1/99 4/14/99 53 Linda Hyman[25%],Deb Linder[25%]
55 Internal Review: RWD-SDSU 7 days 4/15/99 4/23/99 54 Linda Hyman[29%],Deb Linder[29%],RWD
56 Client Report 9 days 4/26/99 5/4/99    
57 Preview Report to Client 1 day 4/26/99 4/26/99 55 Linda Hyman,Deb Linder
58 Final Report 5 days 4/27/99 5/3/99 57 Linda Hyman[25%],Deb Linder[25%]
59 Present Report to Client 1 day 5/4/99 5/4/99 58 Linda Hyman,Deb Linder
60 Internal Review: Strategies for diverging clients 14 days 5/5/99 5/18/99    
61 Analysis and Model Design 3 days 5/5/99 5/7/99 59 Linda Hyman[50%],Deb Linder[50%]
62 Draft of Diverging Clients Strategy Report 5 days 5/10/99 5/14/99 61 Linda Hyman[50%],Deb Linder[50%]
63 Final Diverging Clients Strategy Report 2 days 5/17/99 5/18/99 62 Linda Hyman[50%],Deb Linder[50%]
64 White Paper Development 20 days 5/19/99 6/7/99    
65 Draft purpose, ojective, intended audience for White Paper 1 day 5/19/99 5/19/99 63 Linda Hyman,Deb Linder
66 Draft Stucture of White Paper 3 days 5/20/99 5/24/99 65 Linda Hyman,Deb Linder
67 Draft White Paper 5 days 5/25/99 5/31/99 66 Linda Hyman[50%],Deb Linder[50%]
68 White Paper Final 5 days 6/1/99 6/7/99 67 Linda Hyman[50%],Deb Linder[50%]
69 Presentations 1 day 6/8/99 6/8/99 64  
70 SDSU 1 day 6/8/99 6/8/99   Linda Hyman,Deb Linder
71 RWD 1 day 6/8/99 6/8/99 68 Linda Hyman,Deb Linder

 

Budget

Item Sep Oct Nov Dec Jan Feb Mar Apr May June Totals
Salary 5,833 5,833 5,833 5,833 5,833 5,833 5,833 5,833 5,833 5,833 58,330
Benefits 1,167 1,167 1,167 1,167 1,167 1,167 1,167 1,167 1,167 1,167 11,666
Airfare   800       800     800   2,400
Lodging           100         100
Meals   50       100     50   200
Ground Transportation, Parking, Tips   55       110     55   220
Office Supplies 10 10 10 10 10 10 10 10 10 10 100
Postage, Fedex, Shipping, Freight 3 3 3 3 3 3 3 3 3 3 30
Copying, Printing 3 3 3 3 3 3 3 3 3 3 30
Software Expense 400                   400
Utilities 70 70 70 70 70 70 70 70 70 70 700
TOTALS 7,486 7,991 7,086 7,086 7,086 8,196 7,086 7,086 7,991 7,086 74,176

[ Table of Contents ]

__________________________________________
This project is the work of former San Diego State University edtec students,
Deb Linder and Linda Woods Hyman.
Last revised April 23, 1999.