NewYork2008:Evaluation in PM

Evaluation and Project MGMT Pain

Why we're here

1. Determining how to know confirm that sinking feeling that the project isn't going well
2. What do do with evaluation results to determine
3. Constructive ways of salvaging
4. To talk about structured evaluations - documentation, etc.

Projects should have time for evaluation built into the budget.

Ongoing evaluation leads to ongoing improvement

Evaluation Strategies

• Build it into the budget
• Educate a project team up front - make them evaluation friendly
  • Team members should be comfortable with having outcomes evaluated
  • Goals that follow SMART - Realistic, Time-Based
• At the beginning determine what will make the project a success
• Treat each phase as a separate project
• At the end of each phase, evaluate to determine if goals have been met
• Have a weekly, regular meeting of the core group
• Close every meeting with a round-the-table reaction and documenting those feelings
  • One participant mentions how his team closes every meeting with a "good though" from each member
• Be wary of one person being able to skew evaluations and projects
• Set up evaluation metrics and structure
• Be aware of different evaluation needs? Is an outside party going to evaluate this as well?
• Layout the evaluation plan (like a grant) with a logic model of Goals, Inputs and Outputs
• Move towards having a structured evaluation along the way
  • Document Check-ins
  • Look at website statistics at each phase

Structures and Methods

• Make sure that you check any changes against the initial set of requirements and agreements

Risks

• Track lists of risks
• Brain-storm risks - participants not allowed to stop until they get silly
• Keep people aware that there can be positive risks and negative risks
• Determine the low-medium-high probability of risks
• Determine strategies for risks (mitigate/tolerable)
• If this is done well, then it can be reported

How to Get Buy-In on Evaluation Time

• 15 minute stand-up meetings
• Off the top evaluation in the budget
• Work with a project-friendly organization :)
• Luck

Client/Vendor Difficulties

• Vendors and Clients should be open and talk about how communication itself is going.
  • Clients want to know if they're difficult to deal with
  • Clients want vendors to tell them what they need and to keep them informed
• Making sure that the client know that their inputs are understood
• Vendors may be fearful of disappointing the client - this needs to be overcome
• Be up front and clear about building a relationship between client and vendor - set the tone and maintain it
• Transparency
• 'Evaluate' the relationship

J's Project

For the five that hadn't heard: J came into a job as the chief writer and editor for the "rebuild of a an 8 or 9 year old website". On the second day they showed screen shots of the new website and it was horrible.

J set up a meeting with a vendor. There was minimal, esoteric documentation. The decisions on features and changes that the vendor had made were signed off on by a passive project manager.

J had an outside tech evaluation of the site and found that the vendor had been fleecing them - hosted solution, custom CMS.

J is now asking them for a itemized list to determine what was obtained for their money.

Vendor sent them an email in October saying Pay us. Then in November said "Pay us or we'll unplug you."

Vendor refuses to provide files/MySQL tables because it's a custom CMS.

Group Advice

• This is a failed contract - J is now having all contracts reviewed by counsel
• Leverage the contract to get your data
• People have to heal and be heard and J is in a good position because she isn't to blame for any of the projects faults