Measuring Success

There are two important reasons for evaluating the success of citizen and consumer engagement:

  1. To assess what has been achieved
  2. To improve future practice

 

Effective evaluation is not something that can be tagged onto the end of an engagement process. It needs to be considered from the start, begin as early as possible and continue throughout the process. A number of techniques can be used to measure success.  For example:-

  • Attitudinal, behavioural and demographic data (managers and users), to see the different types people who were involved.
  • Process observation, to see how people participated and interacted.
  • Content analysis, to see the outputs of people’s participation.
  • Site analytics (e.g. Google Analytics, Counters, Referrers), to see how many people participated, where did they come from, and how long did they stay for.
  • Pre- and post-activity surveys or interviews, to see peoples’ experiences of participation and the affect it had on them.
  • Search Engine Ranking / Search volumes, to see how easily people can find out about the participation opportunities.

 

The following facets can be measured to determine the level of success:-

  • Extent and manner of use (effectiveness).
  • Range of users (representativeness).
  • User and stakeholder satisfaction (quality, what changed?).
  • Input costs relative to outputs.
  • Level of stakeholder support (barriers to continuity).
  • User and stakeholder perception about design (process).
  • Repeat visits and ‘up-stepping’ of citizens in the engagement process.
  • Who was/wasn’t involved (public/stakeholder groups) and why/why not.
  • Overspill in terms of increased participation on other channels.

 

and the following democratic criteria:-

  • Representation – who did and did not participate?
  • Political equality – were any groups excluded from participating?
  • Engagement – what was the quality and quantity of participants’ involvement?
  • Exposure – to what degree was the process publicised?
  • Transparency – how open was the process?
  • Conflict and concensus – did the process cause participants opinions to diverge or converge?
  • Community Control – did participants have or take ownership of the process?

 

The CLEAR model is useful to determine when participation is most effective (and subsequently in the design phase) by ensuring citizens:-

  • C an do – have the resources and knowledge to participate.
  • L ike to – have a sense of attachment that reinforces participation.
  • E nabled to – are provided with the opportunity for participation.
  • A sked to – are mobilised through public agencies and civic channels.
  • R esponded to – see evidence that their views have been considered.

Participation projects can also be analysed using GAP analysis .  This should include the application gap (missing technologies and resulting exclusivity) , usgae gap (difference between current take-up and market potential) and gap vision (difference in terms of achievements versus your original inspiration).  Use PEST  analysis for impact determination.