Is your survey calibration process destroying agent morale? Part 1 of a 3-Part Series on IRR

///Is your survey calibration process destroying agent morale? Part 1 of a 3-Part Series on IRR

Is your survey calibration process destroying agent morale? Part 1 of a 3-Part Series on IRR

Share on LinkedIn0Share on Google+0Tweet about this on TwitterShare on Facebook3Pin on Pinterest0Email this to someone

As the economy has contracted over the two years, many organizations have focused on minimizing costs by reducing (if not eliminating) on-going training, quality initiatives, hiring, promotion, etc.  The impact has been a decline in employee engagement which had a direct and measurable impact on the way they treat your most valuable asset – your customers.

According to the Gallup Q12 employee-engagement survey, the following questions represent the largest drivers of employee engagement (correlating to employee productivity, customer loyalty, bottom-line growth:

  1. Do I know what is expected of me at work?
  2. Do I have the materials and equipment I need to do my work right?
  3. At work, do I have the opportunity to do what I do best every day?
  4. In the last seven days, have I received recognition or praise for doing good work?
  5. Does my supervisor, or someone at work, seem to care about me as a person?
  6. Is there someone at work who encourages my development?
  7. At work, do my opinions seem to count?
  8. Does the mission/purpose of my company make me feel my job is important?
  9. Are my co-workers committed to doing quality work?
  10. Do I have a best friend at work?
  11. In the last six months, has someone at work talked to me about my progress?
  12. This last year, have I had opportunities at work to learn and grow?

Your quality assurance team, and more specifically, your calibration process has direct impacts on the questions which appear in bold above (questions 1, 3, 4, 6, 11, and 12).  How do you think Agent Joe feels when QA Jerry tells him he did great on his last call and QA Ben gives him a mediocre call monitoring score the very next week on a nearly identical call?  This scenario of inconsistency is exactly what kills employee engagement, along with the credibility of your Quality Assurance team.


Click to download the Case Study

I am not suggesting that you eliminate calibration because consistency and accuracy are vital components of any high functioning team.  But in order to (quantifiably) attain high levels of accuracy and consistency within your Quality Assurance team, you leap over the current calibration process to a more rigorous and measurable process.  The risk from inadequate calibration is substantial to the organization and to you as a leader.  Reduce risk by engaging in what we at Customer Relationship Metrics (CRM) call Inter-Rater Reliability (IRR).

In the next post of this series I will explain the difference between calibration and IRR.  But in the meantime, I’ll leave you with some food for thought:

  • A highly tenured QA team migrating conducting their very first Inter-Rater Reliability test yielded a 32.32% failure rate in accuracy of call scoring.  Such a failure rate is hardly unusual.
  • This same team reached agreement (consistency) in call scoring only 56.5% of the time (and this is WITH well-defined scoring instructions).
  • The area in which the least consistency in scoring existed was in skills collectively called customer service or soft skills.  These skills are key in driving a positive customer experience and brand loyalty.
  • This highly tenured team is made up of only four Quality Assurance agents.  Our research has shown that the larger the QA team, the higher the initial failures in accuracy and consistency.

About Jim Rembach

Jim Rembach is a panel expert with the Customer Experience Professionals Association (CXPA) and an SVP for Customer Relationship Metrics (CRM). Jim spent many years in contact center operations and leverages this to help others. He is a certified Emotional Intelligence (EQ) practitioner and frequently quoted industry expert. Call Jim at 336-288-8226 if you need help with customer-centric enhancements.

Visit My Website
View All Posts
By | 2016-12-05T15:15:16+00:00 November 16th, 2010|Inter-Rater Reliability (IRR)|0 Comments