Please Don't Shout. ML vs Operators: 1-0

Eduard Khusnutdinov27 May 2023

Problem

CFT runs speech analytics for its call center: a model transcribes the operator's audio track, then a checking block verifies that the operator says what is required and avoids what is forbidden. The department that monitors operators asked for a new check: find operators who shout, because shouting counts as emotional pressure on the customer. Recordings run from a few seconds to about 10 minutes, and labeled shouts were almost absent.

Example from the article: the monitoring department's reports held only about 5 shouting recordings, and staged shouting does not match real anger.

Solution in brief

Tap the first transformer layer of the speech-to-text model already in production and train a small fully connected "shout / no shout" classifier on its outputs. Mine training candidates with a public 7-emotion audio model and let the monitoring department label them. Tune the threshold to shrink the pile of recordings to review, not to catch every shout.

Questions this case answers
  • How to detect a rare acoustic event in long audio recordings when almost no labeled examples exist and a production speech model must be reused
  • How to choose which layer of a pretrained speech model to use as features for a new classifier without running a second large model
  • How to detect shouting operators in call recordings when almost no labeled examples exist
  • How to add a new audio classifier without deploying another large model

Tools and methods

  • wav2vec-style speech model (already in production) — frozen feature extractor, no new large model to deploy
  • Public 7-emotion audio classification model — mines shout candidates from conversation history
  • t-SNE — visual comparison of class separation across layers
  • Small multilayer perceptron — binary classifier on first-layer outputs
  • Probability threshold — controls the balance between missed shouts and false positives
Continue reading — it's free

Google or a one-time email code. No password.