Salary Horrors and How to Avoid Them

Vladimir Davidko28 Oct 2023

Problem

Job seekers do not know what salary to ask for, and employers do not know what salary will fill a vacancy in reasonable time. Existing salary services behave like kNN: they take similar postings and show a mean or a range. The range is too wide to act on, shows nothing about what drives it, and breaks when a profession has no vacancies. A posting that demands industrial experience and PyTorch for a salary of about 5000 collects no applications.

Example from the article: a small company with a budget of about 200 thousand rubles needs an analyst within a month. An estimator that shows each feature's contribution lets it test options, such as hiring remotely in Voronezh instead of Moscow, or looking for someone who builds dashboards instead of requiring Tableau.

Solution in brief

Train a linear regression on vacancy and resume data: profession, region, work schedule, tags and skills. Use FastText embeddings for professions and skills so the model covers many professions. Choose linear regression despite worse metrics because its coefficients show the user what makes up the estimate.

Questions this case answers
  • How to estimate a fair pay range for an object from free-text attributes when many categories have very few observations, and explain each contribution to the user
  • How to make an interpretable linear model cover many categories by embedding them and clustering similar ones
  • How to estimate a fair salary range for a vacancy or a resume from job-board data
  • How to explain a salary estimate to a user feature by feature

Tools and methods

  • Linear regression — salary estimate whose coefficients explain each feature's contribution
  • FastText embeddings — vector representations of professions and skills
  • Clustering of profession embeddings — groups similar professions to share skill effects
  • Streamlit — demo with feature contributions
Continue reading — it's free

Google or a one-time email code. No password.