Is the Platform Part of the Measurement? A Protocol and Simulation Study for Cross-Platform Equivalence of Machine-Labelled Policy Sentiment

Key points

• These comparisons rest on an assumption that the classifier measures the same construct, on the same scale, on every platform.1
• Some studies check a classifier's validity platform by platform, but to our knowledge none has modelled the platform as a facet of the classifier's measurement.1
• We adapt the psychometric equivalence toolkit (generalizability theory, measurement invariance and differential item functioning, and score linking) to machine-labelled policy sentiment, treating the platform as a survey mode, and specify a six-step protocol with anchor events fixed in advance, an anchor-density audit, and a two-classifier convergence control.1
• A case study of the instrument from the first author's MSc dissertation, a fine-tuned BERT classifier over 255,446 UK economic-policy comments from Twitter/X, Reddit, YouTube, and Quora, motivates the design and illustrates the instrument-dependence the protocol addresses.1

Sources (1)

• [1]Is the Platform Part of the Measurement? A Protocol and Simulation Study for Cross-Platform Equivalence of Machine-Labelled Policy Sentiment
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 01:22 PM

  Comparing machine-labelled sentiment across social media platforms is now routine: differences in classifier scores between platforms are read as differences in public opinion.

  These comparisons rest on an assumption that the classifier measures the same construct, on the same scale, on every platform.

Extractive summary: sentences quoted from the sources.