@dansup new way of looking at the #CollectiveActionProblem - what a great problem to have!
#collectiveactionproblem
0 posts0 participants0 posts today
Continued thread
Long thread/8
Continued thread
2/3 But of course I know why they are not doing it, each tangled up in all sorts of webs of obligations, intererests, etc. This is why I am, right now, even more incredulous and angry how little #PoliticalScience is doing. The #ClimateCrisis is above all a massive political challenge - so step up! There is SO much political science can do. Like, it's the mother of all #CollectiveActionProblem s. Help solve it! Representative Democracy, Party Politics, Lobbying: these
Replied in thread
Long thread/5
Continued thread
Long thread/11
Replied in thread
Long thread/11
Replied in thread
Long thread/6
Study of #mTurk workers concludes that #dataQuality and #replicability are no longer ensured by mTurk's worker quality metrics (HITs completed and HIT acceptance rate) because "researchers do not reject HITs containing poor-quality data" (N = 900).
Some researchers block workers who produce poor-quality data, but unless we share/aggregate our block lists, we allow known-bad work to infect our colleagues' research. It's a #collectiveActionProblem!
SpringerLinkEvaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk - Behavior Research MethodsMaintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concerns, CloudResearch, a third-party website that interfaces with MTurk, has assessed ~165,000 MTurkers and categorized them into those that provide high- (~100,000, Approved) and low- (~65,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch’s vetting. In a pre-registered study, participants (N = 900) from the Approved and Blocked groups, along with a Standard MTurk sample (95% HIT acceptance ratio, 100+ completed HITs), completed an array of data-quality measures. Across several indices, Approved participants (i) identified the content of images more accurately, (ii) answered more reading comprehension questions correctly, (iii) responded to reversed coded items more consistently, (iv) passed a greater number of attention checks, (v) self-reported less cheating and actually left the survey window less often on easily Googleable questions, (vi) replicated classic psychology experimental effects more reliably, and (vii) answered AI-stumping questions more accurately than Blocked participants, who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss how MTurk’s Approval Rating system is no longer an effective data-quality control, and we discuss the advantages afforded by using the Approved group for scientific studies on MTurk.