New #surveyMethods paper replicates and extends differences in #dataQuality, attention, naivety, decision style, etc. by
- online #research recruitment platform (#mTurk, #Prolific, #Qualtrics, #Pollfish)
- device (#mobile v. #desktop)
- person's incentive

https://doi.org/10.3758/s13428-025-02618-1

In press: Reflection-#Philosophy Order Effects & Correlations Across Samples

Thought experiments caused reflective decisions?
Reflection tests didn't change philosophical decisions!
18X more junk data from #mTurk (vs other sources)!

#Preprint + audio: https://byrdnick.com/archives/28438/upon-reflection-ep-13-reflection-philosophy-order-effects-and-correlations-across-samples

Excited to share YEARS of research about how to get people to think reflectively and how reflection impacts philosophical judgments at the 2025 #APA in #NewYorkCity (January 8 to 11): https://www.apaonline.org/mpage/2025eastern

Can't make it?
- More about my talk: https://researchgate.net/publication/370132037
- More about my poster: https://researchgate.net/publication/371248872

Thanks to the #APA, James Beebe, and the Experimental Philosophy Society for the opportunity!

Are moral decisions impacted by economic environment?

Perhaps! “…in low-income nations, tournament-based compensation increased deontological commitments [but] in higher-income nations, the effect on deontological commitments reversed …consistent with the historical development of the doux commerce thesis.”

A fun #economics paper that reminds me of the early days of #xPhi: https://doi.org/10.1093/jleo/ewae016

A question for the #HCI folks (please repost for more reach):

Since the #LLM cat is definitely out of the bag now, do you feel that #Prolific, #MTurk, etc. are still a reliable source of survey participants and consequently, survey data?

What's the difference between microtask platforms like #mTurk or Prolific and market-research panels?

CloudResearch recently published an #openAccess paper that explains. (The paper also includes a step-by-step tutorial for using their "Prime Panels" in conjunction Qualtrics). 

https://doi.org/10.1177/25152459221140388

For the #polisci & public opinion crew:

In psychology there are papers that tally what types of samples are most often used in the field (E.g., XX% college students, YY% Mturk etc). Are there any examples of this from your own fields?

Essentially, I'm trying to put some approximate numbers to the different norms in types of sample that I see

@politicalscience @socialpsych #research #sample #Mturk

ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks

with Meysam Alizadeh & Maël Kubli

We find that zero-shot #ChatGPT:
> has better accuracy than #MTurk
> has better intercoder agreement than MTurk and trained coders
> is 20x cheaper than MTurk

https://arxiv.org/abs/2303.15056

RT @turkopticon

It's time to recognize the challenges we face, take into account the autonomy & well-being of those working on #MTurk & create solutions that ensure the fairest treatment of Turkers worldwide.

Next Wednesday @ #Mozfest
:undefined:https://mozillafestival.org/en/tickets
:undefined:https://schedule.mozillafestival.org/session/U7L8W8

RT @turkopticon

Are you curious about the challenges of virtual global organizing in the #GigEconomy? Our session at #Mozfest on "Uniting the Amazon #MTurk Workers Behind AI" is in less than 3 weeks!

Don't miss it!
https://mozillafestival.org/en/tickets
https://schedule.mozillafestival.org/session/U7L8W8

What conclusion do you draw, if any, when all (and only) the people who wrote in "non-binary" to the question, "What is your gender?" failed a survey's attention check?

Why would someone who puts in the effort to write a (vs. select a pre-written) response have been inattentive earlier in the survey?

I doubt that people reporting non-binary gender are generally less attentive on surveys.

So was it most likely a fluke?

mTurk HITs completed

Can you get them automatically added to a dataset when running a survey?

Sort of like you can automatically add Worker ID, Assignment ID, and HIT ID to the dataset with certain URL parameters.

I’m seeing that “Too good to be true” mturk paper uncritically shared on mastodon. That paper was not peer reviewed and the editor who approved it was asked to resign for similar editorial practices (accepting un-reviewed, one-sided commentaries)

Here is another paper pointing out all the deficiencies of that paper that would have been caught in peer review: https://psyarxiv.com/w7qy9

Study of #mTurk workers concludes that #dataQuality and #replicability are no longer ensured by mTurk's worker quality metrics (HITs completed and HIT acceptance rate) because "researchers do not reject HITs containing poor-quality data" (N = 900).

Some researchers block workers who produce poor-quality data, but unless we share/aggregate our block lists, we allow known-bad work to infect our colleagues' research. It's a #collectiveActionProblem!

https://doi.org/10.3758/s13428-022-01999-x