@andrewgordon.bsky.social
Staff Researcher (Behavioural Science) @Prolific Previous dabbler in the cognitive neuroscience of misinformation Once bitten by the worlds rarest goose Was once bitten by the worlds rarest goose
@andrewgordon.bsky.socialThis is a super interesting and unique approach to assessing LLM use in online platforms - highly worth a read! 👏
@andrewgordon.bsky.socialSo many thoughts 😅 There is a tonne of work that goes into the platform side to protect against this @sebastiansauppe.bsky.social (too much for a bsky thread).
The key here though is that there are extremely effective detection methods that exist for this type of agent, they just arent tested here
@andrewgordon.bsky.socialAn agent completes a survey, this is not ground breaking news, yet its led to another set of 'online testing is dead' posts. Remember capability ≠ deployment. Just because it can be done doesn't mean it is being done Also, mouse tracking catches it easily. Design studies with this in mind.
@andrewgordon.bsky.socialThis is so important - we've been warning against relying on the types of checks we would use to catch bad participants as a way to catch agents for a long time. Behavioural signatures are critical and researchers need to get used to integrating these types of measures into their studies
@andrewgordon.bsky.socialSo are AI agents infiltrating surveys? Not really. Meaningful detections were almost exclusively on MTurk (11-16%). Every other platform: at or below 1% for our primary detection flag [5/9]
@andrewgordon.bsky.socialNew preprint out today (osf.io/preprints/ps...). We tested whether AI agents are actually infiltrating online surveys.
Spoiler alert: they aren't
Thread 🧵
[1/9]