explore | Grilled Cheese

@emmharv.bsky.social Oct 14, 2026, 7:47 AM

I will be presenting "Data Annotation as Measurement" at #AIES2026 in Poster Session 3 (today @ 3:15PM CEST)!

ojs.aaai.org/index.php/AI...

A poster with the following content:

Data Annotation as Measurement
Emma Harvey, Allison Koenecke, Rene Kizilcec

Data annotation...
- is the augmentation of data for use in some downstream task
- is a process that is under-documented and under-interrogated
- is centered on maximizing agreement between multiple annotators
- often produces low-quality data

Through an interview study (N=10) and a literature review (N=132), we explore: How can we improve data annotation?

Determine the sources of annotation issues before addressing them:
[image of a dog mislabeled as a cat]
What’s wrong with this label? Maybe…
(A) The annotator:
- is not following instructions (error)
- is misrepresenting themselves (identity)
(B) The instructions:
- lack factual information (ambiguity)
- lack value information (subjectivity)
(C) The data instance:
- lacks necessary information (impossibility)

Use ideas from measurement theory to interrogate annotation reliability and validity:
- Move beyond agreement to ask: are repeated annotations by the same annotator consistent? (test-retest reliability)
- Clearly systematize and operationalize concepts that are being annotated so that it is possible to ask: are annotations (un)correlated with other measures of (un)related concepts? (convergent & discriminant validity)
- Formalize the checks that researchers already conduct before working with annotated data to ask: do annotations pass the “sniff test” at scale? (face validity)
- Use qualitative methods from the social sciences to ask: what are the societal impacts of using a set of annotations? (consequential validity)

@nnkennard.bsky.social Oct 14, 2026, 7:04 AM

Irina Shklovski’s thought provoking keynote talked about whether common computing practices and tacit shared values might fundamentally conflict with the demands of Responsible AI 😲#aies2026

@pranav-nlp.bsky.social Oct 13, 2026, 7:57 AM

Today in #AIES2026, I will talk about my journey of my PhD. What I did and what I plan to do. I highly value society's and broader community input for my work, so if you wanna chat about surveillance, come by at poster S-100!

Title slide with a teal-to-blue gradient background reading “Resisting Surveillance AI: Mechanisms of Accountability Evasion and Community Engagement”, presented by A Pranav (S-100), with Trustworthy AI Hamburg and University of Hamburg logos. A rounded banner poses the question “How can technology and society hold each other accountable in the policies they jointly shape?” Below it, a bar running from yellow to magenta is labeled “How technology informs society?” on the left and “How society informs technology?” on the right. Three boxes branch from it: Technical audits, Governance audits and Community audits.

@davidmanheim.alter.org.il Oct 13, 2026, 7:22 AM

Fantastic story from Stefan Larsson @lund-university.bsky.social giving the keynote today at #AIES2026

He thought he was done working on copyright law, and had moved on to AI issues - then found out that *his book on copyright* had been pirated to train LLMs via the Anthropic class-action lawsuit.

@emmharv.bsky.social Oct 12, 2026, 2:46 PM

I will be presenting "Context and Symmetry in Auditing: A Case Study of Skeleton Inference in Motion Capture" at #AIES2026 in Poster Session 1 (today @ 5PM CEST)!

ojs.aaai.org/index.php/AI...

A poster with the following content:

Context and Symmetry in Auditing
Emma Harvey, Emanuel Moss, Hauke Sandhaus, Abigail Jacobs, Mona Sloane

AI auditing compares how a system should behave (nominal outputs) to how it does behave (actual outputs).
- What inputs are relevant for identifying patterns in a system’s actual outputs?
- What should a system’s nominal outputs look like?
- Against which “ground truth” should accuracy be measured?

We introduce the contextual audit, which makes claims about the accuracy of system outputs in the context of the practices that produce them and:
- uses qualitative methods to uncover the context and assumptions underlying a system
- context reveals actual practices used to produce system inputs and evaluate system outputs
- assumptions point to ideal system behavior, and to facets across which system outputs should be evaluated
- uses symmetry (from STS) to enable audits when ground truth is unknown or contested
- symmetry requires treating competing theories of truth as if either might be true and points to where where measurements are in tension/where assumptions should be further investigated

A Case Study of Skeleton Inference in Motion Capture
- Mocap systems use sensors and algorithms to infer measurements about bodies
- We audited the accuracy of body segment parameters (BSPs), e.g., the distance between body landmarks
- Through field work at a mocap lab, we documented the context in which mocap for creative design is conducted
- Through a historical literature review, we uncovered a key assumption: mocap systems are optimized for bodies that are thin and male
- We tested whether BSPs are equally accurate across the facets of body size and sex
- We followed standard practices used in mocap for creative design
- The true value of BSPs is unknown, so we compared BSPs produced by a mocap system to those produced with a tape measure
- We measured how differences between mocapped and tape measured BSPs vary across facets

@alexanderloth.bsky.social Oct 12, 2026, 9:12 AM

Today at #AIES2026 in Malmö I joined the panel on Participation, Sovereignty, and Power in AI Governance with The Mitigation Paradox, joint work with Mina Schütz, Martin Kappes and Marc-Oliver Pahl. 3 takeaways:
https://doi.org/10.5281/zenodo.23302284

AIES 2026 Discussion Session 1 in Malmo Live: panel on stage under the conference screen, audience in the foreground.

@uhleeeeeeeshuh.bsky.social Oct 12, 2026, 1:06 AM

Are algorithm audits meaningfully fixing issues in algorithmic systems? In our paper at #AIES2026 this week, @davidthewid.bsky.social and I argue that it’s time to deprioritize algorithm audits & instead shift scrutiny to the larger systems of power in which audits sit. 1/n

image of first page of the AIES 2026 paper titled "No More Plaudits for Algorithm Audits: A Provocation" by Alicia DeVrio & David Gray Widder

@alexanderloth.bsky.social Oct 11, 2026, 6:40 PM

Our #AIES2026 paper (Malmö, tomorrow): 54 experts from 21 countries agree on the threats of AI disinformation but split on the solutions. Government regulation: 30% call it most effective, 15% least effective. The Mitigation Paradox.

Preprint: doi.org/10.5281/zeno...

@pranav-nlp.bsky.social Oct 10, 2026, 11:54 AM

On Wednesday, I will present my very personal work on auditing AI startups. Here we look at how AI startups used AI-washing to attract investment for surveillance based products. #AIES2026

Title slide styled as a retro computer window with a salmon title bar, cream body and decorative sparkles. The title reads “Auditing AI-Washing in German Startups: A Mixed-Methods Study of Marketing Claims and Perceptions”, with a box below stating “Presentation by Pranav A”.

@pranav-nlp.bsky.social Oct 10, 2026, 11:52 AM

On Tuesday, I will give an overview of my PhD research which focuses on resisting surveillance AI. I will present my completed and upcoming auditing research on surveillance, age estimation and benchmarking (and what can you do about it!). #AIES2026

@pranav-nlp.bsky.social Oct 10, 2026, 11:50 AM

On Monday, I will present this poster on critically analyzing mass surveillance datasets. I empirically show here that collecting CCTV style images is nonsense idea and does not even improve the model performance. #AIES2026

Poster titled “Training on Mass Surveillance Datasets Provides No Empirical Benefits”, by A Pranav and Anne Lauscher, with University of Hamburg and Trustworthy AI Hamburg logos and a black-and-white illustration of a surveillance camera labeled 1086. At top right, three sample image strips from the Market-1501, CUHK03 and CUHK-SYSU datasets show pedestrians, each with one query crop outlined in red and two matches outlined in green. Beneath them the text reads “Re-identification datasets are built from non-consensual surveillance footage.” A horizontal bar chart of pooled tracking IDF1 (%) shows Pretrained at 28.1 and Finetuned at 28.5, both inside a shaded band from -2 to +2 around the pretrained value. The closing text reads “Finetuning on re-identification datasets does not improve downstream tracking.”

@pranav-nlp.bsky.social Oct 10, 2026, 11:45 AM

I will be in #AIES2026 next week, presenting the following papers. Come by if you want to talk about surveillance, benchmarking, and age estimation!

Table with two columns, “Session” and “Time and Date”, listing five rows: (#1086) “Training on Mass Surveillance Datasets Provides No Empirical Benefits”, Monday, 5 pm; “Student Panel on Accountability and Autonomy”, Tuesday, 9 am; (#S-100) “Resisting Surveillance AI: Mechanisms of Accountability Evasion and Community Engagement”, Tuesday, 5 pm; (#1090) “Auditing AI-Washing in German Startups: A Mixed-Methods Study of Marketing Claims and Perceptions”, Wednesday, 10.45 am; and “Queer in AI socials”, all days after 6 pm.

@pranav-nlp.bsky.social Oct 9, 2026, 12:21 PM

"AI-powered", "smart", "agentic", cockroaches doing surveillance, tracking: these are marketing terms used by startups to exaggerate claims of AI, which we term as 'AI-washing'.

In our #AIES2026 paper, we explore these marketing copies and perception of such advertisements on users and public.

Table of six excerpts from German startup websites, each introduced by a startup type in bold followed by an italic quotation. A legal-research startup claims its AI invents nothing and has no hallucinations. A digital-transformation advisory startup says its AI delivers company-wide solutions without workshops, consultants, or IT projects. A bio-robotic surveillance startup describes releasing cockroaches fitted with a backpack for control, sensing, and secure communication. A call-centre voice-AI startup says its digital call center workers match or beat human agents on customer satisfaction at scale. A specialty-chemicals startup advertises AI twins for chemical formulations. An automotive voice-AI startup describes digital employees that think, talk, and act alongside human counterparts.

Table of five startup excerpts, each labelled with a startup type and a risk category in parentheses. Three are labelled surveillance: a geospatial-AI startup that detects unusual activity and monitors crowd movements for law enforcement, a parking-surveillance startup that tracks persons, animals, and personal data, and a defence bio-robotics startup that deploys silent sensors for covert surveillance in politically sensitive urban areas. One is labelled manipulation: a voice-AI call-centre startup that combines LLMs, human-like voices, and background call center music to reach a CSAT score above 90 percent. One is labelled autonomous decisions: an AI fertility-clinic startup whose computer vision tool selects the highest-quality embryo from clinical and lab data.

@sauvik.me Sep 26, 2026, 6:01 PM

New #AIES2026 paper led by @hankhplee:

Agentic AI devs focus on product and business risks over downstream risks like job displacement and privacy. Lacking mature controls, they contain risk by constraining the same capabilities that make agents useful.

sauvikdas.com/papers/73/s...

@resilientcoders.bsky.social Sep 9, 2026, 2:27 PM

We're excited to announce that we are community partners for the 4th Annual Baddie Con: AI Edge Summit on September 25th in #Atlanta!

You can use our code BCPARTNER30 for a special discount on tickets. Learn more: baddiesintech.com/ai-edge-summit

#BaddieCon2026 #baddiesintech #AIES2026 #Tech