90% of US working hours are spent on tasks not covered well by AI benchmarks

來源原標題

90% of US working hours are spent on tasks not covered well by AI benchmarks

作者

speckx

原始標籤

hackernews

來源訊號

HN points 581　HN 留言 525

原始連結

https://www.mercor.com/research/labor-index/benchmark-map/

摘要原文

AI benchmarks have multiplied in recent years, but coverage of economically valuable work remains limited. To measure this, we compiled relevant benchmarks to measure their coverage of American jobs. This Benchmark Map is our first step toward building a comprehensive, continuously updated resource for understanding AI capabilities across the economy. Each tile represents one of the 923 O*NET occupations making up the US labor force, grouped into 22 job families. Occupations are shaded according to the share of their tasks which are represented in our initial list of AI benchmarks. A single benchmark prompt can be mapped to multiple O*NET tasks. In our data set: only 208 of 923 occupations have a task with at least 5 benchmark prompts , and 6 of 22 job families have no such occupations. Our findings suggest that 90% of American working hours are spent on tasks with little or no coverage from existing benchmarks . This makes it very hard to know how good AI models are at the work Americans actually do. The coverage of existing benchmarks is not random, either. Our map suggests that the AI community is focusing on the largest labor markets first: Adjusted probability of task coverage Annualized occupation wage bill (log scale) Each point shows the estimated probability that an O*NET task with a given AI-exposure tag belonging to an occupation with a given wage-bill will be covered by at least 5 prompts in our benchmark index. AI-exposure tags are binary and come from Eloundou et al. (2023) . Occupation wage-bills are calculated from BLS estimates of the mean wage and number of Americans working in each occupation, assuming a 40 hour work week. Estimated coverage probabilities come from a task-level logistic regression of coverage probability on AI-exposure and log occupational wage bill. Error bars are pointwise 95% delta-method confidence intervals based on standard errors clustered at the occupation group level. For example, benchmarks in our initial list heavily cover AI-exposed lawyers’ tasks, but barely cover any of the O*NET tasks of similarly AI-exposed legal secretaries or court clerks. The long tail of smaller, middle-income knowledge work jobs appears undercovered. As we build out our coverage of model capabilities across the economy, we hope to both surface and help fill more gaps in our understanding of what models can do. To break down the economy into sectors, occupations, and tasks, we used the US Department of Labor’s O*NET taxonomy. O*NET divides the US workforce into 923 occupations; for each occupation, O*NET attempts to provide a comprehensive list of “tasks” which make up the occupation, totaling 18,797 tasks across all occupations. To measure benchmark coverage, we start with a list of 29 benchmarks selected for their quality and economic relevance. (The full list and selection criteria are in the Benchmark Index tab). In selecting benchmarks for this list, we prioritized: Following work by Wang et al. , we then ask an LLM judge to tag each benchmark prompt with as many O*NET tasks as it thinks correspond to that prompt. We validate our LLM tagging with human expert tags on Mercor’s APEX benchmarks, a subset of this list of 29. On the Benchmark Map, an O*NET task counts as “covered” if our LLM judge has mapped it to at least 5 prompts from our 29 benchmarks. To estimate the total share of working hours Americans spend on tasks covered by benchmarks on this list, we combine our coverage estimates with Hatgis-Kessell et al. ’s LLM-derived estimates of the working time share of each O*NET task. A more detailed account of our methodology can be found in the Methodology tab. Our estimates of total economic coverage, and of patterns in coverage, all come from the initial list of 29 benchmarks we chose. This is not a complete or perfectly representative sample of the broader universe of unsaturated, public, agentic economic benchmarks. Any incomplete list will necessarily bias down estimates of economic coverage, and it is difficult to bound that bias. We want to stress, therefore, that these results are provisional. We will update them as we expand the initial benchmark list. We also capped the number of included benchmarks in each job family in order to maximize breadth of coverage. We think this makes our estimate of economy-wide benchmark coverage more accurate, at the cost of biasing down our estimates of coverage in the most-benchmarked job families (e.g. software engineering). By selecting for unsaturated benchmarks, we risk understating benchmark coverage on O*NET tasks which are easier for LLMs to complete. On the other hand, including saturated benchmarks would risk overstating benchmark coverage if saturation is downstream of lower task realism. The latter claim seems likely, given that benchmarks often overstate LLM performance in real-world deployments. We think that the bias from including saturated benchmarks would be greater than the bias from excluding them, but neither option is ideal. Finally, while we outline our benchmark selection criteria in the Benchmark Index tab, several criteria (like “Economic Realism”) are difficult to formalize and introduce some subjectivity in our choices. The benchmarks in our index have relatively deep and narrow coverage of the overall economy. They disproportionately focus on the economic tasks most exposed to AI automation (as measured by Eloundou et al. ), with O*NET tasks rated as “exposed” about 6x more likely to be represented on benchmarks than tasks rated “unexposed”. This makes sense: benchmarks are designed to measure AI progress, and AI progress is most plausible on the most AI-exposed tasks. This explains why benchmarks are full of tasks in Software Engineering and Financial Analysis while sparse on tasks in Truck Driving and Gas Plant Operation. But even after controlling for AI-exposure, O*NET tasks from occupations with larger wage bills are significantly overrepresented among benchmark prompts: Adjusted probability of task coverage Annualized occupation wage bill (log scale) Each point shows the estimated probability that an O*NET task with a given AI-exposure tag belonging to an occupation with a given wage-bill will be covered by at least 5 prompts in our benchmark index. AI-exposure tags are binary and come from Eloundou et al. (2023) . Occupation wage-bills are calculated from BLS estimates of the mean wage and number of Americans working in each occupation, assuming a 40 hour work week. Estimated coverage probabilities come from a task-level logistic regression of coverage probability on AI-exposure and log occupational wage bill. Error bars are pointwise 95% delta-method confidence intervals based on standard errors clustered at the occupation group level. After controlling for occupation and AI-exposure score, O*NET “task importance” is another predictor of benchmark coverage. In particular, O*NET tasks come with domain expert estimates of the importance of the task for its occupation on a scale of 1-5. We find that the tasks scored most important are about 6–10x more likely to be represented on benchmarks than those scored least important: Adjusted probability of task coverage O*NET task importance (1–5 rating scale) Each point shows the estimated probability that, controlling for occupation, an O*NET task with a given AI-exposure tag and a given quintile O*NET importance score is covered by at least 5 prompts in our benchmark index. AI-exposure tags are binary and come from Eloundou et al. (2023) . Importance quintiles are calculated from O*NET importance scores. Estimated coverage probabilities come from a task-level logistic regression of coverage probability on AI-exposure and importance quintiles, with occupation fixed effects. Error bars are pointwise 95% delta-method confidence intervals based on standard errors clustered at the occupation group level. (581 points, 525 comments on Hacker News)