OpenAI has introduced SWELancer, a new benchmark to test whether frontier large language models (LLMs) can successfully complete real-world freelance software engineering tasks—and even earn up to $1 million in total payouts. The evaluation is based on 1,488 freelance software engineering jobs from Upwork, collectively valued at $1 million.

SWE-Lancer comprises over 1,400 software engineering tasks, with projects ranging from $50 bug fixes to $32,000 feature implementations. 

“Introducing SWE-Lancer: our most realistic coding benchmark to date. Still some limitations, but better than evals we had before,” said Tejal Patwardhan, who works on the benchmarks and preparedness team at OpenAI.

These tasks are divided into independent engineering tasks, where models must complete technical work, and managerial decision-making tasks, where models evaluate and choose between implementation proposals.

By mapping AI model performance to real-world monetary value, SWE-Lancer provides a crucial tool for studying the economic impact of AI in software development. More research can be accessed here.

Anthropic, the company behind the Claude model series, also released a survey highlighting AI’s influence on the workplace.

The findings revealed that approximately 36% of all occupations incorporate AI for at least a quarter of their tasks. Moreover, 57% of AI applications enhance human capabilities, while 43% focus on automation. However, only 4% of occupations rely on AI for at least 75% of their tasks.

The study identified software development and technical writing as key areas where AI is utilised. In contrast, AI plays a minimal role in tasks that involve physical interaction with the environment.

The post OpenAI Thinks LLMs Can Earn $1M from Freelance Software Engineering Tasks appeared first on Analytics India Magazine.