01
Single-turn track
One chance to clarify
The system may make at most one clarification turn before returning its implementation.
ICSE 2027 Competition · Dublin, Ireland
Can your coding assistant ask the right question?
Loading competition summary…
Final submission deadline TBA Final competition submission cutoff →Public benchmark
Leaderboard
Rankings below are for the public validation set. Final private-test results remain hidden until the competition ends.
Results
Track
Clarification systems are evaluated on the public validation benchmark.
Rank by TDS ↑ · tie-break by nDCG ↑
| # | Algorithm | Model used | TDS ↑ | nDCG ↑ tie-break | Pass@1 ↑ | Clarify ↑ | Over-ask ↓ | Avg. cost ↓ | Submission |
|---|---|---|---|---|---|---|---|---|---|
| Loading leaderboard… | |||||||||
How to read the metrics
↑ higher is better · ↓ lower is better
TDSTurn-discounted success. Measures task success while discounting extra clarification turns. Primary ranking metric.
nDCGClarification quality/order relative to the ideal ordering. TDS tie-breaker only.
Pass@1Share of first generated solutions that pass the tests.
ClarifyShare of tasks where the system asks at least one clarification.
Over-askShare of already-specified tasks where the system asks unnecessarily.
Avg. costAverage model/API cost per evaluated task.
Only TDS and, for ties, nDCG affect rank. Baselines are reference rows and remain unranked; submissions below the strongest baseline are also shown without a rank.
Private test results are sealed
Results on the private test set will be evaluated and published after the competition closes.
Competition rules
What you need to know
A few simple requirements keep submissions comparable and reproducible. Technical details live in the repository.
02
Multi-turn track
Clarify iteratively
The system may ask multiple targeted clarification questions within the organizer-defined turn budget.
Implement the SDK interface
Submit an algorithm inheriting from ClarificationAlgorithmBase and use the provided competition environment.
Submit through GitHub
Add your algorithm and required team metadata, then open a pull request before the final deadline.
Make the run reproducible
Keep dependencies and model configuration explicit so organizers can reproduce the submitted system.
Keep the private test set private
Develop on the released data and public validation set. Organizers run the hidden test set only for final evaluation.
Evaluation
What gets measured
TDS determines leaderboard rank, with nDCG used only to break TDS ties. The remaining metrics are reported for transparency.
TDS
Turn-discounted success
Primary ranking metric. Leaderboard positions are ordered by TDS from highest to lowest.
nDCG
Clarification quality
Tie-breaker. When two systems have the same TDS, the higher nDCG ranks first.
P@1
Pass@1
Reported only. Raw percentage of generated implementations that pass the developer tests.
CR
Clarification rate
Reported only. Percentage of tasks for which the system asks at least one clarification question.
OA
Over-asking rate
Reported only. How often the system asks a question even though the task is sufficiently specified. Lower is better.
$
Average cost
Reported only. Average LLM/API cost per task; it does not affect leaderboard rank.
Prizes & recognition
Compete, publish, and present at ICSE 2027
Strong competition entries are recognized not only on the leaderboard, but also through participation in the ICSE 2027 program in Dublin.
Winner prize
Free ICSE registration
The winning team receives one complimentary registration to ICSE 2027.
For every ranked entry
Solution paper & ICSE presentation
Every eligible candidate — i.e. every submission that receives a leaderboard rank — can submit a solution paper for the ICSE 2027 Competition Track and present the approach in the competition session in Dublin.
ICSE solution papers
4 pages, reviewed for technical correctness
ICSE 2027 Competition Track solution papers are limited to 4 pages including references and follow ICSE formatting. Papers are reviewed for technical correctness, with an opportunity to address reviewer comments before final acceptance. Accepted papers appear in the conference proceedings. Official ICSE Competition Track details ↗
Competition sponsor
University of Luxembourg
The Clarification Challenge is sponsored by the University of Luxembourg.
Schedule
Important dates
The final submission deadline is the key cutoff for competition entries. Dates remain data-driven so organizers can update them without changing the page layout.
Competition timeline
TBA
Loading…
ICSE 2027 solution-paper timeline
4 Dec 2026
Solution papers
ICSE Competition Track deadlines are 23:59 AoE (UTC−12) unless stated otherwise. Competition-specific deadlines above are maintained separately.
Get started
From idea to leaderboard
Participation follows the repository workflow, so teams can prototype locally and submit the final algorithm through GitHub.
Open competition repository ↗- 01Fork & install
Fork the repository, install the SDK with
uv, and configure your model API key. - 02Implement
Add a single clarification algorithm that inherits from
ClarificationAlgorithmBase. - 03Evaluate
Run response generation and evaluation on the available train/validation splits.
- 04Submit
Open a pull request with the required team and licensing header before the final submission deadline.
- 05Final evaluation
The organizers freeze eligible submissions, evaluate them on the hidden private test set, and publish the final leaderboard.
FAQ
Common questions
Short answers for participation and evaluation. The repository remains the source of truth for implementation details.
Loading FAQ…
FAQ entries are loaded from data/competition.json.
Organizers
Competition organizers
The competition is organized by researchers from the University of Luxembourg and Michigan Technological University.
Amal AKLIUniversity of Luxembourg
Jie JW WuMichigan Technological University
Cedric RichterUniversity of Luxembourg
Mike PapadakisUniversity of Luxembourg
For contact details and the latest competition instructions, use the competition repository ↗.