NLBSE 2027

Multitask Code Classification

Competition Overview

Automatically identifying low-quality and insecure code is an important software-maintenance challenge. Relevant evidence is often distributed across two complementary artefacts: source code and the natural-language comments written by developers.

The NLBSE'27 Code Classification Competition invites participants to build a model that uses both artefacts to address two related tasks jointly:

• vulnerability classification for source code; and
• Self-Admitted Technical Debt (SATD) classification for developer comments.

Models are assessed not only by their predictive performance, but also by their execution time and computational cost in a shared evaluation environment.

Participants must develop one multitask model that receives a composite code-and-comment artefact and jointly predicts two binary labels. Each input contains a normalised source-code function together with its associated developer comments.

The expected output is a two-element binary vector in the following order:

Output	Meaning
[0, 0]	No vulnerability and no SATD
[1, 0]	Vulnerability and no SATD
[0, 1]	No vulnerability and SATD
[1, 1]	Vulnerability and SATD

The primary predictive objective is to maximise the arithmetic mean of the vulnerability and SATD F1-scores. Participants are encouraged to investigate representations and integration strategies that exploit the complementary information in code and comments.

Dataset

The competition uses a multi-annotated dataset based on MADE-WIC, a collection of functions and comments mined from open-source projects. Each instance contains source code and its associated comments, together with the following annotations:

1. a vulnerability label associated with the source-code component; and
2. an SATD label associated with the comment component.

The competition dataset is divided into three partitions:

Partition	Availability	Intended use
Training set	Public	Train the submitted model
Validation set	Public	Measure the performance of the trained model
Test set	Hidden	Final evaluation by the competition organisers only

The hidden test set will be used to determine the final winner. It is not disclosed to participants and is withheld to measure how well submissions generalise beyond the public training and validation data.

Requirements

You must train, tune, and evaluate your model on the provided data. We look forward to solutions that outperform our baseline model.

Detailed instructions about the competition (data, rules, baseline, results, etc.) can be found in our (GitHub Repository) and a (Google Colab Notebook).

Competition Organizers

The competition is organized by: Moritz Mock (momock@unibz.it), Thomas Borsani (tborsani@unibz.it), and Barbara Russo (brusso@unibz.it)

Participation Requirements

To participate in the competition, you must train, tune, and evaluate your model using the provided training and validation sets.

Additionally, you must write a paper (2-4 pages) describing:

• The architecture and details of the classification model
• The procedure used to pre-process the data
• The procedure used to tune the classifier on the training set
• The results of your classifier on the validation set
• A link to the code/tool with proper documentation on how to run it and replicate the results

Submit the paper by the deadline using our submission form. All submissions must conform to the (ICSE'27 formatting and submission instructions) and need not be double-blind.

Submission Acceptance

Submissions will be evaluated and accepted based on correctness and reproducibility, defined by the following criteria:

• Clarity and detail of the paper content
• Availability of the code/tool, including the training/tuning/evaluation pipeline, released as open-source
• Correct training/tuning/evaluation of your code/tool on the provided data
• Correct report of the metrics and results
• Clarity of the code documentation

We will use a formula to rank the competition submissions and determine a winner. Details will be provided in the Google Colab notebook.

The accepted submissions will be published in the workshop proceedings.

Skill Classification Competition

Competition Overview

The 2027 competition consists of building and assessing multi‑label classifiers that predict, for each issue, the set of domains and sub‑domains representing the skills required to solve it.

Dataset

We release a dataset mined from 7,245 merged pull requests across 11 popular Java repositories (57,206 source files; 59,644 methods; 13,097 classes) annotated with 217 skill labels composed by domain/sub‑domains. You may find the dataset in a SQLite database, named skillscope_data.db, in our GitHub repository. This dataset is an updated version of the one submitted in the original SkillScope paper.

• Ready-made Table: Inside the database you will find a table named nlbse_tool_competition_data_by_issue that joins each pull request's textual and code‑context features with its canonical domain/sub‑domain labels per issue. There is also a view vw_nlbse_tool_competition_data_by_file that labels each filename/function associated with each issue.
• The nlbse_tool_competition_data_by_issue table contains a column for each domain and subdomain with an integer count of the number of matching APIs found for that issue. A value greater than zero indicates that domain/subdomain is present in the issue.
• The vw_nlbse_tool_competition_data_by_file is present for convenience purposes only and will not be used in evaluation of the model.

The dataset can be found in our Github repository.

Additional Data Usage

Your model is allowed to use input data that is outside what is given in the database. You may use additional GitHub APIs to fetch more metadata, or download relevant files in an issue for further analysis. However, you must not use any third-party classification engine or the outputs present in skillscope_data.db as direct inputs to the model.

Baselines

Your models will be compared against the SkillScope Random‑Forest + TF‑IDF baselines reported in the paper by evaluating the overall prediction metrics against the issue classifications as recorded in the nlbse_tool_competition_data_by_issue table. The models you create should return a multi-label classification encoded in an one-hot encoded vector. However, it is the metrics of your models which will be the subject to the evaluation instead of the specific model output. Random Forest Baseline is provided in the folder competition_baseline with README and requirements.txt.

Goal

Train, tune and evaluate your models on the provided splits and improve at least one of precision, recall, or micro‑F1 while not decreasing the remaining metrics relative to the best baseline.

Competition Organizers

The skill classification competition is organized by: Fabio Santos (fabio.deabreusantos@colostate.edu), and Jacob Penney (jmp458@nau.edu)

Submission Acceptance & Competition

Submissions will first be filtered to ensure they do not lower any of the three core metrics (precision, recall, micro‑F1) compared with the baseline. Among qualifying submissions, ranking is determined by the largest positive improvement in micro‑F1. Ties will be broken by (1) precision, then (2) runtime.

Submissions will be judged on:

• Clarity and completeness of the paper.
• Availability and reproducibility of code/tool (open‑source required).
• Correct reporting of metrics.
• Quality of documentation.

Accepted papers will appear in the NLBSE '27 proceedings.

Ranking Details

Participants submit a single multi‑label classifier. Rankings proceed in two stages:

1. Eligibility check: The submission must not reduce any of precision, recall, or micro‑F1 compared with the baseline.
2. Ordering: Qualifying submissions are ordered by the greatest positive ∆micro‑F1.

Citing Relevant Work

General Competition Citation

Please cite if participating:

@inproceedings{nlbse2027,
  author = {Mock, Moritz and Borsani, Thomas and Russo, Barbara and Santos, Fabio and Penney, Jacob and Valenzuela{-}Toledo, Pablo and Kehrer, Timo and Panichella, Sebastiano},
  title={The NLBSE'27 Tool Competition and Challenge},
  booktitle={Proceedings of The 6th International Workshop on Natural Language-based Software Engineering (NLBSE'27)},
  year={2027}
}

Multitask Code Classification Citations

Please also cite the work underlying the dataset and baseline for the multitask code classification competition:

@inproceedings{MockEtAl2024MadeWIC,
                author = {Mock, Moritz and Melegati, Jorge and Kretschmann, Max and Diaz Ferreyra, Nicolas E. and Russo, Barbara},
                title = {MADE-WIC: Multiple Annotated Datasets for Exploring Weaknesses In Code},
                year = {2024},
                isbn = {9798400712487},
                publisher = {Association for Computing Machinery},
                address = {New York, NY, USA},
                url = {https://doi.org/10.1145/3691620.3695348},
                doi = {10.1145/3691620.3695348},
                booktitle = {Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering},
                pages = {2346--2349},
                numpages = {4},
                location = {Sacramento, CA, USA},
                series = {ASE '24}
              }

@INPROCEEDINGS{RussoEtAl2025VulSATD,
                author={Russo, Barbara and Melegati, Jorge and Mock, Moritz},
                booktitle={2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC)},
                title={Leveraging Multi-Task Learning to Improve the Detection of SATD and Vulnerability},
                year={2025},
                pages={01-12},
                doi={10.1109/ICPC66645.2025.00017}
              }

@inproceedings{feng-etal-2020-codebert,
                title = "CodeBERT: A Pre-Trained Model for Programming and Natural Languages",
                author = "Feng, Zhangyin and Guo, Daya and Tang, Duyu and Duan, Nan and Feng, Xiaocheng and Gong, Ming and Shou, Linjun and Qin, Bing and Liu, Ting and Jiang, Daxin and Zhou, Ming",
                booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2020",
                month = nov,
                year = "2020",
                address = "Online",
                publisher = "Association for Computational Linguistics",
                url = "https://aclanthology.org/2020.findings-emnlp.139/",
                doi = "10.18653/v1/2020.findings-emnlp.139",
                pages = "1536--1547"
              }

Skill Classification Citations

Please cite if participating in the skill competition:

@inproceedings{carter2025skillscope,
    title        = {SkillScope: A Tool to Predict Fine-Grained Skills Needed to Solve Issues on GitHub},
    author       = {Carter, Benjamin C. and Contreras, Jonathan Rivas and Llanes Villegas, Carlos A. and Acharya, Pawan and Utzerath, Jack and Farner, Adonijah O. and Jenkins, Hunter and Johnson, Dylan and Penney, Jacob and Steinmacher, Igor and Gerosa, Marco A. and Santos, Fabio},
    year         = 2025,
    booktitle    = {2025 IEEE/ACM International Workshop on Natural Language-Based Software Engineering (NLBSE)},
    volume       = {},
    number       = {},
    pages        = {9--12},
    doi          = {10.1109/NLBSE66842.2025.00007},
    keywords     = {Random Forest;Training;Java;Large language models;Semantics;Retrieval augmented generation;Machine learning;Open source software;Software engineering;Software development management;software engineering;skill categorization;open source software (OSS);machine learning;large language models}
}

@article{santos2023tag,
    title        = {Tag that issue: applying API-domain labels in issue tracking systems},
    author       = {Santos, Fabio and Vargovich, Joseph and Trinkenreich, Bianca and Santos, Italo and Penney, Jacob and Britto, Ricardo and Pimentel, Jo{\~a}o Felipe and Wiese, Igor and Steinmacher, Igor and Sarma, Anita and others},
    year         = 2023,
    journal      = {Empirical Software Engineering},
    publisher    = {Springer},
    volume       = 28,
    number       = 5,
    pages        = 116
}

@inproceedings{vargovich2023givemelabeledissues,
    title        = {Givemelabeledissues: An open source issue recommendation system},
    author       = {Vargovich, Joseph and Santos, Fabio and Penney, Jacob and Gerosa, Marco A and Steinmacher, Igor},
    year         = 2023,
    booktitle    = {2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)},
    pages        = {402--406},
    organization = {IEEE}
}