Practical overview
When it fits
Use DialoGLUE when an experiment needs a shared set of task-oriented dialogue datasets and a documented process for preprocessing, baseline evaluation or leaderboard submissions.
Who it is for
- Researchers comparing task-oriented dialogue NLU models.
- Teams studying transfer between dialogue domains or learning from limited labeled data.
- Developers preparing predictions for the DialoGLUE benchmark leaderboard.
Documented uses
- Download and normalize the benchmark’s dialogue datasets.
- Train or evaluate baseline models across benchmark tasks.
- Format model predictions for leaderboard evaluation.
- Compare few-shot results using the benchmark’s reduced training splits.
Requirements & limits
The repository is public and licensed under Apache-2.0. The README lists Python 3.7 as tested and functional and points to requirements.txt for dependencies.
- The benchmark covers seven datasets with different task formats and dataset licenses.
- The README says run.py handles almost all models, while MultiWOZ uses a modified TripPy implementation.
- Few-shot experiments use approximately 10% of training data; the README recommends five runs per task.
- The repository includes scripts for benchmark preparation and evaluation, rather than describing a hosted model-training service.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Download and prepare DialoGLUE datasets
Create the benchmark’s processed dataset files for model work or evaluation.
Before you start
- Python 3.7 is listed as tested and functional.
- Install the dependencies listed in requirements.txt.
- Use the DialoGLUE repository.
Steps
Open the repository’s data_utils directory.
Run bash download_data.sh from that directory.
Use the resulting dataset files, including the training, validation and test splits.
Expected outcome
The script downloads and processes benchmark datasets into the directory structure described in the README.
Guide limitations
- The benchmark incorporates previously published datasets with differing licenses; check each dataset’s terms.
- The README describes data preparation, not a guarantee that every external dataset source will remain available.
Prepare outputs for a DialoGLUE leaderboard submission
Format model predictions as a benchmark submission file.
Before you start
- Model outputs for the seven DialoGLUE datasets.
- For the documented checkpoint-based workflow, edit and run dump_outputs.py; the few-shot workflow uses dump_outputs_fewshot.py.
Steps
Arrange predictions in the task-specific formats described in the README.
For the standard workflow, edit and run dump_outputs.py to generate a JSON submission file.
For few-shot experiments, provide outputs from five model runs and use dump_outputs_fewshot.py to generate the submission file.
Expected outcome
A JSON file formatted for submission to the DialoGLUE leaderboard.
Guide limitations
- Intent classification, slot filling, TOP and MultiWOZ use different output formats.
- The README recommends five runs for few-shot tasks and leaderboard scores average those runs.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 16 October 2026
- Original website title
- alexa/dialoglue · Official homepage title · retained in its source language