Generative AI Tutorials

GenAI Research Connection

User’s Guide

Generative AI Research Resources

Reporting Use of Generative AI in Research

The rapid advancements in Generative Artificial Intelligence (GenAI) have ushered in a wide variety of applications across diverse fields. As researchers experiment with GenAI, comprehensive reporting of GenAI use in publications is essential for research reproducibility and transparency.

We developed these guidelines by analyzing a number of publications employing GenAI (many can be found on our Generative AI resources page). These guidelines are not definitive and are expected to evolve over time. 

(Depending on the specific journals, some information may be included in the main body of the paper, some may be more appropriate as supplementary materials.)

Guidelines for Reporting the Use of Pre-trained Models such as ChatGPT

  1. The function of the GenAI model in your study. Describe the purpose of using GenAI in your study and how it fits in your project workflow.
  2. Provide the accurate name, version and developer of the model such as “ChatGPT 3.5 (Open AI, 2023)”
  3. Prompts. For Large Language Models, provide the prompts used in the study. Detail prompt selection criteria if applicable.
  4. Dataset. Describe the dataset in your study that you applied the GenAI model on. Provide a reference to the dataset.
  5. Code. If you used code to apply the GenAI model or evaluate its output, provide a reference to the original code, and specify any modifications made for the current research.  If you wrote the code, place it in a public repository, if possible, and then provide a reference.
  6. Evaluation Methods. Describe your method to evaluate the model output, such as the metrics that you used and who (or which software) served as evaluators.
  7. Hardware & Computational Assets. Provide a description of the computational resources and hardware used in your study.
  8. Performance Benchmarking. If you compared the performance of the GenAI model with the performance of other research methods, describe how the comparison was done (metrics, specifics of each method).
  9. Caveats & Biases. Describe the unknown aspects of the GenAI model that could make it challenging for others to reproduce your study.

Guidelines for What to Include When Reporting a Model You Have Trained

  1. The function of the GenAI model in your study. Describe the purpose of using GenAI in your study and how it fits in your project workflow.
  2. Training Data Details – Dataset size and origin, with details on data preprocessing, augmentation, and data issues that may impact the model such as missing data, non-representativeness, class imbalances in the dataset.
  3. Test Data Details – As above.
  4. Model Architecture – Details the specific arrangement of layers and neurons, types of activation functions, and attention mechanisms employed. Explain the reasoning for any unconventional choices made.
  5. Training Method – Learning rate, Batch size, Epochs, Optimizer, and any adaptive learning rate strategies, early stopping criteria, or custom optimizers used. The time used for training the model.
  6. Loss Function & Metrics – Specify what loss functions or metrics were chosen, and why
  7. Model Evaluation – Provide an assessment of performance using standard metrics such as accuracy and F1 score.
  8. Hardware & Computation Assets. Provide a description of the computational resources and hardware used in your study. GPU / CPU power.
  9. Performance Comparison – If you compared the performance of the GenAI model with the performance of other research methods, describe how the comparison was done (metrics, specifics of each method).
  10. Model Limitations. Describe scenarios or data types where the model might not perform optimally.
  11. Code. Place it in a public repository, if possible, and then provide a reference.
  12. Dataset. Provide a reference to the dataset, if possible.

If you are fine-tuning a pre-trained model, the guidelines from both of the above sections should be considered.