My Research
I am a motivated researcher; since July 2021, I have been a member of BILSEN (Bilkent University Software Engineering and Data Analytics Research Group) of the Computer Engineering Department at Bilkent University, led by Dr. Eray Tuzun. Mainly, I am focused on the source code generation capabilities of Large Language Models (LLMs).
Although I focused on Empirical Software Engineering in my undergraduate studies, my research interests are not limited to this subject. I am also interested in other areas, such as Data Science, and Machine Learning, which can be proven by my projects, internships, and some of the courses I have taken in my undergraduate studies. These can be viewed on my CV and homepage.
Publications
Conference Papers
Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT
April 21, 2023
Burak Yetiştiren, Işık Özsoy, Miray Ayerdem, and Eray Tüzün. 2023. Evaluating the code quality of AI-Assisted Code Generation Tools: An empirical study on github copilot, Amazon CodeWhisperer, and chatgpt. https://doi.org/10.48550/arXiv.2304.10778
Assessing the Quality of GitHub Copilot’s Code Generation
November 17, 2022
Burak Yetistiren, Isik Ozsoy, and Eray Tuzun. 2022. Assessing the quality of GitHub copilot’s code generation. In Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE 2022). Association for Computing Machinery, New York, NY, USA, 62–71. https://doi.org/10.1145/3558489.3559072
Invited Talks
Microsoft PROSE Team
January 18, 2023
"Assessing the Quality of GitHub Copilot’s Code Generation"
The introduction of GitHub’s new code generation tool, GitHub Copilot, seems to be the first well-established instance of an AI pair-programmer. GitHub Copilot has access to a large amount of open-source projects, enabling it to utilize more extensive code in various programming languages than other code generation tools. Although the initial and informal assessments are promising, a systematic evaluation is needed to explore the limits and benefits of GitHub Copilot. The main objective of this study is to assess the quality of generated code provided by GitHub Copilot. We also aim to evaluate the impact of the quality and variety of input parameters fed to GitHub Copilot. To achieve this aim, we created an experimental setup for evaluating the generated code in terms of validity, correctness, and efficiency. The results suggest that GitHub Copilot was able to generate valid code with a 91.5% success rate. In terms of code correctness, out of 164 problems, 28.7% were correct, while 51.2% were partially correct, and 20.1% were incorrectly generated. Our empirical analysis shows that GitHub Copilot is a promising tool based on the results we obtained, however further and more comprehensive assessment is needed in the future.