It’s easy to see why ChatGPT (or other prompt-based AI) would draw the interest of software engineers, especially when it comes to accelerating or streamlining processes through prompts. Yet, AI and their human counterparts are still learning and evolving.
So, the question becomes:
Is ChatGPT ready to help software engineers?
A recent study suggests – not really.
A Purdue University study shows that when you ask ChatGPT questions related to Stack Overflow questions, it’s likely to reply with a high volume of incorrect responses.
Out of the 517 Stack Overflow questions asked by researchers.
The results:
• 52% of ChatGPT's answers were wrong
• 77% of the answers were extremely wordy
• 65% provided extensive answers, answering all facets of the questions
The extensive responses are relevant for several reasons. The study pointed out that wordiness caused software engineers to miss mistakes.
The concerns:
Sabrina Ortiz from ZDNET said the results should concern software engineers and their employers when it comes to the spread of misinformation. She said the low accuracy scores and associated risk should be reasons to reconsider using ChatGPT for certain prompts.
The spread of misinformation is already a critical ethical consideration when utilizing AI for software engineering purposes.
John Lopez at Tech Report noticed how the research showed that ChatGPT responded with more positive responses than Stack Overflow and considered the ethical challenges that it creates. He said these positive tones could contribute to user trust in ChatGPT's answers when they possess inaccuracies.
If anything, the study provides software engineers and aspiring software engineers with another reason to believe that AI won’t replace their jobs. Humans are going to have to remain part of the process consistently.
Software engineers interested in AI should focus on grasping fundamental machine learning concepts, mastering Python programming, and becoming proficient with popular AI frameworks like TensorFlow and PyTorch. They should delve into neural networks, deep learning architectures, data preprocessing, and model evaluation techniques.
Additionally, gaining expertise in specific ways to apply AI is essential. This includes learning to automate routine tasks, create effective prompts, integrate AI into your workflows and analyze ethical considerations. The important thing for you to do is choose your path and keep learning!