ChatGPT: A scientific proofreader’s thoughts on using AI to read and summarize the literature
There’s been a huge amount of interest and concern about ChatGPT in the scientific and academic communities since this AI tool was launched in November 2022.
ChatGPT, one of several AI tools developed by OpenAI, is a conversational AI that is designed to “answer follow-up questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests”.
There is alarm about the potential misuse of AI in scientific research, with some people fearing that AI tools like ChatGPT could be used to write an entire scientific paper or thesis/dissertation.
In fact, some scientific papers have already been published with ChatGPT as an author. Moreover, abstracts written by ChatGPT have been reported to fool other scientists.
Several journal publishers, including Nature, have produced guidelines regarding the use of ChatGPT and other AI tools due to the concerns that these tools threaten the integrity of science itself.
These new guidelines state that publishers will not accept papers that include an AI tool as an author and require authors to openly state and describe any use of AI in their research. These seem like sensible developments given the massive potential of AI to be misused.
But these developments made us wonder: Does ChatGPT have any potential as a tool to help scientists read the literature? Or can it help with rephrasing text to avoid plagiarism?
All scientists need to evaluate the literature and produce new written work, so we thought it would be interesting to carry out some experiments.
Here, we provide our thoughts on the quality of the output of ChatGPT from our perspective as scientific proofreaders and English editors. We also examine whether ChatGPT can be used to save you time when reading the literature.
Experiment 1: Can ChatGPT summarize the findings of scientific papers based on its abstract?
Reading abstracts and papers can be very time-consuming, so maybe ChatGPT has the potential to save some time and effort? Let’s find out.
We asked ChatGPT to summarize three abstracts. We do not have any relationship with these papers; we simply selected some recently published papers at random.
You can read the original abstracts by clicking the link for each paper. And here’s ChatGPT’s summary of the first abstract:
Overall, the quality of the output is relatively clear, with each answer providing a brief summary of the abstract used as the input. I’m even impressed with the style of language, which is appropriate for scientific writing and does not contain obvious grammatical errors.
However, if we compare the output with the three abstracts used as input, it becomes clear that the AI can only produce a basic summary and also that the summaries are quite repetitive.
I’m pretty sure that any student or researcher who simply combined the output of ChatGPT summaries into their thesis/dissertation would soon be discovered. Like all safety warnings: please don’t try this at home (or in the lab, or office)!
Firstly, the AI might help to summarize the text relatively well, but this does not actually help us to understand the work or identify any limitations. Basically, we still need to read the paper to upload it into our human intelligence system in order to thoroughly evaluate the quality, novelty, and limitations of the study.
Conclusion of experiment 1: ChatGPT can summarize the findings of scientific papers based on their abstracts, but the output is simplistic and becomes repetitive.
Overall, I conclude that ChatGPT can summarize papers relatively well, but the output is repetitive and you still need to read the paper to understand and evaluate the work properly.
Moreover, ChatGPT may offer some scope to make life easier, but I would strongly recommend that you don’t write a paper or thesis by simply combining a number of ChatGPT summaries.
Experiment 2: Can Chat GPT rephrase the text of scientific papers to reduce the risk of plagiarism?
Plagiarism is a constant issue that worries students, supervisors, and journal editors. So, in our next test, we investigated whether ChatGPT could be useful for rephrasing text to reduce plagiarism.
We asked ChatGPT to rephrase the summary it produced for Paper 1: Comparison of RNA Marker Panels for Circulating Tumor Cells and Evaluation of Their Prognostic Relevance in Breast Cancer
Here is the input:
And here’s the output generated by ChatGPT when we asked it to rephrase:
From my perspective as a scientific proofreader, the output is quite good and the AI tool mimics the steps I would actually take when paraphrasing.
Firstly, ChatGPT did not change any of the important scientific names or concepts in the text such as circulating tumor cells, RNA marker panels, or sensitivity.
It’s crucially important to keep using the same names for defined scientific concepts when rephrasing; otherwise, the meaning of the text could change or we could confuse our readers.
Instead, ChatGPT mostly rephrases the text by changing other less crucial nouns. For example, it changed the research paper in the first sentence to study and The paper in the last sentence to The authors.
This strategy is an easy way to rephrase text without changing the meaning, and ChatGPT does the job in much the same way as I would approach this task.
ChatGPT also adopts one of my favorite strategies for paraphrasing: changing verbs to other verbs with similar meanings.
For example, ChatGPT changed assesses in the first line to examines and changes investigates in the same sentence to assesses. In the last sentence, the AI tool changes recommends to concludes.
Again, this is exactly what I try to achieve when rephrasing: changing verbs is a simple strategy to break up the length of the sections of text with 100% similarity.
ChatGPT does a good job of replacing the verbs with other words that have similar meanings, and most importantly, it uses verbs that are appropriate for scientific writing.
Interestingly, when we ask ChatGPT to rephrase the same summary again, it actually starts to warn about the risks of using the tool to rephrase:
Conclusion of Experiment 2: Chat GPT can rephrase the text of scientific papers to reduce the risk of plagiarism, but you need to be very careful about adopting this strategy
Overall, I have to admit that ChatGPT’s output of this rephrasing task is surprisingly good, both in terms of using grammar and style that is appropriate for scientific writing and keeping the original meaning of the text.
However, as a caution, it is important to remember that there is a lot of controversy about ChatGPT and some publishers are developing tools to detect the output of AI tools like ChatGPT. So you need to be very careful!
As I mentioned for the first task above, while the output of the rephrasing task is good, I would not recommend that you write your thesis or paper by asking ChatGPT to summarise or rephrase the text.
After all, we still need to read the paper ourselves to understand and evaluate the content and make sure we are describing the work appropriately.
Welcome!
At Science Editing Experts, we help scientists like you to submit well-written journal papers with confidence and complete your thesis without headaches, so you can focus on your research and career.
Andrea Devlin PhD
Chief editor and owner of Science Editing Experts
The essential list of "Red Flags" in scientific writing:
348 words and phrases that scream "Written by ChatGPT or AI!"
The essential list of "Red Flags" in scientific writing:
348 words and phrases that scream "Written by ChatGPT or AI!"
2 Drumgrannon Heights, Moy, Northern Ireland, BT71 7TW, UK.
© Science Editing Experts 2022-2024. All Rights Reserved.