How did ChatGPT Perform as an Author?

I know what you’re thinking. You’re wondering whether or not I used ChatGPT for this article. That’s a fair thought… but I’d wager you never had that thought before the recent popularization of artificial intelligence in the form of ChatGPT. (For those unfamiliar with ChatGPT, take a look here.)

You should also realize that a lot of people are starting to have this same thought about your emails, your documents, your papers, and your reports as well. I know I had some of those thoughts as my students just handed in their first assignment of the semester. Last week, I asked ChatGPT to write an article, joining at least 100 million people using and exploring ChatGPT as a tool, a game, a resource, a teacher, and, most worrisome for some people, a huge time saver of writing term papers. Everybody is excited about the potential for this new technology, but also scared about it replacing some, or many, jobs throughout society. Doctors, lawyers, accountants, journalists, and yes, even design engineers, will have some aspects or part of their careers altered or completely removed because of the AI capabilities that are starting to propagate across the globe.

Is this potential wave of change inevitable, or is it overhyped?

ChatGPT: A Typical Hype Cycle Technology

One of the earliest articles on the BYU Design Review was about the Hype Cycle. In that article, we posted a figure of a hype cycle generated in August of 2018. It had a number of different AI technologies climbing up to the Peak of Inflated Expectations (see below). Today we are definitely at the peak for AI.  Everyday this year I read, or at least see, another article claiming a novel application of AI. It’s hard to not come across somebody with a claim so inflated that one could quickly believe that AI will cure world hunger, grant perfect health to every human, solve the water and energy crises, and bring peace to every nation before 2024.

Inflated expectations? I think so. Potential to still make a real difference for good? I hope so.

Reposted Hype Cycle from 2018. Note the AI references on the steep upward climb in the innovation triggers phase.

Even if the trajectory of AI must pass through the Trough of Disillusionment first, I’m ultimately bullish on AI as it climbs up the Slope of Enlightenment and to the Plateau of Productivity.

Interestingly, AI isn’t even that new. For a long time, you’ve probably had your email server read your emails and suggest phrases, or even complete sentences, of what to respond to save you some typing time. Most definitely you’ve seen search engines complete your search phrases and smart phones recommend an appropriate reply to your mom’s text. Chatbots have been around for a long time and various scripts online have been running automatic functions for many decades throughout the Internet.

Regardless of the decades-long history and efforts by scientists and engineers in many fields to bring us to ChatGPT, for some reason, the tipping point of awareness and adoption of AI technology happened in late 2022. This is typical as most technologies are evolutionary in nature and then seem to come out of nowhere, like a highly non-linear function. When these technologies reach some inflection point, everybody suddenly takes notice. This is how it was for computers, touch screens, cell phones, electric vehicles, and countless other things. Innovators were quietly plugging along making progress without most of the world taking notice. Then seemingly overnight everyone wanted the technology.

My predictions: There will be early adopters and late adopters. Younger people, in general, will adopt sooner but older people will catch on eventually and appreciate the benefits. New “apps” or ways of using ChatGPT, or other instantiations of AI, will be invented, monetized, and exploited. Start-up companies will be born and other companies will die - all due to AI. Significant advances in medicine and learning will occur but abuses will also arise.  AI will solve countless problems but it will also be used for nefarious purposes, such as cheating, cutting corners, and other malicious ventures. Some leaders will say we should avoid it entirely, while others will embrace it in their role to further growth and progress. It might both save lives and cut them short, just like it will make some new jobs and completely remove other ones. It will, at least, end with a film documentary in a few years, trying to be honestly objective but with an ulterior agenda that eventually seeps through…

You shouldn’t be surprised when my predictions come true. These are all very easy to make and each one follows the trend for almost every other technology invented in the last few hundred years.

Well, I predict all of the above will unfold in the future. But how does it perform today?

A Review of ChatGPT as an Author

Last week, I asked ChatGPT to write an article for the BYU Design Review. For the remainder of this article, I will share my evaluation of this experiment. Spoiler: I was both impressed and underwhelmed. I’ll start with the parts with which I was underwhelmed.

First of all, I asked for a 1000-word article. ChatGPT only gave me 542 words. Was my prompt only worth its half effort? Was this the first passive-aggressive insult I received from an artificial intelligence? Perhaps. Now, I wasn’t expecting exactly 1000 words (although that would have been impressive) but I thought ChatGPT would approach closer to that number or maybe surpass it. Why not try and exceed my expectations? I know some students and my own children try to cut corners with their assignments and chores respectively, but I didn’t expect an AI to do that as well. Perhaps this is how it was trained. After all, ChatGPT was training on documents from the Internet. Maybe some of those are incomplete too.

Speaking of not finishing, ChatGPT also didn’t even finish the article. Go check it out. It stopped mid-sentence. Now there might be many reasons for this: my own Internet connection dropped, too much traffic for their servers to support, or other bugs or errors in the beta-version code, but it was a little disappointing regardless. This of course makes my first claim a little unfair as ChatGPT might have reached 1000 words eventually. I realize I may be amplifying this apparent weakness, which was probably just an unlucky first attempt by a beginner. However, there is the option to regenerate answers from the same prompt. I tried it and the number of words was even less than 542. Another slap in the face?

My third issue was the overall tone of the article. It was good but not exciting. It was generally correct but not thrilling to read. It was adequate but not really breaking new ground. I’ve read similar sentiments expressed by others that sometimes the thing missing from ChatGPT output is the human touch. This is not a surprise. ChatGPT is not human. My analogy is that it wrote something, similar to how documents are written by committees. This is also not much of a surprise. In these types of committee documents, one can often tell that 10, 12, 24, or many more humans all had to reach a compromise in order to agree on the wording of the policies, recommendations, or standards presented in some long, dry list or table, where the statements are an amalgamation of everyone’s ideas in the room (or from the email chain). These are no doubt useful and important documents but they aren’t ever written in the voice of just one person. They are in the voice of a committee - usually quite neutral, generally politically correct, repetitive sentence structure, perhaps even boring, and legally signed off by the lawyers on the committee. Since ChatGPT was trained on the multiple voices of millions of people from the internet, it has a very committee-esque feel to it.

Now for the good things. ChatGPT did in fact write an article. It was logically structured, flowed reasonably well, had no typos (except the premature last sentence ending), and included some good content to share with the world. As an introductory article to what ChatGPT is and what it can do for design engineers, I give it pretty good marks for answering my prompt. Although I was harsh on the dry tone above, I probably could have asked for a certain style as part of my prompt. My other experiments have asked ChatGPT to write a poem in the style of Shakespeare. Thus, perhaps it isn’t ChatGPT’s fault but user error, and in this case, the user is me. We all have to learn to use technology; perhaps this is my next task.

I was also impressed that some of the statements seemed to indicate it was trained on current thoughts and worries of real people. For example, it wrote “ChatGPT is a tool and not a final solution”, and that “Engineers must be aware of ethical considerations such as bias and interpretability, when using the model” suggesting that, even though it is not-self aware, ChatGPT knows that it’s accuracy is subject to what data it’s been trained on and that the data could have limitations. These various qualifying statements were reassuring to read.

Lastly, I enjoyed being reminded of certain things that I hadn’t thought about recently. For example, I haven’t recently followed the latest progress in using AI to develop CAD or 3D modeling. I now plan to investigate the latest research in that area. I also appreciated the paragraph that listed some things that engineers will need in order to use ChatGPT successfully. Now, I didn’t necessarily ask for those answers in my prompt, but ChatGPT suggested designers will need to learn some software and machine learning skills to properly make use of AI technology. What I take away from this is that ChatGPT won’t be a panacea for our engineering design problems anytime soon, but it will be one more advanced tool that we need to learn how to properly use to solve our problems or we’ll be left behind.

Conclusion

Overall, I think ChatGPT will become a useful tool (if not a better author) over time.  I can see myself using it to brainstorm ideas and at least sparking my memory of details for my own writing. After all, we get ideas from reading other people’s work and then build upon them with hopefully new ideas. In essence, ChatGPT is doing the same thing. It’s building on the works of the collective internet which was also built by people. I’m fine adding my bricks to the Building of Progress where some of the bricks were added by humans and some of the bricks were added by a machine.

Oh, and for the record, I didn’t use ChatGPT for any of this article. But could you tell? And then, do you believe me?