Is ChatGPT for email marketing subject lines the Messiah or a fraud?

So with Intuit Mailchimp running nationwide radio advertising announcing its new subject line analytics tool, for the record we built something very similar back in 2018 and you can read about it here. Subject line optimisation and the role ChatGPT could have in it has become increasingly prevalent. Every second post on LinkedIn seems to be about ChatGPT and how it’s going to revolutionise content writing and that includes subject lines.

Having tested ChatGPT for subject line content generation exhaustively, here are my thoughts. I preface this by saying these are my thoughts, not that of the business but those of one cynic who works for the company, just in case Microsoft come after us. Needless to say my conclusion is… ChatGPT is a fraud! There I said it.

ChatGPT does not generate better subject lines for open rates. I can’t write email copy using it without having to edit it enormously. Actually taking more time on the edit than it would if I wrote my stream of consciousness!

For the purposes of this blog let’s concentrate on subject lines. I asked it for 5 subject lines for an electronics company selling TVs and it came up with the below.

The results were as follows

Now I thought these a bit “Stateside” and as I am based in the UK and wanted to compare it to UK companies’ subject lines I changed the question to reflect that…

ChatGPT thought it important to inform people that this was for UK TVs showing a scant understanding or “intelligence” of the actual requirement. Now some of you will say that perhaps I should have been smarter with my question but and here’s the rub – if I can be smart enough to ask the question in a way that ChatGPT spits out a killer set of subject lines, then I can probably write those subject lines myself way quicker. I’ve already spent more time than I’d like to have done asking the 2 questions and not getting answers I could use.

So then I took a real life situation.

I took 5 subject lines from a client which had been used in specific campaigns and asked which one it thought would be most successful. The first time I did it, it ignored the first subject line because I hadn’t put it on a separate line from the question. It then subsequently said the following…

So I asked my question again but this time put the first subject line on its own in the question. ChatGPT then contradicts itself from the first set of questions and decides that the subject line that was missed out was in fact the best but the one it previously said was the best is now not so good because it’s too generic. But it doesn’t move it down from 1st to 2nd but from 1st to 3rd!

So I asked it to rank them in order of effectiveness and it said this

What happens if you apply real intelligence instead of artificial?

Now I have real life data, I have a subject line tool which we built in house, which I can ask the above questions and get real answers. You can read about it here or Dela Quist’s SubjectLinePro or even Intuit Mailchimp’s version of our tool but ChatGPT, well I found it wasn’t useful at all. In fact it got things completely wrong.

So in order of their effectiveness in real life marketing solutions the rankings were…

1. Superb Sale savings… starting online today!

2. Our best 85” TV megadeal ever…

3. Samsung, LG TV and soundbar offers for an improved home cinema experience

4. Inflation-busting TV megadeals including a Samsung 70″ at £699…

5. Bring your TV to life 

And to remind you what ChatGPT said

1. Samsung, LG TV and soundbar offers for an improved home cinema experience
2. Inflation-busting TV megadeals including a Samsung 70″ at £699…
3. Superb Sale savings… starting online today!
4. Our best 85” TV megadeal ever…
5. Bring your TV to life

At least we all agree that last one was the least effective. So my advice is, while AI is great and ChatGPT in particular is fun to play around with I will take real intelligence over artificial intelligence any day of the week!