If you want ChatGPT and other AIs to recommend your book, they must first know it exists. They must also know enough about it in order to recommend it to readers.
I recently released an episode on how to optimize your book’s marketing for AI. But this begs an important question. How do you tell if your AI optimization is working? What does ChatGPT actually know about your book?
Answering this question is harder than you might think.
If you ask Grok or ChatGPT about your book, if they don’t know the answer, they’ll cheat. More specifically, they will do a quick web search to get the answer that way. This is a bit like an open-book test in school. It doesn’t test the LLM’s knowledge; it only tells you what the LLM can look up.
So you may be wondering, “Isn’t searching for my book enough?”
No. It’s not.
If a reader is looking for the kind of book you wrote but doesn’t prompt ChatGPT with your specific book’s title, ChatGPT won’t know what to search for. And, if knowledge about your book is not in the training data, ChatGPT will recommend another book to that reader.
If you want AIs to recommend your book to strangers, they must first be familiar with your book deep down in their training data. To be clear, this doesn’t mean they need to have read your book. While that helps, you would be surprised how much large language models can learn from metadata, sales pages, and Goodreads reviews.
It’s true that large language models may have already read your book, which US courts recently ruled does not violate American copyright law. According to these judges, the Fair Use Doctrine has washed away AI’s original sin. To hear more about these rulings, check out my other podcast and YouTube channel, Author Update.
But just because LLMs can legally read your book doesn’t mean they have. It doesn’t even mean they know anything about it. This is particularly true since one of the judges indicated that the AI companies may have to purchase your book to use it for training.
How do you find out what the AI knows about your book?
How do you find out what the AI knows about your book and where it learned it?
One option is to use the following prompt: “Without doing a search, tell me what you know about the book [Book Title] by [Author Name].”
This prompt works some of the time. The problem is, even if you ask the AI not to do a search by toggling it off, sometimes it will still do a search. For example, when I tested this prompt on Grok, it searched the web and loaded 25 web pages before providing me with an answer.
The best way to test the AI’s knowledge is to use an API to take away the LLM’s web searching abilities. If that sounds technical, don’t worry!
I built some tools to do this for you. I designed them to get large language models to confess what they know about you, your book, and how they learned it. The tools prevent the LLMs from searching the web and can only provide answers based on the knowledge they already have. Think of this as a closed-book examination of the LLM’s context.
Let me show you how they work.
Go to the Novel Marketing Patron Toolbox
First, go to www.PatronToolbox.com. I have created dozens of tools to help authors with marketing. They cost me money every time they are used, which is why they are for Novel Marketing patrons only. If you are not a patron, you can become a patron here. If you are already a patron of the Novel Marketing Podcast, thank you! Just go to PatronToolbox.com and scroll down to the AI Knowledge Checker Section. My goal is to add a new tool every week, and there are already more than 50 tools available.
Book Page Scanner Tool
I’ve added over a dozen brand-new tools since the last time I talked about the Patron Toolbox. The Book Page Scanner is particularly helpful if you’re trying to optimize for AI. Your website is one of your best tools for informing large language models about your book.
The scanner will analyze the book page for your specific book and check if it is AI-friendly, reader-friendly, and search-friendly.
Understanding the AI Knowledge Checkers
Right now, the AI Knowledge Checker tools are at the very bottom of the toolbox. I have six of them that check the “big six” AI models. In my experience, this is more or less their ranking:
- ChatGPT and Grok are the most likely to know about your book.
- Gemini is also fairly likely to know about it.
- Claude, DeepSeek, and Llama are the least likely to know.
You’ll know you’ve done a good job with your AI optimization if all six tools are familiar with your book.
Test: AI Knowledge of Jonathan Shuerger’s Book
ChatGPT
First, we’re going to test ChatGPT with Jonathan Shuerger and his book Shades of Black and Darkness Cast.
GPT will first generate an overview of the author. It will tell you what it knows about Jonathan Shuerger, and then it will describe what it knows about the book.
One quick way to gauge how much it knows is whether it can provide the ISBN number. If it doesn’t recognize the ISBN, that’s a sign it doesn’t actually know about the book. Sometimes, the AI will hallucinate and just make up answers to these questions.
Read through the plot overview and check to make sure it isn’t hallucinating. In this case, I’d say it is hallucinating about Jonathan Shuerger’s book. The information is pretty patchy.
OpenAI’s knowledge of this book is hit or miss. But you can see that OpenAI has trained on Amazon and Goodreads, so it does have summaries from those sources.
I also coded the tool so it identifies gaps in the AI’s knowledge. It will list what it doesn’t know, and at the end, it will display a “Further Reading and Sources” section. This shows what websites the AI has used for training in the past.
Keep in mind that each AI model has a different knowledge cutoff, and they’re constantly being updated. But this still gives you a good idea of what pages to optimize.
If Jonathan wanted to optimize for OpenAI, the pages to focus on would be:
- His Amazon author page
- His Goodreads profile
- His official book page on his website
These are exactly the pages the AI is referencing.
LLaMA
Let’s see what LLaMA knows about Shades of Black and Darkness Cast and compare the results.
At the time of this recording, LLaMA is the worst-performing of all the large language models. However, Mark Zuckerberg is investing hundreds of millions of dollars and hiring top AI engineers from competitors, especially OpenAI.
In fact, as I’m recording this, there are rumors of signing bonuses over $100 million for some OpenAI folks who have joined the LLaMA team. Their work has not yet appeared in the current version of LLaMA, so we’ll have to wait and see how it evolves.
You’ll also notice that LLaMA is pretty slow. It took several minutes to produce results.
Gemini
I’ve found that Gemini is the second most knowledgeable about books. It performs well alongside Grok and GPT. Let’s see what Gemini knows about this book by Jonathan Shuerger.
Each of these tools follows the same format and first gives information about the author. Then it provides a summary of the book. You’ll want to read through all the information to see if it’s accurate.
Gemini hallucinates a little less than others, but it’s still making up quite a bit. This particular book seems to be only somewhat familiar to the AI, which actually makes it a great title to test. It sits in the middle ground. Some AI tools will say they don’t recognize the book at all, others may give a detailed summary, and some fall somewhere in between.
Different AIs perform differently depending on their training data. Also, how long your book has been published will affect results. If your book is brand new, don’t expect the AIs to be familiar with it right away, especially without allowing them to do a search.
Grok
Grok explicitly tells us it has not read the book. It says:
“While specific character names and detailed descriptions are not universally provided in public summaries to avoid spoilers…”
In other words, it’s saying it hasn’t read the book but has read reviews about it. So, its knowledge is limited to what readers have said.
It also avoids listing characters, since many reviews don’t name them specifically. But if we scroll down, it lists its sources of information:
- The author page
- The Goodreads profile
All the AIs reference Goodreads. Grok, however, also looks at X (formerly Twitter) more than other platforms. It mentions scanning Reddit’s r/Fantasy as well, though I doubt there’s much about this book there.
That’s a quick summary of what Grok knows.
Claude
Claude by Anthropic responds with:
“I apologize. I have very limited information about Jonathan Shuerger and his book Shades of Black and Darkness Cast. Here’s what I can provide…”
It basically continues to say, “I don’t have good information,” over and over. Claude is listing all the places in its memory where it looked for information and failed.
Claude doesn’t know about Jonathan Shuerger at all.
Here is the revised version of the transcript, edited for clarity, grammar, and usage. Headings and paragraph breaks have been added to improve readability while preserving the original wording as much as possible:
Compare: AI Knowledge of Larry Correia’s Book
Let’s compare those results to Larry Correia, author of Monster Hunter International. It’s a slightly older book, and it’s also more popular. It has sold millions of copies.
Claude
Claude is much more familiar with Larry Correia. It provides a detailed background of his life. It’s listing his major series, including Monster Hunter International and his other series. It even mentions the awards he’s won.
When we get down to the plot summary, Claude is not hallucinating the main characters. It accurately lists various characters in the book. That suggests that Anthropic has read the book and loaded it into its knowledge banks, or it has read enough reviews and analysis of the book from the open web to learn about it.
My guess is that it has read Monster Hunter International.
You’ll also notice it includes sections on critical reception, pulling commentary from critics and news articles. The more media coverage your book has, the better.
A Wikipedia summary of your book is especially helpful for training AI. Most authors aren’t notable enough to be included in Wikipedia. However, having a strong website can also help the AI find and understand your work.
You may find that these AIs are completely unaware of you and your book. If that’s the case, I encourage you to read or watch my episode called Does ChatGPT Recommend Your Book? It covers AI optimization for authors.
This episode goes hand in hand with that one. One tells you what to do, and the other helps you measure whether your actions are working.
Be Patient
Give yourself some time. AIs don’t update every day, so it can take months for improvements on your website to show up in tools like ChatGPT.
The AI knowledge tool is available to all Novel Marketing patrons at the Patron Toolbox level and above.
Featured Patron
Danny O’Malley, author of Lemmy & Lizzy Search for the Tastiest Fruit
When two hungry jungle friends set out to find the legendary Tastiest Fruit, they embark on a juicy adventure. This beautifully illustrated children’s book serves up important lessons about courage, trying new experiences, and finding joy in unexpected places.
Related Episodes
