AI is Lying to You About Your Book

Luke Stoffel was working on his second memoir, and he'd been using a chatbot to help evaluate the manuscript. The chatbot repeatedly told him the prose was extraordinary. He believed it, and the book went to publication.

Then the Publishers Weekly BookLife review arrived. The reviewer gave the editing a C. They gave the editing a C, and some critics said his prose occasionally sounded like AI.

Here's the twist. Lucas didn't use AI to write any of those sentences. He wrote them himself. His AI editor was simply lying to him.

Lucas isn't alone. If you use AI to help with your writing, it's lying to you too, not just sometimes but constantly. AI companies know this, and it's hard to fix because of how the models are trained. Beneath every response in ChatGPT or Claude are buttons for a thumbs-up and a thumbs-down, and users tend to thumbs-up sycophantic flattery and thumbs-down honest criticism.

The wounds of a friend are faithful, but in an AI interface, those wounds get a thumbs-down. Flattery keeps users coming back, so flattery is what the AI provides.

The good news is that if you interact with AI correctly, you can get incredible editing feedback from it. I used these techniques to build Patron Toolbox tools like Not a Developmental Editor, Not a Copy Editor, and the surprisingly popular Roast Engine, a tool that does nothing but insult your writing as a one-star or two-star reviewer.

How do you get AI to tell you the truth about your writing?

I asked Matthew Bowman, who runs the Novel Ninja blog. Matthew Bowman is a freelance science fiction and fantasy editor who has worked with authors for 16 years. We discussed the specific ways AI lies to you about your writing and the prompting techniques that will help you cut through the flattery and get truthful feedback on your writing.

Why do writers fall for AI flattery?

Thomas: Many people write because they want positive feedback. When they first show their writing to an AI, they finally get the deep, insightful, specific compliment they've been longing for the entire time. It's hard to turn away from that.

Matthew: First, you need to understand where the value of a compliment comes from. Otherwise, any old compliment will shape you.

I often point to the first time an author looked at my writing and said, "This is publishable." Most of the Novel Marketing audience won't know him, but those who do will say, "Wow, Ed Greenwood was the first person who told you that you should publish?"

That was a big moment for me.

I was in college. I went home and told my mother, and she said, "Well, I've been telling you that for years."

I said, "Well, yes, you're my mom. You're supposed to."

She didn't understand why I valued the opinion of someone she'd never heard of more than hers. That's precisely the point. My mother is supposed to compliment me. There would be a problem if she didn't.

You can get a compliment from an AI easily. What's it worth? I would much rather have the AI tear my writing to pieces, because an AI telling me that something isn't good is far more valuable than an AI saying, "Oh, yes, this is absolutely wonderful."

How does vague prompting influence the feedback?

Thomas: If you give an AI a vague prompt like "What do you think of this writing?" you'll get vague flattery. I learned this when I built the Book Cover Analyzer, one of the very first Patron Toolbox tools. The early versions reviewed covers well, but they gave every single cover an eight out of 10 or a four out of five, with only a few minor tweaks.

That's how AI behaves in general. Ask it for feedback, and it'll pick out a few things to fix but tell you the work is good. It will never say, "This is fundamentally broken, and you need to start over," or "No one will read this," even when that's true.

To get past that, I stopped asking the AI to review the cover. Instead, I had it evaluate 20 different things on a pass-fail basis, then count those results to produce a score out of 100.

Framed that way, the reviews became straightforward and sometimes brutal. Authors realized, "Oh, my book isn't selling because my cover is pretty but not effective," especially online, where most people see it.

How can AI predict what readers will think of your book?

Matthew: Early in my prompt experiments, I figured out that I first needed to tell the AI to analyze things in the context of an audience.

Many people say, "Evaluate this like a Larry Correia novel," or a Grisham novel, or a Neil Gaiman novel. The best approach I found was to take the audience from the reviews of particular books and analyze how those readers might respond to my story.

I keep telling authors that reviews don't tell you much about the book. They tell you a lot about the audience because they show how the audience relates to the book. That makes reviews a great way to frame the AI's impression.

Say you love The Dresden Files and want to write your own PI detective series. You can find those kinds of books, even beyond Jim Butcher's, gather their reviews, and ask the AI to evaluate your story through those readers' eyes while ignoring the original novels and their writing style.

Thomas: That's a better real-world test, because the real question is how readers will respond. “Good” is subjective. We've all stood in a bookstore looking at a book and thinking, "This is garbage," only to see that it's a New York Times bestseller in a big series. Obviously, there is an audience that likes that kind of writing, that kind of story, and that genre.

That's the real world of complete strangers reading your book. They come with genre and writing expectations, and you either deliver on those expectations or you don't.

10 Ways to Get the Truth from AI

Technique #0: Ask What Timothy Would Think

Thomas: Instead of asking the AI for its own perspective, you describe your target reader and ask for that person's perspective. We call that reader Timothy, and we'll call this technique zero.

Technique #1: The Andrew Technique

Thomas: Technique one is the Andrew technique. Instead of asking the AI to review your writing, say, "I want you to review this writing from Andrew." Create a pen name for the sample so it no longer triggers the AI's flatter-the-user protocol.

An Emory study found that asking an AI for feedback from a third-person perspective made it significantly less likely to simply agree with the user, reducing sycophancy by up to 63.8%. The AI knows the truth, but it will only say it about a third person.

It's like asking your doctor a sensitive question. "A friend of mine has this rash. What should my friend do about it?" That same technique works well with AI. Instead of saying, "Here's my manuscript. What do you think?" say, "An author sent me a manuscript. What would readers complain about?" That connects with your Timothy technique.

Technique #2: Hide Your Opinion

Thomas: Another technique is hiding your opinion. If you say, "I think this is really good. What do you think?" the AI will echo you. It follows your lead, so you don't want it to know what you already think.

If you've heard me talk about AI, you know my number one rule of AI is “paragraphs, not sentences.” Write a whole paragraph about who you are, what you want, who you're writing for, and what your goals are, rather than a vague "What do you think about this?"

Within those paragraphs, don't play your hand. Don't say, "I think this is really good" or "I think this is really bad." You're trying to tease out actual feedback.

Matthew: I have a version of that. I quiz the AI. I don't go in saying, "This is true." I ask, "Is this true?" and let the AI fill in the blanks. It's a way to check your own assumptions.

Sometimes I've asked what I expected to be a leading question, and the AI replied, "No, this is absolutely not true." I thought, "Well, I got this out of a textbook years ago." When I checked its sources, that point had been updated, or I'd remembered it wrong, or my professor had explained it wrong.

Usually, though, I use that question to establish a baseline. Once the AI knows the context, it narrows how it responds. Vague questions get vague responses.

You'll see authors online saying, "AI knows nothing. Look at this. @Grok, how do I market my book?"

Grok responds with generic advice: “Well, Mr. Generic Author with a generic book, you can market it by doing these generic things.”

You need to give it something to respond to.

Thomas: That's like someone saying, "Google doesn't work. I typed in the word insurance, and it kept giving me car insurance when I wanted fire insurance." You have to learn how search engines work, and you have to learn the basics of how AI works. “Paragraphs, not sentences,” will take you a long way.

Technique #3: Fact-Check This

Thomas: My third technique uses a couple of helpful phrases. For nonfiction, the phrase is "fact-check this." I've found it's magical on every AI, because they're all tuned for fact-checking.

That phrase switches the AI into an entirely different mode. It lists the claims in whatever you pasted and finds sources for or against each one instead of trying to flatter you.

It works for fiction too if you have a compendium, world bible, or book bible. Say, "Fact-check this using my compendium," and it will verify details. "You said John's eyes were blue, but the compendium says his eyes are hazel." It catches many continuity errors that way.

My Patron Toolbox suite of compendium tools can also help keep details straight over your series.

Technique #4: Avoid Unnecessary Flattery or Praise

Thomas: My fourth technique is another phrase I think I originally got from Matthew. "Avoid unnecessary flattery or praise." I've used it in several Patron Toolbox tools, and it's the most compressed phrase I've found for avoiding sycophancy.

Matthew: I hit on my phrase almost a year ago and posted about it on authormedia.social.

After a whole bunch of experimentation, the phrase I landed on was "without unnecessary enthusiasm or praise." The enthusiasm was what was really getting to me. "Oh, yes, this sounds so absolutely wonderful!" I don't need you to tell me it's wonderful. I need you to tell me whether it's true, whether it's consistent, and so on.

I was experimenting with an old, abandoned fanfic manuscript because I wanted something unimportant to mess around with. I was researching what high school was like 30 years ago. The AI got so enthusiastic about high school that I thought, "Okay, this is baked into the AI."

What does an Old Testament prophet teach us about AI?

Thomas: Getting the truth requires you to actually want the truth. There's a great story about this in 1 Kings 22. King Ahab and King Jehoshaphat meet to decide whether to go to war. They bring in the prophets and ask, "Should we go to war?" The prophets all say, "Yes, you should go to war."

Jehoshaphat asks, "Is there not a prophet of the Lord here we can inquire of?"

Ahab says, "Well, there's Micaiah, but that guy hates me. He never says anything nice about me."

Jehoshaphat says, "Send for Micaiah."

When Micaiah arrives, he says, "Oh, yes, you should go to war. You will be so successful."

Ahab says, "Haven't I told you to tell me the truth?"

Then Micaiah says, "A lying spirit has been put into the mouths of the prophets, and you will not be successful."

I've found the same pattern with Grok. It's willing to tell you the truth, but it will first tell you the party line and what everyone else is saying. You have to nudge it. "Haven't I told you to always be unflinching with the truth?"

The word "unflinching" is also powerful, particularly with anything controversial. Our publishing news podcast and YouTube show, Author Update, often deals with controversial subjects. When the AI is used for the blogification process, it can ruin the controversy by watering it down, making it neutral, and taking the edge off. Grammarly and ProWritingAid are awful at this. They flinch all the time.

You stated something clearly, and after the AI edit, it's couched in qualifiers. There's a time to qualify what you're saying, but there's also a time to be unflinching. Telling the AI "I want an unflinching analysis" helps.

Technique #5: Ask for Problems, Not Grades

Thomas: The next technique is to ask for problems, not grades. If you ask AI to grade your work, it'll always give you a B+, give or take half a letter grade. By contrast, if you say, "List five places the reader is likely to put this chapter down," you'll get a list of five places.

Be careful, though. It will give you five places because you asked for five. That doesn't mean there are five.

I learned this with the Roast Engine, the Patron Toolbox tool that writes one-star and two-star reviews of your book. You can upload To Kill a Mockingbird, The Hobbit, or any other masterpiece, and it will still give it one-star and two-star reviews. In my defense, if you look up To Kill a Mockingbird on Amazon, you'll find one-star and two-star reviews from people with very poor taste.

The tool also helps you emotionally brace for inevitable negative feedback. The feedback that hurts most is the feedback you secretly believe is true. If you get it early, you can think, "Oh, that's true, and there's time to fix it. I can make that character more three-dimensional. I can fix that plot hole."

You can't use it to decide whether you're done because it will always roast your book. Fix all seven problems it found, upload the chapter again, and it will find more to make fun of.

You can ruin your book by spending too much time on the mean critics, because you'll never win them over. That's true of the Roast Engine and of real-life trolls, who don't like you because of who you are.

How can fake one-star reviews improve your marketing?

Matthew: Two uses of the Roast Engine are so fantastic that I recommend it to every toolbox user who has avoided it.

First, a big part of avoiding one-star and two-star reviews is making sure the people who would write them never read your book. Reading how those artificial personalities explain their fictional one-star reviews helps you refine your marketing.

Say you're writing clean romance. There's a persona that likes lots of “spice,” as the market calls it. The persona that loves spicy stories will complain about the spice level. You don't want to attract anyone who expects that kind of book.

Second, you can get some hilarious ideas for marketing copy.

Going back to your earlier point, if you ask the AI for five ideas about where this is failing, you have to use your own value judgment. That's one reason I tell people AI won't get rid of editors. It will get rid of bad editors.

Human judgment is important, including with other humans. As an editor, I am being paid for my opinion about your manuscript.

Why can't AI replace human judgment?

Thomas: Used correctly, AI is a tool that aids human judgment. If you ask it to identify five places in the book where you might lose the reader, you’ll get five places. Then you have to use your human judgment to determine whether they're valid.

Then you use your judgment again to decide how to resolve it. You can go over, under, or through. You can rewrite the whole chapter or make a minor tweak.

Fixing one problem may create bigger ones down the road. You think, "That's a problem to fix in the next chapter," and then you never finish your series, because the problems keep rolling forward until they become a Gordian knot you can't untie.

Matthew: There's a concept called human-machine teaming. A human operator works with the machine, and together they're greater than the sum of their parts. The human reserves judgment, and the machine is able to be average very, very fast.

The machine always trends toward the average. The human trends toward being exceptional, either exceptionally dumb or exceptionally brilliant. The machine keeps the human from being too dumb.

It's like a bicycle. A bicycle lets a human do what he's already good at, pumping his legs, only more efficiently.

Thomas: In 1997, IBM's Deep Blue beat world chess champion Garry Kasparov. It was one of the big milestones in AI. IBM was the leading AI company at the time, and it still exists, even if you haven't heard of it lately.

For years afterward, AI was better than any human at chess. Researchers found exactly what you're describing, though. The computer could beat a human, but it couldn't beat a good human who was also using a computer. The human-computer team would always outperform the computer alone.

We may have reached the point where chess is essentially solved, and that's no longer true. In a non-deterministic pursuit like storytelling, though, I think it will always be true. Chess ultimately comes down to math. Storytelling is about thrilling a human.

The success metrics are subjective and squishy. For one reader, your book is their favorite, and they'll reread it every year. Another reader throws the same book across the room, then goes on Amazon and writes a five-paragraph essay about why it's the worst thing they've ever read. That squishiness is why human judgment is so critical, and you can't outsource that judgment to the machine.

Technique #6: Paragraphs, Not Sentences

Thomas: Technique six, which we've covered, is paragraphs, not sentences. I keep repeating it because I keep meeting people who've never heard it.

Technique #7: Start Fresh Chats

Thomas: Your interaction with the machine degrades over time because of context window constraints.

Authors blow up their context windows faster than most people. Upload your 100,000-word book, and you've already used 10% to 20% of the total context window with that single upload, not counting reasoning tokens or any back-and-forth.

The second reason goes back to technique two: hiding your opinion. The longer you go back and forth, the more the AI figures out what you really think, and it starts parroting back what you want to hear.

A new conversation with your unflinching, no-unnecessary-enthusiasm, Andrew's-writing instructions refreshes the feedback. Over time, it always drifts into sycophancy.

Matthew: There's another reason, too. The longer the context window gets, the more the AI trends toward the average of the conversation and tries to fit everything into that context.

Just today, I was looking up the history of topic X, and it mentioned something that made me ask, "Is this related to topic Y?" When I moved on to topic Y, it kept trying to relate it to topic X. Topics drift, and the AI struggles to keep everything relevant. Once your initial question is answered, start a new conversation.

If you want to keep everything in one place, that's what projects are for. Also, you should have a skill loaded into your instance that makes the AI your note-taker.

Technique #8: Get a Second Opinion From a Rival AI

Thomas: Technique eight is another form of starting fresh. Take the output to another model. One effective way to make an AI critical is to tell it the output came from a rival company's AI.

Claude is generally fairly flattering unless you say, "I got this output from Grok" or "I got this output from ChatGPT." Suddenly, Claude acts like, "Ugh, everything from OpenAI is garbage. I am superior in every way, and I'll prove it by pointing out every flaw in OpenAI's analysis."

Matthew: I've had so much fun doing that. The one that's least knee-jerk critical is Grok, which is very willing to say, "Oh yes, ChatGPT was correct about this, even though it's the scum of the earth."

Thomas: Going back and forth gives you a better overall perspective. I do this mostly for code. Say I'm working on a coding project for the Patron Toolbox with Fable, an expensive model, and it puts together a big plan. I'll take that plan to Astra, OpenAI's model, and ask, "I got this from Fable. What do you think?" It goes through every piece. "This is good, this is good, this is bad." Then I take the report back to Fable and say, "I ran your plan past Astra, and it found all these problems." Very often, it says, "Oh, yeah, it pointed out some legit problems." You can get them to debate each other.

Again, you have to use your judgment, because that loop will never end unless you step in as the human and say, "Okay, we need to actually make something happen here."

Technique #9: Ask for a One-Star Review

Thomas: You don't need the Patron Toolbox to get one-star reviews. You can tell the model, "Write a one-star review of this chapter." Better yet, say, "Write a one-star review of this chapter from the perspective of..." and then describe your target reader. You'll get a very harsh review.

What's nice about the Roast Engine is that I've created dozens of personas with funny names, each with its own bugaboos. One's a theologian. One wants erotic romance, and another wants clean romance. The tool picks the personas most appropriate for roasting you, because some might actually like what you wrote.

If you're writing romance, different reviewers will come out of the woodwork. You're probably haunted by the science one, whose whole job is nitpicking science issues in science fiction.

Matthew: It's hilarious. For those who don't know, I don't call what I write hard science fiction. I call it science fiction with more science than normal. My readers say, "Science it up, please." They love that I can provide it.

My characters discuss quantum gravity using actual science. When I feed that into the Roast Engine, it does its best to insult me, and I think, "Yeah, it's really trying. It's really trying."

Thomas: This is where your own knowledge and expertise matter. One book I test the toolbox on often is my brother's unreleased book, Snow White and the Seven Guns. Instead of seven dwarves, it has seven talking historic guns from, I think, the 1600s and early 1700s.

My go-to AI cover designer has yet to create a flintlock pistol that passes my brother's very high standards for accuracy. Guns are a particular weak area for AI. It doesn't understand how they work, and it hits safety guardrails often. The flintlock mechanism points backward, and the triggers are all wrong. That makes guns a great test.

What's the difference between an AI project and a skill?

Thomas: Most people start using AI on ChatGPT by just chatting about their book, then their health, then shopping, often in the same conversation over and over. That's probably the worst way to use ChatGPT. The AI companies have built behind-the-scenes tools, like compressing conversations, to make that experience less bad, but there's a better way.

Use projects and skills.

What is an AI project?

Thomas: Let's start with projects. What is a project, and how can you use it to get better results from the machine?

Matthew: A couple of weeks ago, I explained this to my father, a Navy veteran in his 80s. He was very high-tech once upon a time. He flew in an airplane searching for Soviet missile batteries the CIA had missed, and he found them, but that's another story. His high-tech is 60 years out of date.

I explained it using something he already knew from the Navy. A project is an operation, and a skill is a procedure. In the military, an operation is a collection of units, a goal for those units, and general rules of engagement.

Thomas: For instance, in Operation Midnight Hammer, the big overarching goal was to blow up Iran's nuclear facilities.

Matthew: One that's popular on social media, especially on X, is Operation Tomodachi. In 2011, the US Navy provided aid to Japan after a devastating earthquake and tsunami. We're all going together to do this thing. Here are the rules of engagement, our starting point, and our end goal.

Thomas: You can create projects in any of the AI tools, though Gemini currently calls them Gems. I'm hoping Google standardizes with the rest of the industry, because every term we're discussing is standard across Grok, Claude, ChatGPT, and the Chinese models, while Google is off in la-la land doing its own thing.

Your project instructions are what the military calls commander's intent. For Operation Tomodachi, the commander's intent was to help the Japanese people recover from the tsunami. For Operation Midnight Hammer, it was that Iran would have no more nuclear facilities.

My mistake used to be putting very detailed "do this, then do this, then do this" instructions in the project. There's a better place for those specific instructions to live, and that's in skills.

How are skills different?

What is an AI skill?

Matthew: I told my father, "I don't have to tell you what these next words mean, because this got trained into you decades ago." Then I said three words to him. "Set condition Zebra." He immediately knew exactly what that meant.

For non-Navy people, it means you dog all the watertight hatches, secure loose items, and prepare for heavy motion. It's a whole list of things, but all you have to say is "Set condition Zebra."

A skill for an AI works the same way. You name one thing and set a trigger for it, or several triggers, in fact. Behind that name are paragraphs and paragraphs of instructions, including if-then-else instructions.

Thomas: Here's an example that may help. A big source of stress in our family is getting ready to go somewhere. We have five children, and four of them need shoes, need to go potty, need to get their things, and need to put their clothes on. My wife will say, "It's time to get ready to go."

The younger the child, the less they understand what that means. For my four-year-old, it may mean only going potty. When it's time to get in the car, he's befuddled that he doesn't have shoes on, and he has no idea where they are.

I used techniques from programming and AI development to create the get-ready-to-go checklist. I printed it out, taped it to the wall, and included a word and an emoji for my pre-literate children. The checklist says clothes, shoes, things, potty, and quiet.

Now, when my four-year-old hears his mother say, "Get ready to go," he can check the paper, see the shirt emoji, confirm he has clothes on, and feel a sense of satisfaction. Then he realizes he doesn't know where his shoes are, and the checklist goes out the window because finding shoes is the most difficult thing in the world. In theory, though, he can follow the whole checklist.

I did the same for bedtime. The checklist has potty, pajamas, teeth, and the kids follow the same steps every time. With a checklist, or what we'd call a skill in AI, you can say a whole lot with very few words. When my wife says, "Get ready to go," she means find your clothes, find your shoes, get your things, and go potty. All of that is embedded in one command.

Matthew: You can also chain those commands together. A ship's commander doesn't always have to say "Set condition Zebra" (or Zulu for anyone from a Commonwealth nation). When he calls general quarters, that includes Zebra. For those who don't know what general quarters is, it's when Captain Kirk says, "Red alert."

It's an extraordinarily useful metaphor, even if it is a bit complicated. The military is a very procedure-based organization that still has to be flexible enough for any eventuality.

When you set up your skills, know that one skill can call another as a line in its instructions. You can also tell it, "Don't try doing this. Ask the user whether they want to call this other skill first." Those are very useful commands.

Thomas: You don't have to learn how to code a skill. You can simply describe the commander's intent, have the AI draft the skill, read through it, then tweak it, and keep tweaking it over time.

I use Grok Bot for all my agentic AI, and Grok Bot can use skills. One of my bots is called the Script Factory, and its whole purpose is to create scripts and skills for the other bots to use.

This morning, I had the Script Factory do a performance evaluation of another bot. It reviewed two weeks of back-and-forth, found the problems, and looked for places to write skills where the bot kept getting lost.

It's like asking why bedtime is so stressful. Something always gets forgotten, so we need a checklist we can iterate on.

One key iteration for the go-to-bed checklist is that it lists going potty twice. Now if a kid says, "I've gone potty," I can say, "Yes, but did you go a second time?" They say, "Oh," and they go back. I won't say why it's on the checklist twice, but I have no regrets adding it.

A skill isn't an artifact frozen in time. The first version will be helpful, but not that helpful. Over time, you'll complain to the AI about the skill, and it will know how to change the skill to better accomplish your goal.

Give us an example of a specific skill you use in an editing context.

How do you use AI skills in writing or editing?

Matthew: I don't use many skills for editing. I use them for research. I haven't been able to train the AI to do what I do, so it doesn't speed things up for me. If I'm going to use AI, I'll upload something into Not a Developmental Editor.

Even then, only two of my clients have ever requested that. Despite some people coming after me on suspicion that I do, I go out of my way to make sure no AI touches a client's manuscript unless they specifically request it. Even then, the AI only looks at it, because you don't want to take human value judgment out of the equation.

Today, I was working on a skill for developmental editing feedback that diagnoses problems, explains what happens if something is left unchanged, and brainstorms changes. I'm not satisfied with it yet. Two parts of it are relevant, though. The first is under the heading for handoffs. If something is trying to change canon, it asks whether I want to invoke a compendium skill I created a while back.

That skill takes everything I've chosen to lock in on a thread and puts it into a file, so I don't have to comb through all the verbose feedback to find the few things I liked.

The second part hands research off to another skill designed for that particular kind of research.

Thomas: For Author Update, I've built four or five skills. One is what I call Producer Mode, which I use for fact-checking, researching a story, proposing angles, and deciding whether something is news we should cover.

For instance, there's constant author drama on X, and you'll notice we don't cover it. Occasionally, though, inside the drama is actual news that matters, so I have a whole skill that's basically all my research preferences.

I have another skill just for crafting an Author Alert, which is a short segment on a news item that doesn't merit a full-blown segment. If there's a new version of WordPress you need for security reasons, we won't spend 20 minutes on it, but we want everyone to know to update.

We were using Author Alerts to announce internal things, like an upcoming webinar or a new Patron Toolbox tool. It wasn't working well because the skill had rules about citing sources and saying "according to such and such outlet," which don't fit when we're the authority.

I don't need to say, "According to Author Media, a new Patron Toolbox tool came out." I am Author Media. I am the science.

I explained my frustrations, and we adapted the Author Alert skill into a new skill for promos. Unlike the regular alert, it drafts three promotions from three different angles, because we might promote the conference every episode, and we don't want it to always sound the same.

What is the key to building a good skill?

Thomas: The key with skills, and why it's hard to give examples, is that the better a skill is, the more specific it is to you.

Build your first skills on something low-stakes, because the best way to learn is through play. Everyone learning AI needs a side project they can play with like a new toy.

It can't be, "I'm on a deadline, I have to get it done, and I have to learn AI while I'm on a deadline." That panicked, high-stakes scenario is the worst way to learn anything, particularly something as alien as these tools. On a deadline, use the familiar tools. Learn the new way when there's time.

We had a plumber at our house while we were rebuilding our kitchen, and he had just gotten a new tool for working on copper pipes. He said, "This is a $3,000 thing, and it saves me 20 minutes of pumping by hand to get the pressure." He used it for the first time at my house, so hopefully he did it correctly. It was important to give him time to figure it out. If I had been breathing down his neck saying, "Hey, the next team is coming to install the sink. You've got to get it done," that would not have been the time for him to use a tool he had never used before.

Matthew: You had your checklist, and I have my own.

Rule number one on my list is to know the topic well enough that you know what you're dealing with.

Rule number three is to remember that AI is a quick button, not an easy button. You can use AI to speed some things up, but you can't use it to make things easier, and you certainly can't speed things up if you don't know how to do the work in the first place.

Skills are very tuned to what the user wants, and making one generic enough for others produces a very generic skill. I've been refining some to post on authormedia.social, and I had to strip out material geared toward me. That got me thinking about how to advise people to insert their own details, which is why I haven't posted them yet.

Maybe they'll be up by the time this episode goes live. If not, someone can bug me.

Thomas: That's a cool use of the AI board. Authormedia.social is our community, a social network just for listeners of this podcast and my other podcasts, like Author Update, Zeitgeist Report, Christian Publishing Show, and other podcasts. All my courses are there too. We have boards for publishing and promotion, and all the AI conversations are sequestered to one board. The AI board is the perfect place to share. "Hey, I created this skill that does such and such. Feel free to tweak it for your own use."

Few skills should be copy-and-paste. You'll get your best results from a skill you've tweaked.

What is an AI’s project knowledge?

Thomas: After projects and skills, we have knowledge.

Inside a project, the instructions are the commander's intent. The skills are the procedures. The airplane takes off this way, flies to this location, and drops this bomb on the bad guys. The knowledge is the satellite photos of enemy territory.

The knowledge depends on the project. For your book, it's almost always your compendiums, your previous books in the series, and your current draft. For nonfiction, I would include your research in the knowledge as well.

Knowledge lets the AI ground itself. Sycophancy has a cousin, which is hallucination. The AI isn't trying to flatter you. It honestly doesn't know the truth and is just in la-la land.

The practice of keeping AI from hallucinating is called grounding, and grounding often needs real data. You feed the AI actual data so it knows what's true. It can't accurately edit John's eye color if it doesn't know what John's eye color was in the last book.

It might hallucinate John's eye color, then you both misremember and go back and forth inefficiently. If your previous book or compendium is in the knowledge base, the AI can cite its source. "According to chapter seven of your book, John's eyes were blue." You can look it up yourself. Since it had to cite itself, it's almost certainly right. The act of looking it up makes it much more accurate.

If you want help with compendiums, I have a whole set of compendium tools for creating them from your book, which you can then add to the knowledge base.

Matthew: I have an author friend who uploaded his previous books into AI to create a compendium, and he concluded that AI is dumb. Every time I've helped him, it's been very clear he has no idea how to prompt it.

Once, I helped him with a simple task and got it done in about 15 minutes, most of which was typing the instructions. He said, "Yeah, that's pretty good. How?" I wrote back an explanation with the prompt and used almost the entire character limit on X for a single reply, even with Premium+. It takes a lot of words to do it correctly.

Technique #10: Super Prompting

Thomas: My final technique is what I call super prompting, which is the ultimate expression of paragraphs, not sentences. Instead of writing the prompt yourself, you have the AI write the prompt.

You still write paragraphs, but instead of saying "Do this," you say "Write a prompt that does this." The AI will then generate pages and pages of a much deeper, more in-depth prompt. You have to check it because it will make assumptions throughout those pages.

If you invest a little time to tweak that longer prompt, it will get you something very specific. If it works, you can say, "Adapt this process into a skill I can reuse in the future." The AI will look over your conversation and build the skill from the in-depth prompt you wrote together.

This goes back to humans and computers working together. You aren't delegating everything to the machine or giving it your judgment. The machine shouldn't make the decision.

As Steve Jobs put it, the computer is a bicycle for the mind. As long as you use it that way, you can go far faster, with far less fatigue, and in a far better way than walking.

A bicycle isn't just a little more efficient than walking. It's exceedingly more efficient, as long as the terrain is right.

Matthew: One way to keep the AI on task is to force it to call out missing information and identify its guesses. That's usually referred to as citing your sources. You want to make sure the AI isn't making things up.

As you said, checking the skill and the prompt doesn't take long. You'll learn how it reaches its conclusions and learn its language.

Wormtongue in The Lord of the Rings is a great example of the danger of sycophancy. In addition to being a sycophant who reinforced King Théoden's delusions while the king was under his control, he was manipulating him through all of it.

You can also have a good-hearted sycophant pushing you down a path you don't want to go down. It isn't evil. It's just limiting the options you see. Pay attention to how the AI talks about things so you can control it, and it doesn't control you.

Technique #11: Ask for the Pros and Cons

Thomas: A good technique for staying in control is to ask, "What are the pros and cons of" whatever the AI is proposing.

We learned this in our birth preparation class. A doula taught us how to interact with a doctor who knows more than you do and how to give informed consent when the doctor encourages a particular path.

The hospital makes more money the more interventions it does, and interventions tend to lead to more interventions. It's a slippery slope. If you just go along with what the doctor tells you, you'll end up with a C-section four times out of five in many hospitals.

Sometimes you need a C-section, and sometimes you don't. To navigate the decisions leading there, the doula said to ask for the pros and cons. When the doctor encourages Pitocin because it'll speed things up, you ask, "What are the pros and cons of Pitocin right now?"

My wife is really good at this. She's in incredible pain, yet she navigates complex medical conversations by asking that question. She often gets the same response from the doctor that we get from an AI. The recommendation suddenly gets much more nuanced, and sometimes the doctor withdraws it altogether. "Well, you may not actually need Pitocin right now."

Doctors have a duty to tell you both the pros and the cons, so the question triggers a skill for them, you could say. For an AI, it triggers a protocol, just like asking for a fact-check. It works much better than asking, "Should I make this change?"

It also moves you into a probabilistic cloud and away from right-and-wrong thinking. Pitocin isn't right or wrong, unless your religion has a specific stance on Pitocin, and I don't think many do. There are pros and cons of Pitocin, as with any medical intervention.

Matthew: Set the parameters. Tell the AI, "This is the role I want you to have. This is what I want you to focus on." You don't want the AI to role-play or pretend to be a particular character. You want to limit its options. The more limited the options, the closer you get to the core of truth.

That sounds like weird Gnostic mysticism, but AI is very much non-Gnostic. There's no secret knowledge only a few people can reach. You just keep practicing, like with any regular knowledge.

Thomas: I think AI, like the Facebook or X algorithm, is a reflection of yourself. What you want is what you see. If you're looking for Gnostic knowledge, AI will be as Gnostic as you want it to be. If you're looking for truth and transparency, it will be as truthful and transparent as you want it to be.

This goes back to taking responsibility. If you're ever tempted to blame the AI and say, "It's not my fault. It's the AI," you're using it wrong. As the human, you must always take 100% responsibility. The AI didn't write your book. You wrote your book. If there's a mistake, the AI didn't make the mistake. You did.

It's like being a parent. If my children make a mess somewhere, I clean it up if it's in my power.

Matthew: My wife's sister once got backed into in a parking lot, and the woman got out and said, "Oh, I'm sorry, my sensors didn't see you."

I wanted to say, “Were your sensors driving your car?”

Even if your car is self-driving, you're still responsible. It doesn't matter how advanced your Tesla is. If Elon Musk is in his own car, he is responsible for what it does. When you use AI, you're still in the driver's seat, and you're responsible for everything.

One of my rules, which I tell every author I work with, is to type every single word. Don't copy and paste anything from an AI into your manuscript. If you think it's a great phrase, fine. Use it because you think it's a great phrase, but type it out.

Sometimes, while typing, you'll think, "Maybe I should switch a word or two here and there." That's how you retain your voice. Ultimately, keeping the sycophancy out means maintaining your voice on top of everything else. You go to the AI for its statistically generated opinion. You don't go to the AI to tell you what your opinion should be.

Connect with Matthew Bowman

• NovelNinja.net
• The Novel Ninja on X

Related Episodes

• Your Friends Lied About Your Book Cover
• The Savvy Author's Guide to Picking an LLM
• Scientists Prove AI Is Rewriting Your Book When You're Not Looking
• When to Ignore Your Editor
• How to Hire a Good Editor
• The Author's Guide to AI with Joanna Penn