Tag: AI Writing

Posted on by

AI-generated citations pointing to nonexistent publications

A fake citation can wear a very convincing suit.

It may have plausible authors, a plausible journal, a plausible year, a plausible title, and formatting that looks exactly like a reference copied from an academic paper. The weakness appears only when somebody tries to find the source.

That failure mode has been measured rather than merely complained about. A 2023 Scientific Reports study examined 636 citations generated in short literature reviews by GPT-3.5 and GPT-4. In that specific experiment, 55 percent of the GPT-3.5 citations and 18 percent of the GPT-4 citations referred to works the researchers could not verify as having actually been published. See Fabrication and errors in the bibliographic citations generated by ChatGPT.

Those numbers should not be treated as timeless error rates for every later model. They describe particular models, prompts, and tasks. What they demonstrate is the mechanism: fluent text generation can produce the shape of scholarship without the underlying publication.

Formatting is cheap; existence is the hard part

Bibliographic references are unusually easy for language models to imitate because they are highly patterned.

Author names. Year. Article title. Journal. Volume. Issue. Pages. DOI.

A model can assemble those pieces into something statistically convincing even when no database record sits underneath them. Real journals and real researchers can even be combined into an imaginary paper, which makes casual inspection less useful.

The mistake becomes dangerous when later writers copy the fabricated reference without checking it. A false citation can migrate from an AI answer into a blog post, report, student paper, reference manager, or another model’s training material. Repetition then makes the nonexistent work look increasingly established.

Verification means following the reference

The practical test is boring and effective: try to locate the original publication.

Search the journal or publisher. Resolve the DOI. Check Crossref, PubMed, a library catalog, or the relevant scholarly database. Confirm that the title, authors, year, and publication details actually match.

Even a real paper can be misrepresented, so verification should not stop at proving the paper exists. The source also has to support the claim being attached to it.

This is where synthetic citations fit Dead Internet Theory unusually well. The problem is not simply that a machine made a mistake. It is that the web can acquire references to intellectual objects that never existed, and those references can then circulate like ordinary scholarship.

A citation is supposed to point backward to evidence.

When the pointer leads nowhere, the polished formatting is just scenery.

Posted on by

AI-written book summaries without reliable source grounding

A summary has one job that matters more than elegance: remain faithful to the thing being summarized.

That sounds obvious, but generative systems make it easy to produce a polished account of a book without showing where any particular claim came from. A paragraph can sound exactly like literary criticism while quietly inserting an event, motive, quotation, or theme that is not actually in the text.

The problem is not unique to books. Researchers studying abstractive summarization have documented a general phenomenon usually called hallucination: generated summaries can contain information that is not supported by the source. An ACL 2022 paper, Hallucinated but Factual!, examined precisely this problem and distinguished unsupported additions that happen to be true from additions that are not.

For a book summary, even a true addition can still be misleading if the task is supposed to describe what the book itself says.

A summary is a claim about a source

Suppose a generated summary says a novel is primarily about guilt after war. Maybe that is a reasonable interpretation. Maybe the book never addresses war at all and the model has blended it with another title. The sentence alone does not tell you which happened.

The same problem appears with nonfiction. A summary may attribute an argument to an author because the argument is common in books on the subject, not because it appears in that particular book.

This is why source grounding matters.

A better workflow gives the summarizer the actual text, chapter excerpts, notes, or a reliable edition and preserves a route back to the source. Important claims can be tied to chapter numbers, page ranges, quotations, or at least clearly identified sections. The more specific the summary becomes, the more useful those anchors are.

Confidence is not traceability

Readers often use summaries because they have not read the original. That makes the summary unusually powerful: there may be no immediate contradiction available in the reader’s own memory.

A generated synopsis can therefore create a strange secondhand literature in which thousands of people know what a book supposedly says without anyone checking the book.

AI can be helpful here. It can compress chapters, compare sections, extract recurring concepts, or turn notes into a study guide. But the useful version is grounded in the source rather than merely sounding like somebody who read it.

A book summary should leave a trail back to the book.

Otherwise the internet gains one more confident description of a source that nobody in the publishing chain can prove was actually followed.

Posted on by

Machine-written product descriptions across vast retail catalogs

A catalog with twenty products can be written by hand. A catalog with twenty thousand products creates a different incentive.

Every item needs a title, description, features, materials, compatibility notes, dimensions, care instructions, and search-friendly language. That is exactly the kind of repetitive work automated text generation can accelerate. The danger begins when the system stops rephrasing supplied facts and starts filling gaps with plausible ones.

Shopify’s own documentation for its AI product-description feature, Shopify Magic, makes the distinction unusually clear. Merchants can provide a title and a few keywords, then generate a complete description. But Shopify also warns that generated copy can introduce product benefits or facts that the merchant never supplied, including details borrowed from similar products. Its guidance says merchants remain responsible for the accuracy of what they publish and should review generated text closely. See Shopify’s documentation on automatically generating product descriptions.

That warning gets more important as the catalog grows.

Plausible specifications are still invented specifications

A language model is very good at knowing what a product description usually sounds like. If the item is a jacket, the copy may naturally mention weather resistance. If it is a cable, the model may invent compatibility language. If it is a kitchen tool, it may add claims about dishwasher safety or materials because those details are common in similar listings.

The prose can sound more complete than the underlying record.

That creates a subtle reversal. Instead of the description being a readable version of verified product data, the description becomes a source of new claims that somebody now has to investigate after the fact.

The Federal Trade Commission’s general advertising guidance is boring but useful here: advertisers are responsible for express and implied claims, and material claims need a reasonable basis. Automation does not transfer that responsibility to the model.

The source record has to remain authoritative

The safest workflow is simple. Structured product data comes first: manufacturer specifications, measured dimensions, tested compatibility, ingredients, materials, warranty terms, and other facts that can be checked. Generated copy can then reorganize those facts into readable prose.

When the generated description adds something not present in the source record, it should be treated as an unverified suggestion, not as a discovered fact.

This matters because scale changes the consequences. One invented sentence on one listing is a correction. One invented attribute propagated across ten thousand SKUs becomes a catalog-level data problem.

Machine-written product descriptions are not inherently deceptive. They are a publishing tool. The problem begins when the smoothness of the language hides the difference between information supplied by the merchant and information guessed by the machine.

A huge catalog can be automated. Responsibility cannot.