In M.C. Escher's House, Maybe

Potted this on the Facebooks this morning:

The idea, of course, is a good one, at least as long as people remember to put the plates back in the right place.

What I object to is the AI presentation of this scenario.

First of all, it's pretty odd to have a blank plate to cover a box in a wall. Sure, I've got a few in my house, but a junction like this isn't likely to be set up this way -- there should be a switch. Unless this was the result of a remodel, but even then, why not a switch if you're adding new wiring?

But on to the AI.

Look at the baseboard on the floor. Then look at the open box in the wall.

The perspective does not match. Behold:

It's clear the photo with the box was taken straight on, or near enough straight on. The lines for the box don't match the perspective line for the baseboard, which is probably a reason the plate covers the bottom of the box, which is closer to the baseboard.

Also, that hole in the wall is HUGE compared to other elements in the photo. A plate ain't gonna cover that.

I guess if you're hungry for content and engagement, AI is a good way to get people to comment on the slop, and then argue about it. Either way, it's sloppy.

Harry Mudd Warned Us. He Warned Us All.

Researchers from Texas A&M, the University of Texas, and Purdue University conclude in a study that "continual pre-training on junk web text induces lasting cognitive decline in LLMs."

A few news outlets, including Gizmodo and Ars Technica, have written about the study.

Most of what I quote here on out is from Ars Technica.

The results showed that adding more “junk data” to the training sets had a statistically significant effect on the reasoning and long-context benchmarks across models. The effects were more mixed on the other benchmarks, though. For example, a 50/50 mix of “junk” and control data used for the Llama 8B model generated better scores on some benchmarks (ethical norms, high Openness, low Neuroticism, and Machiavellianism) than either “fully junk” or “fully control” training data sets.

Based on these results, the researchers warn that “heavily relying on Internet data leads LLM pre-training to the trap of content contamination.” They go on to “call for a re-examination of current data collection from the Internet and continual pre-training practices” and warn that “careful curation and quality control will be essential to prevent cumulative harms” in future models.

Gizmodo takes it a bit further:

And wouldn’t you know it, it turns out that consuming directly from the internet landfill that is X isn’t great for thinking clearly. All four models tested—Llama3 8B, Qwen2.5 7B/0.5B, Qwen3 4B—showed some forms of cognitive decline. Meta’s Llama proved the most sensitive to the junk, seeing drops in its reasoning capabilities, understanding of context, and adherence to safety standards. Interestingly, a much smaller model, Qwen 3 4B, proved more resilient, though still suffered declines. It also found that the higher the rates of bad data, the more likely a model was to slip into “no thinking” mode, failing to provide any reasoning for its answer, which was more likely to be inaccurate.

This, of course, tracks. Humans, consuming a lot of "junk" will also exhibit these traits. It all goes back to the old computer standard of "Garbage In, Garbage Out."

Harry Mudd, of course knew this all along. Once Spock showed him the way:

As usal, my friend Curtis Clark nails it:

AI is Here to Stay, Unfortunately

I think what's clear about the use of AI in the classroom is this: It's here, and it doesn't look like we can do much about it.

I say this not with a defeatist attitude, but with the bent of thinking, "We can use this to our advantage."

I think we need an assignment where we challenge students to use AI as a writing tool, but with a few caveats:

1. They have to tell us what AI tools they're using and, importantly, why they're using it.

2. They have to cross-check what AI is writing with more, ahem, traditional methods (their own research and their own writing).

3. They have to be held responsible when their cross-checking allows something erroneous to slip through.

I've been doing some extracurricular reading on AI use in the classroom, and a lot of it comes back to the fundamentals of teaching: Students don't learn from the curriculum, they learn from instructors who care. We're showing we care about the quality of work by banning the use of AI -- but clearly, based on the discussions in this group, that's not reaching all students.

We need to find out why our students are using AI, and see if there are tools and methods we can use to fill in the reasons for AI use. I think we need to drop the accusatory tones and recalibrate -- catching them and calling them out doesn't seem to be working.

Of particular interest: Stanford University researchers think they've detected bias against non-native English speakers in AI detectors. The detectors found that in their sampling, more than half of essays written by non-native speakers were incorrectly flagged as being AI-generated, while a control group of native speakers showed less bias.

Researchers recommend teachers get to know their students' writing style and ability early on so they can recognize it. That works unless they start off using AI to write.

So I don't know what to do or think, other than just blow the whole thing up.

Conclusion II: Also Inconclusive

In the interest of science (cue Dexter) I conducted a little experiment on Facebook, using my friends as guinea pigs.

As I explain in my post (linked here) I asked my friends to post a bit of writing, either from them, from a friend, or created by AI. I'd then use Edward Tian's AI language detector to do some detectoring.

Results were . . . mixed.

This experiment is hardly conclusive, but it's clear Tian's AI detector has trouble with "creative" writing (most of my friends who participated are published authors, in the publishing business, or extremely well-read). The detector consistently failed to differentiate between "creative" AI writing and human-originated writing. It got it right a few times, but more often than not, it failed.

Admittedly, some of the samples included were a bit short for the AI to chew on -- but there were bits in the middle-range of length that the detector basically said "I don't know. Not enough information here," while it confidently identified shorter texts as AI- or human-generated, both right and wrong.

What does this mean? I don't know. But it's interesting. I'll give it that.

Conclusion: Inconclusive

I finally had a few minutes to play around with Edward Tian’s AI-written text detector (explained here). The results leave me satisfied that, under the right conditions, AI-written text can be detected.

But I received enough errors from the analyzer to surmise Tian and those like him have a bit to go before the detection is reliable enough to be useful.

I tried Tian’s detector with five different bits of text:

1. Text generated by AI, which I copied and pasted into a Word document (seen here) and rearranged slightly to make a coherent essay (by rearranged, I mean I re-ordered the paragraphs. I did not alter the text at all).

2. A smaller portion of the same AI-generated text.

3. A portion of an essay I wrote a few years ago on the same subject I tasked the AI to write about.

4. An excerpt from a short story I wrote earlier this week

5. An excerpt from an essay submitted by one of my English students last semester.

Here are the results:

1. Tian’s GPTZero returned a cryptic error (see sample below).

2. Successfully identified the text as AI-generated.

3. Successfully identified the text as human-generated.

4. Successfully identified the text as human-generated.

5. Returned a cryptic error, similar to the sample already mentioned.

I suspect one source of the errors could be metadata carried over by the copy/paste from Microsoft Word, though I did get the errors when I pasted normally and when I pasted with the plain text option.

I’d like to keep conducting experiments with this website, using text I’m unfamiliar with, and certainly using text I’m unsure is AI-generated or not. I’ll keep playing with this, and see if the error rate goes down, or if I can figure out what is causing the errors. On the surface of it, I can see less-cryptic errors would be beneficial.

And with only two tests to differentiate between human and AI, there are likely ways those who want to cheat using AI can game the detector. Of the two, “perplexity,” as explained in the linked article, is a bit cryptic; I’m not exactly sure what AI finds perplexing. And “burstiness,” or variation of sentences in complexity and length, seems easy to game, though you have to suspect it might have to be humans gaming rather than AI, at least for the time being. 

While I’m interested to see where this technology goes (both in AI-generated text and in AI-text-detection, I’m more interested in the general reaction I’m seeing locally from English teachers. The consensus seems to be that we’ll eventually be able to detect it on a consistent basis, and that nothing we’re doing in our courses should change to counter AI text development.

While cheating cheaters will always cheat, I’ve seen from experience that students get more out of writing assignments they’re invested in, rather than just another run-of-the-mill ivory tower essay. But that’s just me using new technological developments to drag out the same ol’ soapbox, so maybe take what I say with a grain of salt.