michael-dean-k/

A fiction plot came to me in the car: an ASI constructs an airtight waterproof bubble around a town, and everyone is puzzled why, until suddenly it usheeschatrs in a Biblical flood that kills everyone in the world, except the people inside the bubble. They choose this town because someone inside of it was determined to be "the supreme human," a genetic and moral code that is exemplary of how all humans should be and live. It turns out it was just a regular guy who said "please" and "thank you" to this chatbots, a kind of "reverse sycophant." We find out, in a very Vince Vaughn-esque apocalyptic romcom, that he's a mediocre fallible guy, but more remarkably, also immune to the crooning and praise from both his neighbors and overlords. He has every opportunity to step into the role of messiah, but would really rather not, and instead continue his pre-flood existence.

The issue with AI chatbot dependency might be that people are outsourcing their judgment.

"Feedback skepticism,” the ability to critically reflect on external judgments, is consequential for the future. If you go to design school, you learn not to trust anyone (students, teachers, online forums). Someone might give you a helpful suggestion, but never will you blindly follow someone else's praise or suggestion, for doing so erodes your own ability to evaluate. You have to hold ambiguity, test multiple paths, and then come to that decision yourself. It probably helped that in an architecture crit, you had multiple judges, and they all have different ideas for you and argued among themselves, so there often wasn't a single source of feedback.

But these chatbots are a single source, trained to default to positive feedback, and so over time you'll feel more validated and less sure of your own opinions. The most important frame here is so view every response with skepticism, but not so much skepticism that you won't even consider it.