Large language models really seem to be everywhere nowadays.
ChatGPT is ubiquitous. Every Google product has built-in Gemini integration. Bing search leads to a response from Microsoft Copilot. Meta AI is baked into Facebook and Instagram to the point where it is difficult to avoid triggering it by accident. X (formerly Twitter) is full of controversy surrounding Grok. News stories abound about misleading financial, medical and legal advice generated by chatbots. “Vibe coding” is extremely popular. Apps like Sora show feeds of nothing but AI-generated videos. Some skeptics believe the entire United States economy seems to be held up by just a few companies spending exorbitant quantities of cash on graphics processing units (GPUs) for data centers. “Slop” was Merriam-Webster’s Word of the Year for 2025.
Yet through this all, when logged in to your Brighton account, something is different: the Gemini icon does not appear anywhere, and attempts to visit Google Labs to try out new artificial intelligence (AI) features lead to a “We are sorry, but you do not have access to Early Access Apps” message. Why?
This may be an anti-cheating measure, as generative AI is often abused by students, but it could also have to do with the fact that the district knows and wants to (indirectly) inform students that large language models (LLMs) must be used responsibly. Brighton High School encourages teachers to use LLMs to assist them in the classroom responsibly, and it should encourage students to be responsible with them as well. Making students complete writing assignments without LLMs has a similar justification to making students complete math assignments without using a calculator: though the tool can be quite useful, you must avoid becoming dependent on it (after all, the only difference between asking your friend to write your essay and asking ChatGPT to write your essay is that ChatGPT is much less likely to say no). It is possible to avoid overusing the technology, but it is not feasible to completely ignore it.
While it’s one thing for me to admit that LLMs have legitimate and beneficial uses, though, it’s another to subscribe to the idea that humanity will be doomed before I graduate from college. In fact, the limitations that current generative AI models possess make it unlikely that they are going to take over humanity in the near future.
Namely, LLMs currently lack proper models of the world, are prone to making stuff up and are very easily goaded into breaking their safety guardrails. This is because these systems do not have any real understanding at all: they are statistical models trained to predict the most likely continuation of the input, usually text. Another common oversimplification is that these systems process words. They do not. They process bits and pieces of words called tokens; direct access to large language models (i.e. making a direct call to the AI model, bypassing the web chatbot interface) is priced per token, not per word. Third, a large language model does not have inherent goals or a personality; the reason each AI chatbot behaves differently is because a special block of text, called the system prompt, is inserted before every query you make. On top of this, the notion of a conversation with a chatbot is an illusion; to generate a reply, the entire conversation thus far, including your replies and the model’s responses, up to a maximum limit called the context window, is fed back into the model with the instruction “continue this conversation.”
This is why ChatGPT can give you a beautifully worded explanation of the origin of the word “strawberry” yet cannot reliably tell you how many letter ‘r’s are in the word itself. This is why Claude can explain to you how to set up and solve an equation yet fail to do basic arithmetic. This is why lawyers who use LLMs get sanctioned, as their AI-generated opinions look good at first glance but contain citations to legal cases that do not exist. What these models can parrot in language does not match up with what they can actually do. Essentially, ChatGPT is a much more sophisticated version of your phone’s autocomplete, and we all know not to trust autocomplete, right?
All of these limitations are inherent to LLMs. “Hallucinations” are not errors or bugs; they are actually the result of the model functioning correctly. What is statistically likely is not always what is true.
In short, you should avoid making yourself reliant on generative AI tools, as said tools are inherently unreliable. This is in addition to the dangers posed to learning by submitting work that is not your own. An AI model should not be treated like a human; it should be treated like a tool, because it is a tool that can be incredibly helpful when used correctly but disastrous when used incorrectly. You cannot use a hammer to screw a screw; you need a screwdriver for that. You cannot get through high school using just AI; you need your brain for that. Embrace the tool, but use it cautiously.
Everyone should take a moment to familiarize themselves with the limitations of large language models before using them, as the increased awareness can be used to elicit higher quality responses. In this way, students at BHS can responsibly adapt to new technologies while ensuring that they continue to make the most out of their education.
In the meantime, humans can sleep tight knowing that generative AI probably won’t be taking over the universe any time soon.
