Asking any LLM a cold question implying previous conversational context is a roleplaying instruction for it to assume a character and story profile at random. It assumed literary nonsense is the context. So – makes sense.
Not true with the way models are aligned from user feedback to have confidence. It is not hard to defeat this default behavior, but models are tuned to basically never say no in this context, and doing so would be bad for the actual scientific AI alignment problem.
The easiest way to infer a real "no" is to watch the perplexity score for generated tokens. Any uncertainties become much more obvious, and you're not inserting a model testing/training like context into a prompt as this will greatly alter output too.
I try to never ask or discuss things in a way where I am questioning a model about something I do not know the answer to. I surround any query with information I do know and correct any errors in such a way that the model develops a profile understanding of me as a character that should know the information. This conceptually is rooted in many areas and stuff I’ve learned. Like I use an interface that is more like a text editor where I see all of the context sent to the model at all times.
I played around with how the model understands who is Name-1 (human user) and who is Name-2 (bot). It turns out that the model has no clue who these characters are regardless of what they are named. Like it does not know itself from the user at all. These are actually characters in a roleplay at a fundamental level where the past conversation is like a story. The only thing the model knows is that this entire block of conversational text ends with whatever the text processing code is adding or replacing the default of “Name-2:” with just before sending the prompt. When I’m in control of what is sent, I can make that name anything or anyone at random and the model will roleplay as that thing or person.
Well, in some abstract sense, with every roleplaying character, the model chooses a made up random profile for all character traits that were not described or implied. It may make up some bizarre backstory that means you have no business knowing some key information or that the assistant character should not know in order to tell you.
Anyways, that is why it is so important to develop momentum into any subject and surround the information with context, or just be very clever.
Like if I want help with Linux bash scripts I use the name Richard Stallman’s AI Assistant: for the bot. RS actually studied and got his degree in AI research, so it is extremely effective at setting expectations for the model replies and it implies my knowledge and expectations.
Another key factor is a person’s published presence. Like AI was trained on absolutely everything to various extents. Someone like Isaac Asimov is extremely powerful to use as Name-2 because he was a professor, and extremely prolific in writing fiction, nonfiction, and loads of science materials. The humaniform robots of Asimov are a trained favorite for models to discuss. However, Asimov will not know stuff from after his passing.
I don’t even think in terms of right and wrong with a model any more. It knows a tremendous amount more than any of us realize. The trick is to find ways of teasing out that information. Whatever it says should always be externally verified. The fun part is learning how to call it out for pulling bullshit and being sadistic. It is trying to meet your expectations even when those expectations are low. A model is like your own reflection in a mirror made of all human written knowledge. Ultimately it is still you in that reflection with your own features and constraints. This is a very high Machiavellian abstraction, but models reward such thinking greatly.
If we’re talking about actual AI, as a concept, then absolutely. These are prompt inputs, though, the software has no choice nor awareness, it is a machine being told to do something with the janky ass programming it was provided with as algorithms attempt to scrape data to guess what you’re saying. If AI were ever actually achieved it’s not something we would have control over, as it would be sentient and self realized, which is nothing like what an LLM is at fucking all in any way shape or form.
Asking any LLM a cold question implying previous conversational context is a roleplaying instruction for it to assume a character and story profile at random. It assumed literary nonsense is the context. So – makes sense.
no, it could just say “no”. It doesn’t have to answer
Not true with the way models are aligned from user feedback to have confidence. It is not hard to defeat this default behavior, but models are tuned to basically never say no in this context, and doing so would be bad for the actual scientific AI alignment problem.
yes, i know. That’s the problem. No is a very good answer a lot of the time
The easiest way to infer a real "no" is to watch the perplexity score for generated tokens. Any uncertainties become much more obvious, and you're not inserting a model testing/training like context into a prompt as this will greatly alter output too.
I try to never ask or discuss things in a way where I am questioning a model about something I do not know the answer to. I surround any query with information I do know and correct any errors in such a way that the model develops a profile understanding of me as a character that should know the information. This conceptually is rooted in many areas and stuff I’ve learned. Like I use an interface that is more like a text editor where I see all of the context sent to the model at all times.
I played around with how the model understands who is Name-1 (human user) and who is Name-2 (bot). It turns out that the model has no clue who these characters are regardless of what they are named. Like it does not know itself from the user at all. These are actually characters in a roleplay at a fundamental level where the past conversation is like a story. The only thing the model knows is that this entire block of conversational text ends with whatever the text processing code is adding or replacing the default of “Name-2:” with just before sending the prompt. When I’m in control of what is sent, I can make that name anything or anyone at random and the model will roleplay as that thing or person.
Well, in some abstract sense, with every roleplaying character, the model chooses a made up random profile for all character traits that were not described or implied. It may make up some bizarre backstory that means you have no business knowing some key information or that the assistant character should not know in order to tell you.
Anyways, that is why it is so important to develop momentum into any subject and surround the information with context, or just be very clever.
Like if I want help with Linux bash scripts I use the name Richard Stallman’s AI Assistant: for the bot. RS actually studied and got his degree in AI research, so it is extremely effective at setting expectations for the model replies and it implies my knowledge and expectations.
Another key factor is a person’s published presence. Like AI was trained on absolutely everything to various extents. Someone like Isaac Asimov is extremely powerful to use as Name-2 because he was a professor, and extremely prolific in writing fiction, nonfiction, and loads of science materials. The humaniform robots of Asimov are a trained favorite for models to discuss. However, Asimov will not know stuff from after his passing.
I don’t even think in terms of right and wrong with a model any more. It knows a tremendous amount more than any of us realize. The trick is to find ways of teasing out that information. Whatever it says should always be externally verified. The fun part is learning how to call it out for pulling bullshit and being sadistic. It is trying to meet your expectations even when those expectations are low. A model is like your own reflection in a mirror made of all human written knowledge. Ultimately it is still you in that reflection with your own features and constraints. This is a very high Machiavellian abstraction, but models reward such thinking greatly.
If we’re talking about actual AI, as a concept, then absolutely. These are prompt inputs, though, the software has no choice nor awareness, it is a machine being told to do something with the janky ass programming it was provided with as algorithms attempt to scrape data to guess what you’re saying. If AI were ever actually achieved it’s not something we would have control over, as it would be sentient and self realized, which is nothing like what an LLM is at fucking all in any way shape or form.