2024-12-23 misc

LLMs are too confident and they need to not be

they need to know their limits and they need to know to what threshold they can make intuitive jumps vs need to break it into smaller parts

good human reasoners do this but actually this is not a closed problem for humans too
like . for something like doing the problem of 572 * 205 ... we know we cant immediately do it so we don't just immediately say "oh the answer is 170380" or wtvr. we have to break it up into like 572 * 2 * 100 + 572 * 10 / 2 = blah
that is a simple example
a more complicated example is . for things like physics hw pset or physics exam . sometimes (either for the solution or for some bigger intermediate step) we think we know Oh obviously we do it using XYZ method/formula . but then as it turns out, it was the wrong formula/approach, and we missed some subtlety .
how do you remedy this in humans? well, we don't have a full solution . but .. next time, they know to account for that subtlety. also, when a peer or teacher reviews their work, we hope to catch any subtle errors by being extra conscious about subtleties . also, when we're solving the problem, i feel like there's some sort of sense of "does this have any subtleties that i might maybe be doing wrong? if so, let's remember these subtleties, and then double check that im accounting for the subtleties correctly"

this is also a skill that we have to train in humans .
salena-like humans (just anyone that is bad at math) have some tendency to be overconfident in applying the wrong formula . or they'll know they're probably wrong, but they just do it anyway bc to them it feels like an attempt at a solution .

===

when breaking down a problem, it's a sort of recursive process
you know you cant make this immediate leap, so you break it down a bit then you know you cant make that immediate leap still so you break it down a bit then etc etc .
but that is so automatic
when you are asked smth . and you dont know the answer . you immediately just start breaking it down . you dont have to consciously stop urself from producing an answer
wait is that true?
how do you make an LLM be uncertain and then make it act differently based on that uncertainty
can i do that with the token probabilities or smth?? that feels very naive. and incomplete.

===

gosh . openai is so ahead . theyre thinking of things that im thinking about .

===

i remember last night or two nights ago i thought curiosity was rly important. i forgot exactly why

===