This raises some questions for me. On the Devil's letter, I agree that I should not open it. (The way to deny this would be a sheer preference-satisfaction view of rationality combined with being very, very curious about what's inside. See the Dutch movie The Vanishing (1988) for a depiction of what something like that might look like.)
I think the Bayesian can kind of account for this. First, it's worth noting that the ideal Bayesian answer should plausibly be indifference to opening it. Nothing good can come from it, by hypothesis, but also nothing bad can come from it, since the Bayesian will know that nothing better will come from actions that are based on that information relative to the actions they would have taken without it, and will rationally take the first actions. (This probably won't be just not conditionalizing on the information, but doing so and then also conditionalizing on the expected value of outcomes that would follow from acting on the information, balancing out. Not positive about this.)
If we permit the possibility that the Bayesian is not perfect, for example an embodied creature with memory that can't perfectly distinguish different information and its sources, and might even make some imperfect EV-calculations once in a while, then they should obviously not open the letter, because it is negative EV. Any diversion from the ideal action will lead to negative outcomes, by using the Devil's information.
This gets me to a more general points. This text seems to move pretty freely between talk of decision theory and psychology. For example, the above example seems partly motivated by the fact that we humans have psychologies that would fail the Bayesian ideal, so it's unwise to be indifferent to opening the letter. I think these should probably be kept more separate, or rather: the normative appeal of the decision theory as a proposal of rationality, I think, should not be too influenced by intuitions of what human or other psychologies are like. However, if we motivate the decision theory, we can then assess whether we can live up to it. (This is my pet peeve with the Allais paradox, for example.)
Finally, I'd like to see a clearer articulation of when we enter the Knightian holes. This seems vague, in the philosophical sense, but also deserving of a different normative treatment. The fact that Bayesians don't require such a shift seems like one of its prime virtues.
"First, it's worth noting that the ideal Bayesian answer should plausibly be indifference to opening it. Nothing good can come from it, by hypothesis"
"If we permit the possibility that the Bayesian is not perfect ... then they should obviously not open the letter, because it is negative EV.
All the action here is in figuring out how to pin down such reasoning (e.g. to a level of specificity that we could design a version of AIXI which did it).
In particular, the problem with the former is that you can't represent a set of mutually exclusive collectively exhaustive hypotheses about the devil himself. The problem with the latter is that you can't represent a set of mutually exclusive collectively exhaustive hypotheses about your own cognitive mistakes.
Without that, you're no longer justified in using probabilities, and you need to do something different. That's what Knightianism is for.
Re separating decision theory and psychology: see this exchange with Jess Taylor (https://www.lesswrong.com/posts/pYFBD2SnqiWkuNns5/explaining-knightianism-on-one-foot?commentId=TPJm3pFybybfiAHbE). In particular, I don't want to separate them because "I think there's an elegant theory of boundedly rational agents which does apply across many scales, such that many things we currently think of as human-specific phenomena will actually turn out to be facets of this deeper structure".
Re Knightian holes: yepp, I'm working on characterizing them better. But I don't think Bayesianism does any better, it just ignores non-realizable hypotheses. So being able to talk about such holes seems like a step forward. (Not sure what you mean by "entering" them though—from a third-person perspective the holes are in one's world-model; while from a first-person perspective you're always in a Knightian hole.)
Okay interesting. I'll check out your exchange, which might respond to the point below.
Nonetheless, I'll just mention that I think that on one interpretation of Bayesianism, and probably the one I'd endorse, it's not really a theory of reasoning at all. Rather, it's something like a theory of what state should obtain, given some prior state and some new evidence, however one goes about arriving at that state (i.e. what reasoning one uses). So in a sense, I'm not sure such Bayesianism is even trying to be a theory of good reasoning, as opposed to a theory of rational attitude-transitions, or something like that.
I'm not sure why I can't represent MECE hypotheses about the Devil. For example, he's either bearded or not. It seems true that I can't represent good ones that will help me represent the outcome space well though. And that seems partly like a constraint on the concepts I have available to do so, which I suspect you agree with. Not sure if my own cognitive mistakes introduce new issues.
And on the "entering", yea I probably misunderstood this. I guess I imagined that from my current perspective, there are some things that I can have plausible guesses about and some that I can't, according to Knightianism. And then we can construct a Sorites case by adding a bit more context, making it more and more understandable until we hit the domain of kosher use of Bayesianism. And if so, that raises the question whether the boundary is sharp or gradual. Gradual seems more appealing, but I'm not sure how one would spell that out.
This raises some questions for me. On the Devil's letter, I agree that I should not open it. (The way to deny this would be a sheer preference-satisfaction view of rationality combined with being very, very curious about what's inside. See the Dutch movie The Vanishing (1988) for a depiction of what something like that might look like.)
I think the Bayesian can kind of account for this. First, it's worth noting that the ideal Bayesian answer should plausibly be indifference to opening it. Nothing good can come from it, by hypothesis, but also nothing bad can come from it, since the Bayesian will know that nothing better will come from actions that are based on that information relative to the actions they would have taken without it, and will rationally take the first actions. (This probably won't be just not conditionalizing on the information, but doing so and then also conditionalizing on the expected value of outcomes that would follow from acting on the information, balancing out. Not positive about this.)
If we permit the possibility that the Bayesian is not perfect, for example an embodied creature with memory that can't perfectly distinguish different information and its sources, and might even make some imperfect EV-calculations once in a while, then they should obviously not open the letter, because it is negative EV. Any diversion from the ideal action will lead to negative outcomes, by using the Devil's information.
This gets me to a more general points. This text seems to move pretty freely between talk of decision theory and psychology. For example, the above example seems partly motivated by the fact that we humans have psychologies that would fail the Bayesian ideal, so it's unwise to be indifferent to opening the letter. I think these should probably be kept more separate, or rather: the normative appeal of the decision theory as a proposal of rationality, I think, should not be too influenced by intuitions of what human or other psychologies are like. However, if we motivate the decision theory, we can then assess whether we can live up to it. (This is my pet peeve with the Allais paradox, for example.)
Finally, I'd like to see a clearer articulation of when we enter the Knightian holes. This seems vague, in the philosophical sense, but also deserving of a different normative treatment. The fact that Bayesians don't require such a shift seems like one of its prime virtues.
"First, it's worth noting that the ideal Bayesian answer should plausibly be indifference to opening it. Nothing good can come from it, by hypothesis"
"If we permit the possibility that the Bayesian is not perfect ... then they should obviously not open the letter, because it is negative EV.
All the action here is in figuring out how to pin down such reasoning (e.g. to a level of specificity that we could design a version of AIXI which did it).
In particular, the problem with the former is that you can't represent a set of mutually exclusive collectively exhaustive hypotheses about the devil himself. The problem with the latter is that you can't represent a set of mutually exclusive collectively exhaustive hypotheses about your own cognitive mistakes.
Without that, you're no longer justified in using probabilities, and you need to do something different. That's what Knightianism is for.
Re separating decision theory and psychology: see this exchange with Jess Taylor (https://www.lesswrong.com/posts/pYFBD2SnqiWkuNns5/explaining-knightianism-on-one-foot?commentId=TPJm3pFybybfiAHbE). In particular, I don't want to separate them because "I think there's an elegant theory of boundedly rational agents which does apply across many scales, such that many things we currently think of as human-specific phenomena will actually turn out to be facets of this deeper structure".
Re Knightian holes: yepp, I'm working on characterizing them better. But I don't think Bayesianism does any better, it just ignores non-realizable hypotheses. So being able to talk about such holes seems like a step forward. (Not sure what you mean by "entering" them though—from a third-person perspective the holes are in one's world-model; while from a first-person perspective you're always in a Knightian hole.)
Okay interesting. I'll check out your exchange, which might respond to the point below.
Nonetheless, I'll just mention that I think that on one interpretation of Bayesianism, and probably the one I'd endorse, it's not really a theory of reasoning at all. Rather, it's something like a theory of what state should obtain, given some prior state and some new evidence, however one goes about arriving at that state (i.e. what reasoning one uses). So in a sense, I'm not sure such Bayesianism is even trying to be a theory of good reasoning, as opposed to a theory of rational attitude-transitions, or something like that.
I'm not sure why I can't represent MECE hypotheses about the Devil. For example, he's either bearded or not. It seems true that I can't represent good ones that will help me represent the outcome space well though. And that seems partly like a constraint on the concepts I have available to do so, which I suspect you agree with. Not sure if my own cognitive mistakes introduce new issues.
And on the "entering", yea I probably misunderstood this. I guess I imagined that from my current perspective, there are some things that I can have plausible guesses about and some that I can't, according to Knightianism. And then we can construct a Sorites case by adding a bit more context, making it more and more understandable until we hit the domain of kosher use of Bayesianism. And if so, that raises the question whether the boundary is sharp or gradual. Gradual seems more appealing, but I'm not sure how one would spell that out.
Super cool, thanks. I loved the devil/angel thing.