I like that little smile Eliezer gives when Andrew wonders "what tests were like when you went to college". My understanding is that Eliezer is an autodidact who never went to college, or even high school for that matter.

I'm glad someone was able to enjoy this one... Regression analysis? Sure, it's used a lot in my field. Thirty cases? Sure, you can't study too many variables with that "N". Other than that, no clue. Did they talk about hierarchical regression? I never remember whether that's the one in which you pick the order of the variables to be used in the analysis. Just kidding... definitely clueless. Enjoy!

laughed at that part too. eliezers shows that a middle school education is a sufficient condition to being a public intellectual.

I don't understand why Eliezer says that after tossing 99 heads in a row you can't realize that anything is wrong within the model (fair coin). The probability of a 5-run of tails is very high (>80%) for instance. So by the 99th toss you've already violated a bunch of well-known statistics for the behavior of a fair coin. Why wouldn't you realize that something is amiss?

Quoting Unit: I don't understand why Eliezer says that after tossing 99 heads in a row you can't realize that anything is wrong within the model (fair coin). The probability of a 5-run of tails is very high (>80%) for instance. So by the 99th toss you've already violated a bunch of well-known statistics for the behavior of a fair coin. Why wouldn't you realize that something is amiss?

I think that, here, "the model" starts out saying that the coin is fair with 100% certainty, in the same way that the model holds with 100% certainty that there's a coin there at all. But, to realize that something is amiss, you have to start to doubt that the coin is fair. This means, by definition, that you are changing models. You are not working entirely within the model. For this reason, it's better for an agent to use a model that starts out only assuming that the coin has some amount of bias (possibly zero). Then, when the model observes the 99 heads-in-a-row, the agent can use Laplace's rule of succession, or something like that, to put bounds on what the bias probably is. The point is that, under this approach, the agent

I looked up information on the netflix contest, thought it was pretty interesting. http://bits.blogs.nytimes.com/2009/0...ew-contest/?hp I am really glad netflix did that contest and is doing another. You could apply that to books as well in theory. People have so many ratings they offer, but something that always nagged at me was the reality that there were countless gems, in all sorts of areas, that would escape our notice as we would never be exposed to them.
With this getting more and more accurate, it increases the chances of discovering things you like and would NEVER have encountered as it may have been outside your network of awareness. This is the kind of hard application of statistical models that should be encouraged more.
I wonder how many areas these models are attempted in? I know Gelman mentioned doing models to predict voting patterns, what else of interest are these advanced models being used for to predict human choice and behavior? Is there more predictability when dealing with human behavior and choice vs say more chaotic natural phenomenon like climate prediction? Maybe that gets too far into details I'd have no clue on deciphering.

Quoting Tyrrell McAllister: I think that, here, "the model" starts out saying that the coin is fair with 100% certainty, in the same way that the model holds with 100% certainty that there's a coin there at all. But, to realize that something is amiss, you have to start to doubt that the coin is fair. This means, by definition, that you are changing models. You are not working entirely within the model.

For this reason, it's better for an agent to use a model that starts out only assuming that the coin has some amount of bias (possibly zero). Then, when the model observes the 99 heads-in-a-row, the agent can use Laplace's rule of succession, or something like that, to put bounds on what the bias probably is.

The point is that, under this approach, the agent never has to doubt anything that it believed with certainty at the beginning. So the agent is always using the same model, or, in other words, always working within the model.

Thanks for that, that was alot clearer there than what I heard.

Indeed, that clears a lot up including comparisons with Taleb.

Math with a cliffhanger ending! Will Gelman finish that thought? Does he have an objection to Yudkowsky's methods? The Blogginghead Community Demands a Rematch! (Also, Gelman should have a 15-yard penalty for the college comment. A planned slight is ungentlemanly.)

I didn't find Eliezer's opening gambit compelling at all. It essentially posits facts that are prima facie inconsistent, and in that sense is similar to the "proof" that all triangles are isoceles (i.e. you start by picking a vertex and dropping the perpendicular to the midpoint of the opposite side). As a classical probabilist, it makes no sense to me to assert that I am somehow honor-bound not to question the veracity of the claim of fairness once the string of heads exceeds a certain length. In fact a classicist would continuously update at least the sign test as the experiment proceeded, and in fact is able to call on any number of significance tests for randomness. Any classicist worth his salt would call into question the fairness of a coin that showed a sequence HHHHHHHHHHTTTTTTTTTTHHHHHHHHHHTTTTTTTTTT.... Eliezer's notion that this sequence is "just as likely" as a sequence HHTTTHTHHTTTHHHTTTHTHTTTTHHHTHTHHHTHTHTT.... expresses a willful ignorance of those classical tests of randomness. The question I have is whether a Bayesian has any way to analyze, say, a sequence of 50 heads followed by 50 tails -- in my limited understanding of Bayesianism, starting with an uninformative prior, this

in my limited understanding of Bayesianism,
...
My sense is that there is a fundamental sickness at the heart of Bayesianism. The sickness is the lack of rigor. Oh, goodness. Look, there are a lot of brilliant people who dislike Bayesianism, but: 1) they know how it works and 2) no one criticizes its lack of rigor! Analytically, there is no distinction between Bayesian and frequentist methods: Bayes' Rule is Bayes' Rule. If you're looking for a cocktail party criticism of Bayesianism, you're supposed to complain that "it's too subjective". But consider, for example, the Bayesian probability distribution you get from flipping a coin (with an uninformative prior) and getting 6 heads and 4 tails. What on earth does this distribution mean? Well, speaking of rigor, let's be clear about our terminology. You're referring to the posterior distribution. It's the updated probability distribution for p, where p=P(coin=Heads), after having seen the data. Is there some test I can do that will accumulate a histogram that is consistent with that distribution? I'm not sure what you mean here. Consistency is a formal property of a sequence of estimators.

Quoting SaraK: in my limited understanding of Bayesianism,
...
My sense is that there is a fundamental sickness at the heart of Bayesianism. The sickness is the lack of rigor.

Oh, goodness. Look, there are a lot of brilliant people who dislike Bayesianism, but: 1) they know how it works and 2) no one criticizes its lack of rigor!

I'll admit it, I think I don't get the Bayesian thing. I mean, I think I get it, but it doesn't seem like a big enough deal to get worked up about pro or con. Some people ascribe quasi-mystical properties to it. To me it just seems like basic probability. You're still left with garbage in, garbage out. If your prior is "the moon is made of green cheese" Bayes ain't going to help, e.g., Robin Hanson and his conditional belief in cryogenics.

Again, I'm not a Bayesian, I'd just prefer to not see people -- especially those who identify as "classical probabilists" -- reveal a total lack of knowledge about the field they're criticizing. It's naive to think that frequentist statistics is objective. When we chosen an estimator with small bias but small MSE over an unbiased estimator with larger MSE, that's subjective. When we select alpha=.03 as a cutoff for a hypo test, that's subjective. Garbage in, garbage out?

This was kind of an embarrassing dialogue. It sort of reminds me of the infamous Gary Shandling episode of Ricky Gervais meets. Just... unpleasant to watch. How did this happen? Did both participants know what the point of the interview was? Eliezer seems to have been more interested in the lay audience side of it, Gelman was more like fireside academic chat.

Okay, unfair. It was a great talk. I braced myself, sat through the uncomfortable part of it and then it got really interesting. With intermittent periods of uncomfortableness.

Quoting SaraK: in my limited understanding of Bayesianism,
...
My sense is that there is a fundamental sickness at the heart of Bayesianism. The sickness is the lack of rigor.

Oh, goodness. Look, there are a lot of brilliant people who dislike Bayesianism, but: 1) they know how it works and 2) no one criticizes its lack of rigor!

Analytically, there is no distinction between Bayesian and frequentist methods: Bayes' Rule is Bayes' Rule. If you're looking for a cocktail party criticism of Bayesianism, you're supposed to complain that "it's too subjective".

But consider, for example, the Bayesian probability distribution you get from flipping a coin (with an uninformative prior) and getting 6 heads and 4 tails. What on earth does this distribution mean?

Well, speaking of rigor, let's be clear about our terminology. You're referring to the posterior distribution. It's the updated probability distribution for p, where p=P(coin=Heads), after having seen the data.

Is there some test I can do that will accumulate a histogram that is consistent with that distribution?

I'm not sure what you mean here. Consistency is a formal property of a sequence of estimators.

You are conflating Bayes' Theorem with the Bayesian interpretation of probability, I think. Bayes' theorem is an uncontroversial proposition in both frequentist and Bayesian camps, since

Quoting bbbeard: ... I can check this, first of all, with a computer program that generates random numbers uniformly in [0,1) in groups of ten, and keeping tabs on what fraction of samples have exactly seven numbers less than 0.5. Obviously I can do this for any (m,n)...BBB

There is no such thing as computer that can calculate true random numbers; only pseudo-random numbers, as they all start with a seed. Which is totally not relevant, to your argument, as the pseudo-random numbers would serve the purpose as used in this discussion. I did some work for a professor researching this in the 90's. they may have advanced the field since then but at the time that was considered as highly unlikely possibility.

Quoting bbbeard: The Bayesian interpretation is certainly not what we use in physics. Suppose we lived at a time before the speed of light was measured accurately. You could poll a bunch of people, even "experts", and get a range of guesses about the value of the speed of light. A Bayesian would construct a prior from this information. But what happens when you go do the experiment? A "pure as the driven snow" Bayesian would update the prior with the test data and generate a posterior probability distribution for the speed of light -- which, I contend, would be meaningless. A scientist not contaminated with the Bayesian paradigm would simply discard all the prior opinions, and use the experimental value with a rigorously determined standard error. What am I not getting here?

I'm among those who don't understand your question. Why would the Bayesian's posterior probability distribution be meaningless? A Bayesian, like your uncontaminated scientist, could conclude that the new tests make the old "expert" testimony irrelevant. Let E1 be the old expert testimony, and let E2 be the new test results. Let H by the hypothesis that the speed of light is within certain fixed bounds. Then you might very well have p(H | E1) > p(H), but

Quoting piscivorous: There is no such thing as computer that can calculate true random numbers; only pseudo-random numbers, as they all start with a seed. Which is totally not relevant, to your argument, as the pseudo-random numbers would serve the purpose as used in this discussion.

I did some work for a professor researching this in the 90's. they may have advanced the field since then but at the time that was considered as highly unlikely possibility.

You might have a look at HotBits, on John Walker's site, if you don't already know about it. Pretty interesting concept -- the short version is he offers, as a service, what he claims are true random numbers. I haven't ever had occasion to do more than play around with it, but it seems to me that if you couldn't get enough data from him, you could at least use what data you could get to build seeds which would be truly random, which seems as though it would make the output of pseudo-random number algorithms give something much closer to true randomness, and practically speaking, often quite good enough. This is, as you'll see, somewhat of an old project

I am vaguely aware of the service and of some others that have designed hardware specifically to create true random numbers. I don't know of their real efficacy though.

Quoting Tyrrell McAllister: I'm among those who don't understand your question. Why would the Bayesian's posterior probability distribution be meaningless?

A Bayesian, like your uncontaminated scientist, could conclude that the new tests make the old "expert" testimony irrelevant. Let E1 be the old expert testimony, and let E2 be the new test results. Let H by the hypothesis that the speed of light is within certain fixed bounds. Then you might very well have p(H | E1) > p(H), but p(H | E1 & E2) = p(H | ~E1 & E2) = p(H | E2). That is, when E1 came in, it provided evidence for the hypothesis H, but now that we have new evidence E2, E1 no longer has any bearing on the probability of H.

I'm clueless about what you are saying. Are you saying it's a theorem that p(H | E1 & E2) = p(H | ~E1 & E2) = p(H | E2)? Or that it might be the case but also might not? I frankly don't even understand what you mean by the "probability that H is true given E1". You measured the speed of light. The measurement has some uncertainty associated with, say, the precision of the yardstick with which you measured a distance for light to travel. In what universe could the "probability" of H depend on the prior opinions of experts? That is, how could P(H | E1) > P(H)? [BTW no one has pointed out that the speed of light is now defined to be exactly 299,792,458 m/s, that is, the meter is defined in terms of c and a standard clock; we no longer try to measure c with a meterstick and a clock. But that wasn't always the case...] The "old" relative accuracy for c was about 4E-9, which I infer

Quoting bbbeard: The question I have is whether a Bayesian has any way to analyze, say, a sequence of 50 heads followed by 50 tails -- in my limited understanding of Bayesianism, starting with an uninformative prior, this sequence would give exactly the same posterior distribution as an actually fair coin.

Classical tests have implied alternatives against which they have power. In this case, the classical test must be looking at autocorrelations in the sequence to detect the non-randomness of your sequence of 50 heads followed by 50 tails. The Bayesian equivalent is a prior distribution over the stochastic process generating the data. If we admit at the start that the data might be generated by, say, a first-order Markov chain, then we can easily describe a Bayesian procedure that could sharply distinguish this sequence from one generated by an i.i.d. process. (See example B of this paper by Diaconis and Rolles.) I should mention that Andrew Gelman dislikes Bayesian tests with discrete alternatives, so rather than the procedure in the Diaconis paper, he would probably prefer something like just fitting the Markov chain model and checking how far away it is from an i.i.d process.

Quoting bbbeard: My sense

Quoting bbbeard: Are you saying it's a theorem that p(H | E1 & E2) = p(H | ~E1 & E2) = p(H | E2)?

No. I'm saying that the conjunction of p(H | E1) > p(H), and p(H | E1 & E2) = p(H | ~E1 & E2) = p(H | E2) might very well happen in this particular case.

I frankly don't even understand what you mean by the "probability that H is true given E1".

I'm using "the probability that H is true given E1" to mean "the amount of credence that you ought to have in H if E1 is your only evidence."

In what universe could the "probability" of H depend on the prior opinions of experts?

Again, probability is the proper amount of credence. So, if the prior opinions of experts is the best evidence you have on the matter, your credence should depend on those opinions.