Maximum Entropy Methods

29 Aug 2007 12:49

It's heresy, but I really don't believe in the maximum entropy principle. Or at least: this is heresy among physicist who are interested in their intersection of their subject with information theory and computation, as I am.

Now, of course I believe that states of thermodynamic equilibrium are states of maximum entropy. I Am Not a Crank, or at least not that much of one. But I don't believe that this is due to some general fact about inductive inference or incomplete information, which is the view propagated by the late, great E. T. Jaynes. I guess I should explain what I do believe about equilibrium thermodynamics and statistical mechanics, and then what it is that I don't believe about entropy maximization. But another time.

I. Csiszár, "Maxent, Mathematics, and Information Theory", pp. 35--50 in Kenneth M. Hanson and Richards N. Silver (eds.), Maximum Entropy and Bayesian Methods: Proceedings of the Fifteenth International Workshop on Maximum Entropy and Bayesian Methods
E. T. Jaynes
- "Information Theory and Statistical Mechanics I," Physical Review 106 (1957): 620--630
- "Information Theory and Statistical Mechanics II," Physical Review 108 (1957): 171--190
- Papers on Probability, Statistics, and Statistical Physics [Reprints both those papers, with many other important ones by Jaynes]
Robert E. Kass and Larry Wasserman, "The Selection of Prior Distributions by Formal Rules", Journal of the American Statistical Association 91 (1996): 1343--1370 [PDF reprint]
Teddy Seidenfeld [Demonstrations that max-ent methods are, in fact, plagued by the same problems as the old Principle of Insufficient Reason, and not consistent with Bayesian inference]
- "Why I Am Not an Objective Bayesian: Some Reflections Prompted by Rosenkrantz", Theory and Decision 11 (1979): 413--440
- "Entropy and Uncertainty", pp. 259--287 in I. B. MacNeill and G. J. Umphrey (eds.), Foundations of Statistical Inference (1987)
Jos Uffink
- "Can the Maximum Entropy Principle be explained as a Consistency Requirement?", Studies in History and Philosophy of Modern Physics 26B (1995): 223-261 [Abstract, with links to PDF and PS]
- "The Constraint Rule of the Maximum Entropy Principle," Studies in History and Philosophy of Modern Physics 27 (1996): 47--79 [I can describe my reaction to this very simply: Word. Abstract, with links to PDF and PS]

CRS, "The Backwards Arrow of Time of the Consistently Bayesian Statistical Mechanic", cond-mat/0410063

Ariel Caticha, "Questions, Relevance and Relative Entropy", cond-mat/0409175
P. Dias and A. Shimony, "A Critique of Jaynes' Maximum Entropy Principle," Advances in Applied Mathematics 2 (1981): 172--211
K. Friedman, and A. Shimony, "Jaynes' Maximum Entropy Prescription and Probability Theory," Journal of Statistical Physics 3 (1971): 381-384.
Peter Grunwald and A. Philip Dawid, "Game Theory, Maximum Entropy, Minimum Discrepancy and Robust Bayesian Decision Theory", Annals of Statistics 32 (2004): 1367--1433
Patrick Haffner, Steven Phillips and Rob Schapire, "Efficient Multiclass Implementations of L1-Regularized Maximum Entropy", cs.LG/0506101
Prakash Ishwar and Pierre Moulin, "On the existence and characterization of the maxent distribution under general moment inequality constraints", cs.IT/0506013 = IEEE Transactions on Information Theory 51 (2005): 3322--3333 ["A broad set of sufficient conditions that guarantees the existence of the maximum entropy (maxent) distribution consistent with specified bounds on certain generalized moments is derived. Most results in the literature are either focused on the minimum cross-entropy distribution or apply only to distributions with a bounded-volume support or address only equality constraints. The results of this work hold for general moment inequality constraints for probability distributions with possibly unbounded support, and the technical conditions are explicitly on the underlying generalized moment functions."]
Oliver Johnson and Christophe Vignat, "Some results concerning maximum Renyi entropy distributions", math.PR/0507400
Jill North, "Symmetry and Probability", phil-sci/2978 [I heard Prof. North talk about this at PSA 2006, and it sounded good, but I need to read the details.]