This Thread Has A 65% Chance Of Sucking

[acropolis]

Probability question. My weatherman says there is a 30% chance of rain today. A computerized ranking system I saw gave Pitt a 65% chance of beating Syracuse. In each case, the potential event will either happen or it won't; their won't be 30% of a rain today and Pitt won't lose 35% of tonight's game. Making the quantum assumption; that in each event there is a certain % chance of each outcome and God will simply use his random number generator to determine which does in fact happen, then the ideal output of both the meteorological and sportsological models is actually a %. How do we measurehow accurate a given model is, based on past results (I'm sure this has been done, and can probably be answered with a wiki link, but I think it's worth thinking about anyway).

To start with, an intuitive approach would be to get a large sample, and then divide it up into 'bins', each with the same prediction. So if there had been 20 "60% chance of rains", then ideally it would have rained 12 times. Do this for each % prediction, and make the comparison. As it turns out, this is terrible for two reasons. First, looking at the past year, maybe it rained 178 times. Then if you predict 50% chance of rain every day for the next year, this system will score you as ~perfect (assuming it rains 177-178 times again this year). That's not how it should work. Second, I have to assume that the 'ideal model output' isn't always an integer %, or even rational, or even constructible. Likely you will have as many different outcome predictions as you have results, which pretty much kills whatever hope was left.

Couple things, on relative and absolute measurements. In theory it should be easy to make comparisons between different models to determine which is better (relative comparison), and I'm not sure why I don't see more of it done (you'd think someone would do comparisons of weathermen accuracy, and if they did, you'd think whoever did best would brag about it in commercials). Absolute results are, however, tricky. At first glance, my absolute system would put its 0 at predicting the system's average outcome every time (e.g., predicting that there is a 60% chance of rain every day in an area where it rains 60% of the time, or predicting that every team has a 50% chance of winning each game (or, more interestingly, making utterly random predictions that average out to 50%)), and it would put its 1 at predicting either a 100% or 0% chance every time, and getting it right every time. Obviously negative model values are possible, but are really kind of better than 0 because you could just inverse all your predictions and suddenly you have a non-trivial model.

The problem with this is that you'll never know if your model is optimal, again making the original assumptions that events have some non-zero, non-100% probability of occurring. If every event has a 70% chance of occuring, and everytime you predict 70%, your model is perfect but you have no way of knowing that approaching purely from a statistical standpoint.

My real interest in this is purely from the standpoint of abusing computer models to make money gambling on sporting events.
A) How would I determine which computer model to use
and
B) Would it work?

i just think that if it did work, someone would be doing it already.

anyway.

[pig]

The house always wins.

[JonnyBGood]

You need more/better information. This is why some people can actually make money on sports betting, they know what they're talking about more than the guys making the odds. I don't think there's a "brute force" way of exploiting the odds as you seem to suggest.

[Dante Hicks]

Past performance is not indicative of future results.

[JonnyBGood]

Originally Posted by Dante Hicks

Past performance is not indicative of future results.

Would the demise of induction by any other name smell as sweet?

[acropolis]

Originally Posted by Dante Hicks

Past performance is not indicative of future results.

only for mutual funds.

as a general rule, past performance is the best indicator of future results.

[Cannon_Fodder]

Because it's the only one?

[Alessio]

When you have all the facts, like when playing a cardsgame, you can make accurate probability models

When you don't, like in a sportsmatch when you have human factors, the probability stays a gamble
and then it's more important to have insight about the situation

That's why more people get rich and banned from playing at casino's then betting at sporting events
If I would want to make money through gambling, then I wouldn't pick sporting events

[queball]

Originally Posted by acropolis

The problem with this is that you'll never know if your model is optimal, again making the original assumptions that events have some non-zero, non-100% probability of occurring. If every event has a 70% chance of occuring, and everytime you predict 70%, your model is perfect but you have no way of knowing that approaching purely from a statistical standpoint.

You can compare it to other models though, in a way that doesn't require any experimentation at all, as you have done. (Assuming that that probabilities given are empirically accurate)

My real interest in this is purely from the standpoint of abusing computer models to make money gambling on sporting events.
A) How would I determine which computer model to use
and
B) Would it work?

i just think that if it did work, someone would be doing it already.

anyway.

You've just given an artificial estimate of how useful a statistic is. This kind of estimate is harder to construct for weather prediction than for betting because usefulness depends on personal preferences about the inconvenience of carrying an umbrella, etc. It's actually far easier to evaluate a betting scheme - notice there's things like celebdaq that turn predictions into betting to make some kind of competition out of it. So a better measure would be to see how much money your prediction makes.

I think one effect your score mightn't pick up is that a system that is good at betting on long odds is better than a system that's good at betting on short odds.

[Nodrog]

Quantum mechanics just confuses things and is irrelevant here; dont even bring it into consideration. Everything either happens or it doesnt, but probabalistic statements are a way to encapsulate the uncertainty we have about an outcome due to our inherent lack of knowledge of all the relevant factors (this is the Bayesian interpretation of probability, and the only one I personally think is coherent).

Originally Posted by acropolis

How do we measurehow accurate a given model is, based on past results (I'm sure this has been done, and can probably be answered with a wiki link, but I think it's worth thinking about anyway).

In order to measure how accurate a given model is, you have to provide it with new observations and see how many it predicts correctly. One unsophistivated way of doing this is to split your set of past observations into a training set, and a test set. Develop your model using the training set without letting it see the test set, and then once youve got a model that you think is decent, try it out on the test set and see how many observations it classifies correctly. If you get a relatively low error rate, youve probably got a decent model. This is a fairly simplistic approach to model testing thought, and there are more sophisticated methods such as cross-validation (where you divide repeatedly dividie the set of observations into subsets of size 1 and N-1 and train on the N-1 set), and bootstrapping.

an intuitive approach would be to get a large sample, and then divide it up into 'bins', each with the same prediction. So if there had been 20 "60% chance of rains", then ideally it would have rained 12 times.

This is seems a strange way of thinking about things. Your classifier should either say that itll rain or it wont, and the '60% chance' will denote that confidence you have that it is predicting correctly. This will be a function of (amongst other things) the error rate (the % of observations its classfied correctly in the past). The error rate can be estimated from your test set of observations, and refined by seeing how well it copes with future observations. You can break the error rate down into false positives/false negatives if you need a more accurate description of the model.

Couple things, on relative and absolute measurements. In theory it should be easy to make comparisons between different models to determine which is better (relative comparison), and I'm not sure why I don't see more of it done

Its done a lot in proper model testing, and theres a whole host of inconclusive theory about which ways of making comparasions is best. The problem is that different methods of model validation can favour different models (eg a cross validation procedure might say model X is better, and a bootstrap might say model Y). If a model is prone to overfitting (adapting too well to data) then you could end up with a model which performs well on training data but doesnt generalise properly since its too specific. This is why testing the model on some observations you withheld from it while training is so important.some observations.

Theres also the problem of deciding whether your results are statistically significant - if model X performs (say) 10% better than model Y on your training data, does this necessarily mean that its likely to be perform better on future observations, or could this just be a statistically irrelvant consequence of using a finite training set? Again, theres a lot of theory devoted to this sort of thing (I beleive Mcnemar's test is fairly popular in the literature).

The problem with this is that you'll never know if your model is optimal

You cant know whether your model is optimal unless you know the complete form of the underlying distribution producing the observations (including any parameters). Given full knowledge of the distribution, you can calculate the Bayes error rate which tells you the theoretical minimum error rate that a classifier can get on the data. But since in real life you cant normally get access to this, you have to rely on empirical methods instead (ie trying lots of things and seeing what works).

A) How would I determine which computer model to use

Simplistically, you'd analyse the data and try to work out which distributions(s) it came from. If you could find this then you could use a parametric estimation technique to find the unknown parameters of the distribution and test how well your model performed. If you couldnt get a good idea of what the underlying distribution is, then there are many methods for doing non-parametric estimation (eg neural networks, kernel estimation, nearest-neighbour methods etc).

B) Would it work?

Depends how talented you are I suppose. Assuming the data is complex, it wouldnt be easy.

[Kurashima]

Nods here

This threads chances of sucking just went up a good 15%

[Nodrog]

Originally Posted by JonnyBGood

You need more/better information. This is why some people can actually make money on sports betting, they know what they're talking about more than the guys making the odds. I don't think there's a "brute force" way of exploiting the odds as you seem to suggest.

Yeah, pretty much. However if all the odds were being compiled by one person (or one team of people, or by one computer) over a long enough period of time, then its possible that the compiler might be making a systematic mistake (like constantly giving slightly too long odds to underdogs, or being overgenerous with the prices of a 'first goalscorer' bet, etc). I suppose its possible that you could pick this sort of thing up if your analysis were sophisticated enough, but youd have to put a _lot_ of work into it, and import a huge amount of domain-specific knowledge.