Alpha
Alpha is a calculation of rating spread or the amount the rating differences between two players tend to differ, computed as ratio, from their "true" rating differences. An alpha value of 1.0 is considered perfect. A lower value suggests that players true ratings are further apart than their computed ratings. Likewise a higher value implies that the true ratings are closer together.
In the simulations I have conducted, if you have K-factors large enough to give good results for your rating system, you will tend to get alpha values of about 0.6 to 0.8 implying that a player with the lower rating in a match has a higher chance of winning games than the differences in their ratings would suggest. You can fix this somewhat by lowering the K-factor. Unfortunately, if you lower the K-factor, it will improve the alpha values but usually with a significant cost to rating accuracy. In general low alphas are less of a problem than low accuracy. Low alphas reduce the predictive effect of rating differences for who is likely to win the game, but they do not reduce the accuracy of the relative ranking of the players.
Alpha can be computed as follows. Take a fixed rating difference. I use 300 in the simulation code. Take a small interval around that rating difference. I use plus or minus 20 so that the interval for rating differences is [280, 320]. Track the results all matches where the rating difference was inside this interval. In the simulation code all matches with rating differences bigger than 280 and smaller than 320 are tracked. Use this accumulated data to compute a winning percentage. Use the winning percentage and use a reverse lookup on your rating model to determine the rating difference that should produce this winning percentage. This should give an estimation of the true rating difference. In the case of the simulation that I wrote, the model used is the Elo Rating Formula. Take the fixed rating difference and divide it by the estimated true rating to get the computation of alpha. In the simulation, 300 is divided by rating calculated by the reverse lookup of the Elo rating formula on the actual winning percentage.
This article's definition came from an article written by Mark Glickman, who has written a number of articles on chess ratings for the U.S Chess Federation. On the web site Chessmetrics, Jeff Sonas has a web page where he presents what he believes is a better formula for computing ratings for elite chess players. He implicitly takes alpha into account by squeezing the ratings towards 2300 which tends to reduce the rating spread and increase alpha by brute force. Jeff Sonas also argues that the current K-factor used by the World Chess Federation is way too low and a much more reasonable value of 24 should be used. I suspect that 24 is also too low and a more reasonable value would be 40 and even stronger compensation for alpha should be used. This estimate comes from the rating simulations that I have done. See my blog article more more.