Tuesday, October 28, 2008

A Simple Election Simulation

As of today, Real Clear Politics lists Pennsylvania as "Strong Obama" with a +10.8 spread (51.4 to 40.6). However, those two percentages only add up to 92%, leaving 8% independents. If all of the independents go for McCain, then the race is much closer, 51.4 to 48.6. Recently there was an article in Salon arguing that independents will, in fact, break heavily for McCain. The article cites poll numbers and voting outcomes from a few recent elections to argue that for African American candidates, "what you see is what you get." In other words, if Obama is polling at 51.4% in Pennsylvania, then on election day he'll get very close to 51.4% of the vote. Hence, Obama needs to stay above 50% in key state polls, regardless of McCain's numbers.

If we take this perspective, then the race is much closer than the spread numbers would seem to indicate.

I did a quick simulation with numbers from Real Clear Politics and Pollster.com. Here's what I did and how it turned out.

I simulated the national election 10,000 times. For each iteration, I went through the 50 states (plus the District of Columbia) one by one, and for each state, I had the computer flip a biased coin to see which candidate would win the state's electoral votes on November 4. To bias each state's coin, I used a gaussian distribution with a mean equal to Obama's polling percentage and a standard deviation of 3 percentage points. So for example, in Pennsylvania Obama's chance of victory was governed by a gaussian distribution of mean 0.514 and standard deviation 0.03. If the number drawn from this distribution was greater than 0.5, I put the state in Obama's column; otherwise, McCain's.

The results were sobering. Obama wins the election in only 70% of my simulations.


Based on the simulations, state priorities for the two candidates emerge as follows.

Obama's Priorities. I identified these as all of the states that participate in more than 70% of Obama victories in the simulation and yet have poll numbers below 52% for Obama. New Mexico is not here because it only participates in 62% of Obama victories, Ohio is not here because it only participates in 59% of Obama victories, and Florida is not here because it only participates in 30% of Obama victories.

The first number is the percent of Obama victories in which the state goes for Obama. The second number is Obama's poll number for the state.

PA, 81.5, 51.7
IA, 80.3, 52.3
VA, 76.5, 51.7
WI, 76.5, 51.8
MN, 71.9, 51.3
NH, 71.2, 51.5
CO, 70.7, 51.3

These numbers may explain why Obama is circling back to Iowa this week.

McCain's Priorities. I identified these as all of the states which participate in more than 30% of McCain's victories in the simulation and yet have poll numbers below 52% for McCain (defined here as poll numbers above 48% for Obama). Iowa is not here because it only participates in 29% of McCain victories (just below the arbitrary threshhold I set), and Georgia is not here because Obama is only polling at 44.7% in Georgia (and 100-44.7 = 55.3, a comfortable victory for McCain). Florida is not here because Obama is only polling at 47.7%, which puts McCain above 52% according to the "what you see is what you get" thesis.

The first number is the percent of McCain victories in which the state goes for McCain. The second number is McCain's poll number for the state (here defined as 100 minus Obama's poll number).

NC 81.9 51.2
OH 76.8 50.1
RI 76.7 51.8
NV 67.3 51
PA 54.1 48.3
NM 45.9 49.3
MN 44.0 48.7
CO 42.6 48.7
VA 42.6 48.3
WI 38.2 48.2
NH 34.1 48.5

The true "battlegrounds" - the states on both candidates' lists - are:

New Hampshire

The output for each state is shown below.


SportsBook.com is giving odds of 4-1 for a bet on John McCain, i.e. a 20% chance of winning. Since my simulation gives John McCain a 30% chance of winning, I figure McCain is undervalued by the betting market. Maybe I should put ten bucks on the old gambler to win.


Nothing has been proved here, of course. The whole thing is much too complicated to compute. The only thing to do is fight until the end.

Even mathematically, within the assumptions of the model, the 70% figure is sensitive to the assumption of a 3 point margin of error in the polls. Here is the sensitivity analysis:

So my result would agree with the oddsmakers if I had assumed a margin of error of 2 points, not 3. To me that sounds like cutting it fine. The reason I chose 3 points is that many of the pools have about 1000 respondents, which makes the noise level (1 over the square root) about 3 percentage points. And systematic errors are bound to be at least this large.

Saturday, October 25, 2008

Git R Done

These are the worst economic times most of us have ever seen. In my own household, I have watched the market crash wipe out most of my daughter's college fund. Fortunately, college is still a long way off.

Right now, my wife and I believe the best investment in our daughter's future we can make is to repair the country she will inherit, and preserve the liberties we want her to enjoy long after we're gone. That's why we have just contributed $1,000.00 to help make sure that Barack Obama is elected president, and not John McCain (and not Sarah Palin).

Last month we contributed $500.00. But we think it's time to double down and get this thing done. The attacks on Obama are working in Florida and Ohio - two states with 47 electoral votes between them. McCain needs these votes desperately and will do anything to win them. The Obama-Biden campaign needs contributions to reinforce the message, get out the vote, and fight voter suppression on election day.

We can't afford to lose this fight. Let's win this goddam thing.

P.S. No pressure - but in case you want to add your own contribution, you can do so here.

Monday, October 13, 2008

The NACAC Report on Standardized Testing in College Admissions

The National Association for College Admission Counseling (NACAC) recently released a major report on the use and misuse of standardized tests in college admissions. The report has gotten a lot of press, including an editorial in the New York Times.

The major tests used nationwide for college admissions are the SAT and the ACT. I was dismayed by the extent to which the NACAC report treats these two tests as interchangeable. The two tests differ in a number of ways - for instance, in their approach to math. The ACT math test measures a specific set of math skills and techniques sampled from the early high school curriculum. The SAT math test, on the other hand, demands more of the kind of mental gymnastics we commonly call "reasoning."

(I think psychometricians agree with me that the two tests "measure different constructs." But honestly, all you have to do is work some practice tests and you'll see what I mean.)

The difference between the two exams is occasionally recognized in policy: the U.S. Department of Education has allowed Illinois to use the ACT for accountability purposes under NCLB (the test was tweaked to align better to Illinois's state standards), whereas to my knowledge the SAT has not been approved for state use. Even FairTest, no friend of either exam, admits that the ACT is more closely aligned with high school curricula than the SAT.

The NACAC report nevertheless treats the two exams as identical. Here's how the Executive Summary breaks down:
Number of times the acronym ACT appears in the Executive Summary in phrases like, "the ACT and the SAT": 13
Number of times the acronym ACT appears outside of such phrases: 0
Now let's compare this treatment with the research coverage in the bibliography of the report:
Number of titles in the bibliography: 82
Number of titles in the bibliography mentioning the acronym ACT: 3
Number of titles in the bibliography mentioning the SAT, PSAT, ETS, or College Board: 41
Half of the bibliography is specifically about the SAT and related programs; only 4% of the bibliography is specifically about the ACT. The heavy weighting of the evidence base towards the SAT does not seem to justify the drawing of identical conclusions about the two exams.

By the way, one of the three specific references to the ACT in the bibliography is a document entitled, "NACAC Commission on the Use of Standardized Tests: ACT's Response and Recommendations Regarding the Critical Issue Areas Identified by the Commission." Oddly, the author of this response is misidentified in the bibliography as NACAC itself. But in case anybody wants to read ACT's reply, the document can be found here. ACT actually affirms many of the report's recommendations. I have not seen this fact noted in press coverage of the commission's report.

There is a revealing article by Scott Jaschik at Inside Higher Ed about the NACAC report and the association's meeting in Seattle. Jaschik refers to the report as "the SAT report"; perhaps he read the bibliography too. At one point he writes of the goings-on at the association meeting:
Others who spoke at the forum and elsewhere at the meeting generally agreed. Some focused broadly on the SAT, while others had specific complaints - fees charged by the College Board, a new College Board policy making it easier for students to take the SAT repeatedly without reporting that to colleges, lack of oversight of the College Board. (While the NACAC commission’s recommendations were put forth to apply equally to the SAT and ACT, most of the anti-testing rhetoric at the meeting was directed at the SAT and the College Board, not the ACT.)

The following passage in Jaschik's article caught my eye:
To applause, [NACAC member Susan Tree] cited "fundamental flaws in a test that has become one that continues to correlate more highly with family income and educational background than academic promise."
The association between poverty and test scores has been much-studied and is not in doubt. What's more, this is not simply a case of correlation without causation. Believe me: it's causation. Poverty causes low test scores. But what people forget, or maybe don't notice, or just skip past, is that this is not a case of direct causation. Think like a physicist with me here for a moment. How are test scores generated? By scanning bubble sheets. So what, physically, are the direct causes of very low test scores? I can think of two.

(1) A very bad scanner malfunction.

(2) A lot of wrong answers.

Now, I really don't think the problem is that poor school districts are ending up with all the crappy scanners. Although we maybe should look into that. After all, they get most of the crappy other stuff. But pending the outcome of that investigation, I think we must believe that what's leading to the low test scores is that poor children are offering a lot of wrong answers.

And why is it the case that poor children give so many wrong answers?

The testing literature details many factors that lead to wrong answers. Test anxiety would be one example. On an individual level, such an explanation makes a lot of sense. We all remember times when we flubbed a test because of anxiety or a bad night's sleep. But when we're talking at the aggregate level, these explanations suffer from an order-of-magnitude problem. In our poorest districts, vast numbers of students score very near rock bottom. It would take a perfect storm of high anxiety, low motivation, and misaligned bubble sheets, not to mention a citywide spate of fire alarms and barking dogs on the day of the test, to explain a phenomenon of this magnitude. No. My inclination is to stick to an obvious and simpleminded explanation. Poor children are giving lots of wrong answers because they don't know very many of the right ones.

And why don't poor children know very many of the right answers?

Well. That's the question, isn't it.

The academic underperformance of poor children is one of the most persistent and serious failures of American public education. I don't know all of the mechanisms that contribute to the problem, any more than anyone else does. But I do know that if the tests go away, then we are flying blind. Last year, 12% of African Americans taking the ACT scored high enough to indicate likely success in college math; whites were four times as likely to score at college-ready levels. (See here, Figure 3.) If the ACT goes away, then so does this damning statistic; and so also disappears our ability to chart growth in response to interventions.

When progressives argue that the tests have to go away because of the way they correlate with poverty, they are cruelly wrong. The time to drop the tests is when they no longer correlate with student poverty.

Friday, October 3, 2008

Listen to the Martingale

Just a short post to recommend Jordan Ellenberg's article in Slate about the financial meltdown. Ellenberg uses a simple explanation of martingales to illustrate the tradeoff between risk and reward.
The [martingale] is a kind of "upside-down lottery." If you play the Powerball, you'll probably lose the cost of a ticket, but you might win big. In the martingale, you'll probably win a little, but if all six numbered balls match your ticket, then the bank comes around and takes away everything you've got.

In the article, Ellenberg refers to a little-known but interesting tidbit, which I also noted in my pamphlet Randomness and Reason - namely, that the ordinary process of coin-flipping slightly biases the coin to land the same way it started (51% vs. 49% or so). The link in Ellenberg's article is to Stanford's press release publicizing the research. Here is the actual study in case anyone wants to see:

Diaconis, P., S. Holmes, and R. Montgomery, "Dynamical Bias in the Coin Toss," http://www-stat.stanford.edu/~susan/papers/headswithJ.pdf