5 Non-Obvious Takeaways from 4 Hours of Conversations with NCARB

I chatted with different senior NCARB officials at the AIA National Conference in Chicago. Below are my takeaways (number four is scandalous!)

1. Most people who fail an exam division fail it by just a little bit.

I’ve suspected this for years, but it was confirmed for me on Thursday. When an organization like NCARB administers an exam to a large number of humans, scores will tend to cluster in the middle ranges.

Graphed out, this makes a “normal curve” (bell-shaped curve). Because the cut scores NCARB has set for the ARE happen to fall so that about half of test-takers pass and half fail, the passing threshold sits near the peak of the curve, near where most people’s scores cluster.

Therefore, many people (perhaps most people) who pass only barely pass; and many people (perhaps most people) who fail also fail by a narrow margin.

This feels non-intuitive to test takers who assume that when they see a passing score, they must have done everything right; when they fail they assume that they must have studied incorrectly. If you fail, you likely failed by just a bit so don’t start over!

Schedule a division retake for as soon as possible– before you forget all that you learned the first time around.

2. How do I know if I failed by just a bit?

NCARB wanted to let you know, in your fail report, how many questions you failed by, but their psychometricians threw a fit. Psychometricians are the experts who measure the repeatability, validity, and fairness of an exam, mostly using numbers.

NCARB has long been beholden to their recommendations as testing professionals, which maddens me, but, in fairness, I can think of no important national or state-wide exam that reveals to you your raw score. (SAT, GRE, ACT, and the state test scores your kids have to take all translate your raw score into a non-intuitive number. Why is 1600 a perfect SAT score?).

Exams report mysterious scaled scores to normalize results across different test forms. You’ll need to answer about two-thirds of the test items correctly to pass, but each of the six exam divisions, and each of the forms within a division are a bit different. Using a test bank of more than 10,000 questions, NCARB assembles and launches four new test forms for each of the six divisions every year.

Based on past results, test makers (modestly) adjust cut scores to account for difficulty variations across exam forms, so a more difficult version of the exam requires fewer correct answers to pass. This makes for a fairer exam experience but also makes for an exceptionally opaque score reporting system. 

Where can I find my scaled score and what does it mean?

You find it here, where the red box sits. 

550 or above is passing, but you’ll never see your scaled score when you pass (feeding into the myth that you must have done everything right to pass, when in reality, you likely barely passed).

A score of 520 roughly translates to “missed by one question”

A score of 508 roughly translates to “missed by two questions”

A score of 500 roughly translates to “missed by three questions”

. . . And on down the mysteriously-weighted scale. I asked NCARB to publish this scale (or at least share the raw math with me so that I could translate it for the community) and they seemed to think that was a good idea. I got an email suggesting that such an explanation was pending. I hope their psychometricians don’t kill the idea.

3. Volunteer architects instead of experts decide what technical content should be on the exam.

NCARB, in part blinded by the seemingly incontrovertible math put forward by their psychometricians, does not recognize what I believe to be the fatal flaw in the ARE: letting volunteer architects instead of experts decide what technical content should be on the exam.

This, at first blush, appears to be righteous democracy at work, but I’ll illustrate its substantial downside with an example. Building scientists tell us that, in approximate order of importance, a low-energy building should prioritize:

First, limiting window-to-wall ratio; you almost certainly can’t create a low-energy building if more than 30% of the wall is glazed.

Second, air-tightness; some new buildings, by virtue of their detailing and construction oversight are twenty times leakier than others.

Third, window performance; shading or low-e films for windows that see sun and low U-values for windows in mixed or cold climates.

And a distant fourth, insulation and R-value in the opaque skin.

If you’ve taken an ARE exam you’d think number four on that list–insulation in opaque walls–gets you most of the way to an efficient building, and you won’t see number one or number two anywhere. Why? Because that’s what architects believe.

Why? Because that is what they were tested on years ago when they took the exam.

Why? Because 50 years ago, when many buildings had no insulation in their opaque skin, and high-performance windows weren’t available, and we didn’t air-condition nearly as much, and the field of building science was less mature, we saw adding insulation as a panacea.

And misplaced priorities echo down through the generations of architects making their own exams without expert oversight. So we make well-insulated buildings that hemorrhage air with low-performance windows and too much glass (and lord knows, I love full-height glass too , but still).

4. "There’s some truth to that rumor, kind of.”

Over breakfast, I was lamenting the conspiratorial zeitgeist: that someone on Facebook last week suggested that NCARB must have sold me names and contact information for test-takers (which, of course, never happened). . .

When the NCARB official raised his eyebrow, leaned in perceptibly, and said, “You know, there’s some truth to that rumor, kind of.”

It seems that another test prep provider sent Freedom of Information requests to a number of state boards (not NCARB) threatening legal action if the state boards didn’t hand over lists of test-takers. Many of the more courageous state boards resisted but some were cowed by the letters and released the lists of your names!

I’m not allowed to reveal which provider it was, but you can ask your state board or other state boards (perhaps with your own Freedom of Information request, or maybe a simple query is all that’s needed) who sent the letter looking for your information. . .  and publish your own blog post in the NCARB forum or ARE Facebook group. Crazy, right?

5. I was again struck by the well-meaning nature of the three NCARB officials I spoke with.

I hate the way they decide what subjects to test on, both because of the lack of experts and because the balance of subjects seems way out of whack favoring pro-practice . . . and no gatekeeper in any field earns popularity. . . . but to me, NCARB staff seem consistently earnest, transparent, professional, and dedicated to the practice.

Fam, please don’t turn on me for publishing number five. Don’t hate the player, hate the game.