Methodology

The sample of respondents interviewed for NAES08-Online is a random subsample of KnowledgePanel, a nationally representative panel of US households maintained by Knowledge Networks. Below is a description of the methodology used to recruit and manage KnowledgePanel, followed by a description of the NAES-specific procedures for sampling and surveying KnowledgePanel members.

It is important to note that Knowledge Networks conducts research to improve KnowledgePanel, and has modified aspects of the panel methodology over time. The description below reflects KnowledgePanel procedures that were in effect during the period applicable to when NAES08-Online was fielded.

KnowledgePanel is a nationally representative, randomly sampled panel of US households, recruited on an ongoing basis, who agree to complete periodic Internet-based opinion surveys on a variety of topics. Households are provided Internet access to participate in KnowledgePanel if they do not already have it.

Although KnowledgePanel members complete surveys online, they were initially recruited through random-digit dialing of telephone numbers. The following procedures were used to identify and recruit households for KnowledgePanel through random-digit dialing:

As a starting point for KnowledgePanel sampling, a list of 10-digit US telephone numbers was randomly generated, inclusive of the 50 states and Washington, DC. All combinations of random digits were allowed, except those representing telephone banks (that is, groups of 100 numbers sharing the same first eight digits) known to contain no working numbers. Known cell phone numbers were excluded.

Randomly generated telephone numbers were matched against directories of business and government numbers, and screened to determine if they were non-working. Non-residential and non-working numbers were removed from the study.

For randomly generated telephone numbers believed to potentially represent households, an attempt was made to match numbers with postal addresses. Approximately 70% of telephone numbers were matched with postal addresses, and advance mailings were sent to these addresses. The advance mailings informed households that they had been selected to participate in KnowledgePanel and would be receiving a telephone call to join the panel.

Interviewers called randomly generated telephone numbers believed to potentially represent households in an attempt to confirm they actually represented households (those that did not were removed from the study), and to elicit household participation in KnowledgePanel. Each number was dialed up to 10 times over 90 days.

Upon reaching a household, interviewers administered a short recruitment questionnaire to determine the household's willingness to participate in KnowledgePanel, enumerate household members, and ensure Internet access:

  • If a household agreed to participate in KnowledgePanel and already had a home computer with Internet access and email, the email addresses of household members were collected for subsequent contact.
  • If a household agreed to participate in KnowledgePanel but did not have a computer or Internet access, a free WebTV and monthly Internet connection were provided. The WebTV was preconfigured with email addresses for the household, and installation support was made available as needed.
  • If a household refused to participate in KnowledgePanel, it was called back in an attempt to re-administer a recruitment interview and elicit participation.

Once a household agreed to participate in KnowledgePanel, individual household members were contacted by email to complete an initial profile. Completion of the initial profile established a respondent as an active member of KnowledgePanel, eligible to be sampled for surveys.

While it is possible for multiple household members to participate in KnowledgePanel, for NAES08-Online, only one member per household was sampled (see below).

KnowledgePanel began recruitment in 1999 and has continued recruitment on an ongoing basis thereafter. Ongoing recruitment replenishes the panel as members voluntarily leave. Ongoing recruitment also introduces variance in the length of panelist tenure (see below).

In addition to the random-digit dialing procedures described above, two sampling adjustments were incorporated in an attempt to maximize KnowledgePanel's representativeness and recruitment efficiency:

  • Oversampling of black and Hispanic telephone exchanges: Black and Hispanic respondents have historically been more difficult to recruit and retain on KnowledgePanel. To compensate for this, randomly generated telephone numbers were divided into two strata based on 2000 US Census data for race and ethnicity by telephone exchange. Numbers in exchanges with higher-than-average proportions of black and/or Hispanic households were oversampled at a rate of approximately 2:1 relative to numbers in exchanges with lower-than-average proportions of black and/or Hispanic households.
  • Undersampling of telephone numbers not matched to postal addresses: Among telephone numbers that were attempted to be matched against postal addresses, approximately 70% of numbers were successfully matched, and all address-matched telephone numbers were sent advance mailings and called to participate in KnowledgePanel. Among the approximately 30% of telephone numbers not matched to a postal address, a random subsample of between 33% and 75% of numbers was called. In addition to being undersampled, non-address matched numbers were not sent advance mailings, as no address was available.

Oversampling and undersampling introduce variation in the probability with which respondents were selected for KnowledgePanel. Sample weights are provided to statistically correct for this variation (see below).

Assignment to surveys: Once a panelist was established as an active member of KnowledgePanel, he or she was eligible to be randomly assigned to surveys. In general, panelists were assigned to no more than one survey per week (and additional restrictions were put in place for NAES08-Online; see below).

Panelists received notification of assignment to a survey via email, and a personalized home page on the KnowledgePanel portal listed all surveys currently pending for a panelist.

Incentives for households with Internet access: For households that had their own Internet access, points redeemable for cash were provided at regular intervals, contingent on survey participation, as an alternative incentive to the free WebTV and Internet connection provided to households that did not have Internet access.

Knowledge Networks' KnowledgePanel is the parent sample of NAES08-Online. NAES respondents were randomly selected from KnowledgePanel.

A total of 28,985 members of KnowledgePanel participated in NAES08-Online. NAES respondents participated in at least one of five NAES interviewing waves:

  • Wave 1 — pre-primary: Interviews conducted starting October 2, 2007, through January 1, 2008; 19,190 respondents participated in this wave
  • Wave 2 — primaries: Interviews conducted January 1, 2008, through March 31, 2008; n = 17,747
  • Wave 3 — spring and summer 2008: Interviews conducted April 2, 2008, through August 29, 2008; n = 20,052
  • Wave 4 — general election: Interviews conducted August 29, 2008, through November 4, 2008; n = 19,241
  • Wave 5 — post-election: Interviews conducted November 5, 2008, through January 31, 2009; n = 19,234

Participation in two waves: 23,033 NAES respondents participated in at least two NAES waves:

Waves

n

1

2

 

 

 

15,528

1

 

3

 

 

14,170

1

 

 

4

 

13,359

1

 

 

 

5

12,929

 

2

3

 

 

14,524

 

2

 

4

 

13,558

 

2

 

 

5

13,127

 

 

3

4

 

16,293

 

 

3

 

5

15,447

 

 

 

4

5

16,242

Participation in three waves: 18,666 NAES respondents participated in at least three NAES waves:

Waves

n

1

2

3

 

 

13,051

1

2

 

4

 

12,258

1

2

 

 

5

11,933

1

 

3

4

 

12,263

1

 

3

 

5

11,884

1

 

 

4

5

11,809

 

2

3

4

 

12,628

 

2

3

 

5

12,263

 

2

 

4

5

12,081

 

 

3

4

5

14,252

Participation in four waves: 14,308 NAES respondents participated in at least four NAES waves:

Waves

n

1

2

3

4

 

11,465

1

2

3

 

5

11,174

1

2

 

4

5

11,032

1

 

3

4

5

11,076

 

2

3

4

5

11,449

Participation in five waves: 10,472 NAES respondents participated in all five NAES waves.

NAES08-Online survey respondents were randomly selected from KnowledgePanel at the household level. Within households represented on KnowledgePanel, a maximum of one adult age 18 or older was invited to participate in the NAES.

If a selected household member did not respond to or refused the invitation to participate in the NAES, another household member was not eligible to be invited in his or her place.

The panel of NAES08-Online respondents was assembled dynamically over the course of the study:

  • For Wave 1, a random sample of KnowledgePanel members was invited to participate.
  • For Wave 2, all Wave 1 participants were invited to be reinterviewed. In addition, a random sample of KnowledgePanel members who were not invited to participate in Wave 1 was invited to participate.
  • For Waves 3 through 5, all participants of at least one previous NAES wave were invited to be reinterviewed. In addition, a random sample of KnowledgePanel members not previously invited to participate in the NAES was invited to participate.

(For example, if a respondent first participated in Wave 2 and declined to be reinterviewed for Wave 3, he or she was invited to participate in Wave 4, and regardless of participation in Wave 4, was invited for Wave 5.)

The following table shows the proportion of each wave's sample that constitutes respondents who were reinterviewed and those who were interviewed for the NAES for the first time:

Wave

Reinterview

First NAES interview

1

19,190

2

15,528

2,219

3

15,643

4,409

4

17,526

1,715

5

17,782

1,452

Each NAES wave consists of a mix of respondents who joined KnowledgePanel relatively recently — that is, within two months of the wave start — and respondents with longer tenures on KnowledgePanel:

Wave

Joined within two months of wave

Longer tenure

1

1,967

17,223

2

2,219

15,528

3

4,409

15,643

4

1,715

17,526

5

1,434

17,800

Variable TENURE indicates the length of tenure on KnowledgePanel.

Respondents were invited to take NAES surveys on a rolling basis, with a fixed period in which to accept the invitation.

For each NAES wave, invitations were issued to a random sample of respondents daily, and respondents were allowed 14 days to complete the survey:

  • Day 1: Initial invitation by email
  • Day 4: Email reminder
  • Day 7: Second email reminder
  • Day 11: Third email reminder
  • Day 15: Survey no longer available

These procedures were expanded for some difficult-to-reach demographic groups (see below).

The number of daily interviews completed for Waves 1 – 5 averaged between 105 and 295, depending on the month. On all except nine days, a minimum 50 interviews were completed:

Wave / Month

Interviews completed

Daily average

Daily maximum

Days less than 50

Wave 1

October 2007

7,146

238

321

0

November 2007

6,592

220

299

0

December 2007 (*)

5,452

170

257

2

Wave 2

January 2008

6,094

197

295

0

February 2008

5,835

201

295

0

March 2008

5,818

188

277

0

Wave 3

April 2008

4,004

138

221

0

May 2008

4,415

142

232

0

June 2008

4,404

147

198

0

July 2008

4,176

135

191

0

August 2008

3,053

105

190

4

Wave 4

September 2008 (**)

9,743

295

388

1

October 2008

8,412

271

345

0

November 2008

1,086

272

315

0

Wave 5

November 2008

7,043

271

615

0

December 2008

5,987

193

275

0

January 2009

6,204

200

289

2

* Includes January 1, 2008

** Includes August 29 – 31

Some demographic groups have historically been more difficult to recruit to KnowledgePanel and maintain as active on the panel. Difficult-to-reach groups include black and Hispanic respondents, young respondents age 18 – 29, and respondents with a high school diploma or less than high school education.

To maximize participation among difficult-to-reach groups, these respondents were offered a $5 incentive to participate in each NAES wave. The incentive was offered starting in the second email reminder sent on day 7 of the 14-day invitation period (see above).

In addition, for Waves 3 – 5, respondents in difficult-to-reach groups were permitted 21 days to accept an NAES survey invitation, instead of the standard 14, and on day 9 of the invitation period, they received an automated telephone call reminding them to participate in the survey.

As members of KnowledgePanel, NAES respondents typically were invited to participate in one new survey per week, be it an NAES survey or, more often, surveys for non-NAES research purposes. As a practical matter, two restrictions were placed on the spacing of interviews of NAES respondents:

  • KnowledgePanel members participating in NAES08-Online were not eligible for non-NAES surveys of KnowledgePanel related to the 2008 election.
  • To prevent the possibility of respondents completing an NAES survey at the end of a wave and being invited for an NAES reinterview at the beginning of the wave immediately following, NAES reinterview invitations were spaced in thirds. Respondents were classified by the third of the wave they initially participated in, and invited for reinterviews during the same thirds of all subsequent waves. (For example, a respondent who completed a first NAES interview during the second third of Wave 3 was invited to be reinterviewed during the second thirds of Waves 4 – 5.)

NAES questionnaires were presented in English. NAES respondents also completed profile questionnaires (Waves A and B) in English.

The data reported as profile Waves A – B are not specific to NAES. They are derived from interviews of KnowledgePanel that were designed and fielded by KnowledgeNetworks independently of NAES:

  • Wave A — core panel profile: Knowledge Networks administered the core panel profile, represented by Wave A, when a respondent joined KnowledgePanel, and re-administered the profile to each respondent annually. The data provided for Wave A represent the core panel profile completed most recently before the respondent's first interview for an NAES wave (that is, within one year before the respondent's first interview for an NAES wave).
  • Wave B — public affairs profile: Knowledge Networks administered an optional public affairs profile, represented by Wave B. This interview was typically conducted shortly before a respondent's first NAES interview, although in some cases it was conducted after the first NAES interview, and in cases of KnowledgePanel members with particularly long tenures, it was conducted more than one year in advance of the first NAES interview.

Two response rate calculations are provided for each of NAES Waves 1 – 5:

Response rate without retention: The response rate for an NAES wave, not factoring in retention on KnowledgePanel, is the product of the following:

  • Household recruitment rate — the response rate for the random-digit dial telephone survey used to recruit KnowledgePanel panelists
  • Household profile rate — the percentage of households recruited for KnowledgePanel in which at least one household member completed an initial profile and was established as an active member of KnowledgePanel
  • Wave completion rate — the percentage of respondents sampled for a wave who completed the wave

Response rate with retention: The response rate for an NAES wave, factoring in retention on KnowledgePanel, is the product of the above and the household retention rate.

The following table shows the response rate calculations for NAES Waves 1 – 5:

 

Wave 1

Wave 2

Wave 3

Wave 4

Wave 5

A. Household recruitment rate (*)

23.7%

22.4%

21.0%

21.2%

21.0%

B. Household profile rate (*)

55.3%

54.9%

55.1%

55.5%

55.7%

C. Wave completion rate

74.5%

77.4%

74.3%

73.1%

69.1%

RX – Response rate without retention

9.8%

9.5%

8.6%

8.6%

8.1%

D. Household retention rate (*)

40.6%

44.1%

47.5%

44.5%

44.8%

RT – Response rate with retention

4.0%

4.2%

4.1%

3.8%

3.6%

* The household recruitment, profile, and retention rates are estimated from the overall KnowledgePanel response rates for the respondent cohorts sampled for each NAES wave.

For more information about response rate calculations for online panel surveys, see Callegaro, M., DiSogra, C. Computing response metrics for online panels. Public Opinion Quarterly 2008;72(5):1008 – 1032.

Sample weights are provided to correct for the following known sources of error in the unweighted sample:

  • Oversampling of telephone exchanges with higher proportions of black and/or Hispanic households (see above)
  • Undersampling of telephone numbers not matched to a postal address (see above)
  • The higher probability of recruitment for KnowledgePanel among households with multiple telephone lines, and the higher probability of invitation to the NAES for respondents in households with fewer members
  • Undersampling of certain small areas of the United States where standard KnowledgePanel Internet equipment cannot be used, and where alternative equipment must be contracted for
  • Oversampling of certain geographic areas early in KnowledgePanel's construction, due to specific research needs at that time (the cities and states oversampled included Chicago and Los Angeles; California, Florida, New York, and Texas; and the central region states)

In addition, the weights correct for demographic deviation from the US Census Current Population Survey, given non-response and panel attrition. Through an iterative convergence process, weights were derived to optimally fit marginal distributions for the following demographics:

  • Sex
  • Age
  • Education
  • Household income
  • Race/ethnicity
  • Census region
  • Residency in a metropolitan area
  • Internet access (estimated from KnowledgePanel recruitment data)

Sample weights are optional and can be added to the data files of interest; separate weights are provided for each combination of NAES Wave(s) 1 – 5:

Wave(s) combination

n

Minimum weight

Maximum weight

Design effect

1

 

 

 

 

19,190

0.14

6.44

2.17

 

2

 

 

 

17,747

0.14

6.46

2.27

 

 

3

 

 

20,052

0.14

6.57

2.35

 

 

 

4

 

19,241

0.12

7.33

2.65

 

 

 

 

5

19,234

0.14

6.98

2.52

1

2

 

 

 

15,528

0.10

7.13

2.57

1

 

3

 

 

14,170

0.11

7.13

2.63

1

 

 

4

 

13,359

0.10

7.46

2.69

1

 

 

 

5

12,929

0.10

7.33

2.65

 

2

3

 

 

14,524

0.10

7.50

2.69

 

2

 

4

 

13,558

0.09

7.86

2.74

 

2

 

 

5

13,127

0.09

8.34

2.78

 

 

3

4

 

16,293

0.11

8.18

2.85

 

 

3

 

5

15,447

0.11

8.33

2.83

 

 

 

4

5

16,242

0.11

8.06

2.83

1

2

3

 

 

13,051

0.11

6.96

2.61

1

2

 

4

 

12,258

0.10

7.40

2.65

1

2

 

 

5

11,933

0.10

7.32

2.64

1

 

3

4

 

12,263

0.11

7.33

2.63

1

 

3

 

5

11,884

0.11

7.07

2.60

1

 

 

4

5

11,809

0.10

7.22

2.61

 

2

3

4

 

12,628

0.10

8.29

2.75

 

2

3

 

5

12,263

0.10

8.14

2.75

 

2

 

4

5

12,081

0.09

8.24

2.74

 

 

3

4

5

14,252

0.11

8.37

2.80

1

2

3

4

 

11,465

0.10

7.34

2.63

1

2

3

 

5

11,174

0.11

7.19

2.61

1

2

 

4

5

11,032

0.10

7.33

2.62

1

 

3

4

5

11,076

0.11

7.21

2.60

 

2

3

4

5

11,449

0.10

8.10

2.72

1

2

3

4

5

10,472

0.11

7.15

2.59