SAMPLING DESIGN:
Sampling design refers to the method used to select respondents to interview. This also is known as sampling method or sampling strategy.
The method of choosing telephone numbers must be statistically valid. The resulting sample must be a probability sample so that information obtained from the sample can be used to generalize results to the total population in the state, as well as to the nation as a whole. Samples used in the BRFSS must be probability samples in which all households with telephones have a known, non-zero chance of inclusion. Beginning in 1999, the BRFSS began to rely on CDC for its telephone sample purchased from GENESYS. This aimed to calculate and improve the probability of selection for inclusion in the survey.
The sampling currently used is a Disproportionate Stratified Sample Design (DSS). Disproportionate stratified random sampling is a variation of cluster sampling. For DSS, information obtained from other sources is used to classify 100 number blocks of telephone numbers into two strata based on the presumed density (high or low) of residential telephone numbers (strata that are either likely or unlikely to yield residential numbers). Telephone numbers in the "likely" strata are sampled at a higher rate than numbers in the unlikely strata. The rate at which each stratum is sampled is called the sampling rate. The GENESYS sample is divided into zero banks and one-plus banks. These values are determined by analyzing all possible 100 blocks for an area. The ratio of the sampling rate of one stratum to sampling rate of a reference stratum is called the sampling ratio. The recommended sampling ratio between one-plus blocks and zero blocks is 1.5:1.
The DSS design attempts to find a way of differentiating, before sampling begins, between a set of telephone numbers that contains a large proportion of target numbers (the high-density block) and a set that contains a smaller proportion of target numbers (the low-density block). It is possible to create more than two groups, but for BRFSS, only two groups are used. In this way, sampling telephone numbers is more efficient compared to simple random sampling.
From 1999 to 2001 and 2003 to 2009, the North Dakota BRFSS was conducted using DSS methodology with two strata. In 2002, the sampling method was slightly modified; DSS methodology with three strata was used.
STRATIFICATION:
Currently, the North Dakota BRFSS is conducted using disproportionate stratified sampling methodology that considers the entire state as a single geographical stratum. North Dakota does not stratify its sample by geography or racial/ethnic group. However, methodology that over samples certain minority populations can provide more accurate and representative health data on adults living in the state of North Dakota. In addition, over sampling specific geographical areas can ensure more useful estimates at the local level. Stratification is considered annually.
CELL PHONE SAMPLING:
Changes in telephone technology have resulted in more and more households that have cellular telephones and no traditional land lines in their homes. Prior to 2008, these households were not in the sampling frame for BRFSS. However, six states participated in a pilot cell phone sample survey in 2008. North Dakota began collecting cell phone surveys in 2009. For 2009 and 2010, the North Dakota BRFSS completed a minimum of 250 state-wide BRFSS interviews with cell phone only users. It became mandatory in 2011 for every state to complete a minimum of 10 percent of their total BRFSS completions with cell phone only users, with a goal of working towards 20 percent of their total BRFSS completions being completed with cell phone only users.
CELL PHONE METHODOLOGY:
Beginning in 2009, the land-line sample has been supplemented by a smaller cell phone only sample. Telephone numbers are sampled from the set of cell phone numbers assigned to North Dakota and activated. Unlike the land-line sample, cell phone sample cannot be tested for working numbers and business numbers, so yield from a bank of 100 numbers is much smaller for the cell phone only sample.
If the number called is incorrect or the person is younger than 18, the interview is not conducted. If it is not a safe time to conduct the survey (i.e., the person is driving), the respondent is called back at a later date. In addition to about 10 minutes of the BRFSS core health questions, a variety of questions are asked about cell telephone usage and whether the respondent lives in a household with a landline to establish if he or she was a dual-user (cell telephone and landline) or cell telephone-only user.
The cell phone survey protocol is available for downloading in PDF format.
RESPONDENT ELIGIBILITY:
Participation is random, anonymous and confidential. Respondents randomly are selected from among the adult members of the household. Only those living in households are surveyed. Those living in institutions (i.e., nursing homes, dormitories) are not surveyed. Criteria for inclusion in the survey includes the following:
- Confirm the correct telephone number was dialed;
- Live in a private household;
- Are 18 years of age or older;
- Current resident of state from which their telephone number was selected; and
- Confirm that we reached them on their cell telephone, not business only phone (for cell phone survey).
SAMPLE SIZE:
Since 2005, the North Dakota BRFSS has completed a minimum of 4,000 interviews with North Dakota adults annually. Sample size and response rate information is available HERE.
RESPONSE RATE:
The Council of American Survey Research Organizations (CASRO) formula is based on the number of interviews completed, the number of households reached, and the number of households with unknown eligibility status (e.g., households that were called 15 times but where no one in the household was reached). The CASRO response rate is used because in addition to those persons who refused to answer questions, lack of response also can arise because household members were not available despite repeated call attempts, or household members refuse to pick up the phone based on what they discern from caller I.D. Although some North Dakotans choose not to participate, North Dakota has one of the highest response rates in the nation. The Sample Size and Response Rate table includes the CASRO response rates for the North Dakota BRFSS for 1996 to 2009 by survey year.
DATA COLLECTION:
BRFSS data are collected by calendar year from January through December and follows standard data collection procedures. Since 1999, the North Dakota Department of Health has contracted out the BRFSS data collection to a survey company. Data are collected through monthly telephone interviews, and the Department of Health regularly monitors interviews to maintain and ensure data quality. Data are collected via computer using Ci3 CATI (Computer Assisted Telephone Interviewing) software. Approximately the same number of people are called each month to reduce bias caused by seasonal variation of health-risk behaviors. The interviews are conducted everyday including evenings and weekends. If there is no answer, the interviewer redials up to 15 times across a period of time before the telephone number is replaced. If the interviewer reaches a nonworking number or a business, calls to that number stop. If the interviewer reaches a household with more than one adult 18 or older, one of them is randomly selected for the interview. If the selected respondent is not available, an appointment is made to call at a later time or date. Because respondents are selected at random and no identifying information is requested, all responses to the survey are anonymous.
At the completion of the interviewing cycle each month, the survey company sends the data to CDC. At the end of the year, the CDC aggregates monthly data for the entire year. The Department of Health usually receives the annual dataset from the CDC in late spring following the survey administration year.
QUESTIONNAIRE DESIGN:
The BRFSS survey conducted by all states consists of three sections: the core modules (which include demographics), optional modules, and state-added modules. The Core section of the survey is consistent across all states as this section includes questions prescribed by the CDC. The optional modules are selected by the states from a bank of CDC-supported modules. Each state also may design its own state specific modules (state-added questions). The questionnaire covers such topics as Health Status, Health Care Access, Nutrition, Physical Activity, Diabetes, Tobacco Use (including Smokeless Tobacco), Alcohol Use, Demographics, Women's Health, Injury Prevention and HIV/AIDS Awareness.
DATA RELIABILITY:
Telephone interviewing has been demonstrated to be a reliable method for collecting behavioral risk data and can cost three to four times less than other interviewing methods such as mail-in interviews or face-to-face interviews. The BRFSS methodology has been utilized and evaluated by the CDC and other participating states since 1984. Content of survey questions, questionnaire design, data collection procedures, surveying techniques and editing procedures have been thoroughly evaluated to maintain overall data quality and to lessen the potential for bias within the population sample.
DATA WEIGHTING:
Unweighted data are the actual responses of each survey respondent. After the data collection is completed, data are cleaned and weighted by CDC. The data are weighted or adjusted to compensate for the overrepresentation or underrepresentation of persons in various subgroups. The data are further weighted to adjust the distribution of the sample data so that it reflects the total population of the sampled area.
Data Weighting Definition: Data weighting is an important statistical process that attempts to remove bias in the sample.
Purpose: Corrects for differences in the probability of selection due to non-response and non-coverage errors.
- Adjusts variables of age, race and gender between the sample and the entire population.
- Allows the generalization of findings to the whole population, not just those who respond to the survey.
- Allows comparability of data (to other states, to national data, etc.).
Implications: Design factors affect weighting. In the BRFSS, these factors include:
- Number of residential telephones in household.
- Number of adults in household.
- Geographic or density stratification.
In addition, there is a post-stratification by age and gender that adjusts for non-coverage and non-response and forces the sum of the weighted frequencies to equal population estimates for the state.
DATA ANALYSIS:
BRFSS data contain information on non-institutionalized North Dakotan adults ages 18 years and older. Data collected by BRFSS are edited using PCEdits software produced by the CDC. Edit reports are produced monthly and corrections made. Corrected data files and edit reports are sent to the CDC monthly. At the end of each survey year, data are compiled and weighted by CDC, and cross tabulations and prevalence reports are prepared.
SAS/SUDAAN OR SPSS software is recommended for use in conducting data analysis because BRFSS is not based on simple random sampling. Confidence intervals are important measures of the role of chance.
LIMITATIONS:
Sampling: The BRFSS survey samples the population using a technique that is discussed above. Sampling yields results that are an estimate of the true answer for the entire population. Survey response rates also may affect the potential for bias in the data. The reliability of a prevalence estimate depends on the actual, unweighted number of respondents in a category or demographic subgroup (not a weighted number). Interpreting and reporting weighted numbers that are based on a small, unweighted number of respondents can be misleading. The degree of precision increases if the sample size is larger and decreases if the sample size is smaller. In some cases, the estimate may become too uncertain to be of value.
Prevalence estimates usually are not reported for those categories in which there are less than 50 respondents or the confidence interval half width is greater than ten.
The BRFSS uses telephone interviewing for several reasons. Telephone interviews are faster and less expensive than face to face interviews. The one main limitation of any telephone survey is that those people without phones cannot be reached and are not represented. Since phone ownership is highly correlated to income, persons without a phone are more likely to have low incomes than persons with a telephone. This potentially will affect questions with responses that are highly dependent on income (e.g., health insurance) more than other questions. In addition, because the questionnaire is asked in English in North Dakota, adults who are not able to be interviewed in English are not included in the sample. As a result, BRFSS findings only can be generalized to English speaking adults living in households with telephones. However, because phone ownership is high in North Dakota (greater than 95 percent), it is unlikely that failing to reach these persons will substantially alter results. National BRFSS results correspond well with findings from other surveys conducted in person.
(Unlisted telephone numbers are included in the sample through the random dialing method that is used.)
Questionnaire Administration: The BRFSS relies on information reported directly by the respondent. As such, this self-reported data may be subject to a number of sources of possible error. How questions are worded may influence responses in a certain way and can result in what is called "measurement error." Similarly, the ability of individuals to accurately recall details is subject to "response error."
Not all the questions used in the survey have been tested to ensure that all persons understand the intended meaning. Those that come from modules created by the Centers for Disease Control and Prevention usually have been tested, while those in state modules may or may not have been tested, depending on the source of the question. Furthermore, not all questions are equally easy for respondents to answer. While it may be easy for a respondent to provide a personal opinion, it may be much harder to recall a past event (last mammogram) or provide factual information (household income).
Interviewers are trained and monitored to ensure that they administer the survey in a neutral voice and read the written question verbatim and without comment. Nonetheless, it is possible for the interviewer to bias the results through tone of voice or administration technique. Coding errors also may occur if the interviewer types in the wrong response to the question. In addition, the person being interviewed may alter his or her response to give the interviewer the most socially acceptable answer. This may be a problem especially for questions that may have a perceived stigma (e.g., HIV risk).
Response Rate: The bias from non-response cannot be removed and it is not possible to know if those who refused to respond would have answered the questions in approximately the same ways as those who responded.
Confounding and Causation: Personal characteristics that are presented on this website are univariate (i.e., examine each risk factor in relationship to only one characteristic at a time); however, the complexity of health associations are not fully represented by examining single relationships. For example, an examination of heart disease and employment status might show a greater prevalence of heart disease among persons who are retired than among persons who are employed. However, persons who are retired are expected to have a greater average age than persons who are employed; consequently, this relationship might entirely disappear if we removed the effects of age. (If this were the case, we would say that the relationship between heart disease and employment status was being confounded by age.)
Likewise, this website does not attempt to explain the causes of the health effects examined. For instance, BRFSS data might show a higher prevalence of heart disease among smokers, but one should not conclude from this that smoking causes heart disease. That smoking is indeed a causal factor for heart disease is apparent from a large body of scientific data, but that is not a conclusion that can be drawn from a cross-sectional survey such as this. Rather this is a "snapshot" of disease, risk factors and population characteristics for adult residents of North Dakota at a point in time.
QUALITY CONTROL:
Quality control issues are very important to the North Dakota Behavioral Risk Factor Surveillance System (BRFSS). The data we collect must be as accurate as possible to ensure that it represents the self-reported health and beliefs of the people of North Dakota. To provide consistent and timely quality control, a number of tasks are performed before and during the data collection process, and after data collection is complete.
First, prior to data collection, interviewers are extensively trained. Newly hired interviewers go through an intense two-day training session that covers basic rules and interviewing techniques, the importance of voice quality, respectful probing and refusal conversion techniques. Interviewers also are provided with an introduction to the computer-assisted telephone interviewing (CATI) system, an overview of BRFSS and a project specific briefing. Confidentiality is emphasized, instruction in human subjects’ criteria is given and cultural competency training helps interviewers understand respondents with diverse backgrounds. Training ends with mock interviews. Trainers and mentors/monitors are stationed near new interviewers when they begin to conduct interviews with the project sample, to provide answers to questions about scripts, the CATI system, or workstations; to help with any problems; and to provide feedback and encouragement. New interviewers are monitored closely during their first several assignments and complete a 90-day probationary period during which performance is evaluated. In addition, all interviewers participate in semi-annual drama training/voice coaching from a professional.
Second, prior to programming the North Dakota BRFSS survey questionnaire in the CATI system, the contractor conducts a comprehensive review of the survey instrument and data layout. Assembly of the survey into the CATI system occurs next followed by three rounds of pretesting to ensure that there is understanding of the questions being asked, the possible responses to each question and proper pronunciation. Throughout data collection, emphasis is placed on quality interviewing, a productive environment and time management.
Next, data are edited on a monthly basis. The contractor performs comprehensive data inspection procedures at the end of each monthly field period using a data-editing program developed by the CDC specifically for BRFSS. Following verification procedures, the data are reviewed again using a comprehensive frequency checker. This allows for the identification and correction of any data errors and/or outliers in a timely manner. These processes are thorough and allow for monthly submission of BRFSS data in accordance with CDC requirements. The BRFSS User’s Guide and numbered memos found on the CDC website are strictly followed. Data collected from the cell phone survey (deleted asthma part since it is not on website currently) are submitted on a quarterly basis, also in accordance with CDC requirements.
Finally, the contractor utilizes a comprehensive systemic daily monitoring process, administered in an unobtrusive manner to validate completed surveys, verify the accuracy and completeness of data entered and provide timely feedback to interviewers. The North Dakota BRFSS program director also monitors interviews conducted by the contractor via a remote monitoring system that allows the program director to view the interviewer’s screen while listening to the audio via a secure website. In addition, the program director makes an annual site visit to the contractor to ensure operations are carried out according to CDC guidelines.
These processes and the CDC’s BRFSS protocol are important to ensure quality control. Training, monitoring and data editing ensure that the sample we collect is of the best quality possible. Quality control is an ongoing task, which helps to guarantee that the BRFSS data are highly representative of the health behaviors and opinions of the people in North Dakota.