BASICS- JMU data 2009-2010- a bit of a dissapointment

In Fall 2009, the University Health Center’s Substance Abuse Prevention started to formally evaluate the effectiveness of BASICS at JMU. The evaluation includes a pre-test, 1-month posttest, and 3-month posttest design.  I wrote to JMU and requested under the Virginia Freedom of Information Act a copy of their data evaluating the effectiveness of the BASICS program at JMU. I received the report and below are some highlights.

From the report here is a description of the BASICS program:

James Madison University’s (JMU) Health Center Substance Abuse Prevention program coordinates the Brief Alcohol Screening and Intervention for College Students (BASICS). BASICS is an evidence based, early intervention alcohol abuse program developed for a college student population (Dimeff, Baer, Kivlahan, & Marlatt, 1999). The intervention targets college students who drink heavily and have experienced or are at great risk for experiencing negative consequences as a result of their excessive alcohol consumption. The BASICS program consists of two fifty-minute interview sessions during which students engage in an empathetic, non-confrontational, and non-judgmental discussion with a BASICS program representative. During the first BASICS meeting session, students provide general information to the BASICS representative concerning their substance use by responding to and discussing a self-report instrument. The second BASICS meeting session allows the student and BASICS representative to engage in a feedback session that encourages students to evaluate their drinking behavior, potential risks or consequences associated with this behavior, and potential changes they may make in their behavior to reduce future substance abuse related issues.   JMU students may refer themselves to the BASICS program or be referred by JMU administrators, faculty/staff, coaches, friends. Students may also be mandated to participate inBASICS programming by JMU Judicial Affairs as a result of a JMU alcohol policy violation or a court mandate to complete substance abuse prevention hours. Students who participate in BASICS complete a mandatory pre-test, are encouraged to complete a 1-month post-test, and an additional 3-month post-test. Each assessment contains identical measures including the Daily Drinking Questionnaire (Collins, Parks, & Marlatt, 1985), the Alcohol Use Disorders Identification Test (AUDIT; Babor, Biddle-Higgins, Saunders, & Monteiro, 2001; Saunders, Aasland, Babor, de la Fuente, & Grant, 1993), the Rutgers Alcohol Problem Index (RAPI; White & Labouvie, 1989), and the Situational Temptations Scale – short form (Maddock, Laforge, & Rossi, 2000), and Student Make-Changes Ruler(modified from the Readiness Ruler; Center for Evidence-Based Practices, Case Western Reserve University, 2010).

The EXECUTIVE Summary notes:

Overall, students with complete pre-, 1-month post-test, and 3-month post-test data either maintained or decreased their levels of drinking, negative consequences associated with alcohol consumption, and level of temptation to drink excessive amounts of alcohol.
One-Way Repeated-Measure ANOVA and post hoc dependent t-test results indicated a significant decrease in student mean drink quantity, peak BAC, and meant total AUDIT scores. Although differences were non-significant, there were decreases in student mean drink frequency, mean total RAPI score, and mean total Situational Temptations Scale score from pre-, to post-test 1, and post-test 2. Descriptive statistics indicated that student self-reported binge drinking remained the same across pre-, post-test 1, and post-test 2. Furthermore, descriptive statistics indicated that student responses specifying their current pattern for cutting back or abstaining from drinking did differ across pre-, post-test 1, and post-test 2.

DETAILS IN REPORT

Table 1 presents the number of participants who completed the BASICS assessments across the three time points. A total of 59 participants completed the BASICS pre-test, post-test 1, post-test 3, or any combination of the three assessments. Note that only 19 participants had complete data for the pre- post-test 1, and post-test 2 items, thus these were the participants used in the analyses reported below. There seems to be a major issue with participants completing the entirety of the tests across the three test administrations. This could be an issue of program attrition, student failure to respond to all pre- and post-test assessment items, or incorrectly entering identification numbers across test administrations. Several participants were removed from the data set due to invalid and missing student identification numbers. However, the major issue seemed to be due to student ID mis-match across assessment administrations. This data merge issue indicates that students either entered their student ID numbers incorrectly or did not complete assessments at all three time periods. As the small sample size severely limits the analyses in this report, we strongly encourage BASICS administrators to take steps in the future to ensure that a larger proportion of participants complete the assessment at all three time points.

Table 1.
Sample Sizes for Pre-Test, Post-Test 1, and Post-Test 2.

Time Point                   Total Sample                 Complete Responses      Matched Pre and Post
Pre-Test                              65                                      56                                    N/A
Post-Test 1                         68                                       67                                     21
Post-Test 2                         59                                       59                                     19

********************************
As I read this report it appears that of the 59 participants only 19 completed all three phases of the evaluation of effectiveness of the BASICS program.  Later in the report it emphasizes that:
     "As noted previously, the analyses are exploratory in nature and the results should be interpreted with       caution."
Using the data from the 2009-2010 Judicial Affairs report this would mean that out of the 3575 sanctions, only 59 led to referral to the BASICS program.  With complete data sets on only 19 of the 59, the effectiveness of JMU's BASICS program is really unknown at this time.  I found myself disappointed in the quality of this evaluative research.  Even if the evaluative research demonstrated valid data about the effectiveness of the JMU BASICS program, it provided services to only 59 students.  This is a very small number of students considering the number of sanctions and the size and scope of the negative alcohol culture at JMU.