BACKGROUND


Sabrina is guest lecturing in a course on Global Health at the Mount Sinai Medical School on March 9th from 6-8 pm. Sabrina has delivered a similar lecture three other times to different cohorts of medical students at Mount Sinai over the past few years. The lecture is titled Designing and Implementing a Survey, and is based off of a week-long course offered over the summer that she created for professionals at Mailman (Epidemiology and Public Health summer Institute at Columbia - EPIC).

General Class Structure

The course runs throughout the spring semester and students are required to attend some sessions and others are optional (Sabrina's is required). Students receive course credit for the class and most importantly, use skills learned in these sessions in their summer research experiences. Student make-up include medical students and MPH-MD dual degree students. There are about 50 - 80 students per cohort, but as the class is recorded attendance is usually around 50. About half of the sessions are guest speakers (as Sabrina's will be) and the other half are delivered by faculty that are affiliated with Mount Sinai Medical School.

Pre-observation Day Activities


Faculty and administration (email February 5, 2015) and students (emailed February 9, 2015 with administration at Mount Sinai) have been primed for Eric's participation in the course as a peer observer.

OBSERVATION DAY


On March 9th, Eric will observe Sabrina's class from 6:00 to 8:00 pm (meeting in the Mount Sinai lobby at 5:45pm).


Session Description:

Study design and data collection are the foundations of public health and population-based research. Competence in measurement and survey design are essential for conducting studies and critically evaluating published literature. This session exposed students to basic skills required for conducting standardized scientifically based observational research, specifically quantitative survey instrument design. Students will have an opportunity to apply concepts and techniques from class directly to the development of their own draft observational study instrument section

Learning objective for this class:
  • By the end of the session, students will be able to:
    • Identify key considerations when selecting between common modern data collection methods
    • Discuss advantages & disadvantages of different question types
    • Identify common pitfalls in survey design, especially focused on designing surveys for use outside the United States


How learning goal helps the instructor define his teaching:
  • Sabrina would like to make very concrete the idea that survey design is a science, art, and skill. That due diligence and research should go into the preparation, testing, and implementation of all surveys. In addition to teaching skills, this lecture needs to impart that skills are needed for the task at hand. Sabrina has at times struggled to adequately frame or describe the rationale behind this topic, taking for granted that all audiences understand and adequately respect the need for survey research science. The introduction of the online tool inclass will hopefully enable Sabrina to show in a very concrete fashion challenges with question writing so that the challenges stay with students longer than the content learned in the lecture.

Sabrina's vision of how students will step towards that goal:
  • Students will work towards this goal of understanding the rationale behind survey design and some basic components to consider when designing and implementing a survey through first understanding that this is an important topic (focus of evaluation), then through applied in-class pair work will get a sense for how challenging survey design can actually be, thus leading to their eventual understanding that there are core components of survey design (taught in the lecture) and that central to any successful study is the thoughtful and scientific-based implementation of a survey.

Eric's understanding of what exactly he is tracking:
  • Eric’s main task is to observe whether in-class computer surveys are useful in keeping students engaged in lecture. Sabrina will introduce a new technology learning tool she has never used before—computer surveys—and wants to know whether these tools are effective in keeping students engaged with the lecture material, or if they’re are more of a distraction. During the class, students will be taught specific kinds of survey methods—how to write knowledge-based surveys, attitude-based ones, and behavior-based ones. Student will then be given sample survey questions of each kind which they’ll answer on their own laptop. Eric will be observing the students and taking down notes on a pre-made list of factors that Sabrina will make. For instance, Eric will measure whether students are paying attention just before a survey is asked of them. If they’re engaged up until the point of the survey, then that suggests that if students end up not doing the survey when Sabrina prompts them, they may have got derailed on the internet.

Eric's understanding of how the tracked session or assignment will be measured:
  • Sabrina will provide Eric with a basic list of factors she thinks will tell her whether the computer-based survey is keeping students engaged or distracts them. The factors will include things like how many students appear to be engaged right before, during and after each of the three surveys is introduced. Other factors include issues like how well Sabrina appears to explain the survey method she was introducing; the time it took students to do each survey; and even a few mundane factors, like problems with the room that may have made it difficult for students to engage.

Observation Activities

In-class Observation
  • Eric will complete the evaluation check-list on student engagement during the new online survey component of the lecture (see attached excel for appropriate coloring and formatting).
  • After the lecture has concluded Sabrina will put one final poll on the board and ask students to reply before leaving:
    • What stood out to you most about tonight's lecture? (open field)
    • Did you use your computer/phone/tablet for off-topic purposes during class? (yes/no)



Theme (intervention condition)
Slide #
In general, students in the classroom are …
Time slide goes up
Time students responses are in
Qualitative notes on classroom engagement, participation, distraction
Knowledge (no online polling)
10
1. completely distracted
2. somewhat distracted
3. as engaged as they were when the previous slide went up
4. somewhat more engaged with the class than when the previous slide went up
5. very much more engaged with the class than when the previous slide went up





1. participating less than when the previous slide
2. participating the same amount as when the previous slide
3. participating more than when the previous slide went up



11
1. completely distracted
2. somewhat distracted
3. as engaged as they were when the previous slide went up
4. somewhat more engaged with the class than when the previous slide went up
5. very much more engaged with the class than when the previous slide went up





1. participating less than when the previous slide
2. participating the same amount as when the previous slide
3. participating more than when the previous slide went up



12
1. completely distracted
2. somewhat distracted
3. as engaged as they were when the previous slide went up
4. somewhat more engaged with the class than when the previous slide went up
5. very much more engaged with the class than when the previous slide went up





1. participating less than when the previous slide
2. participating the same amount as when the previous slide
3. participating more than when the previous slide went up



Attitude (online polling at beginning)
15
1. completely distracted
2. somewhat distracted
3. as engaged as they were when the previous slide went up
4. somewhat more engaged with the class than when the previous slide went up
5. very much more engaged with the class than when the previous slide went up





1. participating less than when the previous slide
2. participating the same amount as when the previous slide
3. participating more than when the previous slide went up




16
1. completely distracted
2. somewhat distracted
3. as engaged as they were when the previous slide went up
4. somewhat more engaged with the class than when the previous slide went up
5. very much more engaged with the class than when the previous slide went up






1. participating less than when the previous slide
2. participating the same amount as when the previous slide
3. participating more than when the previous slide went up



17
1. completely distracted
2. somewhat distracted
3. as engaged as they were when the previous slide went up
4. somewhat more engaged with the class than when the previous slide went up
5. very much more engaged with the class than when the previous slide went up






1. participating less than when the previous slide
2. participating the same amount as when the previous slide
3. participating more than when the previous slide went up




Behavior (online polling at end)
30
1. completely distracted
2. somewhat distracted
3. as engaged as they were when the previous slide went up
4. somewhat more engaged with the class than when the previous slide went up
5. very much more engaged with the class than when the previous slide went up






1. participating less than when the previous slide
2. participating the same amount as when the previous slide
3. participating more than when the previous slide went up"




31
1. completely distracted
2. somewhat distracted
3. as engaged as they were when the previous slide went up
4. somewhat more engaged with the class than when the previous slide went up
5. very much more engaged with the class than when the previous slide went up





1. participating less than when the previous slide
2. participating the same amount as when the previous slide
3. participating more than when the previous slide went up"




32
1. completely distracted
2. somewhat distracted
3. as engaged as they were when the previous slide went up
4. somewhat more engaged with the class than when the previous slide went up
5. very much more engaged with the class than when the previous slide went up






1. participating less than when the previous slide
2. participating the same amount as when the previous slide
3. participating more than when the previous slide went up






After-class Review
  • Eric and Sabrina will review quantitative information from the online program to get a sense of how many participants responded to each item and conduct item-level analysis (Sabrina) to see if certain items, order, etc., were responsible for lower item response rates. Open-ended item about what stood out about class most will be analyzed along with data on off-topic use of electronic devices.

Potential implications of observation findings
  • Sabrina teaches this exact lecture twice a year and has a summer short-course on the topic but has yet to introduce this level of technology for fear of introducing more distractions into the classroom. If students maintain or improve engagement after the online survey items then Sabrina will make this part of her regular teaching. Information on timing of the online survey will also guide Sabrina when deciding how many such exercises to include in one lecture.

PROVISIONAL SCHEDULE


February 5th - Sabrina and Eric discuss observation plan
February 10th - Final draft posted

February 12th - Discuss observation plan at Teagle meeting
- Observation plan locked

March 9th - Eric will observe Sabrina's class
5:45 pm - Eric arrives at class and discusses final details of observation with Sabrina, Eric confirms that everything is set for class evaluation
6:00 pm - Sabrina starts class
6:00 - 8 pm - Eric completes observation instrument while Sabrina teaches.
6:30 - Knowledge (no online polling) begins

6:50 - Attitude (online polling at beginning) begins

8:15 - Behavior (online polling at end) begins



March - Sabrina & Eric debrief
April - Final Observation report due
TBD - Final observation report submitted