2013 Logo 3d gradient 
SEGway       June  2016
Research and Assessment News from SEG Measurement   
In This Issue




Follow us on Twitter




View our profile on LinkedIn



31 Pheasant Run

New Hope Pennsylvania 18938

800.254.7670

 267.759.0617


First, I want to thank Anne Wujcik, editor of EdNET Insight, for recognizing SEG Measurement's SEGway newsletter in the most recent issue of EdNET Insight.  We rely on EdNET at SEG as a critical source of news in education.
"In the April issue of SEGway, Scott Elliot offered insights on the different types of evidence that can be gathered to provide proof of product effectiveness, provided a quick review of test validation and answered some basic questions about conducting an efficacy study. Scott follows this pattern in each monthly issue and I always come away having learned (or been reminded) of something useful." 

We hope that you find SEGway to be a valuable resource of information about assessment and effectiveness research.  Please let us know if you have any questions or topics of interest for future editions.  Also, a warm welcome to all of our new subscribers.  We appreciate your interest and all that you do in education, research, and assessment. 

In this issue, we discuss the importance of usage analytics in educational programs and we address the issue of defining and measuring engagement. We also continue our series of providing answers to your questions. This installment focuses on recruiting schools and teachers to participate in research studies.

We encourage you to learn more about our work in assessment and efficacy research by connecting with us at conferences. We look forward to continuing the discussion with you through the newsletter, on Twitter (@segmeasure) and on LinkedIn (SEG Measurement).  

Ta ke a look at our website at  www.segmeasurement.com , as it is frequently updated with developments in the field.  And,  feel free to email me at selliot@segmeasurement.com . I always look forward to hearing from you.

Sincerely,

Scott Signature  


Scott Elliot
SEG Measurement 
Four Benefits of Usage Analytics in Effectiveness Research
Using teacher and student usage data to improve instruction

Initially, educators and vendors used usage data to manage subscriptions.  But, usage analytics have grown into a rich source of information that can be used to investigate engagement, predict performance, and provide other valuable feedback regarding usage.

Usage analytics are a very valuable tool in effectiveness research. Here are four key benefits:
  • Monitor fidelity of use: Ensuring that the product is actually used in the way it is prescribed as the "treatment" in a research study. Typically when studying the effectiveness of a product, there is a prescribed amount of usage defined by amount of time used, extent to which certain features are used, completion of a specified amount of content, or some combination of factors. Usage analytics help to quickly identify participants who may have fallen behind in compliance. Acting upon this information quickly helps to prevent attrition of participants for lack of fidelity as a result of insufficient use of the product.
  • Identify needs for product enhancements: In an environment in which the participants are able to use the product in their own way that best meets their instructional or learning goals, patterns in usage can alert you to features that folks are missing or avoiding. Usage data can effectively identify underused tools and help in pinpointing tools that are hard to find, too cumbersome, misunderstood, or not functioning properly. This can help to inform future product development updates.
  • Investigate and define best practices for successful implementations: Combining usage data with performance data or other outcomes can provide a picture of patterns of implementation that are associated with achieving optimal outcomes of interest. These best practices can then be shared with others so that other users can benefit from the findings.
  • Build predictive models for student course completion and performance: Usage data can be incorporated with other data to build predictive models that can be used to provide information in real time regarding whether someone is likely to complete or achieve success in the course based on the usage data.
If your product does not have usage analytics or usage reports, or if your product is not digital, we have alternative methods for regularly tracking usage of participants in effectiveness studies. These include regular check-ins and interviews with participants in which they can detail their recent use, brief surveys in which the teachers or students attest to the features used and the time spent on a regular basis, and direct observations of usage. We can also help you to develop plans for collecting the most meaningful usage data within your product.

Visit our website for more details at: www.segmeasurement.com.
Defining and Measuring Engagement
How do we know whether students are engaged with a product or instruction?
 
Many educators believe that engagement in school and specific course content is central to overall academic and life success.  In the past couple of years, we have seen a dramatic increase in the number of publishers who are interested in measuring engagement and whether their product helps to increase students' engagement. Engagement has many components, including affective, behavioral, and cognitive:

Affective- This includes whether students have an interest or liking towards the product itself, the subject matter included in the product, class, or school in general.

Behavioral- This includes whether students behave in ways that show engagement with the product, the subject, or with school.

Cognitive- This includes whether students persevere and are willing to exert cognitive effort to learn the subject matter and use the product.

SEG has designed measures that help to gather information regarding these components of engagement. Measures typically involve data collection from multiple sources including the product itself (i.e., usage data, performance data), the students (through surveys, interviews, or observations), and teachers (through surveys, interviews, or observations).

We typically recommend gathering baseline engagement data prior to the start of the study and then again at the end of the study and during the course of the study as appropriate. We also recommend gathering engagement data regarding the users of the product as well as those students in the control classes who are not using the product.

There are challenges in gathering accurate engagement data and the interpretation of the data must be conducted with great care to avoid drawing conclusions that are not fully supported by the data.

Please contact us to help you plan and execute this work at info@segmeasurement.com or 800-254-7670. 

Recent Questions Asked by Educators and Publishers Considering Conducting an Efficacy Study

Here is the latest installment of this popular section of the newsletter with common questions and answers. This month we are focusing on recruitment questions.  If you have specific questions that are not covered here or if you are interested in exploring any of these questions further, please contact us.  We are here to help.
 
Do you have any tips for recruitment success?
We have a few recommendations that will help ensure a successful recruitment of participants in a study:
  • Do the work up front to invite schools and teachers that are truly engaged with your product; they are most likely to be interested in participating.
  • Spread the word to announce the call for participants through multiple channels and make it easy for participants to find you.
  • Highlight the reasons for the study and what will be done with the results, and highlight how students and teachers participating will benefit directly.

  • Show that you value the participants' time and energy and have planned the study to minimize the impact on instructional time.  

  • Provide reasonable product and monetary incentives to make participation in the study attractive.

  • Anticipate possible objections to participation and work to minimize expected concerns or anticipated roadblocks to participation.

You do not need to take on recruitment on your own.  We can work with you to help with recruitment.

 

Should we use current users or new users for our treatment group?
The answer to this question depends on a few factors including the size of your experienced user base, the extent of the learning curve for a successful implementation, and the design and research goals for your study.  For practical purposes, many of our clients decide that they wish to include current customers who are already familiar with the product.  This minimizes initial training, technical support, and other logistical needs.  The potential downside is that new users may not find the same results as experienced users.  With some of our clients, we have recruited entirely new participants and randomly assigned them to treatment (using the product for the first time) or control (not using the product during the study).  This model benefits from the randomization of the sample and investigating new users, yet requires more support and has inherent challenges with ensuring that the teachers are prepared to integrate the tool into their instruction.

Do we have to include treatment and control students from the same schools?
It is not a requirement that a school must have both treatment and control classes participating.  In many cases, an entire school or grade within a school participates as either treatment or control participants.  In those cases, for each treatment school participant, a similar school is selected to participate as part of the control group.  Depending on the overall targeted sample size and school/district/state/region representation, we can recommend the most appropriate approach to meet the needs of your study.

What questions do you have?  Let us know what questions you want answered.  Contact
info@segmeasurement.com  to share your feedback.

 


Contact SEG today to find out how we can help you establish the effectiveness of your education product or service:
hrickert@segmeasurement.com      267-759-0617 ext 102
SEG At Upcoming Conference s
Let's Meet!
 
We are looking forward to seeing our colleagues and meeting new friends at the upcoming conferences.  We are participating in several sessions & we invite you to join us.

   Look for us at these upcoming conferences:
  • CCSSO National Conference on Student Assessment, June 20 - 22, Philadelphia, PA
  • AACE EdMedia 2016, June 27 - 30, Vancouver, Canada
  • EdNet 2016, June 25 - 27, Dallas, Texas
  • NJEA Convention, October 10, Atlantic City, NJ
  • NCTM Regional Conference, October 31 - November 2, Philadelphia, PA
  We would love to meet with you and discuss how we can help you build strong assessments   and get the proof of effectiveness you need for success.  

If you would like to meet with a representative from SEG Measurement to discuss how we       might help you with your assessment and research needs, please contact us           at  info@segmeasurement.com.
About SEG Measurement
Building Better Assessments and Evaluating Product Efficacy
medical-team-portrait.jpg
SEG Measurement conducts technically sound product efficacy research for educational p ublishers, technology providers, government agencies and other educational organizations, and helps organizations build better assessments. We have been meeting the research and assessment needs of organizations since 1979. SEG Measurement is located in New Hope, Pennsylvania and can be accessed on the web at