Variations2
Usability Studies:
A Report for the Project Midterm Review
Mark Notess
All-Investigators Meeting, Bloomington, 12 November 2002

Agenda
Accomplishments to date
Publications and presentations
Future plans

Accomplishments
Studied current user practices in order to
Establish a usability baseline for future comparison
Guide design decisions
Investigated emergent interaction technologies
Evaluated Variations2 usability via
Contextual interviews with prototypes
Lab-based usability testing of prototypes and finished releases
Pilot studies of actual use

A Preliminary Definition
Usability = ease of use + usefulness
Ease of Use
Effectiveness:  accuracy and completeness of goal achievement, quality of solution
Efficiency:  time to achieve goal
Satisfaction:  attitude towards system
Usefulness
Fit with users
Fit with tasks
Fit with contexts of use

Accomplishments
Baseline established
Usability test of Variations (June 2001)
Contextual inquiry into
music classroom pedagogical practices (July 2001)
Variations use in library (January 2002)
Information habits of voice students (October 2002)
Variations satisfaction survey of current users (October 2002)

Accomplishments

Other Baseline Findings
High satisfaction with Variations (5.6, 1-7  scale)
For most students, if it’s not online, it isn’t there.
Complaints about Variations:
Long retrieval time for non-cached recordings
Delays when moving around within a recording
Restricted to use within the library

Other Baseline Findings
Example of student having to adapt to tool limitations:  how to find song length
Move slider carefully
to end of track and
look at number--a
difficult and inaccurate
operation.

Other Baseline Findings
Design impact:
We gave track times
higher visibility in the
new system.
(Typical of results from contextual studies.)

Accomplishments
Study current user practices in order to
Establish a usability baseline for future comparison
Guide design decisions
Investigate emergent interaction technologies
Evaluate Variations2 usability via
Contextual interviews with prototypes
Lab-based usability testing of prototypes and finished releases
Pilot studies of actual use

Information Visualization
Experimented with off-the-shelf visualization tool, SpotFire DecisionSite, using 20,000+ item database of musical works from classical.net.
Prototyped customized visualizations based on user research.

Information Visualization
Off-the-shelf tool results
Users must be experts
Entire database must reside on client
No viable pricing model

Information Visualization
Concept Prototypes
Visual search results
Visual course reserve list
Based on observed needs--common music DL usage scenarios
Require new integration of currently disparate systems

Information Visualization
Visual search results

Accomplishments
Study current user practices in order to
Establish a usability baseline for future comparison
Guide design decisions
Investigate emergent interaction technologies
Evaluate Variations2 usability via
Contextual interviews with prototypes
Lab-based usability testing of prototypes and finished releases
Pilot studies of actual use

Prototype Interviews
Low-fidelity paper prototypes of design ideas
Paper levels the playing field--users can confidently become co-designers
Real users in their real work context (library) “interact” with the prototypes
Rapid design/feedback iterations speed design process

Usability Tests
Multimedia Music Theory Teaching (MMTT) prototype lessons (March 2002)
Variations2 pre-release version (July 2002)

Usability Tests
MMTT prototype lessons
Complexity is a challenging opportunity
Students see high value

Usability Tests
Variations2 pre-release version
Tested with both music and non-music students
Liked new capabilities, especially fast navigation of audio, having score access
Revealed issues with
entering and retrieving search queries
bookmarking
controlling scores

Pilot Studies
Designed two Variations2 pilot studies for Fall 2002 (http://www.dml.indiana.edu/pilot/)
Graduate voice student Schubert listening assignment
Undergraduate theory class sonata form analysis assignment
Data collection methods
Satisfaction survey
Automated usage logging

Pilot Studies
Schubert listening assignment
7 songs
Scores, texts, recordings
30 students
In progress

Pilot Studies
Sonata form analysis
1 movement
Students create bookmarks and annotate
Can check answers
200 students
Starts this week

Pilot Studies
Data collection

Publications and Presentations
Publications
Dillon, Andrew. Technologies of information: HCI and the digital library.  In Carroll, J. (ed.) HCI and the New Millenium, Addison-Wesley, 2001.
Notess, Mark. Applying contextual design to educational software development.  To appear in Armstrong, A. (ed.) Instructional Design in the Real World, Idea Group, 2003.
Conference Paper/Presentation
Notess, Mark, and Minibayeva, Natalia. Variations2: towards visual interfaces for digital libraries.  Presented at Second International Workshop on Visual Interfaces to Digital Libraries at the ACM+IEEE Joint Conference on Digital Libraries, July 18th, 2002, Portland, Oregon.

Future Plans
Pilot studies - broaden scope, work with satellite sites
Usability tests of current and future versions
Detailed studies of multimedia complex interaction issues (will use eye-tracking system)
Field studies:
Variations2 use
Instructors doing lesson preparation
Classroom technology use
Build and test functional prototypes of visual interfaces
Longitudinal user satisfaction studies examining predictive value of lab test satisfaction results

Usability Staff
Co-PI: Andrew Dillon (now at UT-Austin)
Usability Specialist: Mark Notess
Graduate Assistant: Margaret Swan
For more information, contact mnotess@indiana.edu

Grant Information
This material is based upon work supported by the National Science Foundation under Grant No. 9909068.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.