In my user research methods class, I worked in a group where we had the University of Michigan's College of Engineering as a client. In the latter part of 2016, the College of Engineering is to undergo a redesign. My team was tasked with finding usability issues with the current graduate education section of the greater College of Engineering site. We used various user research techniques to accomplish our goals. We created an interaction map of the current graduate education website, conducted user interviews, developed personas and scenarios, conducted a competitive analysis, created and distributed a survey, conducted heuristic evaluations, and conducted usability tests.
The Problem
When my team and I met with stakeholders, we learned that the existing site is not allowing students access to desired resources. They found this out through a focus group, but could not disclose the specific details because they could not get consent from the participants to share details. My team interviewed users of our primary target audience, current graduate students, to find out what resources were important to them (e.g. financial, research opportunities, courses, careers etc.).
Our Goals
Our team had a few goals for this overall project:
1) Find out what resources were important to graduate students
2) Find usability issues regarding access to these resources.
My Roles
Helped create questions for user interviews Conducted and took notes for user interviews, contributed in creating personas and scenarios, identified competitors and compared competing sites to ours, participated in creating a survey sent out to graduate students, conducted a heuristic evaluation using Jakob Nielson's 10 heuristics, and conducted and took notes for usability tests.
Personas, Interviews, and Scenarios
Our first assignment was to identify our user audiences, interview representatives of this user base, and create personas and scenarios off of the data we obtained. We determined that our primary user audience was current graduate students. Other user audiences include prospective students, current College of Engineering faculty and staff, and alumni. We made sure that we had a mix of domestic and international users as well as having female user representation.
Our primary way for recruiting users was through our team's network of graduate student friends. Following recommended guidelines set by our instructor, we interviewed six participants, five students and one stakeholder. Our students came from a variety of engineering majors, including mechanical engineering, industrial and operations engineering, electrical engineering, and computer science.
Listed below are the questions that we asked each of users. Depending on the answer to these questions, we probed with additional questions to learn more about their search behavior.
1. In the last two weeks how many times did you use the engineering academic website?
2. What is your major?
3. What is your main purpose for using this website?
4. When you registered for the courses, what kind of information did you look for in the website?
5. Have you looked for information regarding faculty/ staff on this website?
6. Have you looked at this website for information regarding funding resources?
7. Did you experience any frustrations and confusion when you used the website?
8. How do you find information about your program?
9. How do you find out about research opportunities at the college of engineering?
Through our interviews, we found that very few students use the College of Engineering Graduate Education website. All of the majors offered in the College of Engineering have their own separate website and four out of our five students used the website of their major to attempt to find desired information.
Our team made a list of recommendations based off of these findings. Major recommendations include grouping similar information spread across many of the Graduate Education pages into related sections, make all forms of financial resources available online, update and maintain the current calendar with all upcoming events, and create a page specifically for international student information.
Our team did have a few limitations and issues when recruiting and interviewing users. The first is that our sample size was small and thus not statistically significant. Being that this project is done in a learning context and thus having a short time frame to complete this assignment (around 2 weeks), we were told to only interview 6-8 users. This inherently did not include users of all backgrounds. We were only able to interview current students and one stakeholder per assignment guidelines and thus unable to include other target user populations. We were also not able to cover the spread of ethnic and racial diversity of users in the College of Engineering. We also chose friends of teammates to interview as we believed this was the most time efficient way of getting results.
Since the student users were friends of one or more of teammates, this could have biased some of the answers we received.
The second limitation involved our interview questions. We initially designed our questions to learn about our user's habits when using the site. Due to most of our student users not being familiar with the site, we had to modify our questions as we could not get accurate information about how they use the website.
Competitive Analysis
From our initial stakeholder meeting, we learned that the College of Engineering wanted to be competitive with the top ten engineering programs in the county, including their website. This information provided the basis of our choice in competing schools. Part of the requirements for this assignment included us choosing competitors that are either indirect, partial, parallel, and analogous. We chose one parallel system, the University of Michigan's Ross School of Business, and one analogous system, the University of Michigan's Biomedical Engineering department. Defined, a parallel system is one that offers the same kind of service to a similar audience via a similar channel and an analogous system is a system that provides has similar functions and can provide ideas as to how our client can improve their service. For our direct competitors, we chose seven of the other nine schools that were listed in U.S. World News and Report (The University of Michigan's College of Engineering was listed in the top ten). We chose the University of California Berkeley, Carnegie Mellon, California Institute of Technology, Purdue, Stanford, and the University of Illinois Urbana-Champaign. We based our comparisons on a list of prioritized elements given to us by one of our stakeholders and we created a chart comparing our competing schools with U of M's College of Engineering.
From our comparisons, we were able to make a few recommendations. The first was to improve the responsiveness of the website. On mobile screens, there were overlapping text and pictures. A big recommendation was to create a separate section dedicated to career resources. All of our direct competitors had a career resources page. Many of our competitors did not have a list of student engineering organizations and this is an area our group believed could give the College of Engineering an edge.
Survey
The creation and distribution of a survey was our third major assignment. We asked questions we felt would be beneficial to our analysis of the site. Topics included: awareness of the existence of the Graduate Education website (based off of our earlier findings), how often they visited if they were aware of the website's existence, how helpful the website was in answering certain topics, how much effort it takes to find resources, and the importance of certain resources. We piloted the survey on users before distributing the final survey. The final survey was sent out through the College of Engineering. A reward was offered at random to one of the respondents after the deadline for responses.
Survey Questions
Below are the survey questions we asked:
Q1. Are you currently enrolled as an engineering graduate student at the University of Michigan?
Yes
No
Q2. Were you aware of the existence of the College of Engineering's Graduate Education Website?
Yes
No
Q3. During this semester how many times did you access the Graduate Education website?
0
1 - 2
3 - 5
6 - 10
> 10
Q4. How helpful have you found the Graduate Education Website for finding relevant information for each of the 8 sections below? Please enter a selection for each section. (1: not very helpful - 4: very helpful; 5: not applicable)
Academics
Funding Resources
Research
Workshops
Events
Student Life
Support Services
Admissions
Q5. Typically how much effort does your search require on the Graduate Education Website before you find your answer?
A great deal
A lot
A moderate amount
A little
None at all
Q6. How likely are you to direct other engineering graduate students to the Graduate Education Website for information?
Extremely likely
Somewhat likely
Neither likely nor unlikely
Somewhat unlikely
Extremely unlikely
Q7. Please rank the following 8 sections based on their importance to you (1 represents the most important and 8 represents the least important) irrespective of whether you have used the Graduate Education Website or not. These are the sections currently available on the Graduate Education Website. A number once entered, cannot be repeated for another section.
______ Academics
______ Funding Resources
______ Research
______ Workshops and Events
______ Student Resources
______ Student Life
______ Support Services
______ Admissions
Q8. How do you find information on the following topics (see Q7). Please select all that apply (see below).
Graduate Education Website
Engineering Department Website
Search Engine (Google, Bing)
Friends
Alumni
Email
Other
We found that the majority of users were not aware of the websites existence, or were aware of its existence, but rarely used it. Of those who were aware of the website, the majority of participants found the website to be only moderately helpful at best. This trend carried through when we asked how easy it was to find desired information about select resources. We asked all participants, whether they knew of the website's existence or not, what resources were most important to them. This was asked by ranking the given categories. Categories included: academics, funding resources, research, workshops and events, student resources, student life, support services, and admissions. Academics, workshops and events, research, and financial resources were the top desired resources for our website. Users generally needed to spend some effort looking for resources and many were indifferent as to whether they would recommend using the site to find information.
Heuristic Evaluation
We conducted a heuristic evaluation on the College of Engineering Graduate Education website using Jakob Nielson's ten heuristics. Each team member evaluated the website separately first using each of the heuristics and we came back together as a team to discuss our findings.
In conducting our heuristic evaluations, we performed many of the common tasks that a user would take to access the required information. One of the biggest findings that we found was that many of the web pages did not have a consistent layout throughout the site. This is because the various web pages are maintained by different departments. By having constantly varying layouts, it requires the user to give extra effort in looking for desired information or links to information on each page.
The other major finding involved the layout of the website.
There is an a lot of content laid out on the home page that gives users information overload. Much of the content in many of the sections is either not labeled well or is not appropriate for the content block or site (e.g. undergraduate student research in a graduate student section of the greater College of Engineering site).
Usability Test
Our final assignment was to conduct a usability test on the website. Each usability test was conducted in rooms similar to conference rooms. Our users were current graduate students who were not familiar with the College of Engineering Graduate Education section. Each member of the team was given a chance to be a moderator who read off of a pre-made script and as a note taker. A pre and post survey was handed out during each session. Our team created a list of tasks that would be relevant to what graduate students would perform. Tasks included: finding financial information, registering for classes, setting up what is called a CAEN account (needed to gain access to College of Engineering computer labs), finding workshops and events, and career resources.
We measured the duration of time needed to complete each task and how the path they took to find the desired information (if they could find it).
Pre Survey
What is your highest education level?
What is your field of study?
Which computer system are you familiar with, PC or Mac?
How much time do you spend on a computer per day on average in the last two weeks?
Which kind of information do you think you will find on this website?
Moderator Transcript & Tasks
Hi [user’s name], my name is [moderator’s name], and thank you so much for coming in today. We really appreciate it. I'm going to be walking you through this session. I’ll be reading from a script today to make sure I don’t miss anything and that I provide clear and consistent instructions. You probably already know, but let me explain why we've asked you to come here today: we are working on the University of Michigan College of Engineering’s Graduate Programs Website and want to understand how users feel when they go through the website.
The first thing I want to make clear right away is that we’re testing the site, not you. You can’t do anything wrong here. In fact, this is probably the one place today where you don’t have to worry about making mistakes. You can stop the test at any time, if you choose to.
As you use the site, I’m going to ask you as much as possible to try to think out loud: to say what you’re looking at, what you’re trying to do, and what you’re thinking. This will be a big help to us. Also, please don’t worry that you’re going to hurt our feelings. We’re doing this to improve the site, so we need to hear your honest reactions.
If you have any questions, please feel free to ask. However, I may not be able to answer them right away. We would like to know how you would use the site when no one else is around. When you feel that you have completed a task, or when you want to stop trying, please let me know. We have a list of tasks and I might urge you to move on. It is perfectly alright if a task is left incomplete.
You may have noticed the camera. With your permission, we're going to videotape the computer screen and what you have to say. The video will be used only to help us figure out how to improve the site, and it won't be seen by anyone except the people working on the project.
If you would, I'm going to ask you to sign something for us. It simply says that we have your permission to tape you, but that it will only be seen by the people working on the project. Do you have any questions before we begin?
Task 1
Imagine you are an engineering graduate student studying biomedical engineering. You feel tuition is way too expensive. Your parents have told you that you need to find ways to reduce your expenses for the program.
Task: On the website please find the names of at least 1 person as well as 1 program that can provide you with more information on this topic.
Task 2
You want to register for classes next semester.
Please find the name of a 500 level class from any department other than biomedical engineering worth 3 credits.
Please find the syllabus for that class as well.
Task 3
In order to use the computers in the engineering computer labs, you need to have a CAEN account set up. Please find information on how you would get the CAEN account set up.
Task 4
You are looking to find fun things to do on campus. You heard of a really fun event called ‘Quantum Coffee Break’. Find information about the ‘Quantum Coffee Break’ on the website.
Task 5
You are in your second semester of your graduate program. It is time to start applying for internships.
Please find a campus resource to have your resume reviewed.
On the website please find a page with information for career fairs.
Post Survey
How do you think the session went?
What are you impressions of the website?
Here are the tasks we went over today. What did you think of each of them:
Financial Support
Course List
Setting up CAEN Account
Social Event
Internship Support
Please name three changes you would like to make to the website.
Do you have any other comments or suggestions?
From our usability tests, we found that users first looked at the top navigation bar. Many were unsuccessful at finding the information they needed through the navigation bar. Using the navigation bar ourselves, only two of our five tasks could be completed through the navigation bar. All of our tasks were based on the content blocks and links found farther down the page. We recommended that highly desired content like financial and career resources be found in the main navigation of the page.
Our team also found that career resources in particular were particularly hard to find as there wasn't a designated section anywhere on the page. We recommended that the redesigned website have a career resource page in the navigation.
Users also often felt lost when they needed to backtrack. This is because many of the links found take a user to different sections of a website altogether. These sections had different layouts then the main Graduate Education page. We recommended that these resources be consolidated into one site and that a breadcrumb be provided so users know where they are when exploring the site.
Another point that should be brought up is that after unsuccessful attempts to look in the navigation bar and content sections on the page, users used a provided search bar to complete the tasks.
What I Learned
Through this experience, I learned valuable skills other than those taught in the classroom. My time management skills were tested with the project. My team had five members and coordinating schedules was at times a challenge. Additionally, we also had to work around the schedules of our participants in various assignments. Each of the projects only had two weeks to be completed. All of these factors made each assignment deadline seem as real-world as it could be.
My team was culturally diverse, which allowed me to experience different ways of thinking and different viewpoints at multiple times when working on these assignments.