This project deals with the analysis of the impact of Netflix's Feedback System on it's overall User Experience


UX Researcher in a team of 4

Inciting Incident

Netflix recently changed its feedback system. In 2017, Netflix switched from the traditional ‘5-star’ rating system for its shows and movies to a ‘Thumbs-Up/Thumbs-Down’ system. In 2018, Netflix removed all user reviews from its platform. As a Netflix user, we wanted to understand the impact of these changes in the long-standing feedback system. Hence, we decided to research how it will impact the overall user experience of Netflix.

Rising Action

We browsed through papers and articles for existing research on how feedback systems impact user experience. Collected data from Google Trends. Leveraged surveys, interviews and focus groups to better understand the mindset of Netflix users for the feedback system and its effect on the overall user experience. Formulated relevant questions for surveys, interviews and focus groups on identifying the kind of information required from the subjects and phrasing the questions aptly to comply with ethics and standards of surveying people. Scheduled interviews and assigned team-members to act as facilitators and note-takers during interviews for smooth transitioning of the process.

Observation Method

Survey: We selected survey to understand user’s perspective about how they feel about the recent changes in the Netflix’s feedback and recommendation system. The survey contained 7 questions. We reached out to 100 participants and received 82 responses which is a conversion rate of 82%.
Interview: We recruited 8 people for an interview in a social setting that helped us get an objective approach to determine how Netflix users traversed their website and mobile application. The interviews also helped us understand how participants find relevant reviews and recommendation for selecting TV shows and movies that match their interests.

Sampling procedure and selection of participants

1. Cluster sampling: We selected groups of people who use online streaming services with respect to their geographical locations.
2. Convenient sampling: We reached out to participants for interviews that are easily accessible to us such as students at the University of Washington.
3. Purposive sampling: We reached out people groups of people that are pro-Netflix and people who are critical about Netflix.

Criteria for sample selection

Inclusion Criteria:We included a mix of people who use Netflix and those who don’t primarily from the US.
Exclusion Criteria: We excluded those who didn’t match the inclusion criteria.

Data Collection Procedure

Survey:Survey: The survey was structured using Qualtrics with a strong emphasis on anonymity to have an objective set of answers from the users. We used normal data responses along with comparative response questions. The data collected was then stored in password protected MS Excel spreadsheet. Finally, we used Tableau to draw inferences from the data.
Interview: We took notes on participant responses and reactions to our set of questions. Furthermore, we used pictorial representations of the participant’s flow of responses. To make the participants more comfortable with the questions, we made our interview more flexible based on their responses

output output
Collecting and Analyzing Research Data

Determined that the recent changes to the feedback system by Netflix have not directly impacted the user experience. This was reflected in our survey results where close to 75% of participants mentioned that they didn’t notice the removal of user reviews from Netflix. However, when presented with the screenshots of the Old and New implementations of Netflix’s feedback system, close to 70% of the users preferred the old implementation with ‘5-star’ rating system and user reviews. This was in alignment with our findings from the interviews where participants mentioned that having the older implementation of the feedback system can significantly improve their user experience on Netflix.

Findings from Quantitative analysis:

1. 65% of the users considered reviews and ratings before selecting the next show they want to watch which highlights the importance of a good review and rating system for delivering a quality user experience.
2. 77% of the users mentioned that review and rating system plays an important role while deciding the choice for a video streaming platform. This strengthens the claim that a feedback system plays an important role in the overall user experience.
3. However, about 75% of users did not notice that Netflix removed its user reviews from the platform. This indicated that a majority of the users were not actively using the reviews on Netflix
4. On the contrary, when presented with a choice between the old and new feedback systems of Netflix, 75% of the users preferred the old implementation with user reviews and ‘5-star’ rating.

Findings from Affinity analysis:

1.Review System People expressed the need for user reviews but did not notice that Netflix removed them from its platform. This was in line with the survey results we received. The major reason observed was that users accessed Netflix through the Mobile App 80% of the times and reviews were only available on the website. We identified this as consistency issues with user experience across platforms. Strong sentiments like ‘Ridiculous Move’ , ‘Ways to avoid criticism’ etc. were also recorded during the interviews.
2. Rating SystemIn context of the rating system, the interview participants stood divided with respect to their choices. Some participants preferred the old 5-star rating system since it gave them a comprehensive view about other people’s views and the flexibility to express their views in a better way. However, some users thought that the ‘Thumbs-Up/Thumbs-Down’ system is easier to use and interpret.
3. User Suggestions: Majority of the interviewees indicated that they follow reviews and suggestions from friends and would like to see the same on Netflix. This will enable them to get an aggregated view of suggestions from friends without even contacting them personally. We marked this as an additional theme that came out only from the interviews because we didn’t involve any suggestion related questions in the survey.
4. Organization Perspective: ONetflix’s official statement about removing user reviews was centered around their declining use. The same can be confirmed by searching ‘How do I post reviews on Netflix’ on their help page. It also claimed that writing a bad review never had any bearing on whether a title was recommended to a user or not.

Conclusion and Recommendations

The research suggested that Netflix can better its user experience by bringing back user reviews and the ‘5-star’ rating system. To ensure that these features are used to their maximum potential, Netflix can launch a new marketing campaign for the same. Netflix can invest in bringing consistency in its feedback system across all its platforms like the Website, Web Application, and Mobile Application.