Expert review and data analytics integration for a data-aware design

One of the design challenges we often encounter at OpenKnowledge is the need to analyze an existing digital product and gather insights for its redesign. To address this need, one of the most frequently applied methodologies is the UX expert review, a qualitative technical survey that allows us to identify product strengths and weaknesses. The goal is to identify usability issues and/or friction points through a cross-sectional analysis of digital product components, using different lenses of inquiry; key areas include: visual design, interaction, and accessibility.

A UX Expert’s analysis differs from other research methodologies such as interviews or focus groups (a qualitative research method involving moderated discussion among a small group of people) in that it does not involve direct user participation. However, it offers several advantages, including:

• Quickly and inexpensively providing valuable information for product (re)design.
• Enabling in-depth exploration of specific aspects of user experience.

Given the above advantages, our experience led us to consider how to build on this established methodology to develop a new framework that addresses its weaknesses, such as the lack of user involvement and the absence of supporting quantitative data; to further extend its potential and fields of application.

We were guided by the belief that alternating multiple research methodologies and combining different skills allows to achieve a holistic view of the usability issues of a digital product and a comprehensive, informed interpretation of the available data. With this perspective, it is possible to redesign experience from a human-centered but data-aware perspective, combining the informational potential of qualitative data with the interpretive richness of quantitative data.

This way of approaching a research process is very close to the framework of data-aware design, a strategic approach that emphasizes the integration and effective use of data so that it becomes a real design material. Unlike the more familiar data-driven design, this approach does not simply view data as the sole source of information but recognizes it as one of the decision-making drivers of the design process, which, therefore, must be collected, analyzed, and critically read in harmony with all sources of information.

Designing according to the data-aware design approach means developing an awareness that not all friction points identified during the analyses performed will have the same impact on end users: what an expert analysis suggests might have a higher priority for action may instead go unnoticed in the eyes of real users. Therefore, a quantitative assessment of actual user browsing behavior allows us to consciously advance a prioritization of these elements in view of digital product (re)design.

In the initial stages, to identify evidence in an unbiased manner, both expert review and analytical study are conducted at the same time. This approach allows for an overall assessment free of bias and enables the identification (each using its own method) of the strengths and weaknesses of the digital product.

The data collected in expert evaluation allows data analysts to focus on specific sections or elements that are considered pain points in the user’s browsing flow. At the same time, an overview of the analysis (which includes traffic sources, behavior patterns, pages or sections of greatest interest, etc.) can be used to determine the most important elements in the user’s browsing experience. The results of a wide-ranging analysis then allow for progress in prioritizing the most relevant pain points, as well as guiding the next stage of detailed analysis.

Then, the results are shared and discussed within the research team to initiate a detailed analysis in which how users interact with specific components of the product will be assessed to determine areas of overlap in the initial evidence (by monitoring, for example, the number of clicks on a call-to-action, scroll rate, number of “dead clicks“, or dwell time).

Especially from what emerged from the expert analysis, the navigation data allows us to confirm or reject the hypotheses advanced, integrating a supporting quantitative dataset, as well as to map a number of elements that may have been omitted from a design assessment, while still needing due analysis. Indeed, it is important to point out how a distance may exist between these assessments and actual user behavior. A distance that, in our experience, can be reduced by the proposed framework.

The new data collected in the analytics are then examined along with the design evidence to link the latter with graphical/interactive elements that might be responsible for certain patterns of user behavior, thus integrating a qualitative interpretation over the pure numerical data recorded.

These perspectives lay the groundwork for a comparison that takes design and analytics into account, allowing both methodologies to be enriched in a process that reinforces their strengths and limits their weaknesses, resulting in a single, integrated and more comprehensive view.

It allows a cross-sectional analysis of the digital product to be conducted and action prioritized on one or more sections, thus facilitating the success of the (re)design process. In addition, insights are defined in a shorter time and at a lower cost (compared to other research methodologies that directly involve users).

The collaboration between these two areas of expertise does not simply merge the collected data but proposes an iterative process in which qualitative and quantitative analyses influence each other, producing a set of concrete, actionable insights. The results obtained from the expert review form the basis for quantitative analyses and vice versa, until they converge into a structured analysis that allows the data to be highlighted in their various perspectives.

Data, collected, analyzed and interpreted from a multidisciplinary perspective, thus become the basis for data-aware (re)design decisions and successful product strategies.

Stefano Lissoni, Stella Maria Ventura

Rochelle King, Elizabeth F. Churchill e Caitlin Tan «Designing with Data: Improving the User Experience with A/B Testing» O’Reilly Media, 2017