There are different types of usability testing taking place online or in-presence. Due to the current situation and regulation of Covid-19, the user evolution study will be a monitored usability testing where the result of the case study will be observed remotely. The user testing will take place in the prototyping phase and will be performed by the co-design participant/user.
User evaluation
The scope of this evaluation is to gauge the effectiveness of the proposed design solution by testing whether the intended goal of the product is met. The purpose of the wearable is to keep the user aware of their own conversations as well as provide a physical confirmation that the user has already made his/her point. The intended outcome is a reduction in repetition from the users side during conversations which is facilitated by the use of the product.
Scope of the evaluation
Dependent variable
-
DV1 - Effectiveness: The number of times the participant repeats herself during a specific length of a conversation.
The fundamental problem that the participant faces is the tendency to repeat herself during interactions, so the primary function of the wearable is to assist the participant in curbing her own behavior. If this outcome is not met, the remaining touchpoints such as aesthetics, ergonomics, and ease of use do not serve any additional value. Thus, the fundamental touchpoint to evaluate and measure is effectiveness in terms of repetition.
Independent variables, Relevance, Confounding Variables, and Practicality
Independent variable
-
IV - The presence of the product.
The aim of this study is to determine if the use of the product facilitates a reduction in repetition. Thus, the average number of repetition should be compared between a scenario in which the participant does not use the device and one in which she does.
Confounding variables
-
The length of the two different conversations should be equal.
-
The topic of conversation should be equally engaging in both conversations.
-
The definition of ‘repetition’ used in both scenarios should be the same.
-
The experimental environment should be the same.
Dependent variable & Relevance
If there is a difference of +/-2 number of repetitions between the two conversations, then this result will be considered significant. If the difference is less, we cannot conclusively infer that there is a difference in her repetition with and without the device.
Dependent variable and Relevance
In the first scenario, the social worker and the participant will have a routine daily discussion without the presence of the prototype, this conversation will be monitored remotely by one of the designers. This conversion will go on for 20 min and the designer will evaluate and count the number of the times where she repeated herself during this conversation.
Next, the second case study begins where she will continue talking with the social worker for 20 minutes while using the prototype. In this scenario the designer will again observe and note down the number of times the participant repeats herself, so that the results of the two scenarios can be compared.
Evaluation method and assessment type
In order to evaluate the prototype in terms of the dependent variable effectiveness, two scenarios need to be defined which is one the user having a conversation in presence of the prototype and one without.
Insights from the co-design studies shows that the participant feels more comfortable speaking in her own language (Dutch), therefore the conversations will take place in Dutch. This case study will be done offline for the user as such she will have a real life conversation with the second person in the room known as her social worker. He visits her weekly to ensure she is not facing too many challenges and not feeling alone. This experiment will be monitored by one the design team members virtually through a social platform (Zoom meetings) in order to keep track of the effect of the prototype in the participant's verbal repetition.
Repetition can be defined very widely among people on the autism spectrum. For the purpose of this user test study, it was defined in a way that related to the participants form of repetition. Verbal repetition for the participant is repeating the same topic more than once in a short span of the conversation. Note that this can occur by using the exact words or paraphrasing.
For the dependent variable, ‘effectiveness: the number of times the participant repeats herself during a specific length of a conversation’, the number measured in the scenario with the use of the product will be lower than in the scenario without the product.
Hypotheses
Planning and Procedure
Once the prototype has been completed, the participant will be contacted and informed about the test plan. Once she approves the plan, her social worker will be contacted and invited to the evaluation session. Here, they will be briefed again about how the session will proceed, and asked to sign a consent form for the data collection. Both participants must be assured that they are not being tested, but rather the product is, and that they are free to stop the evaluation at any time. Moreover, the participants should be aware that any data used will only be shared with the team and the group tutor, and if published, all personal data will be anonymised.
The main focus will be on the product experience for the user and this will be done through a simulation. Using the ‘service design tools’ [1], the most appropriate testing methodologies were explored and narrowed down by implementing the above mentioned criteria (prototyping phase, performed by user, experience, simulation). Experience prototypes, also called mockups, allow designers to test the prototype with an active participation of the final user [2].
References :
[1]"Experience Prototypes | Service Design Tools", Servicedesigntools.org, 2021. [Online]. Available: https://servicedesigntools.org/tools/experience-prototypes. [Accessed: 28- Mar- 2021].
[2]"Experience Prototypes | Service Design Tools", Servicedesigntools.org, 2021. [Online]. Available: https://servicedesigntools.org/tools/experience-prototypes. [Accessed: 28- Mar- 2021].