User-oriented design of virtual reality supported learning factory trainings: methodology for the generation of suitable design elements

The concept of the physical learning factory’s concept includes numerous requirements to meet existing learning theory principles. Nevertheless, despite the versatile learning factory concepts available today, the high degree of complexity of the industrial production environment makes it challenging to transfer the competencies learned into the operational application situation. With Virtual Reality, training participants have the opportunity to learn in physical learning environments with transfer-oriented action tasks in virtual space directly after the training. The learning process can be personalized and adapted in the virtual learning environment. Each training participant can individually determine elements of the learning situation. For example, training participants choose the entire learning environment adapted to their unique real production environment. Virtual Reality enables new forms of reflection, e.g. recording the learning process and the associated actions. However, this sets new requirements in the context of design, which creates different challenges in Virtual Reality compared to the physical concept. The research project PortaL pursues a two-stage procedure to tailor the virtual learning environment to users in the best possible way. A pre-selection of potential design elements was presented to various stakeholders and evaluated by them in a Delphi study regarding their respective suitability for representing different aspects. A list of rated design elements resulted from this evaluation. This paper shows the methodology which was used for the design element identification and evaluation.


Introduction
In the production environment, it is important to master complexity, act flexibly, and have a high problemsolving ability [1]. Learning factories offer the possibility to develop the competencies of employees in a realistic production environment [2]. The technical and methodological competencies of employees are a key competitive factor. Learning factories therefore became very popular to develop students and employees' competencies [2]. Virtual reality (VR) expands the possibilities and application areas both specifically in learning factories [3] and generally in further education. Recent research has revealed that VR offers many opportunities in the field of education [4,5]. Still, only a few learning factories have implemented training scenarios in VR. A reason for this might be that a systematic approach is missing. The authors developed an initial approach to identify and implement requirements agile [6,7]. This shall now be complemented with the identification and evaluation of design elements.
This paper's main focus is to present a methodology for the user-oriented implementation of VR in learning factories. In the first step, various potential design elements are derived in the context of the three-pillar model based on VR design principles [8] and the intended difficulty levels of the learning environment by means of expert brainstorming. In a second step, these are evaluated in a Delphi study by various stakeholders (especially potential users) regarding their probable suitability. For further elaboration, this paper is divided into five sections: after an explanation of the basics of VR in learning factories and Delphi studies (section 2), the methodology of useroriented implementation is explained in detail (section 3). Concluding results are summarized (section 4) and discussed (section 5). Learning factory operators who consider implementing VR in their learning factory concept can use the methodology of this paper. The results are based on the research project PortaL and its workshop content.

Virtual Reality design for learning factories
In learning factories, learning takes place in a realistic factory environment [2], where training participants can perform actions within a real value chain of a physical product. However, the concept of physical learning factories is limited in various ways, mentioned in [9]. Due to these limitations, virtual learning factories have been developed in recent years [10]. Here, learning takes place in a virtual environment [11]. This also reflects a general trend according to which learning environments in VR are increasingly used in vocational education and training as a further education tool [12]. VR is understood as an interactive computer-based simulated setting of reality [13]. VR holds a variety of potentials: processes can be shown in slow motion or fast motion. In addition, depending on the wishes of the training participants, a high number of value chains with different products can be presented. The level of difficulty of the exercises can be varied according to the needs of the participants, and the didactic concept can be supplemented by new types of reflection. The concept of a hybrid learning factory combines the advantages of the physical and the virtual environment [2]. Trainings can be conducted in a real or a virtual environment, depending on its purpose. This opens various possibilities for extending the existing operator model. To design learning factories, three conceptual design levels and two didactical transformations must be considered, which can be used in virtual learning factories but must consider new requirements while implementing them in hybrid or purely virtual learning factories [14]: • The macro-level includes the infrastructure of the learning factory and the curriculum. For hybrid learning factories, the macro-level includes the virtual and physical environment. • The meso-level includes learning modules. This is also the case for hybrid learning factories.
• The micro-level includes teaching-learning scenarios, which are represented virtually or physically.
In the first didactical transformation, the intended competencies are derived from the organizational environment, the organizational targets, and the target group. The second didactical transformation derives the socio-technical infrastructure and didactical aspects from the intended competencies [14]. This systematic approach can also be applied to hybrid learning factories while considering new requirements. The VR experience is generally based on four core elements [15]: • Virtual world describes the environment in which the user is sent to with the VR-headset. This environment has not necessarily to represent an environment that can actually exist in reality. • Immersion describes the effect that causes that the user's consciousness to be no longer aware of the virtual environment [16]. The illusion moves into the background so that the virtual world is perceived as real [17]. • Sensory feedback represents the most significant differentiation from traditional digital media. The user's active movements change the representation of the virtual world [15] • Interactivity is defined as the extent to which users can modify the virtual environment's form and content [18].
These four core elements offer a wide range of possibilities and define the framework for core design principles for developing virtual environments. Individual aspects of these four core elements represent design elements [8]. The choice of design elements allows to generate various ways to implement a single aspect or detail of the virtual world (e.g. stopwatch vs. infobox (high vs. low interactiveness)). This can help developers to integrate action tasks or environments which are not accessible in the real world. In VR, it is therefore possible to train specific action sequences that are difficult to test in reality. Experiences that are limited or difficult to experience can also be presented (e.g. manipulation of time and speed). VR brings flexibility to time and place. For example, historical moments can be simulated [4]. With VR, it is possible to address different types of learners simultaneously [19].
In the research project "Virtual action tasks for personalized adaptive learning" (PortaL), a personalized training scenario is developed in VR. The focus is on adapting and personalizing the learning process. After a training course in the physical learning factory, the participants conduct a personalized exercise in the virtual environment.
The developed approach will initially be implemented in the process learning factory Center for industrial productivity (CiP) using the example of an existing learning module about the value stream analysis. The division UReality of Kirchner GmbH develops the software agilely for the virtual environment. In addition, a guideline will be developed to implement virtual exercises for different target groups, learning factories and learning content. The presented methodology will be part of this guideline.

Methodology
VR offers the developer of a learning factory a wide range of possibilities. These options can also show potential in the context of learning factories. However, to exploit these, a structured methodology is required which deals in particular with the design elements within VR and can identify suitable design elements for the respective application purpose in a learning factory.
The starting point for the development of this methodology is on the one hand the mentioned principles for the design of virtual reality environments (see section 2). On the other hand, an existing competency-oriented workshop concept is taken as given. In this concept personalized learning scenarios for competency development are already identified and documented. Both aspects will not be discussed in detail as they are already discussed in existing literature [14,15]. The whole methodology consists of four steps, which are performed one after the other. The focus of this publication is on generation steps of the methodology dashed framed in Fig. 1 below.

Identification
In the first step, a selection of design elements is identified which are potentially suitable for mapping the action tasks originating from the elaborated competence-oriented workshop concept. In the PortaL research project, a combination of group brainstorming and expert interviews from the field of VR software development was chosen for the generation. The latter served to be able to incorporate best-practice insights from VR development directly.  A three-pillar model (see Fig. 2) was developed to identify suitable design elements and create a design element list to implement the previously defined competency-based action tasks. The model differentiates the form of expression based on activity level of the user. In addition to factors influencing the success of learning environments (e.g. authentic task visualization), this model also includes the design principles in VR and the relevant action tasks derived from the competence-oriented approach. VR design principles cover all the common design and interaction types that VR as a technology can technically enable (e.g. grab elements by pressing a physical controller button, by a control menu or with your finger movement tracked by the VR headset). Design elements can occur in the forms of proactive, active and passive. If the action task of taking time in the context of value stream mapping is considered as an example, a proactive form is, for example, an unlabeled stopwatch that the user can take to each station in order to take times. An active form is a stopwatch, which is implemented at the respective station and thus stimulates the user to stop a time at this station. However, the time taking itself must be actively performed by the user. A passive form is a stopwatch that is implemented by displaying the time in the form of a screen overlay without the user's involvement. The user does not have to be proactive himself, nor does he have to stop the time himself.

Classification
The identification of possible design elements is followed by classification. This step is necessary for the later query of the design elements in the form of matrices, which compare action tasks and design elements. One difficulty that arises is that categories are mapped containing sufficiently homogeneous design elements. If the represented design elements are too inhomogeneous, an evaluation next to each other is only possible with difficulty and reduces the clarity for the survey participant.
The identified design elements were therefore classified into a total of seven categories (data recording, interface, illustration, feedback, communication, interaction and support). These categories are differentiated by technical (e.g. interface), content-related (e.g. illustration) and didactical (e.g. feedback) categories. Individual design elements affect several categories: therefore, individual elements can be classified into more than one category. It can occur that design elements are related or interdependent to others. This circumstance needs to be taken into account and checked after the design element determination. Table 1 shows an exemplary list of design element categories and included design elements.

Delphi Study
In its classic form, a Delphi study is, in its classic form, a systematic, multi-stage survey procedure with feedback and is an estimation method used to estimate future events, trends, and technical developments. In a Delphi study, a group of experts is presented with a list of questions or theses in the relevant field. The respondents can assess the theses in two or more survey rounds. In the first round, the experts evaluate the questions and theses directly. From the second round on, feedback is given on how other experts have answered in the first round, usually anonymously. In this way, an attempt is made to counteract the usual group dynamics with very dominant individuals [20].
A two-stage Delphi study was conducted with 19 workshop participants and trainers. Several design elements structured in matrices were presented to the participants. In these, the participants were asked to rate the design elements in their suitability to represent the assigned action tasks on a scale of 1-5 (1 = completely unsuitable; 5 = completely suitable). The results were then analyzed, and the participants were returned their questionnaires with comments on deviations from the mean. It was not indicated whether the participant's assessment deviated positively or negatively from the mean value of the answers, so the possible correction is not influenced. The participants are asked to reconsider their assessment and to think again about possible interpretations of the design element. The multiple iterations are intended to ensure that participants evaluate items with a consistent basis of understanding. Subsequently, the feedback was evaluated again, and suitable design elements were selected based on the feedback.

Results
The statistical analysis of the study was carried out with the statistical software R. Four hypotheses were formed for each design element, asking whether the mean value of the expert assessment was significantly less than 2, significantly less than 3, significantly greater than 3 or significantly greater than 4. This results in a total of 880 null hypotheses for the 220 considered design elements. The significance level was set at 5 percent. To determine the test procedure, a test had to be carried out to determine whether the collected data is normally distributed. For this purpose, a Shapiro-Wilk test was conducted [21], which showed that there is no normal distribution. For this reason, a Student's t-test cannot be used; instead, a sign test was used to test the null hypotheses [22]. This showed that 29 design elements were significantly less than 2, 25 significantly less than 3 but not less than 2, 87 not significant, 64 significantly greater than 3 but not greater than 4 and 15 significantly greater than 4. Tab. 2 summarizes examples of the significance of the design elements. For example to measure times, using a virtual stopwatch is recommended.  Table 3 shows a detailed example for the design matrix "support", in which 15 design elements are included. The rows indicate the different cases where support is needed during the learning scenario. The columns show the possibilities for support. The mean values of the expert interviews are shown in the cell entries. The colors indicate the significance. From this result, recommendations for the implementation of the virtual environment can be derived. For example, according to the experts interviewed, it is useful that individual questions, questions about the learning scenario and the learning content should be asked to the trainer. It needs to be mentioned, that Tab. 3 shows the stakeholder-rated suitability, while the actual suitability still has to be evaluated.

Discussion
Through the presented procedure, 220 design elements for the user-oriented implementation of virtual learning scenarios could be identified and classified. To decide which design elements should be implemented, user tests are not suitable in this case because the large number of possible test scenarios (2 220 ) cannot be mapped and developed. For this reason, a Delphi study was conducted. Recommendations could be made for 133 (60.5 %) design elements. This simplifies the decision-making process for choosing the most useful design elements. Nevertheless, user tests and competence measurements should be carried out after implementation, as these reflect the participants' direct experiences and learning successes. The design elements provide evidence that the role of the learning factory trainer changes in virtual learning exercises, for example, because further media competences are necessary. Based on this research, a guideline for the implementation of VR in learning factories can be created.

Summary and outlook
In this publication, a methodology for the identification, classification, and evaluation of design elements for VR learning environments was presented. The methodology can be a useful tool especially for large numbers of design elements to be defined. This can reduce the implementation effort enormously, since a decision or preselection regarding the design elements to be used can be made before the actual implementation. The methodology is principally suitable for the application in augmented (AR) and mixed reality (MR) contexts as well.
Further research will consist of user tests and further evaluation. As mentioned in section four, the design elements are stakeholder-rated while the actual suitability must be proven in further evaluation. This will be done in the final evaluation where the VR application will be tested in an actual training context. Additionally, the procedure will also be evaluated by experts with regard to its applicability based on various criteria. In this context, considerations can also be made as to how far the presented approach needs to be adapted for AR and MR contexts.