Quality of Online Learning Participation in a Context of Crisis

Objectives: The COVID-19 pandemic has forced educational institutions to adopt online tools to remotely teach and efficiently use virtual learning situations during the emergency. However, although these environments may serve to improve teaching processes, several issues must be considered to ensure quality student learning. The purpose of our study was to examine types of participation in virtual learning settings by analyzing the level of information contained in message posts and the depth of contributions made by students. Method: We analyzed data from a computer programming module taught online using a learning management system during the first year of a computer science degree program at a Chilean university. We conducted a content analysis of the messages posted in the forums, followed by a statistical analysis of the codified data. For the latter, we used quantitative methods to identify relationships between the level of information contained in student contributions (information level) and a series of covariates, such as message length, perception of achieved learning, final grade, and message depth. Results: Results show that Number of Words (B = 0.02, SE = 0.002), Final Grade (B = 0.45, SE = 0.22), and Student Self-Perceived Learning (B = -0.82, SE = 0.40) predict higher Information Level. Chavez et al., 2021 Open Access Higher Learning Research Communications 73 Conclusions: A clearer understanding of the relationship between forms of participation in online collaborative environments and the quality of participants’ contributions would support activities that are conducive to improved participant learning. Implication for Practice: Results revealed the need for guidelines that define online classroom activities, as these have a considerable influence on the generation of dialogue that is conducive to the attainment of new knowledge.


Introduction
In the context of the ongoing international COVID-19 crisis, higher education institutions have been forced to adopt online environments as key spaces for interaction between teachers and students (Hodges et al., 2020). Although these online contexts may contribute considerably to preserving interaction between teacher and pupil during the pandemic, a number of other issues must be taken into consideration in order to ensure effective teaching and learning processes (Sánchez & Reyes Rojas, 2020). Two of the main hindrances to learning that are observed in virtual environments are the small number of occasions on which students participate and the poor quality of the messages that they post during collaboration (Dillenbourg et al., 2009;Hrastinski, 2009;Reimers & Schleicher, 2020).
Our study is framed in the research area of computer-supported collaborative learning (CSCL) and seeks to explore the idea that participation is one of the fundamental variables that determine learning in online environments (Sivapalan & Cregan, 2005;Kent et al., 2016). This may be because (1) greater participation means a greater probability of student interaction, (2) greater participation means more communicative exchanges, or (3) greater participation leads to a better learning experience (Isohätälä et al., 2017;Kim & Ketenci, 2019). We define participation as the capacity for students to involve themselves in virtual contexts in a variety of ways and to differing degrees. This capacity is expressed through the contributions that students make as part of a collaborative process (Chávez & Romero, 2014;Hratiski, 2008). The scientific evidence regarding collaborative learning in virtual environments is categorical. On one hand, levels of student participation are low or at least unequal. On the other, in the majority of cases, the quality of contributions is not sufficient for participants to achieve profound learning (Dlab et al., 2020;Schellens & Valcke, 2006;Sintonen et al., 2017;Stahl, 2015).
The scientific evidence suggests that in order to enhance student learning in online environments, the quality of participation must be improved (Hrastinski, 2009;Kurucay & Inan, 2017;Michailidis et al., 2018). However, there is disagreement as to how this may be achieved. In general terms, participation can be defined and thus measured at two levels: the structural level and the content level. At the structural level, it can be defined as, for example, the number of times a student accesses an online platform, the number of responses to a given contribution, and/or the time spent on a given application. At the content level, participation is generally associated with the type of contribution made by the student or the depth of these contributions.

Open Access
Higher Learning Research Communications 74 Few researchers choose to adopt a more complex approach or, as in our particular case, to define participation as a complex activity that may include aspects of both structure and content (Dillenbourg et al., 2009;Stahl, 2015).
In relation to structural elements, analysis tends to be based primarily on the assumption that participation is associated with the number of times that a student accesses a platform, the number of messages that they post, and the length of these messages (Fu et al., 2016). However, it is difficult to discern from these indicators the degree of depth of participants' contributions (Chen et al., 2018;Dillenbourg et al., 2009).
Aspects of content are generally associated with the level of depth of messages or the progression achieved by participants in terms of the meanings discussed during the collaborative activity (Ding et al., 2017). Proponents of this position suggest that participation is a process of establishing, maintaining, and growing relationships with others (Wenger, 1998). This is associated with the type or form of communication in which students engage and reflected in the quality of messages posted (Hrastinski, 2009;Michailidis et al., 2018).
Each of these structural and content-related aspects puts emphasis on different elements that are considered important to quality participation in online environments. However, there is a need to determine the role played by the different variables involved in students' participation. Research shows that forms of student participation in such contexts may have consequences in terms of the frequency and quality of participation and, in turn, of the quality of student learning (Dlab et al., 2020;Janssen et al., 2007;Phielix et al., 2010). More frequent participation may mean higher levels of interaction; however, in order to stimulate greater depth in terms of the ideas expressed by participants, this interaction must take a certain form.
We believe that identification and quantification of those variables that can help predict the quality of the messages posted by students as they interact in online learning contexts is vital. Progress on this issue is relevant as progress would (1) enable explanation of the learning that takes place when people collaborate in online learning contexts, (2) help guide and orient teaching practices in online environments, and (3) help with the resolution of certain problems currently faced by those teachers who find themselves required to conduct online teaching activities in the context of the COVID-19 pandemic.

Purpose of the Study, Research Questions, and Hypotheses
In view of the background presented in the previous section, the purpose of our study is to examine types of participation in virtual learning processes by analyzing the level of information contained in message posts and the depth of contributions made by students. As such, our research questions are as follows: 1. What types of messages are posted by students as they participate in collaborative activities in an online environment?
2. Is there a relationship between the type of message posted and (a) the depth of these messages, (b) the length of these messages, (c) the student's final grade, and (d) the student's perceived learning?
3. Which variables serve to statistically predict the type of message posted by students when they participate in collaborative activities in an online environment?
There are three hypotheses corresponding to the research questions: H1. There is a relationship between messages that rank higher in the hierarchy and the total number of messages posted by students during collaboration.

Higher Learning Research Communications 75
H2. There is a relationship between messages that rank higher in the hierarchy and a set of structural participation indicators.
H3. Certain structural participation variables are more relevant than others in the formulation of higherranking messages, and they may serve to predict the quality or type of message posted by students.

Nature of the Study
A case study design was adopted for the present research (Spector et al., 2013;Stahl et al., 2011). In particular, the study analyzed two discussion forums on different subjects pertaining to a computer programming module offered during the first semester of a bachelor's degree in computer science at a public university in Chile. Messages were gathered directly from the forums, which themselves are part of the virtual course environment. We conducted a content analysis of the messages posted in the forums, followed by a statistical analysis of the codified data (Johnson et al., 2007). For the latter, we used quantitative methods to identify relationships between the level of information contained in students' contributions and a series of covariates, such as message length, perception of achieved learning, final grade, and message depth. Subsequently, we fitted a theoretical ordinal logistic regression model that would make it possible to predict the types of message that the students had posted in the module forums.

Context and Participants
The present research was conducted in the context of a computer science degree module that consists of eight hours of online teaching via the Moodle platform. A total of 38 students took the module and all of them agreed to participate in the study. All participants were requested to sign an informed consent. This included a brief explanation of the research objectives and specified that participation was voluntary and anonymous. The objective of the module was to strengthen computational thinking for the implementation of algorithms designed to solve problems within the discipline and in everyday life. The module design included two forums. The first is intended for students to demonstrate their ability to efficiently address a computational problem. The second is for discussion of data searching and sorting methods. Each forum was open for four weeks and one of the most important rules of participation is that ideas must not be repeated. Another fundamental rule is that, in order to encourage a high frequency and depth of messages, each student must comment on at least three posts made by their peers.

Instruments
In order to measure type of participation, a number of structural and content-level indicators were used. At the structural level (independent variables), a set of quantitative data was collected relating to the activity conducted via the platform. Activity was measured using the following structural indicators or independent variables: a) the number of messages posted by the participant; b) the number of words per message (i.e., message length); c) the final grade awarded by the teacher; and d) the student's self-perception of achieved learning. All of these variables, with the exception of perceived learning, were extracted from the Moodle platform. Perceived learning was measured by means of a survey that the students were required to complete at the end of the module. This covered (1) their management of the time given to complete the various tasks; (2) their management of the tasks themselves and the difficulty of the content; (3) their management of social participation or organization of the joint activity in order to complete the tasks; and (4) ) their perception that these aspects served to help or hinder their learning over the course of the module. In terms of content or message type (dependent variable), a qualitative analysis of the contributions made was conducted. This was Higher Learning Research Communications 76 achieved by means of the framework presented in Appendix A, which was designed to facilitate differentiation between the different types of message posted and the level of depth of each of them.

Data Gathering and Analysis Procedure
All of the messages posted by the students in the two forums were gathered. We then conducted two phases of analysis. The first was a content analysis in order to codify the messages, which were classified according to content type and level of depth. Qualitative analysis of the messages was conducted by three specialists using an adapted version of the method proposed by Chávez et al. (2016) for analysis of discussion forum messages (see Appendix A). Messages were categorized according to a hierarchical structure consisting of five levels (lowest to highest): analysis of peer information (PI); idea contribution (IC); learning content processing (CP); situation of the task or problem within a broader framework of knowledge, experiences, and information (TS); and motivation to comply with and understand the task (M). The highest-ranked message type in the hierarchy was task situation (TS), as it indicates the student's capacity to take into consideration other pieces of knowledge in the resolution of a given problem. The motivation (M) type focuses on the student's attribution of meaning to the task rather than specific content creation, which was the central element of the study. Consequently, M was not included in the analysis.
The classification hierarchy was validated by experts, and all data were entered into a quantitatively codified database. Each message was classified according to its level of depth using a five-point Likert-type scale, where 1 indicates superficiality and 5 indicates profundity. Given the quantity of information obtained, we ultimately decided upon a three-level depth classification: Low for messages categorized as 1 or 2, Medium for messages categorized as 3, and High for messages categorized as 4 or 5. In order to establish perceptions of achieved learning, we applied an ad hoc self-perception survey at the end of the semester. The survey employed a five-point Likert-type scale ranging from 1 (very low) to 5 (very high).
Once all data codifications had been validated by the experts according to the message categorization system (Appendix A), a series of quantitative analyses were conducted. In this second phase of the research, we calculated bivariate correlations and analysis of variance (ANOVA) in order to explore Research Questions 1 and 2. Ordinal logistic regression was used to explore Research Question 3, the purpose of which was to assess the behavior of those structural variables that best predict the level of depth and quality of messages. Logistic models are suitable for situations in which there is a need to explain the probability of the occurrence of a given event by means of certain independent or explanatory variables. Given that we have information as to which are the structural participation variables that best reflect the level of message depth, this should permit us to measure the type of message posted by participants. These models reveal the probability, according to the different independent variables, of achieving a certain level of information, or the logit of the level in relation to previous ones. We calculated five different models, whose dependent variable was message type (i.e., level of information). Number of words, final grade, and perceived achievement were then added sequentially as covariates. The process resulted in an optimal model with low standard deviation, in which the estimated parameters enabled prediction of the type of message that participants are able to formulate, according to the covariates.

Results
The results presented in this section are organized according to the research questions specified earlier. We begin by describing the types of messages constructed by students when they participate in collaborative course activities. We then explore the relationships between the different variables addressed in the study. Finally, we present the linear regression results.

Open Access
Higher Learning Research Communications 77 Question 1: What types of messages are posted by students as they participate in collaborative activities in an online environment?
Message classifications according to type and depth are presented in Table 1 and Figure 1. As shown in Table  1, half of the messages posted fall into the lowest information level category (PI = 48.2%). The following two categories contain a similar number of messages (IC = 18.1%; CP = 18.6%). These three levels indicate that the student posts primarily low-ranking messages involving limited argumentation or idea development that reflect little more than response to information provided by their peers (PI).
A much smaller number of messages fell into the two highest information level categories. Messages situating the task or problem within a broader framework of knowledge (TS) accounted for 13.6% of all posts. Messages motivating students to comply with and understand the task (M) accounted for only 1.5% of posts. As each student was permitted to post more than one message, we decided to include the total number of participants at each level. The majority of students (66%) fall into the two lowest information level categories (PI and IC).   (Chávez et al., 2016). Almost all message types can be seen to behave in a similar way, beginning with low percentages at the lowest depth level and gradually increasing to higher percentages at the highest depth level. The only exception is CP-type messages, where distribution remains the same for the first two depth levels and increases only at the high level. Based on all of the above, we can see that the largest number of messages posted by students fell into the highest depth level category, regardless of message type. As such, the students made contributions that, for the most part, involve more profound ideas.

Figure 1: Frequency of Messages According to Information and Depth Level
Note: the colored lines represent the different types of messages posted by participants. Low includes messages categorized as "superficial" and "somewhat more than superficial." Medium includes messages categorized as "neither superficial nor deep." High includes messages categorized as "somewhat deep" and "deep." Table 2 presents the average message word count for each information level, along with their respective standard errors. The results show a gradual increase in the average number of words from the bottom to the top of the message type hierarchy. In other words, messages higher up the hierarchy require a greater number of words. As shown in Table 2, PI-type messages have the lowest average number of words ( ̅ = 69.02, SD = 4.2), while TS-type messages have the highest ( ̅ = 232.29, SD = 44.6). Closer analysis of the results reveals statistically significant differences between average number of words per message type (F = 19.63, p < 0.01). From this we can conclude that the level of information achieved in a message is directly related to the number of words used. We then analyzed for statistically significant relationships between Message Type and the Grade achieved by students at the end of the module. The Chilean grade system ranges from 1 (lowest) to 7 (highest). presents the average grade and standard error for each message type. The grade is awarded by the teacher at the end of the module based on the student's performance in the various tasks. The grade assigned was not related to the level. As the differences were statistically significant (F = 3.53, p = 0.016), we conducted a Duncan test in order to specifically identify them. The test revealed differences between PI and IC messages (which had average final grades of 4.3) and TS-and CP-type messages (with higher average final grades of 4.6 and 4.8, respectively). We also found differences between CP messages and PI messages and between CP messages and IC messages. Despite these differences, the analysis only enables us to conclude that the students who achieved a higher final grade also posted messages that were categorized towards the top of the proposed hierarchy.  Table 4 presents self-perception of achieved learning for each information level. The results show no significant association ( 2 =6 = 8.06; p > 0.203) between self-perceived learning and the level of information contained in messages.

Table 4: Self-Perception of Achieved Learning According to Information Level
Question 3: Which variables serve to statistically predict the type of message posted by students when they participate in collaborative activities in an online environment?
Five logistic regression models were tested. Ordered logit models are used to estimate relationships between an ordinal dependent variable and a set of independent variables. The dependent variable is Information Level. The first contained no covariates; then, each of the remaining four models added one covariate sequentially (Message Depth Level; Number of Words; Final Grade; and Student Self-Perceived Learning.) Table 5    An example to explain the above is presented in Figure 2, which shows the probabilities of obtaining different information levels according to variations in message word count, that is, message length (Wn) and average final grade 1 (Fg) at a constant medium depth level and low self-perception of achieved learning. The probability of obtaining a TS-type message (the highest information level in the hierarchy) rises with the number of words and the average final grade. As such, when messages have 69 words and an average final grade of 4.3, the probability of obtaining TS-type messages is 9.4%, increasing to 11.5% when the average final grade is 4.8. When messages have 232 words (higher than the total mean) and an average final grade of 4.3, the probability of obtaining TS-type messages is 54.5%, increasing to 60% when the average final grade is 4.8.
By contrast, the probability of obtaining a PI-type message (the lowest information level in the hierarchy) falls as the number of words and the average final grade increase. As such, when messages have 69 words and an 1 It should be noted that the values chosen for the number of words and for the average final grade were based on the cut-off points in Table 2 and Table 3, respectively. This shows that, at a medium depth level and a low self-perception of achieved learning, the effect of message word count on the probability of obtaining TS-type messages is significant. Average final grade also has an effect, but this is considerably smaller.

Discussion
Our aim was to analyze participation and learning in a group of students as they engaged in a virtual environment. Specifically, we analyzed how collaborative participation relates to depth of the messages that they exchange and to other variables associated with a variety of structural aspects: message length, final grade awarded, and self-perception of achieved learning. The scientific evidence published to date indicates that students' participation in collaborative activities designed by the teacher does not guarantee learning (Isohätälä, 2017). In accordance with our work, Hrastinski (2008;2009) highlighted the need to consider participation not only as synonymous with speaking or writing, but as a complex process of participation and relationships with others, together with the support of attractive activities. This points to a need to identify the principal aspects that must be developed as part of an online teaching process. In this same line, we assert that further research is needed into the specific characteristics of a form of participation that would yield improved levels of depth of students' contributions and, in turn, a better quality of learning (Hrastinski, 2009;Järvelä, 2016).

Higher Learning Research Communications 82
Our main finding is that it is possible to predict the type of participation or level of message depth in a virtual environment using a generalized linear model. By means of this model, we sought to identify a group of variables that increase or decrease the likelihood of obtaining a higher-ranking message type. Our model took into consideration a number of variables, which we have termed structural participation variables. This enabled us to identify key variables, such as message length, students' final grades, and self-perception of achieved learning. The modeling results show, for example, that the contribution of message length is significant. In other words, students who write longer messages also produce messages of a higher level in terms of information and depth, while shorter messages are less likely to demonstrate a high degree of complexity. However, it is important to note that these messages must demonstrate certain other characteristics, such as simplicity and precision of the ideas expressed. This implies the need for students not only to develop ideas as they collaborate, but also for these ideas to be articulated as clearly as possible in order for them to be of use to their peers. As such, it would be interesting for future works to explore the usefulness to peers of messages posted in a given forum (e.g., Järvelä et al., 2016).
Analysis of other variables that could help to predict messages that rank more highly within the proposed hierarchy revealed an association between students' grades, message types, and message quality. We were able to identify significant differences between the levels of information contained in messages in relation to the average final grades achieved. Students who participate more actively with longer messages also obtain higher grades at the end of the module. In other words, students whose grades are below the total average post messages that rank lower in the hierarchy (PI and IC), while those whose grades are above the total average post higher-level messages (TS and CP). As such, there is a positive relationship between the teacher's evaluation of student learning and the level of information contained in messages posted by the students.
Another prominent covariate identified by the study is self-perception of achieved learning, which refers to the students' own evaluation of their knowledge construction during the module. The results show that students with a strong perception of their own learning are not necessarily those who make the highest-level contributions or acquire the most complex pieces of knowledge. A possible explanation for this is that these students may be highly demanding of themselves when it comes to tackling a task and, as such, are more critical when they come to self-evaluation. This is evidence of the need for teachers to provide motivation and acknowledge the value of messages posted by students during collaboration in an online environment.
Teacher presence is relevant, as it contributes to students' sense of self-worth.

Limitations of the Study
The study has a number of limitations. Firstly, given the small sample size, the results need to be replicated in larger populations. Secondly, additional covariates or structural activity indicators must be considered in order to explore in greater depth the relationship between these and the quality of contributions. Thirdly, the presence of the teacher in these types of activities and the mediating role of the teacher and fellow students over the course of the task are also important factors which we were unable to explore in the present study.

Implications for Theory and Practice
The present study contributes by considering some of the key elements that should be taken into consideration when designing a training process for use in an online environment. These include task elements such as the guidelines that steer dialogue within the activity, as this must be conducive to the construction of new pieces of high-quality knowledge. There are also a number of criteria that serve to increase levels of participation and the quality of contributions. These include guidelines that provide students with a clear idea of concepts, deadlines for participation, and the link between contributions and a satisfactory grade, all of which are key factors in the construction of complex pieces of knowledge (Ludvigsen et al., 2016). As a final point, we believe that the Higher Learning Research Communications 83 organization of the activity, the rules of participation, and acknowledgment by the teacher of that participation are key aspects in the planning of an online learning process.

Conclusion
The present study suggests that online environments are becoming increasingly demanding. This is particularly noticeable in relation to the level of information and quality of contributions required to participants (Chávez, 2020). Greater clarity as to the relationship that exists between forms of participation in online collaborative environments and the quality of participants' contributions would support the generation of activity models that are conducive to improved quality of participant learning.
One of the main challenges posed by the COVID-19 pandemic is for students to engage in online learning environments. Our results suggest that technology is a particularly important resource in the current context in that it enables people to continue their educational activities with as little disruption as possible. Furthermore, they point to the existence of certain elements that contribute to improving the quality of participation in online environments and, as such, to more productive learning processes (Dlab et al., 2020;Schellens & Valcke, 2006;Sintonen et al., 2017;Stahl, 2015). same idea using different words