Indirect Feedback: A Dialoguing Approach to Assessment

Although there is consensus in the current literature that feedback plays a fundamental role to student performance and learning, there is debate about what makes it effective. Particularly, some assessment instruments, like the National Student Survey in the United Kingdom, reveal that evaluation and feedback are systematically among the areas that students are less satisfied with. The aim of this article is to describe the indirect feedback technique, which was devised and used by the principle author in his previous tenure as a professor at the University of Cadiz in Spain and to reflect on how it can be applied to overcome some of the limitations presented in a different context of practice. It is argued that indirect feedback meets many of the principles of good practice (facilitation of self-assessment skills, delivery of quality information about the students’ learning, encouragement of dialogue, and improvement of teaching) identified by Nicol and McFarlane-Dick (2006).


Introduction
Feedback has been defined in a variety of forms and from multiple perspectives, though the main reference for the present work will be Nicol and McFarlane-Dick (2006), who regarded it as "information about how the student's present state (of learning and performance) relates to [academic] goals and standards" (p. 200). In the academic context, numerous studies have shown that feedback, when effective, contributes greatly to student learning (Chaqmaqchee, 2015;Deeley, 2018;Kifle & Alauddin, 2016;Tan, Whipp, Gagné, & Van Quaquebeke, 2019) and that benefits are consistent across disciplines, knowledge, educational levels (Black & William, 1998), socioeconomic status, race, and school setting (Bellon, Bellon, & Blank, 1991).
Despite being touted as universal panacea by some for improving performance, there are issues with application of feedback (Steen-Utheim & Wittek, 2017). The concept of what makes it effective is subject to discussion. Orsmond, Maw, Park, Gomez, and Crook (2013) stated that in the current situation, "there are continuing challenges surrounding the delivery of effective feedback in higher education" (p. 241). In addition, students quite often fail to understand the feedback they receive (Orsmond, 2011;Ouahidi & Lamkhanter, 2017;Pentassuglia, 2018) or find it difficult to translate it into action (e.g., Kreonidou & Kazamia, 2019). Research shows the existence of "significant mismatches between tutors' and students' conceptions of assessment criteria" (Nicol & McFarlane-Dick, 2006, p. 206) and between their perception of detail and efficacy of the feedback provided (McCarthy, 2017).
Not surprisingly, the United Kingdom National Student Survey of 2018 (Office for Students, 2018) revealed that assessment and feedback received the lowest level of satisfaction (overall satisfaction rating of 73%) among the five main areas analyzed (the teaching on my course, assessment and feedback, academic support, organization and management, and learning resources). Moreover, this result mirrors those of the successive editions of the survey since its inception in 2005. Tutors as well as students have expressed frustration with the fact that many of those in their care fail to respond to the feedback provided (Pereira, Flores, Simão, & Barros, 2016;Sellbjer, 2018).
While teaching at a major university in the United Kingdom (see Context of Practice), the principal author perceived that the feedback provided failed to meet the well-known seven principles of good practice suggested by Nicol and McFarlane-Dick (2006): (1) helps clarify what good performance is (goals, criteria, and expected standards); (2) facilitates the development of self-assessment (reflection) in learning; (3) delivers high quality information to students about their learning; (4) encourages teacher and peer dialogue around learning; (5) encourages positive motivational beliefs and self-esteem; (6) provides opportunities to close the gap between current and desired performance; and (7) provides information to teachers that can be used to help shape the teaching. (p. 205) This study describes the simple indirect feedback (IF) procedure that the principal author devised and used in his previous job as a professor at a university in Spain and explains how it can be applied to overcome some of the limitations presented in the next section.

Context of Practice
When this article was written, the Research Methods and Statistics and Advanced Statistics and Methods modules were part of two master's programs taught at the School of Psychology at a major university in the United Kingdom. Although differing in content, program specifications indicate that both modules aim to demonstrate how important professional ethics and research are in psychology for the improvement of learners' understanding of the variety of psychological research methods (both qualitative and quantitative) that are available to them, as well as to familiarize learners with the concepts, key principles, and main objectives that data analysis has in psychology. At the end of each module, students are expected to be able to evaluate in critical ways the key principles of both qualitative and quantitative research methodologies, to demonstrate deep understanding of the criteria they should consider for the selection of a method to give an answer to their research question and, finally, to apply the techniques that are related to both descriptive and inferential data analyses.
Assessment of the Research Methods and Statistics module is undertaken by a 2-hr openbook exam and a 3,000-word essay submitted via Turnitin. Both components are worth 50% of the overall mark. In the 2014/15 edition of the module, the time between taking the exam and receiving the marks was 35 days. Feedback was provided in the form of individualized comments on each exam as well as a single-page document with some general notes on the common strengths and areas of improvement across the cohort. The essay submission period started immediately after the communication of exam marks and concluded 1 week later, with feedback and marks provided 28 days later.
The Advanced Statistics and Methods module assessment included two 2,000-word essays submitted via Turnitin, each counting for 20%, and a subsequent 3-hr final exam worth 60% of the final mark. The period between deadlines for both essay and feedback was 21 and 29 days, respectively, in 2014/15. The exam took place the day after the feedback from the second Indirect Feedback: A Dialoguing Approach … 3 essay had been provided. The period between completion of the exam and publication of marks was 28 days.

Indirect Feedback at the University of Cadiz
During the principle author's last year at Universidad de Cadiz in Spain, he devised and used a simple technique to provide feedback on performance in written exams, which he dubbed IF. This technique consisted of the following steps: (a) 15 to 30 min after completion of the exam, all students were invited to reenter the class and were provided with a printed copy of their answer sheet. (b) A PowerPoint (or similar) presentation was then delivered to the entire class, in which each item/question was presented in the same order as it appeared in the exam. (c) The possible answers to each item and their rationale were successively discussed between teacher and students, until the correct answer was identified and agreed upon. (d) As the discussion progressed, students were asked to review their answer sheet and to assign themselves a mark. This was not shared with others and had no bearing on the final mark, which was awarded by the tutor in the normal way. (e) Individual feedback in the form of comments on the students' assignments accompanied the final results when these were delivered some days after the exam.
This procedure can be considered as a form of feedback because the information provided can be easily and readily used by the students to compare their own performance with the standards presented (McGuinness & Vlachopoulos, 2019) and they can participate in a discussion where their views and knowledge-and their subsequent performance in the exam-will be challenged (Ma, 2018). It is an indirect procedure because the analysis and discussion focuses on the exam questions and not on the students' specific performance.

Advantages of Indirect Feedback
The impact of the application of IF at the University of Cadiz was not formally assessed; therefore, this article is not presented as a research report, but as a reflection on its potential benefits and limitations. Nicol and McFarlane-Dick's (2006) seven principles of good feedback practice will be used to inform the following discussion. However, at least one of the principles, encouraging positive motivational beliefs and self-esteem, is peripheral to the focus of this work and will not be discussed.

Information on the Process
IF provides detailed information on not only the correct answers to the questions in the exam (Principle 1) but also the goals and standards of the task as well as alternative procedures (processes) to achieve those goals (Principle 6). Numerous studies have shown that making assessment criteria and standards explicit is a difficult task, especially when they are complex and multidimensional (Handley & Williams, 2011;Menéndez-Varela & Gregori-Giralt, 2018). As a result, they frequently remain "'tacit' and unarticulated in the mind of the teacher" (Nicol & McFarlane-Dick, 2006, p. 206). In contexts like lessons of statistics, most goals seem unambiguous, but in practice, different procedures may lead to identical results, whereas similar procedures may yield different results depending on choices made by the student. For example, students are required to write their essays in American Psychological Association format "as for publication." Although this appears to be a straightforward criterion, it involves complex decision making, a certain degree of creative freedom, and editorial choice. Further, by focusing on processes and not only results, IF helps to overcome the complaint common among many teachers that students do not care much about the feedback provided unless the task is relevant to future assessments. The mark appears to be their main concern (Dlaska, & Krekeler, 2017). However, it is important to note that this lack of interest could be ascribed to a lack of training or

Dialogue Versus One-Directional Feedback
One of the most relevant features of IF is that it is dialogic rather than one directional (Principle 4). It is a common concern that feedback in higher education is still largely seen as the unilateral transmission of information from teachers to students about what is right and wrong in their work (Nicol & McFarlane-Dick, 2006;Planar & Moya, 2016). Nevertheless, the mere reception of IF is not sufficient for the feedback to be effective and improve learning. The message must be interpreted, constructed and internalized, and actively engaged by the recipient (i.e., student; Cathcart, Greer, & Neale, 2014;Steen-Utheim & Wittek, 2017).
Regarding feedback as a dialogue (e.g., as "all dialogue to support learning in both formal and informal situations"; Askew & Lodge, 2000, p. 1), IF appears as a useful technique. Discussions with the teacher help students to actively engage with feedback and "to develop their understanding of expectations and standards, to check out and correct misunderstandings and to get an immediate response to difficulties" (Nicol & McFarlane-Dick, 2006, p. 208). Not surprisingly, this perspective has been considered essential for the effectiveness of feedback in higher education (Nicol, 2010;Steen-Utheim & Wittek, 2017).
This dialogue must also include the peer group. They help to expose students not only to their teachers' views but also to alternative perspectives and forms of addressing and solving problems (Van den Berg, Admiraal, & Pilot, 2006). They also provide an opportunity to discuss marking criteria (Broadbent, 2018). Further, as Nicol and McFarlane-Dick (2006) highlighted, "students who have just learned something are often better able than teachers to explain it to their classmates in a language and in a way that is accessible" (p. 211).

Shaping the Teaching (and the Assessment)
Multisided dialogue in IF can also facilitate Principle 7 and lead to an improvement in assessment and marking procedures and criteria. Ambiguous exam questions are relatively frequent, and teachers sometimes make mistakes in their estimations of the "correct answers." In the principle author's experience, open class discussions help the teacher to identify and acknowledge these errors.

The Development of Self-Assessment (Reflection) in Learning
As indicated, the suggested IF technique requires the students to reflect on their processes and outcomes, comparing them to the contributions in the class discussion, critically understanding the standards and criteria and marking their own work (Ajjawi & Boud, 2019). Selfassessment, self-evaluation, and self-grading are therefore involved.
Self-assessment is one of the most interesting topics in the literature on assessment in higher education. It has been consistently reported that it "makes it easier for the student to monitor, direct, and regulate actions toward goals of information acquisition, expanding expertise, and self-improvement, [therefore] constituting a fundamental element for effective learning both during and after the university years" (Tejeiro et al., 2012, p. 792). It seems obvious that students will only be able to act upon the feedback received to close the gap between actual and expected performance if they share some of their teachers' evaluation skills (Snead & Freiberg, 2019). Although it is often argued that students' self-assessment skills should be much more actively strengthened by their teachers (Bourke, 2018), this is still far from reality in many educational Self-assessment, expressly included in Principle 2, can nevertheless be seen as part of the more general concept of self-regulation, defined as "an active, constructive process whereby learners set goals for their learning and then attempt to monitor, regulate, and control their cognition" (Pintrich & Zusho, 2002, p. 250). In fact, Nicol and McFarlane-Dick (2006) indicated that the seven principles of good feedback practice share the overarching aim of facilitating selfregulation. Self-regulation has consistently been found to affect how efficiently students use feedback in their learning (Lawson, Vosniadou, Van Deur, Wyra, & Jeffries, 2019;Orsmond & Merry, 2009). In this regard, and drawing on Hounsell's (2007) notion of sustainable feedback, Carless, Salter, Yang, and Lam (2010) defend the need of "dialogic processes and activities which can support and inform the student on the current task, whilst also developing the ability to selfregulate performance on future tasks" (p. 397).

Timing in Feedback
In the Context of Practice section, the period between the completion of each assignment or exam and reception of marks or feedback ranged from 21 to 29 days, with 1 to 7 days between feedback reception and the completion of the following assignment or exam. This strongly contrasts the recommendations by those who indicate that feedback should be timely-given as soon as possible after submitting work (e.g. Mahsood et al., 2018;Pereira et al., 2016).
The IF technique used in the Cadiz example was applied immediately after the exam, when students may have a clearer memory of their answers and of the processes that led to them. Providing the feedback has a greater impact on students' interest in learning than providing only marks, or both marks and feedback at the same time, and "one might predict that grades will induce an extrinsic orientation that should undermine subsequent interest" (Butler, 1988, p. 480). In this regard, Gibbs (1999) suggested that marks on written work should only be issued after students have responded to feedback comments. However, the desire to know their marks as soon as possible actively engages students in the self-assessment process, which facilitates the ultimate goal of feedback-namely, improving their work (Dlaska & Krekeler, 2017).
Further, IF is provided far before students are required to complete the next exam. This links to the notion of "effective learning" from feedback, which in turn relates to the concept of "feed forward," in which "tutor feedback on a completed piece of work can be used by the student to inform their efforts in future assessments" (Orsmond et al., 2013, p. 242). Finally, it has already been indicated that the dialoguing process during IF may lead the teacher to modify criteria ("correct answers") or even to suppress items before the marks are given and communicated (Bostanci & Sengul, 2018)-as was the case with the principle author of this article.

Optimizing Resources
Students' disagreements with their teachers' marks and criteria are frequent and imply a high number of postexam discussions and appeals (Khanna & Goyal, 2016;Watty et al., 2013). These are time consuming and repetitive because, frequently, the comments and complaints tend to be similar. In our own experience, IF drastically reduces the number of claims to the teacher, thus facilitating a more efficient use of the always-scarce resources. This is especially relevant with the current trends of increasing class sizes, reducing available resources, and the accumulating assignments in the final weeks of modularized courses (McGuinness & Vlachopoulos, 2019). Additionally, it must be noted that IF is particularly efficient with large groups like those at the University of Cadiz.

Barriers to Indirect Feedback
Simply implementing IF will not automatically solve all problems associated with feedback in the context of practice or similar contexts. Specifically, we believe there are some barriers that need to be overcome before the technique can be effective. First, students may not be used to discussing their answers in public or participating in debates; this can be tackled by practicing with short, formative tests during the course, which will also serve to increase reflection and discussion about criteria before the assignment. Second, some teachers may not feel comfortable discussing their exams in the proposed group format. The discussion may reveal deficiencies in their knowledge, procedures, and criteria, as well as in the items of the exam. Rather than a challenge, this should be considered an advantage. As has been already suggested, IF can help teachers improve their practice and refine their assessment of the students' work. It also promotes a sense of fairness because the teacher may be led to acknowledge mistakes and thus to modify marking criteria. Finally, after a 3-hr exam, students and teacher may experience fatigue and not be willing to continue with another session of undetermined length. In our experience, a 15-to 30min break will help with recovering energy and clearing the mind.

Conclusions
IF is not a new theoretical paradigm or a complex educational model. The authors acknowledge that there is little mystery behind this technique, which is quite easy and straightforward to prepare and apply. However, our own experience suggests that it may substantially contribute to overcoming many of the problems and limitations found in educational settings-like the one described in the Context of Practice section. IF may facilitate the development of self-assessment skills and the delivery of timely, quality information about students' learning. It may also facilitate the improvement of teaching and the reduction of complaints and appeals.
Although IF is easy to apply in short-answer exams, it would be interesting-and probably challenging-to adapt it to other forms of assessments like essays, oral presentations, or practical exercises. Perhaps more importantly, we encourage our colleagues to translate the reflections here into evidence, through the adequate research procedure.
A complementary approach not yet developed for this type of feedback but that has great potential is the use of classroom technologies. Via handset devices, mobile phones, or the Internet, these technologies may help collate student responses to the questions discussed and feed them back in the form of histograms or other visual displays. This procedure has been used as a trigger for peer discussion and teacher-managed discussion in large classes (e.g., Zher, Hussein, & Saat, 2016) and appears to be a good complement to the IF technique.