Procedure

This is a cross-sectional study aimed at assessing supervision outcomes. This study was conducted from November 2022 to December 2022. To reach a broader range of study subjects, we coordinated our research objectives with school counselors and professional education program principals at some of the organizing university school counselor professional education programs. Data collection in this study used Google Forms, where the introduction, purpose, and online links of research instruments were conveyed by the school counselor professional education program coordinator to the counselor trainee participants through the WhatsApp group of each class.

Approval for ethical considerations, under the number 95665/EC/IKI/VI/2022, was secured from the Indonesian Counselor Association, Jakarta, Indonesia, a recognized body in the field of professional counseling. There is no compulsion for the respondents to participate in this study, and all respondent's personal information is credentialed. All respondents involved in this study agreed and will use the data for academic purposes or activities.

The demographic profile of respondent’s is shown in Table 1. The respondents of 227 people involved in this study were all counselor trainees who attended school counselor professional education in 2022 and had completed internship activities.

Table 1: Demographic data of counselor trainee (N = 227) 

Instrument

The questionnaire survey was conducted to get a more precise and better understanding of the results of clinical competence and multicultural competence obtained by counselor trainees from the supervision process provided by university supervisors and clinical supervisors. ^{Spence et al. (2002}) mentioned that questionnaires are an option or alternative solution because it is almost impossible to interview respondents individually in an organization to obtain relevant data and information.

We have used the Supervision Outcome Scale (SOS) developed by ^{Tsong and Goodyear (2014}; Appendix A). Regarding reliability and validity ^{Tsong and Goodyear (2014)} have tested and obtained the results the Cronbach's alpha coefficients were .90 for overall supervision outcome, .86 for Clinical Competence subscale, and .94 for Multicultural Competence Outcome. Concurrent validity of the SOS was supported through significant correlations with the SWAI-TS (r = .77, N = 357, p < .001), and its subscales including task (r = .78, p < .001), bond (r = .68, p < .001), and goal (r = .71, p < .001). with these results the SOS is very feasible to use.

This instrument respondents to describe the degree to which supervision with the counselor trainee's current (or most recent) individual supervisor (university or field site) has contributed to the improvement of the following: 1) Client symptoms (decrease in symptoms); 2) Counselor trainee relationship with clients; 3) Counselor trainee counseling skills; 4) Counselor trainee case conceptualization ability; 5) Counselor trainee multicultural counseling skills (e.g., skills that are culturally appropriate in working with diverse clients); 6) Counselor trainee multicultural beliefs/attitudes/awareness (e.g., awareness of your own worldviews); 7) Counselor trainee multicultural knowledge (e.g., knowledge of worldviews of culturally different clients). Five alternative answers are provided, 1 means not helpful at all and 5 extremely helpful.

Data analysis and measurement model

The data obtained are analyzed using the Rasch model, referring to the guidelines created by ^{Bond and Fox (2013}) and ^{Sumintono and Widhiarso (2014}) via Winsteps Software (^{Linacre & Wright, 2000}) as the primary choice for analyzing research data. The benefit of utilizing the Rasch model lies in its ability to generate a measurement scale that maintains consistent intervals, thereby offering precise insights into the participants' characteristics and the caliber of their work (^{Perdana, 2018}). As previously stated, the ranking criteria used in the current study are five ratings (deal rate from Not helpful at all to Extremely helpful), a statistical separation of Likert type ratings 1, 2, 3, 4, and 5.

As shown in Table 2, the average size (logit) of items is 0.00 logit, and the standard deviation is relatively low (0.47), indicating that the size dispersion is not wide across the logit scale in terms of item difficulty. For Person, the average logit is 3.76 logit, indicating in general, respondents tend to get a lot of maximum outcome supervision; the standard deviation of 2.22 indicates a wide degree of dispersion. The average statistics of squared clothing reach an expected value of more than 1 for items and person, and a chi-squared value is significant, indicating uniform conformity with the model (^{Boone et al., 2016}; ^{Engelhard Jr, 2013}). Strata index (equal to or more than three) and reliability (more than .67) of items and statistics of person show excellent reliability (^{Fisher, 2007}).

Table 2: Summary statistics of person and ítem 

*p < .01

Item difficulty

Item difficulty refers to the extent to which an item (question or statement) is difficult for respondents to answer or agree with in the context of research. Technically, item difficulty is measured by the probability that a respondent of a particular ability will answer the item correctly or exhibit the expected behavior (e.g., expressing agreement with a positive statement) (^{Sumintono & Widhiarso, 2014}).

Table 3 describes the classification of items according to the difficulty of the Supervision Outcome Scale item or logit value of item (LVI) according to the Rasch model analysis. The classification of items is divided into four difficulty levels, carried out by dividing the distribution of the item's logit score based on its average value and standard deviation.

Table 3: Supervision outcome item calibration 

In the opinion of the counselor trainee to the University Supervisor, there were four items (57.14 %) in the difficult category for approval by respondents (+0.47 ≥ LVI ≥ 0.00), namely 1 item in the Clinical competence (General) subscale item A1: Client symptoms (decrease in symptoms). There were three items in the Multicultural competence outcome subscale item numbers A5: Your multicultural counseling skills (e.g., skills that are culturally appropriate in working with diverse clients), A6: Your multicultural beliefs/attitudes/awareness (e.g., awareness of your own worldviews), and A7: Your multicultural knowledge (e.g., knowledge of worldviews of culturally different clients). Then there is 1 item (14.28 %) in the Easy category for respondents to approve (0.00 ≥ LVI ≥ −0.4 7), namely in the Clinical competence (General) subscale (item A2). There are two items (28.57 %) in the category of very easy to agree with respondents (LVI < −0.47 logit), namely in Clinical competence (General) subscale items A3: Your counseling skills and A4: Your case conceptualization ability.

In the counselor trainee's opinion of the Supervisor's field site, there were 2 items (28.57 %) in the category of very difficult for respondents to agree with (LVI > 0.4 7 logit), namely in Clinical competence (General) subscale (item number B1) and in Multicultural competence outcome subscale item B5: Your multicultural counseling skills (e.g., skills that are culturally appropriate in working with diverse clients), then two items (28.57 %) in the category difficult for respondents to approve (+0.47 ≥ LVI ≥ 0.00), namely in the Multicultural competence outcome subscale items B7: Your multicultural knowledge (e.g., knowledge of worldviews of culturally different clients) and B6: Your multicultural beliefs/attitudes/awareness (e.g., awareness of your own worldviews). Meanwhile, there are three items (42.85 %) in the Easy category for respondents to approve (0.00 ≥ LVI ≥ −0.47), all in the Multicultural competence outcome subscale items B2: Your relationship with clients, B3: Your counseling skills, and B4: Your case conceptualization ability.

From the results above, it can be seen that the supervision outcome seen from the university supervisor (A) side, especially in the multicultural competence outcome subscale, is more difficult for respondents to approve, and likewise, the supervision outcome seen from the field site supervisor (B) side. From the university supervisor and field site supervisor sides, it is clear that the clinical competence (general) subscale item, especially in item A1: Client symptoms (decrease in symptoms), is difficult to approve, even from the field site supervisor's side, it is a very difficult thing to approve. Meanwhile, from the University Supervisor side and field site supervisor in the competence (General) subscale section, it is an easy part to be approved and even very easy for respondents to approve looking at the University Supervisor side items A3: Your counseling skills and A4: Your case conceptualization ability. A complete summary of supervision outcome rates is shown in Table 4.

Table 4: Distribution of supervision outcome measurements based on Supervision Outcome Scale (SOS) items (N= 227, I = 7) 

^Note: Items with the letter A numbers are intended to be respondents' assessments of supervision outcomes to the university supervisor, and code B is intended to be respondents' assessments of supervision outcomes to field site supervisors.

Differences between respondents’ demographics

An item-person map of the research was generated by Winstep software in which 227 respondents answered 14 items of supervision outcome (7 items intended for the University Supervisor and 7 items intended for the field site supervisor). The left side of the map showed the difficulty level of each item, ranging from the one that respondents easily agree with in the bottom left (logit score -0.82 for item A3: Your counseling skills) to the most difficult to agree to in the upper left (logit score 0.68 for item B1: Client symptoms (decrease in symptoms)). The item worked well and separated respondents with a unidimensional raw variance index of 31.5 %.

The right side showed the distribution of respondents according to their logit score, ranging from a high level of outcome supervision acceptance statement in the upper right (logit score > + 8.99 ) to a low level of outcome supervision acceptance statement in the bottom left (logit score < - 0.88). The number code on the person in question is P: respondent from University 3; M: respondent from University 4; G: respondent from University 2; S: respondent from University 1; R: respondent from University 5.

Table 5: Distribution of supervision outcome based on demographics (university aspects) that have been tested through (N= 227) 

From Table 5, it is known that the highest mean measure came from respondents at university 2 (Mean 4.00) and University 5 (Mean 5.01) and the mean measure came from respondents from university 3 (Mean 3.28). Looking at the mean measure person of respondents from all universities (Mean 3.76), it can be known that there are two universities whose respondents have a low level of supervision outcomes, namely University 1 and University 4. Generally, there is no significant difference in DIF between universities (0.0981 > 0.05).

Table 6: Distribution of supervision outcome based on demographics (years of service) that have been tested through (N= 227) 

Table 6 shows the distribution of supervision outcomes by year of service demographics that were tested with a total of 227 respondents. The F-test yielded a value of 0.36, indicating no significant difference in the supervision outcome scores between the different categories of years of service. The Prob>F value was 0.7827, much higher than the expected threshold of 0.05, confirming that the difference in mean supervision outcomes between the groups based on years of service is not statistically significant. This suggests that, according to these data, there is no strong relationship between the number of years of service and the supervision outcomes measured in the study.

The following analysis reveals differences between respondents' demographic variables (university and years of service) on the supervision outcome items. This was analyzed using differential item functioning (DIF) analysis, which showed that respondents from different universities and with different years of service gave different responses to some items when measuring specific supervision outcome at the item level. F1 F2

Figure 1: Person DIF plot based university on supervision outcome 

Figure 2: Person DIF Plot Based years of service on supervision outcome 

Discussion

The issue of supervision, mainly how the results of supervision have been carried out in professional education programs by supervisors, needs to be carried out on an ongoing basis to improve the quality of education by improving the competence of counselors (^{Johnsen et al., 2021}; ^{Reising & Daniels, 1984}). This issue is important because counselors are recognized as educators in the national education system and have a crucial role in developing potential and empowering counselees to make productive decisions and choices (^{Gunawan & Wahab, 2015}). Like other professions, counseling should also have supervision efforts to maintain its existence and role in society (^{Garrett, 2023}). Supervision in counselor professional education still needs to be improved. It has yet to provide many reports that can be used as reflection and evaluation for the improvement and development of the counselor profession itself.

Referring to article 2 of the regulation of the Minister of Education and Culture number 38 of 2020 concerning procedures for obtaining educator certificates for teachers in positions/service, states that “certification aims to improve the competence of teachers in-service training as a professional in an education unit to fulfill pedagogic, personality, social, and professional competencies following the provisions of laws and regulations”. Then in article 13 states that fulfilling the learning load of the school counselor professional education program in-service training program through recognition of past learning as many as 24 semester credit systems and learning is taken as many as 12 credits. With the number of credits, refer to the internship implementation schedule set by the director general of teachers and education personnel. Especially for practical school counselors in the school counselor professional education program, the in-service training program activities internship is only carried out within 27 days. The minimal time for carrying out internships in in-service counselor professional education programs is a big obstacle and challenge for counselor professional education programs. Also, supervisors should be able to help prospective counselors develop and improve the competencies they must have.

In Article 4 regarding the requirements for prospective students in the professional education program in the position, it is explained that prospective students must fulfill the requirements, including having bachelor academic qualifications; Teachers in positions in educational units organized by the central government, regional government, or educational units organized by the community; registered in the primary education data of the Ministry of Education and Culture; have a Unique Educator and Education Personnel Number; and have completed the required documents. Regarding academic qualifications, they are further regulated by the Director of Professional Teacher Education regulations regarding linearity in the Teacher Professional Education Programs Field of Study in Position in 2022. It is stated that to take part in a professional counselor education position, prospective school counselors are required to have academic qualifications or are graduates of a bachelor's degree from a related study program, including namely Guidance and Counseling, Guidance Counseling, Islamic Guidance and Counselling, Islamic Guidance and Counseling, Islamic Counseling Guidance, Guidance and Counseling, Islamic Education, Christian Counseling Guidance, Christian Guidance and Counseling, Psychology, Educational Psychology and Guidance, Education Guidance and counseling. This shows the requirements for multiple programs; even psychology graduates can participate in an in-service counselor professional education program. Apart from that, it shows that there are various names or designations for study programs or similar field of study in Indonesia. It will also be a challenge to provide supervision for them to achieve the expected competencies.

With the ongoing program, guidance and counseling academics, including lecturers or counselor educators, are trying their best to carry out the educational process for prospective candidates' school counselors. In addition to coursework activities, practicum is more critical in internship activities. Supervision activities are essential to internship activities to ensure counselor trainees carry out clinical counseling practices (^{Li & Peters, 2022}; ^{Nelson & Johnson, 1999}; ^{Sanders, 2022}). Training or educational activities will be successful if one of them, namely supervision, is carried out properly and correctly, providing development or support impacts to trainee counselors.

Data found in this study found that the supervision carried out in the school counselor professional education program can be pretty good. This is evidenced by the mean measure, which shows the value above the logit item value. However, several items are difficult for respondents to agree with, where at the first level, the supervision outcome of the field supervisor can impact client symptoms (decrease in symptoms). Related to supervision that impacts client symptoms (decrease in symptoms) should be important because this is related to professional competence and is directly related to the client (^{Li et al., 2022}). Field supervisors' lack of role and influence in this regard should be a serious matter and must be considered. In clinical practice, the role of a field supervisor, or it can also be called a clinical supervisor, is ideally very large and has a high influence. Every planning and action of the trainee counselor should be known and under the direct supervision of the field supervisor (^{Tipton Fisler et al., 2022}).

Field supervisors were also revealed in this study in terms of multicultural counseling skills (e.g., skills that are culturally appropriate in working with diverse clients) less influential in their supervisory roles. By being the second item that is difficult for respondents to agree on again becomes a severe emphasis. This is mainly an item of the second of the two existing subscales. It gives a complete picture that, in the overall aspect, the field supervisor needs to be considered. As explained by ^{Calloway and Creed (2022}), ^{Ivers and Johnson (2022}), ^{Kim and Cho (2022}), and ^{Ramli et al. (2024}) in the current era, multicultural issues are very vital and impossible to avoid in relationships or interactions between fellow human beings, including in the most crucial relationship (counseling).

Seeing the striking differences in some of the items in the supervision outcome, judging from the comparison of several supervisors in several universities providing education for the school counselor profession, it seems that it also needs to be attention that it is important for professional organizations or forums providing counselor professional education to continue to build communication and make standardized or standardized guidelines in the process supervision. Unfortunately, standards for implementing special supervision for school counselor education in Indonesia have not been found. Regarding the implementation of internships, supervisors are only limited to referring to the internship implementation guide rather than a guide on becoming a professional supervisor. The existing internship implementation guidelines created by the director general of educational personnel are general for school educators. There are no specific ones for supervisors in counselor professional education. ^{Wiyono et al. (2022}) stated that to create optimal competence, it is necessary to carry out structured supervision and apply comprehensive supervision principles. The initial stages of the supervision process involve carefully evaluating the needs and character of the individual teacher or counselor. With a deep understanding of these needs, an effective supervision program can be developed and implemented successfully. Continuous evaluation of the program's results is the next step, allowing for necessary improvements and adjustments. Through this process, it is hoped that there will be an increase in the professionalism of teachers or counselors in carrying out their roles.

Considering the facts above, if referring to some professional education standards, especially school counselors, for example the Accreditation of Counseling and Related Educational Programs (Cacrep) standards, school counselor professional education in Indonesia it is still necessary to do a lot of structuring and development. These efforts continue to be made by professional organizations, in this case the Indonesian guidance and counseling association (Asosiasi Bimbingan dan Konseling Indonesia: ABKIN). However, professional organizations can play a role in the professional development of their members through training, workshops, and the provision of resources; such organizations can help school counselors improve their competencies (^{Sofyan et al., 2023}). ^{Sakinah et al. (2020}) stated that in addressing the condition of counselor professional education, ABKIN should be firm and encourage the government to immediately reform the existing regulations and structuring of counselor professional education. After all, the formation of the identity of the counselor requires the role of various parties, not only professional organizations but also the government as policy makers, and other parties as well as practitioners and academics.

This study has several limitations that need to be noted. First, using a quantitative survey approach may not fully capture the complexity and nuances of the supervision process in counselor professional education. Second, this research only focuses on the clinical and multicultural competency results of the supervision process without considering other factors, such as emotional and social aspects, that might influence the effectiveness of supervision. Third, more than the limited time for internships in professional counselor education programs may be needed for supervisors to provide maximum and comprehensive supervision.

Based on existing limitations, further research can take several directions. First, qualitative or mixed methods research involving in-depth interviews or case studies can be used to better understand intern counselors' and supervisors' experiences and perceptions during the supervision process. Second, future research can explore other factors influencing supervision effectiveness, including emotional, social, and organizational factors. Third, research examining the impact of longer internship duration on the development of intern counselor competencies could provide valuable insight into the structure of an ideal counselor professional education program. Finally, research focusing on developing and validating supervision models specific to the context of counselor professional education in Indonesia can help improve the quality of supervision and educational outcomes.