1. Introduction

In any modern business or organization, project objectives should not be established in an isolated manner. Usually, organizations establish business goals or objectives at strategic, tactical and operational levels [17]. Basili et al. [4] indicate that a critical issue in an organization is the lack of linkage between goals formulated at a strategic level (or management level, considering their own words) with those objectives formulated at an operational (or project) level. Specifically, authors stated that “the problem is not that businesses fail to achieve their objectives, but rather that they do not always state these objectives explicitly or clearly enough to verify that they have indeed achieved those objectives. It is even less clear how a business translates its objectives into its lower organizational levels and into individual projects. At present, no methodology bridges the gap between business strategies and their project-level implementation”.

Considering the first part of the above quote, two decades before Gilb et al. [14] had indicated, as the principle of fuzzy targets, that “projects without clear goals will not achieve their goals clearly”. Considering the second part of the above quote, a decade before Park et al. [33] had said that the right establishment of business goals at different organizational levels determines much of the success in carrying out projects. As a consequence, if goals at an operational level are not in alignment with goals at tactical or strategic levels, the organization can direct its effort and resources in a wrong way.

In addition to business goals, it is also necessary to have valuable information which allows to know if the business goals were achieved. An information need goal is a support goal which is always related to a main or business goal [33]. In our proposal, a particular kind of information need goal is the measurement and evaluation (ME) information need goal or objective, which is always driven by ME activities and methods. In summary, information need goals permit knowing the level of achievement of business goals as well as to give the necessary information to reach them. Information need goals are grounded in different and suitable types of analysis that can be made using qualitative and quantitative data and contextual information yielded by ME activities. It is important to remark that ME should not be an end in itself but a key factor to reach information need and business goals in an organization. As pointed out in [4], “quantitative data is a prerequisite to understanding the relationships between the business and project-level goals and verifying the achievement of objectives”.

On the other hand, commonly, organizations arrange their work by means of projects in order to operationalize the established goals. An engineered way to organize the work is by performing project management. Project management allows planning, executing and controlling the activities and resources of the project regarding the adopted life cycle [34]. For a given project life cycle, strategies should be used in order to help in the goal achievement. A strategy defines a specific course of action to be followed for a specific goal purpose [9, 24]. That is, it specifies concretely what should be done and how should be performed. Strategy is a frequently used and broad term, so for our purposes, we have defined it as: “principles, patterns, and particular domain concepts and framework that may be specified by a set of concrete processes, in addition to a set of appropriate methods and tools, as core resources, for helping to achieve a project's goal purpose” [5].

Regarding ME strategies, we conducted a systematic literature review [31], in which we have observed that very few approaches supported integrated strategies. Also, the research allowed us to detect that just one integrated strategy (GQM+Strategies [4]) included multilevel goals. The premise for an integrated strategy is that it supports three integrated capabilities simultaneously, namely: (1) the domain conceptual base and framework, (2) the process perspective specifications, and (3) the method specifications. In [31], another integrated strategy was GOCAME (Goal-Oriented Context-Aware Measurement and Evaluation) [26]. For this strategy, the first capability, is named the C-INCAMI (Contextual-Information Need, Concept Model, Attribute, Metric and Indicator) conceptual framework, which explicitly specifies the ME terms, properties, relationships and constraints, in addition to their grouping into components. The second capability is the process specifications [5], which describes a set of activities, tasks, inputs and outputs, artifacts, roles, and so forth. Besides, process specifications can consider different process perspectives such as functional, behavioral, informational and organizational [11]. Usually, process specifications primarily state what to do rather than indicate the particular methods and tools (resources) used by specific activity descriptions. The third capability provides the ability to specify methods (such as metric and indicator specification templates), which ultimately represent the particular ways to perform the ME tasks.

In addition to the GOCAME strategy, we have defined a set of strategy patterns [35, 38] for the measurement, evaluation and change (MEC) domain. A strategy pattern represents a reusable solution for instantiating the suitable strategy considering the project's goal purpose, and the amount of included quality views. (Note that the quality view concept [36, 38] stems from the relationship between an evaluated entity supercategory -e.g., resource, process, software product, system, among others- and its quality focus -e.g., resource quality, process quality, internal quality, external quality, among others. In turn, for each quality focus a tailored quality model which includes characteristics and attributes should be used for evaluation, selection or improvement purposes).

Regarding the above mentioned research ([4, 33, 34]) and raised issues such as “... At present, no methodology bridges the gap between business strategies and their project-level implementation” [4], we also consider that it is paramount for software organizations to have a systematic and holistic approach which is able to establish and align goals at different organizational levels, and arrange work by means of ME/MEC projects and strategies for helping to achieve these goals. In this direction, Basili et al. have developed GQM+Strategies. However, we observe some opportunities for improvement to this approach, as we highlight in Section 2. As a result, we have developed a systematic approach that considers and relates multilevel goals, projects, strategies and strategy patterns, which is called the Holistic Quality Multilevel and Multipurpose Evaluation Approach (hereafter, for short: Quality Evaluation Approach). This approach embraces four key aspects or principles: i) the definition of multilevel business and information need goals; ii) the definition of different evaluation purposes for goals that also include quality views and their relationships as well; iii) the formulation of ME/MEC projects for operationalizing goals; and iv) the adoption of strategy patterns for the instantiation of specific strategies that help to achieve goal purposes.

These four principles of the Quality Evaluation Approach rely also on conceptual bases which are structured into ontologies. For instance, the abovementioned C-INCAMI conceptual framework deals with the ME domain and represents components such as nonfunctional requirements, measurement, evaluation and analysis. More recently, we have developed the quality multiview modeling framework that includes an ontology of quality views [36]. A year ago, we added the goal and project components [37] for linking business and information need goal concepts with project, strategy and nonfunctional requirements concepts. Therefore, the main contributions of this research are summarized in the following list:

i) The specification of a couple of new terms included into the goal and project conceptual components -which were left implicit in [37]- such as types of organizational levels and strategy pattern. As a result, the enhanced conceptual base is able to relate business and information need goal terms with ME information need, entity category, quality focus, and quality view terms in addition to with project, strategy and strategy pattern concepts.

ii) The definition of the Quality Evaluation Approach stepwise, which lists the necessary activities to establish goals and projects at different organizational levels. The step-by-step applicability can help stakeholders in the process of relating business goals with ME information need goals in addition to formulate and perform ME/MEC projects by using the suitable strategies. Also, while exemplifying the step-by-step sequence, an evaluation scenario for improving the Facebook mobile app is used.

iii) The documentation of evaluation scenarios for business/information need goals at operational level supported by strategies and strategy patterns. For this end, we specify a template which contains a set of items such as the business goal purpose, the evaluation strategy pattern in conjunction with its process specification, among others. We have identified different evaluation strategies that realize purposes aimed at understanding, improving, monitoring and controlling, comparing and selecting entities regarding also the amount of quality views. Examples of scenarios in which each strategy helps to achieve a given purpose are documented as well.

Following to this Introduction, Section 2 motivates our proposal and analyzes related work on ME/MEC approaches and integrated strategies that consider multipurpose and multilevel goals, which in turn are supported by conceptual bases. Section 3 deals with the goal and project conceptual components and their relations with the previously developed C-INCAMI conceptual components. Section 4 defines the step by step of our Quality Evaluation Approach and illustrates its applicability through a proof of concept using as scenario the improvement of the Facebook mobile app. Section 5 documents evaluation scenarios for business/information need goals at operational level for any organization, by addressing well-established purposes supported by strategies and strategy patterns. Finally, Section 6 summarizes the main contributions of this research and outlines future work.

2. Motivation and Related Work

In Software Engineering there exists research that addresses the importance of linking and aligning measurement goals usually formulated at an operational level of an organization, with business goals formulated at this level or often at higher levels such as tactical or strategic [2, 4, 15, 33, 40]. This alignment is paramount for sound and robust decision-making endeavors since ME information need goals can give meaningful information for learning and knowing in what extent a business goal has been achieved. In some cases, ME information need goals can also give information on how to reach the business goal.

To engineer these issues, it is ultimately paramount for software organizations to have a holistic and systematic evaluation approach. This approach should be able to establish and align goals at different organizational levels, and arrange work by means of ME/MEC projects and strategies for helping to achieve business goals, information need goals, and ME information need goals for different purposes such as to understand, monitor and control, improve, compare and select, among others. In any case, by running a specific ME/MEC strategy for a given purpose produces data (by using metrics) and information (by using indicators) which are input for analyzing results, and ultimately, for decision making.

As indicated in the Introduction Section, such a holistic approach should consider core aspects such as the definition of multilevel business and information need goals; the definition of different purposes for evaluation goals that also include quality views and their relationships; the formulation of ME/MEC projects for operationalizing goals; and the instantiation of specific strategies that help to achieve goal purposes. (Note that ME/MEC strategies can stem from the instantiation of strategy patterns). Additionally, as the reader can surmise, given the abundant multidomain terminology that the holistic approach embraces, a minimum set of terms, attributes and relationships should be explicitly specified and agreed.

However, reviewing the related literature, we have not detected approaches with robust conceptual bases (i.e., ontologies) that integrate terms such as business goal, organizational level, information need and ME information need goal, strategy, strategy pattern, project, ME/MEC project, among others, except in Barcellos et al. [2] who include some of these terms. On the other hand, considering that a strategy is an important resource in helping to achieve a project's goal purpose, it should be noted, as discussed in [31] that there exist few documented strategies that integrate simultaneously the process specifications, the method specifications, and the ME conceptual base capabilities. As commented previously, we conducted a systematic literature review as research method, in which the main aim was the selection of integrated strategies for evaluation and comparison purposes. Basically, among the preselected strategies were GQM+Strategies and GOCAME, among others. It is important to remark the research allowed us to detect that only GQM+Strategies supports multilevel goals. Furthermore, there are different ME purposes documented elsewhere [6, 8, 12, 18, 20, 34] such as to understand, improve, monitor and control, compare and select, among others. Consequently, different strategies should be considered for each purpose, varying slightly each other in its own process and method specifications, as we discuss and illustrate in Section 5. Therefore, having well-specified processes and methods for each strategy can foster their applicability and communicability. These issues have often been neglected in the literature as well.

Regarding the ME approaches, GQM+Strategies is an integrated strategy. It includes a goal-oriented framework for the design and implementation of software measurement projects at different organizational levels. Unlike its predecessor, GQM (Goal Question Metric) [3], the business goals that GQM+Strategies defines can be aligned at different organizational levels through the establishment of strategies. In [4], strategies define objectives for reaching goals and require the definition and fulfillment of lower level goals. Therefore, business goals are linked to measurement goals using GQM. GQM+Strategies has a terminological base structured as a glossary where the main used terms are defined. But it lacks the semantic richness that an ontology provides. Also, well-established process specifications for different strategies regarding different goal purposes are missing.

Another measurement approach widely accepted in the industry that helps manage software development projects is PSM (Practical Software Measurement) [23]. It is an information-oriented approach that describes a software measurement process, also being part of a comprehensive management program and software development project management. PSM describes how to define and integrate measurement requirements, collect and analyze measurement data and implement the entire measurement process in an organization. PSM was one of the sources for the development of the ISO/IEC 15939 standard [19], as indicated in http://www.psmsc.com/iso.asp. And after the formal appearance of this ISO document, PSM was updated according to this standard as well. We consider PSM an integrated strategy since includes process specifications, method specifications, and a glossary of terms for the ME domain. However, the explicit support to multilevel goals and multipurposes is missing. That is, PSM (likewise GQM+Strategies) considers just one strategy for all the purposes.

Another related work is the Goal-Driven Measurement approach [33], which describes a process for the definition of measurement goals aimed at helping to understand aspects of the organizational goals. The process begins indicating that the organization should establish business goals at any organizational level. From these goals, questions or issues related to what stakeholders want to know or learn emerge, as also pointed out in [39]. These issues allow identifying quantitative information through the decomposition of the business goal into related subgoals. With the list of subgoals and issues, entities and attributes are identified, following the GQM model and templates. This approach offers guidelines that serve as an important reference for engineers and practitioners, since these detail the approach process through the goal decomposition to the measure quantification and analysis. It also uses the terms that GQM defines. But authors in [33], neither explicitly define concepts such as business goal, organizational level, information need goal, strategy and strategy pattern, among others, nor specify different strategies regarding the abovementioned ME purposes.

Additionally, Goethert and Fisher [15] describe the GQ(I)M (Goal-Question-Indicator-Measurement) approach that combines the most prominent aspect of the strategy described in [33] with the Balanced Scorecard [21] paradigm for the decomposition of strategic goals into subgoals. GQ(I)M approach is used to systematically establish organizational goals for each quadrant that Balanced Scorecard defines. Also, it helps identifying and defining measures and indicators. GQ(I)M includes a glossary with the definition of some terms, which are not present in GQM. It is worthy to remark that GQ(I)M does not deal with the use of integrated strategies to fulfill business goals from information need goals.

In [2], Barcellos et al. define a measurement goal subontology. They argue that the measurement should be aligned with organizational goals in order to produce useful data for decision making. This contains terms, relationships and restrictions related to measurement, goal and organization concepts. Also authors state that indicators are the measures which can be used to assess the level of goal achievement. However, the use of integrated strategies are not described in their research. Unlike [2], our proposal formally establishes the use of integrated strategies for helping to achieve both business and information need goals for different evaluation purposes.

The business goal subontology (specified in the next section) discusses some concepts which are not modeled in [37]. For example, the strategy pattern concept which represents a knowledge asset. A ME/MEC strategy pattern embeds a reusable and customizable solution for a recurrent ME/MEC project problem in similar contexts [38]. Likewise the process guideline offered in [33], we describe in Section 4 the step-by-step applicability of our approach, which in [37] was not explicitly considered. Lastly, a contribution of this work, which was left apart in [37], is the documentation of evaluation scenarios for business/information need goals at operational level supported by strategies and strategy patterns. Hence, in Section 5, we specify a template which contains a set of items such as the business goal purpose, the evaluation strategy pattern in conjunction with its general process specification. We have identified different evaluation strategies that deal with purposes aimed at understanding, improving, monitoring and controlling, comparing and selecting entities regarding also the amount of quality views.

3. Linking Business and Information Need Goals with Project, Strategy and Strategy Pattern: A Conceptual Base DescriptionTable 3

As commented above, the proposed Quality Evaluation Approach is grounded on robust conceptual bases as a way to formally define the terms and relationships involved in quality measurement, evaluation and improvement issues. There exist different ways to structure conceptual bases such as glossaries, taxonomies and ontologies. In the Quality Evaluation Approach, conceptual bases are structured as ontologies. The benefits of ontologies are well known. Just to write down an often cited quote: “An ontology may take a variety of forms, but necessarily it will include a vocabulary of terms, and some specification of their meaning. This includes definitions and an indication of how concepts are inter-related which collectively impose a structure on the domain and constrain the possible interpretations of terms. An ontology is virtually always the manifestation of a shared understanding of a domain that is agreed between a number of agents. Such agreement facilitates accurate and effective communication of meaning, which in turn leads to other benefits such as inter-operability, reuse, and sharing” [41]. Therefore, we can state that an ontology is a richer mechanism than others for structuring a conceptual base, which can also be specified by means of ontological languages such as OWL (Ontology Web Language). Hence, ontological languages are able to support semantic processability.

Originally, the C-INCAMI conceptual framework was composed of six conceptual components such as non-functional requirements, measurement, evaluation, context, quality view and project (see these packages in Fig. 1). The terms, attributes and relationships of these components arose from the ME ontology documented in [25, 26] and from the quality view ontology formalized in [36, 38]. In a complementary research published in [5], the measurement and evaluation components were semantically enriched by the process ontology. Recently, in [37], we have argued that C-INCAMI had not the necessary terms for linking ME information need goals with business goals. This is important for representing the alignment between goals at different organizational levels, such as strategic, tactical and operational. Hence, we have added the business goal component and enlarged the project component as shown in Fig. 1.

Basically, these components specify both business and information need goals at different organizational levels, which can be operationalized by projects and achieved by means of strategies. The terms, attributes and relationships of these conceptual bases were defined considering documents such as [1, 2, 4, 10, 16, 19, 30, 34, 40] with the aim of having some adherence or contrast to well-known sources. However, in [37] a couple of terms were left implicit, e.g., the specific levels of the Organizational Level term and the Strategy Pattern term and its attributes. Therefore in Fig. 1, we add these terms for the business goal and project components in addition to the previous ones included in [37]. In tables 1, 2 and 3 the definition of all terms, relationships and attributes are presented. In the sequel, descriptions of these terms are made. Note that terms are highlighted next in italic the first time they appear in the text.

An organization is an entity that comprises people and is structured and managed to establish and pursue goals. The organization establishes goals which contain an explicit declaration (statement) about the major purpose that should be achieved in a period of time (timeframe). The purpose of an organizational goal is the rationale for achieving it (e.g., to develop, maintain, increase, reduce, understand, monitor, control, improve, predict, analyze, among others). The established time frame for the achievement of a goal can range from short and intermediate term to long term. Also, goals can be classified into business and information need goals. Business goals are the main or primary goals that an organization sets considering its mission and vision. Goals can be formulated at different organizational levels. An organizational level represents a management and decision-making level. Commonly, three levels are identified in the literature [4, 17, 21] such as strategic, tactical and operational levels. In turn, a business goal can be divided into business subgoals.

On the other hand, information need goals are support goals for business goals. Usually, they provide useful information in order to know the degree of achievement of business goals. An information need goal can also require ME information need goals. The latter is a more specific type of information need goal which is driven by ME activities. Note that a ME information need specifies an object to be evaluated (entity category) considering also a quality focus (see in Fig. 1 the nonfunctional requirements component, which includes the quality view one).

Figure 1: Key concepts from the business goal domain (business goal package) and relationships with some concepts for the project and nonfunctional requirements packages. Note: PO means Process Ontology; ME, Measurement and Evaluation; and MEC, Measurement, Evaluation and Change  

Tabla 1: Term definitions for the business goal and project components  

Table 2: Attribute definitions for the business goal and project components 

Table 3: Relationship definitions for the business goal and project components 

An organization arranges work by means of projects. In turn, projects operationalize established organizational goals. There exist different types of projects such as development, maintenance, evaluation, among others. For example, a development project operationalizes a business goal that has as purpose to develop a new software product or system. Particularly, in this work we focus on ME projects, which operationalize ME information need goals, as well as on MEC projects which operationalize both business goals and their related ME information need goals with the purpose of improvement. Fig. 1 shows that a MEC project is composed by a ME subproject and a change subproject in which changes are driven by measurement and evaluation. A change project operationalizes a business goal with the purpose of changing or improving the current state of an entity.

Additionally, project management is the set of managerial processes aimed to achieve the goal operationalized by a project. In [34], it is defined as “the application of knowledge, skills, tools, and techniques to project activities to meet the project requirements”. The project management process produces a project management plan, that is, the document which describes how the project should be executed, monitored and controlled. Besides, a project adopts a project life cycle which indicates the stages the project goes through from its beginning to its end. The project life cycle involves at least resources and work definitions and uses strategies. We have defined the strategy term in the Introduction Section, so we consider that definition in Table 1. Hence, a strategy is a resource which helps to achieve a goal.

It is worthy to remark that our main line of research was devoted to ME and MEC strategies. Therefore, our developed strategies are intended to help to reach goals that are operationalized by ME or MEC projects. In addition to strategies, we have built recently a set of strategy patterns [32, 35, 38]. A strategy pattern is a knowledge asset that includes a reusable and customizable solution to a recurrent project problem in similar situations. Particularly, a ME/MEC strategy pattern is a reusable and customizable solution which deals with ME/MEC project problems. Fig. 1 shows that a strategy may instantiate a strategy pattern. In turn, a strategy pattern has a structure compound of three integrated capabilities, namely: the domain conceptual base, process specifications, and method specifications. The domain conceptual base embraces a terminological base for a given domain, for instance, the ME domain. The second capability describes what to do by means of a model which relates a set of process elements such as activities, tasks, inputs, outputs, pre- and post-conditions, artifacts and roles. A process specification can also consider different process perspectives [5, 11]. The third capability represents how an activity should be carried out using a method specification based on a procedure and rules. A specific strategy pattern is documented in Appendix I, which includes the following items: name, alias, intent, motivation, applicability, structure, known uses, and scenario of use.

Finally, note that the terms included in the business goal component are the minimum and necessary ones for describing goals. The same occurs with the project component which relates terms as project, strategy and strategy pattern. The reader can surmise that, for instance, for the project management term there could be more specific related terms, but are not represented in our subontology due to its intended scope and objective.

In the next section, we instantiate these key terms for a MEC project, aimed at illustrating the main activities of the proposed Quality Evaluation Approach.

4. Quality Evaluation Approach: Step-by-step Applicability

As commented in the Introduction Section, we consider software organizations should foster a systematic approach which is able to establish and align goals at different organizational levels, and arrange work by means of projects and strategies for helping to reach these goals. In this direction, we have developed the Holistic Quality Multilevel and Multipurpose Evaluation Approach. This approach is based on four principles, viz. i) the definition of multilevel business and information need goals; ii) the definition of different evaluation purposes for goals that also include quality views and their relationships as well; iii) the formulation of ME/MEC projects for operationalizing goals; and iv) the adoption of strategy patterns for the instantiation of specific strategies that help to achieve goal purposes. It also relies on conceptual bases (subontologies) related to these aspects as analyzed in Section 3.

In this Section, we present the step by step of our approach which defines the necessary activities to establish goals and projects at different organizational levels. The step-by-step applicability can help stakeholders in the process of relating business goals with ME information need goals in addition to formulate and perform ME/MEC projects. Next, we list the approach main steps (S) or activities:

S.1. Establish a business goal at any organizational level.

S.2. Refine the business goal, if necessary, into tactical/operational business goals.

S.3. Establish information need goals for each business goal at the corresponding organizational level.

S.4. Formulate ME/MEC projects for those goals that require ME activities.

S.4.1. Select a strategy pattern for each ME/MEC project. For this selection, look at the amount of quality views and purpose involved in the project's goal statement.

S.4.2. Per each selected strategy pattern, identify the concrete ME information need/business goals from the pattern process specification.

S.4.3. Instantiate a strategy appropriately from each strategy pattern. Also, schedule this resource into the project life cycle accordingly.

S.5. Perform the ME/MEC projects.

S.6. Check the achievement of business goals by analyzing information need goals.

In order to illustrate the approach steps, we employ a proof of concept for the evaluation and improvement of the Facebook mobile app considering multilevel goals, a specific strategy and its corresponding strategy pattern. This particular Facebook evaluation scenario is shown in Fig. 2 using BPMN (Business Process Model & Notation). Additionally, many terms of the analyzed components are instantiated in Fig. 3 and 4 following the same scenario.

Figure 2: Main instantiated activities of the Quality Evaluation Approach for the Facebook mobileapp scenario, using BPMN  

Let's suppose “Facebook Inc.” organization establishes at strategic level (S.1 in Fig. 2) the following business goal “Increase 20% the number of the Facebook mobile app' users for the 2016 year”, which is operationalized by a specific project (see also Fig. 3). For this business goal, two business subgoals are established in S.2:

(a) “Increase 5% the Facebook mobileapp advertising” (at tactical level), and;

(b) “Improve 10% the Facebook mobileapp usability in 6 months” (at operational level).

With regard to the (b) business subgoal, it can be achieved by making changes on the Facebook mobile app, which are driven by ME activities. Therefore, this subgoal is operationalized by a MEC project (see Fig. 3). To give supporting information to this business subgoal an information need goal can be established in S.3. The statement of this information need goal is “Analyze if usability has improved 10% after changes” across the 6-month time frame. Therefore this information need subgoal will allow to know the extent to which the (b) business subgoal has been achieved after making MEC activities. (Notice that the (a) business subgoal here is not analyzed since surely it will require subgoals, projects and strategies related to the marketing area, which is outside the scope of this article. Consequently, its respective step for the information need goal is not modeled in Fig. 2).

Going a step forward (S.4.1), a strategy pattern may be selected for the (b) subgoal. To this end, the business goal statement (“Improve 10% the Facebook mobileapp usability in 6 months”), which embeds the “improve” purpose is compared against the intent field in the template of each strategy pattern stored in the catalog. After performing this matching -which also considers the amount of quality views, as we see later on- the selection of the suitable strategy pattern is made.

Specifically, the “Improve 10% the Facebook mobileapp usability...” statement in the (b) business subgoal involves one quality view, i.e., the “System Quality View”. This is so, because the stated concrete entity is the “Facebook mobile app” that belongs to the “System” entity category, and the “Usability” characteristic that is related to the “External Quality” focus. Therefore, GoMEC_1QV (Goal-oriented Measurement, Evaluation and Change for One Quality View) [35, 38] is the suitable strategy pattern to be selected. This pattern is applicable to MEC projects in which the purpose is to improve the quality focus of the evaluated entity for one quality view. Note that GoMEC_1QV is documented in Appendix I.

Once S.4.1 was performed, the S.4.2 step considers the ME information need/business goals that the pattern process specification determines. Basically, GoMEC_1QV establishes three subgoals viz. (i) understand the current quality state of the entity, (ii) make changes on it, and (iii) understand the ulterior quality state (the improvement) after changes. This situation is also analyzed in subsection 5.2.

Particularly, in Fig. 3, the three concrete ME information need/business goals are: (i) “Understand the Facebook mobile app usability weaknesses”, (ii) “Apply changes on the Facebook mobile app current version”, and (iii) “Understand the Facebook mobile app usability after changes”. Analyzing this figure, we also see that the (ii) subgoal is a business goal, which is operationalized by a change project, while the rest of the subgoals are ME information need goals, which are operationalized by ME projects. In consequence, the three subprojects compose the MEC project which is carried out by using a particular strategy.

In the S.4.3 step, we use the GOCAMEC (Goal-Oriented Context-Aware Measurement, Evaluation and Change) strategy that instantiates the GoMEC_1QV strategy pattern. GOCAMEC customizes the process and method specifications defined in this pattern (see Fig. I.2, and tables I.1 and I.2, in Appendix I).

Figure 3: Scenario instantiation where a Business Goal from the strategic level is decomposed in subgoals which are supported by Information Need Goals and ME Information Need Goals. Note: FmApp means Facebook mobile Application and ME, Measurement and Evaluation  

Figure 4: Instantiation of terms for the quality_view and nonfunctional requirements components for the given ME Information Need Goal. Note: FmApp means Facebook mobile Application and ME, Measurement and Evaluation  

In order to illustrate the above (i) ME information need goal, Fig. 4 instantiates some terms of the quality_view and nonfunctional requirements components, which were specified in Fig. 1. For the “Understand the Facebook mobile app usability weaknesses” subgoal, the quality focus is “External Quality”.

The External Quality focus is represented by a quality concept model which includes the “Usability” characteristic (Calculable Concept), as well as the “Understandability” and “Operability” subcharacteristics. In turn, the subcharacteristics combine attributes. Note that Fig. 4 includes just one attribute (i.e., the “Permanence of main controls” attribute) in order not to clutter the diagram. The complete requirements tree specification for this case study can be found in [29]. Furthermore, the above ME information need also specifies the object to be evaluated, which is the “Social network application”. This is an instance of the entity category which pertains to the “System” entity super category. Finally, the association between the quality focus and the entity super category determines the quality view. For our proof of concept, we instantiated the “System Quality View”.

Once the ME/MEC projects, for those goals that require ME activities were planned and scheduled -by assigning resources such as the suitable strategy-, the S.5 step is performed.

Lastly, in the S.6 step, evaluators should check the achievement of business goals by analyzing information need goals. In our approach, this is a bottom-up analysis in which the interpretation of the information need goal at operational level allows to inform not only that level, but also higher levels as the information is aggregated and rolled up.

To our scenario, using the measure and indicator values that the MEC project yielded in the ME activities, we can “Analyze if usability has improved 10% after changes” and then to understand if the linked business subgoal, i.e., “Improve 10% the Facebook mobile app usability in 6 months” has been achieved. If the case were that it was not achieved and there would be time within the 6-month time frame, a new change and evaluation cycle can be performed using GOCAMEC.

Ultimately, because all multilevel goals are to some extent linked, measurement, evaluation and improvement planning and results are organization-wide rather than limited to a single project or department, as Basili et al. state in [4].

5. Evaluation Scenarios for Business and Information Need Goals at Operational Level supported by Strategies and Strategy Patterns

As described above, the Quality Evaluation Approach promotes the decomposition of main business goals. That is, a business goal at top level should be refined into operationalizable business goals at lower level, specifically at operational level. Also, it promotes the establishment of information need goals for business goals, and the formulation of ME/MEC projects using strategy patterns accordingly. In order to select the suitable strategy pattern, two main issues should be identified from the statement of an operational business goal, viz. the purpose and the amount of quality views to be considered. With regard to goal purposes in the software measurement literature, different purposes are documented [6, 7, 10, 12, 13, 18, 20, 34]. These evaluation purposes are basically to understand, characterize, improve, predict, assess, select alternatives, change, monitor and control.

Looking at the above literature, it is worthy to remark that there is no broad consensus about the meaning of purposes. Most of these works consider that characterize and understand are the same purpose, which involve

understanding or forming a snapshot of the current state of an entity for establishing baselines for future assessment [7, 13, 18]. Also, the improve purpose is related with the identification of risk, root causes, inefficiencies and other opportunities for improving the entity quality. Sometimes, the improve purpose also implies introducing changes [18].

On the other hand, the monitor purpose aims at following the trends or evolution of the performance or state of an entity [6]. Monitoring is defined in [20] as “continual checking, supervising, critically observing or determining the status in order to identify change from the performance level required or expected”. Often, related to the monitor purpose is the control purpose. The latter is devoted to identifying deviations that influence the state or performance of processes and products in order to alleviate risks [6].

Additionally, control is defined in [34] as “comparing actual performance with planned performance, analyzing variances, assessing trends to effect process improvements, evaluating possible alternatives, and recommending appropriate corrective action as needed”. In some cases, monitoring and controlling are considered as a single purpose [6, 18], which involves a continual evaluation and sometimes introducing changes in order to meet the expected quality level.

In summary, we observe an opportunity to gain consensus on the meaning of purposes. To this aim, an ontology for evaluation purposes should be developed. This section is not devoted to develop such an ontology but rather, it is focused on identifying a set of evaluation scenarios or situations related to evaluation purposes commonly used in the literature.

Therefore, we will specify scenarios for understanding, improving, monitoring and controlling, comparing and selecting purposes. Specifically, we specify a template which contains items such as the description of the scenario, the business goal purpose which is supported by an analyze purpose from the information need goal standpoint, the amount of involved quality views, an example of the evaluation scenario, the suitable strategy pattern to be instantiated regarding the purpose type and the amount of quality views. Besides, the template includes the generic process specification of the strategy pattern and the concrete evaluation strategy to be applied in order to achieve the operational business goal.

In the following subsections we document six evaluation scenarios. For all scenarios, the corresponding business goal purpose can be achieved by using the suitable strategy, which is driven by measurement, evaluation, analysis and, when necessary, change activities. So in subsection 5.1, we consider the scenario for the understanding purpose and one quality view; in subsection 5.2, we describe the scenario for the improving purpose and one quality view. Also, the situation for the same improving purpose but for two related quality views is documented in subsection 5.3. In subsection 5.4, we consider the scenario for the monitoring and controlling purpose for one quality view. Then, in subsection 5.5, we document the scenario for the selection of one alternative from a set of competitive entities. Finally, the scenario for comparing competitive entities and adopting the best capabilities or characteristics to a target entity is described in subsection 5.6. At the end of each subsection we summarize some specific issues for a better comprehension of the evaluation scenario. Additionally, in subsection 5.7, we highlight some general issues for all of them.

Ultimately, the illustration of these scenarios can help the reader to be aware what the S.4.1 (“Select a strategy pattern for each ME/MEC project”) step involves considering business goal purposes.

5.1 Evaluation Scenario for the Understanding Purpose

Description: The business goal purpose at operational level is to understand the current state of an entity, in a given context, for a set of characteristics and attributes related to a quality focus, through the systematic use of an understanding strategy driven by measurement, evaluation and analysis activities. The measurement activity is performed by quantifying attributes by means of the selected metrics. The evaluation activity is performed by interpreting characteristics and attributes by means of indicators. The analysis is based on determining strengths and weaknesses of the evaluated entity in a given moment, which produces a conclusion and recommendation report.

Business Goal Purpose: Understand

Amount of Quality Views: One. Note: Table 4 shows examples of quality views

Example of Evaluation Scenario:

* Business Goal statement: Understand the Security level of the SIU Guaraní System in the Engineering School at UNLPam (Universidad Nacional de La Pampa)

* Entity Category: System

* Concrete Entity: SIU Guaraní, in the context of the Engineering School at UNLPam

* Quality Focus: External Quality

* Characteristic: Security (Subcharacteristics: Confidentiality, Integrity and Authenticity)

* Quality View: System Quality View

Strategy Pattern to be instantiated: GoME_1QV (Goal-oriented Measurement and Evaluation for One Quality View) [38]

Process Specification in GoME_1QV:

Figure 5: Functional and behavioral process perspectives for the GoME_1QV pattern 

Abridged Process Specification:

Figure 6: Behavioral process perspective for the GoME_1QV pattern 

Strategy to be applied: GOCAME (Goal-Oriented Context-Aware Measurement and Evaluation). This strategy was used in [28] for the exemplified Security evaluation scenario.

In Fig. 5 we have specified the behavioral and functional process perspectives for the GoME_1QV strategy pattern. The specification shows sequences, parallelisms in conjunction with inputs and outputs of the activities. In order not to clutter the process model with all this information, we use an abridged process specification (see Fig. 6) which considers just the behavioral process perspective for GoME_1QV. Also, it includes the business goal with its purpose and the information need goal -whose purpose is always analyze. Hereafter, in the following scenarios, we just consider the abridged process specification for each pattern. Besides, in order to reduce the complexity for the reader, “Design the Measurement” and “Design the Evaluation” activities from Fig. 5 were integrated into the A2 activity in Fig. 6 (i.e., the “Design Measurement and Evaluation”). Likewise, “Implement the Measurement” and “Implement the Evaluation” activities from Fig. 5 were integrated into the A3 activity (i.e., the “Implement Measurement and Evaluation”).

The A4 ("Analyze Results") activity permits to draw conclusions and recommendations about the strengths and weaknesses of the evaluated entity in a given moment, just like a snapshot analysis of the current state of the evaluated entity. Usually, in this scenario, the frequency of ME (A3 activity) is not an issue, as does it is in the monitoring and controlling scenario, which is illustrated in subsection 5.4.

5.2 Evaluation Scenario for the Improving Purpose

Description: The business goal purpose at operational level is to understand and improve the current state of an entity, in a given context, for a set of characteristics and attributes related to a quality focus, through the systematic use of an improvement strategy driven by measurement, evaluation, analysis and change activities. The measurement activity is performed by quantifying attributes by means of the selected metrics. The evaluation activity is performed by interpreting characteristics and attributes by means of indicators. The analysis is based on determining weaknesses or vulnerabilities of the evaluated entity in a given moment, which produces a conclusion and recommendation report about opportunities for improvement. The improvement can be achieved by means of changes in the entity and/or in its context. Once the changes were performed, the new entity version (and/or context) is re-evaluated for analyzing the actual impact and gain of the improvement.

Business Goal Purpose: Improve

Amount of Quality Views: One. Note:Table 4 shows examples of quality views

Example of Evaluation Scenario:

* Business Goal statement: Improve 10% the Facebook mobile application Usability in 6 months

* Entity Category: System

* Concrete Entity: Facebook mobile application

* Quality Focus: External Quality

* Characteristic: Usability (Subcharacteristics: Appropriateness Recognizability, Learnability, Operability, User Error Protection and User Interface Aesthetics)

* Quality View: System Quality View

Strategy Pattern to be instantiated: GoMEC_1QV (Goal-oriented Measurement, Evaluation and Change for One Quality View).

Notice that this pattern is fully documented in Appendix I.

Abridged Process Specification:

Figure 7: Behavioral process perspective for the GoMEC_1QV pattern. The functional and behavioral process perspectives for this pattern is shown in Fig. I.2 

Strategy to be applied: GOCAMEC (Goal-Oriented Context-Aware Measurement, Evaluation and Change). This strategy was used in [35, 38] for the exemplified Usability evaluation scenario. Note that this evaluation scenario was also used as proof of concept to illustrate some steps for the Quality Evaluation Approach in Section 4.

Unlike the GoME_1QV strategy pattern, the GoMEC_1QV pattern includes activities devoted to develop a new entity version (see A5 and A6 activities in Fig. 7). Also, it includes re-evaluation cycles as depicted by the flow from A6 to A3 activities. Changes are designed and implemented according to the improvement recommendations given in the A4 activity. Aimed at knowing the quality impact produced after changes on the entity, a re-evaluation should be performed. If the planned improvement gain on quality after the implemented changes was not achieved, i.e., the operational business goal was not achieved considering the information need, then new change and re-evaluation cycles should be performed.

As commented in Section 4, the process pattern specification allows to indentify concrete subgoals which are useful for realizing the S.4.2 (“Per each selected strategy pattern, identify the concrete ME information need/business goals from the pattern process specification”) step. Particularly, the GoMEC_1QV pattern establishes three subgoals viz. (i) understand the current quality state of the entity -by performing the A1 to A4 activities; (ii) make changes on it -by performing the A5 and A6 activities; and (iii) understand the ulterior quality state (the improvement) after changes -by performing again the A3 and A4 activities. These subgoals were also illustrated in Fig. 3 for the Facebook case.

5.3 Another Evaluation Scenario for the Improving PurposeTable 4

Description: Considering that a given quality view depends on another quality view (see the ‘dependent view’ and ‘independent view’ roles specified in the quality view component of Fig. 1), the purpose is to improve the current state of a concrete entity belonging to a dependent quality view by applying evaluation-driven changes to other entity belonging to a related independent quality view. Hence, the business goal purpose at operational level is to understand and ultimately improve the current state of an entity, in a given context, for a set of characteristics and attributes for a dependent view's quality focus by applying evaluation-driven changes to other entity for an independent view's quality focus, through the systematic use of an improvement strategy driven by measurement, evaluation, analysis and change activities. The measurement activity is performed at least twice: First, by means of the quantification of attributes of the entity belonging to the dependent view; Second, by means of the quantification of attributes of the entity belonging to the independent view. The evaluation activity is performed at least twice as well. Evaluation interprets characteristics and attributes by means of indicators. The analysis is based on determining strengths and weaknesses of characteristics and attributes related to the dependent view. Those weakly benchmarked indicators of the dependent view permit to identify problems and derive related attributes for the entity of the independent view. For the independent view, the analysis proposes recommendations for change. When the changes were performed in the entity of the independent view, the dependent view's entity is therefore re-evaluated for analyzing the improvement gain.

Business Goal Purpose: Improve

Amount of Quality Views: Two. Note:Table 4 shows examples of quality views. Also, Fig. 8 shows ‘influences’ and ‘depends on’ relationships between typical quality views in software development and evaluation projects. The direction of the ‘influences’ relationship is from the independent quality view to the dependent quality view. Conversely, the direction of the ‘depends on’ relationship is from the dependent quality view to the independent quality view.

Table 4: Quality Views examples in Software Engineering's production lines  

Example of Evaluation Scenario:

* Business Goal statement: Improve the Quality in Use of the JIRA considering changes in the JIRA system in 4 months time frame

* Entity Category for the Dependent View: System in Use

* Entity Category for the Independent View: System

* Concrete Entities: JIRA in use, in the context of the ABC company, and JIRA web application

* Quality Focuses: Quality in Use and External Quality

* Characteristics of the Quality-in-use focus: Actual Usability (Subcharacteristics: Effectiveness, Efficiency, and Learnability in use). Note: The characteristics related to the independent quality view are derived from problems detected in the JIRA system-in-use evaluation. In the case study performed in [22], the derived characteristics were Usability and Information Quality for the External Quality focus.

* Quality Views: System-in-Use Quality View and System Quality View

Strategy Pattern to be instantiated: GoMEC_2QV (Goal-oriented Measurement, Evaluation and Change for Two Quality Views) [38]

Abridged Specification for the Generic Process:

Figure 9: Behavioral process perspective for the GoMEC_2QV pattern. Note that DQV means Dependent Quality View, and IQV means Independent Quality View  

Strategy to be applied: SIQinU (Strategy for understanding and Improving Quality in Use). This strategy was used in the exemplified JIRA evaluation scenario.

At first, the SIQinU strategy was conceived for those yellow-colored quality views highlighted in Fig. 8. However, the GoMEC_2QV strategy pattern can be applied for any couple of related quality views. In the sequel, for a better comprehension of Fig. 9, we comment the JIRA evaluation scenario which was performed in the context of a real company [22]. In this case, two quality views were involved namely the System-in-Use Quality View and the System Quality View. Starting with the Quality in Use focus -the dependent quality view-, the evaluated "entering a new defect" task and context of use were specified in the A1_DQV activity. Also, the Actual Usability characteristic, and attributes combined to Effectiveness, Efficiency, and Learnability-in-use subcharacteristics were established as nonfunctional requirements.

In the A2_DQV activity, metrics and indicators were designed. Then, in A3_DQV the ME were performed yielding measure and indicator values for characteristics and attributes. The results allowed us to understand the Quality in Use acceptability levels achieved and to perform a preliminary analysis. During the A4_DQV (“Analyze Results for the DQV”) activity we found that some Actual Usability attributes were not satisfied. This enabled us to derive External Quality attributes -the independent quality view- that can influence on Quality in Use. (Note that the System-in-Use Quality View ‘depends on’ the System Quality View, and in turn the latter ‘influences’ the former, as shown in Fig. 8). Specifically, in A1_IQV we specified subcharacteristics and attributes related to Usability and Information Quality for the External Quality focus. Then, the A2_IQV and A3_IQV activities were carried out.

In A4_IQV, we analyzed and proposed recommendations for changes in the JIRA system, regarding weakly benchmarked External Quality indicators. Once changes, using re-parameterization as change method were designed and performed (A5_IQV and A6_IQV activities), the new JIRA version was re-evaluated from the External Quality point of view. Finally, we re-evaluated the Quality in Use of the new JIRA in the same context performing the A3_DQV activity again. So the Quality in Use improvement gain was determined in the A4_DQV activity.

In summary, we were able to gauge how the quality of JIRA in use (the dependent quality view) was enhanced by improvements made in the JIRA web application (the independent quality view). It is important to remark that in the GoMEC_2QV process specification re-evaluation cycles are allowed for the two involved quality views.

5.4 Evaluation Scenario for the Monitoring and Controlling Purpose

Description: The business goal purpose at operational level is to monitor and control the state of an entity, in a given context, for a set of characteristics and attributes related to a quality focus, through the systematic use of a monitor and control strategy driven by measurement, evaluation, analysis and, if necessary, change activities. The measurement activity is performed by quantifying attributes by means of the selected metrics. The evaluation activity is performed by interpreting characteristics and attributes by means of indicators (also named performance and control variables). The analysis activity is based on a continuous ME and critical observation (monitor) to supervise (control) deviations with respect to the established acceptability levels of indicators which allow to identify preventive and corrective actions and/or to determine predictions and trends. The preventive and corrective actions imply changes in the entity and/or its context. The monitor and control of an entity permit analyzing if the detected performance problems were resolved or if new change actions are necessary.

Business Goal Purpose: Monitor and Control

Amount of Quality Views: One. Note: Table 4 shows examples of quality views

Example of Evaluation Scenario:

* Business Goal statement: Monitor and control that the required acceptability level of Security for the SIU Guaraní system is assured over time, in the context of the Engineering School at UNLPam

* Entity Category: System

* Concrete Entity: SIU Guaraní, in the context of the Engineering School at UNLPam

* Quality Focus: External Quality

* Characteristic: Security (Subcharacteristics: Confidentiality, Integrity, Authenticity, and Availability)

* Quality View: System Quality View

Strategy Pattern to be instantiated: GoMEM_1QV (Goal-oriented Measurement, Evaluation, Monitor and control for One Quality View)

Abridged Process Specification:

Figure 10: Behavioral process perspective for the GoMEM_1QV pattern 

Strategy to be applied: GOCAMEMC (Goal-Oriented Context-Aware Measurement, Evaluation, Monitor and Control). The use of this strategy was not published yet, but we are currently monitoring thru the last three years the evolution of the Facebook mobileapp's user interface Usability.

In this evaluation scenario, the A4 (“Analyze and Control Results”) activity differs to a some extent from those described in Fig. 6 and Fig. 7 for the GoME_1QV and GoMEC_1QV process specifications respectively. Particularly, in Fig. 10, the A4 activity is based on continuous observations, that is to say, a realization of monitoring and control tasks with somewhat high frequency, depending on the specific business/information need goal. Monitoring and control allow to supervise if the results of indicators (performance variables) are under control regarding the expected acceptability levels. The monitoring and control loop includes the A3 and A4 activities. That is, there is a continuous cycle of implementing the measurement and evaluation, and then analyzing and controlling again and again. Furthermore, preventive or corrective actions could be performed concurrently, whether deviations are detected in the A4 activity. If this is the case, changes should be performed in the A5 and A6 activities. Therefore, the follow up cycles of monitoring and control take into account the new entity.

In this scenario, different kind of analysis can be performed such as assessing actual performance with planned performance, critically observing or determining the status, predictive and corrective analysis, in addition to determine and assess trends, among others.

5.5 Evaluation Scenario for the Selection Purpose

Description: The business goal purpose at operational level is to select the best alternative among a set of competitive entities, in a given context, for a set of characteristics and attributes related to a quality focus, through the systematic use of a selection strategy driven by measurement, evaluation and analysis. The measurement activity is performed by quantifying attributes by means of the selected metrics. The evaluation activity is performed by interpreting characteristics and attributes by means of indicators. The analysis is based on determining strengths and weaknesses of the evaluated entities in a given moment, which allows to analyze and compare them with the aim of selecting and adopting the best alternative for the established quality focus.

Business Goal Purpose: Select an alternative

Amount of Quality Views: One. Note: Table 4 shows examples of quality views

Example of Evaluation Scenario:

* Business Goal statement: Select the best academic management system for the Engineering School at UNLPam, which ensures quality criteria such as Functional Suitability, Efficiency, Usability, and Security

* Entity Category: System

* Concrete Entities: SIU Guaraní, webSIA and UCASAL-SAG

* Quality Focus: External Quality

* Characteristics: Functional Suitability, Efficiency, Usability and Security

* Quality View: System Quality View

Strategy Pattern to be instantiated: GoMES_1QV (Goal-oriented Measurement, Evaluation and Selection for One Quality View)

Abridged Process Specification:

Figure 11: Behavioral process perspective for the GoMES_1QV pattern 

Strategy to be applied: GOCAMES (Goal-Oriented Context-Aware Measurement, Evaluation and Selection). The use of this strategy was not published yet.

Unlike other strategy patterns' process specifications described before, the first activity in GoMES_1QV is the A0 activity. This activity is named “Preselect Competitive Entities” in Fig. 11. It consists in filtering and determining the set of competitive entities to be evaluated. Preselection can be based for instance on expert judgment, or on more objective protocols and selection criteria. The A0 primary aim is to reduce the sample to those relevant and suitable competitive entities.

After the preselection is performed, the “Define non Functional Requirements” and “Design Measurement and Evaluation” activities follow up in the workflow. Note that A1 and A2 activities are the same as the ones previously commented for the other strategy patterns. Likewise, the A3 (“Implement Measurement and Evaluation”) activity is the same as before but performed in an iterative way. Hence A3 must be carried out per each competitive entity that have been preselected, producing as outcome measure and indicator values per each entity regarding the same evaluated nonfunctional requirements.

Consequently, the A4 (“Analyze Results and Select Alternative”) activity permits to compare the evaluation results for the competitive entities with the aim of selecting the best alternative. In this sense, the analysis is based on determining strengths and weaknesses of the evaluated entities, which allows to analyze and compare them with the aim of selecting and adopting the highest scored alternative for the established quality focus and requirements.

In a nutshell, the GOCAMES strategy fosters a systematic way to get data and information of competitive entities, which ultimately are the inputs for sound analysis. By means of this analysis, the information need gives support to the operational business goal whose purpose is "select alternative".

As a final remark, another evaluation scenario for the selection purpose which considers both a quality view and a cost view can be specified. In this new scenario, the requirements should be established regarding quality characteristics and attributes in addition to cost factors. Moreover, the analysis and selection model should consider specific quality-cost indicator relations.

5.6 Evaluation Scenario for the Comparing Purpose

Description: The business goal purpose at operational level is to compare characteristics and attributes of a set of representative entities, in a given context, aimed at incorporating (adopting) recommended strengths to a new entity or to an existing one, through the systematic use of a comparison strategy driven by measurement, evaluation, analysis and, when necessary, change. The measurement activity is performed by quantifying attributes by means of the selected metrics. The evaluation activity is performed by interpreting characteristics and attributes by means of indicators. The comparative analysis is based on determining strengths and weaknesses of the evaluated entities in a given moment, which allows recommending the detected strengths and adopting them in a new entity or in one that already exists.

Business Goal Purpose: Compare and Adopt

Amount of Quality Views: One. Note: Table 4 shows examples of quality views

Example of Evaluation Scenario:

* Business Goal statement: Compare a set of integrated ME strategies to adopt the best quality capabilities or characteristics in GOCAME

* Entity Category: Resource

* Concrete Entities: GOCAME and GQM+Strategies

* Quality Focus: Resource Quality

* Characteristic: Quality Capability (Subcharacteristics: Process Capability Quality, Methodology Capability Quality and Conceptual-Framework Capability Quality)

* Quality View: Resource Quality View

Strategy Pattern to be instantiated: GoMECom_1QV (Goal-oriented Measurement, Evaluation and Comparison for One Quality View)

Abridged Process Specification:

Figure 12: Behavioral process perspective for the GoMECom_1QV pattern 

Strategy to be applied: GOCAMECom (Goal-Oriented Context-Aware Measurement, Evaluation and Comparison). This strategy was used in [31] for the exemplified evaluation scenario.

Similarly to the GoMES_1QV process specification (described in the previous subsection), the GoMECom_1QV process starts with the preselection (A0) activity of representative entities to be compared, as seen in Fig. 12. Again, the preselection of representative entities can be based on expert judgment, on existing benchmarks or case studies, or on other agreed preselection criteria. It is important to note that one of the preselected entities could be the entity that we are interested in improving it.

As it occurs in GoMES_1QV, the A3 (“Implement Measurement and Evaluation”) activity must be carried out per each preselected representative entity, producing as outcome measure and indicator values regarding the same evaluated nonfunctional requirements.

The A4 (“Analyze Results and Recommend Strengths to be Adopted”) activity is to some extent similar to the one described for the purpose of selecting an alternative. But in this case, after comparing the evaluation results, the best ranked characteristics and attributes could be taken into account to be adopted in a new or existing entity. If the entity already exists -as it was the situation in the [31] case study-, the necessary changes for incorporating the strengths should therefore be designed and implemented. These activities are coded A5 and A6 in Fig. 12. Conversely, if at that moment the entity does not exist, then A5 and A6 activities are not performed. In this case, a development strategy should be chosen for designing and implementing a new entity taking into account the best capabilities or characteristics that have been detected.

Conclusion and Future Work

In this scientific article, we have presented a holistic quality multilevel and multipurpose evaluation approach which considers the linkage of information need goals with business goals at different organizational levels such as operational, tactical and strategic in addition to different evaluation goal purposes.

Summarizing the first contribution listed in the Introduction Section, we have enlarged the conceptual base of the Quality Evaluation Approach by adding concepts such as strategy pattern and organizational level types in the business goal and project components. We have discussed the enhanced conceptual base in Section 3, which is able to relate business and information need goals with ME information needs, entity categories, quality focuses, quality views as well as with projects, strategies and strategy patterns. It is worthwhile remarking that this conceptual base is structured in subontologies. Considering the ontology scope, the terms included in the goal and project components are the minimum and necessary ones for describing goals, projects, strategies and strategy patterns. Also, they are aimed at adding conceptual robustness to our approach in addition to the ability to support semantic processability, among other benefits.

Regarding the second contribution, we have also defined in Section 4 the step by step of the Quality Evaluation Approach which lists the necessary activities to establish goals and projects at different organizational levels. In addition, we have illustrated the stepwise applicability using an evaluation scenario for improving the Facebook mobile app. Although there are some relevant references for the business goal alignment, as those analyzed in the Related Work Section, the approach we have proposed formally establishes the use of integrated ME/MEC strategies for helping to achieve both business and information need goals, mainly at operational level. Besides, the Quality Evaluation Approach specifies ME information need goals as well, which are linked to information need and business goals. Moreover, this approach fosters the use of strategy patterns as a reusable solution for instantiating the suitable evaluation strategy considering the project's goal statement and the strategy pattern intent.

Lastly, taking into account the third contribution, we have documented in Section 5 a set of six different evaluation scenarios for business/information need goals at operational level supported by strategy patterns and strategies. Specifically, we have identified different ME/MEC strategy patterns and strategies that realize purposes aimed at understanding, improving, monitoring and controlling, comparing and selecting entities regarding also the amount of quality views. Each evaluation scenario was illustrated with a concrete example. But the reader can envision that many examples can be applied for the same purpose considering the same or different quality views. Ultimately, the evaluation scenarios documented for business/information need goals at operational level, can be reused not only for the Software Engineering discipline but also to other disciplines such as Health Informatics, Biotechnology, Ecology, amongst many others. Finally, the established business goal purposes at operational level can be linked and provide feedback to business and information need goals at higher organizational levels.

Considering the semantic processability, an ongoing work is the development of a strategy pattern recommender system as a practical use of subontologies. This recommender system can be useful when an organization establishes ME/MEC projects. Hence, considering the project's goal statement and the strategy pattern intent, the recommender system will suggest the suitable strategy pattern that fits better to a given project. Consequently, the particular strategy will be easier to be customized.