Conditions for applying the method of expert assessments. expert method

Introduction …………………………………………………………………………..3

Chapter 1 Essence, methods and process of expert assessments …………………… 5

1.1 Essence of expert assessments ……………………………………………………5

1.2 The role of experts in management ………………………………………………..9

1.3 Peer Review Process …………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………….

1.4 Methods of expert assessments ………………………………………………..18

1.4.1 SWOT analysis ……………………………………………………………...18

1.4.2 SMART method …………………………………………………………….20

1.4.3 Method of ranking and evaluation ……………………………………..21

1.4.4 Method of direct assessment …………………………………22

1.5 Assessing the consensus of experts ………………………………………….23

Chapter 2 Methods of expert assessments on the example of UAZ OJSC ...…………….24

Conclusion …………………………………………………………………………32

List of used sources and literature …………………………..33

Introduction

In the study of management, the method of expert assessments is widely used. This is due to the complexity of many problems, their origin from the "human factor", the lack of reliable experimental or normative tools.

It is undeniable that in order to make informed decisions, it is necessary to rely on the experience, knowledge and intuition of specialists. After the Second World War, within the framework of the theory of management (management), an independent discipline began to develop - expert assessments.

Methods of expert assessments are methods for organizing work with specialist experts and processing expert opinions expressed in quantitative and / or qualitative form in order to prepare information for decision-making by decision makers.

Many works have been devoted to the study of the possibilities and features of the application of expert assessments. They consider forms of expert survey ( different types questionnaires, interviews), assessment approaches (ranking, normalization, different kinds ordering, etc.), methods of processing the results of the survey, requirements for experts and the formation of expert groups, issues of training experts, assessing their competence (when processing estimates, the coefficients of expert competence, the reliability of their opinions are introduced and taken into account), methods of organizing expert surveys. The choice of forms and methods for conducting expert surveys, approaches to processing survey results, etc. depends on the specific task and conditions of the examination.

Expert methods are now used in situations where the choice, justification and evaluation of the consequences of decisions cannot be performed on the basis of accurate calculations. Such situations often arise during the development contemporary problems management social production and especially in forecasting and long-term planning. In recent years, expert assessments have been widely used in socio-political and scientific-technical forecasting, in the planning of the national economy, industries, associations, in the development of major scientific, technical, economic and social programs, in solving certain management problems.

Chapter 1 Essence, methods and process of expert assessments

1.1 The essence of expert assessments

The possibility of using expert assessments, the justification of their objectivity is usually based on the fact that an unknown characteristic of the phenomenon under study is interpreted as a random variable, the reflection of the distribution law of which is an individual assessment of a specialist expert on the reliability and significance of an event. It is assumed that the true value of the characteristic under study is within the range of estimates received from the group of experts, and that the generalized collective opinion is reliable.

However, some theoretical studies question this assumption. For example, it is proposed to divide the problems for which expert assessments are used into two classes. To lanemy class include problems that are sufficiently well provided with information and for which the principle of a “good measurer” can be used, considering the expert as the custodian of a large amount of information, and the group opinion of experts is close to the true one. Co. second class include problems in respect of which knowledge is not enough to be sure of the validity of these assumptions; experts cannot be considered as “good measurers”, and it is necessary to carefully approach the processing of the results of the examination, since in this case the opinion of one (single) expert, who pays more attention to the study of a little-studied problem, may turn out to be the most significant, and during formal processing it will be lost. In this regard, qualitative processing of results should be mainly applied to problems of the second class. The use of averaging methods (valid for "good meters") in this case can lead to significant errors.

The tasks of collective decision-making on the formation of goals, the improvement of methods and forms of management can usually be attributed to the first class. However, when developing forecasts and long-term plans, it is advisable to identify “rare” opinions and subject them to a more thorough analysis.

Another problem that needs to be borne in mind when conducting a system analysis is the following: even in the case of solving problems related to the first class, one should not forget that expert assessments carry not only narrowly subjective features inherent in individual experts, but also collectively. -subjective features that do not disappear when processing the results of the survey (and can even be enhanced when using the Delphi procedure). In other words, expert assessments should be viewed as some kind of “public point of view”, depending on the level of scientific and technical knowledge of the society regarding the subject of research, which can change as the system and our ideas about it develop. Therefore, an expert survey is not a one-time procedure. This way of obtaining information about a complex problem characterized by a high degree of uncertainty should become a kind of "mechanism" in a complex system, i.e. it is necessary to create a regular system of work with experts.

Attention should also be paid to the fact that the use of the classical frequency approach to assessing probability when organizing expert surveys can be difficult, and sometimes impossible (due to the impossibility of proving the legitimacy of using a representative sample). Therefore, at present, studies are underway on the nature of the probability of expert assessment, based on the theory, fuzzy sets of Zadeh, on the idea of ​​expert assessment as a degree of confirmation of a hypothesis or as a probability of achieving a goal. One of the varieties of the expert method is the method of studying the strengths and weaknesses of the organization, the opportunities and threats to its activities - the method of SWOT analysis.

The collection of expert information depends on the choice of the method of expert assessments. Usually, to collect expert information, special documents are compiled, for example, questionnaires approved by the relevant managers and then sent to the experts.

Processing of expert information is carried out using the chosen method, usually with the use of computer technology. The data obtained as a result of processing is analyzed and used to solve the problems of analysis and synthesis of control systems.

Expert assessments are used for analysis, diagnosis of the state, subsequent prediction of development options:

1) objects, the development of which is either completely or partially not amenable to subject description or mathematical formalization;

2) in the absence of sufficiently representative and reliable statistics on the characteristics of the object;

3) in conditions of great uncertainty in the environment for the functioning of the object, the market environment;

4) in medium- and long-term forecasting of new markets, objects of new industries that are strongly influenced by discoveries in the fundamental sciences (for example, the microbiological industry, quantum electronics, nuclear engineering);

5) in cases where either the time or the funds allocated for forecasting and decision-making do not allow to investigate the problem using formal models;

6) there are no necessary technical means of modeling, for example, computer technology with the appropriate characteristics;

7) in extreme situations.

The tasks solved in the process of expert assessments of control systems can be divided into two groups:

1) tasks of synthesis of new control systems and their evaluation;

2) tasks of analysis (measurement) of existing management systems according to selected indicators and performance criteria.

The tasks of the first group include: formation of the image of the system being created; forecasting technical and economic indicators of the stages of its life cycle; substantiation of the main directions of the reorganization of the social management system; selection of optimal or satisfactory methods of action and outcomes using the created control system, etc.

Some of the expert information obtained in the course of solving these problems is of a qualitative nature and is formed in the form of complex judgments in a descriptive form. However, the tasks of synthesis solved with the help of expert assessments can be quantitative in nature, and their solution will be associated with the justification of numerous parameters (characteristics) of the system being created.

The tasks of the second group include all the tasks of evaluating existing or created variants of control systems using specified indicators and performance criteria. Examples of such tasks are: determining the structural, functional or informational characteristics of the system; evaluation of its effectiveness in the course of performing various operations; determining the expediency of further operation of technical means of control and communication, etc. A significant part of the expert information used in solving such problems is of a quantitative nature or has the form of elementary judgments and is processed using various statistical methods.

MINISTRY OF EDUCATION

RUSSIAN FEDERATION

ALTAI STATE UNIVERSITY

Faculty of Economics

Department of Crisis Management, Business Assessment and Innovation

METHOD OF EXPERT ASSESSMENTS

(course work)

Completed by a student

3 courses, group 277

Strekalova S.B.

Job protected

Barnaul - 1999

Introduction 3

Chapter 1. EXPERTISE IN MANAGEMENT 5

1.1. Role of experts in management 5

1.2. Expert judgment method 7

1.3. Organization of peer review 9

1.4. Selection of experts 9

1.5. Expert Survey 10

Chapter 2. FORMALIZATION OF INFORMATION

AND COMPARISON SCALES 12

Chapter 3. PROCESSING OF EXPERT ASSESSMENTS 16

3.1. Machining tasks 16

3.2. Group evaluation of objects 17

3.3. Assessing the consensus of expert opinions 22

3.4. Handling Pairwise Object Comparisons 25

3.5. Determining the relationship of rankings 27

Conclusion 31

References 32

INTRODUCTION

The modern economy makes new, higher demands on management. Questions of improving management methods are now becoming very important, since it is in this sphere that there are even greater reserves for increasing the efficiency of the national economy.

An essential factor in improving the scientific level of management is the use of mathematical methods and models in the preparation of solutions. However, a complete mathematical formalization of technical and economic problems is often not feasible due to their qualitative novelty and complexity. In this regard, expert methods are increasingly used, which are understood as a set of logical and mathematical-statistical methods and procedures aimed at obtaining from specialists the information necessary for the preparation and selection of rational decisions.

Expert methods are now used in situations where the choice, justification and evaluation of the consequences of decisions cannot be performed on the basis of accurate calculations. Such situations often arise in the development of modern problems of managing social production and, especially, in forecasting and long-term planning. In recent years, expert assessments have been widely used in socio-political and scientific-technical forecasting, in the planning of the national economy, industries, associations, in the development of major scientific, technical, economic and social programs, in solving certain management problems.

In the course of the development of social production, not only the complexity of management increases, but also the requirements for the quality of decisions made. In order to increase the validity of decisions and take into account the numerous factors influencing their results, a comprehensive analysis is needed, based both on calculations and on reasoned judgments of managers and specialists familiar with the state of affairs and development prospects in various fields practical activities. The use of expert methods ensures the active and purposeful participation of specialists at all stages of decision-making, which can significantly improve their quality and efficiency.

The purpose of our work is to study the method of expert assessments - one of the most important stages in making competent management decisions.

1) study of the role of expertise in management;

2) consideration of the procedure for organizing expert assessment;

3) study of the types of scales and the procedure for their use;

4) detailed consideration of the final stage of expert evaluation - processing of expert evaluations.

The abstract consists of an introduction, three chapters, a conclusion and a list of references.

The first chapter deals with the issue of the need for expertise in management, considers the method of expert assessments, the stages of organization of expert assessment.

The second chapter is devoted to the comparison scales, the characteristics of each type of scales and the order of their use in the formalization of information are given.

The third chapter deals with the processing of expert assessments: processing tasks, group assessment of objects, assessment of the consistency of expert opinions, processing of pairwise comparisons of objects and determining the relationship of rankings.

Since the purpose of this work is to consider expert evaluation in a theoretical aspect, practical application is not considered.

In conclusion, the role of the method of expert assessments in making managerial decisions is considered.

Chapter 1. EXPERTISE IN MANAGEMENT

1.1. The role of experts in management

Modern society is developing under the ever-increasing impact of the scientific and technological revolution, which is causing fundamental transformations in production, profound changes in the structure and economy of the national economy. The ongoing scientific and technological revolution in its influence goes far beyond the sphere material production, capturing all aspects of the life of society, predetermining the majority of decisions aimed at its rational economic and social development.

The history of the development of science, technology and production shows that simultaneously with the successive replacement of human functions by machine functions, its role in the field of management increases. The continuous growth in the volume of expenditures on the development of science, on the creation of new technology and the improvement of production significantly increases the significance of decisions taken at all levels of economic management. The future of science. Engineering and economics largely depends on the quality and timeliness of these decisions, and the objective trends of scientific and technological progress can accelerate or slow down under their influence.

Optimization methods based on the use of formal, most often mathematical models, which save time and money in solving many practical problems, are now acquiring particular importance in management. Modeling helps bring the complex and sometimes uncertain factors associated with a decision problem into a coherent scheme, determine what data is needed to evaluate and select alternatives.

In the management process, there is a natural desire to find a solution that is objectively the best of all possible. Mathematical programming is now widely used as an optimization tool. Successes in the application of mathematical programming to solving various kinds of economic, scientific, technical and military problems have given rise to methodological views, according to which a cardinal solution to control problems is possible only when all its aspects are displayed in a system of interconnected mathematical models.

However, the formalization of technical, economic and managerial decisions is complicated by a number of features of the current stage of scientific and technological progress. The life of society is so complex that it is difficult to count on the emergence of models that would fully reflect the nature and quantitative relationships of socio-economic processes. Real reality is always more complex than the most subtle mathematical models, and its development often outstrips formal knowledge. Management tasks require the participation of people as an integral element of the solution. And, finally, the management process itself always involves an orientation not only to numerical data, but also to ordinary common sense. The use of mathematical programming and computer technology makes it possible to make decisions based on more complete and reliable information. But it is also undoubted that under any conditions, choosing a rational solution requires something more than a good mathematical model.

When making decisions, we generally assume that the information used to support them is accurate and reliable. But for many economic and scientific-technical problems, which are qualitatively new and non-repetitive in nature, this assumption is either obviously not realized, or it cannot be proved at the time of making a decision.

The availability of information and the correctness of its use largely determine the optimality of the chosen solution. In addition to data consisting of numerical statistics, information includes other quantities that are not directly measurable, such as assumptions about possible solutions and their results. Practice shows that the main difficulties that arise in the search and selection of business solutions are primarily due to the insufficiently high quality and incompleteness of the information available.

The main difficulties associated with information that arise in the development of complex decisions can be divided into the following groups.

First, the initial statistical information is often not reliable enough.

Secondly, some of the information is of a qualitative nature and cannot be quantified. Thus, it is impossible to accurately calculate the degree of influence of social and political factors on the implementation of plans, to assess the economic effect of future inventions, etc. But, since these factors and phenomena have a significant impact on the results of decisions, they cannot be ignored.

Thirdly, in the process of preparing decisions, situations often arise when, in principle, the necessary information can be obtained, but at the time of making a decision it is not available, since this is associated with a large investment of time or money.

Fourth, there is a large group of factors that may affect the implementation of the decision in the future, but they cannot be accurately predicted.

Fifth, one of the most significant difficulties in choosing solutions is that any scientific or technical idea contains the potential for various schemes for its implementation, and any economic action can lead to multiple outcomes. The problem of choosing the best solution may also arise because there are usually resource constraints, and therefore, the adoption of one option is always associated with the rejection of other solutions.

Sixth, when choosing the best solution, we often encounter the ambiguity of the generalized criterion, on the basis of which it is possible to compare possible outcomes. The ambiguity, multidimensionality and qualitative difference of indicators are a serious obstacle to obtaining a generalized assessment of the relative effectiveness, importance, value or usefulness of each of the possible solutions.

In this regard, one of the main features of solving complex problems is that the application of calculations here is always intertwined with the use of the judgments of managers, scientists, and specialists. These judgments make it possible to at least partially compensate for the lack of information, to make fuller use of individual and collective experience, and to take into account the assumptions of specialists about the future states of objects. The pattern of development of science and technology is that new knowledge, scientific and technical information accumulate over a long period of time. Quite often this accumulation goes on in a latent form in the minds of scientists and developers. They, like no one else, are able to assess the prospects of the area in which they work, and to anticipate the characteristics of those systems in the creation of which they are directly involved.

Experience shows that the use of unsystematized judgments of individual specialists is not effective enough in solving many complex scientific and technical problems due to the variety of relationships between the main elements of such problems and the impossibility of covering them all. When using traditional decision preparation procedures, it is often not possible to consider a wide range of factors, to take into account the entire range of alternative ways of solving problems.

All this forces one to resort to staffing groups of specialists representing various fields of knowledge as experts. The use of group expertise allows not only to consider many aspects and factors, but also to combine different approaches, with the help of which the manager finds the best solution.

1.2. Method of expert assessments

The essence of the method of expert assessments is that experts conduct an intuitive-logical analysis of the problem with a quantitative assessment of judgments and formal processing of the results. The generalized opinion of experts obtained as a result of processing is accepted as a solution to the problem. The complex use of intuition (unconscious thinking), logical thinking and quantitative assessments with their formal processing makes it possible to obtain an effective solution to the problem.

When performing their role in the management process, experts perform two main functions: they form objects (alternative situations, goals, decisions, etc.) and measure their characteristics (probabilities of events occurring, goal significance coefficients, decision preferences, etc.) . The formation of objects is carried out by experts on the basis of logical thinking and intuition. In this case, the knowledge and experience of the expert play an important role. Measuring the characteristics of objects requires experts to know the theory of measurements.

The characteristic features of the method of expert assessments as a scientific tool for solving complex non-formalizable problems are, firstly, the scientifically based organization of all stages of the examination, ensuring the greatest efficiency of work at each stage, and, secondly, the use of quantitative methods both in organizing the examination and and in evaluating expert judgment and formal group processing of the results. These two features distinguish the method of expert assessments from the usual long-known expertise, widely used in various fields of human activity.

Expert collective assessments were widely used on a national scale to solve complex problems of managing the national economy already in the early years. Soviet power. In 1918, the Council of Experts was established under the Supreme Council of the National Economy, whose task was to solve the most difficult problems of reorganizing the country's national economy. In drawing up five-year plans for the development of the national economy of the country, expert assessments of a wide range of specialists were systematically used.

At present, in our country and abroad, the method of expert assessments is widely used to solve important problems of a different nature. In various industries, associations and enterprises, there are permanent or temporary expert commissions that form solutions to various complex non-formalizable problems.

The whole set of poorly formalized problems can conditionally be divided into two classes. The first class includes problems for which there is sufficient information potential to successfully solve these problems. The main difficulties in solving first-class problems in peer review are in realizing the existing information potential by selecting experts, building rational survey procedures and applying optimal methods for processing its results. At the same time, the methods of interrogation and processing are based on the use of the principle of a “good” meter. This principle means that the following hypotheses are fulfilled:

1) an expert is a repository of a large amount of rationally processed information, and therefore it can be considered as a qualitative source of information;

2) the group opinion of experts is close to the true solution of the problem.

If these hypotheses are correct, then the results of measurement theory and mathematical statistics can be used to construct polling procedures and processing algorithms.

The second class includes problems in relation to which the information potential of knowledge is insufficient to be sure of the validity of these hypotheses. When solving problems from this class of experts can no longer be considered as "good measurers". Therefore, it is necessary to be very careful when processing the results of the examination. The use of averaging methods that are valid for "good meters" in this case can lead to large errors. For example, the opinion of one expert, which is very different from the opinions of other experts, may turn out to be correct. In this regard, for problems of the second class, qualitative processing should mainly be applied.

The scope of the method of expert assessments is very wide. We list the typical tasks solved by the method of expert assessments:

1) compiling a list of possible events in various areas for a certain period of time;

2) determination of the most probable time intervals for the completion of a set of events;

3) definition of goals and objectives of management with ordering them in order of importance;

4) identification of alternative (options for solving the problem with an assessment of their preferences;

5) alternative distribution of resources for solving problems with an assessment of their preference;

6) alternatives making decisions in a certain situation with an assessment of their preference.

To solve the above typical tasks Currently, various varieties of the method of expert assessments are used. The main types include: questioning and interviewing; brainstorm; discussion; meeting; operational game; scenario.

Each of these types of expert evaluation has its own advantages and disadvantages, which determine the rational area of ​​application. In many cases, the combined application of several types of expertise gives the greatest effect.

Questioning and the scenario assume individual work of the expert. Interviewing can be carried out both individually and with a group of experts. Other types of expertise involve the collective participation of experts in the work. Regardless of the individual or group participation of experts in the work, it is advisable to obtain information from many experts. This makes it possible to obtain more reliable results based on data processing, as well as new information about the dependence of phenomena, events, facts, expert judgments, which is not explicitly contained in the statements of experts.

When using the method of expert assessments, there are some problems. The main ones are: selection of experts, conducting a survey of experts, processing the results of the survey, organizing examination procedures.

1.3. Organization of expert assessment

The first stage in the organization of work on the application of expert assessment is the preparation and publication of a guidance document, which formulates the purpose of the work and the main provisions for its implementation. This document should reflect the following questions: statement of the problem-experiment; the purpose of the experiment; substantiation of the need for the experiment; turnaround time; tasks and composition of the management group; duties and rights of the group; financial and material support of work.

To prepare this document, as well as to manage the entire work, an examination leader is appointed. It is entrusted with the formation of the management group and responsibility for organizing its work.

After the formation, the management group carries out work on the selection of an expert group in approximately the following sequence: clarification of the problem to be solved; determination of the range of areas of activity related to the problem; determination of the proportion of experts in each field of activity; determination of the number of experts in the group; drawing up a preliminary list of experts, taking into account their location; analysis of the qualities of experts and specification of the list of experts in the group; obtaining the consent of experts to participate in the work; compilation of the final list of the expert group.

In parallel with the process of forming a group of experts, the management group is developing the organization and methodology for conducting a survey of experts. In this case, the following issues are resolved: the place and time of the survey; the number and objectives of survey rounds; the form of the survey; the procedure for recording and collecting survey results; composition of the required documents.

The next step in the work of the management group is to determine the organization and methodology for processing survey data. At this stage, it is necessary to determine the tasks and terms of processing, procedures and algorithms for processing, forces and means for processing.

In the process of directly conducting a survey of experts and processing its results, the management group carries out a set of works in accordance with the developed plan, adjusting it as necessary in terms of content, timing and provision of resources.

The last stage of work for the management group is the registration of the results of the work. At this stage, the results of expert evaluation are analyzed; compilation of a report; discussion and approval of the results; presenting the results of the work for approval; familiarization with the results of the examination of organizations and individuals.

1.4. Selection of experts

To implement the expert evaluation procedure, it is necessary to form a group of experts. A common requirement in the formation of a group of experts is an effective solution to the problem of examination. The effectiveness of solving the problem is determined by the characteristics of the reliability of the examination and the cost of it.

The reliability of expert evaluation can be determined only on the basis of a practical solution to the problem and analysis of its results. The use of experts is precisely due to the fact that there are no other ways to obtain information. Therefore, the assessment of the reliability of the examination can be carried out, as a rule, only on the basis of a posteriori (post-experimental) data. If the examination is carried out systematically with approximately the same composition of experts, then it becomes possible to accumulate statistical data on the reliability of the work of a group of experts and obtain a stable numerical assessment of reliability. This estimate can be used as a priori data on the reliability of the panel of experts for subsequent examinations.

The reliability of group expert evaluation depends on the total number of experts in the group, the proportion of different specialists in the group, and on the characteristics of the experts.

Determining the nature of the dependence of reliability on the listed factors is another problem in the selection of experts.

A difficult problem in the selection procedure is the formation of a system of expert characteristics that significantly affect the course and results of the examination. These characteristics should describe the specific characteristics of the specialist and the possible relationships between people that affect the expertise. An important requirement for the characteristics of an expert is the measurability of these characteristics.

Another problem is the organization of the procedure for selecting experts, i.e. determination of a clear sequence of work performed in the process of selecting experts and the necessary resources for their implementation.

The maximum number of experts in the group is checked against the limitation on financial resources. Having determined the relationship between reliability, the number of experts and the cost of payment, the management team presents this information to management and formulates possible alternative solutions. Such alternatives can be either reducing the reliability of the results of expert evaluation to a level that ensures the fulfillment of the limitation on the cost of paying experts, or maintaining the original requirement for the reliability of the examination and increasing the cost of paying experts.

The next step in the selection of experts is the preparation of a preliminary list of experts. When compiling this list, an analysis of the qualities of experts is carried out. In addition to taking into account the qualities of experts, their location and the possibility of participation of selected specialists in the examination are determined. When evaluating qualities, the opinion of people who know candidates for experts well is taken into account.

After drawing up the list of experts, letters are sent to them with an invitation to participate in the examination. The letters explain the purpose of the examination, its timing, the procedure for conducting it, the amount of work and the terms of remuneration. Questionnaires of expert data and self-assessment of competence are attached to the letters. After receiving the experts' responses, the management group draws up the final list of the expert group.

After the list is compiled and approved, a message is sent to the experts about their inclusion in the expert group. If expert evaluation is carried out by the questionnaire method, then simultaneously with the notification of inclusion in the expert group, all experts are sent a questionnaire with the necessary instructions for filling them out. By informing the experts about their inclusion in the examination, the work on the selection of experts ends.

1.5. Expert Survey

The survey is the main stage of joint work of the management group and experts. The main content of the survey is:

Statement of the problem and presentation of questions to experts;

Information support for the work of experts;

Development by experts of judgments, estimates, proposals;

Collecting the results of the work of experts.

There are three types of tasks that are solved in the survey process:

Qualitative or quantitative assessment of given objects;

Construction of new facilities;

Construction and evaluation of new facilities.

In collective expertise, the following main types of survey are used: discussion, questioning and interviewing, the method of collective generation of ideas, or brainstorming.

The survey can be conducted with or without feedback. When questioning with feedback, experts are surveyed in several stages, with some of the results of the survey at the previous stage being brought to the attention of the experts, including the assessments of individual experts and their arguments.

The main thing in the organization of the survey is to provide maximum information and maximum creative activity, independence of the expert. It is necessary to strive to bring to each expert, if possible, all the information related to the analyzed phenomenon, which is available to both experts and the organizers of the survey, without depriving the expert of creative independence and activity at the same time.

However, the possibilities of an information processing expert are limited. As a result, an expert can make a decision without using all the information at his disposal. In addition, new information is perceived by a person with a certain internal resistance and does not immediately affect the already established subjective assessments. The attitude to new information is more favorable, and the perception and use of it is more complete if it is presented in an intelligible, vivid and compact form.

Of these psychological characteristics there is a need to provide experts with opportunities to capture incoming information by keeping records, using technical means, as well as the need to pre-process information and present it to experts in the most perceptible form.

It is necessary to emphasize the inconsistency of the importance of the exchange of information by experts, since the receipt of such information is fraught with the danger of losing creative independence in building a model of an object by an expert. The resolution of this contradiction is completely impossible, and at each examination its organizers must find a reasonable compromise, first of all, by choosing the type of survey, the form and degree of communication between experts.

Each type of survey has its advantages and disadvantages in building the exchange of information between experts and in organizing their independent creativity. The choice of one or another type of survey is determined by many factors, of which the main ones are:

The purpose and objectives of the examination;

The essence and complexity of the analyzed problem;

Completeness and reliability of the initial information;

The required volume and reliability of the information obtained as a result of the survey;

The time allotted for the survey and examination in general;

Permissible cost of the survey, and examination in general;

Number of experts and members of the management group, their characteristics.

Questioning is the most effective and most common type of survey, because it allows the best way to combine the information support of experts with their independent creativity.

Chapter 2. FORMALIZATION OF INFORMATION AND SCALE OF COMPARISONS

The rational use of information received from experts is possible provided that it is formed into a form convenient for further analysis aimed at preparing and making decisions.

The possibilities of formalizing information depend on the specific features of the object under study, the reliability and completeness of the available data, and the level of decision making. The form of presentation of expert data also depends on the accepted criterion, the choice of which, in turn, is significantly influenced by the specifics of the problem under study.

Formalization of information received from experts should be aimed at preparing a solution to such technical, economic and economic problems that cannot be fully described mathematically, since they are “weakly structured”, i.e. contain uncertainties associated not only with the measurement, but also with the very nature of the goals under study, the means to achieve them and external conditions.

When analyzing prospects, it is necessary not only to present in the form of indirect estimates part of the information that cannot be quantified, and not only to express with the help of such estimates quantifiable information about which there are no sufficiently reliable data at the time of preparation of the decision. The most important thing is to formalize this information in such a way as to help the decision maker to choose from a set of actions one or more that are most preferable in relation to some criterion.

If an expert is able to compare and evaluate possible options for action, assigning a certain number to each of them, then he has a certain system of preferences. Depending on the scale on which these preferences can be set, expert assessments contain more or less information and have a different ability to formalize.

Investigated objects or phenomena can be identified or distinguished on the basis of signs or factors. A factor is a set of at least two elements representing different levels of some of the quantities to be considered. The level of some factors can be expressed quantitatively (in rubles, percent, kilograms, etc.) - such factors are called quantitative. The level of others cannot be expressed with a number, they are called qualitative.

Factors are conditionally divided into discrete and continuous. Discrete are factors with a certain, usually small, number of levels. Factors whose levels are considered as forming a continuous set are called continuous. Depending on the goals and capabilities of the analysis, the same factors can be treated either as discrete or as continuous.

Let's consider the main logical axioms that are used in expert methods in formalizing information using various scales.

Using nominal scales the objects under study can be identified and distinguished on the basis of three axioms of identification:

1) i either there j, or there is not j ;

2) if i there is j, then j there is i ;

3) if i there is j and j there is k, then i there is k .

Factors in this case act as associative indicators that have information that can be formalized in the form of binary estimates of two levels: 1 (identical) or 0 (different).

In cases where the objects under study can be placed in a certain sequence as a result of comparison, taking into account any significant factor (factors), ordinal scales, allowing to establish equivalence or dominance.

Suppose you need to arrange in a certain sequence n objects according to some factor (criterion). We represent this ordering as a matrix where i, j = 1,2,…, n .

Quantities establish relationships between objects and can be defined as follows:

Let us establish the basic axioms necessary for the observance of the ordering conditions. ratio meaning that i preferable j, must be asymmetric, i.e., if then and transitive, i.e., if then

ratio meaning that i and j are equivalent is called an equivalence relation. This ratio should be

reflexive, i.e.

symmetrical, i.e., if then

transitive, i.e., if and then

In addition, these two relations must be compatible, i.e., if and then and also, if and then

And finally, the ordering must be connected, i.e. for any i and j or or or

The use of ordinal scales makes it possible to distinguish objects even in cases where the factor (criterion) is not explicitly specified, i.e. when we do not know the sign of comparison, but we can partially or completely order objects based on the system of preferences that the expert has.

Any set A will be called ordered if for any two of its elements X and Y found that either X preceded Y, or Y preceded X. Sometimes it is not possible to establish strict precedence for all elements of a set, but it is possible to produce a "group" ordering when subsets of equivalent elements are ordered. Next, we can pose the problem of comparing and ordering these subsets.

The use of ordinal scales makes it possible to perform transformations of estimates received from experts that correspond to all monotonically increasing functions. So, for example, positive estimates can either be replaced by their squares, or logarithms, or any other monotonically increasing function.

To formalize the estimates received from experts, one often uses interval scales. When such scales are used for these purposes, almost all the usual statistical measures can be taken. The exceptions are those measures that require knowledge of the "true" zero point of the scale, which is introduced conditionally here.

Interval scales suggest the possibility of transforming scores obtained on one scale into scores on another scale using the equation

Differences between values ​​on the interval scale become measures on the ratio scale, i.e. on the usual numerical scale, because as a result of subtraction, you can get rid of the constant term b .

In a number of cases, when formalizing expert assessments, the property of additivity is used, which is inherent only in the scale of relations. The presence of additivity is expressed by the following axioms:

1) if j = a and i> 0, then i + j > a ;

2) i + j = j + i ;

3) if i = a and j = b, then i + j = a + b ;

4) (i + j) + k = i + (j + k).

A common situation in which a decision needs to be made considering additivity is when there are several (at least two) qualitative factors. If there are several factors that characterize specific objects, there are many real properties and types of object relationships.

For example, the factors (indicators) that characterize the effectiveness of the creation and implementation of new technology, according to their objective content, can be divided into technical, economic and social. On the other hand, these factors can be grouped according to their role in the process of creating and implementing new technology, highlighting, for example, indicators that characterize costs, quality, economic efficiency, etc.

Depending on the nature and purpose of the problem under study, the factors by which objects differ can be quantitatively comparable or incomparable among themselves, partially comparable (i.e. not any with any, but only some of them), ordered by their degree of importance, etc. .d. The incommensurability of various factors is due not only to the need to use different units of measurement, but also to the fact that each factor, expressing a certain property, is at the same time an assessment of the attitude towards this property on the part of the decision maker.

In the practice of management at all its levels, situations often arise when it is necessary to make a decision taking into account many factors. The question of which factors should be considered the most important depends on the qualitative features of the object of the decision and the goals that this decision must meet.

For example, when considering several options for a plan or options for organizational and technical measures, factors of time, costs, technical and social results, economic efficiency, etc. should be taken into account. Usually, they try to lead all the variety of factors to an unambiguous complex assessment, and the most convenient and common such assessment is monetary.

However, since the consequences of any decision, especially decisions related to scientific and technological progress, go beyond cost indicators, indicators are needed that characterize the significance, usefulness of a particular factor (or their complex). Such integrated meters are widely used in assessing the quality of products, the technical and economic level of production, in assessing the performance of scientific organizations, and in a number of other tasks. Although the issue of creating a sufficiently substantiated formalized system of such meters is still far from a final decision, we can point out some common features, providing an approach to the formalization of this process and to the use of one or another logical and mathematical apparatus.

In the case when all factors are given on a nominal scale, i.e. some attribute a and an initial set of elements M are given on this scale, the goal is to choose a subset of elements M(a) that have this attribute. In such cases, the elements, or rather their properties, are compared with a sign - a standard, and the result - a partition of the set - can be considered as an ordering on a two-element scale, according to which each of the elements is assigned a score equal to either zero or one.

In the case when the factors are given on an ordinal scale or on several ordinal scales, the goal is to order the elements of the original set, to identify, with the help of experts, the hidden ordering that, by assumption, is inherent in this set. Necessary condition solution to this problem is the assumption of transitivity. The more completely the elements are ordered, the easier it is to apply logical-mathematical and combinatorial methods to solving such problems.

Depending on the nature or importance of a particular factor, various scales can be used at the stage of preparation and decision-making. Factors such as costs, profits, time, can be evaluated on an ordinal or interval scale (in rubles, days or conventional units). To evaluate such factors as the payback period or the comparative effectiveness of options, an interval scale can be used; qualitative or social factors can be assessed on ordinal or nominal scales.

Chapter 3. PROCESSING OF EXPERT ASSESSMENTS

3.1. Processing tasks

After conducting a survey of a group of experts, the results are processed. The initial information for processing is the numerical data expressing the preferences of the experts and the substantive justification for these preferences. The purpose of processing is to obtain generalized data and new information contained in a hidden form in expert assessments. Based on the processing results, a solution to the problem is formed.

The presence of both numerical data and meaningful statements of experts leads to the need to apply qualitative and quantitative methods for processing the results of group expert evaluation. The share of these methods essentially depends on the class of problems solved by expert evaluation.

The whole set of problems can be divided into two classes. The first class includes problems for the solution of which there is a sufficient level of knowledge and experience, that is, there is the necessary information potential. When solving problems belonging to this class, experts are considered as good average measurers. The term "good on average" refers to the possibility of obtaining measurement results that are close to true. For many experts, their judgments cluster around the true value. It follows that for processing the results of group expert evaluation of problems of the first class, one can successfully apply the methods of mathematical statistics based on data averaging.

The second class includes problems for the solution of which sufficient information potential has not yet been accumulated. In this regard, the opinions of experts can vary greatly from each other. Moreover, the judgment of one expert, which is very different from the rest of the opinions, may turn out to be true. Obviously, the use of methods for averaging the results of a group expert assessment in solving problems of the second class can lead to large errors. Therefore, the processing of the results of a survey of experts in this case should be based on methods that do not use the principles of averaging, but on methods of qualitative analysis.

Considering that the problems of the first class are the most common in the practice of peer review, the focus of this chapter is on the methods of processing the results of the review for this class of problems.

Depending on the goals of expert assessment and the chosen measurement method, the following main tasks arise when processing survey results:

1) building a generalized assessment of objects based on individual assessments of experts;

2) building a generalized assessment based on a paired comparison of objects by each expert;

3) determination of the relative weights of objects;

4) determining the consistency of expert opinions;

5) determination of dependencies between rankings;

6) assessment of the reliability of the processing results.

The task of constructing a generalized assessment of objects based on individual assessments of experts arises in group expert assessment. The solution to this problem depends on the measurement method used by the experts.

When solving many problems, it is not enough to arrange objects according to one indicator or some set of indicators. It is desirable to have numerical values ​​for each object, indicating its relative importance compared to other objects. In other words, for many problems it is necessary to have estimates of objects that not only carry out their ordering, but also allow one to determine the degree of preference of one object over another. To solve this problem, you can directly apply the method of direct evaluation. However, under certain conditions, the same problem can be solved by processing expert estimates.

The determination of the consistency of expert opinions is carried out by calculating a numerical measure that characterizes the degree of similarity of individual opinions. Analysis of the value of the measure of consistency contributes to the development of a correct judgment about the general level of knowledge on the problem being solved and the identification of groupings of expert opinions. A qualitative analysis of the reasons for the grouping of opinions makes it possible to establish the existence of various views, concepts, to identify scientific schools, to determine the nature of professional activity etc. All these factors make it possible to more deeply comprehend the results of the survey of experts.

By processing the results of expert evaluation, it is possible to determine the dependencies between the rankings of various experts and thereby establish the unity and difference in the opinions of experts. An important role is also played by the establishment of the relationship between the rankings built on various indicators of comparison of objects. The identification of such dependencies allows one to reveal related comparison indicators and, perhaps, to group them according to the degree of connection. The importance of the task of determining dependencies for practice is obvious. For example, if the comparison indicators are different goals, and the objects are the means to achieve the goals, then establishing the relationship between the rankings that order the means in terms of achieving the goals allows you to reasonably answer the question of the extent to which the achievement of one goal with these means contributes to the achievement of other goals.

Estimates obtained on the basis of processing are random objects, so one of the important tasks of the processing procedure is to determine their reliability. Appropriate attention should be paid to the solution of this problem.

Processing the results of the examination is a time-consuming process. Performing manual calculations of estimates and indicators of their reliability is associated with large labor costs, even in the case of solving simple ordering problems. For this reason, it is advisable to use computer technology and especially computers. The use of computers raises the problem of developing computer programs that implement algorithms for processing the results of expert evaluation.

3.2. Group evaluation of objects

In this section, we consider algorithms for processing the results of expert evaluation of a set of objects. Let m experts assessed n facilities by l indicators. The evaluation results are presented as values ​​, where j– expert number, i- object number, h– number of the indicator (attribute) of comparison. If the assessment of objects is made by the ranking method, then the values ​​are ranks. If the assessment of objects is carried out by the method of direct assessment or by the method of sequential comparison, then the values ​​are numbers from a certain segment of the numerical axis, or points. The processing of evaluation results depends significantly on the measurement methods considered.

Let us consider the case when the values ​​are obtained by methods of direct evaluation or sequential comparison, i.e., they are numbers or points. To obtain a group rating of objects in this case, you can (use the average value of the rating for each object

(5.1)

where - coefficients of weights of indicators of comparison of objects, - coefficients of experts' competence. Weight coefficients of indicators and competence of objects are normalized values

(5.2)

The weight coefficients of the indicators can be determined by an expert. If is the weight factor h-th indicator given j-th expert, then the average weight coefficient h-th indicator for all experts is equal to

(5.3)

Obtaining a group expert assessment by summing up individual assessments with weights of competence and importance of indicators when measuring the properties of objects in cardinal scales is based on the assumption that the axioms of the von Neumann-Morgenstern utility theory are fulfilled for both individual and group assessment and the conditions for indistinguishability of objects in a group relation, if they are indistinguishable in all individual assessments (partial Pareto principle). In real problems, these conditions are usually met, so obtaining a group assessment of objects by summing individual expert assessments with weights is widely used in practice.

Expert competence coefficients can be calculated from a posteriori data, i.e., from the results of object evaluation. The main idea of ​​this calculation is the assumption that the competence of experts should be assessed by the degree of consistency of their assessments with the group assessment of objects.

The algorithm for calculating the expert competence coefficients has the form of a recurrent procedure:

(5.4)

(5.5)

(5.6)

Calculations start from t=1. In formula (5.4), the initial values ​​of the competence coefficients are assumed to be the same and equal. Then, according to formula (5.4), the group estimates of the objects of the first approximation are equal to the arithmetic mean values ​​of the expert estimates

(5.7)

(5.8)

and the value of the competence coefficients of the first approximation according to the formula (5.6):

(5.9)

Using the competence coefficients of the first approximation, it is possible to repeat the entire process of calculation using formulas (5.4), (5.5), (5.6) and obtain second approximations of the quantities

The repetition of the recurrent procedure for calculating object estimates and competence coefficients naturally raises the question of its convergence. To consider this issue, we eliminate the variables from equations (5.4), (5.6) and represent these equations in vector form

where matrices AT dimensions and FROM dimensions are equal

The quantity in equations (5.10) is determined by formula (5.5).

If matrices AT and FROM are non-negative and indecomposable, then, as follows from the Perron-Frobenius theorem, the prevectors and - converge to the eigenvectors of the matrices AT and FROM, corresponding to the maximum eigenvalues ​​of these matrices

(5.12)

Limit values ​​of vectors X and k can be calculated from the equations:

(5.13)

where are the maximum eigenvalues ​​of matrices AT and FROM .

Condition of non-negativity of matrices AT and FROM is easily done by choosing non-negative elements of the matrix X assessments of objects by experts.

Matrix indecomposability condition AT and FROM practically holds, because if these matrices are decomposable, then this means that experts and objects break up into independent groups. In this case, each group of experts evaluates only the objects of its group. Naturally, it makes no sense to receive a group assessment in this case. Thus, the conditions for non-negativity and indecomposability of matrices AT and FROM, and hence the conditions for the convergence of procedures (5.4), (5.5), (5.6) are satisfied under practical conditions.

It should be noted that the practical calculation of the vectors of group assessment of objects and competence coefficients is easier to perform using recursive formulas (5.4), (5.5), (5.6). Determining the limiting values ​​of these vectors by equation (5.13) requires the use of computer technology.

Let us now consider the case when experts evaluate a set of objects by the ranking method so that the values ​​are ranks. Processing the ranking results is to build a generalized ranking. To construct such a ranking, we introduce a finite-dimensional discrete space of rankings and a metric in this space. Each ranking of a set of objects j The th expert is a point in the ranking space.

The ranking can be represented as a pairwise comparison matrix, the elements of which are defined as follows:

Obviously, since every object is equivalent to itself. The matrix elements are antisymmetric.

If all ranked objects are equivalent, then all elements of the pairwise comparison matrix are equal to zero. We will designate such a matrix and assume that the point in the ranking space corresponding to the matrix is ​​the starting point.

Reversing the order of the ranked objects results in a transposition of the pairwise comparison matrix.

metric as the distance between i-th and j-th rankings is determined uniquely by the formula

if the following 6 axioms are satisfied:

1. moreover, equality is achieved if the rankings and are identical;

2.

moreover, equality is achieved if the ranking "lies between" the rankings and. The concept of "lies between" means that the judgment about some pair of objects in the ranking coincides with the judgment about this pair of either in, or in, or in

4.

where is obtained from some permutation of objects, and from the same permutation. This axiom asserts the independence of distance from the renumbering of objects.

5. If two rankings , are the same everywhere except n-element set of elements that is simultaneously a segment of both rankings, then it can be calculated as if the ranking of only these n-objects. A ranking segment is a set whose complement is non-empty and all elements of this complement are either in front of or behind each element of the segment. The meaning of this axiom is that if two rankings are in complete agreement at the beginning and end of the segment, and the difference is in the ordering of the averages n-objects, it is natural to assume that the distance between the rankings should be equal to the distance corresponding to the rankings of the averages n-objects.

6. The minimum distance is one.

The ranking space for two objects can be represented as three points lying on the same straight line. The distances between the points are equal. With three objects, the space of all possible rankings consists of 13 points.

Using the introduced metric, let's define the generalized ranking as the point that best agrees with the points representing expert rankings. The concept of the best agreement in practice is most often defined as the median and average ranking.

The median is a point in the ranking space, the sum of the distances from which to all points - rankings of experts is minimal. According to the definition, the median is calculated from the condition

The average ranking is such a point, the sum of the squared distances from which to all points - expert rankings is minimal. The average ranking is determined from the condition

The ranking space is finite and discrete, so the median and average ranking can only be any points in this space. In the general case, the median and average ranking may not coincide with any of the expert rankings.

If the competence of experts is taken into account, then the median and average ranking are determined from the conditions:

where are the coefficients of expert competence.

If the objects are ranked by several indicators, then the median is first determined for each expert for all indicators, and then the median is calculated over the set of experts:

(j =1,2,…,m);

where are the weight coefficients of the indicators.

The main disadvantage of determining the generalized ranking in the form of a median or average ranking is the complexity of the calculations. The natural way of finding or in the form of enumeration of all points in the space of rankings is unacceptable due to the very rapid increase in the uniformity of space with an increase in the number of objects and, consequently, the increase in the complexity of calculations. It is possible to reduce the problem of finding or to a specific problem of integer programming. However, this does not reduce computational difficulties very effectively.

The discrepancy between the generalized rankings under various criteria arises when the number of experts is small and their assessments are inconsistent. If the opinions of experts are close, then the generalized rankings, built according to the criteria of the median and the average value, will coincide.

The complexity of calculating the median or average ranking has led to the need for more simple ways building a generalized ranking.

Among such methods is the method of sums of ranks.

This method consists in ranking objects according to the values ​​of the sums of ranks received by each object from all experts. For the ranking matrix, sums are compiled

To take into account the competence of experts, it is enough to multiply each i-th ranking on the coefficient of competence j-th expert In this case, the calculation of the sum of ranks for i th object is produced according to the following formula:

(i =1,2,…,n).

The generalized ranking, taking into account the competence of experts, is based on the ordering of the sums of ranks for all objects.

It should be noted that the construction of a generalized ranking by the sums of ranks is a correct procedure if the ranks are assigned as places of objects in the form of natural numbers 1, 2, ..., n. If the ranks are assigned arbitrarily, as numbers on the order scale, then the sum of ranks, generally speaking, does not preserve the condition of monotonicity of the transformation and, therefore, one can obtain different generalized rankings for different mappings of objects onto a numerical system. The numbering of places of objects can be done in a unique way with the help of natural numbers. Therefore, with good agreement between experts, the construction of a generalized ranking using the rank sum method gives results that are consistent with the results of calculating the median.

Another theoretically more substantiated approach to constructing a generalized ranking is to move from a ranking matrix to a pairwise comparison matrix and calculate an eigenvector corresponding to the maximum eigenvalue of this matrix. Ordering of objects is performed by the magnitude of the eigenvector components.

3.3. Evaluation of the consensus of experts' opinions

When ranking objects, experts usually disagree on the problem being solved. In this regard, there is a need to quantify the degree of agreement of experts. Obtaining a quantitative measure of the consistency of opinions of experts allows a more reasonable interpretation of the reasons for the divergence of opinions.

Currently, there are two measures of consistency of opinions of a group of experts: dispersion and entropy concordance coefficients.

Dispersion coefficient of concordance. Consider the matrix of ranking results n objects by a group of m experts ( j =1,…,m ; i =1,…,n), where is the rank assigned j-th expert i-th object. Compile the sums of the ranks for each column. As a result, we get a vector with components

(i=1,2,…,n). (5.14)

Let us consider the quantities as realizations of a random variable and find an estimate of the variance. As is known, the variance estimate, optimal by the criterion of the minimum mean squared error, is determined by the formula:

, (5.15)

where is the estimate of the mathematical expectation, equal to

The dispersion coefficient of concordance is defined as the ratio of the variance estimate (5.15) to the maximum value of this estimate

The concordance coefficient changes from zero to one, since.

Let us calculate the maximum value of the variance estimate for the case of the absence of related ranks (all objects are different). Let us first show that the estimation of the mathematical expectation depends only on the number of objects and the number of experts. Substituting into (5.16) the value from (5.14), we obtain

Consider first the summed over i at a fixed j. This is the sum of the ranks for j-th expert. Since the expert uses natural numbers from 1 to n, then, as is known, the sum of natural numbers from 1 to n is equal to

(5.19)

Substituting (5.19) into (5.18), we obtain

(5.20)

Thus, the average value depends only on the number of experts m and number of objects n .

To calculate the maximum value of the variance estimate, we substitute the value from (5.14) into (5.15) and square the binomial in parentheses. As a result, we get

(5.21)

Considering that from (5.18) it follows

we get

(5.22)

The maximum dispersion value is reached at highest value the first term in square brackets. The value of this member essentially depends on the location of the ranks - natural numbers in each line i. Let, for example, all m experts gave the same ranking for all n objects. Then the same numbers will be located in each row of the matrix. Therefore, the summation of the ranks in each i-u line gives m-multiple repetition i-ro numbers:

Squaring and summing over i, we get the value of the first term in (5.22) :

(5.23)

Now suppose that the experts give mismatched rankings, for example, for the case n =m all experts assign different ranks to the same object. Then

Comparing this expression with m =n, we make sure that the first term in square brackets of formula (9) is equal to the second term and, therefore, the variance estimate is zero.

Thus, the case of complete coincidence of expert rankings corresponds to the maximum value of the variance estimate. Substituting (5.23) into (5.22) and performing transformations, we obtain

(5.24)

We introduce the notation

(5.25)

Using (5.25), we write the variance estimate (5.15) as

Substituting (5.24), (5.25), (5.26) into (5.17) and reducing by the factor ( n-1), we write the final expression for the concordance coefficient

(5.27)

This formula determines the concordance coefficient for the case of no related ranks.

If the rankings have associated ranks, then the maximum value of the variance in the denominator of formula (5.17) becomes smaller than in the absence of associated ranks. It can be shown that in the presence of related ranks, the concordance coefficient is calculated by the formula:

(5.28)

(5.29)

In formula (5.28) - the indicator of related ranks in j-th ranking, - the number of groups of equal ranks in j-th ranking, - the number of equal ranks in k-th group of related ranks when ranking j th expert. If there are no matching ranks, then =0,=0 and, therefore, =0. In this case formula (5.28) coincides with formula (5.27).

The concordance coefficient is equal to 1 if all expert rankings are the same. The concordance coefficient is zero if all rankings are different, i.e. there is no match at all.

The concordance coefficient calculated by formula (5.27) or (5.28) is an estimate of the true value of the coefficient and, therefore, is a random variable. To determine the significance of the estimate of the concordance coefficient, it is necessary to know the frequency distribution for different meanings number of experts m and number of objects n. Frequency allocation for W for and computed in . For large values m and n known statistics can be used. With the number of objects n>7 the assessment of the significance of the concordance coefficient can be made according to the criterion. Value wm (n -1 ) has a distribution with v=n–1 degrees of freedom.

In the presence of associated ranks, the distribution with v=n-1 degrees of freedom has the value:

(5.30)

Entropy Concordance Coefficient is determined by the formula (coefficient of agreement):

where H is the entropy calculated by the formula

(5.32)

a is the maximum entropy value. In the formula for entropy - probability estimates j-th rank assigned i-th object. These probability estimates are calculated as the ratio of the number of experts who assigned the rank to the object j to the total number of experts.

The maximum value of entropy is achieved with an equiprobable distribution of ranks, i.e. when. Then

Substituting this relation into formula (5.32), we obtain

(5.35)

The agreement coefficient varies from zero to one. When the arrangement of objects by ranks is equally probable, because in this case . This case may be due either to the impossibility of ranking objects according to the formulated set of indicators, or to the complete inconsistency of expert opinions. At , which is achieved at zero entropy ( H=0), all experts give the same ranking. Indeed, in this case, for each fixed object, all experts assign it the same rank j, therefore, , a Therefore, and H =0.

A comparative assessment of the dispersion and entropy concordance coefficients shows that these coefficients give approximately the same assessment of the consistency of experts with similar rankings. However, if, for example, the entire group of experts is divided in opinions into two subgroups, and the rankings in these subgroups are opposite (direct and inverse), then the dispersion coefficient of concordance will be equal to zero, and the entropy coefficient of concordance will be equal to 0.7. Thus, the entropy coefficient of concordance allows fixing the fact of division of opinions into two opposite groups. The amount of calculations for the entropy concordance coefficient is somewhat larger than for the dispersion concordance coefficient.

3.4. Handling pairwise object comparisons

When solving the problem of evaluating a large number of objects (ranking, determining relative weights, scoring), psychological difficulties arise due to the perception by experts of many properties of objects. Experts relatively easily solve the problem of pairwise comparison of objects. The question arises, how to obtain an estimate of the entire set of objects based on the results of pairwise comparison, without imposing the conditions of transitivity? Consider an algorithm for solving this problem. Let m experts evaluate all pairs of objects, giving a numerical estimate

(5.36)

If, when evaluating a pair of experts, they spoke in favor of the preferences of the experts, and the experts consider these objects to be equivalent, then the estimate of the mathematical expectation of a random variable is equal to

(5.37)

The total number of experts is equal to the sum

(5.38)

Determining from here and substituting it into (5.37), we obtain

(5.39)

It is obvious that the set of values ​​forms a matrix on the basis of which it is possible to build a ranking of all objects and determine the coefficients of the relative importance of objects.

Let us introduce the vector of coefficients of the relative importance of objects of the order t the following formula:

where is the matrix of mathematical expectations of estimates of pairs of objects, - vector of coefficients of relative importance of objects of order t . The value is

(5.41)

The coefficients of relative importance of the first order are the relative sums of the elements of the rows of the matrix X. Indeed, assuming t=1, from (5.40) we get

(5.42)

Second-order relative importance coefficients ( t=2) are the relative sums of the elements of the rows of the matrix x2 .

(5.43)

If the matrix X is non-negative and indecomposable, then as the order increases, the quantity converges to the maximum eigenvalue of the matrix X

and the vector of coefficients of the relative importance of objects tends to the eigenvector of the matrix X corresponding to the maximum eigenvalue

The eigenvalues ​​and eigenvectors of the matrix are determined by solving the algebraic equation

where E is the identity matrix, and the systems of linear equations

where k is the matrix eigenvector X corresponding to the maximum eigenvalue . The eigenvector components are coefficients of the relative importance of objects, measured on a ratio scale.

From a practical point of view, it is easier to calculate the coefficients of the relative importance of objects by a sequential procedure according to the formula (5.40) with t=1, 2, … As experience shows, 3-4 consecutive calculations are enough to get the values ​​and k, close to the limiting values ​​determined by equations (5.46), (5.47).

The matrix is ​​non-negative, since all its elements (5.39) are non-negative. A matrix is ​​called indecomposable if it cannot be reduced to a triangular form by permuting rows (rows and columns of the same name)

(5.48)

where are indecomposable submatrices of the matrix X. Matrix representation X in the form (5.48) means the division of objects into l dominant sets

At 1 =n matrix X is indecomposable, i.e., there is only one dominating set coinciding with the original set of objects. Matrix decomposability X means that among the experts there are big disagreements in the assessment of objects.

If the matrix X is indecomposable, then the calculation of the coefficients of relative importance allows you to determine how many times one object is superior to another object in terms of compared indicators. The calculation of the coefficients of the relative importance of objects allows you to simultaneously build the ranking of objects. Objects are ranked so that the first object is the object with the highest relative importance coefficient. The complete ranking is determined by the chain of inequalities

from which follows

If the matrix X is decomposable, then it is possible to determine the coefficients of relative importance only for each set. For each matrix, the maximum eigenvalue and the corresponding eigenvector are determined. The components of the eigenvector are the coefficients of the relative importance of the objects included in the set. According to these coefficients, the objects of this set are ranked. The overall ranking of objects is given by the relation

Thus, if the matrix X is indecomposable, then according to the results of a paired comparison of objects, it is possible both to measure the preference of objects in the scale of relations, and in the scale of order (ranking). If the matrix X is decomposable, then only ranking of objects is possible.

It should be noted that the preference relation can be expressed by any positive number FROM. In this case, the condition must be satisfied. In particular, one can choose FROM=2 so that if , then if then and if , then .

3.5. Determining the relationship of rankings

When processing the ranking results, there may be problems of determining the relationship between the rankings of two experts, the relationship between the achievement of two different goals when solving the same set of problems, or the relationship between two features.

In these cases, the measure of the relationship can be rank correlation coefficient. A characteristic of the relationship of a set of rankings or goals will be a matrix of rank correlation coefficients. Spearman and Kendall's rank correlation coefficients are known.

Spearman's rank correlation coefficient is determined by the formula:

where is the mutual correlation moment of the first and second rankings, are the variances of these rankings. Based on these two rankings, estimates of the mutual correlation moment and variance are calculated by the formulas:

(5.51)

(5.52)

where n- number of ranked objects, - ranks in the first and second rankings, respectively, - average ranks in the first and second rankings. Average rank estimates are determined by the formulas:

(5.53)

Let us calculate estimates of average ranks and variances under the assumption that there are no related ranks in the rankings, i.e., both rankings give a strict ordering of objects. In this case, the average ranks (5.53) are the sums of natural numbers from one to n divided by n. Therefore, the average ranks for both rankings are the same and equal

(5.54)

When calculating the estimates of the variances, we note that if we open the parentheses in formulas (5.52), then natural numbers and their squares will be under the sign of sums. Two rankings can differ from each other only by a permutation of ranks, but the sum of natural numbers and their squares does not depend on the order (permutation) of the terms. Therefore, the variances (5.52) for any two rankings (in the absence of related ranks) will be the same and equal to

(i=1.2). (5.55)

Substituting the value from (5.51) and from (5.55) into formula (5.50), we obtain an estimate for the Spearman rank correlation coefficient

(5.56)

For practical calculations, it is more convenient to use another formula for the Spearman correlation coefficient. It can be obtained from (5.56) using the identity

In equality (5.57), the first two sums on the right-hand side, as follows from expression (5.55), are the same and equal to

Substituting in formula (5.56) the value of the sum from (5.57) and using equality (5.58), we obtain the following formula, convenient for calculations, for Spearman's rank correlation coefficient:

(5.59)

Spearman's correlation coefficient varies from -1 to +1. Equality to one is achieved, as follows from formula (5.59), with the same rankings, i.e., when Value occurs with opposite rankings (direct and reverse rankings). If the correlation coefficient is equal to zero, the rankings are considered linearly independent.

The estimate of the correlation coefficient calculated by formula (5.59) is a random variable. To determine the significance of this estimate, it is necessary to set the probability value , decide on the significance of the correlation coefficient and determine the threshold value using the approximate formula

(5.60)

where n is the number of objects, is the function inverse of the function

for which there are tables. After calculating the threshold value, the correlation coefficient estimate is considered significant if.

To determine the significance of the estimate of the Spearman coefficient, you can use the Student's criterion, since the value

approximately distributed according to Student's law with n- 2 degrees of freedom.

If there are related ranks in the rankings, then the Spearman coefficient is calculated using the following formula:

(5.62)

where is an estimate of the Spearman rank correlation coefficient, calculated by formula (5.59), and the values ​​are

(5.63)

In these formulas - the number of different related ranks in the first and second rankings, respectively.

Kendall's rank correlation coefficient in the absence of related ranks is given by:

where n– number of objects, - ranks of objects, sign x is a function equal to

A comparative evaluation of the Spearman and Kendall rank correlation coefficients shows that the Spearman coefficients are calculated using a simpler formula. In addition, the Spearman coefficient gives a more accurate result, since it is the optimal estimate of the correlation coefficient in terms of the minimum mean squared error criterion.

It follows that in practical calculations of the correlation dependence of rankings, it is preferable to use the Spearman rank correlation coefficient.

CONCLUSION

The dynamism and novelty of modern economic tasks, the possibility of the emergence of various factors affecting the effectiveness of decisions, require that these decisions be made quickly and at the same time be well substantiated. Experience, intuition, a sense of perspective, combined with information, help specialists choose the most important goals and directions of development more accurately, find the best options for solving complex scientific, technical and socio-economic problems in conditions where there is no information about solving similar problems in the past.

The use of the method of expert assessments helps to formalize the procedures for collecting, summarizing and analyzing the opinions of specialists in order to transform them into the most convenient form for making an informed decision.

But, it should be noted that the method of expert assessments cannot replace either administrative or planning decisions, it only allows you to replenish the information necessary for the preparation and adoption of such decisions. The widespread use of expert assessments is justified only where it is impossible to apply more accurate methods to analyze the future.

Expert methods are continuously developed and improved. The main directions of this development are determined by a number of factors, among which one can point to the desire to expand the scope, increase the degree of use of mathematical methods and electronic computers, and also find ways to eliminate emerging shortcomings.

Despite the progress made in recent years in the development and practical use of the method of expert assessments, there are a number of problems and tasks that require further methodological research and practical verification. It is necessary to improve the expert selection system, increase the reliability of group opinion characteristics, develop methods for checking the validity of assessments, and study the hidden causes that reduce the reliability of expert assessments.

However, even today, expert assessments in combination with other mathematical and statistical methods are an important tool for improving management at all levels.

BIBLIOGRAPHY:

2. Bekleshev V.K., Zavlin P.N. Rationing of labor in research institutes and design bureaus. M.: Economics, 1973. 203 p.

10. Dobrov G.M., Ershov Yu.V., Levin E.I., Smirnov L.P. Expert assessments in scientific and technical forecasting. Kyiv: Naukova Dumka, 1974. 263 p.

11. Evlanov L.G. Decision making under uncertainty. M.: IUNKh, 1976. 196 p.

12. Evlanov L.G., Kutuzov V.A. Expert assessments in management. M.: Economics, 1978. 133 p.

13. Kardanskaya N. Management decision making. M.: UNITI, 1999. 407 p.

14. Kemeny D., Snell D. Cybernetic modeling. M.: Soviet radio, 1972. 234 p.

15. Kravchenko T.K. The process of making planning decisions. M.: Economics, 1974. 183 p.

16. Mirkin B.G. The problem of group choice. M.: Nauka, 1974. 256 p. . 17. Mikheev V.I. Socio-psychological aspects of management. Style and methods of work of the head. M.: Young Guard, 1975. 181 p.

18. Pfantsagl I. Theory of measurements. M.: Mir, 1976. 278 p.

19. Tikhomirov Yu.A. management decision. M.: Nauka, 1996. 278 p.

20. Fedorenko N.P. Economy optimization. M.: Nauka, 1977. 236 p.

21. Yampolsky S.M., Lisichkin V.A. Forecasting scientific and technological progress. M.: Economics, 1974. 302 p.

EXPERT DECISION-MAKING METHODS

Decisions can be made either on the basis of objective data (including using optimization methods and probabilistic-statistical models), or on the basis of the opinions of specialists (experts). In the tasks of strategic and operational management, technical and economic analysis, environmental safety, environmental management and environmental protection, etc. various methods of expert assessments are constantly used. They are discussed in this chapter.

The main ideas of methods of expert assessments

Examples of expert assessment methods. How will the economic environment change over time? What will happen to the natural environment in ten years? How will the environment change? Will the environmental safety of industrial production be ensured, or will a man-made desert begin to spread around? It is enough to think about these natural questions, to analyze how we imagined the present day ten or even more than twenty years ago, in order to understand that there simply cannot be 100% reliable forecasts. Instead of statements with specific numbers, only qualitative assessments can be expected. Nevertheless, we, managers, economists, engineers, must make decisions, for example, on environmental and other projects and investments, the consequences of which will be felt in ten, twenty, and so on. years. How to be? It remains to turn to the methods of expert assessments. What are these methods?

It is undeniable that in order to make informed decisions, it is necessary to rely on the experience, knowledge and intuition of specialists. After the Second World War, within the framework of cybernetics, control theory, management and operations research, an independent discipline began to develop - the theory and practice of expert assessments.

Methods of expert assessments are methods for organizing work with expert experts and processing expert opinions. These opinions are usually expressed partly in quantitative, partly in qualitative form. Expert research is carried out in order to prepare information for decision-making by the decision maker (recall, the decision maker is the decision maker). To carry out work on the method of expert assessments, a Working Group is created (abbreviated as WG), which organizes, on behalf of the decision maker, the activities of experts united (formally or in essence) in an expert commission (EC).

Expert opinions are individual and collective. Individual ratings These are the estimates of one specialist. For example, a teacher single-handedly puts a mark on a student, and a doctor makes a diagnosis on a patient. But in difficult cases of illness or the threat of expulsion of a student for poor study, they turn to collective opinion - a symposium of doctors or a commission of teachers. The situation is similar in the army. Usually the commander makes the decision alone. But in difficult and responsible situations, a military council is held. One of the most famous examples of this kind is the military council of 1812 in Fili, at which, under the chairmanship of M.I. Kutuzov, the question was decided: "To give or not to give the French a battle near Moscow?"

Another simple example of expert assessments is the assessment of numbers in KVN. Each of the jury members raises the plywood with their score, and the technical worker calculates the arithmetic mean score, which is declared as the collective opinion of the jury (we will see below that this approach is incorrect from the point of view of measurement theory).

In figure skating, the procedure becomes more complicated - before averaging the largest and smallest scores are discarded. This is done so that there is no temptation to overestimate one athlete (for example, a compatriot) or underestimate another. Such estimates that stand out sharply from the general series will be immediately discarded.

Expert judgment is often used in selection, for example:

One variant of a technical device for launching a series of several samples,

Groups of astronauts from many applicants,

Recruitment of research projects for funding from the mass of applications,

Recipients of environmental loans from many applicants,

When choosing investment projects for implementation among those presented, etc.

There are many methods for obtaining expert assessments. In some, they work with each expert separately, he does not even know who else is an expert, and therefore expresses his opinion regardless of the authorities. In others, experts are brought together to prepare materials for the decision maker, while the experts discuss the problem with each other, learn from each other, and incorrect opinions are discarded. In some methods, the number of experts is fixed and such that statistical methods for checking the consistency of opinions and then averaging them allow making informed decisions. In others, the number of experts grows in the process of conducting an examination, for example, when using the "snowball" method (more on that later).

There are no less methods for processing the answers of experts, including those very rich in mathematics and computerized. Many of them are based on the achievements of the statistics of non-numerical objects and other modern methods of applied statistics.

One of the most well-known peer review methods is Delphi method. The name is given by association with the ancient custom to get support when making decisions to apply to the Delphic temple. It was located at the exit of poisonous volcanic gases. The priestesses of the temple, inhaling poison, began to prophesy, uttering incomprehensible words. Special "translators" - the priests of the temple interpreted these words and noted the questions of the pilgrims who came with their problems. According to tradition, it is said that the Temple of Delphi was located in Greece. But there are no volcanoes. Apparently, he was in Italy - near Vesuvius or Etna, and the predictions themselves described took place in the XII-XIV centuries. This follows from the highest achievement of modern historical science - the new statistical chronology.

In the United States in the 1960s, the Delphi method was called an expert procedure for predicting scientific and technological development. In the first round, the experts called the probable dates of certain future accomplishments. In the second round, each expert got acquainted with the forecasts of all the others. If his forecast was very different from the forecasts of the bulk, he was asked to explain his position, and often he changed his estimates, approaching the average values. These average values ​​were given to the customer as a group opinion. It must be said that real results research turned out to be rather modest - although the date of the landing of Americans on the moon was predicted to within a month, all other forecasts failed - cold thermonuclear fusion and a cure for cancer in the twentieth century. mankind did not wait.

However, the technique itself turned out to be popular - in subsequent years it was used at least 40 thousand times. The average cost of an expert study using the Delphi method is 5,000 US dollars, but in some cases, it was necessary to spend even larger sums - up to 130,000 US dollars.

Somewhat aside from the mainstream of expert assessments lies scripting method used primarily for expert forecasting. Let's consider the main ideas of the technology of scenario expert forecasts. Environmental or socio-economic forecasting, like any forecasting in general, can be successful only under some stability of conditions. However, the decisions of authorities, individuals, and other events change the conditions, and events develop in a different way than previously expected. It is quite obvious that after the first round of the presidential elections in 1996, one could speak about the further development of events only in terms of scenarios: if B.N. Yeltsin, then this and that will happen, if G.A. wins. Zyuganov, then events will go this way and that way.

The scenario method is needed not only in the socio-economic or environmental field. For example, when developing methodological, software and information support risk analysis chemical and technological projects, it is necessary to compile a detailed catalog of accident scenarios associated with leaks of toxic chemical substances. Each of these scenarios describes an accident of its type, with its individual origin, development, consequences, and warning capabilities.

Thus, the scenario method is a method of decomposition of the forecasting problem, which provides for the selection of a set of individual options for the development of events (scenarios), which together cover all possible development options. At the same time, each individual scenario should allow for sufficiently accurate forecasting, and the total number of scenarios should be visible.

The possibility of such a decomposition is not obvious. When applying the scenario method, it is necessary to carry out two stages of the study:

Building a comprehensive but manageable set of scenarios;

Forecasting within each specific scenario in order to obtain answers to questions of interest to the researcher.

Each of these stages is only partially formalized. A significant part of the reasoning is carried out at a qualitative level, as is customary in socio-economic and humanities. One of the reasons is that the desire for excessive formalization and mathematization leads to artificial the introduction of certainty where it does not exist in essence, or the use of a cumbersome mathematical apparatus. Thus, reasoning at the verbal level is considered evidentiary in most situations, while an attempt to clarify the meaning of the words used, using, for example, fuzzy set theory, leads to very cumbersome mathematical models.

The set of scenarios should be visible. We have to exclude various unlikely events - the arrival of aliens, the fall of an asteroid, mass epidemics of previously unknown diseases, etc. In itself, the creation of a set of scenarios is the subject of expert study. In addition, experts can assess the probabilities of the implementation of a particular scenario.

Forecasting within each specific scenario in order to obtain answers to questions of interest to the researcher is also carried out in accordance with the forecasting methodology described above. Under stable conditions, statistical methods for forecasting time series can be applied. However, this is preceded by an analysis with the help of experts, and often forecasting at the verbal level is sufficient (to obtain conclusions of interest to the researcher and decision maker) and does not require quantitative clarification.

As you know, when making decisions based on situation analysis(as they say, situational analysis), including the analysis of the results of predictive studies, can be based on various criteria. So, you can focus on the fact that the situation will develop in the worst, or best, or average (in any sense) way. You can try to outline activities that provide the minimum acceptable useful results in any scenario, etc.

Another option for peer review is brainstorm. It is organized as a meeting of experts, on whose speeches one, but very significant, restriction is imposed - one cannot criticize the proposals of others. You can develop them, you can express your ideas, but you can’t criticize! During the meeting, the experts, "infecting" each other, express more and more extravagant considerations. Two hours later, the session recorded on a tape recorder or video camera ends, and the second stage of brainstorming begins - the analysis of the ideas expressed. Usually, out of 100 ideas, 30 deserve further elaboration, out of 5-6 they make it possible to formulate applied projects, and 2-3 ultimately turn out to bring a beneficial effect - profit, increased environmental safety, improvement of the natural environment, etc. At the same time, the interpretation of ideas is a creative process. For example, when discussing the possibilities of protecting ships from a torpedo attack, the idea was expressed: "Line up the sailors along the side and blow on the torpedo to change its course." After elaboration, this idea led to the creation of special devices that create waves that knock the torpedo off course.

The main stages of the expert survey. Let's take a closer look at the individual stages of expert research. As experience shows, from the point of view of the manager - the organizer of such a study, it is advisable to single out the following stages of an expert survey.

1) Deciding on the need to conduct an expert survey and the formulation by the Decision Maker (DM) of its purpose. Thus, the initiative should come from the management, which will ensure the successful solution of organizational and financial problems in the future. Obviously, the initial impetus can be given by a memorandum of one of the employees or a discussion at a meeting, but the real start of work is the decision of the decision maker.

2) Selection and appointment of the decision maker of the main composition of the Working Group, abbreviated as WG (usually - supervisor and secretary). At the same time, the supervisor is responsible for organizing and conducting the expert study as a whole, as well as for analyzing the collected materials and formulating the conclusion of the expert commission. He participates in the formation of a team of experts and the issuance of a task to each expert (together with the decision maker or his representative). He himself is a highly qualified expert and a formal and informal leader of the expert commission recognized by other experts. The secretary's job is to keep documentation of the expert survey, to solve organizational problems.

3) WG development(more precisely, its main staff, primarily the supervisor and secretary) and approval by the decision maker of the terms of reference for conducting an expert survey. At this stage, the decision to conduct an expert survey becomes clear in terms of time, financial, personnel, material and organizational support. In particular, a Working Group is formed, various groups of specialists are distinguished in the WG - analytical, econometric (specialists in methods), computer, for working with experts (for example, interviewers), and organizational. It is very important for success that all these positions are approved by the decision maker.

4) Development by the WG analytical group of a detailed scenario (i.e. regulations) for the collection and analysis of expert opinions (assessments). The scenario includes, first of all, a specific type of information that will be received from experts (for example, words, conditional gradations, numbers, rankings, splits, or other types of non-numerical objects). For example, quite often experts are asked to speak freely, while answering a number of pre-formulated questions. In addition, they are asked to complete a formal map, choosing one of several gradations at each point. The script should also contain specific methods for analyzing the collected information. For example, the calculation of the Kemeny median, statistical analysis of Lucians, the use of other methods of statistics of non-numerical objects and other sections of applied statistics (some of these methods will be discussed below). This work falls on the econometric and computer group of the WG. The traditional mistake is to collect information first, and then think about what to do with it. As a result, as sad experience shows, information is used by no more than 1-2%.

5) Selection of experts according to their competence. At this stage, the WG draws up a list of possible experts and evaluates their suitability for the proposed study.

6) Formation of an expert commission. At this stage, the WG conducts negotiations with experts, obtains their consent to work in the expert commission (abbreviated as EC). It is possible that some of the experts appointed by the WG cannot be included in the expert commission (illness, vacation, business trip, etc.) or refuse for one reason or another (employment, contract conditions, etc.). The decision maker approves the composition of the expert commission, possibly by deleting or adding some experts to the proposals of the WG. Contracts are being concluded with experts on the conditions of their work and their payment.

7) Collection of expert information. Often this is preceded by the recruitment and training of interviewers - one of the groups that make up the WG.

8) Computer analysis of expert information using the methods included in the script. It is usually preceded by the introduction of information into computers.

9) When applied according to the scenario of the expert procedure from several rounds - repetition two previous stages.

10) Final analysis of expert opinions, interpretation of the results analytical group of the WG and preparation of the final document EC for decision maker.

11) Official the ending activities of the WG, including approval of the decision maker of the final document of the EC, preparation and approval of the scientific and financial reports of the WG on the conduct of an expert study, remuneration of experts and employees of the WG, official termination of activities (dissolution) of the EC and WG.

Let us analyze in more detail the individual stages of expert research. Let's start with the selection of experts: personnel decide everything! What are the experts - such is the quality of the conclusion of the expert commission.

Selection of experts. The problem of selecting experts is one of the most difficult in the theory and practice of expert research. Obviously, as experts it is necessary to use those people whose judgments will most help to make an adequate decision. But how to identify, find, select such people? It must be said directly that there are no methods for selecting experts that will surely ensure the success of the examination. Now we will not discuss the problem of the existence of various "parties" among experts and will pay attention to various other aspects of the procedures for selecting experts.

There are two components to the problem of selection of experts - compiling a list of possible experts and selecting an expert commission from them in accordance with the competence of the candidates.

Compilation of a list of possible experts is facilitated when the type of examination in question is carried out repeatedly. In such situations, it is usually registry possible experts, for example, in the field of state environmental expertise or refereeing figure skating, from which you can choose according to various criteria or using a generator (or table) of pseudo-random numbers.

What if the examination is carried out for the first time, there are no established lists of possible experts? However, even in this case, each specific specialist has some idea of ​​what is required from an expert in a similar situation. To form a list, useful method "snowball" in which a certain number (usually 5 - 10) of the names of those who can be an expert on the subject under consideration are received from each specialist involved as an expert. Obviously, some of these surnames met earlier in the activities of the WG, and some are new. Each newcomer is interrogated according to the same scheme. The process of expanding the list stops when new surnames practically cease to occur. The result is a rather extensive list of possible experts. Method "snowball" also has disadvantages. The number of rounds before the coma build-up process stops cannot be predicted in advance. In addition, it is clear that if at the first stage all experts were from the same "clan", held similar views or were engaged in similar activities, then the "snowball" method will most likely give persons from the same "clan" . Opinions and arguments of other "clans" will be missed. (Here we are talking about the fact that the community of specialists is actually divided into groups called "clans" above, and communication takes place mainly within the "clans". The informal structure of science, to which the "clans" belong, is quite difficult to study. We note here that that "clans" are usually formed on the basis of large formal centers (universities, scientific institutes), scientific schools.)

The issue of evaluating the competence of experts is no less complicated. It is clear that the success of participation in previous examinations is a good criterion for the activities of a taster, doctor, judge in sports competitions, i.e. such experts who participate in a long series of similar examinations. However, alas, the most interesting and important are the unique expertise of large projects that have no analogues. The use of formal indicators of experts (position, academic degree and title, length of service, number of publications ...), obviously, in today's rapidly changing conditions can only be of an auxiliary nature, although such indicators are the easiest to apply.

It is often proposed to use methods of self-assessment and mutual assessment of the competence of experts. Let's discuss them, starting with the self-assessment method, in which the expert himself gives information about in which areas he is competent and in which he is not. On the one hand, who better to know the capabilities of an expert than he himself? On the other hand, self-assessment of competence rather assesses the degree of self-confidence of an expert than his actual competence. Moreover, the very concept "competence" not strictly defined. It can be refined by highlighting the components, but this complicates the preliminary part of the work of the expert commission. Quite often, an expert exaggerates his real competence. For example, most people believe that they are well versed in politics, economics, education and upbringing, family and medicine. In fact, experts (and even knowledgeable people) in these areas is very small. There are also deviations in the other direction, an overly critical attitude towards one's capabilities.

When using the method of mutual assessment, in addition to the possibility of displaying personal and group likes and dislikes, the low awareness of experts about each other's capabilities plays a role. In modern conditions, only specialists who have been working together for many years (at least 3-4) working together, in the same room, on the same topic, can have a fairly good acquaintance with the work and capabilities of each other. It is about such couples that one can say that they " ate a pood of salt together". However, the involvement of such pairs of specialists is not very advisable, since their views, due to the similarity life path too similar to each other.

If the expert survey procedure involves direct communication of experts, a number of other circumstances must be taken into account. Their personal (social-psychological) qualities are of great importance. So, the one and only" talker"can paralyze the activities of the entire commission at a joint meeting. Both the hostile relations of the commission members and the very different scientific and official status of the commission members can lead to a disruption. In such cases, it is important to comply with the work regulations developed by the WG.

It should be emphasized that the selection of experts is one of the main functions of the Working Group, and no selection methods can relieve it of responsibility. In other words, it is the Working Group that is responsible for the competence of the experts, for their fundamental ability to solve the problem. An important requirement is for the decision maker to approve the list of experts. At the same time, the decision maker can either add individual experts to the commission or delete some of them - for his own reasons, which members of the WG and EC do not need to get acquainted with.

There are a number of normative documents regulating the activities of expert commissions in certain areas. An example is the Law Russian Federation"On Ecological Expertise" dated November 23, 1995, which regulates the procedure for the examination of "proposed economic or other activities" in order to identify possible harm that the activity in question can cause to the environment.

On the development of regulations for the collection and analysis of expert opinions. There are many methods for obtaining expert assessments. In some, they work with each expert separately, he does not even know who else is an expert, and therefore expresses his opinion regardless of authorities, "clans" and individual colleagues. In others, experts are brought together to prepare materials for the decision maker, while the experts discuss the problem with each other, accept or reject each other's arguments, learn from each other, and incorrect or insufficiently substantiated opinions are discarded. In some methods, the number of experts is fixed and such that statistical methods of checking the consistency of opinions and then (in the case of sufficiently good agreement of opinions) averaging them allow making informed decisions from the point of view of econometrics. In others, the number of experts grows in the course of the examination, for example, when using the "snowball" method to form a team of experts.

Currently does not exist generally accepted scientifically substantiated classification of methods of expert assessments, and even more so - unambiguous recommendations for their application. An attempt to forcefully approve one of the possible points of view on the classification of methods of expert assessments can only bring harm.

However, to talk about the variety of expert assessments, some working classification of methods is needed. We give one of these possible classifications below, listing the grounds on which we divide expert assessments.

One of the main questions - what exactly should the expert commission provide as a result of its work - information for making a decision by the decision maker or a draft decision itself? The organization of the work of the expert commission depends on the answer to this methodological question, and it serves as the first basis for splitting the methods.

PURPOSE - COLLECTING INFORMATION FOR DMP. Then the Working Group should collect as much relevant information as possible, arguments "for" and "against" certain solutions. The following method of gradually increasing the number of experts is useful. First, the first expert gives his views on the issue under consideration. The material compiled by him is transferred to the second expert, who adds his arguments. The accumulated material goes to the next - third - expert... The procedure ends when the flow of new considerations dries up.

Note that the experts in the method under consideration only provide information, arguments "for" and "against", but do not develop an agreed draft decision. There is no need to strive to ensure that expert opinions are consistent with each other. Moreover, experts with a mindset that deviates from the masses are most useful. It is from them that the most original arguments should be expected.

PURPOSE - PREPARATION OF A DRAFT DECISION FOR DECISIONS. Mathematical methods in expert assessments are usually used specifically for solving problems related to the preparation of a draft decision. At the same time, the dogmas of consistency and one-dimensionality are often uncritically accepted. These dogmas "roam" from one publication to another, so it is advisable to discuss them.

DOGMA OF CONSISTENCY. It is often assumed, without any justification, that a decision can only be made on the basis of the agreed opinions of experts. Therefore, those whose opinion differs from the opinion of the majority are excluded from the expert group. At the same time, both unqualified persons who got into the composition of the expert commission due to a misunderstanding or for reasons not related to their professional level, as well as the most original thinkers who have penetrated deeper into the problem than the majority, are eliminated. Their arguments should be clarified, they should be given the opportunity to substantiate their points of view. Instead, their opinion is ignored.

It also happens that experts are divided into two or more groups that have common group points of view. Thus, there is a well-known example of dividing specialists in evaluating the results of scientific research into two groups: "theorists" who clearly prefer R&D in which theoretical results are obtained, and "practitioners" who choose those R&D that allow obtaining direct applied results (we are talking about R&D competition at the Academic Institute for Control Problems (Automation and Telemechanics)).

It is sometimes claimed that if two or more groups of experts are found (instead of one agreed upon), the survey does not achieve its goal. This is not true! The goal has been achieved - it has been established that there is no consensus. This is very important. And the decision maker should take this into account when making decisions. The desire to ensure the consistency of the opinions of experts of any whole can lead to a deliberate one-sided selection of experts, ignoring all points of view, except for one, the most beloved Working Group (or even "prompted" by the decision maker).

Another purely econometric circumstance is often not taken into account. Since the number of experts usually does not exceed 20-30, then the formal statistical consistency of expert opinions (established using certain verification criteria statistical hypotheses) can be combined with the actual division of experts into groups, which makes further calculations irrelevant to reality. For example, let's turn to specific calculation methods using concordance coefficients (ie, in translation - agreement) based on Kendall's or Spearman's rank correlation coefficients. It should be recalled that according to econometric theory, a positive result of checking consistency in this way means nothing more and nothing less than rejecting the hypothesis of independence and uniform distribution of expert opinions on the set of all rankings. Thus, the null hypothesis is tested, according to which the rankings describing the opinions of experts are independent random binary relations uniformly distributed over the set of all rankings. The rejection of this null hypothesis, according to bad tradition, is interpreted as the consistency of the experts' answers. In other words, we fall victim to misconceptions arising from the peculiar interpretation of words: the consistency check in the indicated mathematical-statistical sense is not at all a consistency check in the sense of the practice of expert assessments. (It is the defectiveness of the considered mathematical and statistical methods of ranking analysis that led a group of specialists to develop a new econometric apparatus for checking consistency - non-parametric methods based on the so-called. lucians and included in the modern section of econometrics - non-numeric data statistics). Groups of experts with similar methods can be distinguished by econometric methods of cluster analysis.

OPINIONS OF DISSIDENTS. In order to artificially achieve consistency, they try to reduce the influence of expert opinions. dissidents, i.e. dissenters compared to the majority. Hard the way to deal with dissidents is to ignore their opinions, i.e. in fact, their exclusion from the composition of the expert commission. The rejection of experts, as well as the rejection of outliers (outliers), leads to procedures that have poor or unknown statistical properties. Yes, known extreme instability classical methods for rejecting outliers with respect to deviations from model assumptions (see, for example, tutorial ).

Soft way to deal with dissidents is to use robust (stable) statistical procedures. The simplest example: if the expert's answer is a real number, then the outlier opinion of the dissident strongly affects the arithmetic mean of the experts' answers and does not affect their median. Therefore, it is reasonable to consider the median as a consensus opinion. However, this ignores (does not reach the decision maker) the arguments of the dissidents.

In either of the two ways of dealing with dissidents, the decision maker is deprived of information coming from dissidents, and therefore can make an unreasonable decision, which will subsequently lead to negative consequences. On the other hand, the submission of the entire set of opinions to the decision maker removes part of the responsibility and labor for preparing the final decision from the commission of experts and the working group for conducting an expert survey and shifts this responsibility and labor onto the shoulders of the decision maker.

DOGMA OF ONE-DIMENSIONALITY. In outdated, and sometimes in modern scientific and technical literature, a rather controversial approach of the so-called "qualimetry" is widespread, according to which the object of examination can always be assessed. one number. Strange idea! Evaluating a person by one number came to mind only in slave markets. It is unlikely that even the most zealous qualimetrists consider a book or a picture as equivalent to a number - its "market value". Almost all real objects are quite complex, and therefore they can be described with any accuracy only with the help of many and many numbers, as well as mathematical objects of a non-numerical nature.

At the same time, one cannot completely deny the very idea of ​​searching for generalized indicators of quality, technical level, and similar ones. So, each object can be evaluated by many quality indicators. For example, a car can be evaluated on the following indicators:

gasoline consumption per 100 km (on average);

reliability (including the average cost of repairs per year);

environmental safety, assessed by the content of harmful substances in exhaust gases;

maneuverability (including turning radius);

the speed of picking up speed of 100 km / h after the start of movement; maximum attainable speed;

the duration of maintaining a positive temperature in the cabin at a low outside temperature (for example, minus fifty degrees Celsius) and the engine is off;

design (attractiveness and "fashionableness" of appearance and interior trim);

weight, etc.

Is it possible to summarize the scores for these indicators together? It is clear that the specific situation for which the car is selected is decisive. The maximum speed achieved is important for the racer, but, as we see it, is of little practical importance for the driver of an ordinary private car, especially in a city with a severe limit on maximum speed. For such a driver, gas mileage, maneuverability and reliability are more important. For machines of various services government controlled, apparently, reliability is more important than for a private trader, and gasoline consumption is the opposite. For regions of the Far North, thermal insulation of the cabin is important, but not for southern regions. Etc.

Thus, a specific (narrow) statement of the problem to the experts is important. But such a setting often does not exist. And then "games" to develop a generalized quality indicator - for example, in the form linear function from the listed variables - cannot give objective conclusions. An alternative to the only generalized indicator is a mathematical apparatus of the type multiobjective optimization- Pareto sets, etc.

In some cases, it is still possible to globally compare objects - for example, with the help of the same experts, you can get an ordering of the objects under consideration - products or projects. Then you can choose the coefficients for individual indicators so that ordering by linear function was as close as possible to global ordering(for example, find these coefficients using the least squares method). On the contrary, in such cases SHOULD NOT evaluate the indicated coefficients with the help of experts. This simple idea has not yet become obvious to individual compilers of methodologies for conducting expert surveys and analyzing their results. They try hard to get the experts to do what they do unable- indicate the weights with which individual quality indicators should be included in the final generalized indicator.

Send your good work in the knowledge base is simple. Use the form below

Students, graduate students, young scientists who use the knowledge base in their studies and work will be very grateful to you.

Posted on http://www.allbest.ru/

MOSCOW SOCIO-ECONOMIC INSTITUTE

on the topic "Methodology for conducting expert assessments"

Students:

Artyushenko Yulia Viktorovna

Group: M10B-D-O-z

Moscow 2014

Introduction

2. Methods of expert assessments

Conclusion

Introduction

In the study of management, the method of expert assessments is widely used. This is due to the complexity of many problems, their origin from the "human factor", the lack of reliable experimental or normative tools.

It is undeniable that in order to make informed decisions, it is necessary to rely on the experience, knowledge and intuition of specialists. After the Second World War, within the framework of the theory of management (management), an independent discipline began to develop - expert assessments.

Methods of expert assessments are methods for organizing work with specialist experts and processing expert opinions expressed in quantitative and / or qualitative form in order to prepare information for decision-making by decision makers.

Many works have been devoted to the study of the possibilities and features of the application of expert assessments. They consider the forms of an expert survey (various types of questionnaires, interviews), assessment approaches (ranking, normalization, various types of ordering, etc.), methods for processing survey results, requirements for experts and the formation of expert groups, issues of training experts, assessments their competence (when processing the assessments, the coefficients of the competence of experts, the reliability of their opinions are introduced and taken into account), methods of organizing expert surveys. The choice of forms and methods for conducting expert surveys, approaches to processing survey results, etc. depends on the specific task and conditions of the examination.

Expert methods are now used in situations where the choice, justification and evaluation of the consequences of decisions cannot be performed on the basis of accurate calculations. Such situations often arise in the development of modern problems of managing social production and, especially, in forecasting and long-term planning. In recent years, expert assessments have been widely used in socio-political and scientific-technical forecasting, in the planning of the national economy, industries, associations, in the development of major scientific, technical, economic and social programs, in solving certain management problems. expert management ranking

1. Essence, methods and process of expert assessments

1.1 The essence of expert assessments

The possibility of using expert assessments, the justification of their objectivity is usually based on the fact that an unknown characteristic of the phenomenon under study is interpreted as a random variable, the reflection of the distribution law of which is an individual assessment of a specialist expert on the reliability and significance of an event. It is assumed that the true value of the characteristic under study is within the range of estimates received from the group of experts, and that the generalized collective opinion is reliable.

However, some theoretical studies question this assumption. For example, it is proposed to divide the problems for which expert assessments are used into two classes. The first class includes problems that are sufficiently well provided with information and for which the principle of a “good measurer” can be used, considering the expert as the custodian of a large amount of information, and the group opinion of experts is close to the true one. The second class includes problems in respect of which there is not enough knowledge to be sure of the validity of the above assumptions; experts cannot be considered as “good measurers”, and it is necessary to carefully approach the processing of the results of the examination, since in this case the opinion of one (single) expert, who pays more attention to the study of a little-studied problem, may turn out to be the most significant, and during formal processing it will be lost. In this regard, qualitative processing of results should be mainly applied to problems of the second class. The use of averaging methods (valid for "good meters") in this case can lead to significant errors.

The tasks of collective decision-making on the formation of goals, the improvement of methods and forms of management can usually be attributed to the first class. However, when developing forecasts and long-term plans, it is advisable to identify “rare” opinions and subject them to a more thorough analysis.

Another problem that needs to be kept in mind when conducting a system analysis is the following: even in the case of solving problems related to the first class, one should not forget that expert assessments carry not only narrowly subjective features inherent in individual experts, but also collective-subjective features that do not disappear when processing the results of the survey (and when using Delphi procedures, they can even be enhanced). In other words, expert assessments should be viewed as some kind of “public point of view”, depending on the level of scientific and technical knowledge of the society regarding the subject of research, which can change as the system and our ideas about it develop. Therefore, an expert survey is not a one-time procedure. This way of obtaining information about a complex problem characterized by a high degree of uncertainty should become a kind of "mechanism" in a complex system, i.e. it is necessary to create a regular system of work with experts.

Attention should also be paid to the fact that the use of the classical frequency approach to assessing probability when organizing expert surveys can be difficult, and sometimes impossible (due to the impossibility of proving the legitimacy of using a representative sample). Therefore, at present, studies are underway on the nature of the probability of expert assessment, based on the theory, fuzzy sets of Zadeh, on the idea of ​​expert assessment as a degree of confirmation of a hypothesis or as a probability of achieving a goal. One of the varieties expert method is a method of studying the strengths and weaknesses of the organization, the opportunities and threats to its activities - the method of SWOT analysis.

The collection of expert information depends on the choice of the method of expert assessments. Usually, to collect expert information, special documents are compiled, for example, questionnaires approved by the relevant managers and then sent to the experts.

Processing of expert information is carried out using the chosen method, usually with the use of computer technology. The data obtained as a result of processing is analyzed and used to solve the problems of analysis and synthesis of control systems.

Expert assessments are used for analysis, diagnosis of the state, subsequent prediction of development options:

1) objects, the development of which is either completely or partially not amenable to subject description or mathematical formalization;

2) in the absence of sufficiently representative and reliable statistics on the characteristics of the object;

3) in conditions of great uncertainty in the environment for the functioning of the object, the market environment;

4) in medium- and long-term forecasting of new markets, objects of new industries that are strongly influenced by discoveries in the fundamental sciences (for example, the microbiological industry, quantum electronics, nuclear engineering);

5) in cases where either the time or the funds allocated for forecasting and decision-making do not allow to investigate the problem using formal models;

6) there are no necessary technical means of modeling, for example, computer technology with the appropriate characteristics;

7) in extreme situations.

The tasks solved in the process of expert assessments of control systems can be divided into two groups:

1) tasks of synthesis of new control systems and their evaluation;

2) tasks of analysis (measurement) of existing management systems according to selected indicators and performance criteria.

The tasks of the first group include: formation of the image of the system being created; forecasting technical and economic indicators of the stages of its life cycle; substantiation of the main directions of the reorganization of the social management system; selection of optimal or satisfactory methods of action and outcomes using the created control system, etc. Some of the expert information obtained in the course of solving these problems is of a qualitative nature and is formed in the form of complex judgments in a descriptive form. However, the tasks of synthesis solved with the help of expert assessments can be quantitative in nature, and their solution will be associated with the justification of numerous parameters (characteristics) of the system being created. The tasks of the second group include all the tasks of evaluating existing or created variants of control systems using specified indicators and performance criteria. Examples of such tasks are: determining the structural, functional or informational characteristics of the system; evaluation of its effectiveness in the course of performing various operations; determination of the expediency of further operation of technical means of control and communication, etc.

1.2 The role of experts in management

Expertise is an opinion, idea, decision or assessment based on the implementation of the valuable experience of a specialist, deep knowledge of the subject of research and qualitative analysis technologies.

Expertise can be individual or group. In group expertise, the selection of a group of experts and the methodology for the final processing of the results of its work are of great importance.

The expert opinion is a document that records the course of the study and its results. At the same time, the conclusions and opinions of experts can have both categorical ("yes", "no"), and probabilistic (in the form of an assumption, ranking, preference coefficient, etc.) form.

In organizing the work of experts, it is necessary to adhere to the following principles:

1. Ideas, opinions and assessments should fit into a pre-prepared scheme. This allows you to generalize, compare, highlight the essential, etc. Such a scheme should not constrain thought and limit fantasy. The scheme may allow and assume the possibility of its modification and addition.

2. The processing of expert opinions must be carried out not only in quantitative generalization, but also through qualitative analysis, highlighting the main, essential, important, relevant, original, new, etc. Expert opinion can be the subject of examination of the second stage.

3. Experts must be independent, i.e. freed from any organizational or conceptual, as well as psychological restrictions. In this case, their experience, knowledge and intuition are realized in the best way.

4. The work of the expert group should be purposeful. Understanding why and why an examination is carried out is an important element of its implementation. In many cases, special training of experts is needed, which plays the role of mobilizing efforts and intelligence.

5. There are various forms of organizing the work of an expert group: either each expert makes an examination individually, then the results are summarized and systematized, or the experts work collectively, interacting with each other.

6. Parallel and multi-stage work of several expert groups is possible. Comparison of expertise provides important information.

There are many methods for obtaining expert assessments. In some, they work with each expert separately, he does not even know who else is an expert, and therefore expresses his opinion regardless of the authorities. In others, experts are brought together to prepare materials for the decision maker, while the experts discuss the problem with each other, learn from each other, and incorrect opinions are discarded. In some methods, the number of experts is fixed and such that statistical methods for checking the consistency of opinions and then averaging them allow making informed decisions. In others, the number of examiners grows during the course of the examination, for example, when using the "snowball" method.

A specialist or group of specialists acting as experts is sometimes identified with measuring device, which has random and systematic measurement errors.

Random errors are due to the subjectivity of expert opinions on the issue under consideration and may deviate in one direction or another from the true value. The impact of such errors is reduced by averaging a sufficient number of estimates.

A systematic error is inherent in the entire team of experts and cannot be eliminated by processing the obtained estimates. This suggests that in some cases it is necessary to approach the results of an expert survey very carefully, which can sometimes express a generally erroneous point of view, depending on the level of knowledge and beliefs of experts.

1.3 Peer review process

The main stages of the peer review process include:

Formation of the goal and objectives of expert assessment;

Formation of a management group and execution of a decision to conduct an expert assessment;

Choosing a method for obtaining expert information and methods for its processing;

Selection of an expert group and formation, if necessary, of survey questionnaires;

Survey of experts (expertise);

Processing and analysis of the results of the examination;

Interpretation of the obtained results;

Compilation of a report.

The task of conducting an expert assessment is set by the decision maker. The stage of forming the goal and objectives of expert evaluation is the main one. The reliability of the result obtained and its pragmatic value depend on it. The formation of the goal and objectives of expert evaluation is dictated by the essence of the problem being solved. Here, the following factors should be taken into account: the reliability and completeness of the available initial information, the required form of presenting the result (qualitative or quantitative), the possible areas of use of the information received, the timing of its submission, the resources available to management, the possibility of attracting specialists from other fields of knowledge, and much more. The task is formalized in the form of a guiding document (for example, a decision to conduct an expert assessment).

To prepare the decision and guide all further work, the head of the examination is appointed. It defines the composition of the management group. The control group provides feedback to experts or the Delphi method.

The management group is entrusted not only with all organizational and planning work to provide favorable conditions for the effective creative activity of experts, but also with analytical work on the selection of an expert group, determining methods for obtaining and processing information, compiling questionnaires - questionnaires, meaningful interpretation of the results.

This large and complex range of tasks to be solved requires the inclusion of highly qualified specialists in the management group both in the field of the problem under consideration and in other areas - psychology, mathematics, medicine, sociology.

The selection of specific experts is carried out on the basis of an analysis of the quality of each of the proposed experts. Various methods are used for this purpose:

assessment of candidates for experts on the basis of statistical analysis of the results of past activities as experts on I problems of the study of SU;

collective assessment of the candidate for expert as a specialist in this field

self-assessment of a candidate for expert;

analytical determination of the competence of candidates for experts.

However, all these methods have certain disadvantages, including: the lack of a single generally recognized assessment methodology; high complexity of the assessment; the emergence of ethical problems when using subjective assessment methods.

In the course of this work, several methods are often used simultaneously: self-assessment and collective assessment of the qualities of the proposed expert. This approach makes it possible to reasonably select experts with the necessary qualities. However, it should be recognized that the method of assessing past performance seems to be more objective than the methods of self-assessments and collective assessments.

In general, the formation of an expert group is preceded by the following activities:

the problem is identified and formulated;

the purpose and scope of the group's activities are determined;

a preliminary list of experts is drawn up;

analysis and selection of experts is carried out (based on the use of one or more methods for selecting them);

the list of experts is specified; . the consent of the expert to participate in the work of the expert group is obtained;

a final representative list of experts is determined. All potential experts, depending on their quality and competence, can be classified into seven classes

An example of the gradation of quality and competence of experts.

The choice of the number of expert quality classes in this case is due to the "rule of seven", which is traditionally used in solving quality management problems.

This gradation makes it possible to select the required experts to work in the expert group. To obtain sufficiently objective results of the study of SU, it is desirable to select from among experts belonging to the 1st-4th quality classes. Candidates for experts of lower quality classes should not be involved in examinations.

Regardless of the chosen method of assessing the qualities of candidates, experts must in all cases meet certain requirements, including:

* professional competence and practical and research experience in the field of management;

* creativity (ability to solve creative problems); . scientific intuition;

Interest in the objective results of expert work;

* independence of judgment;

* Efficiency "discipline" the ability to switch from one type of activity to another, communicativeness, independence of judgment, motivation of actions);

* objectivity;

* non-conformism;

* high general erudition.

Conducting the collection of expert opinions involves determining: the place and time of the collection of opinions; forms and methods of collecting opinions; the number of rounds of opinion gathering; the composition and content of the documentation; the procedure for entering the results of expert opinions into documents.

It is very important to determine the form of collecting expert opinions. Among all known forms of collecting opinions, one can note individual, collective (group) and mixed. Thus, these forms differ primarily in terms of the participation of experts in the work (individual or collective) and each of them has a number of varieties:

* questioning;

* interviewing;

* discussion;

* brainstorm

* meeting;

* business game.

All of them have their own advantages and disadvantages. In many cases, each of these varieties is used in conjunction with others, which often provides greater effect and objectivity. Is the mixed form used when collecting expert opinions in cases of some ambiguity of the problem, in case of disagreement? individual opinions or disagreements of experts in a collective discussion.

After conducting a survey of a group of experts, the results are processed. The initial information for processing is the numerical data expressing the preferences of the experts and the substantive justification for these preferences. The purpose of processing is to obtain generalized data and new information contained in a hidden form in expert assessments. Based on the processing results, a solution to the problem is formed.

The presence of both numerical data and meaningful statements of experts leads to the need to apply qualitative and quantitative methods for processing the results of group expert evaluation. The share of these methods essentially depends on the class of problems solved by expert evaluation.

The whole set of problems can be divided into two classes. The first class includes problems for the solution of which there is a sufficient level of knowledge and experience, that is, there is the necessary information potential. When solving problems belonging to this class, experts are considered as good average measurers. The term "good on average" refers to the possibility of obtaining measurement results that are close to true. For many experts, their judgments cluster around the true value. It follows that for processing the results of group expert evaluation of problems of the first class, one can successfully apply the methods of mathematical statistics based on data averaging.

The second class includes problems for the solution of which sufficient information potential has not yet been accumulated. In this regard, the opinions of experts can vary greatly from each other. Moreover, the judgment of one expert, which is very different from the rest of the opinions, may turn out to be true. Obviously, the use of methods for averaging the results of a group expert assessment in solving problems of the second class can lead to large errors. Therefore, the processing of the results of a survey of experts in this case should be based on methods that do not use the principles of averaging, but on methods of qualitative analysis.

Considering that the problems of the first class are the most common in the practice of peer review, the focus of this chapter is on the methods of processing the results of the review for this class of problems.

Depending on the goals of expert assessment and the chosen measurement method, the following main tasks arise when processing survey results:

1) building a generalized assessment of objects based on individual assessments of experts;

2) building a generalized assessment based on a paired comparison of objects by each expert;

3) determination of the relative weights of objects;

4) determining the consistency of expert opinions;

5) determination of dependencies between rankings;

6) assessment of the reliability of the processing results.

The task of constructing a generalized assessment of objects based on individual assessments of experts arises in group expert assessment. The solution to this problem depends on the measurement method used by the experts.

When solving many problems, it is not enough to arrange objects according to one indicator or some set of indicators. It is desirable to have numerical values ​​for each object, indicating its relative importance compared to other objects. In other words, for many problems it is necessary to have estimates of objects that not only carry out their ordering, but also allow one to determine the degree of preference of one object over another. To solve this problem, you can directly apply the method of direct evaluation. However, under certain conditions, the same problem can be solved by processing expert estimates.

The determination of the consistency of expert opinions is carried out by calculating a numerical measure that characterizes the degree of similarity of individual opinions. Analysis of the value of the measure of consistency contributes to the development of a correct judgment about the general level of knowledge on the problem being solved and the identification of groupings of expert opinions. A qualitative analysis of the reasons for grouping opinions makes it possible to establish the existence of different views and concepts, to identify scientific schools, to determine the nature of professional activity, etc. All these factors make it possible to more deeply comprehend the results of a survey of experts.

By processing the results of expert evaluation, it is possible to determine the dependencies between the rankings of various experts and thereby establish the unity and difference in the opinions of experts. An important role is also played by the establishment of the relationship between the rankings built on various indicators of comparison of objects. The identification of such dependencies allows one to reveal related comparison indicators and, perhaps, to group them according to the degree of connection. The importance of the task of determining dependencies for practice is obvious. For example, if the comparison indicators are different goals, and the objects are the means to achieve the goals, then establishing the relationship between the rankings that order the means in terms of achieving the goals allows you to reasonably answer the question of the extent to which the achievement of one goal with these means contributes to the achievement of other goals. .

Estimates obtained on the basis of processing are random objects, so one of the important tasks of the processing procedure is to determine their reliability. Appropriate attention should be paid to the solution of this problem.

Processing the results of the examination is a time-consuming process. Performing manual calculations of estimates and indicators of their reliability is associated with large labor costs, even in the case of solving simple ordering problems. In this regard, it is advisable to use computer technology and especially computers. The use of computers raises the problem of developing computer programs that implement algorithms for processing the results of expert evaluation.

2. Methods of expert assessments

SWOT analysis

A special kind of expert method, which is very popular, is the original method of SWOT analysis. It got its name from the first letters of four English words, which in Russian translation mean: Strengths and Weaknesses, Opportunities and Threats.

This methodology can be used as a universal one. It has a special effect in the study of processes in the socio-economic system, which is characterized by dynamism, controllability, dependence of internal and external factors of functioning, cyclical development.

According to the methodology of this analysis, the distribution of factors characterizing the subject of research is carried out according to these four components, taking into account whether this factor belongs to the class of external or internal factors.

As a result, a picture of the correlation of strengths and weaknesses, opportunities and dangers appears, which suggests how the situation should be changed in order to have development success.

The allocation of factors to these quadrants or sectors of the matrices is not always easy. It happens that the same factor simultaneously characterizes both the strengths and weaknesses of the subject. In addition, factors act situationally. In one situation, they look like a virtue, in another - a disadvantage. Sometimes they are disproportionate in their significance. These circumstances can and should be taken into account.

The same factor can be placed in several quadrants if it is difficult to unambiguously determine its place. This will not adversely affect the study. After all, the essence of the method is to identify factors, place them in such a way that their concentration suggests ways to solve the problem, so that they become manageable.

In each quadrant, the factors do not have to have the same weight, but they must be presented in their entirety.

The completed matrix shows the real state of affairs, the state of the problem and the nature of the situation. This is the first stage of the SWOT analysis.

The second step is to conduct a comparative analysis of strengths and opportunities, which should show how to use the strengths. At the same time, it is necessary to analyze the weaknesses in relation to the existing dangers. Such an analysis will show how likely a crisis is. After all, the danger increases when it arises in conditions of weakness, when the weak sides do not make it possible to hinder the danger.

Of course, it is very useful to make a comparative analysis of strengths and existing dangers. After all, strengths can be poorly used in preventing a crisis, strengths must be seen not only in relation to favorable opportunities, but also in relation to dangers.

In the study of control systems, the subject of this method can be various problems of control development. For example, efficiency, personnel, style, distribution of functions, structure of the management system, management mechanism, motivation, professionalism, information support, communications and organizational behavior, etc.

The use of specially trained and selected experts or internal consultants makes this method more effective.

SMART Method

There are many modifications of the SWOT analysis method. The most interesting of them is the method of development and analysis of goals.

It is known that the goal of management is a decisive factor in success, efficiency, strategy and development. Without a goal, it is impossible to develop a plan or program. But this concerns not only the goal of management, but also the goal of research. After all, it is also not easy to formulate this goal correctly. The research program, the use of research methods depend on the purpose.

The goal should be developed according to the criteria of achievability, specificity, evaluability (measurability), taking into account the Place and Time. These criteria reflect English words-- Specific, Measurable, Achievable, Relevant, Timed, abbreviated as SMART. That's what this method is called.

The method assumes a consistent assessment of goals according to a set of criteria arranged in a matrix form. Here is a set of comparable factors that reflect the characteristics of the goal: difficult to achieve - easy to achieve, high costs - low costs, has staff support - does not have staff support, has priorities - does not have priorities, takes a lot of time - takes little time, has a wide impact -- has limited influence, high technology oriented - low (conventional) technology oriented, linked to new management organization -- not connected to new management organization.

The next step is to create a problem definition matrix. To achieve the goal, a number of problems must be solved. But for this they must first be defined.

The distribution of problems is carried out according to the following criteria: the existing situation, the desired situation, the possibility of achieving the goal. These criteria characterize the horizontal of the matrix. The following criteria are considered along the vertical: problem definition, problem evaluation (quantitative parameters), organization of the solution (who, where, when), costs of solving the problem.

This matrix allows you to plan research.

Method of ranking and evaluation.

According to the method of ranks, the expert performs ranking (ordering) of the studied objects of the organizational system depending on their relative importance (preference), when the most preferred object is assigned rank 1, and the least preferred is the last rank, equal in absolute value to the number of ordered objects. More precise ordering occurs with a smaller number of objects of study, and vice versa.

With the preferred (by rank) arrangement of objects of expertise by one expert, the sum of ranks should be equal to the sum of the numbers of the entire natural series of the number of objects H, starting from one: H= (H+1): 2.

The resulting ranks of ranking objects according to survey data are determined as the sum of the ranks for each object. In this case, as a result, the first rank is assigned to the object that received the smallest sum of ranks, and the last - to the one with the largest sum of ranks, i.e. the least significant object (an example of determining the resulting rank of three objects by seven experts)

The more experts involved, the higher the objectivity of the evaluation result. However, the involvement of a large number of qualified experts and the high labor intensity of expert work increases the cost of quality assessments. Therefore, in order to reduce the complexity of the work of experts, the rank method is used, which provides only the ranking of indicators, and not their numerical determination by experts.

Nevertheless, this method is used in the practice of studying SU, despite its simplicity and low labor intensity, relatively. This is due to the large number of ranked research objects.

Method of direct assessment

It is an ordering of the objects under study (for example, when selecting parameters for compiling a parametric model) depending on their importance by assigning points to each of them. In this case, the most important object is assigned the highest number of points on the accepted scale (an assessment is given). The most common rating scale range is from 0 to 1; 0 to 5; 0 to 10; 0 to 100. In the simplest case, the score can be 0 or 1.

Sometimes assessment is done verbally. For example, “very important”, “important”, “unimportant”, etc., which is also sometimes translated into a point scale (respectively 3, 2, 1) for greater convenience in processing the survey results.

Direct assessment should be used with full confidence in the professional awareness of experts about the properties of the objects under study. According to the results of assessments, the rank and weight (importance) of each object under study are determined.

Conclusion

Currently, various methods of expert assessments are being increasingly used. They are indispensable in solving complex problems of evaluating and selecting technical objects, including those for special purposes, in analyzing and predicting situations with a large number of significant factors - wherever it is necessary to involve the knowledge, intuition and experience of many highly qualified experts.

Expert methods are continuously developed and improved. The main directions of this development are determined by a number of factors, among which one can point to the desire to expand the scope, increase the degree of use of mathematical methods and electronic computers, and also find ways to eliminate emerging shortcomings.

Despite the progress made in recent years in the development and practical use of the method of expert assessments, there are a number of problems and tasks that require further methodological research and practical verification. It is necessary to improve the expert selection system, increase the reliability of group opinion characteristics, develop methods for checking the validity of assessments, and study the hidden causes that reduce the reliability of expert assessments.

The basis of the expert assessment of the properties and business qualities of the candidate is based on the quantitative parameters and evaluation criteria obtained as a result of the interview. Although there are elements of convention and subjectivity here, however, with a good development of the rating scale and an attentive (professional) approach of experts, it is possible to evaluate the subjects with a high degree of reliability.

List of used literature

1. Grigorov V. M. Experts in the system of public production management // M .: Thought, 1976

2.Demidova A.V. Study of control systems. - M.: Prior-izdat, 2005. - 96 p.

3. Ignatieva A.V. Study of control systems. - M.: UNITI-DANA, 2003. - 157 p.

4. Kafidov V.V. Study of control systems. - M.: Academic Project, 2005. - 160 p.

5. Malin A.S. Study of control systems. - M.: GU VSHE, 2005. - 399 p.

6. Reylyan Ya. R. The basis for making managerial decisions // M .: Finance and statistics, 1989

7. Remennikov V.B. Development of a management solution. Proc. allowance. -- M.: UNITI-DANA, 2000.

8. Smolkin A.M. Management: foundations of the organization. -- M.: INFRA-M, 1999.

Hosted on Allbest.ru

Similar Documents

Problem solving, argumentation and the formation of quantitative estimates of results by formal methods. Components of the method of expert assessments. The method of collective generation of ideas ("brainstorming"). Delphi method, features of the focus group method, SWOT analysis.

presentation, added 03/30/2014


Essence and content, the main stages of expert analysis, the scope and features of its practical application, interpretation of the results. The degree of reliability of this examination. Application of the method of expert assessments to build a tree of goals.

term paper, added 02/25/2012


The concept and features of the application of expert technologies as an integral part of the process of preparing and making important management decisions. Studying the main stages of an expert survey. Selection of experts. Delphi method, PATTERN, brainstorming.

abstract, added 10/09/2016


The use of expert assessments. Application of different methods for solving one problem. Ranking, paired and multiple comparisons, direct evaluation, Thurstone's method are the most commonly used expert measurement procedures. Delphi type methods.

test, added 03/09/2011


The essence and types of expert assessments, the purpose of their use. The main stages of expert research. Characteristics of the methods of collective work of the expert group, as well as methods for obtaining an individual opinion. Processing the results of a survey of specialists.

abstract, added 04/03/2012


Characteristics of expert procedures: features of heuristic methods and models, methods of individual assessments, collective expert assessments. The specifics of the examination, the content and processing of the results. Expert assessment of the level of country risk.

abstract, added 05/10/2010


Methods for obtaining expert assessments. The problem of selection of experts. Normative documents regulating the activities of expert commissions. Decision making under risk and uncertainty. Tasks for decision-making under uncertainty.

control work, added 07/15/2010


Essence and types of decisions in the process of production management. The main requirements for the quality of management decisions. Methods for optimizing managerial decisions. Methods for optimizing decisions by methods of expert assessments.

term paper, added 05/08/2002


The study of development forecasting methods: extrapolation, balance, normative and program-target method. Study of the organization of the work of an expert, the formation of questionnaires and tables of expert assessments. Analysis of mathematical and statistical forecast models.

test, added 06/19/2011


Methodology and stages of system classification according to various signs. Drawing up questionnaires for obtaining expert assessments, their mandatory details and main questions. The essence and construction of the goal tree, the principles of its detailing. Methodology for evaluating complex systems.

The main idea of ​​forecasting based on expert estimates is to build rational procedure of intuitive-logical thinking of a person in combination with quantitative methods for evaluating and processing the results obtained.

The essence of expert assessment methods lies in the fact that the forecast is based on opinion specialist or team of specialists, based on professional, scientific and practical experience.

Individual expert assessments- are based on the use of the opinions of experts-specialists of the relevant profile.

1. Method "interview" involves a conversation between a forecaster and an expert according to the "question-answer" scheme, during which the forecaster, in accordance with a pre-developed program, puts questions to the expert regarding the prospects for the development of the predicted object. The success of such an assessment depends to a large extent on the ability of an expert to give an impromptu opinion on a wide variety of issues.

2. Questionnaire method consists in the fact that the expert is invited to fill out a questionnaire (questionnaire) containing a list of questions, each of which is logically related to the research task.

The following types of questions can be used in the questionnaire:

open - answers to these questions can be formulated in any form;

Closed type - answers are offered, one of which must be chosen by the expert.

The use of closed-type questions in the questionnaire is preferable, since it simplifies the statistical processing of the results of the answer and facilitates the work of the expert when filling out the questionnaire. On the other hand, the list of answers to a question may not contain the opinion of an expert. Therefore, when forming a list of answers to some questions, it should be possible for the expert to put forward his own answer or avoid answering

3. Analytical method(analytical notes) provides for a thorough independent work an expert on the analysis of trends, assessment of the state and development paths of the predicted object. An expert can use all the information he needs about the forecast object. He writes his findings in the form of a memorandum. The main advantage of this method is the possibility of maximum use of the individual abilities of the expert. However, it is not very suitable for predicting complex systems and developing a strategy due to the limited knowledge of one expert in related fields of knowledge.

The main advantage of the methods of individual expert assessments is the possibility of maximizing the use of the individual abilities of experts. However, these methods are not suitable for predicting the most general strategies due to the limited knowledge of one expert about the development of related fields of science and practice.

An example of the use of expert assessments in planning the development of socio-economic systems is the multi-criteria problem of choosing a solution option, which is currently relevant in many areas of human activity.

The multi-criteria selection procedure includes the following steps:

1. Identification of the most significant indicators (criteria) characterizing the object under study;

2. Determining how to quantify indicators;

3. Determination of acceptable limits for changing indicators;

4. Choice of the search method for the best option;

5. Solution of the problem and analysis of the results.

The additive convolution of criteria is most often used as an objective function for evaluating solution options:

Or , (2.18)

where are weight coefficients characterizing the significance of the criterion . Numerical values ​​are determined by experts, while it is desirable to comply with the following condition:

If the criteria have different units of measurement, then they must be reduced to a single dimensionless scale so that the following inequalities are satisfied:

Example . According to experts, the main indicators of economic and social development regions are:

Gross domestic (regional) product;

The level of employment of the population;

Average monthly salary.

An expert assessment of the significance of the criteria on a ten-point scale is presented in Table. 2.2.

The leadership of the region was offered four targeted programs for the development of the region, aimed at priority financing:

1. Agro-industrial complex;

2. Enterprises of the food industry;

3. Branches of the socio-cultural sphere;

4. Housing construction.

The expected values ​​of the main indicators obtained during the implementation of the target programs under consideration are given in Table. 2.3.

Table 2.2

Expert evaluation results

Table 2.3

Expected values ​​of the main socio-economic indicators of the region's development

It is necessary to determine the most appropriate program for the development of the region.

Solution:

Let's determine the values ​​of the weight coefficients:

; ; .

Thus, as a result of processing expert estimates, the objective function has the following form:

Taking into account that the target program No. 3 is obviously inefficient in comparison with the program No. 2 (1500<2000; 80=80; 1000<2000), удалим её из матрицы возможных решений:

Since the values ​​of indicators have different dimensions, they must be reduced to a single dimensionless scale. This is achieved by dividing the elements of each column by the maximum value in the column:

At the final stage, we determine the value of the objective function for the proposed programs:

The maximum value of the objective function corresponds to program No. 1. Therefore, the implementation of this program is the most appropriate.

The most reliable are collective expert assessments - involve determining the degree of agreement between experts' opinions on promising areas for the development of the forecasting object, formulated by individual specialists.

To organize expert assessments, working groups are created, whose functions include conducting a survey, processing materials and analyzing the results of a collective expert assessment. The working group appoints experts who provide answers to the questions raised regarding the prospects for the development of this object.

1. essence method of collective generation of ideas (brainstorming) consists in using the creative potential of specialists in brainstorming a problem situation, which first implements the generation of ideas, and then their structuring, analysis and criticism with the advancement of countermeasures and the development of a consistent point of view.

The method of collective generation of ideas involves the implementation of the following steps:

1. the formation of a group of participants in the "brainstorming" to solve a specific problem. The optimal group size is found empirically. Groups consisting of 10-15 people are recognized as the most productive.

2. The analysis team draws up a problem note, which formulates the problem situation and contains a description of the method and the problem situation.

3. The stage of generating ideas. Each participant has the right to perform many times. Criticism of previous speeches and skeptical remarks are not allowed. The facilitator corrects the process, welcomes an improvement or combination of ideas, provides support, freeing participants from constraint. Duration of "brainstorming" - not less than 20 minutes and not more than 1 hour, depending on the activity of the participants.

4. Systematization of ideas expressed at the generation stage. A list of ideas is formed, features are distinguished by which ideas can be combined, ideas are combined into groups according to the selected features.

5. At the fifth stage, the destructuring (destruction) of systematized ideas is carried out. Each idea is subjected to comprehensive criticism by a group of highly qualified specialists consisting of 20-25 people.

6. In the sixth step, criticisms are assessed and a list of practical ideas is compiled.

Method "635"- one of the varieties of "brainstorming". The numbers b, 3, 5 denote 6 participants, each of which must write down 3 ideas within 5 minutes. The leaf moves around. Thus, in half an hour everyone will write down 18 ideas in their asset, and all together - 108. The structure of ideas is clearly defined. Method modifications are possible. This method is widely used in foreign countries (especially in Japan) to select from a variety of ideas the most original and progressive in solving certain problems.

2. Method "Delphi". The purpose of the method is to develop a program of consecutive multi-round individual surveys. An individual survey of experts is usually carried out in the form of questionnaires. Then their statistical processing is carried out on a computer and the collective opinion of the group is formed, arguments in favor of various judgments are identified and generalized. The computer-processed information is communicated to experts, who can correct the estimates, explaining the reasons for their disagreement with the collective judgment. This procedure can be repeated up to 3-4 times. As a result, there is a narrowing of the range of estimates and a consistent judgment is made regarding the prospects for the development of the object.

Features of the "Delphi" method:

a) anonymity of experts - the interaction of group members when filling out questionnaires is completely excluded;
b) the possibility of using the results of the previous round of the survey;

c) a statistical characteristic of group opinion.

3. Method of "commissions"- based on the work of special commissions. Groups of experts at the "round table" discuss a particular issue in order to agree on points of view and develop a common opinion. The disadvantage of this method is that the group of experts in their judgments is guided mainly by the logic of compromise.

The method of expert commissions can be organized in one of the following forms:

As practice has shown, the "commission" method has significant drawbacks:

The great influence of such a psychological factor as the opinion of authoritative experts, to which other experts join without expressing their point of view;

The unwillingness of experts to publicly renounce their previously expressed opinions;

During the work of commissions, most often there is a dispute between two or three of the most authoritative experts, as a result of which other experts participate in the discussion or do not accept or take into account their opinions.

4. Court method - based on the organization of the work of a team of experts in the form of conducting a trial. The use of this method is advisable in the presence of several groups of experts, each of which defends its point of view. In this case, the object of forecasting acts as the “defendant”. Leaders of groups expressing alternative points of view act as both prosecution and defense (prosecutor, lawyer). Individual experts play the role of witnesses, providing the court with the information necessary to make a decision. The role of the judge is played by an interested person (a group of persons). So, for example, in the television program "The Trial", based on the use of the court method to analyze and predict the development of various socio-economic processes, the role of the judge was played by the audience, voting in the process of transmission by phone calls for the point of view that they supported.

Method of morphological analysis involves choosing the most appropriate solution to the problem from among the possible. It is advisable to use it when forecasting fundamental research. The method includes a number of techniques that involve a systematic consideration of the characteristics of the object. The study is carried out according to the "morphological box" method, which is built in the form of a tree of goals or a matrix, in the cells of which the corresponding parameters are entered. Serial connection of the first level parameter with one of the parameters of the subsequent levels is a possible solution to the problem. The total number of possible solutions is equal to the product of the number of all parameters presented in the "box", taken row by row. Through permutations and various combinations, it is possible to develop the probabilistic characteristics of objects.

Script writing method- based on the definition of the logic of the process or phenomenon in time under various conditions. It involves the establishment of a sequence of events that develop during the transition from the current situation to the future state of the object. The forecast scenario determines the development strategy of the forecast object. It should reflect the general goal of the development of the object, the criteria for evaluating the upper levels of the "tree of goals", the priorities of the problems and the resources to achieve the main goals. The scenario displays a consistent solution to the problem, possible obstacles. In this case, the necessary materials for the development of the forecasting object are used.

A predictive graph is a figure consisting of points-vertices connected by segments-edges. A "goal tree" is a tree graph that expresses the relationship between stage nodes or goal achievement problems. Each vertex is a target for all branches outgoing from it. "Tree of goals" involves the allocation of several structural or hierarchical levels.

Building a "tree of goals" requires solving many problems: forecasting the development of the object as a whole; formulating the scenario of the predicted goal, determining the levels and vertices of the "tree", criteria and their weights in the ranking of the vertices. These tasks can be solved, if necessary, by methods of expert assessments. It should be noted that this goal as an object of the forecast can correspond to many different scenarios.

The scenario usually has a multivariate character and highlights three lines of behavior: optimistic - the development of the system in the most favorable situation; pessimistic - the development of the system in the least favorable situation; working - the development of the system, taking into account the counteraction to negative factors, the appearance of which is most likely. As part of the forecast scenario, it is advisable to work out a backup strategy in case of unforeseen situations.

The finished script must be analyzed. Based on the analysis of information deemed suitable for the upcoming forecast, goals are formulated, criteria are determined, and alternative solutions are considered.