Today it's really too easy: you can walk up to a computer and with little or no knowledge of what you're doing, create sane and nonsense with truly amazing speed. (J. Box)

Expert assessments

Expert opinions often arise in practice, for example, when tasting food. In general, they are typical for sociological surveys, for example, a quality control manager conducts a survey of customers in a supermarket. Using expert method an order scale is often used to evaluate quality. The issue of comparison is being decided on the principle of "better worse", "more less". More detailed information about how many times better or worse is often not required.

Pairwise comparison (objects are compared with each other in pairs). When constructing a scale of order or the so-called ranked series, experts use the method pairwise matching. In tab. 1 shows an example of ranking six objects by pairwise comparison. This is the result of the work of one expert who evaluated objects in a certain way. The preference of one object over another is indicated by 1, the reverse situation is 0.

ranked row(order scale) for objects, the comparative assessment of which is given in tab. 1 will look like:

Q4

If you use several experts, you can get a more accurate result.

You can use more advanced criteria, for example, determine the advantage with a score of 1, determine the worst quality with a score of -1, and determine the equivalent quality with a score of 0. The mechanism for compiling a ranked series remains the same.

Psychologists have proven that pairwise comparison underlies any choice (that is, you choose products by comparing them in pairs), however, the order scale is often compiled in advance (not a ranked series) and reference (reference) points are fixed in it, which called points.

This is how the twelve-point earthquake intensity scale MSK - 64, the Mohs mineralogical scale, the five-point scale for assessing knowledge, points in figure skating, etc. appeared. Table 2 shows the Mohs hardness scale for minerals as an example.

Object number							Outcome

Table 1. Ranking of six objects by pairwise comparison

Table 2. Mohs scale

Each subsequent mineral leaves a scratch on the previous one, i.e. is more solid. The measurement results obtained by pairwise comparison can be refined by successive approximation.

Influence of the composition of experts on the results of the examination

When forming an expert group, it is advisable to conduct testing, mutual evaluation of experts and checking the consistency of opinions.

Testing consists in solving tasks by experts, with results known to the organizers of testing, but unknown to experts, and testing, using the Fisher criterion, the hypothesis that the assessments of different experts belong to the same general set of assessments.

Self-assessment consists in the fact that each expert in a limited time answers the questions of a specially compiled questionnaire. Such a test is carried out on a computer and then a score is obtained. Experts can also evaluate each other, but this requires a trusting environment and experience of working together. The consistency of expert opinion can be assessed by the value of the concordance coefficient:

where S- the sum of the squared deviations of all estimates of the ranks of each object of examination from the average value;
n- number of experts;
m- the number of objects of expertise.

Concordance coefficient varies in the range 0<W<1, причем 0 - полная несогласованность, 1 - полное единодушие.

Example

It is necessary to determine the degree of agreement between the opinions of five experts, the ranking results of which seven objects are given in Table 3. To determine the degree of agreement, a special measure is used - the Kendall concordance coefficient (from lat. concordare- align, arrange.

Examination object number	Expert assessment					Sum of ranks	Deviation from the mean	Deviation square

Table 3. Data for assessing the consistency of opinions of five experts

We estimate the arithmetic mean number of ranks:

Q cf = (21 + 15 + 9 + 28 + 7 + 25 + 35)/7 = 20.

Then we estimate the sum of squared deviations from the mean: S = 630. Determine the value of the concordance coefficient:

W = 12 * 630 / 25 * (343 - 7) = 0,9.

Is it a lot or a little? If we carry out the corresponding calculations in STATISTICS, you can get the following table of results:

Rice. 1. Results of the analysis in STATISTICA

From this table it follows that the different opinions of experts in this example are insignificant: p<0.00014.

Influence on the results of the examination of the quantitative composition of experts. With an increase in the number of experts in the group, the measurement accuracy increases, which is typical for multiple measurements.

Number of experts n, providing a given measurement accuracy, can be established knowing the law of distribution of expert opinions and the maximum allowable standard error of the estimate Sx. Then, using the well-known expression, we can determine the minimum number of experts n, providing the specified measurement accuracy:

where is the standard deviation, which is determined by the formula:

where is the arithmetic mean of expert assessments;
- the number of assessments given by experts.

There are also individual and collective opinions of experts, the latter is considered more accurate, and most importantly, agreed. People with special training, potential consumers and manufacturers of products can act as experts.

In most of the types of research considered, the subject of study is the point of view of consumers (private or corporate). However, there are also studies that are carried out taking into account all the factors that affect a particular market, such as competition, affiliation, general trends, changes in legislation, current and planned projects of players, industry regulation, risks, etc. And neither in publications, these studies are not included in industry statistics. Here, both desk research and consumer surveys can be used as an element, but the main tool in this case is expert interviews with market players, independent analysts, heads of associations, journalists, people involved in competitive intelligence, etc.

Peer review method - This is a type of research in which the respondents are experts - specialists in a particular field of activity.

The main purpose of the peer review method - identification of complex aspects of the problem under study, increasing the reliability of information, conclusions.

Distinctive feature of the method is that it involves the competent participation of experts (expertise) in the analysis and solution of research problems.

Expertise - procedure for obtaining information from experts. Expert assessments are expert judgments about various areas of human activity, involving a procedure for comparing objects and their properties according to selected criteria.

Specifics of the expert survey consists of the following.

There is no need to use indirect or control questions in the questionnaire.

The expert survey program is not detailed and is of a conceptual nature.

In the questionnaire, it is preferable to use open-ended questions with complete freedom to choose the form of the answer.

Basic regulatory requirements for peer review :

careful selection of experts;

assessment of the reliability of the information provided by the experts;

creating conditions for the productive use of experts in the course of the study;

taking into account the factors influencing the judgments of experts;

preservation of expert information without distortion at all stages of the study.

The quality and reliability of expert assessments is reduced by the random selection of experts.

Expert Selection Criteria are:

the degree of competence, the indicators of which can be the presence of an expert's academic degree, academic title, work experience in the specialty, official position, number of published works, etc.;

the ability to navigate the latest achievements of modern science in those areas that are the subject of expertise;

a combination of narrow specialization and the general outlook of an expert;

ability to analyze and synthesize the studied problems,

the ability to process and assimilate qualitatively new information;

high moral character;

a combination of psychologically acceptable to each other in a group of experts of different ages, different scientific schools, etc.

The expert group cannot be numerous. Expert selection methods single out objective - use of special selection techniques - and subjective - Involvement of potential experts themselves in the selection procedure.

The objective approach has two options :

a) documentary method - selection of experts on the basis of socio-demographic data.

b) experimental method - selection based on the testing of the candidate.

The subjective approach also has several variations. ;

a) attestation - the selection of experts is carried out with the help of open or secret voting of potential members of the future expert group (may be carried out in several rounds);

b) method of mutual evaluation in points or ranking;

c) a method of self-assessment of competence.

An expert survey can take different forms. :

one-time individual survey (questionnaire or interview);

one-time collective survey (meetings, brainstorming);

individual survey in several rounds (Delphian technique);

collective survey in several rounds (discussion, meeting, multi-stage selection).

Another form of expert survey - traditional discussion - a method of conversation with a small group of experts on the topic of research. The purpose of the discussion is to develop a group collective opinion. A prerequisite for a successful group discussion is a clear formulation of the subject of discussion and familiarity with it by the vast majority of participants.

To a large extent, the success of the discussion depends on the culture, preparation, conduct and design of the collective opinion of experts.

Referred valuation method - repetition of several cycles of discussion with the identification of the essence of the emerging disagreements and the gradual development of a common opinion of all or the majority of the participants in the examination, while those who disagree retain the right to a private opinion.

Examination methods in the collective work of its participants, they have many obvious advantages, but at the same time they also have a number of disadvantages. The main drawback is related to the influence of experts on each other. This shortcoming is overcome with the help of an individual survey in several rounds. The correspondence version of the "referred valuation" method has acquired the name delphi method, or Delphic technology (from the name of the ancient Greek city, which became famous as the center of oracle predictions).

The Delphi technique guarantees the anonymity of the respondents: experts do not meet each other, they fill out anonymous questionnaires or connect directly to work with a computer.

After the first round, the experts get acquainted with the final characteristics of the position of the group as a whole. In the second round, they get the opportunity either to bring their opinion closer to the position of the majority, or to study the reason for the deviation. In the third round, new information opens up the opportunity to reconsider your point of view once again.

Simplified variations of the Delphic technique (“mini delphi”) allow

collect expert opinions on 2-3 rounds in a few hours or days.

The disadvantages of the Delphi method are :

the complexity of preparing, conducting and processing the results,

relatively large investment of time and money.

Despite the shortcomings, the Delphic technique has received significant distribution; In terms of the scale of application in many countries, it was in the top five most popular methods of social forecasting.

An expert survey in its modern form is often based on the joint use of various methods, forms and procedures. So, based on the use of Delphic technology, one of the American forecasting systems, "Pattern", was built, which allows you to create a system of information models in the form of a tree of goals. The work on creating a system begins with the creation of a scenario, i.e., a description of the state and direction of development of the object under study. At the next stage, a goal tree is built, necessary and sufficient sub-goals are developed for each goal, which are a condition for achieving a common goal. At the third stage, the coefficients of the relative importance of criteria and goals at all levels are determined. Next, specific types of necessary work, resources and deadlines for their implementation are determined. The longest chain is the execution time of the entire complex of works.

Ministry of Education and Science of Russia

Mari State Technical University

Department of control and production of radio equipment

on the topic: Methods of expert assessments

Completed: Art. gr. BZD-41

Kopylova I.V.

Checked by Prof. cafe Cyprus

Skulkin N.M.

Yoshkar-Ola 2012

Introduction

Expert assessment

Stages of expert assessment

Types of expert assessments

Processing the results of the survey of experts

Conclusion

Bibliography

Introduction

Examples of expert assessment methods. How will the economic environment change over time? What will happen to the natural environment in ten years? How will the environment change? Will the environmental safety of industrial production be ensured, or will a man-made desert begin to spread around? It is enough to think about these natural questions, to analyze how we imagined the present day ten or even more than twenty years ago, in order to understand that there simply cannot be 100% reliable forecasts. Instead of statements with specific numbers, only qualitative assessments can be expected. Nevertheless, we engineers must make decisions, for example, about environmental and other projects and investments, the consequences of which will be felt in ten, twenty, and so on. years. How to be? It remains to turn to the methods of expert assessments. What are these methods?

1. Expert assessment

Expert assessment- the procedure for obtaining an assessment of the problem based on the opinion of specialists (experts) for the purpose of subsequent decision-making (choice).

Experts(from the Latin "expertus" - experienced) - these are persons who have knowledge and are able to express a reasoned opinion on the phenomenon under study.

Methods of expert assessments - these are methods of organizing work with expert specialists and processing expert opinions.

The essence of expert assessment methods lies in the fact that the forecast is based on the opinion of a specialist or a team of specialists based on professional, scientific and practical experience. There are individual and collective expert assessments.

Expert judgment is often used in selection, for example:

one variant of a technical device to be launched into a series of several samples,

groups of astronauts from many applicants,

a set of research projects for funding from a pool of applications,

recipients of environmental loans from many applicants,

when choosing investment projects for implementation among those presented, etc.

2. Stages of expert evaluation

1. Setting the goal of the study.

Expert evaluation involves the creation of a kind of mind with greater abilities compared to the capabilities of an individual. The source of the possibilities of multimind is the search for weak associations and assumptions based on the experience of an individual specialist. The expert approach allows you to solve problems that cannot be solved in the usual analytical way, including:

· Choosing the best solution among the available ones.

· Forecasting the development of the process.

· Finding possible solutions to complex problems.

Before starting an expert study, it is necessary to clearly define its purpose (problem) and formulate an appropriate question for the experts. In this case, it is recommended to adhere to the following rules:

· A clear definition of the conditions, time, external and internal constraints of the problem. * The ability to answer a question with human-experienced accuracy.

· The question is better formulated as a qualitative statement than as an estimate of a number. For numerical estimates, it is not recommended to set more than five gradations.

· Experts evaluate possible options, and one should not expect them to develop a complete plan of action, a detailed description of possible solutions.

2. Choice of research form, determination of the project budget.

Existing types of expert assessments can be classified according to the following criteria:

· According to the form of participation of experts: full-time, part-time. The face-to-face method allows experts to focus on the problem being solved, which improves the quality of the result, but the remote method can be cheaper.

· By the number of iterations (repetitions of the procedure to improve accuracy) - one-step and iterative.

· According to the tasks to be solved: generating solutions and evaluating options.

· By type of answer: ideological, ranking, evaluating the object in a relative or absolute (numerical) scale.

· According to the method of processing expert opinions: direct and analytical.

· By the number of involved experts: unlimited, limited. Usually 5-12 experts are used.

The most well-known methods of expert assessments are the Delphi method, brainstorming and the hierarchy analysis method. Each method has its own timing and need for experts. After choosing the method of expert evaluation, it is possible to determine the costs of the procedure, which include the payment of experts, rent of premises, purchase of stationery, payment of a specialist in conducting and analyzing the results of the examination.

3. Preparation of information materials, forms of questionnaires, procedure moderator.

Experts before making a judgment should diversify the presented problem. To carry out this procedure, it is necessary to prepare information materials describing the problem, available statistical data, reference materials, questionnaire forms, inventory. The following mistakes should be avoided: mentioning the developers of materials, highlighting one or another solution option, expressing the management's attitude towards the expected results. Data should be versatile and neutral. It is necessary to develop forms of questionnaires for experts in advance. Depending on the method, they can be with open and closed questions, the answer can be given in the form of a judgment, paired comparison, ranked series, in points or in the form of an absolute assessment.

The procedure itself is carried out by an independent moderator of the procedure, who controls compliance with the regulations, distributes materials and questionnaires, but does not express his own opinion.

4. Selection of experts.

The problem of selecting experts is one of the most difficult in the theory and practice of expert research. Obviously, as experts it is necessary to use those people whose judgments will most help to make an adequate decision. But how to identify, find, select such people? It must be said frankly that there are no methods for selecting experts that will surely ensure the success of the examination.

Two components can be distinguished in the problem of selecting experts - compiling a list of possible experts and choosing an expert commission from them in accordance with the competence of the candidates.

Experts should have experience in areas relevant to the tasks to be solved. When selecting experts, one should take into account the moment of personal interest, which can become a significant obstacle to obtaining an objective judgment. For this purpose, for example, Shar's methods are used, when one expert, the most respected specialist, recommends a number of others and further along the chain until the necessary team is selected.

5. Conducting an examination.

The procedure varies depending on the method used. General recommendations:

· Set and follow rules. Increasing the time to make a decision beyond the optimal one does not improve the accuracy of the answer.

6. Statistical analysis of results . After receiving the answers of the experts, it is necessary to evaluate them. This allows:

1) Assess the consistency of expert opinions. In the absence of significant agreement among experts, it is necessary to identify the causes of inconsistency (presence of groups) and recognize the absence of an agreed opinion (insignificant results).

) Estimate research error.

)Build a model of object properties based on the answers of experts (for analytical expertise). The results of expert evaluation are issued in the form of a report. The report indicates the purpose of the study, the composition of experts, the assessment received and the statistical analysis of the results.

7. Preparation of a report with the results of an expert assessment.

. Types of expert assessments

Methods of expert assessments can be divided into two groups:

§ methods of collective work of the expert group

§ methods of obtaining an individual opinion of the members of the expert group.

Methods of collective work of the expert group involve obtaining a common opinion in the course of a joint discussion of the problem being solved. These methods are sometimes referred to as direct collective opinion methods. The main advantage of these methods lies in the possibility of a comprehensive analysis of problems. The disadvantages of the methods are the complexity of the procedure for obtaining information, the complexity of forming a group opinion on the individual judgments of experts, the possibility of pressure from authorities in the group.

Teamwork methods include brainstorming, scripting, business games, meetings, and judgment.

Brain attack.It is organized as a meeting of experts, on whose speeches one, but very significant, restriction is imposed - one cannot criticize the proposals of others. You can develop them, you can express your ideas, but you can’t criticize! During the meeting, the experts, "infecting" each other, express more and more extravagant considerations. Two hours later, the session recorded on a tape recorder or video camera ends, and the second stage of brainstorming begins - the analysis of the ideas expressed. Usually, out of 100 ideas, 30 deserve further elaboration, out of 5-6 they make it possible to formulate applied projects, and 2-3 ultimately turn out to bring a beneficial effect - profit, increased environmental safety, improvement of the natural environment, etc. At the same time, the interpretation of ideas is a creative process. For example, when discussing the possibilities of protecting ships from a torpedo attack, the idea was expressed: "Line up the sailors along the side and blow on the torpedo to change its course." After elaboration, this idea led to the creation of special devices that create waves that knock the torpedo off course.

Method "635"- one of the varieties of brainstorming. The numbers 6, 3, 5 indicate six participants, each of which must write down three ideas within five minutes. The leaf moves around. Thus, in half an hour everyone will write down 18 ideas in their asset, and all together - 108. The structure of ideas is clearly defined. Method modifications are possible. This method is widely used in foreign countries (especially in Japan) to select from a variety of ideas the most original and progressive in solving certain problems.

business gamesare based on modeling the functioning of the social management system when performing operations aimed at achieving the goal. Unlike previous methods, where expert assessments are formed in the course of a collective discussion, business games involve the active work of an expert group, each member of which has a certain responsibility in accordance with pre-compiled rules and a program.

The main advantage of business games is the ability to develop a solution in dynamics, taking into account all stages of the process under study with the interaction of all elements of the public management system. The disadvantage lies in the complexity of organizing a business game in conditions close to a real problem situation.

Meeting method("commissions", "round table") - the most simple and traditional. It involves holding a meeting or discussion in order to develop a common collective opinion on the problem being solved. In contrast to the "brainstorming" method, each expert can not only express his opinion, but also criticize the proposals of others. As a result of such careful discussion, the possibility of errors in the development of a decision is reduced.

The advantage of the method is the simplicity of its implementation. However, at the meeting, the erroneous opinion of one of the participants may be adopted due to his authority, official position, perseverance or oratorical abilities.

Commission method- one of the methods of expert assessments, based on the work of special commissions. Groups of experts at the "round table" discuss a particular issue in order to agree on points of view and develop a common opinion. The disadvantage of this method is that the group of experts in their judgments is guided mainly by the logic of compromise.

Script writing methodis based on determining the logic of a process or phenomenon in time under various conditions. It involves the establishment of a sequence of events that develop during the transition from the current situation to the future state of the object. A peculiar scenario may be a description of the sequence and conditions for the international integration of the economies of countries, including the following questions:

from which simplest forms to more complex ones this process must pass;

how it will affect the national economy and economic relations of countries;

what are the financial, organizational, social, legal problems that may arise in the course of the internationalization of the economy.

The forecast scenario determines the development strategy of the forecast object. It should reflect the general goal of the development of the object, the criteria for evaluating the upper levels of the goal tree, the priorities of the problems and the resources to achieve the main goals. The scenario displays a consistent solution to the problem, possible obstacles. In this case, the necessary materials for the development of the forecasting object are used.

The scenario should be written in such a way that after reading it, the general goal of the work being carried out in the light of the socio-economic tasks for the forecast period becomes clear.

It is usually multivariate in nature and highlights three lines of behavior:

optimistic - the development of the system in the most favorable situation;

pessimistic - the development of the system in the least favorable situation;

working - the development of the system, taking into account the counteraction to negative factors, the appearance of which is most likely.

As part of the forecast scenario, it is advisable to work out a backup strategy in case of unforeseen situations.

The finished script must be analyzed. Based on the analysis of information deemed suitable for the upcoming forecast, goals are formulated, criteria are determined, and alternative solutions are considered.

court methodis a kind of method of "meetings" and is implemented by analogy with the conduct of the trial.

Chosen solutions act as "defendants";

in the role of "judges" - decision makers;

in the role of "prosecutors" and "defenders" - members of the expert group.

The role of "witnesses" is performed by various selection conditions and arguments of experts. When conducting such a "trial", certain decisions are rejected or made.

It is advisable to use the "court" method when there are several groups of experts adhering to different solutions.

Methods for obtaining an individual opinion of the members of the expert group are based on the preliminary receipt of information from experts interviewed independently of each other, with subsequent processing of the received data. These methods include questionnaire methods, interviews and Delphi methods. The main advantages of the method of individual expert evaluation are their efficiency, the ability to fully use the individual abilities of an expert, the absence of pressure from authorities, and the low cost of expertise. Their main disadvantage is the high degree of subjectivity of the estimates obtained due to the limited knowledge of one expert.

Delphi method.The goal is to develop a program of consecutive multi-round individual surveys. An individual survey of experts is usually carried out in the form of questionnaires. Then their statistical processing is carried out on a computer and the collective opinion of the group is formed, arguments in favor of various judgments are identified and generalized. The computer-processed information is communicated to experts, who can correct the estimates, explaining the reasons for their disagreement with the collective judgment. This procedure can be repeated up to 3-4 times. As a result, there is a narrowing of the range of estimates and a consistent judgment is made regarding the prospects for the development of the object. Features of the Delphi method:

a) anonymity of experts (members of the expert group are unknown to each other, the interaction of group members when filling out questionnaires is completely excluded);

b) the possibility of using the results of the previous round of the survey;

c) a statistical characteristic of group opinion.

This method helps to predetermine the development of long-term problematic situations. Our specialists working in the field of scientific and technical forecasting also develop methods for processing expert assessments. They are called heuristic.

Interview methodinvolves a conversation between a forecaster and an expert according to the question-answer scheme, during which the forecaster, in accordance with a pre-developed program, puts questions to the expert regarding the prospects for the development of the predicted object. The success of such an assessment depends to a large extent on the ability of an expert to give an impromptu opinion on various issues.

Analytical methodprovides for a thorough independent work of an expert on the analysis of trends, assessment of the state and development paths of the predicted object. An expert can use all the information he needs about the forecast object. He writes his findings in the form of a memorandum. The main advantage of this method is the possibility of maximum use of the individual abilities of the expert. However, it is not very suitable for predicting complex systems and developing a strategy due to the limited knowledge of one expert in related fields of knowledge.

. Processing the results of the survey of experts

expert collective individual survey

Let's pass to consideration of the procedures which are carried out at a stage of processing of results of interrogation.

On the basis of expert assessments, generalized information about the object (phenomenon) under study is obtained and a solution is formed, which is set by the purpose of the examination. When processing individual assessments of experts, various quantitative and qualitative methods are used. The choice of this or that method depends on the complexity of the problem being solved, the form in which the opinions of experts are presented, and the goals of the examination.

Most often, when processing survey results, methods of mathematical statistics are used.

Depending on the objectives of the examination, the following problems can be solved when processing assessments:

· formation of a generalized assessment;

· determination of the relative weights of objects;

· establishing the degree of agreement of opinions of experts, etc.

1)Formation of a generalized assessment

So, let a group of experts evaluate some object, then x j - assessment of the j-th expert, , where m is the number of experts.

To form a generalized assessment of a group of experts, average values ​​are most often used. For example, the median (M E ), for which such an estimate is taken, in relation to which the number of large estimates is equal to the number of smaller ones.

A point estimate for a group of experts can also be used, calculated as an arithmetic mean:

2)Determining the relative weights of objects

Sometimes it is required to determine how important (significant) one or another factor (object) is from the point of view of any criterion. In this case, we say that we need to determine the weight of each factor.

One of the methods for determining the weights is as follows. Let x ij - assessment of the factor i, given by the j-th expert, , , n - number of compared objects, m - number of experts. Then the weight of the i-th object, calculated according to the estimates of all experts (wi ), is equal to:

where w ij - the weight of the i-th object, calculated according to the estimates of the j-th expert, is equal to:

3)Establishing the degree of agreement among experts

If several experts participate in the survey, discrepancies in their assessments are inevitable, but the magnitude of this discrepancy is important. A group assessment can be considered sufficiently reliable only if there is good agreement between the answers of individual specialists.

To analyze the scatter and consistency of estimates, statistical characteristics are used - measures of scatter.

Variation range (R):

Xmax - x min ,

where x max- maximum assessment of the object; min - the minimum assessment of the object.

The standard deviation, calculated by the well-known formula:

where xj is the score given by the j-th expert; is the number of experts.

The coefficient of variation (V), which is usually expressed as a percentage:

Approaches to the consistency check used in the evaluation of objects by the ranking method are specific.

In this case, the result of the expert's work is a ranking, which is a sequence of ranks (for expert j): x 1j , x 2j, …, x nj .

Consistency between the rankings of two experts can be determined using the Spearman rank correlation coefficient:

where xij is the rank assigned to the i-th object by the j-th expert; ik is the rank assigned to the i-th object by the k-th expert; i is the difference between the ranks assigned to the i-th object.

The value can vary from -1 to +1. With complete coincidence of estimates, the coefficient is equal to one. The equality of the coefficient minus one is observed with the greatest divergence in the opinions of experts.

In addition, the calculation of the rank correlation coefficient can be used as a way to assess the relationship between any factor and the resulting feature (reaction) in cases where the features cannot be measured accurately, but can be ordered.

In this case, the value of the Spearman coefficient can be interpreted similarly to the value of the pair correlation coefficient. A positive value indicates a direct relationship between the factors, a negative value indicates a reverse one, while the closer the absolute value of the coefficient is to one, the closer the relationship.

When it is necessary to determine the consistency in the rankings of a large (more than two) number of experts, the so-called concordance coefficient is calculated - the overall rank correlation coefficient for a group of m experts:

Note that what is subtracted in parentheses is nothing more than the average sum of ranks (when summed for each object) received by i objects from experts.

The coefficient W varies in the range from 0 to 1. Its equality to one means that all experts assigned the same ranks to the objects. The closer the value of the coefficient to zero, the less consistent are the estimates of experts.

Conclusion

Experience, intuition, a sense of perspective, combined with information, help specialists choose the most important goals and directions of development more accurately, find the best options for solving complex scientific, technical and socio-economic problems in conditions where there is no information about solving similar problems in the past.

The use of the method of expert assessments helps to formalize the procedures for collecting, summarizing and analyzing the opinions of specialists in order to transform them into the most convenient form for making an informed decision. But, it should be noted that the method of expert assessments cannot replace either administrative or planning decisions, it only allows you to replenish the information necessary for the preparation and adoption of such decisions. The widespread use of expert assessments is justified only where it is impossible to apply more accurate methods to analyze the future.

Expert methods are continuously developed and improved. The main directions of this development are determined by a number of factors, among which one can point to the desire to expand the scope, increase the degree of use of mathematical methods and electronic computers, and also find ways to eliminate emerging shortcomings. Despite the progress made in recent years in the development and practical use of the method of expert assessments, there are a number of problems and tasks that require further methodological research and practical verification. It is necessary to improve the expert selection system, increase the reliability of group opinion characteristics, develop methods for checking the validity of assessments, and study the hidden causes that reduce the reliability of expert assessments. However, even today, expert assessments in combination with other mathematical and statistical methods are an important tool for improving management at all levels.

Bibliography

1Orlov A.I. Expert assessments. // Factory laboratory. ? 1996.? T. 62.? No. 1. ? pp. 54-60.

2Orlov A.I. Expert assessments. Proc. allowance. - M.: 2002.

Beshelev S.D., Gurvich F.G. Expert assessments in making planned decisions. Proc. allowance. - M.: Economics, 1976. ? 287 p.

Evlanov L.G., Kutuzov V.A. Expert assessments in management. - M.: Economics, 1978. ? 133 p.

Management. Proc. allowance. / Ed. J.V. Prokofieva. - M.: Knowledge, 2000. - 288 p.

Beshelev S.D., Gurvich F.G. Expert assessments. - M.: Nauka, 1973. - 79 p.

Statistical methods for the analysis of expert assessments. - M.: Nauka, 1977. - 384 p.

Moiseev N.N. Mathematical problems of system analysis. - M.: Nauka, 1981. - 487 p.

Litvak B.G. Expert assessments and decision making. - M.: Patent, 1996.

Characteristics of expert assessment methods [Electronic resource]: #"justify">Expert assessment. / Wikipedia. [Electronic resource]: #"justify">Expert assessments. // StatSoft: SPC Consulting. [Electronic resource]: http://www.spc-consulting.ru/app/expert.htm

Tutoring

Need help learning a topic?

Our experts will advise or provide tutoring services on topics of interest to you.
Submit an application indicating the topic right now to find out about the possibility of obtaining a consultation.

Expertise is an assessment obtained by asking the opinions of specialists. An expert (from the Latin e x p e r t u s - experienced) is a person who is knowledgeable in a particular field of activity, invited to solve an issue that requires special knowledge. Examination can be individual (when one specialist is involved in solving the problem) or group. Experts can express their opinion orally or fill out a special form. Experts are consulted whenever it is impossible or very difficult to make measurements using more accurate methods (V.M. Zatsiorsky, 1982; B.G. Litvak, 1996; V.S. Rubin, 2006; E.R. Yakhontov, 2006) .

For example, expertise is used in the following cases: a) when predicting a situation; b) when analyzing events for which there are no other methods of measurement; c) when justifying the adoption of a particular managerial decision under conditions of uncertainty (V.V. Muzychenko, 2003; N.N. Pilipenko, E.L. Tatarsky, 2007).

Conducting an examination includes the following main stages: the formation of its goal, the selection of experts, the choice of methodology, a survey of experts, processing the information received, including an assessment of the consistency of individual expert assessments. Expert assessments are divided into quantitative and qualitative (BG Litvak, 2002).

The selection of experts is the most important stage of the examination. The main requirements for experts are: competence, interest in the work of the expert commission, efficiency, breadth of views, objectivity and independence of judgment. The accuracy of the expert assessment largely depends on the number of experts. As practice shows, the optimal number of experts is 7-12 people (E.M. Korotkov, 2003). Expert evaluation can be carried out by using the following approaches: a) closed discussion followed by closed voting or filling in special expert forms; b) free speech without discussion and voting; c) open discussion of the issues raised, followed by open or closed voting.

There are many different ways to conduct a quality assessment. The simplest of these is called the preference (or ranking of alternatives) method. Using this method, the experts arrange the assessed objects in order of deterioration in their quality (Table 1).

Table 1. Form of an expert form compiled during the examination by the preference method

TYPE OF TOURISM	Ranking results
Expert number	Sum of points
Rustic
Wellness
Pilgrimage
Informative
Adventure
Entertaining
recreational
Sports
Exotic
Ecological

The place occupied by each object is determined by the number of points scored: the lower the sum of points, the higher the occupied place. For example, Table 1 presents the results of ranking ten types of tourism by six specialists (experts) according to the level of their attractiveness for Russian residents.

Often used and another method of examination - the method of paired comparison. In this case, the expert fills in a table in which all compared objects are indicated both horizontally and vertically (Table 2).

Table 2. An example of an expert form filled in by each expert when conducting an examination by the method of paired comparison

									Sum of points
Bulgaria
Germany

For example, it is necessary to determine the most popular country out of the eight presented, to which Russians would like to make a tourist trip. In Table 2, each cell refers to two compared objects, and one is put down in it to the one that, in the opinion of the expert, has a higher quality or is more important. The other of these compared objects is set to zero. Then the total number of points scored is calculated and the place (rank) of the object of examination is determined.

Evaluation of the results of the examination may have a more complex form. Further, as an example, we will give a phased examination aimed at "Selection of the head of a travel company."

Stage number 1. Filling out an expert form. As we have already established, each of the experts must fill out an expert form, which, in fact, is a preference matrix, and the columns and rows of this matrix are called the selected qualities (Table 3). The meaning of this filling is to compare all the qualities in turn with each other. In this case (i.e. in a mutual comparison), the more preferred quality is given 2 points, and the less preferred quality is 0 points. If it is impossible to give preference to any of the two compared qualities, each of them is given 1 point.

Table 3. Preference Matrix

So, in our example, at the first step, we begin to compare the qualities of the first line, i.e. First of all, we compare the “sociability” of the 1st line and the “obligation” of the 2nd line. Let, in the opinion of one of the experts, for the director of a travel company, whose qualities are compared with each other, “sociability” is preferable to “obligation”. Then the value 2 is entered in "cell 1.2", respectively, "mandatory" is assigned 0 points and this value is entered in "cell 2.1". Thus, at the first step, only the first row and, accordingly, the first column are filled in, and “automatically” depending on the chosen preferences for the first row. Next, the expert compares the qualities on the second line, i.e. compares "obligatory" with all other qualities. Moreover, it is no longer necessary to compare “sociability” and “obligation”, since this comparison has already been made at the first step. Similarly, all the remaining qualities are compared. In our example, the completed preference matrix will look like this (Table 3).

To check the correctness of filling in the preference matrix, you should pay attention to the fact that all elements relative to the main diagonal have mutual correspondence, that is, if the value 2 is written in “cell 1.2”, then the value 0 should be written in “cell 2.1”, respectively, if in “cell 1.3” the value 1 is written, then in “cell 3.1”, respectively, the value 1 should also be written, etc.

Stage number 2. Processing the preference matrix. At this stage, each expert starts processing the preference matrix. First of all, all the values ​​of the matrix cells by rows are summed up. In this way it is possible to calculate the total amount of points received by each alternative quality, i.e. essentially find out the absolute weight of each individual quality (V). Note that the maximum absolute weight of each quality () is equal to:

where N is the number of compared qualities.

In our example, this value is 14. After processing the preference matrix, it can be seen that such a quality as “sociability” has an absolute weight of 5, “obligation” - 9, “punctuality” - 4, etc.

Then it is necessary to determine the total absolute weight of all qualities of the preference matrix according to the formula:

In our example = 8 (8 - 1) = 56.

If several experts take part in the examination, then the average weight of each quality should be calculated using the formula:

where? = ? +? + … + ?; k - number of experts; and 1, 2 ... n is the current line number (quality serial number).

And finally, it is easy to calculate the relative weight of each quality using the formula:

= [ / N (N - 1)] ?100% if several experts participate in the examination, and

100% if one expert participates in the examination.

In our case, we have only one expert and, therefore, the value k=1. Hence, the relative weight corresponding to the quality of "sociability" is equal to (5/56) ? 100% = 8.9%; quality "obligation" - (9/56)? 100% = 16.2%; quality "punctuality" - (4/56) ? 100% = 7.1%; quality "poise" - (8/56) ? 100% = 14.3%; quality "work experience" - (5/56) ? 100% = 8.9%; quality "fairness" - (11/56) ? 100% = 19.6%; quality "competence" - (14/56) ? 100% = 25.0%.

Stage number 3. Analysis of the results of the examination. Knowing the relative weight of each quality, you can rank them, placing them in ascending order of importance. Another important question, which should also be answered using the results obtained, what are the qualities without which the candidate cannot take the proposed position under any circumstances? To find the boundary between the necessary and sufficient qualities, it is recommended to use the boundary coefficient equal to 4/3n (V.V. Muzychenko, 2003). So, if there are n qualities, then the boundary will pass by a weight equal to (4/3) n. In our case, this value is 11 and, therefore, such qualities as "competence" and "fairness" are necessary for the head of a travel company.

Of the more complex methods of conducting an examination, the “Method of Delphi”, “Method of brainstorming”, “Method 6.3.5.” and some other techniques.

Delphi method. Its name comes from the ancient Greek city of Delphi, where, according to legend, at the temple of Apollo in the period from the IX century. BC. according to IV century. AD there was a council of wise men ("the Delphic oracle"), famous for its predictions. The essence of the method lies in the development of agreed opinions by repeated repetition of an individual written survey of the same experts. After the first round of the survey, all answers are analyzed and brought to the attention of each expert in a consolidated form. Then, after each round, the survey data is processed again, and the results are reported to the experts, indicating the location of the assessments. The first round of the survey is conducted without argumentation. In the second, answers that differ from others are subject to argumentation, or the expert can change the assessment. After the assessments stabilize, the survey stops and the decision proposed by the experts is adopted (A. Durovich, L. Anastasova, 2002).

"Brainstorming" is one of the main ones in organizing and conducting an examination (B.G. Litvak, 1996; E.M. Korotkov, 2003; E.R. Yakhontov, 2006). Brainstorming usually consists of two rounds. In the first round, ideas are generated, and in the second round, the identified ideas are discussed, evaluated, and a collective point of view is developed.

The first round is held in such a way that each of the experts can freely express their opinion. Any point of view or idea expressed must be discussed and cannot be declared false. The main task of the first round is to get, perhaps, a more complete picture of the factors that may influence the development of the situation. In the second round, of the factors identified in the first round, only the most significant should be left. To do this, they need to be critically evaluated, so the experts participating in the second stage are divided into supporters and opponents of the opinion expressed. Supporters try to provide the necessary evidence in favor of the expressed point of view, and opponents try to refute them. Then, based on the results of the discussion, a final decision is made.

The method of qualitative expert assessments, as we see, has many options, one of which is the “6. 3.5". Its essence lies in the fact that 6 experts in 5 minutes offer 3 options for solving the problem under study. The experts write their answer on special forms (Table 4). Five minutes later, the next six experts are invited for the same procedure. Thus, in half an hour you can get 108 new proposals.

Table 4. An example of an expert form filled out during the examination using the “6. 3.5"

Researched problem
option number	Suggestions for solving the problem by the expert group
First expert of the group	Second expert		Fourth		group expert
Second option
Third option

Numerous proposals accumulated in a short time are then carefully analyzed, and expert experts draw conclusions and give specific proposals on the problem raised.

Sometimes the specifics of the objects of expert evaluation are such that experts can quantify individual indicators. In these cases, it is more justified to use the methods of quantitative assessment of objects of expertise. Among the methods for obtaining quantitative expert estimates, the most commonly used are "Direct Quantification", "Midpoint Method" and "Churchman-Akoff Method".

Direct quantification is used when it is necessary to determine the value of an indicator that is measured quantitatively. In this case, each of the experts directly indicates the value of the indicator for the assessed object. For example, the estimated capacity of the tourist market is estimated; the price of a unit of production at which it will have a competitive demand; optimal production volume; company value, etc. .

midpoint method. The method is used when there are a lot of alternative options that can be quantified. In this case, if through f () we denote the assessment of the first alternative, and through f () - the assessment of the second alternative, then further the expert is invited to determine the third alternative, the assessment of which f () is located in the middle between the values ​​f () and f () and is equal to f () + f ()/2. Next, the expert indicates an alternative, the value of which is located in the middle between f () and f (), then the option, the value of which f () is located in the middle between the values ​​of f (and f (). The procedure is completed when the comparative preference of all alternatives participating in the examination is determined. options.

Churchman-Akoff method. This method is used in the quantitative assessment of the comparative preference of alternative options and allows for the adjustment of estimates given by experts. All alternative options are ranked by preference, and each of them is assigned quantitative estimates by the expert, as a rule, in fractions of one, while the total sum of the estimates of alternative options should be equal to 1 (or 100%). Next, the expert compares the assessment of the first alternative option f () and the sum of the remaining alternative options by preference. If the first option is preferable than the sum of the remaining alternative options, then it is excluded from further considerations. When less preferable than the sum of the remaining alternatives, it is compared with the sum of the alternatives except for the last one. If an alternative option at some step turns out to be preferable to the sum of other alternative options, then it is excluded from further considerations. This process continues until all alternatives have been sequentially reviewed, and the scores can be adjusted accordingly.

As we can see, tourism covers many areas of human life, and therefore the study of tourism is closely related to human psychology. Most fully, in our opinion, the need to apply methods of psychodiagnostics in tourism was substantiated in his work by M.B. Birzhakov (1999, p. 181): “Which travel direction will the tourist choose, which country will be preferable this season, what type of tourism will be the most popular? What is profitable to offer in the tourism market, where to direct your activity in promoting your tourism product? How to build a tour to best meet the wishes of the client? Many of these questions cannot be answered without studying human psychology and the prerequisites for motivating actions and decision-making.

Send your good work in the knowledge base is simple. Use the form below

Students, graduate students, young scientists who use the knowledge base in their studies and work will be very grateful to you.

Posted on http://www.allbest.ru/

MOSCOW SOCIO-ECONOMIC INSTITUTE

on the topic "Methodology for conducting expert assessments"

Students:

Artyushenko Yulia Viktorovna

Group: M10B-D-O-z

Moscow 2014

Introduction

2. Methods of expert assessments

Conclusion

Introduction

In the study of management, the method of expert assessments is widely used. This is due to the complexity of many problems, their origin from the "human factor", the lack of reliable experimental or normative tools.

It is undeniable that in order to make informed decisions, it is necessary to rely on the experience, knowledge and intuition of specialists. After the Second World War, within the framework of the theory of management (management), an independent discipline began to develop - expert assessments.

Methods of expert assessments are methods for organizing work with specialist experts and processing expert opinions expressed in quantitative and / or qualitative form in order to prepare information for decision-making by decision makers.

Many works have been devoted to the study of the possibilities and features of the application of expert assessments. They consider the forms of an expert survey (various types of questionnaires, interviews), assessment approaches (ranking, normalization, various types of ordering, etc.), methods for processing survey results, requirements for experts and the formation of expert groups, issues of training experts, assessments their competence (when processing the assessments, the coefficients of the competence of experts, the reliability of their opinions are introduced and taken into account), methods of organizing expert surveys. The choice of forms and methods for conducting expert surveys, approaches to processing survey results, etc. depends on the specific task and conditions of the examination.

Expert methods are now used in situations where the choice, justification and evaluation of the consequences of decisions cannot be performed on the basis of accurate calculations. Such situations often arise in the development of modern problems of managing social production and, especially, in forecasting and long-term planning. In recent years, expert assessments have been widely used in socio-political and scientific-technical forecasting, in the planning of the national economy, industries, associations, in the development of major scientific, technical, economic and social programs, in solving certain management problems. expert management ranking

1. Essence, methods and process of expert assessments

1.1 The essence of expert assessments

The possibility of using expert assessments, the justification of their objectivity is usually based on the fact that an unknown characteristic of the phenomenon under study is interpreted as a random variable, the reflection of the distribution law of which is an individual assessment of a specialist expert on the reliability and significance of an event. It is assumed that the true value of the characteristic under study is within the range of estimates received from the group of experts, and that the generalized collective opinion is reliable.

However, some theoretical studies question this assumption. For example, it is proposed to divide the problems for which expert assessments are used into two classes. The first class includes problems that are sufficiently well provided with information and for which the principle of a “good measurer” can be used, considering the expert as the custodian of a large amount of information, and the group opinion of experts is close to the true one. The second class includes problems in respect of which there is not enough knowledge to be sure of the validity of the above assumptions; experts cannot be considered as “good measurers”, and it is necessary to carefully approach the processing of the results of the examination, since in this case the opinion of one (single) expert, who pays more attention to the study of a little-studied problem, may turn out to be the most significant, and during formal processing it will be lost. In this regard, qualitative processing of results should be mainly applied to problems of the second class. The use of averaging methods (valid for "good meters") in this case can lead to significant errors.

The tasks of collective decision-making on the formation of goals, the improvement of methods and forms of management can usually be attributed to the first class. However, when developing forecasts and long-term plans, it is advisable to identify “rare” opinions and subject them to a more thorough analysis.

Another problem that needs to be kept in mind when conducting a system analysis is the following: even in the case of solving problems related to the first class, one should not forget that expert assessments carry not only narrowly subjective features inherent in individual experts, but also collective-subjective features that do not disappear when processing the results of the survey (and when using Delphi procedures, they can even be enhanced). In other words, expert assessments should be viewed as some kind of “public point of view”, depending on the level of scientific and technical knowledge of the society regarding the subject of research, which can change as the system and our ideas about it develop. Therefore, an expert survey is not a one-time procedure. This way of obtaining information about a complex problem characterized by a high degree of uncertainty should become a kind of "mechanism" in a complex system, i.e. it is necessary to create a regular system of work with experts.

Attention should also be paid to the fact that the use of the classical frequency approach to assessing probability when organizing expert surveys can be difficult, and sometimes impossible (due to the impossibility of proving the legitimacy of using a representative sample). Therefore, at present, studies are underway on the nature of the probability of expert assessment, based on the theory, fuzzy sets of Zadeh, on the idea of ​​expert assessment as a degree of confirmation of a hypothesis or as a probability of achieving a goal. One of the varieties of the expert method is the method of studying the strengths and weaknesses of the organization, the opportunities and threats to its activities - the method of SWOT analysis.

The collection of expert information depends on the choice of the method of expert assessments. Usually, to collect expert information, special documents are compiled, for example, questionnaires approved by the relevant managers and then sent to the experts.

Processing of expert information is carried out using the chosen method, usually with the use of computer technology. The data obtained as a result of processing is analyzed and used to solve the problems of analysis and synthesis of control systems.

Expert assessments are used for analysis, diagnosis of the state, subsequent prediction of development options:

1) objects, the development of which is either completely or partially not amenable to subject description or mathematical formalization;

2) in the absence of sufficiently representative and reliable statistics on the characteristics of the object;

3) in conditions of great uncertainty in the environment for the functioning of the object, the market environment;

4) in medium- and long-term forecasting of new markets, objects of new industries that are strongly influenced by discoveries in the fundamental sciences (for example, the microbiological industry, quantum electronics, nuclear engineering);

5) in cases where either the time or the funds allocated for forecasting and decision-making do not allow to investigate the problem using formal models;

6) there are no necessary technical means of modeling, for example, computer technology with the appropriate characteristics;

7) in extreme situations.

The tasks solved in the process of expert assessments of control systems can be divided into two groups:

1) tasks of synthesis of new control systems and their evaluation;

2) tasks of analysis (measurement) of existing management systems according to selected indicators and performance criteria.

The tasks of the first group include: formation of the image of the system being created; forecasting technical and economic indicators of the stages of its life cycle; substantiation of the main directions of the reorganization of the social management system; selection of optimal or satisfactory methods of action and outcomes using the created control system, etc. Some of the expert information obtained in the course of solving these problems is of a qualitative nature and is formed in the form of complex judgments in a descriptive form. However, the tasks of synthesis solved with the help of expert assessments can be quantitative in nature, and their solution will be associated with the justification of numerous parameters (characteristics) of the system being created. The tasks of the second group include all the tasks of evaluating existing or created variants of control systems using specified indicators and performance criteria. Examples of such tasks are: determining the structural, functional or informational characteristics of the system; evaluation of its effectiveness in the course of performing various operations; determination of the expediency of further operation of technical means of control and communication, etc.

1.2 The role of experts in management

Expertise is an opinion, idea, decision or assessment based on the implementation of the valuable experience of a specialist, deep knowledge of the subject of research and qualitative analysis technologies.

Expertise can be individual or group. In group expertise, the selection of a group of experts and the methodology for the final processing of the results of its work are of great importance.

The expert opinion is a document that records the course of the study and its results. At the same time, the conclusions and opinions of experts can have both categorical ("yes", "no"), and probabilistic (in the form of an assumption, ranking, preference coefficient, etc.) form.

In organizing the work of experts, it is necessary to adhere to the following principles:

1. Ideas, opinions and assessments should fit into a pre-prepared scheme. This allows you to generalize, compare, highlight the essential, etc. Such a scheme should not constrain thought and limit fantasy. The scheme may allow and assume the possibility of its modification and addition.

2. The processing of expert opinions must be carried out not only in quantitative generalization, but also through qualitative analysis, highlighting the main, essential, important, relevant, original, new, etc. Expert opinion can be the subject of examination of the second stage.

3. Experts must be independent, i.e. freed from any organizational or conceptual, as well as psychological restrictions. In this case, their experience, knowledge and intuition are realized in the best way.

4. The work of the expert group should be purposeful. Understanding why and why an examination is carried out is an important element of its implementation. In many cases, special training of experts is needed, which plays the role of mobilizing efforts and intelligence.

5. There are various forms of organizing the work of an expert group: either each expert makes an examination individually, then the results are summarized and systematized, or the experts work collectively, interacting with each other.

6. Parallel and multi-stage work of several expert groups is possible. Comparison of expertise provides important information.

There are many methods for obtaining expert assessments. In some, they work with each expert separately, he does not even know who else is an expert, and therefore expresses his opinion regardless of the authorities. In others, experts are brought together to prepare materials for the decision maker, while the experts discuss the problem with each other, learn from each other, and incorrect opinions are discarded. In some methods, the number of experts is fixed and such that statistical methods for checking the consistency of opinions and then averaging them allow making informed decisions. In others, the number of examiners grows during the course of the examination, for example, when using the "snowball" method.

A specialist or a group of specialists acting as experts is sometimes identified with a measuring device that has random and systematic measurement errors.

Random errors are due to the subjectivity of expert opinions on the issue under consideration and may deviate in one direction or another from the true value. The impact of such errors is reduced by averaging a sufficient number of estimates.

A systematic error is inherent in the entire team of experts and cannot be eliminated by processing the obtained estimates. This suggests that in some cases it is necessary to approach the results of an expert survey very carefully, which can sometimes express a generally erroneous point of view, depending on the level of knowledge and beliefs of experts.

1.3 Peer review process

The main stages of the peer review process include:

Formation of the goal and objectives of expert assessment;

Formation of a management group and execution of a decision to conduct an expert assessment;

Choosing a method for obtaining expert information and methods for its processing;

Selection of an expert group and formation, if necessary, of survey questionnaires;

Survey of experts (expertise);

Processing and analysis of the results of the examination;

Interpretation of the obtained results;

Compilation of a report.

The task of conducting an expert assessment is set by the decision maker. The stage of forming the goal and objectives of expert evaluation is the main one. The reliability of the result obtained and its pragmatic value depend on it. The formation of the goal and objectives of expert evaluation is dictated by the essence of the problem being solved. Here, the following factors should be taken into account: the reliability and completeness of the available initial information, the required form of presenting the result (qualitative or quantitative), the possible areas of use of the information received, the timing of its submission, the resources available to management, the possibility of attracting specialists from other fields of knowledge, and much more. The task is formalized in the form of a guiding document (for example, a decision to conduct an expert assessment).

To prepare the decision and guide all further work, the head of the examination is appointed. It defines the composition of the management group. The control group provides feedback to experts or the Delphi method.

The management group is entrusted not only with all organizational and planning work to provide favorable conditions for the effective creative activity of experts, but also with analytical work on the selection of an expert group, determining methods for obtaining and processing information, compiling questionnaires - questionnaires, meaningful interpretation of the results.

This large and complex range of tasks to be solved requires the inclusion of highly qualified specialists in the management group both in the field of the problem under consideration and in other areas - psychology, mathematics, medicine, sociology.

The selection of specific experts is carried out on the basis of an analysis of the quality of each of the proposed experts. Various methods are used for this purpose:

assessment of candidates for experts on the basis of statistical analysis of the results of past activities as experts on I problems of the study of SU;

collective assessment of the candidate for expert as a specialist in this field

self-assessment of a candidate for expert;

analytical determination of the competence of candidates for experts.

However, all these methods have certain disadvantages, including: the lack of a single generally recognized assessment methodology; high complexity of the assessment; the emergence of ethical problems when using subjective assessment methods.

In the course of this work, several methods are often used simultaneously: self-assessment and collective assessment of the qualities of the proposed expert. This approach makes it possible to reasonably select experts with the necessary qualities. However, it should be recognized that the method of assessing past performance seems to be more objective than the methods of self-assessments and collective assessments.

In general, the formation of an expert group is preceded by the following activities:

the problem is identified and formulated;

the purpose and scope of the group's activities are determined;

a preliminary list of experts is drawn up;

analysis and selection of experts is carried out (based on the use of one or more methods for selecting them);

the list of experts is specified; . the consent of the expert to participate in the work of the expert group is obtained;

a final representative list of experts is determined. All potential experts, depending on their quality and competence, can be classified into seven classes

An example of the gradation of quality and competence of experts.

The choice of the number of expert quality classes in this case is due to the "rule of seven", which is traditionally used in solving quality management problems.

This gradation makes it possible to select the required experts to work in the expert group. To obtain sufficiently objective results of the study of SU, it is desirable to select from among experts belonging to the 1st-4th quality classes. Candidates for experts of lower quality classes should not be involved in examinations.

Regardless of the chosen method of assessing the qualities of candidates, experts must in all cases meet certain requirements, including:

* professional competence and practical and research experience in the field of management;

* creativity (ability to solve creative problems); . scientific intuition;

Interest in the objective results of expert work;

* independence of judgment;

* Efficiency "discipline" the ability to switch from one type of activity to another, communicativeness, independence of judgment, motivation of actions);

* objectivity;

* non-conformism;

* high general erudition.

Conducting the collection of expert opinions involves determining: the place and time of the collection of opinions; forms and methods of collecting opinions; the number of rounds of opinion gathering; the composition and content of the documentation; the procedure for entering the results of expert opinions into documents.

It is very important to determine the form of collecting expert opinions. Among all known forms of collecting opinions, one can note individual, collective (group) and mixed. Thus, these forms differ primarily in terms of the participation of experts in the work (individual or collective) and each of them has a number of varieties:

* questioning;

* interviewing;

* discussion;

* brainstorm

* meeting;

* business game.

All of them have their own advantages and disadvantages. In many cases, each of these varieties is used in conjunction with others, which often provides greater effect and objectivity. Is the mixed form used when collecting expert opinions in cases of some ambiguity of the problem, in case of disagreement? individual opinions or disagreements of experts in a collective discussion.

After conducting a survey of a group of experts, the results are processed. The initial information for processing is the numerical data expressing the preferences of the experts and the substantive justification for these preferences. The purpose of processing is to obtain generalized data and new information contained in a hidden form in expert assessments. Based on the processing results, a solution to the problem is formed.

The presence of both numerical data and meaningful statements of experts leads to the need to apply qualitative and quantitative methods for processing the results of group expert evaluation. The share of these methods essentially depends on the class of problems solved by expert evaluation.

The whole set of problems can be divided into two classes. The first class includes problems for the solution of which there is a sufficient level of knowledge and experience, that is, there is the necessary information potential. When solving problems belonging to this class, experts are considered as good average measurers. The term "good on average" refers to the possibility of obtaining measurement results that are close to true. For many experts, their judgments cluster around the true value. It follows that for processing the results of group expert evaluation of problems of the first class, one can successfully apply the methods of mathematical statistics based on data averaging.

The second class includes problems for the solution of which sufficient information potential has not yet been accumulated. In this regard, the opinions of experts can vary greatly from each other. Moreover, the judgment of one expert, which is very different from the rest of the opinions, may turn out to be true. Obviously, the use of methods for averaging the results of a group expert assessment in solving problems of the second class can lead to large errors. Therefore, the processing of the results of a survey of experts in this case should be based on methods that do not use the principles of averaging, but on methods of qualitative analysis.

Considering that the problems of the first class are the most common in the practice of peer review, the focus of this chapter is on the methods of processing the results of the review for this class of problems.

Depending on the goals of expert assessment and the chosen measurement method, the following main tasks arise when processing survey results:

1) building a generalized assessment of objects based on individual assessments of experts;

2) building a generalized assessment based on a paired comparison of objects by each expert;

3) determination of the relative weights of objects;

4) determining the consistency of expert opinions;

5) determination of dependencies between rankings;

6) assessment of the reliability of the processing results.

The task of constructing a generalized assessment of objects based on individual assessments of experts arises in group expert assessment. The solution to this problem depends on the measurement method used by the experts.

When solving many problems, it is not enough to arrange objects according to one indicator or some set of indicators. It is desirable to have numerical values ​​for each object, indicating its relative importance compared to other objects. In other words, for many problems it is necessary to have estimates of objects that not only carry out their ordering, but also allow one to determine the degree of preference of one object over another. To solve this problem, you can directly apply the method of direct evaluation. However, under certain conditions, the same problem can be solved by processing expert estimates.

The determination of the consistency of expert opinions is carried out by calculating a numerical measure that characterizes the degree of similarity of individual opinions. Analysis of the value of the measure of consistency contributes to the development of a correct judgment about the general level of knowledge on the problem being solved and the identification of groupings of expert opinions. A qualitative analysis of the reasons for grouping opinions makes it possible to establish the existence of different views and concepts, to identify scientific schools, to determine the nature of professional activity, etc. All these factors make it possible to more deeply comprehend the results of a survey of experts.

By processing the results of expert evaluation, it is possible to determine the dependencies between the rankings of various experts and thereby establish the unity and difference in the opinions of experts. An important role is also played by the establishment of the relationship between the rankings built on various indicators of comparison of objects. The identification of such dependencies allows one to reveal related comparison indicators and, perhaps, to group them according to the degree of connection. The importance of the task of determining dependencies for practice is obvious. For example, if the comparison indicators are different goals, and the objects are the means to achieve the goals, then establishing the relationship between the rankings that order the means in terms of achieving the goals allows you to reasonably answer the question of the extent to which the achievement of one goal with these means contributes to the achievement of other goals. .

Estimates obtained on the basis of processing are random objects, so one of the important tasks of the processing procedure is to determine their reliability. Appropriate attention should be paid to the solution of this problem.

Processing the results of the examination is a time-consuming process. Performing manual calculations of estimates and indicators of their reliability is associated with large labor costs, even in the case of solving simple ordering problems. In this regard, it is advisable to use computer technology and especially computers. The use of computers raises the problem of developing computer programs that implement algorithms for processing the results of expert evaluation.

2. Methods of expert assessments

SWOT analysis

A special kind of expert method, which is very popular, is the original method of SWOT analysis. It got its name from the first letters of four English words, which in Russian translation mean: Strengths and Weaknesses, Opportunities and Threats.

This methodology can be used as a universal one. It has a special effect in the study of processes in the socio-economic system, which is characterized by dynamism, controllability, dependence of internal and external factors of functioning, cyclical development.

According to the methodology of this analysis, the distribution of factors characterizing the subject of research is carried out according to these four components, taking into account whether this factor belongs to the class of external or internal factors.

As a result, a picture of the correlation of strengths and weaknesses, opportunities and dangers appears, which suggests how the situation should be changed in order to have development success.

The allocation of factors to these quadrants or sectors of the matrices is not always easy. It happens that the same factor simultaneously characterizes both the strengths and weaknesses of the subject. In addition, factors act situationally. In one situation, they look like a virtue, in another - a disadvantage. Sometimes they are disproportionate in their significance. These circumstances can and should be taken into account.

The same factor can be placed in several quadrants if it is difficult to unambiguously determine its place. This will not adversely affect the study. After all, the essence of the method is to identify factors, place them in such a way that their concentration suggests ways to solve the problem, so that they become manageable.

In each quadrant, the factors do not have to have the same weight, but they must be presented in their entirety.

The completed matrix shows the real state of affairs, the state of the problem and the nature of the situation. This is the first stage of the SWOT analysis.

The second step is to conduct a comparative analysis of strengths and opportunities, which should show how to use the strengths. At the same time, it is necessary to analyze the weaknesses in relation to the existing dangers. Such an analysis will show how likely a crisis is. After all, the danger increases when it arises in conditions of weakness, when the weak sides do not make it possible to hinder the danger.

Of course, it is very useful to make a comparative analysis of strengths and existing dangers. After all, strengths can be poorly used in preventing a crisis, strengths must be seen not only in relation to favorable opportunities, but also in relation to dangers.

In the study of control systems, the subject of this method can be various problems of control development. For example, efficiency, personnel, style, distribution of functions, structure of the management system, management mechanism, motivation, professionalism, information support, communications and organizational behavior, etc.

The use of specially trained and selected experts or internal consultants makes this method more effective.

SMART Method

There are many modifications of the SWOT analysis method. The most interesting of them is the method of development and analysis of goals.

It is known that the goal of management is a decisive factor in success, efficiency, strategy and development. Without a goal, it is impossible to develop a plan or program. But this concerns not only the goal of management, but also the goal of research. After all, it is also not easy to formulate this goal correctly. The research program, the use of research methods depend on the purpose.

The goal should be developed according to the criteria of achievability, specificity, evaluability (measurability), taking into account the Place and Time. These criteria reflect the English words - Specific, Measurable, Achievable, Relevant, Timed, in the abbreviated name it is SMART. That's what this method is called.

The method assumes a consistent assessment of goals according to a set of criteria arranged in a matrix form. Here is a set of comparable factors that reflect the characteristics of the goal: difficult to achieve - easy to achieve, high costs - low costs, has staff support - does not have staff support, has priorities - does not have priorities, takes a lot of time - takes little time, has a wide impact -- has limited influence, high technology oriented - low (conventional) technology oriented, linked to new management organization -- not connected to new management organization.

The next step is to create a problem definition matrix. To achieve the goal, a number of problems must be solved. But for this they must first be defined.

The distribution of problems is carried out according to the following criteria: the existing situation, the desired situation, the possibility of achieving the goal. These criteria characterize the horizontal of the matrix. The following criteria are considered along the vertical: problem definition, problem evaluation (quantitative parameters), organization of the solution (who, where, when), costs of solving the problem.

This matrix allows you to plan research.

Method of ranking and evaluation.

According to the method of ranks, the expert performs ranking (ordering) of the studied objects of the organizational system depending on their relative importance (preference), when the most preferred object is assigned rank 1, and the least preferred is the last rank, equal in absolute value to the number of ordered objects. More precise ordering occurs with a smaller number of objects of study, and vice versa.

With the preferred (by rank) arrangement of objects of expertise by one expert, the sum of ranks should be equal to the sum of the numbers of the entire natural series of the number of objects H, starting from one: H= (H+1): 2.

The resulting ranks of ranking objects according to survey data are determined as the sum of the ranks for each object. In this case, as a result, the first rank is assigned to the object that received the smallest sum of ranks, and the last - to the one with the largest sum of ranks, i.e. the least significant object (an example of determining the resulting rank of three objects by seven experts)

The more experts involved, the higher the objectivity of the evaluation result. However, the involvement of a large number of qualified experts and the high labor intensity of expert work increases the cost of quality assessments. Therefore, in order to reduce the complexity of the work of experts, the rank method is used, which provides only the ranking of indicators, and not their numerical determination by experts.

Nevertheless, this method is used in the practice of studying SU, despite its simplicity and low labor intensity, relatively. This is due to the large number of ranked research objects.

Method of direct assessment

It is an ordering of the objects under study (for example, when selecting parameters for compiling a parametric model) depending on their importance by assigning points to each of them. In this case, the most important object is assigned the highest number of points on the accepted scale (an assessment is given). The most common rating scale range is from 0 to 1; 0 to 5; 0 to 10; 0 to 100. In the simplest case, the score can be 0 or 1.

Sometimes assessment is done verbally. For example, “very important”, “important”, “unimportant”, etc., which is also sometimes translated into a point scale (respectively 3, 2, 1) for greater convenience in processing the survey results.

Direct assessment should be used with full confidence in the professional awareness of experts about the properties of the objects under study. According to the results of assessments, the rank and weight (importance) of each object under study are determined.

Conclusion

Currently, various methods of expert assessments are being increasingly used. They are indispensable in solving complex problems of evaluating and selecting technical objects, including those for special purposes, in analyzing and predicting situations with a large number of significant factors - wherever it is necessary to involve the knowledge, intuition and experience of many highly qualified experts.

Expert methods are continuously developed and improved. The main directions of this development are determined by a number of factors, among which one can point to the desire to expand the scope, increase the degree of use of mathematical methods and electronic computers, and also find ways to eliminate emerging shortcomings.

Despite the progress made in recent years in the development and practical use of the method of expert assessments, there are a number of problems and tasks that require further methodological research and practical verification. It is necessary to improve the expert selection system, increase the reliability of group opinion characteristics, develop methods for checking the validity of assessments, and study the hidden causes that reduce the reliability of expert assessments.

The basis of the expert assessment of the properties and business qualities of the candidate is based on the quantitative parameters and evaluation criteria obtained as a result of the interview. Although there are elements of convention and subjectivity here, however, with a good development of the rating scale and an attentive (professional) approach of experts, it is possible to evaluate the subjects with a high degree of reliability.

List of used literature

1. Grigorov V. M. Experts in the system of public production management // M .: Thought, 1976

2.Demidova A.V. Study of control systems. - M.: Prior-izdat, 2005. - 96 p.

3. Ignatieva A.V. Study of control systems. - M.: UNITI-DANA, 2003. - 157 p.

4. Kafidov V.V. Study of control systems. - M.: Academic Project, 2005. - 160 p.

5. Malin A.S. Study of control systems. - M.: GU VSHE, 2005. - 399 p.

6. Reylyan Ya. R. The basis for making managerial decisions // M .: Finance and statistics, 1989

7. Remennikov V.B. Development of a management solution. Proc. allowance. -- M.: UNITI-DANA, 2000.

8. Smolkin A.M. Management: foundations of the organization. -- M.: INFRA-M, 1999.

Hosted on Allbest.ru

Similar Documents

Problem solving, argumentation and the formation of quantitative estimates of results by formal methods. Components of the method of expert assessments. The method of collective generation of ideas ("brainstorming"). Delphi method, features of the focus group method, SWOT analysis.

presentation, added 03/30/2014


Essence and content, the main stages of expert analysis, the scope and features of its practical application, interpretation of the results. The degree of reliability of this examination. Application of the method of expert assessments to build a tree of goals.

term paper, added 02/25/2012


The concept and features of the application of expert technologies as an integral part of the process of preparing and making important management decisions. Studying the main stages of an expert survey. Selection of experts. Delphi method, PATTERN, brainstorming.

abstract, added 10/09/2016


The use of expert assessments. Application of different methods for solving one problem. Ranking, paired and multiple comparisons, direct evaluation, Thurstone's method are the most commonly used expert measurement procedures. Delphi type methods.

test, added 03/09/2011


The essence and types of expert assessments, the purpose of their use. The main stages of expert research. Characteristics of the methods of collective work of the expert group, as well as methods for obtaining an individual opinion. Processing the results of a survey of specialists.

abstract, added 04/03/2012


Characteristics of expert procedures: features of heuristic methods and models, methods of individual assessments, collective expert assessments. The specifics of the examination, the content and processing of the results. Expert assessment of the level of country risk.

abstract, added 05/10/2010


Methods for obtaining expert assessments. The problem of selection of experts. Normative documents regulating the activities of expert commissions. Decision making under risk and uncertainty. Tasks for decision-making under uncertainty.

control work, added 07/15/2010


Essence and types of decisions in the process of production management. The main requirements for the quality of management decisions. Methods for optimizing managerial decisions. Methods for optimizing decisions by methods of expert assessments.

term paper, added 05/08/2002


The study of development forecasting methods: extrapolation, balance, normative and program-target method. Study of the organization of the work of an expert, the formation of questionnaires and tables of expert assessments. Analysis of mathematical and statistical forecast models.

test, added 06/19/2011


Methodology and stages of classification of systems according to various criteria. Drawing up questionnaires for obtaining expert assessments, their mandatory details and main questions. The essence and construction of the goal tree, the principles of its detailing. Methodology for evaluating complex systems.