Statistics FAQ

Regarding (a): Our studies are not based on a random sample. A random sample would require that each element of the population has the same probability of being selected for the sample. Figuratively speaking, this would mean that the names of all board members, sponsors, and project managers who might have an interest in working with the consulting firms under consideration would have to be placed in a large lottery drum in order to randomly draw a sample from them. And the executives drawn, which is another prerequisite, would have to be obliged to actually participate in the survey. It is obvious that this cannot be guaranteed in practice. Sometimes it is possible to pre-sort the population according to certain characteristics – for example, if you know what proportion of the population works in a certain industry, has a certain age, a certain education or a certain gender. In these cases, you form proportional subgroups, for each of which you then draw a random sample – again with the above-mentioned prerequisites, which in the case of the executives we surveyed can hardly be met.

In colloquial terms, a study is sometimes referred to as representative if the sample is not taken at random, but is deliberately designed in such a way that it replicates the distribution of known characteristics in the overall population. For example, if one tries to represent the same percentage of DAX board members in the sample as one would expect in the overall population. Practically speaking, this basically means continuing to look for additional candidates in individual subgroups until the proportion in question is reached. In a strict statistical sense, this would not be called a representative sample.

Regarding (b): All of the above variants – whether strictly statistical or colloquial – will fail in our case anyway, because the elements of a population can only be randomly selected or sorted according to certain characteristics if the population is known in its entirety. This is the case, for example, when considering the students in a particular class, the enrolled attendees of an event, or the people registered in a particular city district. However, it would be almost impossible to compile a complete and accurate list of all managers who could in principle be considered for working with the consulting companies we have looked at. In such cases, attempts are often made to estimate the distribution of certain characteristics in the population as accurately as possible.

In the case of our studies, such a structural comparison proves difficult because – as explained in our comments on representativeness – it is hardly possible to accurately determine certain distributions of characteristics within the overall population. Nonetheless, attempts are repeatedly made to establish at least a good estimate of the relevant structures. In particular, this applies to the distribution of management consultants’ business with clients from different industries. Both the WGMB and the Bundesverband Deutscher Unternehmensberater (BDU) compile respective ratios. Even if these ratios refer to the German consulting market as a whole and therefore do not have to be congruent with the business of the leading management consultants, they are in our view the best available yardstick that can be used for a structural comparison. The following tables compare the share of respondents from different industries in our study from 2021 with the share of revenue generated by consultants in the respective industries – one according to the WGMB estimate (from 2020) and one according to the BDU estimate (for this purpose, we have translated the latest publicly available version from 2018):

So, does the structure of our sample reflect the structure of the overall population properly? This is a question that can only be answered by everyone for him or herself.

No. Due to the limited representativeness, it is not possible to infer the overall population from the results of our studies using statistical methods. At this point, we would like to briefly remind you that statistics is divided into two major subareas: descriptive statistics and inferential statistics. Our studies belong to the descriptive statistics. This means that we focus on describing the data we collect using various measures – such as the mean or the standard deviation. Inferential statistics, on the other hand, deals with the question to what extent such ratios, which were observed in a sample, can be generalized and transferred to the overall population. In the case of the data we are collecting, as noted above, such a generalization is not possible due to limited representativeness. This means, for example, that in our study from 2021 the statement “McKinsey, Bain and BCG are Germany’s best management consultants” must always be accompanied by an indication of whose viewpoint this evaluation is based on. In the present case, this is the view of the 1,063 executives we surveyed.

So, does this mean that the statements made in our studies are of no value due to the lack of statistical reference to the overall population? No. Firstly, they reflect the opinions of over 1,000 highly relevant decision-makers. Decision-makers who have all worked with at least one of the consulting firms surveyed at least once in the past three years. On average, the respondents have worked on projects with 3.5 of the consulting firms in question during this period, many of them more than once. Their experiences and their assessments are a value in themselves. Moreover, while many findings are not transferable to the population in the strict statistical sense, they can reveal important trends and tendencies.

Strictly speaking, a statement on significance cannot be made due to the limited representativeness of our studies. Significance means nothing other than that a result observed in a sample can be applied to the overall population with a certain probability of error – usually five percent. Such a statistical inference from the sample to the population is in fact not possible with our data. Although we point this out very clearly, we are often asked to check the distances between the consulting firms in a ranking for significance anyway. Even though we are fundamentally reluctant to do so, we would like to comply with this request and briefly present some results of the corresponding tests.

In our client satisfaction ranking from 2021, McKinsey is in first place with 402 points, followed by Bain with 388 points and BCG with 386 points. A t-test concludes that the gap between McKinsey and the other two firms is significant, while the gap between Bain and BCG is not. However, this result is questionable not only because of the lack of representativeness of the sample, but also because a t-test assumes a normal distribution of the data. At least in the case of Bain, this requirement is not met. If a representative sample were available, one could resort to so-called parameter-free test procedures in such cases. One such test is the Wilcoxon test. In our example, this test would produce the same result as the t-test.

No. The information provided by the participating executives is based on their experiences and assessments and is submitted in completely anonymous form; it reflects the personal opinions of the respondents. If they are value judgments, the answers cannot be classified as false or true; they can only correspond to or contradict the respondent’s own subjective point of view.

Information on whether respondents have worked with specific consulting firms in the past cannot be verified due to the anonymized participation. However, we obtain all our data from sources we consider trustworthy. However, no guarantee can be given with regard to the truthfulness of the data.