On the Commission's Factual Summary report on the public procurement consultation

Earlier this week the Commission put out a summary of the findings on the public procurement consultation for the revision of the procurement Directives. Albert has a great entry on the findings of the consultation but I will be taking a different view on this one.

In total the consultation received 733 responses distributed across various stakeholders such as contracting authorities, business, trade unions or academia. And herein lies the main problem with this exercise and the one I want to focus on since it affects the usefulness of the whole exercise. The Commission put out the questionnaire stating that "[e]veryone is welcome to provide feedback on this call for evidence and reply to the public consultation." From a methodological perspective this is...not great.

Let's look at contracting authorities to illustrate the problem. Out of 250,000 contracting authorities the Commission received 199 submissions. That's a 0.001% sample of contracting authorities that filled in the form. The sample size is small but that by itself could have been alleviated had this been an exercise in quantitative analysis with a purposeful sampling strategy that would strive to gather a representative set of views of a specific cohort. However, that is not the case here since anyone could reply to the questionnaire so this means that only those contracting authorities really wanting to say something and be part of the process would take the effort to fill in the form.

In consequence this means that whatever the responses contain they represent only that minuscule subset of contracting authorities that filled in the form and not the wider cohort they belong to. Therefore, making decisions based on this data is fraught with danger. We do not know why other contracting authorities did not participate. Were they not aware? Do they not have the resources to spend some time filling in the form? We will never know but the Commission has said publicly they considered the number of responses to be low. We will never know the reasons, but I wonder what outreach was done to target those bodies that usually do not speak up at opportunities like this.

The same caveat can be said for the business responses, although we do not know the size of that cohort to calculate a sample size and at least business associations exist to that information collection and parsing for feeding into processes like this.

Beyond treating this exercise as a mere artifact in a larger policymaking exercise, building policy upon these findings means empowering those who scream the loudest at the expense of the quiet ones.