Measuring

​​​​1. Gathering hard data to drive tangible action

Numbers and facts form a strong basis for setting a baseline and KPIs, as well as demonstrating results. They can also trigger attention and action by stakeholders.

No data is perfect – practitioners advise simply being transparent about the limitations.

Launched in 2011/2012, the incident reporting database of the Maritime Anti-Corruption Network now contains some 40,000 anonymous incidents reported of corrupt demands. MACN used this incident data right from the start to give stakeholders an indication of “what may be happening”.

“MACN’s support and the incident data have been key drivers that resulted in a new regulatory framework in Argentina that reduces discretion in the inspection of holds and tanks on board vessels. In 2018, corruption incidents in Argentina decreased by more than 90 percent according to MACN data submitted through our anonymous incident reporting mechanism.” 

– Cecilia Müller Torbrand, MACN Executive Director

2. Picking useful metrics

Gathering metrics on membership growth and market share can help gauge the success of an initiative. It can also help identify when the “tipping point” is reached, i.e. when the initiative is mature enough to drive long-lasting change.

Whatever the metrics, practitioners recommend being as transparent as possible with stakeholders and reporting on these regularly.

The Banknote Ethics Initiative (BnEI) has a system for measuring impact related to membership and customer support. Such a black-and-white system is possible thanks to the BnEI’s clear and rigid accreditation process but may not be practical for initiatives structured differently.

3. Stories vs. statistics

Capturing real stories and testimonials is a vivid way to demonstrate the impact of a Collective Action programme, particularly as a complement to quantitative results.

The Maritime Anti-Corruption Network shares stories from captains and other frontline staff on their experiences with facing corrupt demands and how the situation is changing. As experienced by one captain: “I have crossed Suez Canal twice since MACN’s Say No campaign was introduced, and it’s much easier to resist now as more companies are participating in the anti-corruption campaign.”

As part of Transparency International’s EU Integrity Pacts project, donors and stakeholders are actively encouraged to give feedback. In Poland, for example, the contractor involved in the Integrity Pact said: “In working with the Stefan Batory Foundation [the independent civic monitor], we have implemented a whistle-blower policy for the first time. Our partnership has helped us to bring our processes in line with our values by strengthening our corporate governance.” Such testimonials not only strengthen some of the core assumptions of the project, but corroborate the credibility and efficacy of the Integrity Pacts model. 

4. Being transparent about challenges

Common challenges include providing hard evidence for an initiative’s effectiveness or realising that an initiative has had unintended or negative consequences. Transparency is the best approach here.

In the EU Integrity Pacts project, several country projects have not identified any irregularity. It is difficult to claim that IPs were responsible for the lack of them, or to claim that this means the procurement system in question is clean. Nevertheless, there is a still a positive story to be told about how Integrity Pacts can boost coordination and accountability among all stakeholders.

5. Setting a baseline

Establishing the status quo is important for setting realistic goals and demonstrating results. But it’s challenging to do this without an existing body of data, particularly if there is limited time for the preparatory phase.

Practitioners advise to use what you have – don’t worry if it’s not perfect – and plan periodic revisions to your baseline and KPIs as your data improves.           

Some compliance-focused initiatives use self-assessment checklists for members to set their own baseline, identify their weaknesses and measure their progress. Examples are TEİD and the Ukrainian Network of Integrity and Compliance. Several certification initiatives also make use of self-assessments in their initial stages.

6. Flexible monitoring and evaluation

Monitoring and evaluation (M&E) systems should be comprehensive, flexible and realistic. In some initiatives, members develop the goals and KPIs organically, while in others donors request these to be set in advance.

Common tips include: defining clear roles and responsibilities; substantiating claims with evidence, and proactively seeking external feedback from donors and stakeholders.       

The leaders of Colombian initiative Hacia la Integridad sought external advice from a consultant to make sure their M&E system was rock-solid.

Metrics in the EU Integrity Pacts project cover the number and type of irregularities and the corresponding responses from government; recommendations to improve procurement mechanisms; and indicators of policy, institutional and behavioural changes. The KPIs were developed and adopted on the project level. In a further attempt to cater to different contexts, each monitor has set tailored non-KPIs to help capture behavioural changes and perceptions in their specific context.

7. Linking cause and effect

The success of an initiative often depends on external factors such as changes in legislation or the political and economic landscape. This makes it tricky to link cause (the work of the initiative) and effect (positive impact and change).  

Transparency International’s impact assessment system for its EU Integrity Pacts project combines yearly change mappings with continuous monitoring. Monitors include relevant external changes such as new procurement laws, which solve an issue they have flagged but to which they haven’t contributed. This approach helps the project leaders to demonstrate transparently which activities have led to which effects.

8. Impact vs. change

Some practitioners advise considering the difference between the quantifiable impacts of an initiative – usually measured with KPIs – and real, long-term change in the operating environment.

Experienced practitioners such as Professor Esteban Arribas Reyes argue that the second may be harder to measure, especially in the short term, and more unpredictable in terms of timing and consequences. Demonstrating that an initiative has led to long-term change may, however, be more powerful in convincing governments, civil society and businesses of the value of the Collective Action.    

An Integrity Pact was deployed in 2018-2019 for a school construction project in the Spanish region of Valencia. The administrators gathered indicators to define the key considerations to select the location. The impact was not only on the building of the school but the fact that the administrators put into practice a more transparent way of making such decisions. A positive impact for sure, but be cautious – this new process applied only to this particular project and does not imply long-term systemic change. See also information (in Spanish) by the Valencian authorities.

9. Changing behaviour through education

Behavioural change takes a long time and is often driven by education. Working with local academic institutions to update curricula to include relevant aspects of ethics and compliance could be a way forward here.

The UNODC Global Integrity Education project aims to “foster ethical decision-making in the private sector by equipping young graduates with ethical mindsets and skills at the start of their careers.”

The UNODC Integrity and Ethics Programme and the UN Global Compact's Principles for Responsible Management Education (PRME) have a range of educational resources that might be useful.

10. Peer learning between members

Building a constructive atmosphere of trust between members can lead to valuable peer learning experiences.

The UNODC Global Integrity Education project has a strong emphasis not only on M&E but on continuous learning through frequent interviews and reflection sessions with the private sector.

One positive impact of the Banknote Ethics Initiative has been the fact that, for the first time, compliance officers have a forum to meet in an atmosphere of trust and raise issues of market misbehaviour with one other. The Basel Institute has seen such positive peer exchanges in other initiatives.

The Turkish Ethics and Reputation Society, TEİD, organises small-group meetings, including sectoral or regional meetings, where TEİD corporate member representatives can share their experiences, challenges and solutions/achievements and learn from others.

Things to think about

Of course, these are just examples of what other practitioners have found effective. It very much depends on the context of your initiative and what you’re trying to achieve. Here are a few general things to think about:

• What data will you collect and how?
• Who could give reliable and honest testimonials?
• What challenges do you face in measuring impact? 
• How can you support others?
• How can members of your Collective Action support each other? Mentoring? Reflection sessions?
• What can you learn from other Collective Action practitioners and initiatives?

If you want to talk over any of this, we’re happy to act as a sounding board. It’s our role under the Siemens Integrity Initiative to act as a free source of guidance and support to Collective Action initiatives. Contact Vanessa Hans.