Tracking progress: The data issues facing South Africa's BRRRs

This blog article discusses the difficulty in tracking South Africa’s Budgetary Review & Recommendation Reports (BRRRs), which check how the government handles finances. One problem is that recommendations don’t have unique IDs, making it hard to track progress and hold the government accountable. Without unique IDs, past recommendations may be forgotten, or repeated unnecessarily, thereby causing inefficiencies. OUTA (Organisation Undoing Tax Abuse) has teamed up with OpenUp to test a system where each recommendation gets a unique ID, starting with the Home Affairs’ portfolio committee as a trial. This will help track recommendations, spot repeated ones, and link them to government responses. The plan is to expand this to all committees from 2019-2025. The new system will also group recommendations by parliamentary values and connect them to global targets (UN SDGs). The goal is to make government oversight and public transparency better.

Understanding Budgetary Review & Recommendation Reports (BRRRs)

Every year, parliamentary committees must submit Budgetary Review & Recommendation Reports (BRRRs) to the National Assembly. These reports assess how government departments manage public funds, helping Parliament review financial practices. The requirements for these reports are outlined in the Money Bills Amendment Procedure & Related Matters Act.

BRRRs have three key goals: to evaluate service delivery, assess financial efficiency, and provide recommendations for better resource use in the future. These reports (for example) are essential for oversight, as they allow Parliament to assess departmental performance, scrutinise the National Treasury’s responses to recommendations, and suggest improvements in budget allocations.

In addition, BRRRs promote transparency and accountability by documenting government performance. They ensure that the public understands how taxpayer money is being spent and how the government’s fiscal strategy is shaped. The Minister of Finance must explain how the budget reflects Parliament’s recommendations, helping build trust in government spending decisions.

Despite the importance of BRRRs, tracking their effectiveness over time remains a challenge. Improvements are needed in how these reports are monitored and evaluated to ensure that recommendations lead to real, long-term changes in government financial management.

The Need for Tracking Recommendations Over Time

Tracking whether government departments follow through on past recommendations is crucial for ensuring they improve their spending and services. Without proper follow-up, departments may ignore recommendations, leading to missed opportunities for improvement. By reviewing past recommendations, Parliament can identify what worked, what didn’t, and push departments to take action when needed. This process also helps shape better decisions for the future, avoiding past mistakes and improving governance.

A major challenge in tracking recommendations is the lack of unique identifiers (IDs) for each one. Without these unique IDs, it becomes difficult to follow up on whether recommendations were acted upon, especially with changing committee members or shifting priorities year to year. This lack of clarity can lead to repeating the same recommendations, rather than addressing new problems. Adding unique IDs would make it easier to track progress, ensure accountability, and maintain continuity in parliamentary oversight.

Aside from tracking issues, parliamentary committees also face challenges such as limited funding, difficulty getting departments to act, and the need for coordination between multiple departments. Despite these hurdles, tracking recommendations is essential for good governance.

A clearer system for monitoring follow-through would help improve accountability, strengthen public trust, and ensure that recommendations lead to meaningful change.

Current Approach for Monitoring & Evaluating BBBRs

OUTA (Organisation Undoing Tax Abuse) works to review how government funds are spent and how money collected by the government is managed by the National Treasury, with the goal of keeping South Africa a healthy society.

They use recommendations from the BRRRs to guide their annual reports on Parliament’s oversight of the government. These reports have shown that Parliament’s oversight is often not strong enough. In their latest report, OUTA explains how they track and evaluate parliamentary recommendations through a detailed process.

OUTA starts by analysing BRRRs from different parliamentary committees, looking at how well these committees oversee the executive branch, the challenges they addressed, and the recommendations they made for future Parliaments. They also compare these recommendations with the performance plans of government departments, past reports, audit results, and meeting minutes from the Parliamentary Monitoring Group (PMG).

This analysis helps OUTA see if the executive has acted on the MPs’ recommendations, and whether the committees have done a good job or not.

However, because there are no unique IDs for each recommendation, comparing them across different BRRRs is a slow and manual process. This also makes it hard to identify trends and draw conclusions using quantitative data analysis techniques.

Towards a Standardised & Automated Tracking System

A standardised, automated system for tracking parliamentary recommendations is crucial for improving transparency, accountability, and the overall effectiveness of parliamentary oversight. By using unique IDs for each recommendation, it ensures that progress can be tracked consistently and accurately across departments and committees, even when recommendations change over time.

Automating this process also improves efficiency by reducing manual input, which is time-consuming and prone to errors. With real-time access to past recommendations, Parliament (and other interested groups, including citizens) can focus on analysis and decision-making rather than tracking each recommendation manually. This streamlines the process and makes oversight more effective.

The system would also enhance transparency by clearly showing whether government departments have acted on specific recommendations and what the results were. This level of openness helps hold the government accountable for its actions and provides the public with insight into how taxpayer money is being managed.

Additionally, a standardised system supports long-term monitoring and data-driven decision-making. With a clear record of past recommendations, Parliament can identify trends, spot recurring issues, and refine policies for better outcomes, ensuring that oversight leads to real, positive change for the public.

Adding Unique IDs — the case of Home Affairs

To explore the idea further and see if it could work, OUTA partnered with OpenUp to test a system for giving unique ID numbers to recommendations. Together, they chose the Portfolio Committee on Home Affairs as a pilot project. For this test, the 2023 BRRR for Home Affairs (dated October 20, 2023) was used as a starting point, with a focus on sections that list recommendations, specifically.

Section 11.2

Section 11.2 lists recommendations along with responses from the Minister of Home Affairs. The recommendations are organised in two ways:

1. By entity – grouping them under different departments of government.
2. By past reports – showing which BRRR they originally came from.

The 2023 BRRR includes recommendations related to the following entities:

• Department of Home Affairs (DHA)
• Government Printing Works (GPW)
• Electoral Commission of South Africa (IEC)
• Border Management Authority (BMA)

Within Section 11.2, the recommendations are further divided into (refer also to Figures 1 & 2):

• Issues Outstanding from the 2021 & earlier BRRRs
• Recommendations from 2022

Section 11.3

Section 11.3 lists recommendations along with responses from the Minister of Home Affairs, but this time they are grouped only by entity. These recommendations are specifically related to the committee’s 2024 budget and planning report (see Figure 3).

Section 12

Section 12 lists recommendations that have not yet received a response from the Minister (see Figure 4). Unlike Section 11.2, which separates recommendations from the 2022 BRRR and those from 2021 and earlier, Section 12 groups all unresolved recommendations together. That’s why it is titled "Issues Outstanding from the 2022 & Previous BRRRs."

Section 13

Section 13 also lists recommendations that have not yet received a response, organised by entity. These recommendations are specifically related to the committee’s 2023 budget review (see Figure 5).

Assigning Unique IDs to Recommendations

In total, the 2023 BRRR contains 94 recommendations (see Figure 6 below). However, we don’t yet know if all of these recommendations are unique. Some might be repeated across different sections or even from past BRRRs. To check for duplicates, we need to:

1. Go through all 94 recommendations and give each one a unique ID.
2. If a recommendation appears more than once in the 2023 BRRR, we must assign it the same ID to track repeats.

This is also a great time to set up a data structure to help break down the 2023 BRRR pdf file in an organised way. As shown in Figure 6 (above), the recommendations are grouped into five categories and relate to one of four entities. Other useful details to track might include:

• The title of the BRRR
• The date/year of the BRRR
• The recommendation number (as listed in the BRRR)
• The recommendation text itself
• The Minister’s response (if available)
• The source document link (URL)

Below is a proposed structure (with examples) that could be used in a tool like Google Sheets.

After we have gathered all the recommendations from the 2023 BRRR into a Google Sheet (using the structure shown above), we can start assigning unique IDs. There are different ways to create unique IDs, but if you're using Google Sheets or Excel, the following formula works well:

="R" & TEXT(RANDBETWEEN(100000, 999999), "000000")

This formula will keep generating a random 6-digit number, with the letter ‘R’ at the beginning. Randomly generated Unique IDs are better than IDs based on some pattern (or type of logic) because they prevent guessing, reduce errors, and stay reliable over time. Logic or pattern-based IDs can reveal information, run into duplicates, or break if the logic changes. Random IDs keep things secure, unique, and future-proof.

Important: Since the formula refreshes automatically, you must copy the generated ID and paste it as a value to keep it from changing. A good approach is to create a separate sheet called "Unique ID Generator" where you use this formula. Then, copy the generated IDs from that sheet and paste them as values into the sheet with the recommendations.

Since we don’t yet know if some recommendations repeat in the 2023 BRRR, we need to follow a careful step-by-step process (see below). This is a slow, manual task because repeated recommendations are often worded differently.

Step-by-step process

1. Generate a Unique ID.
2. Assign it to the first recommendation (copy and paste as a value).
3. Manually check if the same recommendation appears elsewhere in the 2023 BRRR.
4. If it does, assign it the same Unique ID (copy and paste the same value).
5. Repeat this process until all unique recommendations have an ID.

Checking for recommendations in the 2023 BRRR

Once we have assigned unique IDs, it becomes easy to filter and check which recommendations appear more than once in the 2023 BRRR. After reviewing the data, we found that 8 recommendations appear twice. Figure 8, below, provides a summary of this.

Interestingly, all 8 recommendations listed under Section 12 (Outstanding Committee’s Recommendations) have responses from the Minister in Section 11.2.

It’s unclear why these 8 are listed in Section 12, while other recommendations from 2021 and earlier aren’t. To avoid confusion, we remove the duplicates that don’t have responses (i.e., the 8 recommendations under Section 12).

After doing this, we’re left with 86 unique recommendations in the 2023 BRRR, which are spread across the different sections as follows:

In simple terms, in the 2023 BRRR there are (refer to Figure 9):

• 38 recommendations (12 + 26) from previous BRRRs (2022 and earlier) that have responses;
• 22 recommendations for the following year’s (2024) budget and plan, which also have responses; and
• 26 recommendations for the current year’s (2023) budget review, which need responses within 6 months.

Checking previous BRRRs for recommendations found in the 2023 BRRR

Next, we need to check the 2022 BRRR for these 86 recommendations. The goals are:

1. Find out how many recommendations were carried over from 2022 to 2023
2. Identify which groupings they were placed under

Depending on how far back we want to check, we can repeat this process for earlier years. For this trial, we went all the way back to the 2020 BRRR, but only for the 86 unique recommendations in the 2023 BRRR.

As it turns out, even the sections in different BRRRs are not consistent — both their numbering and names vary. So, we followed this rule of thumb for identifying and assigning unique IDs:

1. Search for recommendations from the 2023 BRRR in earlier BRRRs.
2. If a recommendation appears more than once, only record the instance with a response.
3. If it appears only once, record it and put ‘n/a’ for no response.

As shown in Figure 10 (above), out of the original 86 unique recommendations in the 2023 BRRR, 7 can be traced back to the 2020 BRRR. Focusing on these 7 recommendations, we looked at the first BRRR each one appeared in and also checked the most recent 2024 BRRR.

Findings from the trial

As Figure 11 (above) illustrates, a recommendation can:

• Appear in a BRRR long before a Minister’s response comes (for example, R197612 first appeared in 2018 but only got a response in 2022).
• Sporadically appear and disappear in BRRRs over time (for example, R697208 first appeared in 2019, again in 2020, then disappeared in 2021, reappeared in 2022, and finally got a response in 2023, after which it disappeared again).
• Receive a response from the Minister, but this doesn’t mean it has been resolved (for example, all the recommendations listed above got a response in 2023, but only 2 of the 7 appeared again in 2024 — this time without responses and marked as ‘Outstanding Committee’s Recommendations’).

Despite the seemingly patchy results (of when particular recommendations appear in BRRRs and when they are responded to), the method of assigning unique IDs for tracking recommendations over time remains a valid and effective approach for several reasons:

1. Tracking Repeated and Carried-Over Recommendations: The process helps to identify and track recommendations that have been repeated across multiple reports, ensuring consistency in their follow-up. In the case of the Home Affairs Portfolio Committee, the unique IDs allowed recommendations from the 2023 BRRR to be traced back to previous BRRRs (in some instances, as far back as 2018), even when they appear under different sections or have slightly altered wordings.
   
   
2. Consistency in Tracking Responses: Assigning unique IDs ensures that responses from the Minister are properly linked to the correct recommendations, regardless of where or how those recommendations appear across different BRRRs or sections. This is important for monitoring whether recommendations have been resolved, acknowledged, or still require attention. For example, recommendations from earlier BRRRs that were reintroduced in 2023 can be traced and tracked for consistency in responses and progress.
   
   
3. Clarity and Organisation: With unique IDs, it becomes easier to organise the recommendations systematically, making it possible to filter and sort them based on different criteria, such as response status or entity. This structured approach aids in understanding trends, whether a recommendation has been addressed, and if any issues remain outstanding over time.
   
   
4. Managing Duplicates: As the trial using the 2023 BRRR revealed, some recommendations appear multiple times across different sections, or even in previous years’ BRRRs. The unique ID system allows for efficient identification and management of duplicates. Recommendations that appear more than once can be tracked with a single ID, eliminating confusion and ensuring that they are not mistakenly treated as separate issues.
   
   
5. Long-Term Monitoring: The ability to track a recommendation's history across multiple years — whether it’s been raised in earlier BRRRs, responded to by the Minister, and how its status evolves over time — is invaluable for long-term monitoring. This also aids transparency and accountability by showing whether recommendations are being consistently addressed and followed through on, or if they are neglected.
   
   
6. Increased Efficiency: The manual process of assigning unique IDs may seem slow initially, but once set up, it significantly improves the ability to manage and track large volumes of recommendations, making it easier for teams to focus on follow-up actions, rather than searching for recommendations or wondering if they have been dealt with.
   
   
7. Clear Data Structure: The use of a structured data system (such as a Google Sheet) allows all relevant information about each recommendation to be captured in one place. By including details like the recommendation text, response status, source document, and unique ID, the system becomes a comprehensive tool for analysis, reporting, and decision-making.
   
   

Overall, the approach of assigning unique IDs is a practical, reliable, and scalable method for managing recommendations across multiple BRRRs and ensuring they are tracked consistently over time. It enables better oversight, more effective follow-up, and greater clarity in understanding the evolution of recommendations and responses.

Next steps towards the Tracking System

After testing the system, OUTA and OpenUp have decided to extend this process to all portfolio committees' BRRRs from 2019 to 2025. Once the BRRRs for each committee have been organised into a tool like Google Sheets or Excel, the process of assigning unique IDs to each recommendation will begin.

Along with the unique IDs, OUTA will also tag each recommendation with a few keywords (so users can search for them easily), assign categories based on parliamentary values, and link the recommendations to targets, indicators, and dimensions from the Indicators for Democratic Parliaments (IDPs).

These IDPs are aligned with the UN's Sustainable Development Goal targets 16.6 and 16.7. This will allow OUTA to assess how well South Africa's parliament is doing in areas like effectiveness, accountability, transparency, responsiveness, inclusiveness, participation, and representation.

[UPDATE: At the time of writing, all recommendations had been organised into sheets, and are awaiting unique ID assignment — a follow-up piece on this experience will be published]