Evaluation of the Duties Relief and Duty Drawback Programs: Progress [redacted]
4.1 Assessment of progress towards staffing targets
Finding 1: Recruitment of additional staff took longer than anticipated. To date, three-quarters of positions in the DRP and Drawback Program have been filled, but the key region of Ontario still lacks SOTCs.
[redacted] funding for a total of 26.6 incremental FTEs to be added on an ongoing basis, 23 of whom would be directly involved in improving the administration of the DRP and Drawback Program. The projection was to staff all 23 HQ and regional positions by , but the risk of not being able to recruit and train up all staff was assessed as highly probable.
As is shown in Table 1, as of , 58% of all positions in the DRP and Drawback Program were filled. As of , three-quarters of all positions were filled, including nine of the 11 HQ positions. Of note, 40% of DRP/Drawback positions were still vacant in the Ontario region at this time. This is important, because over the past three years, Ontario region finalized almost two-thirds of all DRP licence applications and 43% of all Drawback claims, but conducted only 14% of all DRP verifications. As a mitigation measure, Ontario region assigned DRP and Drawback work to experienced Trade staff in other areas until new staff joined the Programs following the creation of a FB-04 pool in fall 2020.
| NCR | Quebec | Ontario | Western | Total (Regions) | Grand Total | |
|---|---|---|---|---|---|---|
| New | 9 | 5 | 7 | 2 | 14 | 23 |
| Renewed (existing) | 2 | 14 | 23 | 11 | 48 | 50 |
| Total | 11 | 19 | 30 | 13 | 62 | 73 |
| Percentage filled as of | 18.2% | 68.4% | 53.3% | 88.5% | 65.3% | 58.2% |
| Percentage filled as of | 72.7% | 94.7% | 60.0% | 84.6% | 75.8% | 75.3% |
| Source: CAS, plus self-reported data by the regions and HQ. | ||||||
Underlying the recruitment trends are some complicated realities. First and foremost, the DRP and Drawback Program are highly technical, and new staff require one to two years to gain the necessary skills and experience to undertake trade compliance verifications. Such verifications are often technically complex and sensitive where SMGs are involved. In addition, SOTCs are highly specialized officers who are in demand across the CBSA, making recruitment and retention inherently challenging. There has been very high staff turnover at HQ and in some regions [redacted]. For example, while the Quebec region is now close to its recruitment target number of SOTCs, the addition of 15 FTEs over the past three years translated into only a net gain of five FTEs, due to departures. Ontario region has an ageing workforceFootnote 15, a high attrition rate and indicates that Trade more broadly has experienced challenges attracting and retaining staff. Finally, onboarding new employees during the COVID-19 pandemic has been an additional challenge. Given the complex and technical nature of the program, new SOTCs are typically mentored one-on-one, the opportunities for which have been limited over the past 18 months.
4.2 Assessment of progress towards expected results
Finding 2: Progress towards intended outcomes could not be determined, [redacted], and the low quality of the available data.
4.2.1 General observations
It was very challenging to demonstrate progress in realizing expected outcomes using the indicators [redacted]. The evaluation team determined that none of the [redacted] indicators were adequate to demonstrate progress the program has made in the last three years. This was deemed the case as some of the indicators are outside the control of the CBSA and therefore should not be used as measures of performance; other indicators are so broad or general that they can offer little insight into real performance gains; and, for other indicators still, the CBSA’s efforts to strengthen program administration and oversight is actually working in opposite direction of the established targets, which were only based on a straight-line projection. [redacted] indicators have lost relevance as a result of the changing external environment and the implementation of the CF, which is built around incremental compliance interventions rather than a reliance on the full trade compliance verification. Under the CF, the CBSA may well conduct more smaller-scale verifications and fewer full verifications; however, the latter will cover the highest-risk areas.
4.2.2 Mid-point reporting challenges
Both the evaluation team and program experts endeavoured to re-create the [redacted] baselines from existing data and, by extension, determine if progress had been made [redacted]. However, for most of the indicators, it was not possible to recreate the original baselines and, thus determine whether associated targets had been met. Some indicators could not be reported on at all. [redacted], no SOPs have been developed. This has led to a situation whereby internal reporting on the number of DRP applications and drawback claims finalized does not match data published elsewhere, even when using the same methodologyFootnote 16.
The storage and management of data for the DRP and Drawback Program is unwieldy and cumbersome. Data used for performance reporting has to be cleaned and collated manually, and is, therefore, difficult to validate. The Trade Compliance Management System (TCMS) is the working database into which officers enter data as they undertake trade compliance verifications, and it is the main source of DRP data in general. Data on processing drawback claims is entered into the Batch Accounting Reporting K32 System (BARKS). In addition, officers maintain paper files for all claims/applications. Given that the Programs are administered in the regions, there are regional inconsistencies in how data is recorded in the systems. Due to the challenges with systems, one key region keeps data in local databases and periodically uploads the data into TCMS/BARKS, which can cause data gaps and errors.
Given the poor quality of the original [redacted]indicators, the inability to replicate [redacted] baselines, data quality issues, and changes in the external environment, there was little utility in using the original indicators to demonstrate the Programs’ performance over the past three years. At this mid-point, the evaluation team recommends that a full review of the Performance Measurement Framework be conducted and a new set of indicators be put in place (see section 6.2 Recommendations).
4.2.3 Indicator reporting
Table 2 below provides the outcomes and indicators [redacted]. A complete table of the Expected Results [redacted] is contained in Appendix C. Of note, the immediate results indicators had a target to increase the number of applications/claims finalized and verifications conducted by 50%, which stems from the 45% increase in staffing based on the expected volume of work resulting from the surtaxes.
Table 2: Expected results, outcomes and indicators [redacted]
Immediate Results (short-term)
New reviews of Duties Relief applications, reviews of Drawback claims, and compliance verification activities for both programs are conducted, decisions are made in a timely manner and program measures are actively enforced.
Development and enhancement of oversight structure and activities along with revised policies and procedures.
| Outcomes | Indicators | |
|---|---|---|
| Result 1 | Increased number of Duties Relief Program applications are reviewed | Number of applications finalized |
| Result 2 | Increased number of compliance verifications are conducted for the Duties Relief Program | Number of trade compliance verification for the Duties Relief Program completed |
| Result 3 | Increased number of Drawback claims are reviewed | Number of Drawback Program claims finalized |
| Result 4 | Increased number of compliance verifications are conducted for Drawback program claims | Number of trade compliance verifications for the Drawback Program completed |
Intermediate Results (medium-term)
The CBSA conducts its Duties Relief Program and Duty Drawback Program compliance activities in a fair, transparent and timely manner.
| Outcomes | Indicators | |
|---|---|---|
| Result 1 | Stakeholders have access to up-to-date program information on the CBSA’s public web site, including how to access program services. | Percentage of public policies and program information (D-Memoranda) relating to the Duties Relief Program and the Duty Drawback Program available on the CBSA’s public web site that are developed or revised in 2022 to 2023 over same in 2017 to 2018 |
| Result 2 | CBSA administers the Duties Relief Program and Duty Drawback Program in a fair and consistent manner on a national basis with respect to decisions based on the legislative framework | Percentage of program decisions which are under appeal in fiscal 2022 to 2023 over same in 2017 to 2018. The number of decisions appealed is used to measure the perceived fairness and consistency of program administration |
Ultimate Results (long-term)
Canadian industry is protected through the CBSA’s administration of the Duties Relief Program and Duty Drawback Program.
| Outcomes | Indicators | |
|---|---|---|
| Result 1 | To what extent is Canadian industry protected through the CBSA’s administration of the DRP and Drawback Program | Percentage of number of verifications for the DRP & Drawback Program in FY 2023 to 2024 resulting in assessments greater than $1M |
| Result 2 | Canadian jobs and production are protected | Percentage of the total amount of assessments resulting from verifications for DRP & Drawback Program in FT 2023 to 2024 over the total amounts of duties relieved and refunded for the DRP % Drawback Program in FY 2023 to 2024 |
- Immediate Result 1 - Number of DRP applications finalized
- Immediate Result 3 - Number of Drawback claims finalized
The number of DRP applications finalized and the number of Drawback claims finalized are supposed to demonstrate progress in outcomes that are almost identically worded, and which are not outcomes per se, but are measurements of workload. The incoming volume of applications and claims is not controlled by the CBSA. The use of the DRP and Drawback program fluctuates as external environmental factors change. For example, there was a spike in applications for DRP licences in FY 2018 to 2019 as a result of the surtaxes, but this was short-lived as the tariffs and countermeasures were removed within the year.Footnote 17 For this reason, applications and claims finalized are measures of program activity and should not be used as indicators of program performance.
The Agency’s responsibility with regards to applications and claims submitted is to process them promptly, and to grant DRP licences to companies meeting the eligibility requirements. There are internal service standards related to the processing of DRP applications and Drawback claims - to process them no later than 90 calendar days from the date of receipt of a correctly completed and supported application/claim. For DRP applications, the CBSA met this service standard at all times for FY 2018 to 2019 to 2020 to 2021. For Drawback claims, the service standard was met 82% of the time in FY 2018 to 2019 and 88% of the time in FY 2019 to 2020 and 2020 to 2021. Timeliness of processing applications and claims is a better indication of the Agency’s support to Canadian Industry, rather than absolute numbers of claims and applications processed.
There was a general consensus at HQ and in the regions that more rigour is now being applied in issuing DRP licences. Anecdotally, the evaluation team heard instances of where licence applications had been denied, which had previously not occurred. While no data was available to substantiate this, the CBSA has taken specific steps to apply more rigour to application processing.Footnote 18 This is important, because if the CBSA is being more judicious in approving licences and is denying licences in some cases, better program performance [redacted], which is to increase the number of DRP applications finalized.
In addition, as different figures exist on the number of DRP applications and Drawback claims finalized for the same fiscal yearFootnote 19, the program should ensure consistency in reporting on its key metrics.
- Immediate Result 2 - Number of trade compliance verifications for the DRP completed
- Immediate Result 4 - Number of trade compliance verifications for the Drawback Program completed
The indicators on the completion of trade compliance verifications are intended to demonstrate increased compliance activities conducted for the Programs. However, as an absolute number, rather than a percentage of claims or applications for relief that are subject to a verification, they do not account for the incoming volume of claims or applications for relief. They also do not indicate if the verifications are being conducted in a timely manner. Furthermore, in the case of the DRP, the global number of trade compliance verifications is too broad to be meaningful, as there are different types of verifications carried out at different times and for different reasons. In addition, an arbitrary target number of verifications completed puts focus on quantity over quality. Presumably the goal should be to conduct the most meaningful verifications (i.e., highest risk and/or covering the greatest dollar value). As these are likely to be more complex and take longer to complete, putting emphasis here means a smaller number of compliance verifications are likely to be conducted overall. Finally, the absolute number of verifications becomes an even less meaningful measure with the advent of the CF, which is a move towards a targeted, risk-based approach to compliance assurance, relying less on the full trade verification to ensure compliance.
The evaluation team recommends these indicators be discontinued and replaced by much more precise and up-to-date indicators that reflect the new approach to promoting compliance in the CF. v
- Intermediate Result 1 - Percentage of public policies and program information (D-Memoranda) relating to the DRP and the Duty Drawback Program available on the CBSA’s public web site that are developed or revised in 2022 to 2023 over same in 2017 to 2018
The indicator on the availability of D-Memoranda on the public website is supposed to measure the outcome of stakeholders having access to up-to-date program information. However, the existence of new or updated D-Memoranda on the public website does necessarily mean that stakeholders have access to the information they need on the Programs. The clarity and completeness of the program information contained in the D-Memoranda is important and whether any information is missing, is best assessed by the recipients of the information (such as via periodic stakeholder surveys).
The evaluation team recommends that the program still track this indicator at the activity level. However, a new outcome-focussed indicator (or indicators) is required, which is centred on soliciting stakeholders’ views as to whether they have access to the information they need.
- Intermediate Result 2 - Percentage of program decisions, which are under appeal in fiscal 2022 to 2023 over same in 2017 to 2018
This indicator is inadequate to measure its associated outcome, which centres on the CBSA’s ability to administer the Programs in a fair and consistent manner on a national basis with respect to decisions based on the legislative framework. The number of appeals launched against CBSA decisions is outside the Agency’s control. A company may appeal a CBSA DRP or Drawback decision for any reason; conversely, a company may not launch an appeal even if it feels it has not been treated fairly or consistently, for example due to the costs involved.
Over the past five years, there have been mounting legal challenges, which are a direct result of the CBSA ramping up verifications of DRP SMG participants, as the SMG-related applications for relief involve very high monetary valuesFootnote 20. These verifications have revealed significant non-compliance, resulting in companies losing access to large sums of duties relief, which has been challenged in court.Footnote 21 In this way, stronger program performance (e.g., uncovering non-compliance via verifications of high-risk SMG importers) [redacted] to reduce number of program decisions appealed. The outcome of the court cases is more indicative of whether or not the Programs are being administered in a fair and consistent manner, either in terms of companies withdrawing their court challenges, or via the court-of-law rulings made on the challenges.Footnote 22
For the reasons discussed above, a more suitable indicator to capture fairness and consistency with respect to decisions based on the legislative framework should be developed.
- Ultimate Result 1 – Percentage of number of verifications for the Duties Relief Program plus the Drawback Program in fiscal 2023 to 2024 resulting in assessments that are greater than $1M.
This indicator is insufficient to measure its associated outcome, which is that the Program users are compliant. Firstly, the reason for the threshold being set at $1 million is unclear. It does not consider/capture participants who are non-compliant (possibly repeatedly) to the tune of hundreds of thousands of dollars. It also only captures compliance for a subset of program users, as the original baseline was calculated for high-risk verifications only (i.e., SMG-related DRP verifications); it does not take into account the compliance of all the other businesses which use the DRP (as well as the Drawback Program).
The evaluation team also determined that the original baseline was calculated incorrectly for this indicator. The denominator for the baseline was high-risk SMG verifications, and was stated as 40. However, while 40 SMG verifications had been initiated [redacted], only around 20 had concluded, and thus could have had their assessments included in the calculation of the baseline.
The evaluation team recommends this indicator be discontinued and replaced by a more easily understandable and inclusive indicator or indicators of compliance, which are tracked separately for the Programs.
- Ultimate Result 2 - Percentage of the total amount of assessments resulting from verifications for the Duties Relief Program and the Drawback Program in fiscal 2023 to 2024 over the total amounts of duties relieved and refunded for the Duties Relief Program plus the Drawback Program in fiscal 2023 to 2024.
This indicator is not readily understandable. As with the other ultimate results indicator, it does not sufficiently measure its associated outcome - Canadian jobs and production are protected. It does not reflect the number and variety of business supported by the Programs. It also does not consider the efforts of the CBSA to protect businesses not participating in the Programs via compliance assurance on those who are benefitting.
The most basic measure of protecting Canadian Industry is the total amount of duties relieved/refunded under the Programs, but the program could explore an outcome of reducing barriers for industry (i.e., reductions of an unnecessary tax burden), with an indicator being the number of jobs impacted.
The evaluation team recommends this indicator be discontinued and replaced by a more easily understandable and inclusive indicator or indicators of compliance, which should be tracked separately for the Programs.
4.2.4 Performance reporting and CARM
In the near future, CARM is expected to be the system of record for processing drawback claims as well as for accepting submissions of licence applications. To this end, there has been significant involvement both at the HQ and regional level in CARM design activities over the past two years. It is anticipated that CARM will facilitate an improvement in performance reporting for the programs via its dedicated reporting functionality.
- Date modified: