Evaluation of the Duties Relief and Duty Drawback Programs: Appendix B: Evaluation methodology and data limitations

Evaluation timeframe

The substantive work for the evaluation was carried out between March and November 2021.

Evaluation questions

The questions developed for the evaluation were based on [redacted]; these are listed below. An evaluation matrix was used to guide the evaluation, which included the evaluation indicators and sources of data.

[redacted]

Sources of data

Multiple data collection methods and sources were used to answer the evaluation questions, as described below:

Document review

The document review took place throughout the evaluation project, from the planning to the examination phases. Over 50 documents were reviewed, [redacted], CBSA standard operational procedures, program performance reports, past audits, and performance measurement tools from the Trade Incentives Unit. Documents were reviewed systematically and, where appropriate, evidence was compiled and used to help answer the evaluation questions.

HR data

A HR dataset was extracted from the Corporate Administrative System (CAS) by the Human Resources Branch (HRB). However, given that the SOTCs work across multiple programs in addition to the DRP and Drawback Program, and that SOTCs work across multiple divisions at HQ, the evaluation team had to rely on HQ to identify the exact numbers of staffing in the program in the past 3 years. For similar reasons, regional staffing numbers were also provided by the Regions.

Program data

Program data for the DRP and Drawback Program is primarily contained in three separate databases: BARKS, TCMS and the Customs Commercial System (CCS). These databases track the processing of Drawback claims (BARKS), DRP licences (CCS) and claims against DRP licences (TCMS). The evaluation team also analyzed litigation data related to the Programs.

Semi-Structured interviews

Semi-structured interviews were held with key stakeholders from HQ and all regions - Western region (Pacific & Prairie), Ontario and Quebec regions. In all, 11 interviews were conducted with 24 directors, managers and SOTCs.

Data limitations and challenges

The main challenges with the systems data are outlined in section 4.2.2. Mid-point reporting challenges.

An additional challenge was having sufficient processing power to analyze the large Excel TCMS datasets, but this was overcome without any impact on the evaluation timeline.