Internal Audit and Program Evaluation Directorate
Evaluation of the Field Technology Support Program
September 18, 2019

Evaluation of the Field Technology Support Program (PDF, 873  KB)
[help with PDF files]

Table of contents

• Acronyms and abbreviations
• Executive summary
• 1. Background
  • 1.1 Evaluation purpose and scope
  • 1.2 Program description
    • 1.2.1 Logic model
  • 1.3 Key stakeholders and program management structure
  • 1.4 Resources
  • 1.5 Evaluation methodology
• 2. Findings: Effectiveness
  • 2.1 Analytical and forensic services
  • 2.2 Border technology services
    • 2.2.1 Asset management
  • 2.3 Innovation
    • 2.3.1 Pilot projects
    • 2.3.2 Research and development
  • 2.4 Program awareness and utilization
• 3. Findings: Efficiency
  • 3.1 Expenditures and outputs
  • 3.2 Overlap
  • 3.3 CBSA/CRA MOU
• 4. Conclusion and recommendations
• Appendix A: Management response and action plan
• Appendix B: Field Technology Support Program logic model
• Appendix C: Data limitations
• Appendix D: Detection technology asset management model: Current and potential
• Appendix E: Pilot projects timeline

Acronyms and abbreviations

ACROSS

Accelerated Commercial Release Operations Support System

AFSD

Analytical and Forensic Services Division

BSO

Border services officer

BTD

Border Technology Division

CBSA

Canada Border Services Agency

CCTV

Closed-circuit television

CFIA

Canadian Food Inspection Agency

CRA

Canada Revenue Agency

CTO

Chief Transformation Office

DPMC

Detection Program Management Committee

DRAP

Deficit Reduction Action Plan

DSSA

Designated safe sampling area

DTFS

Detection technology field support

FCMB

Finance and Corporate Management Branch

FDE

Forensic document examination

FMM

Functional management model

FTE

Full-time equivalent

FTS

Field technology support

FY

Fiscal year

GTA

Greater Toronto Area

HTS

Highly toxic substances

HRB

Human Resources Branch

IBP

Integrated business planning

I&E

Intelligence and enforcement

ICES

Integrated Customs Enforcement System

ISO

International Organization for Standardization

ISTB

Information, Science and Technology Branch

LASS

Laboratory Analysis Support System

LSI

Large-scale imaging

MOU

Memorandum of Understanding

MRAP

Management Response and Action Plan

NCR

National Capital Region

OGDs

Other government departments

POE

Port of entry

RCMP

Royal Canadian Mounted Police

SED

Science and Engineering Directorate

SESOD

Science and Engineering Support and Operations Division

SOR

Southern Ontario Region

SPB

Strategic Policy Branch

SSI

Small-scale imaging

US

United States

VIMC

Vancouver International Mail Centre

VP

Vice president

VSSA

Variable Shift Schedule Agreement

Executive summary

Evaluation purpose and scope

The purpose of this evaluation was to examine the performance (effectiveness and efficiency) of the Field Technology Support (FTS) Program (the FTS Program/the Program), between fiscal years 2013-2014 and 2017-2018, in accordance with the 2016 Treasury Board Policy on Results. The evaluation was undertaken between June 2018 and September 2019.

Program description

The Information, Science and Technology Branch (ISTB) of the Canada Border Services Agency (CBSA) is responsible for the design and delivery of the Program. The Program, which falls under the Science and Engineering Directorate (SED) within ISTB, employs science and technology to aid the Agency in accomplishing its border facilitation and security mandate while reducing the risk to border services officers (BSOs) and Canadians. The Program is an integral component of commercial-trade facilitation and compliance, traveller facilitation and compliance, and intelligence collection and analysis.

The SED is comprised of 3 divisions. The Analytical and Forensic Services Division (AFSD) is responsible for the scientific examination of suspected contraband, alcohol, tobacco, cannabis, customs analysis and forensic document examination (FDE). The Border Technology Division (BTD) is responsible for border technology development, detection science and engineering, detection technology field support and advanced analytics. Finally, the Science and Engineering Support and Operations Division (SESOD) takes care of corporate functions such as corporate and information services, quality assurance and technical support and integration, which includes two-way radios.

The Agency generally, and frontline staff in the Commercial and Trade, Travellers, and Intelligence and Enforcement (I&E) branches specifically, are the program clients. The Detection Program Management Committee (DPMC) is the decision-making working group within the Commercial and Trade Branch that determines which technologies will be adopted, where and in what priority, in response to regional requests for detection technology.

Evaluation methodology

Data collection and analysis for this evaluation were conducted between December 2018 and May 2019 using both quantitative and qualitative research methods. The evaluation team conducted interviews with CBSA internal and external stakeholders, reviewed key documentation, analyzed operational and financial data, facilitated focus groups in 2 CBSA regions, made comparisons with other government departments (OGDs) and international partners, and administered a survey to frontline staff.

There were a number of data limitations, including incomplete and under-reported data; insufficient data available on the use of detection technology for commercial seizures; lack of systematic performance measurement; and limited open-source documents and information for international comparisons.

Evaluation findings

The FTS Program is a new program under the Departmental Results Framework (DRF) and, as such, a program management framework remains under development. This evaluation serves as a baseline upon which future progress can be measured, and it can help shape the future of management and governance aspects of the program.

Effectiveness: The extent to which the Program achieved the expected results

Overall, the Program is effective in the services it provides to the Agency, including the scientific analysis of samples and pilot projects. The Program holds the only federal capacity for the examination of forensic documents, the expertise of which is sought both domestically and internationally. As a result of the highly specialized work it carries out, the FTS Program faces unique challenges in staffing positions, and is competing with the much more agile private sector for uniquely qualified and highly skilled candidates.

The Program's current service standards for the analysis of suspected contraband, alcohol, tobacco and cannabis, as well as customs samples are not reflective of the Program's capacity nor of frontline service demand. Increased demand for services, paired with limited testing capabilities, have resulted in sustained backlogs for suspected contraband analysis in recent years. Consequently, turnaround times have increased, potentially leading some frontline staff to submit fewer samples for analysis, which may have contributed to the decline in the number of samples submitted for analysis in 2017-2018. Service standards should be revisited and properly communicated to the front line.

With respect to detection technology, the evaluation found that detection technology equipment is not often used in the traveller stream, but its usage is known to be under-reported by BSOs in Agency systems. When it is used, detection technology is associated with higher value seizures. Of note, insufficient data was available for the use of detection technology in the commercial stream, preventing an assessment of a significant portion of the Agency's detection technology use. Detection technology could be better utilized with improved availability of tools and training for frontline staff. Inadequate training on detection technology has led to under utilization as well as misuse causing the breakdown of equipment.

While the time that detection technology assets were unavailable due to breakdown decreased during 2013-2014 to 2017-2018, frontline staff felt that availability did not meet operational needs. Equipment availability targets are measured by asset class at the national level only, preventing regional, port or asset-specific reporting. The Program should re-examine its role in detection technology asset management by focusing on areas where its technical expertise is required and devolving areas that could be managed by other parts of the Agency, in accordance with the Agency's new Functional Management Model. In addition, although not managed by the Program, the acquisition, distribution, and maintenance of closed-circuit television (CCTV) equipment is fragmented across the Agency and managed inconsistently across regions and POEs.

With respect to innovation, pilot projects developed by the FTS Program have made positive contributions to Agency operations. That said, pilot projects could be more effective with clearer governance and management/ownership delineation, including the roles and responsibilities of the various stakeholders. This should lead to more effective stakeholder engagement, as well as better oversight of pilots. Confusion surrounding pilot ownership and responsibility for stakeholder engagement has previously resulted in avoidable inefficiencies.

The evaluation looked at the awareness of services offered by the FTS Program. Overall, these are not well-known in the Agency and clients are not always aware of how to communicate their needs to the Program. The use of laboratory services is also inconsistent across regions, with significant disparities observed across the 3 postal centres, which are the largest clients of laboratory services.

Efficiency: The extent to which expected results were achieved in an efficient manner

Where data exists, the Program was found to be efficient in most areas. Core program expenditures remained generally stable over the 5-year period examined. The evaluation observed, however, that the FDE unit has regularly performed uncompensated work for external clients.

The AFSD outputs fluctuated significantly over the 5-year period, and generally increased for the BTD. The usage of overtime remained relatively steady except in BTD, which saw a significant increase between 2013-2014 and 2017-2018.

Overall, relatively few areas of overlap exist between the services offered by the Program and those provided by OGDs. The evaluation examined the CBSA/Canada Revenue Agency (CRA) Memorandum of Understanding (MOU), which was developed to govern the scientific laboratory which was formerly shared between the 2 agencies. The evaluation found that, because the MOU targets and estimated hours have not been reassessed since they were established 14 years ago, they are outdated and do not reflect current workloads.

Recommendations

The findings of the evaluation led to the following recommendations:

• R1. The Vice-President (VP) of Information, Science and Technology Branch (ISTB), in consultation with clients, should examine the service delivery model for the analysis of suspected contraband, alcohol, tobacco, cannabis and customs samples. This includes:
  • a. reviewing and revising existing service standards
  • b. undertaking a cost-benefit analysis to determine if efficiencies can be realized by partnering with OGDs and private sector labs to eliminate the backlog
  • c. following the revision of service standards, developing a communications plan to review the branding of the FTS Program and to improve the awareness of the FTS Program's roles and responsibilities, services offered and service standards
  • d. As part of the broader performance measurement strategies currently under development, ensuring that service standards are regularly monitored and reported on

• R2. The VP of Finance and Corporate Management Branch (FCMB) should, in consultation with relevant branches, develop a material management strategy for the Agency's detection technology assets (including determination, acquisition, distribution, maintenance and training), leading to the development of an overall material management strategy for Agency assets.
• R3. The VP of ISTB, in consultation with relevant branches and the regions, should lead the development of an alternative model for the ownership and management of FTS-led pilot projects, which clearly articulates the roles and responsibilities of respective branches and/or regions at the various phases (planning, development, implementation, and mainstreaming). Once developed, this model should be proposed to the Executive Committee for approval.
• R4. The VP of ISTB, in consultation with the Chief Transformation Officer (CTO) Transformation Branch and the Strategic Policy Branch (SPB), should review the governance of innovation and research and development in the Agency to ensure that strategic direction and guidance are provided to the FTS Program.
• R5. The VP of FCMB, in consultation with other relevant branches, should develop a national strategy for the management of CCTV.

1. Background

1.1 Evaluation purpose and scope

This report presents the results of the evaluation of the FTS Program. In accordance with the 2016 Treasury Board Policy on Results, the objective of the evaluation was to examine the performance of the Program. The evaluation examined the Program's effectiveness and efficiency between fiscal years 2013-2014 and 2017-2018.

The FTS Program was identified as a priority evaluation in the CBSA's 2018 Risk-Based Audit and Evaluation Plan. The evaluation scope was approved by the Performance Measurement and Evaluation Committee (PMEC) in September 2018. As shown in Table 1 below, an assessment of laboratory facilities and the outcomes of the Radio Program and sustainability funding were not assessed as part of the evaluation.

An in-depth assessment of relevance was not undertaken because the FTS Program fulfills an operational requirement by supporting the Agency's core border facilitation and security mandate. The Program is unique in its responsibility for providing analytical and scientific advice on samples and for managing technology from a border management perspective. The services offered are also integral to frontline operations in facilitating compliance and enforcement actions for suspected counterfeit and contraband. For this reason, the evaluation limited the assessment of relevance to the services provided to OGDs and partners to assess any potential overlap or duplication of work (covered in section 3.2 the report).

1.2 Program description

The FTS Program is a new program under the 2018 DRF, falling under the "Border Management" core responsibility area. The Program is the responsibility of the SED in ISTB. It leverages science and technology to aid the Agency in accomplishing its border facilitation and security mandate, while reducing the risk to BSOs and Canadians. It is an integral component of commercial-trade facilitation and compliance, traveller facilitation and compliance, and intelligence collection and analysis.

The Program's work falls into 3 main areas: Analytical and Forensic Services, Detection Technology, and Innovation, as outlined in Figure 1 below.^{Footnote 1}

Figure 1: Field Technology Support Program FTS Program: Main areas of work

The Program's laboratory in the National Capital Region (NCR) analyzes seized samples of suspected alcohol, tobacco, cannabis, drugs and drug precursors, as well as customs samples sent from POEs. Analyses are carried out by a team of science professionals with backgrounds in chemistry. Intelligence and advice on observed trends are shared with stakeholders. Disputed documents are submitted by CRA^{Footnote 2} and by CBSA officials in the areas of criminal investigations, immigration, and hearing and appeals to enable compliance and enforcement actions (such as seizures, investigations, and criminal and civil prosecutions) and are analyzed at the laboratory by specialists in forensic ink chemistry and forensic technicians.

Scientists, engineers, technologists, and chemists responsible for the procurement and testing of detection technology also operate within the Program's facilities in the NCR, while 6 field technicians responsible for the installation, maintenance, and repair of select detection technology are based in the regions.

With respect to innovation, innovative technologies are first developed through a phased approach and prototyped by a team of scientists, engineers, and technologists at the laboratory in the NCR before being piloted in the field to determine their technical and operational feasibility. There is also an advanced analytics team working on mathematical modelling, including areas such as machine learning, profiling of previous offenders and automated shift scheduling to reduce overtime and overstaffing.

1.2.1 Logic model

At the time of the evaluation, the FTS Program did not have a performance measurement framework. A logic model^{Footnote 3} for the Program is included at Appendix B and identifies the following expected outcomes:

• Immediate outcomes: Stakeholders have the operational capability to make evidence-based admissibility determinations, enforcement and trade decisions. Advice is provided in support of strategic business decisions.
• Intermediate outcome: CBSA stakeholders are provided with innovative scientific and engineering services and solutions that meet their needs
• Ultimate outcome: The CBSA advances and implements effective border science technology and solutions

1.3. Key stakeholders and program management structure

The SED is comprised of 3 divisions:

1. Analytical and Forensic Services Division (AFSD): AFSD is responsible for testing suspected contraband, alcohol, tobacco and cannabis, and customs samples, as well as examining forensic documents. The results of testing lead to admissibility determinations of both goods and people, application of tariffs, and enforcement actions. The majority of suspected contraband analysis, forensic document examination and customs analysis is completed for CBSA clients, while alcohol, tobacco and cannabis is completed primarily for the CRA.
2. Border Technology Division (BTD): BTD is responsible for the research, evaluation, procurement, maintenance, and development of various border technologies, including detection technologies and pilot projects. The division is also responsible for using data analytics to enhance existing enforcement tools, and for providing 24/7 technical and scientific support to the Agency's radiation program (RADNet) that screens all marine containers entering the country. BTD also provides training to the front line on radiation safety and proper handling of illicit goods.
3. Science and Engineering Support and Operations Division (SESOD): SESOD organizes the lab's outreach activities, manages corporate reporting requirements, completes quality assurance of scientific methodology, and provides all aspects of radio communication services to the front line.

Key stakeholders of the Program include the Commercial and Trade, Travellers, and I&E branches.

Both the Commercial and Trade and the Travellers branches use sample testing services to make enforcement determinations, and require detection technology equipment for day-to-day border operations. The Commercial and Trade Branch, in particular, benefits from detection technology, as more sophisticated equipment is required for clearance of commercial goods. Additionally, the Commercial and Trade and the Travellers branches have been able to take advantage of innovative pilot projects that increase facilitation for low-risk travellers and goods.

Within the Commercial and Trade Branch, the DPMC is the director-level decision-making working group that determines which technologies will be adopted, where and in what priority, in response to regional requests for detection technology.

I&E Branch benefits directly from the results of suspected contraband testing. The Program consults with I&E officials when new trends are seen with incoming precursors, chemical derivatives, or smuggling techniques.

1.4 Resources

Total program spending for fiscal year 2017-2018 was $41.4M, and core program spending was approximately $14.5M; annual average core spending was $13.8M over the 5-year evaluation period. In fiscal year 2017-2018, there were a total of 116 full-time equivalents (FTEs) working in the FTS Program.

1.5 Evaluation methodology

An Evaluation Advisory Committee was established to support the evaluation by providing input, advice and suggestions regarding evaluation deliverables. The committee membership was established at the outset of the evaluation and included directors general from relevant branches of the Agency.

The data collection and analysis for this evaluation were conducted between December 2018 and May 2019 using both quantitative and qualitative research methods. See Figure 2 below for a comprehensive list of data collection methods employed in this evaluation.

Figure 2: Data collection methods

Consultations with key stakeholders and a review of key documents during the planning stage assisted in refining the evaluation questions to ensure that the evaluation would provide useful information for decision-making. The evaluation questions focused on an assessment of effectiveness and efficiency, and are as follows:

• To what extent does the FTS Program contribute to CBSA stakeholders' operational capability to make evidence-based admissibility determinations, enforcement and trade decisions?
• To what extent is the advice provided by the FTS Program used to support Agency strategic business decisions?
• To what extent are CBSA stakeholders provided with innovative scientific and engineering services and solutions that meet their needs?
• How effective is the design and implementation of the FTS Program?
• How effective is the life-cycle management plan for detection technology?
• Are there alternative ways of delivering field technology support for the benefit of border management?
• Does the performance measurement strategy support program accountability and decision-making?
• To what extent does the FTS Program support the CBSA Innovation agenda?
• Are Program resources/capacity aligned appropriately across key activities?
• Are the services to OGDs and other partners consistent with the Program's mandate?
• Are the current partnerships with other government departments cost-effective (for example, MOU with the CRA)?

The main limitations of the evaluation were the following: incomplete and under reported data; insufficient data available on the use of detection technology for commercial seizures; and a lack of robust performance measurement metrics. A detailed description of the limitations can be found at Appendix C.

2.0 Findings: Effectiveness

2.1 Analytical and forensic services

The AFSD provides analytical and scientific advisory services for suspected contraband, alcohol, tobacco and cannabis, and customs samples, as well as examines forensic documents to enable compliance and enforcement actions (that is seizures, investigations, and criminal and civil prosecutions). The quality and timeliness of these services were analyzed to assess the effectiveness of the AFSD. In terms of the quality, the evaluation focused on the credibility and reliability of AFSD analysis and advice, as well as the value of their services to frontline personnel. For timeliness, the evaluation examined the current service standards and turnaround times for AFSD services.

The AFSD received ISO (International Organization for Standardization) accreditation for the analysis of suspected contraband samples in December 2018, which provides internationally recognized credibility for the quality of its work. This accreditation has eliminated the need to resubmit samples to Health Canada's Drug Analysis Service to support charges appearing before the courts. The AFSD is currently working towards accreditation of additional testing/analysis methods.

In addition, the AFSD has demonstrated its value domestically and abroad through consistently identifying new chemical analogues, and sharing the data internationally with the United States (US), Australia, New Zealand, and the Netherlands, among others. In 2018, 35 new chemical analogues were identified, including 7 new fentanyl analogues (see Figure 3 below).

Figure 3: New chemical analogues identified by the FTS Program (1998 to 2018)

Source: Program data

The results of the evaluation survey of CBSA frontline staff indicate the value and reliability of services provided by the AFSD to frontline border operations. Approximately 85% of survey respondents agreed/somewhat agreed that the services provided by the Program are important in making determinations and decisions at their POE/location. Additionally, close to 80% of respondents agreed/somewhat agreed that laboratory analysis reports received in the last 12 months were easy to understand and interpret.

As for the examination of forensic documents, services provided by the FDE unit are sought out both domestically and internationally. For example, FDE has provided case assistance to numerous municipal police forces, the Province of British Columbia, as well as OGDs including Immigration, Refugees and Citizenship Canada, Health Canada, the Department of Justice, the Department of National Defence, Employment and Social Development Canada as well as the Competition Bureau of Canada. FDE has also completed analyses for law enforcement agencies around the world (for example: Australia, New Zealand, France, US, and the Netherlands) in an area of specialization known as the Solvent-Loss Ratio Method for ink analysis, which determines the relative age of ink on a document. The FDE's work on this method resulted in a Public Service Award of Excellence in the innovation category in 2007. In exchange, the FDE unit occasionally seeks assistance from other countries when required (for example, accessing the ink libraries of other countries).

The rise in traveller and commercial releases^{Footnote 4} , coupled with more complex samples sent in for analysis, have increased the workload in the contraband analysis unit. As shown in Figure 4 below, there was a sustained backlog in suspected contraband samples between 2013-2014 and 2017-2018, despite deploying all available resources (particularly in 2016-2017) and using overtime. There was also an increase in turnaround times for samples analysis, which had a direct impact on the front line, discussed below.

Figure 4: Suspected contraband sample volumes/volume per full-time equivalent FTE (2013-2014 to 2017-2018)

Source: Laboratory Analysis Support System (LASS) and FTS Program data

Service standards exist for most AFSD services, which were initially established in 2014 as internal targets published to the front line in reaction to operational needs.^{Footnote 5} These standards have not been properly assessed or validated since, and have increasingly not been met.

For alcohol, tobacco and cannabis and customs testing, the service standard to complete analysis is 60 days, with a target of achieving this service standard 80% of the time for alcohol/tobacco samples, and 85% of the time for customs samples. However, on average, only 61% of alcohol/tobacco analysis and 65% of customs analysis were completed within the 60-day service standard from fiscal year 2013-2014 to fiscal year 2017-2018.

In terms of suspected contraband analysis, service standards vary by mode, except for marine, intelligence and investigations, which only have service standards for rush samples (although no definition of "rush" exists). As shown in Table 2 below, service standards for suspected contraband analysis have increasingly not been met—the proportion of cases for which service standards were met decreased from 71% to 33% over the 5 year period examined. Turnaround times over 5 years also show an increase of 86% to 544% in the number of days it takes to complete the sample analysis (depending on the mode). This means that the front line is waiting increasingly longer for the results of suspected contraband analysis to be completed, with postal, intelligence and air (passenger) modes experiencing the greatest increase. The recent fentanyl crisis may be a contributing factor, which has led to a greater demand for more complex analyses, consistent with the experience of other countries, such as the US and Australia.

Focus group discussions revealed that many frontline officers are unaware of the AFSD's services or that service standards exist. Some participants indicated they send their suspected contraband samples to the Health Canada Drug Analysis Service through the Royal Canadian Mounted Police (RCMP). According to survey results, only 23% of frontline officers submitted a suspected contraband sample to the CBSA Laboratory for analysis in the last 12 months, which also suggests potential underutilization of the Program's services.

Delays in receiving test results seem to have led some BSOs to send in fewer samples, considering the proportion of volume increases. Nearly one fifth of survey respondents indicated they had not sent in requests for routine sample analysis or advice in the last year due to the length of time it would have taken to receive the results. This potentially translates into hundreds^{Footnote 6} of resultant samples not being sent to the AFSD for analysis, some of which could be resultant for enforcement purposes. However, BSOs do often hold suspected contraband which, if left unclaimed after 3 months, is sent for destruction.

Partnering with OGD and private sector labs may help manage or eliminate the AFSD backlog and/or keep up with the increased demand for services when volumes exceed the AFSD's capacity. The FTS Program may wish to review its current service delivery model for Analytical and Forensic Services and undertake a cost-benefit analysis to determine if each service line should continue to be delivered internally.

In addition, the 2017 Operational Memo on Fentanyl and Highly Toxic Substances (HTS) instructed BSOs to limit the submission of samples, noting, "only a subset of goods should be sent to the CBSA laboratory when there are a multiplicity of indicators to suspect that narcotics or HTS exist". As a result of this memo, 13% of frontline staff who were surveyed indicated they did not submit samples for analysis when they otherwise would have. They cited impacts on operations such as releasing items they otherwise would have sent for analysis, the inability to conduct controlled deliveries, referring samples to others for analysis (for example, Health Canada and other law enforcement organizations such as the RCMP) and relying on other indicators to make seizures, including the use of detection technology, discussed below.

The Program has experienced, and continues to face, challenges in having sufficient qualified staff to perform its highly specialized work. The recruitment of chemists to work in contraband analysis and forensic document examiners is particularly difficult. In the case of forensic document analysis specifically, very few fully qualified candidates exist, none from within the public service, and few even nationwide. Unless fully qualified, new hires to the FDE unit require 3 years of training and an additional 2 years of limited work before they are considered fully operational. In addition, interview evidence suggests that the highly rigorous assessment processes (set by the Program itself) result in few successful applicants.

Interview observations of program managers are supported by findings that indicate past staffing processes and the length of time involved made bringing in external candidates challenging. The Program has struggled to remain competitive with the private sector, for example when it has been restricted to offering only casual or term contracts to the few candidates identified. Compensation is also not on par with the private sector. A comparison of the average salary of CH-02 Chemists with the average income for all comparable job descriptions based on the employment income statistics (2016 Census) found CH-02 earning 35% less than the industry average. Similarly, the average salary for a Research Engineer (ENG 03) is 19% less than the comparable average salary in the private sector.

2.2 Border technology services

Detection technology refers to equipment used by frontline staff to detect or identify undeclared or inadmissible goods. The Program is responsible for the procurement, testing, installation, and maintenance of the Agency's detection technology. The installation and maintenance of select equipment are contracted out to manufacturers based on individual service agreements. The FTS Program conducts international research on available technologies, and often works with manufacturing companies to specify technical requirements and later modifies the technology to align with government regulations as well as the needs of the Agency and individual POEs. The Program procures technology at the request of the DPMC, a director level working group within the Commercial and Trade Branch. The DPMC determines which technologies will be adopted, and where and in what priority, in response to regional requests for detection technology.

Findings on the effectiveness of detection technology were based on the frequency of utilization and value of seizures when detection technology was used in the traveller stream.

Detection technology tools are supposed to assist the CBSA's frontline officers in doing their jobs; at present, insufficient data exists to corroborate this and to determine the effectiveness of individual tools. Very little data is available on the use of detection technology in seizures in the commercial stream. For Traveller's, the recording of detection technology usage in seizures in the Integrated Customs Enforcement System (ICES) is inconsistent and seems to be under reported by BSOs. A review of ICES data from the Vancouver International Mail Centre (VIMC) for fiscal year 2017 to 2018 indicated that 3,104 non-US mail seizures were completed with "no technical aids". However, all seizures of non US mail during this period would have been X rayed, as 7 new X-rays were installed in July 2016. This suggests that BSOs may not be recording the use of detection technology in the system, as required.^{Footnote 7}

Focus group sessions revealed that BSOs were not aware of the value of accurately entering data, such as detection technology usage. One focus group participant stated, "BSOs don't understand why they need to input information into ICES. The mentality is to get people through quickly instead of filling out data accurately. It's too much extra work.", while another participant noted that "more education needs to be given to officers as to why they are entering data".

Postal and air modes recorded the greatest proportion of personal seizures using detection technology (31% and 14%, respectively) over the 5 year period, as shown in Table 3 below. In contrast, only 8 percent of the almost 60,000 seizures in highway mode reportedly involved detection technology. The variance in usage rates is likely due to a combination of factors that include the commodity types being seized, the availability of detection technology, and more importantly, the level of training (which is outside the Program's control).

Almost 70% of seizures that involved the use of detection technology occurred in postal mode over the last 5 years. There is an indication that other modes may be under utilizing lab services, given that 77% of non postal survey respondents indicated they did not use lab services in the past 12 months, compared to only 14% of respondents working in postal.

Overall in the traveller stream detection technology is only reported to be used in 19% of all contraband seizures. However, the recorded use of detection technology is associated with a higher value of seizures. As shown in Figure 5 below, on average over the past 5 years, the 19% of contraband seizures that involved the use of detection technology accounted for close to 81% of the total value of traveller contraband seizures.

Figure 5: Number and value of travellers contraband seizures with and without use of detection technology (average from 2013-2014 to 2017-2018)

Source: ICES

While the Training and Development Directorate (TDD) currently has many learning solutions in place for numerous detection technology tools that are utilized in the field, the lack of an overarching training strategy for detection technology has translated into a number of training gaps in relation to detection technology usage.

A focus group of frontline personnel pointed to insufficient training on detection technology preventing some officers from using equipment for its intended purpose, specifically the small scale imagers (SSI) and LSIs. Participants noted that, even if detection technology is in good working order at their POEs, often times BSOs do not know how to use the equipment. In addition, based on a review of detection technology availability memos, the lack of proper training often results in the misuse of equipment, causing breakdowns.

Without being prompted, a number of survey respondents submitted comments on the need for additional and ongoing training relating to detection technology. The most common training needs mentioned included training on specific equipment (X-rays, fume hood and IonScan), training on container examinations, as well as general hands on training, and additional or refresher training on detection technology.

"More training needs to be provided as we have a lot of officer turnover…and many do not know how to properly use the equipment" - Survey respondent

Insufficient training may also be leading officers into the habit of only using certain detection technology tools and disregarding others. One focus group participant expressed that "training gets people reinvigorated to use specific tools".

2.2.1 Asset management

Asset management is the responsibility of the Program's Detection Technology Field Support unit within the BTD. This includes preventative maintenance, corrective maintenance in response to tickets, and life-cycle replacement. Detection technology assets are given a 10-year life-cycle following which they are expected to be replaced. Availability of tools and training for detection technology, responsiveness to service requests, breakdown days of individual assets, and roles and responsibilities for CCTV equipment were examined to assess the effectiveness of asset management.

Detection technology breakdowns are caused by various factors including the misuse of equipment, age, maintenance carried out, and external factors such as weather conditions. Having detection technology in good working order is important because effective detection technology leads to more suspected contraband being sent for testing and improved interdiction of contraband.

Figure 6: Maintenance tickets versus breakdown days for detection technology (all regions)

Source: FTS Program data

Between fiscal year 2014 to 2015^{Footnote 8} and fiscal year 2017 to 2018, the number of days detection technology equipment was broken down decreased 40%, while the number of maintenance tickets created increased 55%, as shown in Figure 6 above. In fiscal year 2017 to 2018, the total time detection technology equipment was broken amounted to almost 7,400 days (spread out across 2,428 total assets—1,808 small tools and 620 main assets). This reflects a relative improvement in asset availability (4 years earlier, the total was over 12,000 days).

With respect to specific assets classes, SSIs, LSIs and Trace Detection (IonScans) had the most maintenance tickets created as well as the largest number of breakdown hours between fiscal year 2014 to 2015 and fiscal year 2017 to 2018, as seen in Figure 7 below. Survey results show that 40% of respondents reported LSIs being unavailable more than 10 times in the last 12-month period, mainly due to breakdowns and having insufficient staff to operate the machines.

Figure 7: Maintenance tickets and breakdown duration by asset class (2014-2015 to 2017-2018)

Source: Program data

The asset which had the most significant impact on reducing the total number of breakdown days between fiscal year 2014 to 2015 and fiscal year 2017 to 2018 was SSIs. As can be seen from Table 4 below, the average number of days each of the 148 SSIs was broken down decreased from 36 to 7 days over the 4 year period. A number of other asset classes saw significant decreases in breakdowns. On the other hand, IonScans broke down 34% more often in 2017-2018 compared to fiscal year 2014 to 2015, though significantly less than in fiscal year 2015 to 2016.

The Program attributes the reduction in equipment breakdown days to the increased expertise of their maintenance staff resulting from familiarity with assets, along with carrying out more routine preventative maintenance^{Footnote 9}. The increase in maintenance tickets over the 5 year period can be attributed to the percentage of assets and detection technology utilized beyond life expectancy (30%), as well as changes in the ticket creation process.^{Footnote 10} Despite the decrease in breakdown time for a number of assets between fiscal year 2014 to 2015 and fiscal year 2017 to 2018, according to survey results, only 40% of respondents felt that service requests for detection technology equipment were responded to in a timely manner. In addition, 79% of respondents reported that their LSIs were broken down at their POE at least once in the previous 12 months. Around half of respondents indicated that both their small tools and IonScans were in need of replacement at their POE.

The consequences of equipment breakdowns on POEs vary significantly depending on the number of assets available at the POE and their portability. For example, the 16 LSIs servicing 13 POEs across the country, 4 of which are fixed and 12 are not easily relocated, and are broken down, on average, 16% of the time (as shown in Table 5 below). In contrast, the 125 portable IonScans across 104 POEs are each broken down, on average, 7% of the time. However, POEs with only one IonScan are more significantly impacted by breakdowns, with perhaps only NikTests as an alternative, compared to POEs with multiple IonScans.

Service standards do not exist for detection technology repairs by asset or by region. Rather, there is a 90% national availability target for all asset classes (95% for the fixed site LSI). This has not been an effective way to measure the impact of broken down detection technology at specific POEs, especially for asset classes which have large quantities of equipment spread out across the majority of POEs. For example, there are 125 IonScans deployed in the field, which encompasses over 99% of the asset class "Trace Detection". Using the 90% national availability target, 12 POEs could have a broken down IonScan for the full fiscal year, yet the availability service standard would have been met for this asset class. Evaluating asset availability on a more granular scale would allow the Program to be more responsive to the unique needs of each region, POE and location.

Close to 30% of superintendents, supervisors, chiefs, managers, criminal investigators, and intelligence officers and analysts surveyed felt they lacked sufficient detection technology equipment to perform their duties. The most common reasons cited include equipment being generally unreliable, outdated or non existent; and, insufficient trained staff to operate LSI equipment.

At present, decisions about what detection technology is adopted, where and in what priority in the Agency, are made by the DPMC, a director level committee within the Commercial and Trade Branch. As such, there is a lack of Agency level, strategic oversight in order to establish a systemic approach for a minimal operational posture for detection technology at each POE. Regions have indicated that, while some POEs are submitting requests for the newest and most innovative technology, others are struggling to ensure basic detection technology tools are available and working.

An important part of detection technology asset management is proper training on equipment usage. As mentioned previously, the lack of a comprehensive training strategy for detection technology has resulted in breakdowns due to user error, particularly for more advanced tools and technologies. Inadequate training also results in detection technology not being leveraged to its full potential—BSOs not properly trained on the use of detection technology may be missing opportunities to detect inadmissible items. Particularly as experienced BSOs retire, and their knowledge and expertise on detection technology usage are also lost, a robust training plan is needed.

A performance measurement strategy does not currently exist for the FTS Program, although one is under development. Some performance indicators are in use, such as for detection technology availability, but these tend to be overly broad and do not accurately capture the reality on the ground. An ongoing challenge is data availability and reliability, an issue not uncommon to the rest of the Agency, or elsewhere in the Government of Canada. The FTS related IT systems currently in place do not store information for reporting purposes, and the data that is captured is insufficient, incomplete, and/or siloed. There are plans to replace and upgrade some FTS related systems under the FSTII model; in addition, the creation of a Border Technology Network, which would link various systems and detection technology equipment, is in its infancy.

Detection technology asset management could be improved by having the FTS Program focus its resources on the assessment of technical specifications for detection technology during procurement, and limit its asset management involvement to specialized, technical assets. An alternative model could include a centralized role for the FCMB in procurement and a comprehensive and streamlined approach to training on detection technology provided via the Human Resources Branch (HRB), with the Program focusing on the technical aspects. Appendix D outlines the current model of detection technology asset management, as well as a potential alternative approach.

A review of program documentation in combination with interview data from across the Agency point to the lack of clearly defined roles and responsibilities for the management of CCTV equipment. A national strategy does not exist.

Different branches are responsible for the various facets of identification, acquisition, and distribution of CCTV, while the regions are responsible for individual CCTV maintenance agreements. The FCMB is responsible for procurement of CCTV cameras in consultation with regions. Individual regions also play a role in identifying requirements, and are solely responsible for the commissioning, maintenance and disposal of CCTV equipment. The Program is responsible for identifying the technical specifications for new audio visual monitoring and recording technologies, including CCTV equipment. It also provides functional guidance to regional operations in terms of identifying suitable audio visual technologies based on regional requirements.

In the absence of a national or overarching strategy, CCTV equipment life-cycle management is inconsistent across different regions and POEs, which leads to the inability of many individual POEs to dedicate sufficient resources to CCTV management due to competing priorities. CCTV systems have been installed without validation against technical specifications. Focus group evidence also suggests that the lack of a national maintenance strategy has significant repercussions. One participant pointed to a case of a broken-down camera preventing the submission of footage as evidence for a court case. A clearer Agency vision surrounding CCTV may alleviate these issues moving forward.

2.3 Innovation

The FTS Program is the Agency's driver of innovation in support of the Government of Canada's Innovation Agenda. The Program proactively advances technological innovation at the front line by developing prototypes and piloting border technology solutions, as well as experimenting with new approaches and advising the Agency on next-generation technologies. These new technologies improve stakeholders' operational capability to make evidence-based admissibility determinations, enforcement, and trade decisions.

The evaluation looked at the number of innovative pilot projects implemented within CBSA operations over the 5-year period and whether pilots were moved into full production following the pilot phase. Findings on the effectiveness of the Program's innovative efforts were based on survey responses, stakeholder interviews, and focus groups, on the engagement process for pilot projects and frontline support for their implementation.

2.3.1 Pilot projects

The BTD is responsible for the development of a proof of concept to demonstrate the type of technologies required to achieve the pilot's objectives. Following the proof of concept, the technology is tried as a prototype within a test environment. Upon successful testing, the piloting phase is then rolled out to the field to determine its effectiveness and efficiency in an operational setting. If the Agency wants to adopt a pilot as a permanent program, the pilot is handed over to the relevant program area for full implementation.

Pilot projects have been well received by frontline staff. Three quarters (74%) of survey respondents reported being satisfied or somewhat satisfied with the innovative pilot projects developed by the FTS Program, and 90% supported the permanent implementation of the pilots at their location.

Over the 5 year period examined, the BTD developed 12 innovative pilots (see Appendix E). Of these, the Predictive Modelling for the Previous Seizure Enforcement Risk Tool Offender Wheel Analysis and the Satellite Laboratory (Designated Safe Sampling Area—DSSA) were completed successfully and are being rolled out as part of regular Agency operations. The Remote Traveller Processing Pilot was also successfully completed (and continuously operates), but was not implemented in other locations due to funding constraints.

According to an Agency report^{Footnote 11}, the DSSA pilot at VIMC realized multiple benefits, the most important of which was a significantly reduced turnaround time for sample analysis (average of 52 minutes using the DSSA, compared to 103 days using the AFSD) and an increased acceptance rate for controlled delivery. Having the DSSA also meant sample testing was done onsite, resulting in fewer samples sent to the AFSD for analysis (increase from 5% onsite sample identification capability (using only the IonScan) to 82%). The pilot also pointed to improved BSO morale (due to fast turnaround times) and workplace health and safety (installation of fume hood and ability to identify substances from spills resulting in reduced hazardous materials decontamination).

The Immunoassay Drug ID pilot was ended mainly due to issues with vendor supply, while the Trade Fraud Detection pilot fulfilled a set objective and a continued need for the pilot did not exist. The costs and operational constraints associated with the Container Tracking pilot were found to be excessive, and as such the pilot was ended without plans for a permanent roll-out. The Faces on the Move pilot was not mainstreamed due to the necessary policy and privacy coverage not yet being in place. While pilots are not implemented permanently for various reasons, it is important to note that the ability to explore innovative technology is beneficial in itself in terms of informing future projects and innovation. As such, the success of pilots should not be assessed solely on whether or not they are rolled out as part of regular Agency operations.

In addition, the Advanced Analytics unit within the BTD uses predictive modelling and data analytics to develop pilots which enhance existing programs and methodologies for both traveller and trade facilitation and enforcement. The Variable Shift Schedule Agreement (VSSA) is an example of a pilot project which was created in response to an operational requirement as part of the CBSA Renewal Initiative. This unit also responds to requests from across the Agency for complex mathematical and statistical analyses.

Despite the success of a number of projects, there is limited and inconsistent governance surrounding pilot projects, as well as the lack of a comprehensive model for their management and ownership. Pilots generally stem from ministerial commitments, Agency priorities, or are driven by innovation within the FTS Program. As the Agency moves forward with Renewal and fostering the use of new technologies, pilot projects would benefit from being governed by a VP level body (new or existing, for example Executive Committee) to help prioritize decision-making surrounding pilots and to promote linkages to Agency priorities.

The lack of clarity over pilot project ownership and management and surrounding the roles and responsibilities of different stakeholders has proved challenging. In particular, long term roles and responsibilities as pilot projects end are not clear, as was the case with the Remote Travellers Processing pilot, meaning there is ambiguity with regards to the point at which responsibility for a pilot shifts to the business line in the case that it will be continued as a regular program. In addition, while there have been efforts to engage stakeholders in pilot projects, these have not always been effective. Some stakeholders indicated that they would prefer the Program to engage more with the regions starting from the conceptual design phase, while others noted greater collaboration with the Program during the operational research and analysis phase would provide options more relevant to their operations.

Stakeholders generally hold the FTS Program responsible for the management of the pilots, including the perceived gaps in communication and consultation. Focus group participants provided a couple of examples of such gaps, including the lack of effective consultation or non consideration by the Program of collective agreement requirements for the VSSA pilot (in fact the responsibility of HRB), as well as the lack of effective consultation and planning in the Secure Corridor Concept pilot (which was the responsibility of the commercial business line). In the case of the latter, this necessitated a resource-intensive redesign of the physical layout part-way through implementation. In both cases, the role of the FTS Program vis à vis other pilot partners was not clear to stakeholders, nor was the responsibility for who should be engaging the business line and at what phases of the pilot. In the case of the Secure Corridor Concept, focus group participants pointed out that had frontline staff had been consulted in the planning phase, they would have indicated the non-consideration of health and safety issues, preventing the need for the redesign.

2.3.2 Research and development

The FTS Program collaborates with partners and shares best practices on innovative research and development. It coordinates regularly with OGDs, academic organizations, and industry partners through the Canadian Safety and Security Program^{Footnote 12} on studies, concepts, demonstrations, pilots, as well as research and development for new technology.

According to the survey, almost 6 in 10 frontline staff (58%) felt that they did not have the latest cutting edge technology at their POE. Examples of cutting edge technologies include the US Eye-dentify which uses eye tracking software to identify imposters, the US's current evaluation of Touch-Free Fingerprint Scanner (which allows a traveller's fingerprints to serve as their boarding pass and identity document), and the United Kingdom's ePassport gates (which use facial recognition to quickly and securely process passengers).

While the FTS Program collaborates with partners to provide innovative solutions for increasing interceptions at the border and facilitating contraband identification, the Agency's uptake of new technologies is subject to many considerations (for example: public opinion, funding, length of procurement processes, privacy assessments, etc.).

"The Agency's procurement process takes too long such that by the time detection technology is implemented, it is already out of date. Due to budgetary constraints we seem to only receive the basic equipment to do the job but nothing cutting-edge." - Survey respondent

While the FTS Program continually tests and develops innovative technology, it lacks dedicated resourcing for "strategic and applied" research and development. Based on interviews with the Program managers, less and less time is dedicated to innovation due to increasing work pressures in other areas. To this end, a CBSA Innovation Hub is currently under development. The hub will be a dedicated, multifunctional and sustainable rapid-prototyping team for the implementation of IT-enabled technologies focused on deployable border technology. This would include the addition of 5 permanent FTEs on an incremental basis to the existing 7 FTEs working on innovation and pilots, ultimately creating a core capability of 12 FTEs. The governance for the Innovation Hub is expected to be established in this fiscal year.

In 2023, the Program will be physically relocated to a federal science campus as a result of the Federal Science & and Technology Infrastructure Initiative ("Laboratories Canada"), along with the scientific services of Health Canada, Canadian Food Inspection Agency, Agriculture and Agri-Food Canada, among others. Being housed under one roof in and of itself will provide opportunities for increasing collaboration with OGDs, leveraging technical expertise, and sharing resources.

2.4 Program awareness and utilization

The FTS Program provides a number of different services targeted to various clients within the Agency. Although the Program has made efforts to raise awareness in the Agency in the past^{Footnote 13}, the majority of awareness activities were targeted to small groups or relied on staff's self-interest, which was limited.

Interview evidence indicates that existing clients are sufficiently aware of the lines of service offered by the Program for meeting their existing needs; however, not all are aware of the other lines of service offered. Interviewees not currently utilizing the Program's services had a limited awareness of the Program and what it has to offer. As discussed previously, focus group discussions revealed that many frontline officers are unaware of the AFSD's services, and some participants indicated they send their contraband samples to the Health Canada Drug Analysis Service through the RCMP, instead of sending it to AFSD, as required. Given that only 23% of frontline officers submitted a suspected contraband sample to the CBSA Laboratory for analysis in the last 12 months, there may be an issue of underutilization of the Program's services.

In addition, according to the survey, one third of respondents (32%) felt there wasn't a process in place to communicate their sample analysis needs to the FTS Program.^{Footnote 14}

Few in the Agency refer to the Program by its title of Field Technology Support Program, instead referring to it as the "CBSA Lab". A communications strategy may help with branding, in order to increase awareness about the Program and the services it offers. A new name that better reflects the lines of services offered by the Program could be considered.

The FDE unit is often confused with the Agency's National Document Centre and Regional Document Analysis Units. While all of them examine documents for authenticity, each provides a different level of expertise/analysis. The FDE unit is the only one which provides forensic level analysis. An Internal Audit consulting engagement of March 2019 found that confusion regarding the roles of different document examiners in the Agency presented a risk in terms of program oversight and delivery of services. It determined that roles and responsibilities needed to be clarified between key players and processes needed to be developed to better work together and within areas of expertise. The consulting engagement also noted a lack of awareness among the document examination centres regarding the other units and their level of expertise. As a result, there may have been missed opportunities in the past for the National Document Centre to reach out to the FDE unit for assistance, as well as for the FDE unit to relieve its caseload by passing non-forensic cases to the National Document Centre.

Under CBSA renewal, new data analytics functions and capabilities are being developed in the Agency, such as through the addition of a data analytics group under the Chief Data Officer. The distinction between, and the roles and responsibilities of, these groups and the FTS Program's data analytics function are not yet clear. Opportunities to streamline these areas and to promote collaboration should be considered.

Stakeholders within the Agency, including I&E Branch and regional operations, have indicated that there are opportunities to improve engagement and information sharing with the FTS Program. In light of the Agency's new FMM, the Program's relationship with partners throughout the Agency should be reviewed to ensure regular engagement takes place. Interview feedback suggests that strategic oversight and planning on the acquisition and management of detection technology could be improved. For example, the installation of a fixed site Large Scale Imager (LSI) at one POE may make contraband smugglers shift their operations to other POEs in the region to avoid this form of detection technology.

The usage of the AFSD's laboratory services varies greatly from region to region, even within the same mode. Figure 8 below provides an example of the disparity in samples sent to the FTS contraband laboratory by the 3 postal centres. Over the 5 year period between fiscal years 2013-2014 and 2017-2018, the Quebec mail centre submitted on average one third of the suspected contraband samples submitted by the Greater Toronto Area (GTA) mail centre, despite completing approximately 500 more narcotics seizures than GTA mail centre per year. Pacific sent by far the largest number of samples for analysis and also had the highest narcotics seizure level of all mail centres.

Figure 8: Number of narcotics seizures and samples sent to contraband lab in postal mode

Source: ICES, LASS

ICES data also showed that the GTA mail centre used detection technology in 39% of narcotics seizures in postal mode from fiscal year 2013-2014 to fiscal year 2017-2018, while Quebec only used detection technology in 5 percent of postal narcotics seizures and 16% for Pacific. Some (or much) of the difference may be accounted for by BSOs under recording of the use of detection technology in seizures, which has been discussed previously. It is also likely that, compared to other regions, postal BSOs in Quebec are less aware of, or are unfamiliar with, the process in place to communicate with the FTS Program ^{Footnote 15}.

According to Program and ICES data, at the regional level and for all modes, Pacific and GTA regions consistently send the most suspected contraband samples to the lab for testing; Pacific region utilizes detection technology in seizures more frequently than other regions.

3. Findings: Efficiency

The evaluation's findings on the efficiency of FTS Program were based on the assessment of core program expenditures, the number of FTEs, the volume and cost of outputs, areas of overlap with services provided within and external to the Agency, and overtime hours. Program expenditures and outputs were derived based on data from the FCMB and on the Program's administrative and financial records.

Where data exists, the Program was found to be generally efficient in most areas. The specific elements analyzed to draw this conclusion are detailed below.

3.1 Expenditures and outputs

The Program's expenditures are comprised of core program spending (A-base), the Detection Technology Control Fund (detection technology acquisition) and radio program funding. Total program spending in 2017/18 was $41.4M, over half of which ($21.7M) was on a special investment in new radios.^{Footnote 16} Core expenditures remained fairly stable over the 5 year period, fluctuating between $11.8 million and $14.5 million (see Figure 9 below). Both core program spending and FTEs increased by just under 10% over the 5 year period, although, as noted previously, the demand for services increased significantly during the same time period, most notably in suspected contraband, alcohol, tobacco and cannabis and FDE.

Figure 9: FTS Program A-base expenditures and FTEs (2013-2014 to 2017-2018)

Source: CAS, FTS Program data

Detection technology control fund expenditures fluctuated significantly, depending on asset procurement and replacement needs, and due to the absence of multi year capital expenditure planning.^{Footnote 17} Within the evaluation period, control fund expenditures ranged from $4.5M in fiscal year 2016 to 2017 to $16.9M in fiscal year 2013 to 2014.

For the AFSD, significant fluctuations in outputs were observed for most units year-over-year between fiscal years 2013-2014 and 2017-2018, with a significant decrease recorded in the most recent year examined. ^{Footnote 18} As shown in Figure 10 below, although the output levels for suspected contraband and forensic document testing declined in fiscal year 2017 to 2018, these were still on par with the output levels for fiscal years 2013-2014 and 2014-2015. However, in the case of alcohol, tobacco and cannabis as well as customs testing, the output levels were significantly lower than what they were 5 years previously (they declined by 37% and 34%, respectively over the 5-year period).

Figure 10: AFSD outputs by unit (2013-2014 to 2017-2018)

Source: FTS Program data, LASS

Outputs for the BTD increased overall between fiscal years 2013-2014 and 2017-2018, as reflected in Figure 11 below, except in the areas of training and reports. The number of ongoing pilots more than doubled from 3 to 8 over the 5-year period. However these pilots have all varied in complexity, resource requirements and duration. The number of detection technology field support (DTFS) maintenance tickets resolved and responses to RADnet alarms increased by 55% and 60% respectively over the 5-year period. Outputs related to the delivery of training and reports significantly decreased from fiscal year 2013 to 2014 onwards due to other operational pressures in the division.

Figure 11: BTD outputs by unit (2013-2014 to 2017-2018)

Source: FTS Program data, LASS

Between fiscal years 2013-2014 and 2017-2018, the AFSD's overall salary cost per output increased only marginally, although salary cost per output for the suspected contraband, alcohol/tobacco and FDE units all increased significantly. As shown in Table 6, the percentage increase in the salary cost per output for these 3 units was considerable, and was as much as 58% for the alcohol, tobacco and cannabis unit. For the latter, the increased salary cost per output was driven by a decrease in overall output combined with stable expenditures. The increase for both the suspected contraband and FDE units is attributed to increased salary expenditures while output remained relatively consistent in fiscal year 2017 to 2018 compared to fiscal year 2013 to 2014.

The increase in salary costs, combined with stable outputs for the contraband and FDE units, or stable outputs and decreased salary costs in the case of the alcohol, tobacco and cannabis unit, point to a relative decrease in efficiency per output.

The customs analysis unit was the outlier, as it experienced a 15% reduction in its salary cost per output over the 5-year period. Although this was the only AFSD unit to record a decrease, it was also the unit with the highest average salary cost per output^{Footnote 20}. As a result, the AFSD salary cost per output actually only increased by 6% overall (and the total cost per output increased by only 2% in the division). The reduction in salary cost per output for customs analysis was due mainly to restructuring within the unit and resulting FTE reductions along with the year over year fluctuations in demand from the front line.

An analysis of the average salary cost per output was not conducted for BTD or SESOD. BTD outputs vary significantly in size and complexity and are not comparable as they are for AFSD. SESOD does not produce outputs that are easily quantified or compared.

Overtime usage in the AFSD and the SESOD remained relatively steady between fiscal years 2013-2014 and 2017-2018, as shown in Figure 12. In contrast, overtime hours in the BTD increased 177% (equivalent to 4 FTEs). This is largely attributed to assignment of RadNet responsibilities to this cost centre in fiscal year 2015 to 2016 which came with the requirement for technicians to be on stand by, increased resource requirements for pilot project implementation, and an increased workload related to CCTV technical specifications. The significant increase in overtime for BTD is indicative of the high workload in this area, and should be managed. Continuous overtime, even at moderate levels, often leads to unhealthy workplace cultures.

Figure 12: FTS Program overtime hours by division (2013-2014 to 2017-2018)

Source: CAS

Between fiscal years 2013-2014 and 2017-2018, the FDE unit completed uncompensated work for external clients with a combined estimated value of over $220,000. This represents, on average, 4% of the unit's annual expenditures. In the absence of a cost-recovery mechanism, there were no attempts to recover the costs given that the time/effort required would have exceeded the value of the work in most cases.

While this uncompensated work represents a financial inefficiency for FDE, it was found to hold value in that it was deemed unique, of high legal importance, and/or offered staff skill development or maintenance.

3.2 Overlap

The FTS Program is unique in its responsibility for analyzing samples and managing technology from a border management perspective, and has capabilities and expertise not available outside the CBSA. The Program engages with OGDs, the private sector, and international partners to collaborate and share best practices on methods and emerging trends. In some cases, it also sends samples for third party analysis to leverage additional expertise when the need arises.

As depicted in Figure 13 below, there exists some overlap with Canadian Food Inspection Agency (CFIA) laboratories in terms of food analysis (performed by the Customs Analysis Unit), and with Health Canada for drug analysis (performed by the contraband analysis unit). However, in terms of the latter, Health Canada's mandate is very narrow in terms of what samples it will analyze, and it performs confirmation rather than identification analysis.

Figure 13: Overlap of services offered with OGDs

No overlap exists between FDE and OGDs as the CBSA is the only federal entity conducting forensic document examination. Other forensic and non forensic document analysis units within and outside of the Agency do not have the level of scientific expertise held by examiners in the FDE unit.

3.3 CBSA/CRA MOU

In 2004, following the establishment of the CBSA, a MOU was developed to govern the scientific laboratory which was formerly shared between the CRA and the CBSA. As a result, the CBSA took ownership of the facilities and the CRA funded certain FTEs in exchange for a fixed number of annual sample tests, mainly alcohol, tobacco and cannabis analyses, and forensic document examinations. As part of the MOU, the CBSA agreed to devote 8.5 FTEs to support an expected CRA workload of 2,030 alcohol, tobacco and cannabis analyses each year, and to devote 6 FTEs to examine an expected 85 cases for FDE annually (updated to 3,000 forensic documents in 2008). In 2008, the CBSA requested and received funding for 2.5 additional FTEs for the administrative work associated with the CRA FDE casework, bringing the total to 8.5 FTEs for the examination of forensic documents.

A complete review and updating of the targets in the CBSA/CRA MOU has not been conducted since they were established 14 years ago (aside from the additional resources received for FDE in 2008). While targets are not being met for specific lines of service, the overall resource requirement is in line with total services conducted on behalf of the CRA and is therefore found to be cost effective.

Services provided to the CRA (and other OGDs) generally align with the Program's mandate in that they work to leverage science and technology to accomplish a greater government agenda, however, are only partially aligned with the border facilitation and security mandate.

4. Conclusion and recommendations

The evaluation examined the effectiveness and efficiency of the FTS Program. It is a new ‘program' under the DRF and, as such, a robust program management framework is still under development. In the absence of a performance measurement strategy, the evaluation found that the FTS Program's performance indicators are overly broad and do not accurately or adequately capture progress towards program outcomes. Data availability and reliability is an ongoing challenge for the Program, as is the case elsewhere in the Agency. Improvements in monitoring, analyzing and reporting on program performance are necessary to be able to better assess the effectiveness and efficiency in future evaluations of the Program.

This evaluation provides baseline data and information upon which future progress can be measured, but the Program will need to focus on strengthening the foundational elements that will shape its program management and direction going forward. Recognizing that the Agency operates in a complex and dynamic environment, the Program needs to be agile in order to meet current and future challenges.

Detection technology equipment is aging and there is a perception among front-line staff that they lack cutting-edge detection technology to do their jobs. For the Agency as a whole, the lack of a holistic and coherent model for detection technology asset determination, acquisition, distribution and maintenance has resulted in gaps in detection technology availability and potentially missed opportunities for enforcement. Insufficient training on detection technology has led to preventable breakdowns of tools and technologies (such as those due to user error), as well as potential under usage of detection technology by BSOs. There is a need to review the approach to the detection technology assets and for the FTS Program to re focus on its core role of evaluating and determining the technical specifications of detection technology in the CBSA.

There has been a lack of strategic direction to guide the FTS Program's priorities and activities. Establishing a more robust governance structure will ensure that the Program is better positioned to respond to the Agency's core and strategic business priorities. Pilot projects would also benefit from a comprehensive framework, tailored to each project, that provides prioritization of pilots based on Agency needs, clear guidance and articulation of roles and responsibilities, stakeholder engagement and realistic timelines.

Where data exists, the Program was found to be generally efficient in most areas.

Recommendations:

R1. The VP of ISTB, in consultation with clients, should examine the service delivery model for the analysis of suspected contraband, alcohol, tobacco, cannabis, and customs samples. This includes:

• a. reviewing and revising existing service standards
• b. undertaking a cost-benefit analysis to determine if efficiencies can be realized by partnering with OGDs and private sector labs to eliminate the backlog
• c. following the revision of service standards, developing a communications plan to review the branding of the FTS Program and to improve the awareness of the FTS Program's roles and responsibilities, services offered and service standards
• d. As part of the broader performance measurement strategies currently under development, ensuring that service standards are regularly monitored and reported on

R2. The VP of FCMB should, in consultation with relevant branches, develop a material management strategy for the Agency's detection technology assets (including determination, acquisition, distribution, maintenance and training), leading to the development of an overall material management strategy for Agency assets.

R3. The VP of ISTB, in consultation with relevant branches and the regions, should lead the development of an alternative model for the ownership and management of FTS-led pilot projects, which clearly articulates the roles and responsibilities of respective branches and/or regions at the various phases (planning, development, implementation, and mainstreaming). Once developed, this model should be proposed to the Executive Committee for approval.

R4. The VP of ISTB, in consultation with the CTO Transformation Branch and the SPB, should review the governance of innovation and research and development in the Agency to ensure that strategic direction and guidance are provided to the FTS Program.

R5. The VP of FCMB, in consultation with other relevant branches, should develop a national strategy for the management of CCTV.

Appendix A: Management response and action plan

Appendix B: Field Technology Support Program logic model

Appendix C: Data limitations

For this evaluation, the following limitations should be considered:

1. The analysis on the use of detection technology depended on ICES data which is incomplete and questionable due to BSO input, leading to possible under-reporting or misreporting
2. The use of detection technology in seizures is not able to be recorded in the Accelerated Commercial Release Operations Support System (ACROSS), so its usage could not be measured for the commercial stream
3. As a new program, FTS has not yet established complete performance indicators to support analysis of performance in sample (contraband, alcohol, tobacco, cannabis and customs) testing
4. When maintenance tickets are closed, information is not recorded consistently, as it is based on technician input, with no standard definition of "start time" and "end time" of breakdowns. In addition, ticket creation has evolved over time, which made looking at tickets over a 5-year time period challenging, for example: replacement of consumables and small tools were only counted as tickets in more recent years.
5. The benchmarking research relied upon the limited amount of information available through open source documents and information

Appendix D: Detection technology asset management model: Current and potential

Appendix E: Pilot projects timeline

Date modified:

2020-05-28