Evaluation of the Intelligence Collection and Analysis Program: Part 2 findings, conclusion and appendices

Evaluation findings – efficiency

Consistent with other findings in this evaluation, the lack of usable performance measures also resulted in the inability to fully assess the program's efficiency. Proxy measures were used to illustrate value where possible; however, a thorough analysis of efficiency was not possible. A number of areas impacting program efficiency are discussed below and in preceding sections of the report, including updating outdated systems, improving data access and analysis capacity, and improving the governance relationship between HQ and the regions.

Program efficiency

When comparing commercial and traveller volumes by region to IA and IO full-time equivalents (FTEs), resources appeared to be generally aligned to aggregate volumes as shown in the FTE data below and in figure 11, [redacted].

FTEs per million commercial/traveller transactions

Lowest volume to highest:

• SOR (lowest)
• GTA
• PAC
• PRA
• QUE, NOR
• ATL (highest)

While the alignment to aggregate levels of commercial and traveller volume can be observed, program resources were not allocated on this basis. An assessment of volumes relative to program outputs by mode could not be conducted given the limitations of performance data.

When assessing the distribution of intelligence FTEs across regions, traveller and commercial volumes are only one driving factor. The risk profile of each region is also an important consideration, as is addressing national intelligence priorities. As previously indicated, the program does not conduct an assessment of risk based on the region or mode to inform resource allocation decisions.

Source: COGNOS (December 2021) and CAS (November 2021)

The overall perception from program management is that they are able to generate valuable intelligence products despite limited resources. Nonetheless, there was a general consensus that more resourcing is needed for the program to reach its full potential. For example, regional sentiment was that resources are insufficient for the workload, with some of the smaller regions indicating that IAs and IOs are working on thousands of files, thus affecting their ability to advance any of them in a meaningful way. Insufficient or unreliable data, coupled with the lack of risk assessments by region or mode, restricted the ability to analyze the extent to which resources were allocated optimally; no quantitative data was available to support stakeholder views.

As previously discussed, regional intelligence resources are allocated by the RDGs under the FMM. HQ feels it does not have much oversight in terms of where resources are being directed and what areas require more resourcing.

While return on investment is difficult to quantify, there is anecdotal evidence of value for money generated by the program. For example, from 2016 to 2017 to 2020 to 2021, the program's annual operating costs ranged between $65 and $80 million (see Figure 2 showing refined program expenditures that exclude the cost of the eManifest project and the International Program). Over the same five-year period, the agency's investment in the program yielded a return ranging from 240 to 430% based on only the top 10 intelligence-led seizures per year, and 390 to 750% based on all intelligence-led seizure values for the same period compared to total refined program spending (Figure 12).

Source: ICES, December 2021

Taken at face value, these figures show evidence of a significant return on investment for the program. However, caution should be taken when relying on ICES data as previously discussed (see Performance measurement and data limitations). It should also be acknowledged that the value of the program outputs extends beyond its impacts on seizures, so the above is only one anecdotal example of return on investment in one very specific area. Data limitations prevented an assessment of the impacts of the program on fraudulent immigration, human smuggling and trafficking, and other areas where the program also plays an important role. The program does not currently report on any immigration-related KPIs as part of the corporate reporting processes. Program staff, however, acknowledge the gap and plan to address it in the updated PMF. In addition, as discussed in the section on Performance measurement and data limitations, the program (and the agency as a whole) does not measure the socio-economic impacts that result from customs and immigration enforcement intelligence, which go beyond the street value of illegal drugs seized. This would suggest that the "value" of intelligence-led seizures is indeed much higher than what is captured in agency systems.

Data and systems

Interviewees felt that the CBSA is a data-rich organization, but also indicated that accessing and reconciling data from different sources is challenging and the program lacks the tools to leverage data analytics. [redacted]. Furthermore, restrictions on accessing agency data warehouses render some data inaccessible, particularly for regional intelligence staff. The program has mapped out current data systems against needs, which has highlighted gaps, and has identified more advanced tools that could collect, store, integrate, process, report and share data and intelligence more efficiently.

[redacted]

Interviewees indicated that more advanced software options are in use by partners, including other federal government departments. There is a belief that better access to systems and software tools could create efficiencies in production of intelligence products by automating searches and allowing intelligence staff to focus on anomalies and areas of concern.

The evaluation did not perform a review of IT systems and tools; however, the program has conducted an assessment of data needs versus existing tools, which showed that system gaps exist. The evaluation recognizes that challenges with data reliability, storage, access, extraction and related analytical capabilities are common across the agency, and cannot be resolved meaningfully by focusing solely on the requirements of the Intelligence Program. Notwithstanding the planned work in the agency's modernization priority to leverage technology to optimize the power of data analytics and invest in data literacy, the program is encouraged to work with ISTB and the Chief Data Office to articulate program-specific needs and work to identify possible solutions.

Recommendation 3: The VP of I&E should articulate the data/IT business requirements for the Intelligence Program and engage with the Chief Data Office and ISTB to conduct an options analysis to determine appropriate technology solutions.

Conclusion and recommendations

The CBSA's Intelligence Collection and Analysis Program remains relevant through its provision of tactical, operational and strategic intelligence products, and partnerships with domestic and foreign LEAs. It makes an important contribution to the achievement of the CBSA's mandate by identifying and mitigating safety and security threats while facilitating legitimate travel and trade across the border. However, the inherent sensitive nature of an intelligence function, coupled with data limitations, greatly limited an assessment of the effectiveness of the CBSA's intelligence collection and analysis activities.

While there were gaps and challenges related to data and performance measurement, the evaluation found some evidence of the program's value to the agency. To some extent, the program appears to be generating products that inform key decision makers of threats and trends, disrupt criminal activity, and contribute positively to helping domestic and international LEA partners achieve their mandates.

Several key areas for improvement were identified, including in the areas of training, systems and tools, and oversight by the functional authority under the FMM. As a matter of priority, solutions are needed for measuring and monitoring performance so the program can determine its effectiveness and value-added. Opportunities to strengthen the program have been put forward below in the form of three recommendations, the implementation of which will help strengthen the program.

The findings of the evaluation led to the following recommendations:

Recommendation 1

The VP of HRB, in consultation with the VP of I&E, should develop a roadmap to further advance the development and delivery of core training to intelligence analysts and officers so as to improve access and ensure training needs are met.

Recommendation 2

The VP of I&E should update and implement its Performance Measurement Framework, and use it as a tool for increasing horizontal reporting and accountability from Intelligence Program stakeholders (including the regions) under the CBSA's new Functional Management Model.

Recommendation 3

The VP of I&E should articulate the data/IT business requirements for the Intelligence Program and engage with the Chief Data Office and ISTB to conduct an options analysis to determine appropriate technology solutions.

Appendix A: Management response and action plan

Appendix B: Intelligence collection and analysis logic model

CBSA mandate: The Agency is responsible for providing integrated border services that support national security and public safety priorities and facilitate the free flow of persons and goods, including animals and plants, that meet all requirements under the program legislation.

Core responsibility: The CBSA assesses risk to identify threats, manages the free flow of admissible travellers and commercial goods into, through and out of Canada, and manages non-compliance.

Business lines

• Information Collection
  • Activities:
    • Collect data & information from various internal/external sources, such as:
      • surveillance activities
      • confidential human sources
      • classified and open sources
      • partners (including via CBSA LOs)
  • Outputs:
    • Intelligence Bulletins
    • Intelligence Advisory
    • Operational Bulletins
    • Threat and Vulnerability Assessments
    • Alerts
    • Trend Analysis Reports
    • Lookouts
    • Warning Products
    • Briefing
    • Strategic Threat Assessments
• Analysis and Production
  • Activities:
    • Data and information are evaluated for accuracy, reliability and relevance, and analyzed to produce intelligence
    • Develop timely, accurate and actionable tactical, operational, and strategic intelligence products that inform a variety of decision makers at all levels within the CBSA on issues related to the CBSA mandate and priorities
  • Outputs: Same as Information Collection
• Compliance and Oversight
  • Activities:
    • Maintain strong working relationships with regions and other government departments and provide expertise and functional direction on special cases
    • Develop policies, procedures and guidelines related to the horizontal I&E policies and provide functional direction on same; (CHS, Surveillance, JFO, WPP, Covert Ops)
  • Outputs:
    • Policies and procedural guidelines
    • Annual reviews of JFO's, Surveillance and CHS, and subsequent program recommendations
    • Risk Management framework
• Partnerships and Support
  • Activities:
    • Actively engage with internal, domestic and international partners to increase the intelligence and targeting program profile and increase collaboration (i.e. Traveller, Commercial, CSIS, RCMP, B5, CBP, Frontex)
  • Outputs:
    • Strategic Partnerships
    • Horizontal GoC Strategy reporting
    • Development of MOU's/Annexes
    • Outreach planning and implementation
    • Delivery of MC/TB sub commitments
    • Actioned RFI's
    • Biometrics Exchange

Program outcomes

• Immediate: Timely, relevant and actionable intelligence production to identify threats and heighten situational awareness
• Intermediate: Tactical and Operational decision makers make informed enforcement related decisions; Strategic decision makers make informed planning, prioritization and management decisions , with a focus to the allocation of Agency resources and efforts dedicated to the highest risks and threats to Canada and Canadians
• Ultimate: Intelligence contributes to the identification, mitigation and neutralization of risks and threats to the safety, security, and prosperity of Canadians and Canada

Appendix C: Alternate performance reporting strategies

Based on a literature review of performance reporting strategies done by other intelligence organizations as well as feedback received from law enforcement partners, the following suggestions are put forward to assist the program with alternate performance reporting strategies:

1. Use of proxy measures that provide quantifiable data to evaluate the contributions of the program to achieving expected results without presuming to establish direct links (attribution) to prevention
2. Measure the components and sub-components (outcomes) in the prevention process by focusing on the process of intelligence cycle (i.e., how the unit is performing in the process of information gathering, analyzing, sharing information, etc.)

   However, it is crucial after development of a set of ideal performance measures for each intelligence function to link how accomplishing it contributes to the performance of the entire system

3. Collecting client feedback and satisfaction with intelligence products
4. Use of case studies that anecdotally illustrate the impacts of intelligence products

Appendix D: [redacted]

[redacted]