Audit of Project Management

Audit of Project Management (PDF, 1.2  MB)
[help with PDF files]

Table of contents

1.0 Introduction

1. The Canada Border Services Agency (CBSA) is working to modernize its programs, services, and infrastructure to keep up with global advances in technology and to meet the demands of rising volumes of trade and travel. As of , the CBSA had 26 active information technology (IT) enabled projects and 7 real property projects.Footnote 1 CBSA project spending averaged $226.5 million per year over 3 fiscal years from 2014-15 to 2016-17.Footnote 2

2. The Treasury Board (TB) defines a project as an activity or series of activities that has a beginning and an end.Footnote 3 The TB further specifies that a project is required to produce defined outputs and realize specific outcomes in support of a public policy objective, within a clear schedule and resource plan, within specific time, cost and performance parameters. The CBSA uses this definition in its Project Management Framework (PMF), and goes further to explain that certain investments are not considered projects, including building maintenance or capital repair, regular operations or systems maintenance, unchanging repeatable processes, and IT operational changes. Investments are assessed as potential projects through the CBSA Investment Planning Process, which involves 3 approval phases (gates) consisting of needs identification, opportunity intake and concept. Once an investment is approved as a project, it is then managed through 4 additional gates of the Project Management Lifecycle depicted below.

Figure 1: Project gating

Figure 1 description

Stages or “gates” that a project must undergo from start to finish. The process is divided into 2 parts:

  • Investment planning process
    • Gate 1: Needs identification
    • Gate 2: Opportunity intake
    • Gate 3: Concept
  • Project management lifecycle
    • Gate 4: Initiating
    • Gate 5: Planning
    • Gate 6: Executing
    • Gate 7: Closing

A post-implementation occurs after the project management lifecycle. The scope of this audit is limited to the project management lifecycle.

3. The Enterprise Project Management Office (EPMO), which transferred from the Information Science and Technology Branch (ISTB) to the Finance and Corporate Management Branch (FCMB) in , is the centre of expertise and authority for project management for the CBSA. The EPMO serves as a resource for project managers and other stakeholders offering standardized processes, practices and tools, including the PMF. The FCMB is also responsible for the Benefits Management and Realization Framework, which complements the PMF and provides tools, templates and guidance for the implementation of benefits management practices.

4. With the implementation in of the new Functional Management Model, responsibility for leading and managing IT-enabled projects was transferred to the business owners (program areas) from the ISTB. As such, business lines are now accountable for their projects, including project management responsibilities and business readiness. The ISTB continues to oversee technical aspects of IT-enabled projects as a service provider.

2.0 Significance of the audit

5. This audit is of interest to management given the magnitude of Agency investments in projects and their importance in helping the Agency realize transformation and border modernization initiatives.

6. The audit objective was to assess the extent to which project management governance, processes and controls are established and followed to support the successful delivery of Agency projects. The audit examined 5 active and 3 closed projects between and , which are listed in Annex A.

7. In , the Treasury Board issued a new Policy on the Planning and Management of Investments, along with a Directive on the Management of Projects and Programmes. Given the timing of this audit, we did not assess projects against the new policy and directive. The detailed audit methodology, scope and criteria can be found in Annex B.

3.0 Statement of conformance

8. This audit was conducted in conformance with the Treasury Board Policy and Directive on Internal Audit and the Institute of Internal Auditors’ (IIA) International Professional Practices Framework (IPPF). Sufficient and appropriate evidence was gathered through various procedures to provide an audit level of assurance. The Agency’s internal audit function is independent and internal auditors performed their work with objectivity as defined by the IIA’s International Standards for the Processional Practice of Internal Auditing.

4.0 Audit opinion

9. The Agency has established project management practices that are aligned with best practices and, at the time of the audit, updates were underway to comply with the new Treasury Board policy and to enhance accountability for project delivery and outcomes. Projects examined were found to have proceeded through the various gates without conforming to the project management framework. Opportunities exist to improve on the identification and tracking of projects, and to mature Agency project management practices through continuous learning, adequate stakeholder engagement and strengthened change management. Finally, the Agency needs to attract and develop adequate project management expertise and experience.

5.0 Key findings

10. The Agency has recently made significant changes to the organization and accountability for projects, which will strengthen the accountabilities of key stakeholders.

11. The Agency Project Management Framework (PMF) outlines the project deliverables for each gate in the Project Management Lifecycle, including appropriate consultations. Each of the projects examined had passed gates without having completed all the required gate deliverables. Change management, a key expectation through the lifecycle of a project, was also not found to be established, as minimal documentation was on file.

12. The EPMO has a key responsibility to ensure project readiness prior to proceeding for gate approval. Without a strong challenge and compliance function, projects run the risk of moving through key gates without the appropriate building blocks in place to ensure their success.

13. The Agency Master Project List has not been an effective tool for tracking projects or identifying relative priorities, interdependencies and efficiencies. As well, gate approval decisions were not always properly or consistently documented in project files, committee records of decision, or in the Master Project List and conditional approvals were not followed up on.

14. Efforts to address the known Project management capacity issue are underway. Increased focus on fostering the development of skills and expertise essential to project success will be necessary. Implementing a process to assess lessons learned and continuously improve Agency project management processes would mature project management practices.

6.0 Summary of recommendations

15. The audit makes 3 recommendations relating to:

7.0 Management response

Overall the Vice-President of the Finance and Corporate Management Branch (FCMB) agrees with all 3 recommendations of the audit and will continue to invest in strengthening the discipline of project management within the CBSA. The revised CBSA Project Management Framework supports compliance with the Treasury Board Policy on the Planning and Management of Investments and Directive on the Management of Projects and Programmes to ensure a standardized approach to managing projects and that projects are managed in a manner consistent with complexity and risk. The Agency will further mature the project management discipline by developing and strengthening the project management competencies within the CBSA.

8.0 Audit findings

8.1 Establishing clear accountability

Audit criteria:

Accountabilities of the key stakeholders such as project sponsors, project managers and the Enterprise Project Management Office (EPMO) are understood and exercised.

16. Establishing clear accountabilities is a key factor in the likelihood of success of a project.Footnote 4 The Agency Project Management Framework (PMF) outlines the roles and responsibilities of the main stakeholders:

17. The audit examined whether key stakeholders understood their roles and exercised them as expected. Interviews confirmed that the main stakeholders generally demonstrated an understanding of their roles and responsibilities. However, in practice, some of these roles were not always exercised as expected.

18. Three of 6 project sponsors interviewed had the perception that they were not kept involved in the management of their project; they felt insufficiently consulted by the project managers and not provided with timely information before having to provide feedback on project deliverables. They noted however that this was less and less the case and has been improving recently.

19. At the time of the audit, the EPMO was not carrying out the full extent of its role. The PMF articulates a mandate of centre of authority for the EPMO, relying on it to ensure compliance of Agency projects with the standard project management processes and for providing strategic direction and oversight.Footnote 7 The EPMO did not always ensure gate readiness prior to projects being scheduled for committee approval. For the 7 projects examined that were subject to producing gating deliverables, all proceeded through gates without having completed all the required project deliverables.Footnote 8

20. The Agency has recently made significant changes to the organization and accountability for projects. In , the EPMO was transferred from the Information Science and Technology Branch (ISTB) to the Finance and Corporate Management Branch (FCMB). In , as part of the implementation of the a new Functional Management Model, responsibility for leading and managing IT-enabled projects was transferred to the business owners (program areas) from the ISTB. As such, business lines are now accountable for their projects, including project management responsibilities and business readiness.Footnote 9

21. The new Policy on the Planning and Management of Investments and the Directive on the Management of Projects and Programmes provides the Agency with the opportunity to update the PMF, including clarifying project management roles and responsibilities.

22. The Agency has made significant changes to clarify project management accountabilities and responsibilities. It will be important for the FCMB to ensure that the roles and responsibilities are being carried out effectively, including the role of the EPMO.

8.2 Project management processes

Audit criteria:

Key lessons from the Government of Canada Pay Transformation Initiative include the need to properly define initiatives, to articulate discrete projects and how they interrelate, and outline effective accountability.Footnote 10

Identifying and tracking projects

23. The CBSA Project Management Framework (PMF) was last updated in 2017 and describes the enterprise approach to project management. The PMF defines what is and what is not considered a project. Agency Projects are categorized according to their level of materiality, complexity and risk. At the time of the audit, projects under $1M required only branch level governance and oversight. Larger, more complex projects required different levels of executive governance either through the Investment and Project Management Committee (IPMC), or through the Financial Investment Management Committee (FIMC). Some projects are also subject to Treasury Board oversight.

24. Projects are identified and categorized during the investment planning process (gates 1 to 3), which precedes the project management lifecycle. The EPMO manually tracks Agency projects through a spreadsheet called the “Master Project List” (MPL), which is meant to consolidate multiple sources of information and provide timely status of the project's compliance to senior management, project executives and stakeholders.Footnote 11 The MPL is not used for tracking branch projects and initiatives that fall below a $1M threshold and are not subject to the PMF and IPMC or FIMC governance.

25. At the outset of this audit, the MPL was outdated, was not shared with stakeholders, did not reflect current accountabilities and included a variety of items, which were not projects or were sub-projects of already identified projects.Footnote 12 Project definition issues were found for several of the projects selected from the MPL for this audit, and several data elements were outdated. Additionally, the MPL provided limited information on the linkages of projects to one another, or to other interdependencies. For example, the MPL listed the Entry/Exit project in addition to 5 sub-projects related to the Entry/Exit initiative, but it was impossible to tell from the MPL if these were concurrent or if any needed to be completed prior to others. The MPL did not lend itself well to distinguishing or linking across a portfolio of projects, such as the various Beyond the Border initiatives.

26. Recently, the IPMC requested that the MPL be updated and presented to the committee on a more regular basis. It is expected that this will help those responsible for monitoring projects to have more complete and reliable information on the projects to exercise due diligence.

27. Project management tools are available, such as the CA Clarity project management software and a project management module in CAS, but the Agency has opted not to use these tools and does not have a plan for a proposed tool at this time.

28. Without an effective mechanism to identify, categorize, and track projects, there is a risk of not having the proper level of oversight as well as the potential for missed opportunities to identify relative priorities, interdependencies and efficiencies.

Recommendation 1: The Vice-President of the Finance and Corporate Management Branch (FCMB) should implement improved tool(s) and processes for categorizing and tracking projects and their interdependencies.

Management response: Agreed. The Vice-President of FCMB will continue to address opportunities to improve and streamline project management processes. The Enterprise Project Management Office will strengthen the processes to ensure that data integrity is maintained and that the monitoring and reporting of all aspects of projects, from initiating through to closing, is accurate and timely in order to provide IPMC and FIMC the information required to support informed decisions.

Completion date:

Project management deliverables

29. Departments must ensure that appropriate systems, processes, and controls for managing projects are in place to support the achievement of project and program outcomes, while limiting the risk to stakeholders and taxpayers.Footnote 13 The Agency project management processes are outlined in the CBSA PMF, which also presents the list of project deliverables for each gate in the Project Management Lifecycle, including appropriate consultations.

30. The audit assessed the extent to which the expected project gating deliverables and gate approvals were on file for each of the active and closed projects sampled. For 7 of the 8 projects that had project documentation available to review (note AIP3 was only required to complete monthly dashboards), we found that all 7 projects had passed gates without having completed all of the required project management documents or without evidence of having satisfied approval conditions. For the purpose of this test, documents had to be on file, complete and signed when applicable, to be considered compliant.

Figure 2: Percentages of fully completed gate deliverables found on file

Figure 2 description

The table shows the percentage of completed deliverables filed for the 7 sampled projects by gates. The project that has not yet progressed through the gate is represented by "N/A".

Percentage of completed gate deliverables found on file
  Gate 4 Gate 5 Gate 6 Gate 7
Project 1 70% N/A N/A N/A
Project 2 70% 60% 67% 100%
Project 3 70% 50% 67% N/A
Project 4 80% 80% N/A N/A
Project 5 60% 60% 56% 83%
Project 6 50% 60% 67% 33%
Project 7 30% 40% 44% 50%

31. Frequently missing from the file were task and financial agreements, benefits health assessments, and evidence of a validation of the ongoing business value of investment. According to the PMF, different areas are responsible for producing these documents, such as the “functional subject matter expert”, the Sponsor, or the Benefit Owner. All projects had a Project Charter, and all but 1 had monthly dashboards on file. One project did not complete a Project Management Plan or a Project Complexity and Risk Assessment (PCRA) as it was not originally considered a project. Only 3 out of a total of 12 gate presentations that should have been on file for the 3 closed projects (for each, gate 4 through 7) were found.

Risk management

32. Risk management is important for decreasing the impact and probability of negative risks, but also for identifying opportunities.Footnote 14 The main objective of applying risk management is to identify potential problems and deal with them when it is easier and less expensive to do so before they are problems and a crisis exists.Footnote 15

33. CBSA project managers are expected to actively manage risks and issues throughout the life of their project and to document them, along with appropriate risk mitigation strategies, in a risk log. Risk logs are meant to be included with the monthly submission of project dashboards to EPMO.

34. Most of the 8 projects we examined had documentation of the management of project risks. Four projects prepared monthly risk logs and 3 projects used living risk logs. For 1 of the closed projects, no risk log was found.

35. For the most part, there was evidence that risks were being monitored, modified or removed, however some risks persisted or materialized. For example, several project managers noted that delays in obtaining approvals had significant impacts on project timelines. One of the projects examined included this risk in one of their risk logs and also maintained a spreadsheet that tracked the average time between submission of a change request and subsequent approval by the project board. The average time was over 3 months, with most delays ranging from 4 to 8 months. The same situation was experienced by another project, as noted in the EPMO lessons learned library.

36. The Agency’s project risk management process is aligned with best practices and evidence to show it was being followed and was available for all but 1 of the 8 projects examined. While risk logs are a good tool for project teams to identify and document risks, it is important to ensure that risks are being managed to minimize the potential impact on project timelines, scope/benefits, and budget.

Project change management

37. As per the CBSA PMF, change is inevitable during the life of a project and all projects are expected to follow the change management practices to ensure that every change has full consideration of the impact to the project and is agreed to by the relevant authorities.Footnote 16

38. The audit assessed whether changes were documented, their impact was articulated and assessed, relevant consultations occurred and whether the change was appropriately approved. A total of 31 “Requests for Change” (RFC) forms were on file for 4 of the 7 projects that had project management documentation on file.Footnote 17 Eleven (35%) of the RFCs examined did not have evidence of consultation, which is important because failing to adequately inform and involve stakeholders is a key risk to successful project management. Only 9 of the 31 RFCs (29%) had the endorsement signatures of Project Sponsors on file.

39. Impacts of the requested change were documented on 94% (29/31) of the RFCs examined, however, the impacts of the change were almost exclusively articulated in terms of the impact to project timelines, costs, and scope with limited information on impact to project benefits. Four of the RFCs were enhancements, while 6 RFCs were scope reductions. The 6 scope reductions were all for the same project, for which changes were discussed and approved by a project management board. However, evidence of endorsement by project sponsor was only found for 1 of the 6 RFCs.

40. In , the Agency updated the project change management RFC form to better document the consultations undertaken, the assessment of the impact of the change, and the approvals. In addition, thresholds for the required governance and approval of project changes based on the impact to project scope, schedule or cost were articulated in Annex A of the new RFC form. The Agency has also recently developed a prototype for digital signatures, which should make it easier to obtain timely approvals on project documentation.Footnote 18 It will be important for the EPMO to monitor the effectiveness of the revised RFC process and ensure projects receive timely access for required approvals.

Organizational change management

41. Change management must be planned and implemented as a priority throughout the life of any high risk, complex transformation initiative. Change management and change leadership cannot be considered optional, an add-on, nor expendable when looking for ways to save time or money.Footnote 19

42. Change management activities are weaved into the CBSA project management approach. Gate 5 deliverables include a draft Change Management Plan and a Transition Management Plan, which encompasses all activities related to managing the business change successfully and a Communications Plan, which outlines how information will be distributed to ensure project success. Gate 6 expects a final Change Management Plan and the Gate 7 presentation should include a summary of organizational change management activities.

43. For the 8 projects examined, minimal documentation was on file related to change management. The organizational change management plan included in the Project Management Plans for several projects was not very detailed, and only 1 had a transition plan. Project managers were often not aware of change management activities, as they were the responsibility of the business owner.

44. Four projects had a Benefits Realization Plan and 4 had either a user manual, standard operating procedures or a training strategy. Interviewees highlighted the need to ensure that regional considerations be taken into account throughout the project, especially with respect to change management. Based on interviews and a review of project documentation, organizational change management for Agency projects is not mature.

45. As part of CBSA Renewal, the Agency has recognized the need to improve overall organizational change management and has established a Chief Transformation Officer Branch, which is in the process of developing a Change Management and Culture Framework and offering training. In addition to these efforts, the Agency should mature change management for its projects.

46. The EPMO has a key responsibility to ensure project readiness prior to proceeding for gate approval. Recent efforts show progress: the EPMO started to produce independent assessments of project health towards the end of 2019. However, without a strong challenge and compliance function, projects run the risk of moving through key gates without the appropriate building blocks in place to ensure their success.

Recommendation 2: The Vice-President of the Finance and Corporate Management Branch (FCMB) should strengthen the role of the EPMO to ensure it delivers on its challenge and centre of authority function and provides the expected level of control over project gating.

Management response: Agreed. The Vice-President of FCMB will strive to strengthen the role of the Enterprise Project Management Office (EPMO) to be seen as the centre of expertise and principal authority for project management for the CBSA. An improved project gating process and the IPMC and FIMC support of the EPMO’s recommendations on gate readiness and project performance will provide the expected level of control over project gating as well as help to reinforce the role and authority of the EPMO.

Completion date:

8.3 Oversight and reporting

Audit criteria:

47. Deputy Heads are responsible for monitoring and reporting on the management of departmental projects.Footnote 20 Project dashboards should generate discussion at senior levels and draw attention to areas that might require course correction. Project reporting should be timely, accurate, relevant, transparent and complete.Footnote 21

Information for decision-making

48. At the time of the audit, projects were expected to produce monthly Executive Project Dashboards (EPD) and Project Summary Reports (PSR) to provide updated information on project status and health. The EPD and PSR contained much of the same information, with the PSR providing slightly more detail. The EPMO would then use this information to produce quarterly Integrated Project Reports (IPR) for governance bodies (IPMC/FIMC), presented a month or two after the quarter.

49. Project sponsors expressed concerns about the timeliness of the quarterly reports. Oftentimes, information related to the project health status had evolved by the time it was reviewed by the committees, and the outdated information yielded unnecessary discussions.

Financial information

50. From a financial perspective, financial management advisors (FMAs) noted that re-profiling of funds and delayed time reporting against a project would often skew the financial health reported to committees. Using a list of expenditure codes against the Agency Work Breakdown Structure (WBS) for fiscal years 2017, 2018 and 2019, a pattern was detected for several projects where salary dollars were low for periods 1 through 6 and then a sudden spike in salary dollars spending occurred midway or towards the end of the fiscal year. For example, Biometrics Expansion salary expenditures spiked in periods 9 of 2017, 12 of 2018 and 6 of 2019 as a result of the retroactive coding of time from previous months to the project. Similar spikes were noted for other projects including Nexus electronic application, Trusted Trader Harmonization, Interactive Advance Passenger Information, Master Data Management, and Service Oriented Architecture. The source of this may be attributed to the Task Financial Agreements (TFAs) process.

51. The CBSA established TFAs for allocating funds internally between service requestors and the service providers to supply services necessary to complete projects. The TFA process was adopted to help ensure agreement on the timing for the production of the deliverables and to control the interchange of funds amongst stakeholders. TFAs include the information necessary to perform the budget transfers, including the (WBS) elements, cost centers and type of costs by vote and salary / non-salary.Footnote 22 A delay in establishing TFAs can impact the timeliness of the work of service providers. In many cases, work is performed, but only coded to the project once the funds are received. The Agency recognized the issue with the TFAs and has taken measures in 2019 to ensure that they are put in place at the beginning of the fiscal year.Footnote 23

52. In terms of accuracy of project financial reporting, the audit found several examples of project spending that were not coded to their associated WBS by analyzing Agency expenditures for fiscal years 2016-17 through to fiscal year 2019 to 2020. It was confirmed with the FMAs that several transactions that were not coded against their WBS were in fact project related. The WBS is a soft control, requiring parties to manually “tag” transactions as they see fit to a WBS. Consequently coding errors can and do result in transactions not being properly linked to projects. FMAs raised concerns about the issues with the structure of a project in the financial system and suggested that project managers involve them earlier in the process to avoid mistakes that can be hard to correct later.

Information on project risks and changes

53. For 4 projects examined, the audit was able to compare monthly risk logs against the risks presented in monthly project dashboards.Footnote 24 Discrepancies were found for 2 of these 4 projects: the risk ratings in the dashboard did not always match the rating in the risk log. For 1 project, 3 high-rated risks were not presented in the project dashboard. For another project, in 4 instances, lower risks were presented on the dashboard when higher risks existed, and in 5 instances, the risk ratings presented on the dashboard were lower than the rating in the risk log. In addition, all projects reviewed had instances of missing or incomplete monthly risk logs, such as the absence of the risk owner, a risk impact description, or a mitigation plan.

54. When probed about the culture around project management at the Agency, interviewees indicated that certain improvements could be made to help foster collaboration, good governance and transparency. For example, participants at governance meeting should ensure they are familiar with the projects being discussed, offer helpful questions and seek to challenge projects in order to create meaningful exchanges and an understanding of the impacts of a decision. Additionally, several project teams felt it was not acceptable for them to submit a project dashboard that showed a yellow or red status: some had been asked in the past to change the status to green. Interviewees spoke of a general tone at the CBSA where people do not want to be associated with project failure. Recently, the EPMO has started to provide its own independent assessment of project health status. The organization’s willingness to receive unfavorable or unexpected information will be key to improving the reporting of the actual state of projects.

55. Without complete, timely and accurate information on project risks and health, decision makers may not be able to make informed decisions and ensure proper mitigation of emerging risks.

Documentation of decisions

56. The documentation of key decisions in project management is important for ensuring proper accountability and authority, communicating objectives and direction, creating a repository of corporate knowledge and facilitating continuous improvement.Footnote 25 Decisions made along the lifecycle of a project can impact the outcome.

57. The audit examined whether gate approvals by oversight bodies were on file for the projects examined. For the active projects, a review of the IPMC and FIMC Records of Decision enabled confirmation that the committees had approved or conditionally approved the projects to move onto the next gate. This is an improvement, as gate approvals were not found on file for any of the 3 closed projects reviewed.

Figure 3: Gate approval found on file or in committee records of decision

Figure 3 description

This graphic shows the approval rating of 7 sampled projects found on file or in committee records of decision in each of 4 stages or "gates."

There are 4 possible outcomes at each gate:

  • Approved
  • Partial/Conditional
  • Not found
  • Not applicable
Gate approval found on file or in committee records of decision
  Gate 4 Gate 5 Gate 6 Gate 7
Project 1 Partial/Conditional Not applicable Not applicable Not applicable
Project 2 Approved Approved Partial/Conditional Approved
Project 3 Approved Partial/Conditional Partial/Conditional Not applicable
Project 4 Approved Approved Not applicable Not applicable
Project 5 Not found Not found Not found Partial/Conditional
Project 6 Partial/Conditional Partial/Conditional Not found Not found
Project 7 Not found Not found Not found Not found

58. Gate decision points are typically a go/no-go situation, but a conditional approval is sometimes granted to a project to proceed to the next phase with the expectation that the gate requirements will be subsequently completed. A review of governance committee agendas and records of decision from , to , showed 8 instances of conditional approvals on various projects (not limited to the projects sampled for this audit). For example, the CAED project received gate 4 approval conditional on providing updated financial and benefits information. Another example was a conditional gate approval for the Port of Entry Management System project. Conditional approvals are tracked by the EPMO as well as in governance committee Action Tracking Lists. Decisions were not always recorded the same way by EPMO and the governance secretariat, and evidence of a follow-up was found for only 2 of these 8 conditional approvals. Projects were not required to come back to the committee to demonstrate that conditions were met.

59. When decisions are not properly documented, it can be difficult to demonstrate appropriate oversight and communicate clear direction on the way forward.

8.4 Capacity management and continuous improvement

Audit criteria:

60. To assure its success in achieving objectives, organizations need to focus on building capacity and leadership to ensure they have the necessary people and work environment.Footnote 26 It is also a project management best practice that knowledge gained during a project on how project events were addressed, or should be addressed in the future, be used for the purpose of improving future performance.Footnote 27

Project management capacity

61. Project management skills and expertise are essential to project success and finding the right talent has been a challenge for some time. Examples of capacity issues were provided for several of the projects sampled, and gaps in skills and expertise were also a common theme in the Project Lessons Learned Library.

62. A key responsibility of the EPMO is to act as the centre of expertise for project management within the CBSA and support the community of project management practitioners as of source of leadership and expertise.Footnote 28 In 2014, the EPMO developed a project management competency framework, but it was never formally implemented. Branches have started to develop project management capacity, but overall, the Agency has not inventoried its existing project management capacity or identified the capacity and expertise required.

63. The Agency has a high reliance on consultants for expertise in project management. Interviewees noted that the Agency had an insufficient number of experienced Project Managers resulting in 5 of the 8 projects relying on the use of consultants. In addition, turnover within project teams also results in a loss of knowledge and capacity. A presentation to IPMC in recognized the need to address known gaps in project management competencies and capacity, including training, classification and establishing career paths.Footnote 29

64. New online project management training was made available in .Footnote 30 Several interviewees suggested that Project Managers and Project Sponsors could also benefit from additional training in financial management, relationship management and change management.

Continuous improvement, communication and collaboration

65. The EPMO maintains resources to support continuous improvement, including a Project Lessons Learned Library, a Project Lessons Learned Guide, and templates, which are aligned with best practices as set out in the Project Management Body of Knowledge of the Project Management Institute. However, it is generally left to each project to consider these and there is no process in place to assess lessons at an overall level to adapt and improve Agency project management practices as may be required.

66. Audit analysis of the EPMO Lessons Learned Library showed that over 85 (33%) of the 256 lessons learned were related to stakeholder issues, including stakeholder management (40 entries), service management (20 entries), and communications management (17 entries). Several interviewees corroborated that the lack of timely engagement of internal service providers (for example, procurement) and failing to account for interrelationships and interdependencies between projects can impact project timelines and costs. Depending on the size and complexity of a project, the establishment of a project board, or portfolio project board, is recommended in the PMF as a key way to engage important stakeholders, including internal service providers. Evidence of a project board was found for 2 of the 8 projects examined. There is an opportunity for the EPMO to evolve the PMF in light of this lesson and help determine the type of project board that would be appropriate for each project, according to their size and risk level.

67. Independent Third Party Reviews (ITPRs), which are assessments periodically conducted on projects, are another key source of lessons. Currently, there is no process to communicate ITPR lessons and risks or to integrate them into the Lessons Learned Library.

68. There is an opportunity to mature Agency project management practices to ensure continuous learning. Insufficient project management capacity and expertise leads to reliance on external resources and loss of knowledge, which can put the Agency at risk of not being able to deliver on its portfolio of projects, or deliver projects in a timely manner.

Recommendation 3: The Vice-President of the Finance and Corporate Management Branch (FCMB) should establish a process to continually improve and mature Agency project management practices and implement a plan to identify and address gaps in the capacity, knowledge and expertise of its project management workforce.

Management response: Agreed. The Vice-President of FCMB acknowledges the requirement to identify and address the Agency’s needs with respect to building capacity, competency and knowledge of the project management workforce within the CBSA. The launch of the online course, “Project Management at the CBSA”, will increase awareness and understanding of how projects are managed within the agency. The EPMO will develop a plan to identify the project management community in order to support and promote the professional development of project management practitioners.

Completion date:

69. The Agency has made considerable progress since to improve project management accountability and oversight, and is implementing changes arising from the new Treasury Board Policy on the Planning and Management of Investments. It will be important for the FCMB to continue exercising its function by taking an active role in ensuring project readiness (that is completion of project and benefits management documentation) and validating the accuracy of project reporting prior to governance bodies appearances. Finally, the Agency needs to attract and develop adequate project management expertise and experience in order to be able to successfully deliver on its portfolio of projects.

Annex A: List of projects examined

List of projects examined
Project name and type Description
Canadian Automated Export Declarations (CAED) replacement initiative (IT-enabled)
(At Gate 5 during audit)		 The Canadian Export Reporting System (CERS), a web-based self-service portal used to submit export data, will replace Statistics Canada’s (SC) Canadian Automated Export Declaration (CAED) system.
Master Data Management (MDM) (IT-enabled)
(At Gate 6 during audit)		 A commercial off-the-shelf solution to create a single, unique entity from multiple CBSA data sources providing common data for use in front-line applications. When fully implemented, the MDM solution will ensure that data about people and/or businesses, stored in different systems, can be linked consistently, reliably and efficiently.
Accelerated Infrastructure Program (AIP3) (Real property)
(At Gate 5 during audit)		 Programme of investment in CBSA custodial Ports of Entry to improve operational effectiveness and occupational health and safety. AIP3 was tracked as an overall “project” which was difficult as it consisted of dozens of sub-projects at different phases.
Partners in Protection (PIP) and Customs Trade Partnership Against Terrorism (CTPAT) Harmonization (non-IT)
(At Gate 7 during audit)		 One of several Beyond the Border (BtB) initiatives that was not initially tracked as a project. Harmonization is to align the Canadian and United States programs in the areas of policy, procedures and processing practices, using the Trusted Trader Portal (TTP) as an interoperable communications tool enabling the exchange of program information with U.S. Customs and Border Protection via their CTPAT portal. Project Closure report dated .
Biometrics Expansion (led by IRCC) (IT-enabled)
(At Gate 5 during audit)		 Approved by Treasury Board in , and building upon the Temporary Resident Biometrics Program introduced in 2013, the project is expected to strengthen identity management; improve the ability to prevent inadmissible individuals from entering Canada; and facilitate movement of admissible individuals into Canada.
Recourse Content Management System (RCMS) (IT-enabled)
Closed ()		 Creation of a recourse case management system (Trade and Enforcement) with improved reporting capability and an electronic appeals process that is supported by the Information Science and Technology Branch.
Lacolle Commercial (Real property)
Closed ()		 Construction of a new commercial examination facility with 6 loading docks, including the reorganization of traffic lanes to access the new building. Construction completed .
Nexus electronic application (IT-enabled)
Closed ()		 A BtB initiative to create an enterprise portal for Canadian citizens to submit online NEXUS applications. The forward-facing component was cancelled and back-end database and system enhancements were implemented instead.

Annex B: About the audit

Audit objectives and scope

The objective of this audit was to assess the extent to which project management governance, processes and controls are established and followed to support the successful delivery of Agency projects.Footnote 31

The audit scope included a review of a sample of completed and ongoing projects between , and . The audit covered the Project Management process from initiating a project (gate 4) through to project closure (gate 7). Post implementation activities were excluded as they come after the project management lifecycle. The audit scope excluded the project investment planning gates 1-3 because these were the focus of a review of internal controls with respect to investment planning. The Benefits Realization framework and processes were excluded from the scope of the audit except for verifying that organizational change management plans and activities are considered and developed during the project management lifecycle. The audit did not assess the Service Lifecycle Management Framework.

Risk assessment

A preliminary risk assessment was conducted during the audit planning phase to identify potential areas of risk as well as audit priorities. The following key risks related to the Agency project management were identified:

Approach and methodology

This audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada and the Institute of Internal Auditors Standards for the Professional Practice of Internal Auditing.

The following methodologies and techniques were used during the examination phase of this audit:

Audit criteria

The audit criteria were aligned with the former Treasury Board Policy on the Management of Projects and supporting policies, standards and guidance: the CBSA Project Management Framework; the Project Management Body of Knowledge (PMBOK) of the Project Management Institute; the Core Management Controls and Audit Criteria of the Office of the Comptroller General; and the Committee of Sponsoring Organizations of the Treadway Commission (COSO) Principles of Effective Internal Control.

The following audit lines of enquiry and criteria were selected:

Line of enquiry 1: Accountability and oversight

Audit criteria:

Line of enquiry 2: Capacity management

Audit criteria:

Line of enquiry 3: Project management processes

Audit criteria:

Annex C: List of acronyms

AIP
Accelerated Infrastructure Program
BRF
Benefits Realization Framework
CAED/CERS
Canadian Automated Export Declaration/Canadian Export Reporting System
CBSA
Canada Border Services Agency
CPRD
Corporate Planning and Reporting Directorate
EPD
Enterprise Project Dashboard
EPMO
Enterprise Project Management Office
FCMB
Finance and Corporate Management Branch
FIMC
Financial Investment Management Committee
FMA
Financial Management Advisor
IPMC
Investment and Project Management Committee
IPR
Integrated Project Reports
ISTB
Information, Science and Technology Branch
ITPR
Independent Third Party Review
MDM
Master Data Management
PIP-CTPAT
Partners in Protection – Customs Trade Partnership Against Terrorism
PMBOK
Project Management Body of Knowledge
PMF
Project Management Framework
PSR
Project Status Report
RCMS
Recourse Content Management System
RFC
Request for Change
ROD
Record of Decision
TB
Treasury Board
WBS
Work Breakdown Structure
Date modified: