type: other

jurisdiction: US

text: b' The Integrated Financial System Project Team\n Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a\n Successful Deployment\n\n August 2004\n\n Reference Number: 2004-20-147\n\n\n\n\nThis report has cleared the Treasury Inspector General For Tax Administration disclosure\nreview process and information determined to be restricted from public release has been\n redacted from this document.\n\x0c DEPARTMENT OF THE TREASURY\n WASHINGTON, D.C. 20220\n\n\n\n\nINSPECTOR GENERAL\n for TAX\n ADMINISTRATION\n\n\n\n\n August 13, 2004\n\n\n\n MEMORANDUM FOR CHIEF INFORMATION OFFICER\n CHIEF, MISSION ASSURANCE\n\n\n FROM: Gordon C. Milbourn III\n Acting Deputy Inspector General for Audit\n\n SUBJECT: Final Audit Report - The Integrated Financial System Project\n Team Needs to Resolve Transition Planning and Testing Issues\n to Increase the Chances of a Successful Deployment\n (Audit # 200320043)\n\n\n This report presents the results of our review of the Internal Revenue Service\xe2\x80\x99s (IRS)\n Integrated Financial System (IFS). The overall objective of this review was to determine\n if adequate testing processes and procedures were being followed prior to the\n deployment of the IFS to ensure the system meets expectations. Additionally, we\n determined if the IFS project team was adequately planning for the transition of\n operation and maintenance of the system from the contractor to the IRS.\n In summary, the IFS is intended to provide administrative financial management\n information essential for financial statement preparation. In addition, the development\n of the IFS supports one of the President\xe2\x80\x99s Management Agenda initiatives to improve\n financial performance, which involves ensuring Federal Government financial systems\n produce accurate and timely financial information. The IRS and the PRIME contractor1\n recently announced that the IFS would not be deployed until October 2004, a delay of 1\n year from the originally scheduled deployment date. While the IFS project has\n experienced delays, the project team has made progress in the areas of System\n Integration Testing (SIT),2 transition planning, and security test planning. We\n\n\n 1\n To facilitate success of its modernization efforts, the IRS hired the Computer Sciences Corporation as the PRIME\n contractor and integrator for the Business Systems Modernization (BSM) program and created the Business Systems\n Modernization Office to guide and oversee the work of the PRIME contractor.\n 2\n The SIT ensures all system components (hardware and software) are working correctly and collectively with other\n related or dependent systems. The IFS project is also performing Systems Acceptance Testing (SAT) as part of the\n SIT. SAT independently assesses the quality of a system and the system\xe2\x80\x99s readiness for implementation.\n\x0c 2\n\ndetermined that significant integration testing was underway, and the risk of the IRS\nInformation Technology Services organization not being able to operate and maintain\nthe IFS had been reduced. In addition, the IFS project team became the first Business\nSystems Modernization project team to complete a Transition Management Plan. Also,\na detailed IFS Security Test and Evaluation (ST&E) Plan has been prepared.\nWhile the IFS project team continues to make progress, improvements can be made in\nthe areas of transition planning and testing to increase the chances of a successful\ndeployment. To provide timely feedback during critical transition and testing activities,\nwe communicated the results of our analyses during the audit; these communications\nare discussed throughout the report.\nTo ensure transition planning issues are resolved prior to acceptance of the IFS by the\nIRS and transition management is considered when exiting appropriate milestones, we\nrecommended the Chief Information Officer (CIO) ensure outstanding IFS transition\nissues are documented and tracked to closure and the Enterprise Life Cycle (ELC)3\nMilestone Exit Criteria4 are updated to include completion of the Transition Management\nPlan. To ensure IFS testing results in a high-quality product and future modernization\nprojects are improved through lessons learned by the IFS project team, we\nrecommended the CIO ensure 1) the Requirements Traceability Verification Matrix\n(RTVM)5 is updated to include a verification and validation method for all requirements,\n2) deviation forms6 are updated to include a signature line for all required approving\nofficials, 3) the root cause is investigated and resolved for why defects7 are more\nvoluminous than expected and for why defects are being evaluated and resolved slower\nthan expected, and 4) the ELC is updated to include a requirement for entrance and exit\ncriteria8 in the ST&E Plan. To ensure planning for the upcoming IFS ST&E is as\nthorough as possible, we recommended the Chief, Mission Assurance, ensure all\nneeded elements from the Internal Revenue Manual (IRM) are included in the ST&E\nPlan and trusted recovery9 and object reuse10 testing occurs (or the risk of not\nconducting these tests is documented).\n\n\n\n3\n BSM processes and procedures are documented within the ELC. See Appendix IV for an overview of the ELC.\n4\n The IRS uses Milestone Exit Criteria to determine whether modernization projects are ready to proceed to the next\nlife cycle milestone.\n5\n The RTVM is a tool used to ensure each system requirement is tested by assigning it to one or more test cases.\nThis tool helps ensure all requirements are tested.\n6\n Test Case Deviations and Waivers Forms include requests for verification of a requirement or set of requirements\nto be moved to another phase or for the obligation to verify functionality, such as a requirement or set of\nrequirements, be deleted.\n7\n A defect is a perceived problem found in software, documents, hardware, or other controlled products. Defect\nprocessing is the process of documenting the defect, tracking the defect and its corrective action, and reporting the\nstatus of each defect.\n8\n A best practice in test planning is to create a predefined set of criteria used to determine when a test is ready to be\nconducted (entrance criteria) and when a test has been completed at an acceptable quality level (exit criteria).\n9\n Trusted recovery controls refer to procedures and/or mechanisms available to assure that, after a system failure or\nother discontinuity, recovery without compromise is possible.\n10\n Object reuse controls are designed to ensure the authorized user of the system does not obtain residual data.\n\x0c 3\n\nManagement\xe2\x80\x99s Response: Management agreed with seven of our recommendations\nand disagreed with one recommendation. The IRS responded that it is now including\nthe correct signature line on deviation forms and using an issue tracking log to monitor\ntransition issues. The IRS also plans to update the IFS RTVM and ST&E Plan, ensure\na lessons learned report is generated, and update the ELC to ensure that projects\ndevelop a Transition Management Plan.\nIn addition to responding to our recommendations, the IRS indicated that it has\nassigned the Chief Financial Officer full-time to the IFS Project, negotiated a\ncost-sharing arrangement with the PRIME contractor to balance financial risk, and\nbegun validating the integrity of the system. The PRIME contractor has also bolstered\nits management team with key skills. Management\xe2\x80\x99s complete response to the draft\nreport is included as Appendix VII.\nOffice of Audit Comment: The CIO disagreed with our recommendation to ensure that\ntrusted recovery and object reuse testing occurs, or the risk of not conducting these\ntests is documented. The IRM requires that correct implementation of object reuse and\ntrusted recovery be validated as part of the ST&E. In addition, the IRS is planning to\ntest object reuse capabilities on one of two computer platforms. The IRS responded\nthat object reuse testing is no longer required by the Department of the Treasury. While\nthe Department of the Treasury requirement may have changed, the IRM still requires\nobject reuse testing. Therefore, we believe the IRS should conduct the required testing,\nparticularly since it plans to perform object reuse testing on one computer platform and\nnot the other. The IRS\xe2\x80\x99 response did not address trusted recovery testing.\nThe CIO agreed with our recommendation to update the ELC to include a requirement\nfor entrance and exit criteria in the ST&E Plan. However, the corrective action provided\nwill not adequately address our recommendation. A best practice in test planning is to\ncreate a predefined set of criteria used to determine when a test is ready to be\nconducted (entrance criteria) and when a test has been completed at an acceptable\nquality level (exit criteria). The IRS responded that it would be incorporating changes\ninto the milestone exit requirements. Changing the milestone exit criteria will not create\na requirement for entrance and exit criteria in the ST&E Plan.\nWhile we still believe our recommendations are worthwhile, we do not intend to elevate\nour disagreements concerning them to the Department of the Treasury for resolution.\nCopies of this report are also being sent to the IRS managers affected by the report\nrecommendations. Please contact me at (202) 622-6510 if you have questions or\nMargaret E. Begg, Assistant Inspector General for Audit (Information Systems\nPrograms), at (202) 622-8510.\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n\n\n\n Table of Contents\n\n\nBackground ............................................................................................... Page 1\nWhile the Integrated Financial System Project Continues to Experience\nSignificant Schedule Delays, the Project Team Is Making Progress\nToward Deployment .................................................................................. Page 3\nTransition Planning Is Improving; However, Additional Improvements\nAre Needed ............................................................................................... Page 5\n Recommendations 1 and 2: .......................................................... Page 9\n\nTesting Practices Continue to Need Improvement .................................... Page 9\n Recommendations 3 through 5: .................................................... Page 20\n Recommendations 6 through 8: .................................................... Page 21\n\nAppendix I \xe2\x80\x93 Detailed Objectives, Scope, and Methodology ..................... Page 22\nAppendix II \xe2\x80\x93 Major Contributors to This Report........................................ Page 24\nAppendix III \xe2\x80\x93 Report Distribution List ....................................................... Page 25\nAppendix IV \xe2\x80\x93 Enterprise Life Cycle Overview .......................................... Page 26\nAppendix V \xe2\x80\x93 Test Case Elements Required by the Enterprise\nLife Cycle................................................................................................... Page 31\nAppendix VI \xe2\x80\x93 Defect Tracking and Resolution Time Periods ................... Page 33\nAppendix VII \xe2\x80\x93 Management\xe2\x80\x99s Response to the Draft Report ................... Page 34\n\x0c The Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n The Internal Revenue Service (IRS) is currently engaged in\nBackground an effort, known as Business Systems Modernization\n (BSM), to modernize its systems and associated processes.\n To facilitate the success of its modernization efforts, the IRS\n hired the Computer Sciences Corporation as the PRIME\n contractor and integrator for the BSM program and created\n the Business Systems Modernization Office (BSMO) to\n guide and oversee the work of the PRIME contractor.\n One of the BSM projects is the Integrated Financial System\n (IFS), which will help to modernize the IRS financial\n systems and processes used to generate annual financial\n statements. This new general ledger and accounting system\n is based on commercial software from the Systems Analysis\n and Program Development (SAP) organization.1\n The IFS is intended to provide administrative financial\n management information essential for financial statement\n preparation. In addition, the development of the IFS\n supports one of the President\xe2\x80\x99s Management Agenda\n initiatives to improve financial performance, which involves\n ensuring Federal Government financial systems produce\n accurate and timely financial information.\n This is the Treasury Inspector General for Tax\n Administration (TIGTA) Information Systems Programs\n business unit\xe2\x80\x99s second audit of the IFS project.2 The first\n audit3 found the IFS project team had begun important\n testing activities, ensured compatibility with the Security\n\n\n\n\n 1\n The SAP organization is the vendor selected to implement the software\n application.\n 2\n The TIGTA Headquarters Operations and Exempt Organizations\n Programs business unit has also completed two audits of the IFS:\n Requirements Definition of the Integrated Financial System (Reference\n Number 2003-10-179, dated August 2003); and Improvements Are\n Needed for Subsequent Integrated Financial System Testing (Reference\n Number 2004-10-052, dated March 2004). A third audit is nearing\n completion, which is addressing whether the IFS complies with\n accounting standards and contains certain functionalities\n (Audit # 200410003).\n 3\n Risks Are Mounting as the Integrated Financial System Project Team\n Strives to Meet an Aggressive Implementation Date (Reference\n Number 2004-20-001, dated October 2003).\n Page 1\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n and Technology Infrastructure Release (STIR),4 and ensured\n redundant hardware was planned for implementation.\n However, we also determined that risks5 were mounting as\n the IFS project team strived to meet an aggressive\n deployment date of October 2003. In September 2003, the\n IFS project team announced that IFS deployment was being\n rescheduled for February 13, 2004, and that a delay until\n April 13, 2004, was possible.6 In January 2004, the IFS\n project team announced that the IFS would not deploy until\n October 2004.\n After learning the PRIME contractor would again miss the\n target deployment date for the IFS, the Commissioner\n testified that the delay was a huge disappointment to the\n IRS. The Commissioner indicated the PRIME contractor\n was willing to bear the financial burden for the additional\n delay; however, the IRS needed to take stronger steps. The\n Commissioner stated the IRS would expand competition for\n new enforcement projects as well as for the next phase of\n the IFS.\n This audit was conducted at the BSMO facilities in\n New Carrollton, Maryland, during the period\n September 2003 through March 2004 in accordance with\n Government Auditing Standards. Detailed information on\n our audit objectives, scope, and methodology is presented in\n Appendix I. Major contributors to the report are listed in\n Appendix II. To provide timely feedback during critical\n transition and testing activities, we communicated the\n results of our analyses to the BSMO during the audit; these\n communications are discussed throughout the report.\n During the audit, the BSMO informed us of the results of\n several studies that were performed to obtain fresh and\n\n\n 4\n The STIR project will provide the secure technical infrastructure to\n support and enable the delivery of the IRS\xe2\x80\x99 modernized business\n systems. The STIR is now a part of the Infrastructure Shared Services\n program.\n 5\n We determined that testing practices could be improved, project costs\n were increasing, some functionality had been postponed, and disaster\n recovery would not be optimal or fully tested prior to implementation.\n 6\n The April deployment date was provided as a risk-adjusted date. This\n was the date the project team believed could be met even if all known\n risks occurred (i.e., worst case scenario).\n Page 2\n\x0c The Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n independent assessments from outside experts on the health\n of the BSM program. To address the results of the studies,\n the IRS and the PRIME contractor have developed and are\n implementing an action plan, known as the BSM Challenge\n Plan, designed to address the BSM-related study\n recommendations. Several of the BSM Challenge Plan\n actions, if effective, would address issues within this report\n (e.g., ensuring projects strictly follow the Enterprise Life\n Cycle (ELC),7 tightly managing business requirements, fully\n maturing management disciplines, and ensuring more\n effective integration of IRS and PRIME/subcontractor test\n teams).\n This audit was conducted while changes were being made at\n both the BSM program level and the IFS project level. The\n majority of our fieldwork was completed in February 2004;\n however, we reviewed additional information provided by\n the BSMO and the PRIME contractor during March 2004.\n Any project changes that have occurred since we concluded\n our analyses are not reflected in this report. As a result, this\n report may not reflect the most current status.\n The IRS and the PRIME contractor recently announced that\nWhile the Integrated Financial\n the IFS would not be deployed until October 2004, a delay\nSystem Project Continues to\n of 1 year from the originally scheduled deployment date.\nExperience Significant Schedule\n This delay has been attributed to the following issues: data\nDelays, the Project Team Is\n conversion,8 health coverage tax credit (HCTC),9 interface\nMaking Progress Toward\n development,10 and testing productivity. While the IFS\nDeployment\n project has experienced delays, the project team has made\n progress in the areas of integration testing, transition\n planning, and security test planning.\n\n\n\n 7\n BSM processes and procedures are documented within the ELC. See\n Appendix IV for an overview of the ELC.\n 8\n Data conversion involves testing the ability of computer programs to\n transform data from a prior application into usable IFS data.\n 9\n The IRS is responsible for administering the program in which\n individuals eligible for qualified health coverage pay 35 percent of their\n health plan premium each month. The IRS adds the remaining\n 65 percent to the payment and submits the full 100 percent to the\n individual\xe2\x80\x99s designated health plan. The IFS will receive HCTC\n information and create payment vouchers.\n 10\n An interface refers to equipment or programs designed to\n communicate information from one system to another.\n Page 3\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n \xe2\x80\xa2 Significant integration testing is underway and the\n process for moving and deleting requirements is\n being followed. The IFS System Integration Testing\n (SIT)11 was separated into five parts. As of\n March 31, 2004, one part was complete and the\n remaining four parts were underway. In addition, the\n PRIME contractor informed us that testing productivity\n was increasing because both developers and testers are\n now included in the SIT environment. We also\n determined that the ELC process for moving a set of\n requirements to another part of testing and deleting\n requirements from the current release was being\n documented.\n \xe2\x80\xa2 The IFS project team became the first BSM project\n team to complete a Transition Management Plan\n (TMP). In October 2002, the PRIME contractor\n enhanced the ELC to require a TMP in place of a\n Transition To Support Plan. The TMP, which describes\n the plans for transitioning the initial release of the IFS to\n the IRS, is a result of the accumulation of best practices\n by the PRIME contractor.\n While the Transition To Support Plan and the TMP both\n describe transition activities, the TMP is a much more\n comprehensive document than the Transition To\n Support Plan. For instance, the TMP provides a listing\n of applications, project documentation, hardware,\n software, telecommunications, and database items that\n will be transitioned to the IRS.\n In September 2003, the IFS project team delivered the\n initial version of the IFS TMP. The IFS TMP includes\n transition schedules and transition acceptance criteria, as\n well as information about user training, staffing, and\n other items needed to plan for the transition.\n \xe2\x80\xa2 The risk of the IRS Information Technology Services\n (ITS) organization not being able to operate and\n maintain the IFS has been reduced. The IFS project\n team has been preparing to implement the transfer of the\n\n 11\n The SIT ensures all system components (hardware and software) are\n working correctly and collectively with other related or dependent\n systems.\n Page 4\n\x0c The Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n IFS to the IRS. Initially, the ITS organization did not\n believe enough time was being allocated for its\n personnel to learn how to operate and maintain the IFS\n once it was implemented. The proposed transition time\n was much less than the proposed transition time for a\n similar BSM project. The IFS project team correctly\n documented this as a risk and created a risk reduction\n plan. At the end of our audit, all risk reduction activities\n were on track, and the project delay had provided\n additional transition time.\n \xe2\x80\xa2 A detailed IFS Security Test and Evaluation (ST&E)\n Plan has been prepared. The IRS Office of Mission\n Assurance and its security testing contractor have\n prepared an ST&E Plan to ensure the IFS application\n will meet security requirements. The Plan includes a\n good overview of the tests to be conducted, as well as\n many test cases designed to ensure security requirements\n are met. While the ST&E Plan is a very good start,\n several areas in it could be improved (see The ST&E\n Plan can be improved section later in this report).12\n The actual ST&E for the IFS application will be\n conducted just prior to deployment.\n While the IFS project team continues to make progress,\n improvements can be made in the areas of transition\n planning and testing to increase the chances of a successful\n deployment.\n In September 2003, the IFS project team delivered the IFS\nTransition Planning Is\n TMP, which describes the plans for transitioning the initial\nImproving; However, Additional\n release of the IFS to the IRS. Based on our review of the\nImprovements Are Needed\n TMP and discussions with BSMO, Chief Financial\n Officer (CFO), ITS, and PRIME contractor personnel, we\n communicated our concerns regarding transition\n management to BSMO officials on October 17, 2003.\n\n\n 12\n Our review of the ST&E activity was limited to a high-level review of\n the ST&E Plan and associated documentation. The TIGTA Information\n Systems Programs business unit is currently conducting a review of\n security controls for modernized systems. This review will focus on\n five BSM projects; the IFS is not included. The objective is to assess\n the effectiveness of system security controls for modernized systems.\n We will release the results of this audit in a separate audit report.\n Page 5\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n Our initial impression was that the IFS TMP included a very\n good overview of transitional activities; however, the level\n of detail in the TMP and related documents was insufficient\n and raised a lot of unanswered questions. We were\n informed the BSMO also had come to this conclusion and\n had requested that the PRIME contractor revise the TMP.\n The PRIME contractor delivered a revised version of the\n IFS TMP in December 2003. We reviewed the revised\n TMP and provided additional input to the BSMO on\n January 16, 2004. While our original concerns were not\n completely addressed by the revised TMP, we noted that the\n IFS project team was making progress in planning for the\n IFS transition. The strongest improvement in the revised\n TMP was that many of our original concerns were\n mentioned, whereas the previous TMP was often silent on\n important details.\n The ELC requires that transition activities, such as help\n desk, service level agreements (SLA),13 configuration\n management,14 and training requirements, be covered in the\n TMP. Transition issues, if left unresolved, will result in\n inadequate clarity as to how to operate and maintain the IFS\n once it is implemented.\n We determined that further details are needed to fully\n address several post-implementation transition planning\n activities. These details could be provided in future updates\n of the TMP or as related transition documents are created or\n updated.\n \xe2\x80\xa2 Help desk procedures and staffing levels were not\n defined. Procedures for help desk call handling,\n routing, and escalation were under development. In\n addition, staffing levels for the help desk had not been\n determined. Once the IFS has been implemented,\n procedures will need to be in place to instruct users on\n how to get assistance should they have trouble using the\n\n\n 13\n An SLA defines what is needed and the service level that is expected\n when one organization needs services or resources from another\n organization.\n 14\n Configuration management involves establishing proper control over\n approved project documentation, hardware, and software and assuring\n changes are authorized, controlled, and tracked.\n Page 6\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n IFS. In addition, CFO and ITS organization personnel\n will need to be assigned to answer help desk calls and\n will need to know how to handle calls for assistance.\n \xe2\x80\xa2 Not all SLAs and contracts involving the CFO, the\n ITS, and the SAP organization were finalized.\n Activities had not been finalized for determining how\n the IRS will access the development and test\n environments to support ongoing system operations and\n maintenance once the IFS has been implemented. The\n project team had not completed SLAs for help desk\n support, response time, availability, and security for IFS\n operations and maintenance support. In addition, the\n contract needed to obtain SAP organization support had\n not been finalized. Without finalized SLAs and\n contracts, the IRS may not receive the needed level of\n support once the IFS is implemented.\n \xe2\x80\xa2 Procedures for configuration management were not\n defined. Configuration management processes were\n under development. Configuration management\n processes are needed to ensure changes to the IFS, after\n it is implemented by the CFO and ITS organizations, are\n coordinated with the PRIME contractor. Without proper\n configuration management processes, unapproved\n changes could be made to the IFS, and approved\n changes made in the production environment may not\n result in needed changes to the IFS development,\n testing, and disaster recovery environments. This could\n affect the quality of future releases. In a previous audit,\n we noted that 1 project was delayed by 4 months and\n additional costs were incurred when configuration\n management processes were not followed.15\n Management Action: Upon reviewing a preliminary\n version of this report, BSMO officials informed us that\n they agreed planning for post-implementation\n configuration management processes was significant\n and the PRIME contractor had started working the issue.\n However, BSMO officials wanted to clarify that the\n\n\n 15\n Additional Actions Are Needed to Establish and Maintain Controls\n Over Computer and Hardware Software Changes (Reference\n Number 2004-20-026, dated December 2003).\n Page 7\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n current IFS Configuration Management Plan does not\n cover post-implementation configuration management\n processes because it is not required to be updated to\n cover post-implementation until the project team begins\n working towards Milestone 5. We agree the IRS and the\n PRIME contractor should begin working the\n post-implementation configuration management issues\n now and, as stated previously, further detail could be\n provided in future updates of the TMP or related\n transition documents.\n \xe2\x80\xa2 Training delivery dates were not finalized. Due to\n schedule delays, training delivery dates were not\n finalized. These dates should be finalized as the IFS\n deployment date nears. If not, future operations and\n maintenance personnel may not be trained timely.\n We also determined that completion of the TMP was not\n part of the ELC Milestone Exit Criteria, which are used to\n determine whether modernization projects are ready to\n proceed to the next life cycle milestone. The Transition To\n Support Plan, which has been replaced by the TMP, is part\n of the milestone exit criteria, but the TMP has not yet been\n included in the ELC Milestone Exit Criteria, according to\n BSMO officials, since it is a recent requirement (as\n previously stated, the IFS project was the first project to\n deliver a TMP). If the TMP is not included as part of the\n milestone exit criteria, the IRS may not consider transition\n issues when determining whether modernization projects are\n ready to proceed to the next milestone.\n Management Action: The BSMO stated it was acting on our\n concerns as the IFS project moves toward deployment. In\n January 2004, the ITS organization approved the IFS TMP\n citing 11 conditions, such as help desk and SLAs, that need\n to be improved in the future. Also, BSMO officials\n informed us that they had created a database to track project\n transition issues. According to BSMO officials, the project\n team began using the database in May 2004.\n\n\n\n\n Page 8\n\x0c The Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n Recommendations\n\n To ensure transition issues are resolved prior to acceptance\n of the IFS by the IRS and transition management is\n considered when exiting appropriate milestones, the Chief\n Information Officer (CIO) should ensure:\n 1. Outstanding IFS transition issues are documented and\n tracked to closure.\n Management\xe2\x80\x99s Response: The IRS responded that it is now\n using a team-level issue tracking log to monitor transition\n issues. Any issues not resolved at this level will be elevated\n and monitored at weekly executive management meetings.\n 2. The ELC Milestone Exit Criteria are updated to include\n completion of the TMP.\n Management\xe2\x80\x99s Response: The IRS responded that the new\n ELC framework will require a TMP to be developed at\n different milestones during a project\xe2\x80\x99s life cycle.\n As part of our prior audit,16 we evaluated IFS testing\nTesting Practices Continue to\n activities. We found the IFS project was in the early stages\nNeed Improvement\n of testing and testing practices could be improved as the\n project advanced to more formal testing. We recommended\n IRS management correct these deficiencies prior to the\n SIT,17 to help ensure a high-quality system is delivered.\n As part of the current audit, we followed up on our previous\n testing concerns to determine if progress was being made as\n part of the SIT.18 In addition, we reviewed SIT deviation\n\n\n\n 16\n Risks Are Mounting as the Integrated Financial System Project Team\n Strives to Meet an Aggressive Implementation Date (Reference\n Number 2004-20-001, dated October 2003).\n 17\n The IFS project is also performing Systems Acceptance Testing\n (SAT) as part of the SIT. SAT independently assesses the quality of a\n system and the system\xe2\x80\x99s readiness for implementation.\n 18\n Our review of SIT activities was limited to reviews of documentation.\n Another audit is being conducted by the TIGTA Headquarters\n Operations and Exempt Organizations Programs business unit. This\n audit focuses on observing and analyzing the actual testing of system\n requirements. We will release the results of this audit in a separate audit\n report.\n Page 9\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n forms,19 defect processing,20 and security test planning\n documentation.\n We identified and communicated the following issues to the\n BSMO and Mission Assurance function during our review:\n \xe2\x80\xa2 SIT documentation could be improved to ensure testing\n objectives are being met.\n \xe2\x80\xa2 Defect management processes need improvement.\n \xe2\x80\xa2 The ST&E Plan can be improved.\n SIT documentation could be improved to ensure testing\n objectives are being met\n While the SIT is progressing and certain parts of the SIT are\n near completion, we found the IFS project team could\n improve SIT documentation to ensure testing objectives are\n being met. Based on our review of SIT documentation, we\n communicated the following concerns to the BSMO on\n February 23, 2004, March 3, 2004, and March 18, 2004.\n Entrance criteria were not always met\n In our previous audit, we determined Application\n Qualification Testing (AQT)21 test cases were not fully\n documented before the AQT began. Therefore, the AQT\n entrance criteria had not been met before AQT activities\n were started. We recommended SIT practices be\n strengthened based on lessons learned during the initial\n AQT.\n As part of our current audit, we followed up to determine if\n entrance criteria were being met before each part of the SIT\n was started. The IFS SIT Test Plan states that Test\n\n\n 19\n Test Case Deviations and Waivers Forms include requests for\n verification of a requirement or set of requirements to be moved to\n another phase or for the obligation to verify functionality, such as a\n requirement or set of requirements, be deleted.\n 20\n A defect is a perceived problem found in software, documents,\n hardware, or other controlled products. Defect processing is the process\n of documenting the defect, tracking the defect and its corrective action,\n and reporting the status of each defect.\n 21\n AQT involves testing an application before it is integrated with other\n release components. The intent is to discover and resolve errors prior to\n more formal SIT.\n Page 10\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n Readiness Reviews (TRR)22 are the forum for determining if\n entrance criteria have been met. A successful TRR inspires\n confidence in the testing team and in the adequacy of the\n tests to be performed. A TRR checklist is used to document\n the TRR meeting and the determination of test readiness.\n As required by the ELC, TRRs were conducted for each of\n the five parts of the SIT.\n No determination of test readiness was documented for one\n TRR. For three TRRs, a decision was made to start testing\n even though all activities on the TRR checklist had not been\n completed. For the final SIT part, all TRR checklist items\n were satisfied and the decision to move forward with testing\n was documented.\n In some cases, a reason was documented as to why certain\n checklist items were not met. No reasons were provided for\n others. For the part of the SIT that contains the majority of\n accounting requirements, we noted the decision was made to\n proceed with testing even though all test cases were not\n complete.\n The BSMO informed us all TRR checklist items were not\n being satisfactorily completed prior to testing because the\n PRIME contractor had the final determination as to whether\n to proceed with testing, and the PRIME contractor had\n determined it was willing to assume the risk of proceeding\n without having completed all checklist items. Without\n successful TRRs, the goal of inspiring confidence in the\n testing team and in the adequacy of the tests to be performed\n is not met, and the risk increases that testing may be delayed\n due to incomplete planning.\n The IRS initially planned to complete corrective actions\n from our prior audit regarding strengthening SIT practices\n by October 31, 2003. The IRS has extended the time\n needed to complete prior corrective actions until\n May 15, 2004. Since the IRS is in the process of\n implementing corrective actions to our previous\n recommendation in this area, we are making no additional\n recommendations regarding entrance criteria.\n\n\n 22\n The purpose of a SIT TRR is to assess risks and determine whether to\n proceed with testing.\n Page 11\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n Test cases did not include all required elements\n In our previous audit, we found that not all AQT test cases\n were complete. For example, test cases did not always\n include adequate expected results. We recommended SIT\n practices be strengthened based on lessons learned during\n the initial AQT.\n As part of our current audit, we followed up to determine if\n required test case elements were included in SIT test cases\n (see Appendix V for an explanation of the test case elements\n required by the ELC). We reviewed a random sample of\n 85 test cases to identify whether they were being developed\n in accordance with guidelines. We found that 21 (about\n 25 percent) of the 85 test cases we reviewed did not include\n all required elements, such as expected results.\n Without all required test case elements, the tester may not\n know what the test is meant to achieve and what should be\n done before and after the test. The ELC provides a template\n to make required test case elements easily identifiable. The\n PRIME contractor informed us it was not completely filling\n out the required test case template, but the information\n could be found in the detailed test steps. Since the\n information in the test steps is not labeled, we could not\n determine if the information in the detailed test steps\n included all required test case elements. The PRIME\n contractor also informed us it would be making corrections\n to put the required test case elements in the appropriate part\n of the test case form. At the conclusion of our audit in\n March 2004, this had not been done.\n The IRS initially planned to complete corrective actions\n from our prior audit regarding strengthening SIT practices\n by October 31, 2003. The IRS has extended the time\n needed to complete prior corrective actions until\n May 15, 2004. Since the IRS is in the process of\n implementing corrective actions to our previous\n recommendation in this area, we are making no additional\n recommendations regarding test case elements.\n The Requirements Traceability Verification Matrix (RTVM)\n and related documentation were incomplete\n The SIT Test Plan includes several key testing documents:\n\n Page 12\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n \xe2\x80\xa2 RTVM \xe2\x80\x93 The RTVM is a tool used to ensure each\n system requirement is tested by assigning it to one or\n more test cases. This tool helps ensure all requirements\n are tested.\n \xe2\x80\xa2 Software Documentation Automation (SoDA)\n Report \xe2\x80\x93 The SoDA Report includes a listing of all\n documented test cases.\n \xe2\x80\xa2 Test Execution Schedule \xe2\x80\x93 Test execution schedules are\n developed by the test team to manage testing activities\n by listing the date when each test case is estimated to\n start and complete. The schedules should provide one\n entry for each planned test case.\n In our previous audit, we found the RTVM was not\n complete because the AQT test execution schedule referred\n to test cases that were not in the AQT RTVM. We\n recommended SIT practices be strengthened based on\n lessons learned during the initial AQT.\n As part of our current audit, we followed up to determine if\n the SIT RTVM was complete. Our objective was to\n determine if the SIT RTVM contained all requirements and\n the requirements were properly linked to the appropriate,\n documented test cases as required by the ELC.\n Additionally, the ELC states that the RTVM should map\n requirements to test cases and the test execution schedule\n should provide one entry for each planned test case.\n Throughout the audit, we received RTVMs for each of the\n different parts of the SIT. For the part that contains the\n majority of accounting requirements, we compared the IFS\n test execution schedule to the SoDA Report to determine\n whether scheduled test cases had been documented. We\n determined that 227 (approximately 31 percent) of 731 test\n cases listed in the test execution schedule were not\n documented in the SoDA Report.\n The inconsistency between the test execution schedule and\n the SoDA Report was due, in part, to test case naming\n inconsistencies. For instance, one test case in the execution\n schedule was called IFS-R1-SIT2-TAXES-001. In the\n SoDA Report, this test case was called IFS-R1-SIT2-\n TAXES_1. The PRIME contractor responded this was due\n\n Page 13\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n to the fact that the test execution schedule was created\n manually and typographical errors were always possible.\n In March 2004, we received RTVMs for all parts of the SIT.\n The RTVMs did not always map to test cases. Specifically,\n 85 (approximately 3 percent) out of 2,696 initial IFS\n Release 1.0 requirements23 were not mapped to test cases.\n In addition, we received SoDA Reports for all five parts of\n the SIT. We compared the test case names listed in the\n RTVMs to the SoDA Reports and found the SoDA Reports\n did not contain 472 (approximately 37 percent) of 1,283 test\n cases referred to in the RTVMs.\n While we were unable to determine the cause of each\n inconsistency between the RTVMs and SoDA Reports, a\n recent internal study indicated that the PRIME contractor\n needed to enforce requirements definition and program\n management compliance. In addition, the PRIME\n contractor stated the SIT was a work in progress and the\n final SIT report would contain a complete, correct reference\n to all test cases in an RTVM and SoDA Report. Without a\n complete and accurate test execution schedule, RTVM, and\n documented test cases, the IRS cannot determine if all\n system requirements are scheduled to be tested and if all test\n cases are documented.\n The IRS initially planned to complete corrective actions\n from our prior audit regarding strengthening SIT practices\n by October 31, 2003. The IRS has extended the time\n needed to complete prior corrective actions until\n May 15, 2004. Since the IRS is in the process of\n implementing corrective actions to our previous\n recommendation in this area, we are making no additional\n recommendations regarding inconsistencies between the\n RTVM, test execution schedules, and SoDA Reports.\n\n\n\n\n 23\n As part of our analysis, we took into account any approved changes to\n the initial IFS Release 1.0 requirements, as defined in the IFS system\n requirements report.\n Page 14\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n Some requirements did not include a verification and\n validation method\n The ELC requires that all system requirements include a\n verification and validation (V&V) method.24 We found\n 12 (approximately 26 percent) out of 47 systems\n requirements to assist disabled individuals in using\n technology,25 which were listed in the RTVM and did not\n include a V&V method. Without a V&V method, the IRS\n cannot determine how requirements will be verified and\n validated. The PRIME contractor stated that each of these\n requirements had a verification and validation method;\n however, the documentation was not up-to-date.\n Deviation forms did not include all signatures\n The ELC provides guidance on the signatures required for\n moving and deleting requirements from a release. One\n required signature for the IFS project is that of the IRS\n Internal Management Program Director. During the audit,\n the project team prepared deviation forms for\n 36 requirements. However, the deviation forms did not\n include a signature line for the Internal Management\n Program Director. Without all required signatures, the\n deviation forms are not being adequately approved. The\n PRIME contractor stated it was not aware of the\n requirement to have the IRS Internal Management Program\n Director approve the deviation forms.\n Defect management processes need improvement\n A defect occurs during testing when the actual results differ\n from the expected results documented in the test cases.\n\n\n 24\n To verify and validate that system requirements are being met, a\n system may be tested, analyzed, or inspected, or a demonstration of a\n system\xe2\x80\x99s capability could be conducted.\n 25\n Rehabilitation Act of 1973, Pub. L. No. 93-112, 87 Stat. 355 (codified\n as amended in scattered sections of 15 U.S.C., 20 U.S.C., 29 U.S.C.,\n 36 U.S.C., 41 U.S.C., and 42 U.S.C.). In 1998, the Congress amended\n the Rehabilitation Act to require Federal Government agencies to make\n their electronic and information technology accessible to people with\n disabilities. Section 508 was enacted to eliminate barriers in\n information technology, to make available new opportunities for people\n with disabilities, and to encourage development of technologies that will\n help achieve these goals.\n Page 15\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n Defects discovered during each part of testing are\n documented, fixed, and retested.\n The PRIME contractor\xe2\x80\x99s responsibilities include ensuring\n the defect reporting and tracking process is used to report\n and track defects for all modernization projects. Defects\n encountered during testing are entered into the PRIME\n contractor\xe2\x80\x99s Defect Report Tracking System, which is used\n to verify defect resolution and the closing of defects when\n tests have been successfully completed.\n Based on our review of IFS defect reports, we\n communicated the following concerns to the BSMO on\n February 23, 2004.\n IFS SIT defects have exceeded the average number of\n defects for a project of its size and complexity\n The BSM Performance Management Office performs data\n analyses using a collection of industry data and provides an\n estimate of the total number of defects that will be found for\n each modernization project. The BSM Performance\n Management Office determined that projects similar to the\n IFS in size and complexity produced an average of\n 121 defects.\n As of January 29, 2004, the IFS project team reported that it\n had encountered 190 defects. As mentioned previously, the\n IFS project team has delayed the IFS deployment date until\n October 2004 and is continuing to test. Therefore, the\n number of defects will undoubtedly increase.\n We asked the PRIME contractor if it had performed any\n analyses to determine if there was a reason why there were\n so many defects and what actions could be taken to reduce\n the number of future defects. The PRIME contractor stated\n it had performed a defect causal analysis, but as of\n March 31, 2004, the PRIME contractor had not provided us\n this analysis.26 Without an understanding of why so many\n\n\n 26\n Our audit work was completed in March; we were provided with a\n copy of the IFS defect causal analysis on April 12, 2004. The analysis\n did not provide insight into the entire defect population. However, the\n analysis did provide an explanation for the decision to colocate\n developers with testers, which could reduce the amount of time it takes\n to correct defects.\n Page 16\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n defects have been encountered, the IFS project team may\n continue to experience testing delays, and future\n modernization projects may not learn lessons based on the\n experience of the IFS project team.\n Management Action: The BSMO requested that the PRIME\n contractor review the input that had been provided\n previously on the size and complexity of the IFS project.\n The PRIME contractor responded with revised size figures.\n Based on the revised figures, the BSMO recomputed the\n average number of defects for projects similar to IFS in size\n and complexity. The new average number of defects is 440.\n While the new average number of defects has increased, the\n number of actual defects encountered by the IFS project\n team has also increased. As of April 20, 2004, the BSMO\n estimates the IFS project team has encountered 30 percent\n more defects than comparable projects.\n Defects were not being evaluated and resolved timely\n The ELC provides goals for timely evaluating and resolving\n defects based on the severity of each defect. See\n Appendix VI for a listing of these goals.\n We randomly selected 50 (approximately 26 percent) of the\n 190 defects documented as of January 29, 2004. Six\n (12 percent) of the 50 defects were not evaluated timely, and\n 42 (84 percent) defects were not resolved timely.\n Without an understanding of why defects are not evaluated\n and resolved timely, the IFS project team could continue to\n experience testing delays, and future modernization projects\n may not learn lessons based on the experience of the IFS\n project team.\n The IFS project team documented this issue and created an\n action plan. One of the actions was for the PRIME\n contractor to \xe2\x80\x9canalyze why SIT didn\xe2\x80\x99t meet the throughput\n and develop a mitigation strategy to prevent reoccurrence.\xe2\x80\x9d\n The PRIME contractor could not provide any\n documentation on this mitigation strategy; however, it stated\n the mitigation strategy involved making sure developers are\n onsite with testers to view testing and understand any\n defects that are encountered. The PRIME contractor also\n informed us it was now passing more test cases due to this\n new strategy. We determined that the IFS project team had\n Page 17\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n passed 171 test cases from the last workday of February\n through the month of March, compared to passing 96 test\n cases during the latter part of December and the month of\n January.\n The ST&E Plan can be improved\n The Department of the Treasury Directive 71-10\n (August 23, 1999) establishes the security policy for\n requiring formal reviews of computer systems that will\n become operational. The IRS Internal Revenue Manual\n (IRM) provides policies and guidance to be used by IRS\n organizations to carry out their respective responsibilities in\n information systems security. The IRM indicates that\n testing and evaluation of computer systems must validate\n the correct implementation of security features.\n The Office of Mission Assurance and its contractor are\n responsible for conducting ST&Es for all modernization\n projects. An ST&E involves the planning and execution of\n security tests and the evaluation and analysis of the\n subsequent test results. The goals of the ST&E are to\n identify the security profile of a particular system and to\n assess whether the system configuration and controls meet\n the security requirements and are ready for operation.\n We reviewed the ST&E Plan and found it included an\n understandable outline of the ST&E that would be\n conducted for the IFS application. In addition, the ST&E\n Plan included many test cases designed to ensure security\n requirements are met.\n We also found areas in which the ST&E Plan could be\n improved, and we communicated the following concerns to\n the BSMO and Office of Mission Assurance on\n December 17, 2003, and February 4, 2004.\n The ST&E Plan does not completely comply with the IRM\n The IRM includes a template to be used when developing\n the ST&E Plan, but the IFS ST&E Plan does not follow this\n template. Therefore, we reviewed the IFS ST&E Plan to\n determine if it included the information in the ST&E\n template. For example, the ST&E template states that the\n Plan should describe the requirements, goals, and functions\n of the security system; why the system security is needed,\n\n Page 18\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n what the system security should do, and how well the\n system should do it; and the system security functions as\n they relate to the system requirements. However, we were\n unable to identify these features within the IFS ST&E Plan.\n The Office of Mission Assurance informed us the IRM\n template was just a suggested format and further\n information could be added to the ST&E Plan as the test\n draws nearer. Templates are created to bring about\n standardization. When templates are not followed, needed\n information may not be included and lessons learned that\n led to the creation of the template may have to be relearned.\n Some required security features are not being tested\n The IRM states that testing and evaluation must validate the\n correct implementation of security features, including\n identification and authentication, audit capability, access\n controls, object reuse, user accountability, trusted recovery,\n network connectivity, and transmission encryption (if\n applicable).\n Trusted recovery27 test cases were not included in the IFS\n ST&E Plan, and object reuse28 was being tested on only one\n of two computer platforms. The Office of Mission\n Assurance informed us the trusted recovery controls were\n not being tested because testing these controls was too\n costly. The Office of Mission Assurance also informed us it\n was testing object reuse on both computer platforms;\n however, the test case it referred us to for one platform was\n not a test for object reuse. Furthermore, no other test cases\n for object reuse could be located for this platform. If these\n tests are not conducted, the IRS will not have assurance the\n IFS meets all security requirements.\n Entrance and exit criteria are not documented in the ST&E\n Plan\n A best practice in test planning is to create a predefined set\n of criteria used to determine when a test is ready to be\n\n\n 27\n Trusted recovery controls refer to procedures and/or mechanisms\n available to assure that, after a system failure or other discontinuity,\n recovery without compromise is possible.\n 28\n Object reuse controls are designed to ensure the authorized user of the\n system does not obtain residual data.\n Page 19\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n conducted (entrance criteria) and when a test has been\n completed at an acceptable quality level (exit criteria). The\n ELC requires that entrance and exit criteria be defined for\n all tests, except the ST&E.\n The Office of Mission Assurance informed us it was not\n required to document entrance and exit criteria in the ST&E\n Plan; however, it was aware of the conditions that must\n exist for the ST&E to start and complete. Without\n documented entrance and exit criteria, ST&Es could start\n prematurely or stop prior to expected results being achieved.\n\n Recommendations\n\n To ensure IFS testing results in a high-quality product and\n future modernization projects are improved by lessons\n learned by the IFS project team, the CIO should ensure:\n 3. The RTVM is updated to include a verification and\n validation method for all requirements.\n Management\xe2\x80\x99s Response: The IRS responded that\n the RTVM will be updated. In addition, the new ELC\n framework will require updates to the RTVM during the\n project\xe2\x80\x99s life cycle.\n 4. Deviation forms are updated to include a signature line\n for all required approving officials.\n Management\xe2\x80\x99s Response: The IRS is now including the\n correct signature line on deviation forms.\n 5. The root cause is investigated and resolved for why\n defects are more voluminous than expected and for why\n defects are being evaluated and resolved slower than\n expected.\n Management\xe2\x80\x99s Response: The IRS agreed with our\n recommendation and will ensure that a lessons learned\n report is generated. In addition, the IRS revised its defect\n estimate and found the project was within expectations once\n a corrected defect estimate was calculated.\n\n\n\n\n Page 20\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n 6. The ELC is updated to include a requirement for\n entrance and exit criteria in the ST&E Plan.\n Management\xe2\x80\x99s Response: The IRS will be incorporating\n changes into the milestone exit requirements.\n Office of Audit Comment: The CIO agreed with our\n recommendation; however, the corrective action provided\n will not adequately address our recommendation. Changing\n the milestone exit criteria will not create a requirement for\n entrance and exit criteria in the ST&E Plan.\n To ensure planning for the upcoming IFS ST&E is as\n thorough as possible, the Chief, Mission Assurance, should\n ensure:\n 7. All needed elements from the IRM are included in the\n ST&E Plan.\n Management\xe2\x80\x99s Response: The IRS responded that the IFS\n ST&E Plan or its associated documents will be updated to\n address needed elements as identified in the IRM.\n\n 8. Trusted recovery and object reuse testing occurs or the\n risk of not conducting these tests is documented.\n Management\xe2\x80\x99s Response: The CIO disagreed with our\n recommendation to ensure trusted recovery and object reuse\n testing occurs, or the risk of not conducting these tests is\n documented. The IRS responded that object reuse testing is\n no longer required by the Department of the Treasury.\n Office of Audit Comment: While the Department of the\n Treasury requirement may have changed, the IRM still\n requires object reuse testing. Therefore, we believe the IRS\n should conduct the required testing, particularly since it\n plans to perform object reuse testing on one computer\n platform and not the other. In addition, the IRS\xe2\x80\x99 response\n did not address trusted recovery testing.\n\n\n\n\n Page 21\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n Appendix I\n\n\n Detailed Objectives, Scope, and Methodology\n\nThe overall objective of this review was to determine if adequate testing processes and\nprocedures were being followed prior to the deployment of the Integrated Financial System (IFS)\nto ensure the system meets expectations. Additionally, we determined if the IFS project team\nwas adequately planning for the transition1 of operation and maintenance of the system from the\ncontractor to the Internal Revenue Service (IRS).\nTo achieve these objectives, we:\nI. Determined if testing support processes for the IFS Release 1.0 were being adequately\n planned for deployment.\n A. Determined if all entrance and exit criteria were being met prior to initiating and\n exiting the various parts of the System Integration Testing (SIT).2\n B. Determined if the IFS project team was adequately controlling waivers and deferrals.\n C. Determined if all aspects of the IFS application would be tested during the SIT.\n D. Determined if test cases included required elements and all scheduled test cases were\n documented. NOTE: The PRIME contractor provided 523 test cases for the second\n part of SIT and 103 test cases for the third part of SIT. To accomplish this audit step,\n we chose a random sample of 75 test cases (approximately 14 percent) from the\n second part of SIT and 10 test cases (approximately 10 percent) from the third part of\n SIT for review. We did not use a statistical sample because we were not going to\n project the results to the entire population.\n E. Determined the quality of the product the PRIME contractor3 had delivered for the\n SIT by comparing the actual number of defects with the estimated number of defects.\n F. Determined if the defect reporting process was being followed and resolved in\n accordance with the Enterprise Life Cycle (ELC)4 during each part of the SIT.\n NOTE: The PRIME contractor provided 190 defect reports from SIT as of\n January 29, 2004. To accomplish this audit step, we chose a random sample of\n\n\n1\n We focused only on activities required to transition the system to the Internal Revenue Service for operation and\nmaintenance. See Appendix IV for an overview of the Enterprise Life Cycle.\n2\n The SIT ensures all system components (hardware and software) are working correctly and collectively with other\nrelated or dependent systems.\n3\n To facilitate success of its modernization efforts, the IRS hired the Computer Sciences Corporation as the PRIME\ncontractor and integrator for the Business Systems Modernization (BSM) program and created the Business Systems\nModernization Office to guide and oversee the work of the PRIME contractor.\n4\n BSM processes and procedures are documented within the ELC. See Appendix IV for an overview of the ELC.\n Page 22\n\x0c The Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n 50 defect reports (approximately 26 percent) for review. We did not use a statistical\n sample because we were not going to project the results to the entire population.\nII. Determined if the IRS adequately planned for the Security Test and Evaluation (ST&E) by\n reviewing the ST&E Plan.\nIII. Determined if adequate plans were in place to ensure the Information Technology Services\n (ITS) and the Chief Financial Officer (CFO) organizations would be ready to provide\n technical and infrastructure support for the IFS Release 1.0 after Milestone 5.5\n A. Determined if all training needed to accomplish the transfer of the IFS Release 1.0 to\n the IRS had been identified and scheduled.\n B. Determined if post-Milestone 5 roles and responsibilities had been defined for the\n CFO and ITS organizations and for the PRIME contractor.\n C. Determined if the IFS project team had documented how to handle changes to the\n Release 1.0 software after Milestone 5.\n D. Compared the steps being taken to transfer knowledge about the IFS Release 1.0 to\n that of the Custodial Accounting Project (CAP) Release 1.0 to determine if there\n were any additional items the IFS project team should be taking into account.\n NOTE: The CAP is a similar project that should have similar transition activities.\n E. Reviewed the IFS Transition Management Plan and determined if it included all\n required ELC elements.\n F. Determined if the risk reduction plans regarding ITS readiness were on track and key\n activities were being completed.\n NOTE: Additional work was scheduled concerning the IFS Deployment Site Readiness Test\n (DSRT).6 Due to delays in IFS testing, at the time of our audit the IFS DSRT Test Plan was not\n available for review and DSRT testing had not been initiated.\n\n\n\n\n 5\n Milestone 5 refers to the activities that need to be completed before the IRS formally accepts a release. See\n Appendix IV for an overview of the ELC.\n 6\n The DSRT assesses whether a system as installed is working well enough to support enterprise-wide business use.\n\n\n\n\n Page 23\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n Appendix II\n\n\n Major Contributors to This Report\n\nMargaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs)\nGary V. Hinkle, Director\nTroy D. Paterson, Audit Manager\nMark K. Carder, Senior Auditor\nCharlene L. Elliston, Auditor\nPerrin T. Gleaton, Auditor\n\n\n\n\n Page 24\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n Appendix III\n\n\n Report Distribution List\n\nCommissioner C\nOffice of the Commissioner \xe2\x80\x93 Attn: Chief of Staff C\nAssociate Chief Information Officer, Business Systems Modernization OS:CIO:B\nAssociate Chief Information Officer, Management OS:CIO:M\nAssociate Chief Information Officer, Modernization Management OS:CIO:MM\nDeputy Associate Chief Information Officer, Program Management OS:CIO:B:PM\nActing Director, Internal Management Modernization OS:CIO:B:PM:IMM\nDirector, Stakeholder Management OS:CIO:SM\nChief Counsel CC\nNational Taxpayer Advocate TA\nDirector, Office of Legislative Affairs CL:LA\nDirector, Office of Program Evaluation and Risk Analysis RAS:O\nOffice of Management Controls OS:CFO:AR:M\nAudit Liaisons:\n Associate Chief Information Officer, Business Systems Modernization OS:CIO:B\n Chief, Mission Assurance OS:MA\n\n\n\n\n Page 25\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n Appendix IV\n\n\n Enterprise Life Cycle Overview\n\nThe Enterprise Life Cycle (ELC) defines the processes, products, techniques, roles,\nresponsibilities, policies, procedures, and standards associated with planning, executing, and\nmanaging business change. It includes redesign of business processes; transformation of the\norganization; and development, integration, deployment, and maintenance of the related\ninformation technology applications and infrastructure. Its immediate focus is the Internal\nRevenue Service (IRS) Business Systems Modernization (BSM) program. Both the IRS and the\nPRIME contractor1 must follow the ELC in developing/acquiring business solutions for\nmodernization projects.\nThe ELC framework is a flexible and adaptable structure within which one plans, executes, and\nintegrates business change. The ELC process layer was created principally from the Computer\nSciences Corporation\xe2\x80\x99s Catalyst\xc2\xae methodology.2 It is intended to improve the acquisition, use,\nand management of information technology within the IRS; facilitate management of large-scale\nbusiness change; and enhance the methods of decision making and information sharing. Other\ncomponents and extensions were added as needed to meet the specific needs of the IRS BSM\nprogram.\n\nELC Processes\n\nA process is an ordered, interdependent set of activities established to accomplish a specific\npurpose. Processes help to define what work needs to be performed. The ELC methodology\nincludes two major groups of processes:\n Life-Cycle Processes, which are organized into phases and subphases and which address all\n domains of business change.\n Management Processes, which are organized into management areas and which operate\n across the entire life cycle.\n\n\n\n\n1\n To facilitate success of its modernization efforts, the IRS hired the Computer Sciences Corporation as the PRIME\ncontractor and integrator for the BSM program and created the Business Systems Modernization Office to guide and\noversee the work of the PRIME contractor.\n2\n The IRS has acquired a perpetual license to Catalyst\xc2\xae as part of the PRIME contract, subject to certain restrictions.\nThe license includes rights to all enhancements made to Catalyst\xc2\xae by the Computer Sciences Corporation during the\ncontract period.\n Page 26\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n Enterprise Life-Cycle Processes\n\n\n IRS Governance and Investment Decision Management\n Management\n\n\n\n\n Program Management and Project Management\n\n Architectural Engineering / Development Coordination\n\n Management Support Processes\n\n\n Business Processes\n\n\n\n\n Operations & Support\n Applications\n Vision & Strategy\n\n\n\n\n Development\n Architecture\n\n\n\n\n Deployment\n Integration\n Life Cycle\n\n\n\n\n Data\n\n\n Organizational Change\n\n\n Technical Infrastructure\n\n\n Facilities Infrastructure\n\n\nSource: ELC Guide, Page 2-16.\n\n\n\n\nLife-Cycle Processes\n\nThe life-cycle processes of the ELC are divided into six phases, as described below:\n\xe2\x80\xa2 Vision and Strategy - This phase establishes the overall direction and priorities for\n business change for the enterprise. It also identifies and prioritizes the business or system\n areas for further analysis.\n\xe2\x80\xa2 Architecture - This phase establishes the concept/vision, requirements, and design for a\n particular business area or target system. It also defines the releases for the business area\n or system.\n\n\n Page 27\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n\xe2\x80\xa2 Development - This phase includes the analysis, design, acquisition, modification,\n construction, and testing of the components of a business solution. This phase also\n includes routine planned maintenance of applications.\n\xe2\x80\xa2 Integration - This phase includes the integration, testing, piloting, and acceptance of a\n release. In this phase, the integration team brings together individual work packages of\n solution components developed or acquired separately during the Development phase.\n Application and technical infrastructure components are tested to determine whether they\n interact properly. If appropriate, the team conducts a pilot to ensure all elements of the\n business solution work together.\n\xe2\x80\xa2 Deployment - This phase includes preparation of a release for deployment and actual\n deployment of the release to the deployment sites. During this phase, the deployment\n team puts the solution release into operation at target sites.\n\xe2\x80\xa2 Operations and Support - This phase addresses the ongoing operations and support of\n the system. It begins after the business processes and system(s) have been installed and\n have begun performing business functions. It encompasses all of the operations and\n support processes necessary to deliver the services associated with managing all or part\n of a computing environment.\n The Operations and Support phase includes the scheduled activities, such as planned\n maintenance, systems backup, and production output, as well as the nonscheduled\n activities, such as problem resolution and service request delivery, including emergency\n unplanned maintenance of applications. It also includes the support processes required to\n keep the system up and running at the contractually specified level.\n\nManagement Processes\n\nBesides the life-cycle processes, the ELC also addresses the various management areas at the\nprocess level. The management areas include:\n\xe2\x80\xa2 IRS Governance and Investment Decision Management - This area is responsible for\n managing the overall direction of the IRS, determining where to invest, and managing the\n investments over time.\n\xe2\x80\xa2 Program Management and Project Management - This area is responsible for\n organizing, planning, directing, and controlling the activities within the program and its\n subordinate projects to achieve the objectives of the program and deliver the expected\n business results.\n\xe2\x80\xa2 Architectural Engineering/Development Coordination - This area is responsible for\n managing the technical aspects of coordination across projects and disciplines, such as\n managing interfaces, controlling architectural changes, ensuring architectural compliance,\n maintaining standards, and resolving issues.\n\n Page 28\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n\xe2\x80\xa2 Management Support Processes - This area includes common management processes,\n such as quality management and configuration management that operate across multiple\n levels of management.\n\nMilestones\n\nThe ELC establishes a set of repeatable processes and a system of milestones, checkpoints, and\nreviews that reduce the risks of systems development, accelerate the delivery of business\nsolutions, and ensure alignment with the overall business strategy. The ELC defines a series of\nmilestones in the life-cycle processes. Milestones provide for \xe2\x80\x9cgo/no-go\xe2\x80\x9d decision points in the\nproject and are sometimes associated with funding approval to proceed. They occur at natural\nbreaks in the process where there is new information regarding costs, benefits, and risks and\nwhere executive authority is necessary for next phase expenditures.\nThere are five milestones during the project life cycle:\n\xe2\x80\xa2 Milestone 1 - Business Vision and Case for Action. In the activities leading up to\n Milestone 1, executive leadership identifies the direction and priorities for IRS business\n change. These guide which business areas and systems development projects are funded\n for further analysis. The primary decision at Milestone 1 is to select BSM projects based\n on both the enterprise-level Vision and Strategy and the Enterprise Architecture.\n\xe2\x80\xa2 Milestone 2 - Business Systems Concept and Preliminary Business Case. The\n activities leading up to Milestone 2 establish the project concept, including requirements\n and design elements, as a solution for a specific business area or business system. A\n preliminary business case is also produced. The primary decision at Milestone 2 is to\n approve the solution/system concept and associated plans for a modernization initiative\n and to authorize funding for that solution.\n\xe2\x80\xa2 Milestone 3 - Business Systems Design and Baseline Business Case. In the activities\n leading up to Milestone 3, the major components of the business solution are analyzed\n and designed. A baseline business case is also produced. The primary decision at\n Milestone 3 is to accept the logical system design and associated plans and to authorize\n funding for development, test, and (if chosen) pilot of that solution.\n\xe2\x80\xa2 Milestone 4 - Business Systems Development and Enterprise Deployment Decision.\n In the activities leading up to Milestone 4, the business solution is built. The system is\n integrated with other business systems and tested, piloted (usually), and prepared for\n deployment. The primary decision at Milestone 4 is to authorize the release for\n enterprise-wide deployment and commit the necessary resources.\n\xe2\x80\xa2 Milestone 5 - Business Systems Deployment and Postdeployment Evaluation. In the\n activities leading up to Milestone 5, the business solution is fully deployed, including\n delivery of training on use and maintenance. The primary decision at Milestone 5 is to\n\n\n Page 29\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n authorize the release of performance-based compensation based on actual, measured\n performance of the business system.\n\n\n\n\n Page 30\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n Appendix V\n\n\n Test Case Elements Required by the Enterprise Life Cycle\n\nThe Enterprise Life Cycle requires the following elements to be included in test cases.\n\nTEST CASE IDENTIFIER\n\nThe test case identifier should identify the PROJECT and/or TEST PHASE, RELEASE,\nCONFIGURATION ITEM and/or REQUIREMENTS CATEGORY (Functional, Security,\nPerformance, Regression, etc.), TEST CASE NUMBER (unique alpha numeric or numeric\nidentifier), and, if the same test is to be executed at multiple locations or against multiple\ncomponents, a LOCATION identifier.\n\nTEST SCRIPT IDENTIFIER\n\nThe test script identifier should specify the unique identifier of each script necessary to exercise\nthe functionality and requirements.\n\nTEST CASE DESCRIPTION\n\nThe test case description provides the condition (or object or application state) being tested, use\ncase, use-case scenario, or technical or supplemental requirement from which the test case is\nderived.\n\nREQUIREMENT CATEGORY\n\nThe requirement category helps to categorize test cases. Examples of test types are Performance,\nCycle, Data Conversion, Security, and External Function.\n\nVERIFICATION AND VALIDATION METHOD\n\nThe verification and validation method classifies the approach for ensuring the implementation\nof the requirements(s) such as analysis, inspection, test, and demonstration.\n\nREQUIREMENTS\n\nRequirement number and description, also called test inputs, must be provided to identify the\nfunctionality a test is validating. Examples of test inputs are requirements, model elements,\nspreadsheet values, etc.\n\n Page 31\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\nPRECONDITIONS\n\nA precondition is the description of the constraints a test case requires before it is run. Examples\nof preconditions include:\n\xe2\x80\xa2 Other test case dependencies.\n\xe2\x80\xa2 Input data conditions.\n\xe2\x80\xa2 Input files.\n\xe2\x80\xa2 Preparation activities.\n\xe2\x80\xa2 Account or password setup.\n\nPOSTCONDITIONS\n\nThe postconditions of a test case are actions required to restore the tested system to its original\nstate once the test has passed.\n\nCONFIGURATION\n\nThe configuration refers to the hardware and software components and specific settings required\nfor proper execution of the test case.\n\nACCEPTANCE CRITERIA\n\nThe acceptance criteria are the expected result stated in terms of the output state, condition, or\ndata value(s) that provides evidence or proof a requirement has been satisfied.\n\n\n\n\n Page 32\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n Appendix VI\n\n\n Defect Tracking and Resolution Time Periods\n\nTable 1 depicts the goals for evaluating and resolving defect reports.\n Table 1: Severity and Schedule of Resolution\n Severity Schedule of Resolution\n 1 \xe2\x80\x93 Critical Development team evaluates the defect report (DR) within an hour and works the\n resolution around the clock with the goal of delivering a fix within 24 hours, unless\n otherwise scheduled.\n 2 \xe2\x80\x93 High Development team evaluates the DR within 1 business day with the goal of\n delivering a fix within 3 business days, unless otherwise scheduled.\n 3 \xe2\x80\x93 Medium Development team evaluates the DR within 3 business days with the goal to deliver\n all severity 3 and 4 fixes. The effort for resolving severity 3 and 4 DRs will be\n assessed according to the number of severity 1 and 2 DRs being worked, the\n 4 \xe2\x80\x93 Low\n remaining test schedule, and management decisions to defer fixes to a later release.\nSource: Enterprise Life Cycle (ELC). See Appendix IV for an overview of the ELC.\n\n\n\n\n Page 33\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n Appendix VII\n\n\n Management\xe2\x80\x99s Response to the Draft Report\n\n\n\n\n Page 34\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n\n\n\n Page 35\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n\n\n\n Page 36\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n\n\n\n Page 37\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n\n\n\n Page 38\n\x0cThe Integrated Financial System Project Team Needs to Resolve Transition Planning and\n Testing Issues to Increase the Chances of a Successful Deployment\n\n\n\n\n Page 39\n\x0c'


type: wikipedia

text:
Bhí Laurence Olivier (22 Bealtaine 1907 – 11 Iúil 1989) ar na haisteoirí is mó sa 20ú haois. B'ollmhór an cháil a bhain sé amach mar aisteoir stáitse, aisteoir scannán agus mar stiúrthóir. Tá cáil ar leith air mar gheall ar an aisteoireacht a rinne sé i ndrámaí Shakespeare. Bhuaigh sé an gradam Aisteoir is Fearr sna gradaim Golden Globe i 1949.
 Olivier, Laurence
 Olivier, Laurence
 Olivier, Laurence
 Buaiteoirí gradam Aisteoir is Fearr sna gradaim Academy
 Buaiteoirí gradam Aisteoir is Fearr i nDráma sna gradaim Golden Globe