jurisdiction: US

text: b'March 26, 2004\nReport No. 04-014\n\n\nXBAT Contracting and Project Management\n\n\n\n\n EVALUATION REPORT\n\x0cFederal Deposit Insurance Corporation Office of Audits\n801 17th St. NW Washington DC, 20434 Office of Inspector General\n\n\n\nDATE: March 26, 2004\n\nMEMORANDUM TO: Arthur Murton, Director\n Division of Insurance and Research\n\n Michael E. Bartell, Chief Information Officer and Director\n Division of Information Resources Management\n\nFROM: Russell A. Rau [Electronically produced version; original signed by Russell A. Rau]\n Assistant Inspector General for Audits\n\n Michael MacDermott [Electronically produced version; original signed by Michael\n MacDermott]\n Acting Director, Office of Internal Control Management\n\nSUBJECT: XBAT Contracting and Project Management\n Capital (Report No. 04-014)\n\nThe subject final report is provided for your information and use. Please refer to the Executive\nSummary section for the overall results of our review. Our evaluation of responses is\nincorporated into the body of the report, and the Division of Administration\xe2\x80\x99s written response is\nincluded in its entirety as an appendix to the report.\nIf you have any questions concerning the report, please contact Stephen Beard, Deputy Assistant\nInspector General for Audits, at (202) 416-4217, or Marshall Gentry, Director, Corporate\nEvaluations, at (202) 416-2919. We appreciate the courtesies extended to the evaluation staff.\n\n\nAttachment\n\ncc: Arleas Upton Kea, DOA\n\x0c Table of Contents\n\n\n\n\nTable of Contents\n\n\nEXECUTIVE SUMMARY.....................................................................................................1\n\nBACKGROUND...................................................................................................................3\n\nEVALUATION RESULTS....................................................................................................5\n\n Functional Requirements............................................................................................5\n Procurement Planning and Execution ......................................................................8\n Contract Administration............................................................................................14\n Change Management and Project Integration..................................................\xe2\x80\xa6. 17\n Communications ........................................................................................................22\n System Development Life Cycle ..............................................................................25\n\nCORPORATION COMMENTS AND OIG EVALUATION\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6.29\n\nAPPENDIX I: Objective, Scope , and Methodology...............\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6..\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6.30\n\nAPPENDIX II: XBAT Contracting and Project Management Challenges...................33\n\nAPPENDIX III: Key Events in the XBAT Project.......................................................\xe2\x80\xa6.36\n\nAPPENDIX IV: XBAT Procurement Requirements...................................................\xe2\x80\xa637\n\nAPPENDIX V: Corporation Comments\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6.. .\xe2\x80\xa6\xe2\x80\xa6. 38\n\nTABLES\n\n Table 1: Key Lessons to be Learned From the XBAT Development Effort\xe2\x80\xa6.......2\n Table 2: Call Mod Project Objectives\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6...3\n Table 3: XBAT Functional Requirements\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa66\n Table 4: APM Task Order Requirements\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6..\xe2\x80\xa6\xe2\x80\xa6...9\n Table 5: XBAT Contract Award and Execution Departures\n From APM Provisions\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6.\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa612\n Table 6: SDLC Testing Requirements.....................................................................27\n Table 7: XBAT Invoices and Payment Dates\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6.\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6.\xe2\x80\xa6..27\n\nFIGURES\n\n Figure 1: XBRL Business Analyst Tool Estimated and Actual\n Development Costs\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6..\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6..\xe2\x80\xa611\n Figure 2: Project Integration Timeline\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6..\xe2\x80\xa6\xe2\x80\xa6..\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6.\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6\xe2\x80\xa6..21\n\x0c Executive Summary\n\n\n\n\nExecutive Summary\nAt the request of Corporation senior managers, we evaluated the Extensible Business Reporting\nLanguage (XBRL) Business Analyst Tool (XBAT) contracting and development effort. The\nXBAT is an important component of the larger Call Report Processing Modernization (Call Mod)\nproject, an FDIC Chairman initiative and interagency effort to develop a centralized data\ncollection, validation, integration, and distribution system to replace the existing Call Report\nprocess. A key part of the Call Mod project is the implementation of a Central Data Repository\n(CDR) that will use emerging technologies to effectively collect and process Call Report data.\nFDIC tasked the Institution Data Management project team (IDM team), within the Division of\nInsurance and Research (DIR), with implementing the XBAT and CDR efforts. The FDIC\nengaged a contractor to develop the first version of XBAT to certain specifications and promised\nthe delivery of XBAT under the larger CDR contract. However, following CDR contract award,\nthe winning CDR bidder determined that XBAT did not meet all of the specifications the FDIC\nhad promised to the CDR vendor.\n\nEVALUATION OBJECTIVE\n\n\nFDIC executives expressed concerns that the XBAT product delivered to the CDR contractor\nwas not fully functional and asked us to identify lessons learned in contract administration and\nproject management that could be applied to the larger CDR effort. Accordingly, our objective\nwas to assess the adequacy of FDIC\xe2\x80\x99s contract administration and project management for the\ndevelopment of XBAT. Appendix I discusses our objective, scope, and methodology.\n\nWHAT WE FOUND\n\n\nThe FDIC received some value from the XBAT procurement. The XBAT contractor was to\ndevelop and deliver two products: (1) a framework of taxonomies for use by software vendors to\ndevelop Call Report software for financial institutions\xe2\x80\x99 use and (2) a software product (XBAT) for\ncreating, editing, and publishing taxonomies. Taxonomies contain common terms, definitions,\nand relationships for Call Report data items. DIR and the CDR contractor concluded the\ntaxonomies were well developed and are being used by the software vendors to develop\nsoftware. DIR considered it imperative to deliver the taxonomies to the software vendors early\nin the process to ensure timely implementation of the Call Mod project.\n\nThe development of the XBAT software was not successful. The FDIC determined that XBAT\nlacked the functionality to successfully manage Call Report taxonomies. As a result, FDIC\nissued a change order to the CDR contract for the development of a Metadata Management\nTool (MMT) that will replace XBAT. According to the CDR contractor, development of the MMT\nshould not impact the overall milestones for implementing the CDR.\n\nWe concluded that the XBAT procurement effort was impaired by a number of contracting and\nproject management issues. Specifically:\n\n \xe2\x80\xa2 FDIC did not establish XBAT functional requirements early enough in the contracting\n process. As a result, neither the FDIC nor the XBAT contractor had a clear and\n consistent understanding of the functionality the XBAT should include.\n\x0c Executive Summary\n\n\n\n\n \xe2\x80\xa2 FDIC did not adequately plan or execute the XBAT procurement effort. To meet cost\n and time constraints, the IDM team waived important XBAT requirements intended to\n support the overall CDR effort. Consequently, the quality of the XBAT software suffered.\n Further, because the procurement did not consistently follow the FDIC Acquisition Policy\n Manual (APM), the FDIC was party to a contract that did not protect the Corporation\xe2\x80\x99s\n best interests.\n\n \xe2\x80\xa2 The contracting officer (CO) and oversight manager (OM) did not completely fulfill their\n responsibilities and relied on the technical monitor (TM) for making changes to the\n functional requirements, accepting contract deliverables, and approving a key invoice\n payment. As a result, the procurement did not have adequate checks and balances to\n help ensure the project\xe2\x80\x99s success.\n\n \xe2\x80\xa2 FDIC did not exercise adequate change management or project integration during the\n XBAT development effort and the CDR solicitation process. The CDR solicitation\n promised a fully functioning XBAT at the same time that the IDM team was waiving\n important functional requirements in order to meet cost constraints and product delivery\n milestones. As a result, FDIC promised a software product containing functionality that it\n could not deliver and had to reprocure the effort.\n\n \xe2\x80\xa2 Communication within and between the IDM team, senior-level managers, and the\n acquisition team (CO, OM, and TM) was not effective. Accordingly, DIR executives and\n CDR bidders were not aware of the true status of the XBAT development effort.\n\n \xe2\x80\xa2 XBAT development did not fully comply with FDIC\xe2\x80\x99s System Development Life Cycle\n (SDLC). Most importantly, FDIC paid the XBAT contractor before completely testing\n XBAT. Thus, FDIC had limited financial leverage to compel contractor performance.\n\nThese conditions occurred because the FDIC did not always follow established acquisition\nprocedures, prudent project management practices, or the SDLC. As a result, the FDIC paid for\nsoftware that did not meet all corporate needs or expectations and had to issue a change order\nto redevelop the software. Moreover, the conditions exposed the FDIC to risks for potential\ndelays in the CDR project and to reputation risk in the eyes of the CDR contractor, other\nbanking agencies, and FDIC-insured institutions (see Appendix II).\n\nThis report identifies a number of lessons learned. Table 1 presents the most important lessons\nto be applied to future system development contracting efforts.\n\nTable 1: Key Lessons to be Learned From the XBAT Development Effort\nContracting Project Management\n\nFunctional requirements should be established early in Closely coordinate projects that are interdependent, and\nthe procurement process. promise only what can be realistically delivered.\n\nChanges to system requirements should not be driven Continually evaluate communication channels to ensure\nsolely by resource and schedule constraints \xe2\x80\x94 they are open and effective and provide balanced and\nconsideration must also be given to impact on business complete information about a project\xe2\x80\x99s status and viability\nneeds. at key development stages.\n\nThe CO, OM, and TM should work together as a team Follow an SDLC methodology that is appropriate for the\nin meeting their assigned responsibilities as part of development effort and perform adequate testing of\nestablishing a control framework. deliverables before making payments.\nSource: OIG analysis .\n\n\n\n 2\n\x0c Background\n\n\n\n\nBACKGROUND\nThe FDIC is a member of the Federal Financial Institutions Examination Council (FFIEC), a\nformal interagency body empowered to prescribe uniform principles, standards, and report\nforms for the federal examination of financial institutions and to make recommendations to\npromote uniformity in the supervision of financial institutions.\xe2\x88\x97 The FFIEC has oversight\nresponsibilities for the collection of Call Reports filed electronically by banks on a quarterly\nbasis. The Call Report shows a bank\xe2\x80\x99s condition and income and is used for multiple purposes\nincluding assessing the financial health and risk of the institution.\n\nAll commercial banks insured by the FDIC and all FDIC-supervised savings banks are required\nto submit quarterly Call Reports. Banks have 30 days following the quarter-end to submit their\ncompleted Call Reports electronically, and banks with foreign offices have 45 days. Banks have\nthe option to use vendor-supplied software to compile and submit their Call Reports.\n\nThe FRB processes data for approximately 1,000 banks using a distributed process across 12\nFederal Reserve District Banks. The FDIC processes data for approximately 7,700 remaining\nbanks using a centralized process at its Headquarters location. Although each agency stores\nthe data separately, ultimately the data are shared. The FDIC and FRB validate Call Report\ndata for mathematical errors and logical relationships, and their staff addresses exceptions by\ncontacting respondents and inputting explanations and corrections.\n\nThe FDIC, FRB, and OCC collaborated to improve the collection and management of financial\ndata starting with the information gathered in the Call Reports. This interagency effort, called\nthe Call Mod project, is intended to provide for collection, storage, and distribution of bank Call\nReport data in a central repository shared by the three agencies. The Call Mod project called\nfor the FDIC to award a multi-million dollar contract to a private-sector contractor to design,\nbuild, and manage the CDR. The primary objectives of the Call Mod project are illustrated in\nTable 2.\n\nTable 2: Call Mod Project Objectives\n Project Objectives\n\n \xc3\xbc Decrease the time between the receipt of Call Report data by the regulators and the release of data to\n agency and public users.\n\n \xc3\xbc Increase the FFIEC Call Agencies\xe2\x80\x99 flexibility to adjust systems quickly to accommodate changing\n information needs.\n\n \xc3\xbc Increase consistency in Call Report data available for use within the FFIEC Call Agencies by implementing\n a central repository with consolidated collection and validation processes.\n\n \xc3\xbc Decrease the overall cost of collecting, validating, integrating, and distributing Call Report data.\n\n \xc3\xbc Improve the transparency of FFIEC Call Agency data requirements by facilitating the exchange of\n information through the use of standards to describe both financial and technical reporting content.\n\n \xc3\xbc Improve the efficiency of value-added calculations by performing these calculations one time and using the\n results whenever needed.\n\nSource: CDR Solicitation Draft Statement of Work.\n\n\xe2\x88\x97\n Other members of the FFIEC include the Board of Governors of the Federal Reserve System (FRB), the Office of the\nComptroller of the Currency (OCC), the Office of Thrift Supervision, and the National Credit Union Administration .\n\n\n\n 3\n\x0c Background\n\n\n\n\nThe CDR was premised on using flexible technologies, one of which is a data exchange\nstandard known as XBRL. XBRL is based on the eXtensible Mark-up Language, a set of\nInternet standards, and is geared to tagging each piece of data with enough information to\nspeed its exchange among users. XBRL taxonomy is a description and classification system for\nthe contents of financial statements and other business reporting documents. Taxonomies\nrepresent up to hundreds of individual business reporting concepts, mathematical relationships,\nand definitions.\n\nThe FDIC established the IDM team to identify and implement strategies to improve the\ncollection, validation, integration, and distribution of financial institution data\xe2\x80\x94the IDM team is\nthe program office for the XBAT and CDR contracts. IDM is organizationally located within DIR.\nAppendix III presents a timeline of key events in the XBAT development process.\n\n\nBenefits of and Challenges Impacting the XBAT Developme nt Effort\n\nThe FDIC received some value from the XBAT engagement. DIR and the CDR vendor\nconcluded the taxonomies were well developed and are being used by the software vendors to\ndevelop software. DIR considered it imperative to deliver these taxonomies to the software\nvendors early in the process to ensure timely implementation of the Call Mod project. The IDM\nteam also noted that it learned a great deal about XBRL technology and functional requirements\nfor a business analyst tool from the XBAT effort and that the CDR contractor was able to build\non this knowledge in developing the MMT.\n\nOur conclusions related to the adequacy of contract administration and project management of\nthe XBAT procurement are presented in the context of recognizing that the IDM team faced a\nnumber of challenges during the time the XBAT services were being procured. XBRL\ntechnology was a new standard that had not been used in the federal government, and the IDM\nteam had to gain acceptance from three agencies \xe2\x80\x93 FDIC, FRB, and OCC \xe2\x80\x93 that the XBAT\nwould work for the overall CDR. Further, several IDM team members told us that the team had\nto contend with the pressure of delivering a product within tight time constraints and limited\nfunding concurrent with satisfying the FDIC Chairman\xe2\x80\x99s Call Mod initiative and the corporate\nperformance objective of the FDIC making high-quality banking data available to the public on a\nmore timely basis.\n\n\n\n\n 4\n\x0c Evaluation Results\n\n\n\n\nFunctional Requirements\n\n The FDIC did not establish XBAT functional requirements early enough in the\n contracting process. As a result, neither the FDIC nor the XBAT contractor had a clear\n and consistent understanding of what functionality the XBAT should include.\n\nWhat Should Have Happened\n\nFDIC\xe2\x80\x99s APM, section 4.A.4.a. specifies that the program office\xe2\x80\x94in this case DIR, or more\nspecifically, the IDM team\xe2\x80\x94is responsible for identifying procurement requirements and\nproviding a clear and specific description of the goods or services required. In doing so, the\nprogram office is responsible for preparing a Statement of Work (SOW) that clearly delineates\nwhat is to be procured. The APM states that, in developing the SOW, a thorough understanding\nof the required goods or services and expected results is critical. Key items to be considered\nand conveyed through the SOW include the nature of the services and minimum standards that\nmust be met and the deliverables and scheduled\nmilestones for service delivery. The SOW for the XBAT project\n stipulated that the FDIC would provide\nFDIC Circular 1320.3, System Development Life Cycle, the contractor a draft FRD for the\n business analyst tool. The SOW also\n(SDLC) Version 3, outlines provisions that apply to all stated that, in all cases, the principles,\ninformation technology project efforts, including new phases, and deliverables of the FDIC\nsystem development efforts as well as enhancing and SDLC as described in Circular 1320.3\nconverting existing systems. One of the eight phases of must be applied. The tasks outlined in\n the XBAT SOW included a statement\nthe SDLC is the Requirements Definition Phase, during that, during the SDLC Requirements\nwhich user performance needs are analyzed and user Definition Phase, the FDIC will analyze\nrequirements are formally defined. Requirements for and define system requirements, but the\nfunctionality, data, system performance, security and contractor may be requested to assist in\ninternal control, portability, and maintainability are defining these requirements. The SOW\n also included a statement that the FDIC\ndefined to a level of detail sufficient for system design to will work with the contractor to gather\nproceed. Requirements are formally documented in the requirements to develop an FRD to be\nfunctional requirements document (FRD) and approved used by the contractor to design,\nby the client and developer. develop, and test the system.\n\n\n\nWhat Actually Happened\n\nXBAT functional requirements were not established early enough in the contracting process and\nevolved during the XBAT engagement. The IDM team and the XBAT contractor drafted the first\nFRD in late September 2002, followed by two additional draft FRDs dated October 2, 2002 and\nOctober 16, 2002, respectively. A subsequent FRD was drafted in June 2003, 10 months after\nthe effective date of the contract. The October 2002 and June 2003 FRDs reduced the\nrequirements specified in the September 2002 FRD. Because the functional requirements were\nevolving, neither the FDIC nor the XBAT contractor had a clear understanding of what\nfunctionality the XBAT should include.\n\nXBAT contractor representatives indicated that the FDIC provided a draft FRD in August 2002,\nbut noted the FRD was not detailed enough and that some requirements were unrealistic and\nreflected a lack of understanding about system capabilities and requirements. The XBAT\ncontractor spent about 2 months adding detail to the FRD. The contractor estimated there were\n\n\n\n 5\n\x0c Evaluation Results\n\n\n\n\n5 to 10 drafts of the FRD, the last versions being drafted in October 2002. The contractor was\nnot aware of the June 2003 FRD. Table 3 summarizes the various FRDs and their impact on\nthe XBAT and CDR projects.\n\nTable 3: XBAT Functional Requirements\n Functional Description Impact on XBAT and/or CDR\n Requirements\n\n September 2002 Includes 5 categories of 57 requirements: Provided to CDR bidders on\n FRD functional, hardware, software, September 11, 2002.\n (Draft) performance, and security. Software\n requirements are described as \xe2\x80\x9cTo Be\n Determined.\xe2\x80\x9d Also includes SDLC\n reference and deliverables.\n\n October 2, 2002 Includes the categories of requirements Provided to CDR bidders in the\n FRD mentioned in the September FRD, except Request for Proposal, Amendment 09,\n (Draft) for hardware requirements being with an effective date of October 3,\n described as \xe2\x80\x9cTo Be Determined.\xe2\x80\x9d 2002.\n Software requirements are described.\n Excludes SDLC reference and\n deliverables, except for a Table of\n Contents reference.\n\n October 16, 2002 Includes the same categories of XBAT Task Order 01, signed\n FRD requirements as the October 2, 2002 FRD. December 16, 2002, and effective\n (Draft) Hardware requirements are described as August 9, 2002, included a provision for\n \xe2\x80\x9cTo Be Determined.\xe2\x80\x9d Excludes SDLC the contractor to develop XBAT to meet\n reference and deliverables. Includes two requirements described in the FRD with\n new functional requirements and one new the final date of October 16, 2002.\n performance requirement.\n However, XBAT Task Order 01 also\n waived 18 requirements listed in the\n October 16, 2002 FRD.\n\n\n June 2003 FRD Includes 5 categories of requirements A fully functioning XBAT was not\n (Draft) identified in the September 2002 FRD, but delivered to the CDR contractor. The\n excludes 17 of the remaining specific FDIC issued a $840,000 change order\n requirements listed in the September 2002 to the CDR contractor for an MMT to\n FRD. Some of the exclusions are in the replace XBAT.\n areas of application testing, system\n outputs, and reliability requirements. The\n June 2003 FRD included three software\n requirements and one system sizing and\n growth requirement that were not included\n in the September 2002 FRD. The June\n 2003 FRD also included SDLC\n documentation requirements.\nSource: XBAT FRDs, Task Order 01, and OIG Analysis.\n\nThe CDR contractor assessed the XBAT software delivered by the FDIC and determined that it\ndid not meet the September 2002 FRD promised in the CDR solicitation. The CDR contractor\nconcluded that the XBAT contractor had made significant progress in proving basic XBRL\nconcepts and had also succeeded in capturing the underlying principles involved in creating\nXBRL taxonomies. However, the CDR contractor noted that its bid on the CDR contract was\nbased on all XBAT FRD requirements being met, and the contractor expected to receive design\ndocuments and code as part of the XBAT that could be used in building CDR components.\n\n\n\n\n 6\n\x0c Evaluation Results\n\n\n\n\nThe FDIC agreed with the CDR contractor\xe2\x80\x99s\n The CDR contractor\xe2\x80\x99s assessment of three\nassessment of XBAT and awarded a separate major requirement categories noted the\ncontract to the CDR contractor to replace the following issues relative to the XBAT:\nXBAT with a fully functioning MMT. The MMT\nis a new tool for creating, editing, and publishing \xe2\x80\xa2 The XBAT is able to produce\n taxonomies. However, there are some\ntaxonomies.\n omissions from the requirements in the\n original FRD, such as validation,\n import/export, and print capabilities.\nConsequence\n \xe2\x80\xa2 The XBAT provides management of\n taxonomy source data. The remaining\nBecause functional requirements were evolving meta-data systems management\nduring the XBAT development, FDIC and the requirements are CDR requirements.\nXBAT contractor did not have a clear\nunderstanding of what functionality the XBAT \xe2\x80\xa2 Virtually none of the meta-data testing\nshould include. has been implemented.\n\n\n\nLessons Learned\n\nBased on our discussions with DIR, the Division of Information and Resources Management,\nand Acquisition Services Branch (ASB) officials involved with the XBAT and CDR projects, we\noffer the following lessons learned:\n\n \xe2\x80\xa2 Functional requirements should be established early in the procurement process.\n\n \xe2\x80\xa2 When identifying procurement requirements, provide a clear and specific description of\n the goods or services required.\n\n\n\n\n 7\n\x0c Evaluation Results\n\n\n\n\nProcurement Planning and Execution\n\n The FDIC did not adequately plan or execute the XBAT procurement effort. The IDM\n team waived important XBAT functions that were intended to support the overall CDR\n effort to meet cost and time constraints. Consequently, the quality of the XBAT\n software suffered. Further, because the procurement did not consistently follow the\n APM, the FDIC was party to a contract that did not protect the Corporation\xe2\x80\x99s best\n interests.\n\n\nWhat Should Have Happened\n\nThe APM provisions for preparing a procurement requirements package include providing a\nclear and specific description of the goods or services required and a price estimate for the\ngoods or services being requested, including the base period and all option periods. Preparing\na cost estimate requires an identification of the cost components that apply to the proposed\ncontract. Identifying these components requires a clear understanding of the goods or services\nbeing procured.\n\nThe APM provides for two basic types of pricing arrangements used in FDIC contracts, fixed\nprice and level of effort. The APM suggests that the successful use of a firm fixed price\narrangement requires a clear definition of requirements in the SOW and realistic estimates of\nwork to be performed. A level of effort contract, such as a time and materials contract, is used\nwhen it is difficult to provide a detailed SOW or to estimate the price or duration of time required\nfor performance.\n\nThe APM\xe2\x80\x99s provisions for awarding a contract based on competitive range determination state\nthat, following the initial evaluation of bids, if there is no one successful offeror and if there is a\nneed to hold technical and/or cost discussions with offerors, the CO may establish a competitive\nrange with those firms that have a reasonable chance of being selected for award. After\ndiscussions are completed, offerors are given the opportunity to improve their proposals through\na best and final offer phase. The CO is responsible for ensuring that FDIC personnel do not:\n(1) help an offeror bring its proposal up to the level of other proposals through successive\ndiscussion opportunities; (2) disclose technical information pertaining to one proposal that\nresults in improvement of a competing proposal; (3) indicate to an offeror a price that it must\nmeet to obtain further consideration; or (4) furnish information about other offerors\xe2\x80\x99 proposed\nprices.\n\nThe APM contains various provisions regarding basic ordering agreements (BOAs) and task\norders, SDLC contracts and task assignments, contractual documents, official contract files, and\nlegal review of contracting documents.\n\nBOAs and Task Orders: The APM identifies a BOA as a written agreement between the FDIC\nand a contractor containing terms and conditions that will apply to FDIC-issued task orders\nduring its term, a description of the services to be provided, and the method(s) for the pricing\nand issuing of orders under the agreement. A BOA is not a contract because it does not require\nthat orders be placed and does not contain consideration. Any task order issued against a\nBOA, in accordance with the terms and conditions contained in that BOA, becomes a\ncontractual instrument against which funds are obligated as consideration in exchange for the\ngoods or services specified in the task order.\n\n\n 8\n\x0c Evaluation Results\n\n\n\n\nThe APM procedures for awarding task orders include a requirement that the program office\nprovide the CO with written requests for all task orders under a BOA. The request should be in\nthe form of a memorandum or letter accompanied by a written statement of work specific to the\nassignment being requested. As shown in Table 4, The APM includes the following minimum\nrequirements for requesting task orders and what a task order should include.\n\nTable 4: APM Task Order Requirements.\n\nR\nReeqquueesstt ffoorr TTaasskk O\n Orrddeerr R\n Reeqquuiirreem\n meennttss TTaasskk O\n Orrddeerr R\n Reeqquuiirreem\n meennttss\n\nInclude the nature of services and minimum Incorporate terms and conditions of the BOA.\nstandards that must be met, qualifications necessary\nto perform the work, deliverables and scheduled\nmilestones for their delivery, and standards by which\nthe contractor\xe2\x80\x99s performance will be measured.\n\nDetailed cost estimate. Contain or incorporate a statement of work for the specific task\n to be performed.\n\nTechnical evaluation criteria, if award is to be based Specify milestones with a schedule of deliverables.\non other than price.\n\nWeighting of technical and price proposals, if award State a period of performance.\nis to be based on other than price.\n\nProcurement Requisition. Pricing information, including ceiling prices for labor and travel.\nSource: APM.\n\nSDLC Contracts and Task Assignments: DIRM generally uses contracts with task\nassignments for SDLC activities. These contracts allow the OM to issue individual task\nassignments under the terms of the contract. In this way, the OM can determine the timing and\nscope of deliverables to be provided under a task assignment without having the CO issue the\ntask assignment. The contract and task assignment SOW should refer to the SDLC.\n\nThe program office is responsible for developing the task assignments, ensuring that (1) the\ntotal value of the various task assignments issued pursuant to a contract does not exceed the\noriginal expenditure amount of the contract and (2) that the work detailed in the task\nassignments does not go beyond the scope of work set forth in the contract. The OM signs the\ntask assignments, and a copy is sent to the CO who has responsibility to retain the task\nassignment in the official contract file.\n\nContractual Documents and Official Contract Files: The APM states that the CO shall\nprepare and assemble the appropriate contractual documents. The CO is responsible for\nsending two originals of the contract to the contractor for signature, executing both copies of the\ncontract on behalf of the FDIC, returning one of the originals to the contractor, and retaining the\nother original contract in the official contract file. The CO is also responsible for providing a\ncopy of the contract to the designated OM.\n\nThe APM states that it is good business practice to have fully executed documents in place\nbefore a contractor commences work. However, on an as-needed basis, and only with the prior\napproval of the Assistant Director, Headquarters Acquisition Section, or head of the regional\ncontracting function, the effective date of a contract may be prior to the execution date. This\nprovision should be utilized when the CO intends to orally authorize a contractor to begin work\nprior to having in place a fully executed contractual document.\n\n\n\n 9\n\x0c Evaluation Results\n\n\n\n\nThe APM states that the ASB is responsible for maintaining the official contract file associated\nwith all phases of the contracting process. Each CO is responsible for establishing a contract\nfile for each new contract and ensuring that all documents are filed prior to contract award\naccording to the FDIC Formal Contracting File Checklist, attached as Exhibit XXIII to the APM.\nDocumentation shall include that which is required under the APM relating to pre-solicitation,\nsolicitation, evaluation, selection, pre-award reviews, and the award decision.\n\nLegal Review of Contracting Documents: The APM identifies the Contracting Law Unit within\nthe FDIC\xe2\x80\x99s Legal Division as a member of the team supporting the FDIC\xe2\x80\x99s contracting process.\nThe Unit supports the development of contracting policy and procedures and provides advice\nand legal sufficiency reviews. The APM stipulates procurement responsibilities for the Legal\nDivision, including requirements to (1) review solicitation packages for contracts of $100,000 or\nmore; (2) review complex contracting requirements, as requested by the CO; (3) provide advice\nas required on issues involving contract scope; and (4) provide other assistance as requested\nby the CO. The APM does not specifically require that the Legal Division review contract\ndocuments unless requested by the CO.\n\n\nWhat Actually Happened\n\nASB and the IDM team did not adequately plan the XBAT contract, and the IDM team\nunderestimated the costs to develop XBAT. Although XBAT was a research and development\neffort employing new technology, ASB and the IDM team used a firm fixed price contract with\nshort time frames and evolving requirements. Firm fixed price contracts are generally\nappropriate for low-risk engagements with well-defined requirements. Given the urgent need to\ndevelop the XBAT in time for delivery to the CDR vendor and the evolving functional\nrequirements, the XBAT procurement would most likely not be categorized as a low-risk\nengagement suitable for firm fixed pricing. Appendix IV presents how XBAT procurement\nrequirements changed over the course of the contract.\n\nAs shown in Figure 1, the costs for developing an XBRL business analyst tool increased from\nan early estimate of almost $500,000 to a cost of more than $1.5 million (considering XBAT and\nMMT development costs). We observed that XBAT development costs increased by more than\n80 percent from the BOA to Task Order 01 at the same time that the IDM team was waiving\nimportant functional requirements.\n\n\n\n\n 10\n\x0c Evaluation Results\n\n\n\n\n Figure 1: XBRL Business Analyst Tool Estimated and\n Actual Development Costs\n\n $2,000,000\n $840,000\n Development Cost\n $1,500,000\n\n\n\n $1,000,000 $703,330\n $498,400\n $370,000\n $500,000 $703,330\n\n\n\n $0\n Procurement BOA Task Order 01 XBAT & MMT\n Requisition\n XBAT MMT\n\n Source: XBAT Contracting Files .\n\nXBAT contractor representatives informed us that they provided labor rates and labor hour\nestimates in response to the initial XBAT request for proposal. However, the XBAT contractor\ntold us that during the best and final offer period, FDIC reduced the number of option years and\nimposed a price ceiling based on FDIC budget authority. In response, the XBAT contractor\npresented a best and final offer bid of $370,000 to develop XBAT\xe2\x80\x94almost 26 percent lower\nthan the FDIC\xe2\x80\x99s estimate. FDIC officials we interviewed did not recall providing a price ceiling or\nFDIC budget authority information to the bidders during the best and final offer period.\n\nShortly after the contract had been awarded, the TM questioned the adequacy of contract\nfunding for developing XBAT. Further, just 2 months following the start of work, the XBAT\ncontractor indicated that the option year on the contract would need to be executed because the\nbase year funding had been expended. The XBAT\ncontractor also indicated at that time that work In October 2003, the FDIC awarded a change\ncould be completed by the end of December 2002 order to the CDR contract that specifically\n identifies a breakdown of the firm fixed price\nfor total funding of $1.1 to 1.3 million, nearly of the contract for each major deliverable. The\ndouble the firm fixed price negotiated for the XBAT contract specifies four project deliverables,\ncontract, unless requirements were reduced. The including a master project plan and a design\nXBAT contractor acknowledged that, with the document, and identifies related payments and\n completion dates for each deliverable. In\nbenefit of hindsight, its best and final offer bid was addition, the contract includes an acceptance\nnot sufficient to complete the XBAT contract clause that requires the contractor to obtain an\nrequirements. acceptance signature by FDIC of formal\n deliverables. The acceptance procedures\n state that the FDIC will accept the deliverables\nMoreover, the milestones established for\n as complying with the SOW or provide a\ndevelopment of the taxonomies and XBAT appear written statement that identifies in reasonable\nunrealistic. The TM said that, in hindsight, the detail significant deviations between the\n4-month timeframe to develop and deliver the deliverable and the statement of work. FDIC\nXBAT was unrealistic, especially for new did not include such an acceptance clause in\n the XBAT task order.\ntechnology. The TM further stated that the goal of\ndeveloping a fully functioning XBAT by December 2002 to accommodate the CDR was also\nunrealistic. The IDM project manager informed us that the FDIC awarded bonus points to the\n\n\n\n 11\n\x0c Evaluation Results\n\n\n\n\nXBAT contractor during the best and final offer for early delivery of the taxonomies and\nsoftware. An unsigned July 17, 2002 technical evaluation results memorandum providing\nconclusions regarding the technical merits of bidders\xe2\x80\x99 oral presentations stated that the XBAT\ncontractor provided a reasonable time schedule for performing the required tasks.\n\nThe XBAT contract award and execution also did not fully comply with APM provisions. ASB\nadvertised the XBAT solicitation as a task-order-based contract, but ASB executed a BOA and\nissued task assignments that are typically used with an SDLC contract. The CO issued an\nadvanced authorization letter on August 9, 2002, authorizing the XBAT contractor to proceed\nwith work while contracting documents were being finalized. However, ASB and the contractor\ndid not execute a task order for the BOA until December 16, 2002, 4 days before the XBAT\nproducts were scheduled to be completed for delivery to the FDIC. We saw no evidence that\nthe ASB Assistant Director approved the effective date of the contract, as required by the APM.\nFurther, as illustrated in Table 5, we noted several other contract award and execution activities\nthat were departures from APM provisions.\n\nTable 5: XBAT Contract Award and Execution Departures From APM Provisions\nDescription of Action in Question\n\nThe CO did not provide a copy of the signed contract for the XBAT to the designated OM.\n\nThe task order contained a nearly 2-year period of performance (August 9, 2002 through June 30, 2004) that\ndiffered from the BOA\xe2\x80\x99s 1-year period of performance, starting August 9, 2002 and ending August 9, 2003.\n\nThe firm fixed price of the task order exceeded the BOA compensation ceiling by $30,000.\n\nKey documents associated with the XBAT contract award and execution were missing from the official contract file,\nincluding the unsuccessful bidders\xe2\x80\x99 proposals, a signed selection recommendation report, a SOW for the signed\nBOA, and a SOW for the task assignments and task order.\n\nChanges and waivers of functional requirements were made without the CO\xe2\x80\x99s approval. The OM issued two task\nassignments \xe2\x80\x93 one that waived a number of contracting requirements without authorization from the CO.\nSource: OIG Analysis.\n\nIn regard to the Legal Division\xe2\x80\x99s involvement in XBAT procurement activities, we interviewed the\nCounsel assigned to the XBAT procurement effort. The Counsel indicated that his role was to\nreview the contract and associated task orders for legal sufficiency. The Counsel reviewed the\nsolicitation package, documents associated with the August 2002 advanced authorization letter,\nand the December 2002 Task Order 01. The Counsel noted that ASB and the IDM team were\nin a hurry and that there was a rush to issue the contract documents. We made the following\nobservations about the Counsel\xe2\x80\x99s review of XBAT procurement documents:\n\n\xe2\x80\xa2 The Counsel reviewed the solicitation documents as required by the APM. However, ASB\n issued the XBAT Request for Proposal 5 days prior to receiving the Counsel\xe2\x80\x99s notice of\n review and acceptance.\n\n\xe2\x80\xa2 The CO indicated that he would not honor the TM\xe2\x80\x99s request for an advanced authorization\n allowing the contractor to commence work prior to awarding a contract without the Legal\n Division\xe2\x80\x99s review of the Intellectual Property (licensing) terms of the XBAT software. The\n Counsel provided contract language to the CO regarding XBAT licensing provisions prior to\n the issuance of the advanced authorization letter.\n\n\n\n\n 12\n\x0c Evaluation Results\n\n\n\n\n\xe2\x80\xa2 We saw no evidence in the XBAT contract files of the Legal Division\xe2\x80\x99s review of the BOA,\n although the APM does not specifically require a legal review of the contract unless such a\n review is requested by the CO.\n\n\xe2\x80\xa2 The Counsel informed us that he did not review the XBAT task assignments.\n\n\xe2\x80\xa2 In regard to the task order, the XBAT contracting and project management files indicated\n that the TM developed the task order and sent it electronically to the CO, OM, and IDM team\n for review on October 23, 2002. The XBAT contract files indicated that the CO did not\n request the Legal Division to review the revised XBAT task order until December 10, 2002,\n 6 days prior to the execution of the task order and 10 days prior to the scheduled date of\n XBAT delivery.\n\n\nConsequence s\n\nBecause the XBAT procurement was not adequately planned or executed, the quality of the\nXBAT software suffered. Further, because the procurement did not consistently follow the APM,\nthe FDIC was party to a contract that did not protect the Corporation\xe2\x80\x99s best interests.\n\n\nLessons Learned\n\n \xe2\x80\xa2 Changes to system requirements should not be driven solely by resource and schedule\n constraints\xe2\x80\x94consideration must also be given to impact on business needs.\n\n \xe2\x80\xa2 Research and development efforts do not lend themselves to a firm fixed price\n contracting vehicle, given the uncertain and evolving nature of the services. The type of\n contract awarded should be based on the level of risk to the Corporation, with fixed price\n contracts generally used when risk has been reduced to a reasonable level.\n\n \xe2\x80\xa2 When procuring services for research and development, communicate to everyone\n involved that the project is a research and development effort, requiring closer\n monitoring.\n\n \xe2\x80\xa2 A time and materials level of effort contracting vehicle may be more suitable for a\n research and development engagement, given the difficulty in providing a detailed SOW\n or to estimate the price or time required for a research and development effort.\n\n \xe2\x80\xa2 Involve the Legal Division early in procurement planning, and subject key contracting\n documents such as the contract and SOW to legal review and concurrence prior to\n contract execution. Focus the legal review on both legal sufficiency and protecting the\n Corporation\xe2\x80\x99s interests, particularly for contract provisions that pose the greatest risk for\n the Corporation.\n\n\n\n\n 13\n\x0c Evaluation Results\n\n\n\n\nContract Administration\n\n The CO and OM did not completely fulfill their responsibilities and relied on the TM for\n making changes to the functional requirements, accepting contract deliverables, and\n approving a key invoice payment. As a result, the procurement did not have adequate\n checks and balances to help ensure the project\xe2\x80\x99s success.\n\nWhat Should Have Happened\n\nThe APM stipulates that contract administration encompasses oversight of all relationships\nbetween the FDIC and the contractor relating to contractor performance and provides the\nfollowing responsibilities for the CO, OM, and TM.\n\nContracting Officer: Responsibility for ensuring The BOA executed for the XBAT engagement\ncompliance with the contract rests with the CO, also included the following provisions\nwho may delegate certain authorities to other regarding the OM and CO.\nFDIC personnel. The CO\xe2\x80\x99s responsibilities\ninclude (1) monitoring contract performance for \xe2\x80\xa2 FDIC Contracting Officer. The term\n \xe2\x80\x9cContracting Officer\xe2\x80\x9d means a person\ncompliance with its terms and conditions; (2) designated in writing by the FDIC with\nenforcing contract provisions; (3) executing FDIC delegated authority to enter into,\nmodifications to the contract; (4) assisting the OM modify, administer, and terminate\nor other program office representatives, as contracts and orders. The Contracting\n Officer is the only person authorized\nrequested, in oversight of the technical business\n by FDIC to issue any instructions or\nrequirements of the contract; and (5) jointly with directions that affect any increase or\nthe OM, verifying costs incurred and invoiced to decrease in the cost of this agreement,\nthe FDIC under the contract and monitoring any change in delivery schedule or\nexpenditures against the expenditure ceiling. which change any other term of the\n agreement.\nOversight Manager: The OM is designated by \xe2\x80\xa2 FDIC Oversight Manager. The term\nthe program office and has responsibility for \xe2\x80\x9cOversight Manager\xe2\x80\x9d means the\n(1) overseeing the performance requirements of person designated in writing by the\nthe contract; (2) providing business and technical Contracting Officer to represent the\n FDIC for the purpose of monitoring\nliaison between the FDIC and the contractor; technical performance under any task\n(3) notifying the CO of any need to modify the order awarded. The Oversight\ncontract; and (4) referring all questions regarding Manager is not authorized to issue any\ncontract provisions to the CO. The OM is solely instructions or directions which effect\n any increase or decrease in the price\nresponsible for carrying out all duties and of any task order awarded or which\nresponsibilities set forth in the Letter of Oversight change the delivery date(s) or Period\nManager Confirmation issued by the CO, with of Performance.\ncopies provided to the contractor and the OM.\n\nTechnical Monitor: It may be appropriate in very complex areas of performance to appoint one\nor more TMs for a contract. The duties of a TM are a subset of the duties of the OM, but the\nresponsibility for contractor oversight remains with the OM. TMs are designated by the OM and\nappointed in a Letter of Technical Monitor Confirmation issued by the CO, with copies sent to\nthe contractor and the OM.\n\nThe APM also requires that OMs and TMs be adequately trained to oversee contractor\nperformance within the terms of the contract. The APM requires that OMs and TM attend the\n\n\n\n 14\n\x0c Evaluation Results\n\n\n\n\nFDIC Oversight Management Training course and the FDIC Contract Administration Training\ncourse and pass an examination for each course. Further, OMs and TMs must take a 1-day\nrefresher course every 3 years.\n\nThe APM also requires a Contract Administration Plan (CAP) for contracts over $100,000. The\nobjective of a CAP is to ensure that the CO and OM have a common understanding of both the\ncontractor\xe2\x80\x99s and the FDIC\xe2\x80\x99s obligations under the contract. The CO prepares the CAP with the\nassistance of the OM immediately following contract award.\n\n\nWhat Actually Happened\n\nContract administration was not adequate. The CO and the OM relied too heavily on the TM for\napproving deliverables, changing requirements, and approving invoice payments. Two DIRM\nOMs noted that the XBAT TM was very involved with the project and that both relied on the\nexpertise and \xe2\x80\x9chands-on\xe2\x80\x9d involvement of the TM for acceptance of some of the deliverables.\n\nThe TM sent an electronic message to the OM in early August 2002, requesting agreement on\ncontract management authority and responsibilities. The electronic message was not copied to\nthe CO. The TM acknowledged in the message that oversight management of the contract had\nbeen assigned to DIRM. However, the TM assumed responsibility for:\n\n (1) technical management of the contract,\n (2) the success of the project,\n (3) developing and providing task assignments to the OM for review,\n (4) managing day-to-day contract operations and informing the OM of contract status, and\n (5) directing the contractor\xe2\x80\x99s work.\n\nThe first OM served in his position from August 2002 until late October 2002 at which time the\nsecond OM was appointed. The change in OMs was the result of corporate reorganization\nactivities. The CO did not issue an APM-required Letter of Oversight Manager Confirmation to\nthe first OM. In addition, the CO did not issue an APM-required Letter of Technical Monitor\nConfirmation to the TM. These letters were issued for the current OM and TM.\n\nMoreover, the FDIC\xe2\x80\x99s Training Management Server showed no record of the required OM and\ncontract administration training for the OMs or the TMs appointed for the XBAT contract. We\nsubsequently learned that the current OM completed OM training and took an online refresher\ncourse.\n\nFinally, the official contract files for the XBAT did not contain a CAP. The OM and TM\nconfirmation letters and a CAP would have been beneficial for the CO and the OM to\nunderstand their roles in overseeing the contract, to ensure awareness of specific deliverables\nand milestones, and to monitor changes to requirements, timeframes, and funding.\n\n\nConsequence\n\nBecause the CO and OM did not completely fulfill their responsibilities and relied too heavily on\nthe TM for managing the contract, the procurement did not have adequate checks and balances\nto help ensure the project\xe2\x80\x99s success.\n\n\n\n 15\n\x0c Evaluation Results\n\n\n\n\nLessons Learned\n\n \xe2\x80\xa2 The CO, OM, and TM should work together as a team in meeting their assigned\n responsibilities as part of establishing a control framework.\n\n \xe2\x80\xa2 The OM should involve the CO in addressing contractor performance matters.\n\n \xe2\x80\xa2 OM and TM roles and responsibilities need to be clearly documented and communicated\n in the Letter of Oversight Manager (or Technical Monitor) Confirmation.\n\n \xe2\x80\xa2 A CAP is a good tool for helping ensure that the contractor performs in accordance with\n the terms and conditions of the contract and is required by the APM.\n\n \xe2\x80\xa2 A CAP allows both the OM and CO to identify specific deliverables and due dates and\n track future modifications and funding.\n\n\n\n\n 16\n\x0c Evaluation Results\n\n\n\n\nChange Management and Project Integration\n\n The FDIC did not exercise adequate change management or project integration during\n the XBAT development effort and the CDR solicitation process. The CDR solicitation\n promised a fully functioning XBAT at the same time that the IDM team was waiving\n important functional requirements to meet cost constraints and product delivery\n milestones. As a result, the FDIC promised a software product containing\n functionality that it could not deliver and had to reprocure the effort.\n\n\nWhat Should Have Happened\n\nThe PMBOK\xc2\xae Guide (discussed below) and the APM include the following information relevant\nto change management and project integration.\n\nChange Management: The Project Management Institute (PMI) has conducted extensive\nresearch and analysis in the field of project management and published a standards guide in\n2000, entitled A Guide to the Project Management Body of Knowledge (PMBOK\xc2\xae Guide). The\nguide documents proven practices, tools, and techniques that have become generally accepted\nin the field of project management, including information systems development and\nimplementation. The PMBOK\xc2\xae Guide is an approved standard of both the American National\nStandards Institute and the Institute of Electrical and Electronics Engineers. The guide\nidentifies nine distinct knowledge areas associated with successful project management. The\nnine areas are integration, scope, time, cost, quality, human resources, communications, risk,\nand procurement management.\n\nPMBOK\xc2\xae identifies project scope management as a subset of project management that\nincludes the processes required to ensure that the project includes all the work required, and\nonly the work required, to complete the project successfully. One of the processes involves\ncontrolling changes to project scope. Scope change control is concerned with influencing the\nfactors that create scope changes to ensure that changes are agreed upon, determining that a\nscope change has occurred, and managing the actual changes when and if they occur. Scope\nchange control must be integrated with the other control processes, such as scheduling, cost,\nquality, risk, and staffing.\n\nThe APM contains a chapter of provisions regarding changes and modifications to contracts and\ncategorizes the types of changes as administrative or substantive. Substantive changes include\na change in the amount of fees to be paid to the contractor, a change in the delivery schedule,\nand a change in the quantity and nature of deliverables. The CO is responsible for determining\nwhether a proposed modification is within the scope of the contract. The OM is responsible for\nidentifying the requirement for a modification, determining whether the cost to the FDIC caused\nby the modification will exceed the expenditure ceiling, and preparing a detailed, written\nexplanation of the reason for and nature of the change or modification.\n\nProject Integration: According to PMBOK\xc2\xae, the work of the project must be integrated with\nthe ongoing operations of the performing organization. Project integration management\nincludes the processes required to ensure that the various elements of the project are properly\ncoordinated. It involves making tradeoffs among competing objectives and alternatives to meet\nor exceed stakeholder needs and expectations and includes project plan development, project\nplan execution, and integrated change control.\n\n\n\n 17\n\x0c Evaluation Results\n\n\n\n\nIntegrated change control relates to coordinating changes across the entire project and is\nconcerned with: influencing the factors that create changes to ensure that changes are agreed\nupon, determining that a change has occurred and managing the actual changes when and as\nthey occur. One of the tools and techniques for integrated change control involves additional\nplanning to allow for prospective changes that may require new or revised cost estimates,\nmodified activity sequences, schedules, resource requirements, or other adjustments to the\nproject plan. These changes must be communicated to appropriate stakeholders, as needed.\nThe CDR request for proposal included a copy of the FDIC\xe2\x80\x99s General Provisions for contracts\ndated May 2002. The provisions contained the following language in a section addressing\nFDIC-furnished property:\n\n The delivery or performance dates for this Contract are based upon the expectation that\n FDIC-furnished property suitable for use (except for property furnished as-is) will be\n delivered to Contractor at the times stated in the Statement of Work, or, if not so stated,\n in sufficient time to enable Contractor to meet the Contract\xe2\x80\x99s delivery or performance\n dates.\n\n\nWhat Actually Happened\n\nChange Management: As discussed earlier, the FDIC issued Task Order 01 which waived\ncertain functional requirements to meet the December 2002 deadline for delivery of the XBAT to\nthe CDR winning bidder. Waived requirements included SDLC deliverables and the process to\ntrace requirements and test all aspects of the system. However, the task order did not explicitly\nidentify the SDLC and system testing waivers. The CO needed this information to determine\nwhether the requested changes were within the scope of the original contract, as required by\nthe APM for contract modifications.\n\nFurther, the OM and TM did not cross-reference the requirements waived in the task order to\nthe individual requirements in the original FRD to clearly indicate those XBAT specifications that\nwere no longer required. In October 2003, 10 months after the task order was signed, the IDM\nproject manager prepared a schedule comparing In October 2003, the FDIC awarded a change\nthe waived requirements to the October 2002 FRD. order to the CDR contract to develop an MMT\nNevertheless, the waived requirements should to replace XBAT. The MMT change order\nhave been referenced at the time the task order contained change control procedures as part\n of the contract requirements. The contract\nwas being negotiated especially because the FDIC\n requirements state that, in the event that the\nhad promised that the XBAT would be delivered to FDIC or the contractor wishes to alter the\nthe CDR winning bidder in accordance with the statement of work, various procedures must\nSeptember and October 2002 FRD specifications. be followed, including (1) documenting the\n description and reason for the proposed\n change; (2) coordinating change requests\nIn September 2003, DIR appointed a new OM to with the OM; and (3) investigating the impact\noversee the remaining work on the XBAT contract. of the change request on the price, timetable,\nThe OM prepared a list of requirements from the statement of work, and specifications and\ntask order that had not been completed and relevant obligations under the agreement.\n These change control procedures would have\ndiscussed the requirements with the XBAT\n benefited the XBAT engagement.\ncontractor. The XBAT contractor told the OM that\nthe contractor believed that the FDIC agreed to waive these requirements, but the waivers were\ndocumented only in status reports and electronic messages rather than being handled through\nthe formal contracting modification process. The OM estimated the cost of the unfinished work\nto be $40,000.\n\n\n 18\n\x0c Evaluation Results\n\n\n\n\nProject Integration: XBAT, an important component of the overall CDR project, was being\ndeveloped concurrent with the CDR solicitation process. The FDIC promised to deliver the\ntaxonomies and software, meeting the September and October 2002 functional requirements, to\nthe winning bidder for the CDR contract at the same time that the IDM team was waiving XBAT\nfunctional requirements. However, the IDM team did not assess the impact that the waivers and\nchanges to the functional requirements might have on the CDR project.\n\nAs late as October 2, 2002, FDIC\xe2\x80\x99s responses to questions posed by the CDR bidders indicated\nthat XBAT would be delivered by December 31, 2002. In early November 2002, the XBAT\ncontractor informed the TM and CO that more funding would be needed to complete all\nrequirements of the original XBAT contract, but the \xe2\x80\x9ccore functionality\xe2\x80\x9d could be delivered for the\noriginal contract amount. The changes to the FRD requirements were not factored into the CDR\nsolicitation activities occurring during the period August 2002 through January 2003. Figure 2,\non page 21, illustrates the timing of activities that parallel the XBAT development and the CDR\nsolicitation efforts.\n\nThe XBAT was a relatively minor procurement effort but was also a critical path item to the\nmuch larger CDR effort. In September 2002, FDIC established the Capital Investment Review\nCommittee (CIRC) to implement a systematic management review process for FDIC capital\ninvestments, defined as initiatives with total capital outlays in excess of $3 million. The CIRC\ncurrently monitors 14 initiatives, 1 of which is the CDR project. The XBAT is not a CIRC\ninitiative because its development and maintenance costs are well below the $3 million\nthreshold. However, the FDIC\xe2\x80\x99s capital investment management review process should include\nCIRC initiatives as well as related or interdependent contracts and initiatives.\n\nFurther, the IDM team did not respond proactively to indications that XBAT development was\nexperiencing problems and did not amend the CDR solicitation or notify CDR bidders that they\nwould be receiving the XBAT software \xe2\x80\x9cas is\xe2\x80\x9d as opposed to receiving a fully functioning\nsoftware product. The IDM project manager told us that he empowered the TM and placed too\nmuch reliance on his staff to ensure that the XBAT was progressing as planned and scheduled.\nIn particular, the project manager relied too heavily on the TM to do the right thing and did not\nestablish control points to independently verify that XBAT was on schedule, within cost\nestimates, and meeting performance objectives. The IDM project manager indicated that in\nhindsight, the FDIC should not have promised to deliver to CDR bidders an XBAT software\nmeeting the September and October 2002 functional requirements. Instead, the FDIC should\nhave offered XBAT \xe2\x80\x9cas is\xe2\x80\x9d without specific representations and warranties.\n\n\nConsequence\n\nThe FDIC promised a software product containing functionality that it could not deliver and had\nto issue an $840,000 change order to the CDR contractor to replace XBAT.\n\n\nLessons Learned\n\n \xe2\x80\xa2 Closely coordinate projects that are interdependent and promise only what can be\n realistically delivered.\n\n\n\n\n 19\n\x0c Evaluation Results\n\n\n\n\n\xe2\x80\xa2 A change management process is needed to ensure that modifications are documented,\n communicated, and agreed upon.\n\n\xe2\x80\xa2 In situations where contract requirements are waived, the waived requirements should\n be clearly cross-referenced to the original contract requirements.\n\n\xe2\x80\xa2 In situations where FDIC determines that it is appropriate to split projects among multiple\n contracts, it is important that these projects are closely coordinated and that contracting\n documents specifically identify how deliverables from the separate contracts will be\n integrated.\n\n\xe2\x80\xa2 Bidders should be made aware of any modifications to solicitation documents, including\n requirements or milestone changes of related contracts or projects.\n\n\xe2\x80\xa2 Contracts should not generally be dependent on contract deliverables that have not\n been fully tested and accepted.\n\n\xe2\x80\xa2 When an initiative requires Capital Investment Review Committee review and\n monitoring, the Committee should monitor all contracts related to the initiative.\n\n\xe2\x80\xa2 In situations where FDIC property, such as software, is furnished to a contractor, strong\n consideration should be given to providing the property \xe2\x80\x9cas is,\xe2\x80\x9d rather than \xe2\x80\x9csuitable for\n use,\xe2\x80\x9d especially for property being produced through another contract or research and\n development effort.\n\n\n\n\n 20\n\x0c Evaluation Results\n\n\n\n\nFigure 2: Project Integration Timeline\n XBAT CONTRACT\n\n ASB Advanced Draft XBAT BOA Draft Task Assignment #1 Task Order #1\n Authorization Letter Functional w/ SOW Functional Signed: 10/17/02\n Requirements Requirements Signed 12/16/02\n Signed 8/09/02 Document Effective: 8/09/02 Document Effective: 8/09/02\n Signed: 9/25/02\n Not to Exceed: 9/11/02 (Revision 10) Task Assignment #2 Cancelled and ratified\n $131,733 System 10/16/02 Signed: 10/17/02 task assignments #1 & 2\n development\n estimated at Waived certain System development\n $395,000 functional increased to $712,000\n requirements.\n\n\n\n CDR SOLICITATION\n Request for Amendment to the Amendment to the Proposals Amendment to the RFP Best and Final\n Proposal Issued RFP RFP received Offers\n October 2002 Received from\n 7/31/02 August 2002 September 2002 10/07/02 XBAT\n CDR Request for Quote and Contractor and\n Quality & Assurance Responses, CDR Winning\n promising a fully functioning Bidder.\n XBAT meeting the 9/02 and 10/02\n FRDs. January 2003\n\n\n\n\n August September October December January\nSource: OIG analysis of XBAT and CDR files .\n\n\n\n\n 21\n\x0c Evaluation Results\n\n\n\n\nCommunications\n\n Communication within and between the IDM team, senior-level managers, and the\n acquisition team was not effective. Accordingly, DIR executives and CDR bidders\n were not aware of the true status of the XBAT development effort.\n\nWhat Should Have Happened\n\nThe PMBOK\xc2\xae Guide identifies project communications management as a subset of project\nmanagement that includes the processes required to ensure timely and appropriate generation,\ncollection, dissemination, storage, and disposition of project information. It provides the critical\nlinks among people, ideas, and information that are necessary for success. Everyone involved\nin the project must be prepared to send and receive communications and must understand how\nthe communications in which they are involved as individuals affect the process as a whole.\nMajor processes of project communications management consist of:\n\n \xe2\x80\xa2 Communications Planning \xe2\x80\x93 determining the information and communications needs of\n the stakeholders, the individuals who need the information, when they will need it, and\n how it will be given to them.\n\n \xe2\x80\xa2 Information Distribution \xe2\x80\x93 making needed information available to project stakeholders in\n a timely manner.\n\n \xe2\x80\xa2 Performance Reporting \xe2\x80\x93 collecting and disseminating performance information,\n including status reporting, progress measurement, and forecasting.\n\n \xe2\x80\xa2 Administrative Closure \xe2\x80\x93 generating, gathering, and disseminating information to\n formalize phase or project completion.\n\nThe APM requires that the acquisition team maintain open communications at all times during\ncontracting efforts.\n\n\nWhat Actually Happened\n\nCommunications surrounding the XBAT engagement were not effective on several levels:\n\nWithin the IDM team. The XBAT TM waived contract requirements at the same time that the\nCDR Technical Evaluation Panel (TEP) Chairman was issuing solicitation documents and\nresponding to bidder questions and answers about XBAT\xe2\x80\x99s functionality. The TEP Chairman\xe2\x80\x99s\nfiles included e-mail documentation addressed to the IDM project manager, CDR TEP\nChairman, and XBAT TM from another IDM team member. The e-mail message discussed the\nneed to notify CDR bidders of changes in XBAT functionality to avoid the potential of CDR\ncontract default on the part of the FDIC. However, the IDM team did not act on this warning.\n\nFurther, it became necessary for the XBAT TM and the IDM project manager to recuse\nthemselves from the CDR solicitation due to potential conflicts of interest. The TM\xe2\x80\x99s recusal\nwas necessary because he was working to secure post-retirement employment with a\nsubcontractor to the CDR winning bidder. The TM coordinated his post-employment efforts with\n\n\n\n 22\n\x0c Evaluation Results\n\n\n\n\nthe FDIC\xe2\x80\x99s Ethics Office. We verified that the TM did not violate post-employment ethics\nrestrictions. However, because he was arranging for employment with a subcontractor to the\nCDR winning bidder, the TM was precluded from discussing or being involved in the CDR\nsolicitation and contract award effort.\n\nThe IDM project manager also recused himself from During a January 2003 CIRC meeting, the\nparticipating in the CDR solicitation because he IDM project manager inappropriately\nowned stock in a company that was bidding for the announced the identity of the CDR\n winning bidder and subcontractor. The\nCDR contract. The project manager stated this\n CIRC is to judge proposed capital\nrecusal impaired his ability to communicate with the investments on the merits of the project,\nCDR TEP Chairman. The project manager told us not based on the reputation of the\nthat he announced his recusal to the rest of the IDM vendors implementing the project. Two\nteam. However, the project manager did not discuss CIRC members, the Deputy to the\n Chairman and the DIR Director\nthis issue with the FDIC Ethics Office or communicate immediately recused themselves from\nhis recusal to DIR executive managers or anyone further participation in the CDR CIRC\noutside of the IDM team. process because they owned stock in the\n CDR subcontractor\xe2\x80\x94the same company\n from which the project manager had\nWe verified the amount of stock that the project recused himself.\nmanager owned during the CDR solicitation and\nconfirmed with the FDIC\xe2\x80\x99s Ethics Office that the project manager was not required to (1) recuse\nhimself from the CDR solicitation effort or (2) communicate his recusal to anyone. While the\nproject manager was technically not required to announce his recusal, given his role as project\nmanager for a $40 million dollar contracting effort, it would have been: (1) prudent for the\nproject manager to discuss the stock ownership issue with the Ethics Office and (2) appropriate\nto communicate the recusal to executive DIR management in the spirit of full disclosure. Doing\nthe latter may have resulted in DIR executive managers reorganizing the IDM team to achieve a\nmore open communication environment.\n\nThe IDM Team and Senior DIR and FDIC Management. The XBAT and CDR status reports\ndid not adequately communicate the timing and resource problems that the IDM team was\nencountering with XBAT or the potential impact that XBAT problems could have on the CDR\nproject. For example, the October 2002 IDM status report included a discussion that the XBAT\ncontract was being modified to a firm fixed price contract for delivery of the Call Report\ntaxonomies and the XBAT tool by December 20, 2002. However, the status report did not\ncommunicate that the XBAT FRD had been changed and that the XBAT contractor would need\nadditional funding to complete all functional requirements by December 2002. This information\nwas critical at the time because the XBAT taxonomies and software had been promised as\ndeliverables to the winning bidder on the CDR contract.\n\nSenior DIR and FDIC management told us they relied on the executive and management-level\nacquisition team to oversee the less expensive XBAT contract. There was no evidence that\nthese senior managers were actively involved in validating the information presented in status\nreports especially at key decision points such as CDR solicitation and contract award. An\nexecutive DIR manager told us that during status meetings with the IDM project manager, she\nasked questions about the status of the XBAT contract and any issues that could impact the\nCDR contract, but received information only that the XBAT project was running smoothly and\non-track. Further, we saw no e-mailed correspondence communicating questions or concerns\nabout the XBAT contract above the IDM project manager level.\n\n\n\n\n 23\n\x0c Evaluation Results\n\n\n\n\nThis level of executive oversight may have been reasonable, given the relatively low-dollar\nvalue of this contract and the high-level members of the contract management team assigned to\nadminister the contract. Because it was an integral part of the larger CDR contract, the XBAT\ncontract needed to have effective control mechanisms in place that would have given executive\nDIR management the true status of the XBAT contracting and development effort. The contract\nadministration process, oversight management function, and the SDLC are critical control\nactivities that are designed to provide executive managers a comfort level about a project\xe2\x80\x99s\nsuccess. While each of these control activities was in place, these processes or functions were\nultimately not completely effective.\n\nMembers of the XBAT Acquisition Team. We saw numerous e-mails evidencing inadequate\ncommunication among the CO, OM, and TM. For example, on October 18, 2002, the OM\ninformed DIRM management of problems being encountered on the XBAT engagement,\nincluding the OM not receiving (1) a signed copy of the contract, (2) a Letter of Oversight\nManager Confirmation, or (3) the SOW. As another example, the TM sent a message to DIRM,\nthe OM, and the IDM project manager on August 9, 2002, communicating technical\nmanagement and oversight management responsibilities. This message was not copied to the\nCO or anyone in ASB.\n\n\nConsequence\n\nCDR bidders were not informed about the status of the XBAT product and the change in\nfunctional requirements. Potential ethics issues were not communicated to senior DIR\nmanagers. Communication within the IDM team and acquisition team and between the IDM\nteam and senior DIR managers was negatively affected.\n\n\nLessons Learned\n\n \xe2\x80\xa2 Continually evaluate communication channels to ensure they are open and effective and\n provide balanced and complete information about a project\xe2\x80\x99s status and viability at key\n development stages.\n\n \xe2\x80\xa2 For information technology engagements, there should be a clear line of responsibility\n and communication between DIRM and the requesting program office. For large\n contracts, it may be appropriate to document lines of responsibility and expectations for\n communication in writing.\n\n \xe2\x80\xa2 When members of the project or contracting team encounter personal or ethical conflicts\n that impact communication, the team should develop other means of effectively\n communicating project status and project issues or consider changes to the teams.\n\n\n\n\n 24\n\x0c Evaluation Results\n\n\n\n\nSystem Development Life Cycle\n\n The XBAT development did not fully comply with the FDIC\xe2\x80\x99s System Development Life\n Cycle. Most importantly, the FDIC paid the XBAT contractor before completely testing\n XBAT. Consequently, the FDIC had limited financial leverage to compel performance\n by the XBAT contractor.\n\nWhat Should Have Happened\n\nFDIC Circular 1320.3, System Development Life Cycle (SDLC) Version 3.0, dated July 17,\n1997, defines a uniform process for developing, maintaining, and enhancing all FDIC automated\ninformation systems, and establishes the SDLC Version 3.0 as the standard methodology for\nsystem development at the FDIC. The objective of the SDLC is to employ a standard set of\npractices to ensure the delivery of quality systems that meet user needs in a timely, cost-\neffective manner.\n\nThe SDLC provides a structured framework and organizes The SDLC is divided into the\nroles, responsibilities, and development activities by clearly following eight\ndefined stages and phases. The phases of the SDLC are interdependent phases:\n 1. Planning\ndelineated by formal end products, quality reviews, and 2. Requirements\nmilestone concurrence and approval. Circular 1320.3 requires Definition\ncompliance with the SDLC process described in the Circular 3. External Design\nby all developers and corporate users involved in planning for 4. Internal Design\n 5. Development\nand the development, acquisition, and maintenance of\n 6. Testing\napplication systems, software, and hardware that support (1) a 7. Implementation\nbusiness function of the FDIC, (2) one or more offices within 8. Maintenance\nthe FDIC, or (3) the exchange of data.\n\nDIRM has overall responsibility for the management and development of corporate information\ntechnology. The program manager in information technology projects is the principal user\nrepresentative from the FDIC division or office requesting or relying upon the automated\ninformation system. As the sponsor of the system, the program manager is the final user\napproval authority for the system. As such, the program manager is accountable and\nresponsible for defining the required performance and ensuring that the stakeholders in the user\ncommunity are involved in each phase of activities and that the developed application is\naddressing user needs.\n\n\nWhat Actually Happened\n\nThe XBAT development did not fully comply with the FDIC\xe2\x80\x99s SDLC provisions outlined in\nCircular 1320.3. For example, DIRM and the IDM team did not complete the SDLC test phase\nbefore paying the XBAT contractor. The test phase provides that the user and other designated\ntesters validate that the functional requirements defined in the FRD are satisfied by the\ndeveloped system and that there are no adverse effects on the overall process or other existing\nsystems.\n\n\n\n\n 25\n\x0c Evaluation Results\n\n\n\n\nThe SDLC requirements were included in The September 2002 FRD included the following\nthe request for proposal for the XBAT requirements:\nsolicitation, the BOA executed for the XBAT\n Section 4.5, Testing and Traceability: A process\ncontract, and the September 2002 FRD must be in place to trace requirements and test all\ncontaining functional requirements for the aspects of the system. Specifically,\nXBAT. These documents specifically\nreferenced FDIC Circular 1320.3. The \xe2\x80\xa2 For testing of the developed system (or for future\nXBAT SOW included the provision that, in enhancements), test data must be provided to\n enable a thorough test of the system to be\nall cases, the principles, phases, and completed.\ndeliverables of the FDIC SDLC as\ndescribed in Circular 1320.3 must be \xe2\x80\xa2 Modules for each function must be separate so\napplied to the XBAT application. that testing of one process can be made\nHowever, the SDLC provisions were independent of another process.\nremoved from the revised FRDs prepared \xe2\x80\xa2 Changes and additions to the requirements must\nfor XBAT in October 2002. Further, the be added to the Traceability Matrix for reference\nSDLC provisions were not included in the during future phases.\ntask order awarded for the XBAT\n These requirements were eliminated from the\nengagement.\n October 2002 and June 2003 XBAT FRDs.\n\nIn assessing the XBAT product delivered by the FDIC, the CDR contractor determined that\nXBAT met the SDLC requirements for the external design and internal design phases.\nHowever, the CDR contractor\xe2\x80\x99s assessment indicated that XBAT did not meet the other SDLC\nrequirements. One of the OMs told us that the TM, in consultation with ASB contracting\nrepresentatives, agreed to a partial waiver of SDLC requirements, including the requirement for\ntesting the system.\n\nThe XBAT contractor told us that XBAT user acceptance testing was the responsibility of the\nFDIC. Under the SDLC, user acceptance testing is the final test of the system being developed.\nThe XBAT task order included the following provision related to user acceptance testing:\n\n The FDIC will develop test scripts for and sponsor the User Acceptance Testing [UAT].\n The contractor will assist in UAT which includes fixing bugs or system errors uncovered\n during this process. This will be completed no later than March 31, 2003.\n\nThe FDIC\xe2\x80\x99s Systems Development Life Cycle Manual Version 3.0 requires information\ntechnology development projects to include a test plan to ensure that all aspects of the system\nare adequately tested and that the system can be successfully implemented. The test plan is\nbegun in the requirements definition phase and refined throughout the design and development\nphases. Table 6 presents the various types of tests that should be conducted during system\ndevelopment, the party responsible for performing the test, and whether the test was performed\nunder the XBAT development effort.\n\n\n\n\n 26\n\x0c Evaluation Results\n\n\n\n\nTable 6: SDLC Testing Requirements\nType of Test Description Performed By Performed under\n XBAT?\n\nUnit/Module Validates module\xe2\x80\x99s logic and adherence to FRD and Individual 80 percent\nTest technical specifications. developing the completed;\n code. requirement waived\n in Task Order 01.\n\nIntegration Involves the subsystems that are made up of integrated Developing Completed in\nTest groupings of software units and modules. organization June 2003.\n\nSystem Independent test that determines whether the system DIRM 90 percent\nQualification complies with standards and whether it satisfies completed in\nTest functional and technical requirements when executed on June 2003.\n target hardware using operational data files. Software\n performance, response time, ability to operate under\n stressed conditions, interfaces to other applications,\n security, and integrity control functions are tested.\n\nUser Performed by the user in a nonproduction environment Client who will 90 percent\nAcceptance which mirrors the environment in which the system will be use the system. completed in\nTest used. System interoperability, all documentation, system June 2003.\n reliability, and the level to which the system meets user\n requirements will be evaluated. Performance, recovery,\n and restart, and data quality testing may be performed.\nSource: FDIC SDLC Manual and IDM Information.\n\nThe XBAT contractor submitted three invoices during 2002 under the XBAT contract, as shown\nin Table 7. The FDIC held each invoice for several months and ultimately paid the invoices in\nFebruary and March 2003.\n\nTable 7: XBAT Invoices and Payment Dates\nInvoice Invoice Amount Invoice Date Check Date Purpose\n\n1 $75,690 9/26/02 2/10/03 Progress bill for professional services to support\n the development of (1) XBRL taxonomies, (2)\n XBAT, and (3) XBRL demonstration application.\n\n2 $299,592 11/04/02 2/05/03 Progress bill for work performed 9/1/02 \xe2\x80\x93\n 10/31/02 in support of the development of XBRL\n taxonomies and XBAT. Invoice stated that\n 55 percent of the work related to Task Order 01\n had been completed.\n\n3 $328,048 12/31/02 3/06/03 Progress bill for work performed 11/1/02 \xe2\x80\x93\n 12/31/02 in support of the development of XBRL\n taxonomies and XBAT. Invoice stated that the\n December 20th delivery of work under Task\n Order 01 had been completed.\nSource: Contracting files.\n\nIt appears that the FDIC paid the third invoice before the FDIC had completed user acceptance\ntesting and determined that XBAT was fully functioning. For example, following FDIC payment\nof the third invoice, an XBAT status report, dated April of 2003, notes that 78 issues are\noutstanding, 72 of which pertain to user acceptance testing. The XBAT contractor noted that of\nthe 78 issues remaining, 18 (23 percent) were critical (1 issue) or high priority (17 issues).\n\n\n\n\n 27\n\x0c Evaluation Results\n\n\n\n\nConsequence\n\nBecause the FDIC did not fully test XBAT before paying the XBAT contractor, FDIC had limited\nfinancial leverage to hold the XBAT contractor responsible for fulfilling contract requirements\nsatisfactorily.\n\n\nLessons Learned\n\n \xe2\x80\xa2 System development contracting efforts should follow the FDIC\xe2\x80\x99s System Development\n Life Cycle requirements contained in FDIC Circular 1320.3 or an equivalent.\n\n \xe2\x80\xa2 Contractors should be paid for their services only after deliverables are tested and\n accepted.\n\n\n\n\n 28\n\x0c Corporation Comments and OIG Evaluation\n\n\n\n\nCORPORATION COMMENTS AND OIG EVALUATION\nOur draft report contained no recommendations, and a written corporate response was not\nrequired. However, we solicited comments from DIR, DIRM, and the Division of Administration\n(DOA) on the lessons learned presented in the report and the discussion of the XBAT\ncontracting and project management issues. DIR did not issue formal comments, but met with\nus to clarify certain aspects of the report. We subsequently made minor editorial changes to the\nreport to address DIR\xe2\x80\x99s comments. DIRM did not issue formal comments, but shared the report\nwith DIRM managers responsible for contract administration and project management activities\nfor their consideration on future projects. The Associate Director, ASB, provided DOA\xe2\x80\x99s written\nresponse, dated March 18, 2004. DOA agreed that although the FDIC received value from the\nXBAT procurement, the project\xe2\x80\x99s success was impaired by certain contracting and project\nmanagement issues. DOA\xe2\x80\x99s response summarizes some contracting efforts ASB has in\nprocess or expects to undertake in 2004 to enhance the overall controls governing the\ncontracting process and to minimize the issues identified in our report for future system\ndevelopment efforts. DOA\xe2\x80\x99s response is presented in its entirety in Appendix V to this report.\n\n\n\n\n 29\n\x0c Appendix I\n Objective, Scope, and Methodology\n\n\n\nAppendix I: Objective, Scope, and Methodology\nThe objective of this evaluation was to assess the adequacy of the FDIC\xe2\x80\x99s contract\nadministration and project management for the development of the XBAT. We conducted this\nreview in response to a request from FDIC senior management that the OIG and the Office of\nInternal Control Management evaluate the XBAT contracting and development efforts. The\nrequestors expressed concerns that the XBAT product delivered for the CDR contractor was not\nfully functional and asked that we identify lessons learned in contract administration and project\nmanagement of the XBAT procurement that could be applied to the larger CDR effort.\n\n\nScope\n\nTo accomplish our objective:\n\n \xe2\x80\xa2 We evaluated the effectiveness of the implementation of specific management controls\n related to the contract administration and project management for the development of\n the XBAT. These controls included the policies and procedures contained in the FDIC\n Acquisition Policy Manual as they apply to the XBAT procurement and oversight controls\n such as status reporting of the XBAT development project.\n \xe2\x80\xa2 We reviewed the XBAT procurement activities from fund authorization in May 2002\n through contract award in December 2002. Because the XBAT contract is still in effect,\n we updated information through December 2003 for our report.\n \xe2\x80\xa2 We reviewed the FDIC Acquisition Policy Manual \xe2\x80\x93 Revision 1, effective March 31, 2000,\n for provisions that would apply to the XBAT procurement, which was awarded\n subsequent to the March 31, 2000 date and prior to the July 2003 subsequent update to\n the FDIC Acquisition Policy Manual.\n \xe2\x80\xa2 We conducted our evaluation work at the headquarters offices of DIRM, DIR, and DOA\n in Washington, D.C., during the period October through December 2003.\n\n\nMethodology\n\nWe performed the following activities for our evaluation:\n\n \xe2\x80\xa2 We interviewed headquarters DIRM, DIR, and DOA officials who were responsible for\n contract administration and project management for the development of XBAT.\n \xe2\x80\xa2 We interviewed representatives of the XBAT contractor.\n \xe2\x80\xa2 We interviewed an official in FDIC\xe2\x80\x99s Ethics Office.\n \xe2\x80\xa2 We reviewed key documents related to contract administration and project management\n of the XBAT development project, including:\n\n o FDIC Acquisition Policy Manual.\n o Oversight Manager Files.\n o Technical Monitor Files.\n o XBAT Pre-Award Files.\n o XBAT Contract Files.\n o IDM XBAT Files.\n o IDM Monthly Project Status Reports on XBAT and CDR Projects.\n o DIR Monthly Status Reports.\n\n\n 30\n\x0c Appendix I\n Objective, Scope, and Methodology\n\n\n o DIRM Status Report to the FDIC\xe2\x80\x99s Chief Operating Officer.\n o XBAT Contractor Status Reports.\n o Minutes of Call Modernization Project Steering Committee Meetings.\n o Electronic Messages provided by two Technical Monitors, two Oversight\n Managers, the CDR TEP Chairperson, IDM Team members, and the IDM Project\n Manager.\n o Electronic Messages included in the XBAT Pre-Award and Contract Files.\n o Bidders\xe2\x80\x99 Questions and FDIC\xe2\x80\x99s Answers for the XBAT Solicitation.\n o Bidders\xe2\x80\x99 Questions and FDIC\xe2\x80\x99s Answers for the CDR Solicitation.\n o XBAT Functional Requirements Documents, dated September 2002; October 2,\n 2002; October 16, 2002; and June 2003.\n o Various Versions of the CDR Contractor\xe2\x80\x99s Assessment of XBAT.\n o CDR Change Order 01, executed October 25, 2003.\n o FDIC Training Server Information for Oversight Manager Training.\n\nWe corroborated automated information used to support our evaluation results and lessons\nlearned with other sources to ensure the information was sufficiently reliable. For example, we\ndiscussed information contained in project status reports and electronic messages with key\npersonnel involved in the XBAT procurement activities.\n\nOur work to address the Government Performance and Results Act included reviewing the\nFDIC\xe2\x80\x99s 2003 Annual Performance Plan, in particular, the annual performance goal to maintain\nsufficient and reliable information on insured depository institutions. Embedded in the annual\nperformance goal is a target to develop a more efficient approach to bank data collection and\nmanagement. The CDR project is designed to modify and improve data management\nprocesses to more effectively collect, manage, and share information about insured institutions.\nThe XBAT is an integral part of the CDR.\n\nIn addition, we reviewed the FDIC 2003 Agenda: Corporate Performance Objectives, Third\nQuarter Summary Report. In the Strategic Objective, Stability, the FDIC has established a\nperformance objective to make high-quality banking data available to the public on a more\ntimely basis. As part of the CDR project, the FDIC, FRB, and OCC are actively working with the\nfinancial institutions and trade groups on strategies to improve data quality and timeliness when\nthe new CDR is implemented.\n\nWe did not develop specific evaluation procedures to detect abuse and illegal acts because we\ndid not consider abuse and illegal acts to be material to the evaluation objective. However,\nthroughout our evaluation, we were sensitive to the potential of illegal acts, including fraud,\nwaste, abuse, and mismanagement.\n\nWe did not assess the FDIC\xe2\x80\x99s compliance with applicable laws and regulations because we did\nnot identify specific laws or regulations pertaining to the implementation of contract\nadministration, or project management controls.\n\nWe conducted our evaluation in accordance with generally accepted government auditing\nstandards.\n\n\n\n\n 31\n\x0c Appendix I\n Objective, Scope, and Methodology\n\n\nProject Management Criteria\n\nAlthough the FDIC is not required to comply with the Project Management Body of Knowledge\nPMBOK\xc2\xae Guide, we used it as the primary criteria for assessing FDIC\xe2\x80\x99s project management\ncontrols for the XBAT procurement, especially in the areas of change control management and\nproject integration. This guide contains generally accepted industry practices related to\nsuccessful project management. Generally accepted means that the knowledge and practices\ndescribed are applicable to most projects most of the time and that there is widespread\nconsensus about their usefulness. Generally accepted does not mean that the knowledge and\npractices described are to be applied uniformly on all projects; the project management team is\nalways responsible for determining what is appropriate for any given project.\n\n\n\n\n 32\n\x0c Appendix II\n XBAT Contracting and Project Management Challenges\n\n\n\nAppendix II: XBAT Contracting and Project Management Challenges\n\n What Actually What Should Have Happened Consequences Lessons Learned\n Happened\nFDIC did not establish In initiating procurement actions, Because functional 1. Functional requirements should be established early in the\nXBAT functional requirements should be defined to requirements were procurement process.\nrequirements early provide a clear and specific description evolving during the\nenough in the of the goods or services required. XBAT development, the 2. When identifying procurement requirements, provide a clear and\ncontracting process. APM 4.A.4.a.(3) FDIC and XBAT specific description of the goods or services required.\n contractor did not have\n a clear and consistent\n understanding of what\n functionality XBAT\n should include.\n\nFDIC did not A procurement requirements package Because the XBAT 1. Changes to system requirements should not be driven solely by\nadequately plan or should include a schedule for delivery of procurement was not resource and schedule constraints \xe2\x80\x93 consideration must also be given\nexecute the XBAT goods or performance of services and a adequately planned or to impact on business needs .\nprocurement effort. price estimate, including the base executed, the quality of\nThe IDM team waived period and all option periods . (APM the XBAT software 2. Research and development efforts do not lend themselves to a firm\nimportant XBAT 4.A.4.a. (3) and (5)) suffered. Further, fixed price contracting vehicle, given the uncertain and evolving nature\nfunctions that were because the of the services. The type of contract awarded should be based on the\nintended to support the A BOA is a written agreement between procurement did not level of risk to the Corporation, with fixed price contracts generally\noverall CDR effort to the FDIC and a contractor, containing consistently follow being used when risk has been reduced to a reasonable level.\nmeet cost and time terms and conditions that will apply to acquisition policies and\nconstraints. Further, FDIC-issued task orders during the term procedures, the FDIC 3. When procuring services for research and development,\nthe XBAT procurement of the agreement. (APM 3.B.3.b) was party to a contract communicate to everyone involved that the project is a research and\ndid not consistently that did not protect the development effort, requiring closer monitoring.\nfollow the APM. A System Development Life Cycle Corporation\xe2\x80\x99s best\n contract with task assignments can be interests. 4. A time and materials level of effort contracting vehicle may be more\n used in DIRM contracts that incorporate suitable for a research and development effort, given the difficulty in\n a system development life cycle providing a detailed statement of work or to estimate the price or\n documenting the steps to be taken by duration of time required for a research and development effort.\n the contractor. (APM 3.B.9.a.)\n 5. Involve the FDIC Legal Division early in procurement planning and\n subject key contracting documents such as the contract and statement\n of work to legal review and concurrence prior to contract execution.\n Focus the legal review on both legal sufficiency and protecting the\n Corporation\xe2\x80\x99s interest, particularly for contract provisions that pose the\n greatest risk for the Corporation.\n\n\n\n\n 33\n\x0c Appendix II\n XBAT Contracting and Project Management Challenges\n\n\n\n\n What Actually What Should Have Happened Consequences Lessons Learned\n Happened\nThe CO and OM did The OM is responsible for overseeing Because the CO and 1. The CO, OM, and TM should work together as a team in meeting\nnot completely fulfill the performance requirements of the OM did not completely their assigned responsibilities as part of establishing a control\ntheir responsibilities contract, provides business and fulfill their framework.\nand relied on the TM technical liaison between the FDIC and responsibilities and\nfor making changes to the contractor, reviews and approves relied too heavily on the 2. The OM should involve the CO in addressing contractor\nthe functional invoices for payments, verifies TM for managing the performance matters.\nrequirements, satisfactory delivery of contract items, contract, the XBAT\naccepting contract and notifies the CO of any need to procurement did not 3. OM and TM roles and responsibilities need to be clearly\ndeliverables, and change the contract. (APM 7.B.1.h. have adequate checks documented and communicated in the Letter of OM and TM\napproving a key and i.) and balances to help Confirmation.\ninvoice payment. ensure the project\xe2\x80\x99s\n The CO is responsible for ensuring success. 4. A CAP is a good tool for helping ensure that the contractor performs\n compliance with the contract. (APM in accordance with the terms and conditions of the contract and is\n 7.B.2.a.) already required by the APM.\n\n The duties of a TM are a subset of the 5. A CAP allows the OM and CO to identify specific deliverables and\n duties of the OM, but the responsibility due dates and track modifications and funding.\n for contractor oversight remains with the\n OM. (APM 7.B.1.d.)\n\nFDIC did not exercise Scope change control involves The FDIC promised a 1. Closely coordinate projects that are interdependent, and promise\nadequate change influencing the factors that create scope software product only what can be realistically delivered.\nmanagement or project changes to ens ure that changes are containing functionality\nintegration during the agreed upon, determining that change that it could not deliver 2. A change management process is needed to ensure that changes\nXBAT development has occurred, and managing the and had to issue an are documented, communicated, and agreed upon.\neffort and the CDR changes when they occur. Scope $840,000 change order\nsolicitation process. change control must be integrated with to the CDR contractor to 3. In situations where contract requirements are waived, the waived\nThe CDR solicitation other control processes, such as replace XBAT. requirements should be clearly cross-referenced to the original\npromised a fully scheduling, cost, quality, risk, and contract requirements.\nfunctioning XBAT at staffing. (PMBOK\xc2\xae Guide 5.5 and 4.3)\nthe same time that the 4. In situations where FDIC determines that it is appropriate to split\nIDM team was waiving projects among multiple contracts, it is important that these projects\nimportant functional are closely coordinated and that contracting documents specifically\nrequirements to meet identify how deliverables from the separate contracts will be\ncost constraints and integrated.\nproduct delivery\nmilestones. 5. Bidders should be made aware of any modifications to the\n solicitation documents, including requirements or milestone changes\n of related contracts or projects.\n\n\n\n\n 34\n\x0c Appendix II\n XBAT Contracting and Project Management Challenges\n\n\n\n\n What Actually What Should Have Happened Consequences Lessons Learned\n Happened\n 6. Contracts s hould not generally be dependent on contract\n deliverables that have not been fully tested and accepted.\n\n 7. For major projects involving multiple contracts, the CIRC should\n consider all related contracts.\n\n 8. In situations where FDIC property, such as software, is furnished to\n a contractor, strong consideration should be given to providing the\n property \xe2\x80\x9cas is,\xe2\x80\x9d rather than \xe2\x80\x9csuitable for use,\xe2\x80\x9d especially for property\n being produced through another contract or research and\n development effort.\n\nCommunication within Project communications management CDR bidders were not 1. Continually evaluate communication channels to ensure they are\nand between the includes the processes required to informed about the open, effective, and provide balanced and complete information about\nproject team, senior- ensure timely and appropriate status of XBAT and the a project\xe2\x80\x99s status and viability at key development stages.\nlevel managers, and generation, collection, dissemination, change in functional\nthe acquisition team storage, and disposition of project requirements. Potential 2. In information technology engagements, there should be a clear line\nwas not effective. information. (PMBOK\xc2\xae Guide 10) ethics issues were not of responsibility and communication between DIRM and the\n communicated to senior requesting program office. For large contracts, it may be appropriate\n DIR managers. to document lines of responsibility and expectations for\n Communication within communication in writing.\n the IDM team and\n acquisition team and 3. In situations where members of the project or contracting team\n between the IDM team encounter personal or ethical conflicts that impact communication, the\n and DIR senior team should develop other means of effectively communicating project\n managers was status and project issues or consider changes to the teams.\n negatively affected.\n\nXBAT development did FDIC Circular 1320.3, System Because FDIC did not 1. System development contracting efforts should follow FDIC\xe2\x80\x99s\nnot fully comply with Development Life Cycle (SDLC) fully test XBAT before System Development Life Cycle requirements contained in FDIC\nthe FDIC\xe2\x80\x99s System Version 3.0, included in the solicitation paying the XBAT Circular 1320.3, or an equivalent.\nDevelopment Life for XBAT and the Functional contractor, FDIC had\nCycle. Most Requirements Documents, defines a limited financial 2. Contractors should be paid for their services only after deliverables\nimportantly, the FDIC uniform process for developing, leverage to hold the are tested and accepted.\npaid the XBAT maintaining, and enhancing all FDIC XBAT contractor\ncontractor before automated information systems, and responsible for fulfilling\ncompletely testing the establishes the FDIC System Develop- contract requirements .\nXBAT. ment Life Cycle Manual Version 3.0 as\n the standard methodology for system\n development at the FDIC.\nSource: OIG Analysis.\n\n\n\n 35\n\x0c Appendix III\n Key Events in the XBAT Project\n\n\n\nAppendix III: Key Events in the XBAT Project\n\nMonth Event\n\nDecember 2001 FDIC Chairman announces goal of providing accurate and timely Call Report data to the\n public as soon as it is available.\n\nJune 2002 XBAT request for proposal issued by FDIC.\n\nJuly 2002 FDIC developing requirements for the XBAT tool.\n\nAugust 2002 FDIC issues advance authorization letter to XBAT contractor while contracting documents\n are undergoing review.\n\nSeptember 2002 FDIC and Contractor sign Basic Ordering Agreement for XBAT.\n\nOctober 2002 FDIC OM and XBAT contractor sign Task Assignments 01 & 02.\n\nNovember 2002 FDIC continues working on a Task Order.\n\nDecember 2002 FDIC and XBAT contractor sign Task Order 01 effective August 9, 2002. Work completed on\n version 1 of XBAT.\n\nJanuary 2003 Best and Final Offer for the CDR Contract. XBAT promis ed as a deliverable to the winning\n bidder on the CDR Contract. XBAT undergoing quality assurance testing.\n\nFebruary 2003 XBAT Basic Ordering Agreement is modified to increase funding by $30,000.\n\nMarch 2003 Call Report analysts participate in quality assurance testing for the validation criteria\n component of XBAT.\n\nApril 2003 Version 1.3 of XBAT undergoing quality assurance testing.\n\nMay 2003 FDIC awards CDR contract.\n\nJune 2003 XBAT given to CDR contractor for analysis of XBAT architecture and outputs.\n\nJuly 2003 CDR contractor completing a detailed analysis of XBAT, focusing on business and technical\n requirements.\n\nAugust 2003 CDR contractor conducts an analysis of the status of the XBAT tool against the Functional\n Requirements Document posted as part of the CDR Request for Proposal.\n\nSeptember 2003 XBAT contractor continues to correct nine issues/problems identified in the XBAT application.\n\nOctober 2003 CDR Contract Change Order 01 awarded for design and development of a Meta-data\n Management Tool to replace XBAT.\nSource: XBAT Project Management Files.\n\n\n\n\n 36\n\x0c Appendix IV\n XBAT Procurement Requirements\n\n\n\n\n Appendix IV: XBAT Procurement Requirements\nProcurement Request for Proposal BOA signed Task Assignment 01 Task Assignment 02 Task Order 01 signed\nRequirement Issued June 7, 2002 September 25, 2002 Signed October 17, 2002 Signed October 17, December 20, 2002,\n 2002 effective August 9, 2002\nScope of Provide consulting No change. Analyze and present an Develop XBAT to By 12/20/02, develop XBRL Call\nWork support for the optimal architectural create and maintain Report frameworks for the\n development of a set of solution for the XBRL Call Report reporting periods of 3/2001 and\n XBRL taxonomies and a development of an XBRL frameworks. 3/2002, and analyze, design,\n business analyst tool for Business Analyst Tool and develop an application\n FFIEC Call Reports. and complete called XBAT.\n development of XBRL\n Call Report frameworks By 6/30/04, provide technical\n for period March 2001 input to FDIC for developing\n through December 2002. frameworks for 6/2001, 9/2001,\n 12/2001, 6/2002, 9/2002,\n 12/2002, and 3/2003.\nFunctional The FDIC will provide No change. Contractor is required to The contractor will The XBAT will be developed to\nRequirements draft requirements for a deliver a written functional develop the XBAT to meet requirements in the FRD\n FFIEC XBRL business requirements document meet requirements as with the final date of October 16,\n analyst tool. for XBAT by September 6, described in the FRD 2002.\n 2002. with the final date of\n September 30, 2002.\nSDLC In all cases, the principles, No change. SDLC requirements were SDLC requirements SDLC requirements not\n phases, and deliverables not specifically mentioned were not specifically specifically mentioned in Task\n of the FDIC SDLC as in Task Order 01. mentioned in Task Order 01 or referenced in the\n described in FDIC Circular Order 02, but were FRD.\n 1320.3 must be applied. referenced in the\n September 2002 FRD.\n\nFunding 5/08/02 procurement September 2002 BOA for N/A N/A Task Order covering 23 months\nProvisions requisition for $1.19 $682,330 covering 4 for $712,330, of which $703,330\n million covering 6.5 years. years. was related to development.\n --Development --$498,400 --Development--$395,200\n --Maintenance --$693,456 --Maintenance--$287,130\n\nPeriod of June 3, 2002 through August 9, 2002 through August 13, 2002 through September 18, 2002 August 9, 2002 through June 30,\nPerformance December 31, 2008. August 9, 2003 with three March 31, 2003. through March 31, 2004.\n 1-year options. 2003.\n\n Source: FDIC Contracting Files for XBAT.\n\n\n\n 37\n\x0c Appendix V\n Corporation Comments\n\n\nCORPORATION COMMENTS\n\x0c Appendix V\n Corporation Comments\n\n\n\n\n39\n\x0c'