Steering toward analytics certification

INFORMS turns the ship in short order with launch of first-of-its-kind CAP™ program.

By Gary Bennett and Jack Levis

It is said that it takes time and care to turn a ship in a different direction. INFORMS did just that in a remarkably short time frame with its new analytics certification program. Three years ago the program was just a thought of a few members of the Board of Directors. In April 2013 it became a reality with the designation of INFORMS’ first Certified Analytics Professionals (CAP™).

Analytics certification fits nicely into the INFORMS mission statement of “advancing the development and dissemination of all aspects of OR/MS.” Of course, to agree with this position you need to subscribe to the theory that analytics and OR/MS are very closely related – in fact near synonyms. According to research performed by Liberatore and Luo in 2011, most members believe this to be true. The Board in 2011 went even further and said that INFORMS “will be recognized as the leading association for advanced analytics professionals by advancing the practice, research, methods and applications of advanced analytics, and identifying and serving analytics professionals with products and services they value.” The Board of Directors in 2010 made a strategic decision to embrace and serve the analytics movement. One of the first products considered – one that would be a differentiator for INFORMS – was analytics certification.

The paragraphs that follow provide a record of INFORMS’ role as the certification provider in the analytics profession, much like the Project Management Institute (PMI) serves the project management profession and the International Institute of Business Analytics (IIBA) serves the business analysis profession.

Spring 2010: An
Certified Analytics Professionals
alytics Market Study Begins

The Board decided to sponsor a study of the analytics marketplace and INFORMS’ place in it. VP Practice Jack Levis, (then) VP Marketing, Communications & Outreach Anne Robinson and Director of Marketing Gary Bennett were assigned to implement the analytics market study. This three-person team researched various consulting firms, drafted the RFP, participated in “get-to-know-you” calls with many of the candidates, reviewed proposals, attended candidate firms’ presentations and participated in the scoring and selection of the eventual winning firm, Capgemini.

The kickoff meeting with Capgemini and the analytics market study steering committee was held on April 7, 2010. Following internal discussions, it was decided that Capgemini would base their initial findings on interviews with representatives of the analytics community not affiliated with INFORMS and draw on their own experience of developing an analytics center of excellence at their headquarters in the United Kingdom.

Summer 2010: Analytics Market Study Completed

Capgemini completed the Analytics Market Study in early June 2010. The results were presented at the summer Board meeting and concluded that INFORMS should serve this constituency and in doing so provide a great service to the overall profession. The study uncovered several potential products and services worth pursuing. The one that really stood out as a differentiator for INFORMS and the one that would have the most immediate impact on the analytics market was certification – a voluntary process through which an organization grants recognition to an individual after verifying that he or she has met minimum criteria, including but not limited to passing an assessment. The No. 1 issue faced by the analytics market was “how do I find good analytics talent?” Certification was seen as a possible tool to help them in this endeavor.

Fall 2010: Analytics Market Study Socialized

The study leaders, Anne Robinson, Jack Levis and Gary Bennett, embarked on a socialization of the study findings within INFORMS. More than 20 committees and subdivision boards were briefed on the findings and asked to provide feedback. Consensus was to pursue several short-term tactics to begin changing INFORMS’ culture while making plans for a bigger long-term strategic “ask” for the Board to consider in November of 2010.

Anne, Jack and Gary wrote a thorough recap of the study for the October issue of OR/MS Today. Entitled “INFORMS to Officially Join Analytics Movement” [1], the article was designed to encourage membership to read and react to this overall shift in INFORMS strategy. It was also hoped that officers and committee leaders across INFORMS would point to this article in their efforts to socialize the study and gather feedback.

Anne and Jack also presented a session at the INFORMS Annual Meeting entitled “Analytics: The Future of INFORMS,” which also provided a recap of the study and discussion of next steps.

Winter 2011: Certification Feasibility Study Plans

At the November 2010 Board Meeting, the Board approved funds to complete a more in-depth study into launching a certification program, a training/continuing education program and to enrich the spring Practice (renamed Analytics) Conference with more analytics-related programming.

Gary Bennett was charged with finding a credentialing consultant who could assist with further study into the demand for an INFORMS-run analytics certification program. After researching and considering several consultants, INFORMS selected and retained the services of Michael Hamm, principal, Michael Hamm and Associates. A kickoff call was held with Hamm on Jan. 10, 2011.

Hamm recommended an in-depth study of approximately six months. At the conclusion of the study, INFORMS would gain an understanding of the overall need and demand from both potential candidates and employers for an analytics certification program, an estimated five-year return on investment based on findings from the study coupled with Hamm’s expertise, and a clear recommendation on whether INFORMS should pursue certification.

Hamm’s overall study plan included a review of existing research, one-on-one telephone interviews with key staff and volunteer leadership to gather opinions, two focus groups with candidate and employer stakeholders, two quantitative online surveys with candidate and employer stakeholders, and follow up telephone interviews with stakeholders to better understand the quantitative findings. Hamm presented these findings and results in a final written report and an additional in-person report.

Spring 2011: Certification Feasibility Study Commences

At the spring 2011 INFORMS Board Meeting, much progress was reported on the certification study. Hamm had completed: 1. the review of existing research – including all work done in 2008 and 2009 by the previous INFORMS Credentialing Committee – and a thorough review of the Capgemini Analytics Market Study; 2. one-on-one telephone interviews with 14 INFORMS leaders, representing a cross-section of INFORMS Board members, members of the ad hoc Analytics Market Study Committee and others; and 3. a focus group of employers of potential analytics credential candidates at Predictive Analytics World in San Francisco in March.

The following points recap the findings from the 14 interviews with INFORMS leaders:

  • Enthusiasm and interest was expressed in offering a certification for the analytics community. Most of the concerns noted revolved around economic issues and the challenges of the design/marketing of a certification that would appeal to the largest analytics market segment.
  • The majority of leaders preferred a certification program vs. a curriculum certificate program as the first development priority, but many expressed a desire for INFORMS to pursue both options.
  • Perceptions about the size of the market for an analytics credential varied from 20,000 to 300,000 people in the United States, with a larger global candidate population.
  • Most leaders believed that there was no agreed upon body of knowledge in analytics, but some indicated that this may change very soon.
  • Leaders indicated a belief that certification candidate costs/fees would likely be split evenly between applicants and employers in most cases, but this practice would vary considerably depending upon the employer.
  • The typical candidate would likely have the title “analytics” in his/her job description, this individual would probably have a BA/MA degree, and potential candidates would probably have 3-5 years of analytics job experience.
  • One leader thought the current demand for an analytics certification in India might be stronger than in the U.S.

The following recaps the findings from the employer focus group held at Predictive Analytics World San Francisco in March 2011:

  • There was definite interest in a certification but acknowledgement that very specific skills and abilities would be expected from such credential holders.
  • Large employers could derive more benefits from a national/international certification program.
  • Most participants favored a certification over a curriculum certificate program but expressed some interest in seeing both models available for this field.
  • Most participants agreed the certification fee would be shared by the candidate and employer.
  • Participants agreed that they would like the certification to focus at least to some degree on all three levels of analytics – descriptive, predictive and prescriptive – but that the credential should probably be targeted toward a mid-level to senior analytics position.
  • Most of the participants were not familiar with INFORMS and were not sure whether INFORMS is the best/proper sponsor for a new certification program. During the discussion no other organization was suggested as a better candidate for this role, but participants did suggest that the lack of awareness of INFORMS would be a significant marketing challenge in this endeavor.
  • Participants were not sure if their organization would endorse an INFORMS certification program even if it met all of their needs.
  • Participants saw a clear need for standardization and uniformity in defining analytics competencies.
  • Participants acknowledged that defining the scope of the certification would be a significant challenge, and academic politics could interfere with the process of agreeing upon an acceptable scope/content.

Summer 2011: Certification Feasibility Study Concludes

After six months of in-depth study, Hamm concluded that INFORMS should pursue offering a professional certification in analytics. Two of the feasibility criteria that should be met in any potential new certification program are: 1) a pool of likely candidates in the 1,000+ range; and 2) a strong demonstration of interest by at least 50 percent of feasibility study participants. The completed quantitative study pointed to about 12,000 likely candidates and strong interest by more than 60 percent of the study participants.

Hamm estimated that an INFORMS-sponsored analytics certification program could be reasonably expected to start covering and exceeding annual operating costs by year five, could help to increase membership as long as member discounts are offered, could help to increase meeting participation as long as meetings are used for official continuing education for recertification purposes, and would provide INFORMS with a legitimate platform to claim professional leadership in the analytics community.

Respondents preferred professional certification (mid- or higher-level) over entry-level certification. Roughly three-quarters of all study respondents who believed they might be candidates for this certification expressed substantial interest and said the certification would be valuable. Roughly two-thirds of all employers expressed similar opinions.

An analysis of all of the 740 candidate respondents who indicated an interest in the “Professional Certification” category revealed the following traits:

  • Most candidates anticipated that they likely would pay for certification.
  • 88 percent believed that INFORMS is an appropriate sponsor.
  • 69 percent believed that their peers will seek certification.
  • 18 percent of this group already maintains some perceived certification in analytics.
  • Most of these individuals have an “analyst” or “consultant” job function.
  • Most work in many different industries, but they are more likely to be employed in a consulting firm.
  • 43 percent (317 individuals) left contact information indicating an interest in working on the development of a new analytics certification program, which is considered a large number with this specific level of interest.

Non-INFORMS members make up the largest share of the potential applicant pool, but nearly nine in 10 INFORMS practice members surveyed indicated strong interest in the certification. Most study respondents said INFORMS was an appropriate sponsor for an analytics certification program. Even when not aware of INFORMS, most respondents could not identify a more appropriate sponsor. Even though potential joint ventures for an analytics certification were found to be unlikely, the study did indicate a good possibility of a potential joint endorsement program. Although most study respondents preferred a certification fee as low as possible, a strong number of respondents noted the appropriateness of fees in the range of $400-$500. (INFORMS subsequently priced certification at $495 for members.)

The INFORMS Board of Directors accepted the research findings, combined with the favorable five-year projection, and approved the establishment of an INFORMS-sponsored analytics certification program in November 2011.

Task Force Appointed

Shortly after the November Board meeting, a certification task force was appointed by Anne Robinson, Jack Levis and Bill Klimack (who also serves as a Board representative for this project).

Members of the INFORMS initial certification task force included: Anne Robinson, Verizon Wireless; Jack Levis, UPS; Bill Klimack, Chevron; Terry Harrison, Penn State; Lisa Kart, Gartner; Dave Leonhardi, Boeing; Polly Mitchell-Guthrie, SAS; Scott Nestler, U.S. Army; Michael Rappa, North Carolina State University; Melissa Moore, INFORMS; and Gary Bennett, INFORMS.

The certification task force was charged with making strategic decisions about the program until the formal appointment of an official governing body. The task force held an in-person meeting at INFORMS headquarters in late November 2011 with Hamm to lay out the high-level strategic specifications and assumptions such as name of the credential, vision, mission, values, goals, governing body structure and roles, eligibility requirements, fees and assessment process.

Highlights of the task force’s strategic decisions included:

  • Name of certification program: CAP (Certified Analytics Professional).
  • Vision: Advancing the use of analytics by setting agreed upon standards for the profession.
  • Mission: Advance the analytics profession by providing a high-quality program of certification and promoting continuing competence for practitioners.
    Goals:
  1. Help organizations identify and develop qualified analytics professionals.
  2. Promote the use of best practices in the analytics profession.
  3. Contribute to the career success of analytics professionals.
  4. Improve the credibility of the analytics profession.
  5. Improve the visibility of the analytics profession.
  6. Increase the use of analytics in the private and public sectors.
  7. Improve the impact of analytics decision-making in organizations using these techniques.
  8. Achieve global recognition for the INFORMS analytics certification program.
    Governing body:

The certification program will be governed by the Analytics Certification Board (ACB.)

Composition of the ACB governing body will include: 11 total voting members plus ex-officio staff support (director/manager of certification position.) Three of the 11 voting positions will be INFORMS volunteer leadership including VP Sections/Societies, VP Practice and president of the Analytics Section.

The ACB will have the following administrative independence from INFORMS:

  • Final authority for all certification decisions without INFORMS Board approval.
  • Authority to develop and change certification process.
  • Authority to develop all policies and procedures for the certification program.
    Fees:

Initial certification fee will be $495 for members and $695 for non-members. There will be a $100 annual maintenance fee in lieu of a renewal fee at the end of the three-year certification cycle. Other revenues will come from an education provider program and sales of study guides.

    Business Plan Approved

After the Task Force meeting, Hamm developed a high-level draft business plan to guide the program over its first five years. This business plan draft included a five-year budget estimate. The business plan considered 2012 to be a strictly developmental year with the following tasks to be completed:

  • development and execution of the job task analysis which will inform the certification assessment instrument,
  • development of a detailed governance and management process,
  • development of formal policies and procedures,
  • development of a detailed marketing and communications plan,
  • development of marketing materials and website,
  • development of a candidate handbook, and
  • development, testing and roll out of accounting and IT systems.

2013 would mark the first year of offering an analytics certification examination to potential candidates. A director/manager of Certification would also be hired in 2013.

Winter 2012: Job Task Analysis Developed

A committee of analytics subject matter experts was chosen by the certification task force to help complete a job task analysis (JTA) that identifies all the major tasks performed by analytics professionals and the underlying knowledge and skills needed to successfully perform those tasks. When done correctly, the JTA links what is done on the job with what is measured by the certification examination. This linkage is necessary to establish the validity of the examination.

The subject matter experts included: Michael Rappa, North Carolina State University; Rita Sallam, Gartner; Bill Klimack, Chevron; Sam Savage, Stanford University; Jerry Oglesby, SA; Daymond Ling, CIBC; Mike Gorman, University of Dayton; Freeman Marvin, Innovation Decisions; Scott Nestler, U.S. Army; Glenn Wegryn, Procter & Gamble; Jeff Camm, University of Cincinnati; Arnie Greenland, IBM; Tim Rey, Dow Chemical; and Jack Levis, UPS.

The subject matter experts (SMEs) met near INFORMS headquarters on Jan. 9 and 10, 2012 for two days of deliberations led by Hamm and his colleague, psychometrician Dr. Gerald Rosen. The SMEs completed their work and drafted the JTA. The final document was circulated among the group for comments. This phase of the certification project was completed on time on Jan. 15, 2012 and on budget.

The certification effort moved forward with finalizing the JTA document, which serves as the basis for actual exam questions, and drafting a Policies and Procedures governance document (P&P) that would be brought before the Board at the summer 2012 Board meeting.

Job Task Analysis (JTA) Document

In March 2012, the completed JTA document, which serves as the basis for certification exam development and the test blue print, was subjected to wider comment and review by a representative sample of professionals to further validate its contents. The JTA was sent out to a random sample of INFORMS members and non-members on March 8 as part of a survey package consisting of a cover letter, set of instructions, definitions of terms, JTA and a response form. The survey closed two weeks later on March 23. A total of 220 analytics professionals answered the survey, which exceeded the original goal of 200. The 220 respondents were comprised of 57 INFORMS members and 163 non-members. The non-member respondents came from previous respondents of the certification feasibility study and registered subscribers to Analytics magazine. In general, the JTA was accepted by the wider analytics community as being representative of job activities and descriptive of the end-to-end analytics process.

Draft Governance P&P

The certification task force, which serves as a steering committee for this project and makes all high-level decisions until an Analytics Certification Board (ACB) is chosen, met in February to finalize the draft P&P document to govern activities of the ACB. The P&P document is posted in the governance pages of the INFORMS website as Section 17.

In general, the task force decided the ACB should remain part of INFORMS as more of a super committee than the equally prevalent practice of making the Board administratively independent from the Board of Directors. The American National Standards Institute (ANSI) provides certification accreditation according to the international standard (ISO/IEC 17024) and is generally ambivalent about the preferred model but stresses independence in decision-making, especially in the decision to award certification to any person. The Task Force decided to slow the transition to the ACB and to take up the matter again later in the year. The P&P provides detail on the preferred composition of the eventual ACB, criteria for inclusion, orientation, dismissal, and terms of office, elections and other administrative details.

Hamm also created a draft job description for the planned certification manager who was to come on board in 2013.

Summer 2012: Item Writing Commences

On June 7, 2012, staff, volunteers and consultants held a two-hour Webex training session for about 35 members and non-members who volunteered to begin writing draft certification exam questions (items). Bennett and Levis led the discussion that traced the origins of INFORMS getting into the analytics certification business, Hamm reviewed the Job Task Analysis on which questions would be based, and consulting psychometrician Dr. Gerald Rosen

provided detailed guidance on how to write good, defensible certification questions. The writers were given a goal to collectively write more than 200 questions by July 12. They met this goal. Rosen edited all completed questions to ensure proper form, fairness, validity and reliability of the questions.

More than 50 item writers were recruited to perform this task including: 1) some volunteers who helped draft the Job Task Analysis, 2) non-member respondents to the certification feasibility study, and 3) nominees from many subdivisions. With support from Barry Thomas, then VP of Sections and Societies at INFORMS, a call for nominations of item writers from all subdivisions was sent out in mid-May. To their great credit, subdivisions came through and supplied about 30 of the 50 recruits (see Table 1).

Table 1: Volunteer item writers.

  • Zahir Balaporia, Schneider National
  • Russell Barton, Penn State University
  • Carrie Beam, Carrie Beam Consulting
  • Peter Bell, University of Western Ontario
  • Rob Benson, Spinnaker
  • Margery Connor, Chevron
  • Tapa Das, University of South Florida
  • Gavin DeNyse, Hewlett-Packard
  • Pooja Dewan, BNSF
  • Rob Ende, Reanalyze
  • Venu Gopalakrishna, Remani
  • Arnie Greenland, IBM
  • Terry Harrison, Penn State University
  • Mary Helander, IBM
  • Shrikant Jarugumilli, BNSF
  • Stefan Karisch, Jeppesen, a Boeing Company
  • Lisa Kart, Gartner
  • Diego Klabjan, Northwestern University
  • Russ Labe, Bank of America
  • Jack Levis, UPS
  • Irv Lustig, IBM
  • Rakesh Kulkarni
  • Jennifer Leong, Booz Allen Hamilton
  • Fang Liu, Northwestern University
  • Heping Liu, Auburn University
  • Paul Messinger, University of Alberta
  • Polly Mitchell-Guthrie, SAS
  • Larry Myers
  • Ranga Nuggehalli, UPS
  • Scott Nestler, U.S. Army
  • Ivan Oliveira, SAS
  • Doug Samuelson, InfoLogix
  • Subhashish Samaddar, Georgia State University
  • Steve Sashihara, Princeton Consultants
  • Sam Savage, Stanford University
  • Mathew Schall, Catalysis
  • Durai Sundaramoorthi, Washington University-St. Louis
  • Lee Schruben, University of California-Berkeley
  • John Toczek, Aramark
  • Chen (Mavis) Wang, University of Wisconsin
  • Glenn Wegryn, Analytic Impact
  • Paul Wicker, Decision Strategies
  • Jim Williams, Land O’Lakes
  • Greg Zaric, University of Western Ontario
  • Bo Zhang, Penn State University

A subcommittee of the item writers, along with other subject matter experts who have been involved in the certification project, met for a two-day, in-person review meeting in Baltimore on July 30 and 31, 2012. The purpose of this meeting was to review, critique and re-write the draft exam questions as needed. By the end of the second day of deliberations, volunteers had: 1) considered each question, 2) re-written more than half, 3) deleted about 40 that were deemed to not be of high enough quality, were confusing or covered subjects not covered by the Job Task Analysis, and 4) written 10 new questions. The volunteers submitted a final total of 172 approved questions that staff and consultants entered into item bank software.

Members of the in-person review team included: Carrie Beam, Freeman Marvin, Arnie Greenland, Jennifer Leong, Fang Liu, Heping Liu, Scott Nestler, Venu Remani, Steve Sashihara, Sub Samaddar, Durai Sudaramoorthi and Bo Zhang.

Fall 2012: Certification Moves Ahead on Many Fronts

Rosen, the consulting psychometrician, drafted an initial examination form that was reviewed by the SMEs during two conference calls in late October/early November. After the calls, the consultants edited the examination and returned it to the SME committee for final approval.
At the November 2012 Board Meeting, the Board considered and passed P&P 17.1 establishing the ACB and how they would perform their duties. This P&P was set to go into effect after the first examination in April 2013. Until that time, the INFORMS Certification Task Force continued to operate developmental activities.

Bennett and Hamm collaborated on developing a certification marketing plan to guide the program through the rest of 2012 and throughout 2013. The marketing plan stresses that the certification is serious, rigorous, will be a career differentiator and was developed by the best minds in analytics.

Specific tactics would include the following: word of mouth/spokesmanship to engage likely early adopters of the program such as sister societies and large consulting firms; a bundled price with INFORMS’ large meetings; a full vetting of potential marketing partnerships; heavy leveraging of Analytics magazine as a carrier of certification news; various road shows to speak one-on-one with candidates such as attendees to Predictive Analytics World and other key conferences; and care and feeding of INFORMS members to ensure they understand the goals and will help us carry the word forward.

The certification website (http://www.informs.org/Build-Your-Career/Analytics-Certification) was established that provides an overview of the program. Sub-pages include applications, certification maintenance, reviewing the job task analysis, reviewing the overall candidate handbook and frequently asked questions.

In addition, “The Candidate Handbook” was produced that provides the overall centerpiece of the CAP program’s marketing effort that is the go-to source for candidates exploring sitting for the exam. It includes items such as sample questions, schedule of exams, application form, soft skills confirmation form, fee structure information and lists of references and available training programs. Download it from http://
www.informs.org/Build-Your-Career/Analytics-Certification/Candidate-Handbook.

Winter 2012-2013: It’s a ‘Go’

At the Fall 2012 Board Meeting, CAP officials reported the program had a bank of more than 170 questions or items that were heavily weighted to domains of practice that INFORMS members are quite familiar with: method selection and model building. A call was sent out to the best question writers from the first round and new question writers suggested by Anne Robinson, Terry Harrison and Jack Levis. This set of question writers supplied roughly 100 new questions that are spread more evenly across the seven domains of practice: business problem framing, analytics problem framing, data, method selection, model building, deployment and model life-cycle management.

These additional question writers are to be commended for their efforts: Carrie Beam, Rob Benson, Margery Connor, Tapas Das, Gavin DeNyce, Rob Ende, Mary Helander, Stefan Karisch and his staff at Jepessen, Fang Lui, Ivan Oliveira, Ranga Nuggehalli, John Toczek and Paul Wicker.

The review committee then classified, reviewed and edited each and every question. The review committee met on 15 different occasions between early November and mid-February (with still more sessions scheduled) to review each question. The committee did a heroic job of reaching consensus on the validity of each question. They set aside inappropriate questions, those with ambiguity or practices that were not industry-wide and re-wrote and corrected many others.

The review committee deserves much credit for their dedication to this project. Thanks go to: Terry Harrison, Jack Levis, Scott Nestler, Lisa Kart, Arnie Greenland, Sam Savage, Polly Mitchell-Guthrie and Sub Samaddar.

Go/No Go Meeting

A Go/No Go meeting was held at INFORMS headquarters on Jan. 21, 2013. Since the last Board meeting in October, a significant amount of work was completed on many fronts that allowed VP Practice Jack Levis and Past President Terry Harrison to make a “go” decision for the first exam to be administered at the INFORMS Conference on Business Analytics and Operations Research in April 2013.

In terms of quality control, the draft exam underwent an important quality control step in February – an actual self-administration of the test by eight trusted pilot testers and subject matter experts who were charged with providing their comments on the overall validity of the test.

The pilot testers included: Irv Lustig, Jim Williams, Diego Klabjan, Glenn Wegryn, Peter Bell, Zahir Balaporia, Russ Labe and Paul Messinger.

Results from the pilot testers showed that the exam was close to completion but could benefit from some tweaking and minor edits. This feedback was shared with the review committee that went about addressing all concerns.

In December 2012, INFORMS welcomed Louise Wehrle to the INFORMS staff as certification manager. Louise came to INFORMS with years of experience in certification and hit the ground running. She is well known in certification circles, knows and has worked with INFORMS’ two consultants on the project and is active in ANSI, the accrediting body for certification. Louise took over all day-to-day operation of the program.

Spring 2013: Certification Activities

The final administrative step was completion of the passing score study. This study set the program’s initial passing score and is a leading quality control step in certification development.

On March 9 of this year, 12 volunteers met in Baltimore for a one-day passing score workshop. Participants included four veterans of the certification development process to date – Jack Levis, Sub Samaddar, Scott Nestler and Arnie Greenland – along with eight newcomers to the process who were recruited from a WINFORMS meeting where Levis spoke on certification. The eight newcomers that provided a fresh perspective on the process included: Zach Waltz (IBM), Russell Barton (Penn State), Jonathan Taylor (University of Maryland), Steven Wilcox (SERCO), Mike Smith (ICFI), Bob Lucas (SAS), Cat Truxillo (SAS) and Ranga Nuggehalli (UPS).

The eight newcomers began the day by taking the exam under strict exam conditions. Those passing the exam based on the passing score were subsequently awarded the CAP™ designation. The first hurdle was passed when the 100-question exam was completed by all within the allotted three-hour time period. The first to finish completed the exam in about two hours. Next, Rosen trained all participants in the modified Angoff Method, which is a passing score determination methodology. The Angoff Method requires each participant to assign a percentage to each question that represents the likelihood that a certificant “just good enough to pass the exam” would get that question correct.

The training included practicing on the first 10 exam questions as a group. Ratings typically ranged from 40 percent to 90 percent. Those giving high and low ratings were asked to defend that rating. A lively discussion ensued. Once all became comfortable in the use of the Angoff Method they applied it to the remaining 90 questions independently. Rosen then took all the data provided by the 12 participants and computed the overall passing score (which is confidential information). The passing score fell in line with many other certifications that our consultants and staff had been involved in.

After the workshop, participants shared their overall thoughts about the exam with Levis and staff. Overall, most participants were complimentary that the exam was right on target or very close to target. They did point out some minor problems with a few questions. Those questions were subsequently adjusted by the exam review committee.

Readiness Review

INFORMS President Anne Robinson traveled to INFORMS headquarters on March 22 for a certification readiness review. The program passed with flying colors. The readiness review considered such items as pilot test completion, exam form readiness, passing score set, code of ethics completion and online application system working well.

The first formal administration of the examination took place on April 7 in San Antonio, Texas, prior to the 2013 INFORMS Conference on Business Analytics and Operations Research. Those who passed were announced on April 20. The initial group of certificants also includes those who developed the exam, those who validated it and the successful cut-score participants.

INFORMS received comments from the test takers that the exam fairly represented the end-to-end analytics process. It addresses the analytics process from business and analytics problem formation, through data challenges, to methodology and model selection, deployment of the selected model and life-cycle management.

Thirty-eight individuals comprised the first group of recipients to earn the CAP™ designation (see Table 2).

Table 2: First group to earn the CAP™ designation from INFORMS.

  • Alan L. Austin (Provo, Utah)
  • Zahir Balaporia (Green Bay, Wis.)
  • Russell R. Barton (University Park, Pa.)
  • Donald Buckshaw (Ellicott City, Md.)
  • Thomas W. Chestnutt (Encinitas, Calif.)
  • Thomas W. Fletcher (Buford, Ga.)
  • Arnold Greenland (Bethesda, Md.)
  • Terry P. Harrison (University Park, Pa.)
  • Steven Harrod (Dayton, Ohio)
  • Christopher Michael Heiden (Columbus, Ohio)
  • Amanda Humbert (Grove City, Ohio)
  • Lisa Kart (Austin, Texas)
  • Zain Khandwala (Columbus, Ohio)
  • Diego Klabjan (Evanston, Ill.)
  • Srikanth Krishnamurthy (Malden, Maine)
  • Russell P. Labe Jr. (Pennington, N.J.)
  • Robert M. Lucas (Cary, N.C.)
  • Irvin J. Lustig (Short Hills, N.J.)
  • Paul R. Messinger (Edmonton, Alberta, Canada)
  • Rami Musa (Claymont, Del.)
  • Scott Nestler (Carlisle Barracks, Pa.)
  • Hua Ni (Chantilly, Va.)
  • Elizabeth Nielsen (Durham, N.C.)
  • Ranganath S. Nuggehalli (Timonium, Md.)
  • Subhashish Samaddar (Atlanta, Ga.)
  • Sam L. Savage (Stanford, Calif.)
  • Satjeet Singh (Lewisville, Texas)
  • Michael Anthony Smith (Arlington, Va.)
  • Alan Taber (Cinnaminson, N.J.)
  • Jonathan Taylor (Greenbelt, Md.)
  • Deepak Tirumalasetty (Modesto, Calif.)
  • Catherine Truxillo (Raleigh, N.C.)
  • Zachary Waltz (Arlington, Va.)
  • Glenn Wegryn (Cincinnati, Ohio)
  • Steven P. Wilcox (Rockville, Md.)
  • James T. Williams (St. Paul, Minn.)
  • Kevin Windham (Fairfax, Va.)
  • Gigi Yuen-Reed (Tampa, Fla.).

INFORMS now turns its attention to future administrations of the exam. This includes continually monitoring the performance of each test item or question during administration, developing new items to ensure a rich bank to refresh the examinations, convening the Analytics Certification Board, and providing opportunities for certificants to maintain their certifications through continuing education in analytics topics. The ACB will preside over, among other activities, the creation of a study guide for personal or chapter use and the creation of guides to the Analytics Body of Knowledge, or A-BOK. If you wish to participate in exam development in some way, contact certification@informs.org. If you wish to sit for the exam, apply here: https://www.informs.org/Build-Your-Career/Analytics-Certification/Apply-for-Certification. All exams in 2013 are given as in-person, paper and pencil exams at one of the following locations:

  • June 23 preceding the INFORMS Healthcare Conference in Chicago
  • July 13 at the McLean, Va., office of Booz Allen Hamilton
  • Aug. 3 at the University of Washington-Bothell, Bellevue, Wash.
  • Oct. 3 at Predictive Analytics World Boston
  • Oct. 5 preceding the INFORMS Annual Meeting in Minneapolis

INFORMS certification leaders are contemplating offering computer based testing in 2014, which will allow the exam to go international. The future looks bright.

Gary Bennett (Gary.Bennett@INFORMS.ORG) is the director of marketing for INFORMS.

Jack Levis (jlevis@ups.com) is a division manager on the corporate engineering staff at UPS and vice president-practice activities for INFORMS.

Reference

  1. https://www.informs.org/ORMS-Today/Public-Articles/October-Volume-37-Number-5/INFORMS-News-INFORMS-to-Officially-Join-Analytics-Movement