by Antonio Rodriguez (N.I.H.)
As presented at the 2018 Winter Simulation Conference
Abstract
Even the best scientific minds cannot repeatedly produce desired results when working in sub-optimal systems. Complex enterprises are difficult to understand and manage. Cause-and-effect relationships are often separated in time and space, making real improvements challenging. To understand how complex systems work it is essential that we employ tools that accurately map and quantify the dynamics that drive results. Computer modeling and simulation (CMAS) is a valuable design, planning, management, and overall analytical decision-support tool to achieve effective and efficient results. CMAS could become a ubiquitous tool in the lengthy and complex environment of the clinical research (CR) enterprise. Without the comprehensive understanding gained when applying CMAS, organizations may continue to be overwhelmed by problems such as unnecessary bottlenecks, high costs, low productivity, and the inability of retaining critical staff. The approach explained here may complement or even replace traditional methods when organizations pursue greater enterprise capability.
Introduction
The continued development of computer modeling and simulation (CMAS) products have facilitated new capabilities for understanding interactions among multiple factors across producing spectrums. The use of computer models provide access to new understanding unavailable in the physical world—an understanding that leads to improvements required to achieve and sustain efficient systems where contributors can proudly realize their objectives.
The clinical research (CR) field is complex and could benefit from the application of CMAS to drive new understanding and improvement in its diverse enterprises over time. CMAS tools today are more robust and user friendly than ever, and include accurate animations to better communicate with all stakeholders. Simulations are more realistic when they include quality, scheduling, resource utilization, unexpected events, and constraints in the model. These topics will be addressed in the discussion section and will be integrated in the proposed approach to use CMAS in the CR enterprise
Discussion
In a tight fiscal or highly competitive environment, efforts to accomplish critical enterprise results could be guided using methods and tools that take into account variability in 1) the demand for outputs, 2) the capacity and constraints of its processes, 3) production schedules, and 4) unexpected events. Efficient and effective end-to-end processes result mainly when a pre-defined sequence of actions is well-balanced in terms of capacity and demand. To achieve a sustainable balance of end-to-end processes, the study and improvement effort may also need to include critical support functions.
Inadequate capacity throughout the sequence of activities and decisions will cause unnecessary delays and quality problems in some areas, only to be compounded with costly overcapacity in other areas. The cumulative effect of these prevalent unbalanced systems produces unnecessarily high-cost products and services, and frustrated stakeholders.
Like the private sector, federal government is frequently challenged by limits in budget and human resources. My experience in the pharmaceutical and electronics industries is that more effort was placed on improving operations on the manufacturing side than I have witnessed in operations, research and development, and support service areas in federal government agencies. This was done, for example, 1) through changing facility layouts; 2) changing inspection methodologies; 3) applying statistical methods to demonstrate that changes made to raw materials, processes, or equipment accomplished desired results; and 4) applying new technologies to fundamentally change established processes.
However, even in manufacturing facilities the problem of unbalanced processes or activities caused the cost of converting inputs to outputs (e.g., production planning, purchasing raw materials, production scheduling, assembly, filling, testing, sterilizing, packing, distribution) to be higher than necessary. Staff has to be hired, trained, and provided with an adequate work environment; equipment has to be purchased, calibrated, maintained, and replaced; production has to be planned; new products have to be pilot tested; facilities have to be secured, sterilized, operated, maintained, renovated, and in some cases expanded; products have to be packed, distributed, or scraped and disposed of; electricity, fuel, and water have to be purchased and in some cases even generated; and regulatory requirements have to be met and documented. The aforementioned activities are performed regularly and describe what may need to be done to produce outputs.
Knowing what is needed to produce a product is necessary, but not sufficient, to achieve effective and efficient end-to-end processes. Multiple bottlenecks will occur unless a balance between capacity and demand in most activities is carefully managed. This does not mean a one-to-one ratio between capacity and demand, it means sufficient capacity to meet the demand without creating bottlenecks larger than what the customers or participants are willing to tolerate—this is a moving target.
Consistent efficiency and effectiveness are a function of multiple factors such as balanced capacity and demand in the sequence of activities and decisions (i.e.
Production schedule production schedule
CMAS is an excellent tool to model different scheduling scenarios quickly to inform decision-makers on the likelihood of success (Pegden 2017). Different schedules for the same products will require resources at different times. Experimenting with the availability of limited resources in a computer model and seeing the impact on various scheduling scenarios helps decision-makers select more predictable scenarios. The complexity rapidly increases as the quantity and mix of products increase. Complex schedules are subject to change. Being able to anticipate the impact these changes will have on the availability of resources and the delivery of materials is essential to minimize scheduling disruptions and effectively execute production plans. Schedule changes may impact the balance needed for efficiency by altering available capacity or by lowering expected internal demand throughout the process. Schedules could be monitored periodically to determine if changes will impact resources needed to reduce bottlenecks, or even better, achieve bottleneckfree processes.
Quality
Many have attempted to define quality. It is difficult to do. Juran mentioned “fitness for use” (Juran 1989, p. 15) as a definition that received wide acceptance. He also stated that fitness for use “does not provide the depth needed by managers to choose courses of action” (Juran 1989, p. 15). W. Edwards Deming stated that “quality can be defined only in terms of the agent” (Deming 1986, p. 168). Different agents may view or define quality in different ways. Therefore, any serious attempt to provide quality requires operationally defined terms agreed on by all significant agents or stakeholders (Ackoff 1994).
Shewhart examined the subject of quality from different perspectives (Shewhart 1931). He addressed, for example, the popular conception of quality—quality of product, quality of process, and quality as a relationship. The subject is complex, but Deming summarized the various constructs addressed by Shewhart as “the difficulty of defining quality is to translate future needs of the user into measurable characteristics, so that a product can be designed and turned out to give satisfaction at a price that the user will pay” (Deming 1986, p. 169).
Process quality impacts production capacity. Rework (i.e., making changes to something already done) and scrap (i.e., having to throw away work done and start over again) diminish available production capacity and add unnecessary cost to the final product. If a finished product in the hands of a consumer adds value or satisfaction for the price paid, it should be considered efficient and effective. However, the performance and cost of competing products or providers of the same product may change the final verdict of the customer or sponsor. Rework and scrap levels, and their impact on capacity and demand throughout processes, can be easily assessed using CMAS.
Without consistent quality upstream, enterprise resources will be consumed without producing the expected demand in an organization’s domain of the process. Also, rework and scrap in one step of the process will waste some available capacity and deliver an inadequate amount of output downstream. This inefficiency unnecessarily increases the cost of work already performed. Therefore, every step in the process must perform well on a consistent basis if efficiency and effectiveness are to be sustained
Clinical research
The process of “making important discoveries that improve health and save lives” (NIH Who We Are 2017) is lengthy and complex. The National Institutes of Health (NIH) director, Francis Collins, described it as follows:
The translation of basic biological discoveries into clinical applications that improve human health is an intricate process that involves a series of complex steps: the discovery of basic information about the pathogenesis of a disease; an assessment of whether that information has the potential to lead to a clinical advance; development of candidate diagnostics, devices, or therapeutics; optimization of the candidates in preclinical settings; regulatory assessment of the data to determine the potential for human use; testing in human clinical trials; application for approval for widespread clinical use; and, ultimately, the assessment of approved diagnostics, devices, and therapeutics during widespread use in real-world settings (Collins 2011, p. 1).
Since this overall translational effort is a complicated progression, it may be helpful to start by focusing on a critical and sometimes limiting portion of this whole effort: clinical research. Information gained by following the approach depicted in this document could be helpful in guiding efforts upstream and downstream. “Through clinical research, clinician-investigators translate laboratory discoveries into better treatments, therapies and interventions to improve the nation's health” (NIH Clinical Center 2018). The CR enterprise is expected to accomplish patient safety and research goals with fewer bottlenecks and more agility.
As applied to CR, what is the meaning of quality? What is a quality research protocol (final product)? What is a quality research process (all the steps and decisions that contribute to the final product)? Does the meaning of quality change throughout the process? To what extent can quality be achieved without operational definitions? (Lewis 1929). Without operationally defining the outputs of the different steps in a process, multiple agents will not have shared knowledge and actionable definitions to achieve optimal enterprise results (Deming 1986).
The NIH is expected to make “important discoveries that improve health and save lives” (NIH Who We Are 2018) Part of the NIH mission is met through CR. “Clinician–investigators translate laboratory discoveries into better treatments, therapies and interventions to improve the nation’s health” (NIH Clinical Center 2018).
What are the results expected from the CR enterprise? How many completed protocols should be expected and when? How many protocols will be in production or active every year? What are the resources needed for each one? Should a complete support system be provided to every Principal Investigator (PI), to a group of PIs, or should they be centrally provided for all? If a partial support portfolio is going to be provided, what should it include and in what quantities? How long should a research effort be funded and how should it be supported to provide for a fair likelihood of success? What should be the yield of approved protocols that reach the treatment phase? What impact does an approved protocol have on the retention of investigators and support staff, if it is canceled due to unanticipated constraints?
As in manufacturing, the CR support system is also complex. It includes staff from a myriad of professions that must be recruited and retained such as credentialed physicians, dentists, PhD researchers, research nurses, nurse practitioners, study coordinators, bioethics specialists, protocol navigators, fellows, QA/improvement specialists, contract research organization representatives, pharmacists, dietitians, medical technologists, imaging technologists, therapists, regulatory affairs staff, patient representatives, medical records and medical supply staff, patient recruitment and patient services personnel, radiation safety officers, human resources staff, and other administrative and acquisition professionals. Determining the level of staff support needed, when and where, can be better determined by using CMAS than by using other available practices such as adjusting the level of resources annually based on the previous year’s results. CMAS is also a more robust practice than making resource allocation decisions based on benchmarking studies, or industry standards developed for other institutions, or organizational history.
Professionals depend upon, for example, laboratory and hospital equipment, libraries, information technology, websites, medicines, transportation services, language interpreters, pharmacies, amenities, visa services for international patients and caregivers, background materials for reviewers, climate control systems, aseptic processing areas, protective apparel, microbiological agents, and radioactive materials. Therefore, planners and managers in the CR enterprise should address many of the aforementioned questions to adequately source them on an on-going basis.
What and how many of each type of resource is needed throughout the steps of the process to minimize unwanted bottlenecks? Identifying, prioritizing, and investing in some of these resources that seem scarce, without examining the impact on the whole process, may only shift bottlenecks. It may bring temporary relief to a part of the process, while making the whole less efficient, or even worse, less effective.
In addition, the CR system depends on scientists with outstanding credentials from outside the NIH who evaluate the quality of research and advise scientific directors, institute and center directors, the NIH deputy director for Intramural Research, and the NIH director. The contribution that they make happens before and during treatment of patients participating in protocols. Scientists also have to be identified and recruited; their participation requires a great deal of coordination. Their recruitment and coordination effort is another part of the overall process that needs to be orchestrated to avoid bottlenecks in the review and approval of protocols
Optimizing every function independently seldom, if ever, achieves overall net optimization (Deming 2000). It usually results in the creation of many bottlenecks and frustration throughout the overall process. Understanding and managing the whole CR process as a system of interdependent components is essential to maximize efficiency, while achieving the effectiveness desired. CMAS provides an excellent and efficient alternative to view and experiment with the whole process (Deming 2000).
Approach
What follows is not a cookbook, but a methodology to better understand the CR enterprise as a system and to determine the necessary resources to achieve sustained net efficiency and effectiveness.
First CR needs to be recognized as a sequence of interdependent steps and decisions and not as a sum of independent activities. The following depiction of a high-level CR process provides a framework for the discussion on how to apply CMAS to improve the end-to-end process (Figure. 1). The steps used here are: protocol design and development, protocol review, protocol approval, patient recruitment, study conduct and treatment, data collection, analysis, and report results (Dagalar 2016).
Figure 1: CR enterprise as a system
In an actual model, more granularity would be needed to capture the different process steps of the various types of research protocols (e.g., interventional/clinical trials, natural history, training, screening, and phases of clinical trials [phase I – phase IV]) (NIH Clinical Trials and You. The Basics. 2018).
Once the sequence of steps is known, we will need the production plan (units or protocols) that will be implemented throughout the whole process for a specific period of time, as well as the production capacity of each step (Figure. 2). Historical demand data for each type of protocol, and the different phases of clinical trials, are needed to determine the quantity of each resource (e.g., staff, specialized equipment, materials, drugs, operating rooms) in every step (Figure. 3). In the absence of sufficient historical data, pseudorandom values can be generated from appropriate probability distributions, after estimating parameters using techniques such as the Monte Carlo method (MCM) (Simio 2017). Some process steps will need to be broken out in greater detail (e.g., treatment) to make a useful CR simulation model.
Figure 2: Production plan view
As an example, assume that the goal is to work with 900 protocols in any given year. Some new trials will start, while others will already be in progress at different steps of the process. If the overall process is to run efficiently, then adequate capacity to process the demand at every step is needed. The depiction in Figure 2 is an over simplification of the overall process, and includes notional numbers for illustration purposes. If these numbers were representing the number of protocols currently distributed throughout the overall process, then CMAS could be used to determine the expected bottlenecks given current available capacity of all relevant resources.
Action plans can be put in place to reduce or eliminate these bottlenecks without creating additional ones elsewhere in the process. On the other hand, the simulation model could be used to better match capacity for a given level of demand throughout the whole process to reduce bottlenecks and overcapacity in the near future. Part of the process of matching capacity and demand would have to include an inventory of the tasks involved per staff category in every significant step, and the amount of time it would take to complete tasks. For example, what are the tasks of the Lead Associate Investigator during the protocol review step? How much time is needed to complete them? Data could be obtained from available systems, or generated using the MCM to account for random variation in completing tasks.
Figure 3: Capacity, demand, and bottlenecks.
If there are meaningful differences in the productivity among staff members in any given task, those differences can also be included in the simulation model. Differences in the processing time between similar equipment and availability, as well as data on the accessibility of essential materials, agents, or drugs should also be entered in the simulation model. The inclusion of all significant variables and their data are necessary for a robust analysis of the overall process. If a limited analysis is conducted, it may result in an inaccurate model, one that fosters many misleading decisions and actions. For example, having the right number and type of professional staff can be of limited value if bottlenecks are created by equipment, drugs, or other resource constraints not included in a model.
The uncertainty inherent in recruiting sufficient patients should also be factored in the CMAS since patient availability can be a significant source of bottlenecks (Mahon et al. 2016). Insufficient participant numbers can be estimated by reviewing historical data and assigning yields to the relevant steps of the different protocol types and phases, or using the MCM to generate them
Determining the resources needed to meet a given demand is relatively easy in a world of unlimited resources. Unfortunately, the challenge is significantly more difficult since many critical resources are becoming scarce. The real problem is to balance a number of moving parts that includes a high mix of protocol demand in a given time period while tackling limitations such as operating rooms, specialized staff, special equipment, and available research participants. CMAS is a tool that is versatile and can be used to assist decision makers as they reconcile these aspects by running multiple scenarios to maximize throughputs based on limited resources, while avoiding unnecessary bottlenecks throughout the process.
Clinical Research hospitals may have multiple specialized research sources (see figure 4 for a notional perspective) feeding into and receiving from the treatment/data collection phase (i.e., research hospital). These sources will differ in how they break down their CR process steps prior to and after the treatment phase; however, for a smooth overall process to take place, each specialized source (specialized internal supplier) could also apply the approach explained in this paper
Suppliers of specialized protocols should provide periodic information to help improve the production (i.e., treatment) planning and scheduling of the CR hospital. On an emergency basis, priorities may have to be established throughout the CR process based on, for example, overall critical need, rare research opportunity, or resource availability. However, on-going prioritization should take into consideration multiple factors simultaneously to avoid sub-optimizing the results of the overall CR enterprise (Deming 2000; Christopher 1993).
Figure 4: Clinical research with multiple specialized sources.
CMAS can account for important performance indicators and display them in every process step of interest. For example, some important variables to track may be 1) productivity loss due to quality problems such as adverse drug responses; 2) resource utilization and idle time of specialized equipment; 3) process output such as treatments completed, treatment delays caused by facility problems, or patient waiting time due to new drug availability, and 4) cumulative delay time per protocol.
Once the model is tested and demonstrates that it is valid and reliable, the modeling team can run a number of experiments in the CMAS (Simio 2018) environment to assess, for example, the impact of greater or less variability in demand, number of unexpected events, availability of resources, schedule, and the number of changes to approved protocols. This experimentation will facilitate new learning and innovation in the process of conducting CR (Porter and Teisberg 2006).
Additionally, a large number of replications (Simio 2018) can be executed in a short period of time to increase the reliability of the results (i.e., reduce the confidence interval of the different statistical variables). When all is done, the CMAS will prove to be an invaluable tool in determining the best mix and quantity of resources needed to achieve end-to-end efficient and effective CR.
The approach discussed here is thorough, complex, and somewhat time consuming. However, it should be conducted in several phases, adding one or a few layers of complexity at a time (Pegden 2017). The knowledge and technology are available today to implement cost reduction and accelerate productivity not only in the CR environment, but in all research and technology transfer, and in providing treatment to patients.
Conclusion
Traditional improvement efforts are usually focused on components of systems, without assessing the impact on the whole system itself. Unfortunately, at best, the results are usually short-lived or negligible, and, at worst, they make the bottom line significantly worse. The CR enterprise is not immune to the limitations of short-term and isolated interventions. CMAS is a versatile predictive analytical tool that has been used in several industries for many years. It provides for a comprehensive understanding of causeand-effect relationships throughout the whole system. Adopting the use of CMAS in the CR enterprise can enhance decision-making and improve overall results. The implications are far reaching and include, but are not limited to, reduction of stakeholder frustration and improvement in critical CR staff retention, reduction of overall costs, attraction of new scientists, and more importantly, achievement of faster translation of “discovery into better treatments, therapies, and interventions to improve the nation’s health” (NIH Clinical Center 2018).
Author biography
ANTONIO R. RODRIGUEZ is the director of the Office of Quality Management. He graduated from the Engineering School of the University of Puerto Rico in 1984. In December of 2003, he received the Department of Defense Chief Information Officer Certificate from the National Defense University in Washington, DC. He also completed the Leadership for a Democratic Society program from The Federal Executive Institute in September of 2011. Since the 1980’s, he has led many organizational improvement initiatives in both the private and public sectors. He has worked for Intel, Baxter International, Navy Personnel Research and Development Center, Under Secretary of the Navy’s TQL Office and Business Performance Office, Department of the Navy Office of the CIO, the California School of Professional Psychology (CSPP), and the National Institutes of Health. He helped those organizations make significant improvements through the application of contemporary management and leadership improvement efforts such as the Toyota Production System, Balanced Scorecard, Continual Improvement, Computer Modeling and Simulation, Time Series Analysis of Product Demand, and Group Decision Making using Expert Choice. As an Adjunct Professor at CSPP, he taught Process and Organizational Analysis II to Ph.D. candidates. He has won numerous awards for his leadership in transforming the way federal government organizations plan and improve their performance. His email address is rodrigan@ors.od.nih.gov.