Application of DMAIC Cycle and Modeling as Tools for Health Technology Assessment in a University Hospital

This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Associated Data

Supplementary Materials: Details on the multiple regression model assumptions check (as briefly summarized in Table 5 of the manuscript) are reported in the attached Supplementary Material file.

Data are not present in a publicly accessible repository. Data could be made available upon reasonable request to the authors.

Abstract

Background

The Health Technology Assessment (HTA) is used to evaluate health services, manage healthcare processes more efficiently, and compare medical technologies. The aim of this paper is to carry out an HTA study that compares two pharmacological therapies and provides the clinicians with two models to predict the length of hospital stay (LOS) of patients undergoing oral cavity cancer surgery on the bone tissue.

Methods

The six Sigma method was used as a tool of HTA; it is a technique of quality management and process improvement that combines the use of statistics with a five-step procedure: “Define, Measure, Analyze, Improve, Control” referred to in the acronym DMAIC. Subsequently, multiple linear regression has been used to create two models. Two groups of patients were analyzed: 45 were treated with ceftriaxone while 48 were treated with the combination of cefazolin and clindamycin.

Results

A reduction of the overall mean LOS of patients undergoing oral cavity cancer surgery on bone was observed of 40.9% in the group treated with ceftriaxone. Its reduction was observed in all the variables of the ceftriaxone group. The best results are obtained in younger patients (−54.1%) and in patients with low oral hygiene (−52.4%) treated. The regression results showed that the best LOS predictors for cefazolin/clindamycin are ASA score and flap while for ceftriaxone, in addition to these two, oral hygiene and lymphadenectomy are the best predictors. In addition, the adjusted R squared showed that the variables considered explain most of the variance of LOS.

Conclusion

SS methodology, used as an HTA tool, allowed us to understand the performance of the antibiotics and provided variables that mostly influence postoperative LOS. The obtained models can improve the outcome of patients, reducing the postoperative LOS and the relative costs, consequently increasing patient safety, and improving the quality of care provided.

1. Introduction

Healthcare seeks to give improvements in the prevention, control, and treatment of diseases, but at the same time, it also deals with complications, inefficiencies, and other problems that put patients' safety at risk. Therefore, it is necessary to monitor the health services provided by applying management methods and tools to control quality [1]. Nowadays, several methodologies and approaches are used in healthcare to help in the clinical decision-making process [2–8], to aid physicians in defining the diagnosis and prognosis of patients [9–11], and to analyze quality improvement in hospital processes [12, 13]. A useful methodology for these purposes is the Health Technology Assessment (HTA), a multidisciplinary process for medical-clinical, social, organizational, economic, technological, ethical, and legal implication analysis of health technology through the evaluation of efficiency, security, costs, and social and organizational impact [14, 15]. The technologies could be drugs, medical devices, vaccines, procedures, and, generally, all systems developed to solve a health problem and to improve the quality of life.

Parmar and Chan [16] used HTA methodology in urologic oncology. As a result of the rapid development of new cancer therapies, it is important to have a decision-making tool which leads to the choice of the right therapy in a short period of time. In this study, HTA was used as an approach that could help to guide value-based decision-making. An HTA model was developed for the evaluation of generic pharmaceutical products. This tool allows us to compare, both qualitatively and economically, equivalent drug preparation. HTA was employed to evaluate a new health technology for the thyroglobulin assay in patients with differentiated thyroid cancer. The authors used the Dynamic AHP as an HTA tool to reach the goal [17]; this paper proved also the utility of combining HTA with other managerial approaches.

Another promising tool to improve the quality of healthcare processes is Six Sigma (SS) [18–21]. Initially introduced in the manufacturing sector, today, it is widely developed in the health sector. SS relies on the “Define, Measure, Analyze, Improve, Control” cycle (DMAIC), which is a five-step procedure related to quality management and process improvement that exploits both statistical and managerial tools. Through this problem-solving strategy with a fixed structure, it is possible to analyze a process in order to improve its performance reducing the “natural variability” and carry out the “systematic control” of the critical variables to obtain a better result. The procedure is divided into the following phases: defining the project goals and customer (internal and external) requirements, measuring the process to determine current performance, analyzing and defining the root cause(s) of relevant defects, improving the process by eliminating defect root causes, and controlling future process performance. For the first time, Bill Smith developed this methodology in 1986 with the aim of reducing product or process defects that did not satisfy customers [18, 22]. DMAIC is then a framework used to enable the team to define and achieve set objectives [1, 23, 24].

From literature studies, it stands out the success that the strength of SS is founded not only in the manufacturing field but also in the health sector, where the SS DMAIC approach has been applied, for example, to improve first aid processes [25] and in the paramedical services [26]. Mahesh et al. [27] demonstrated how to reduce patients' waiting time to receive a specialist medical visit at the Out-Patient Department of Cardiology in a private hospital in the city of Bangalore, and El-Eid et al. [28] have confirmed SS as an efficient and effective management tool to improve the patient discharge process, reducing patient discharge time. As well, other studies confirmed the validity of the methodology [13, 29–33], also in combination with other methods such as the Agile [34]. Ricciardi et al. [12] analyzed the introduction of the Diagnostic Therapeutic Assistance Path (DTAP), employing Lean Thinking and SS methodology based on the DMAIC cycle. Furthermore, several studies show that the SS is often associated with Lean Thinking: this approach aims to improve services to meet customer needs by eliminating wastes and reducing costs [35–37]. The use of these methodologies has reported multiple benefits in healthcare; in fact, they have been used to improve clinical decision-making processes and to reduce the risk of healthcare-associated infections in surgery departments [38], while others have conducted studies to introduce prehospitalization to perform the necessary tests and examinations for hip and knee prosthetic surgery [29, 39].

The problem of healthcare infections is of great interest in many surgery departments, and it is an indicator of hospital efficiency, safety, and quality. Scotton et al. [40] conducted a study whose purpose was to analyze infections in patients after Salvage Laryngectomy (SL) and review the potential impact of the antibiotic prophylaxis adopted. The results showed that infection rates after SL were high, and univariate analysis demonstrated risk variables that had a significant correlation with infection, so the antibiotic regimen is probably ineffective. Other authors [41–48] presented an overview of current evidence-based best practices in the use of prophylactic antibiotics in head and neck cancer surgery; indeed, this type of patient is at high risk of developing complications after surgery. Thus, they reported that prophylactic antibiotics helped significantly reduce the risk of infection [49]. However, short four-dose antibiotic regimens for 24 hours are as effective as prolonged cycles, regardless of the complexity of the procedure [50–53]. In the same framework, the research of Egan et al. [54] discusses the use of the SS focusing on therapy with antimicrobial gentamicin, which requires good practice in selecting the dose and monitoring serum levels. They found a new dosage with a standardized sampling, a monitoring program, and a new timing of drug delivery that maximized local capacities. In light of the above-mentioned studies, it emerges the importance of choosing correct prophylactic antibiotics to manage patients appropriately after surgical interventions.

To this aim, in our recent study [55], SS was employed to compare the use of antibiotics in patients undergoing oral cancer surgery on bone tissue. Starting from the previous promising results, in this work, two antibiotics, ceftriaxone and the combination of cefazolin and clindamycin, are compared in order to understand which one reduces the postoperative length of hospital stay (LOS) for patients undergoing oral cavity cancer surgery on the bone tissue. In this study, it is taken into consideration the clinical factor because the two antibiotics are quite similar from a safety, legal, ethical, economic, and technological point of view. Six Sigma (SS) methodology is applied as a tool of HTA in order to achieve the aim. SS was used to analyze the influence of some clinical variables (ASA score, age, gender, oral hygiene, diabetes, and cardiovascular diseases) on the Critical to Quality (CTQ) (postoperative LOS). Patients' postoperative LOS can be described as the duration of time after a patient's surgery until the day of discharge.

The novelty of this new study is the use of the DMAIC cycle as an HTA tool including a modeling phase. This would enable healthcare providers to understand the performance of antibiotics, improving patients' outcomes, reducing postoperative LOS and related costs, consequently, increasing patient safety, and improving the quality of care provided. After applying DMAIC, a modeling study was conducted through a multinomial linear regression; in particular, it was applied to obtain two models capable of predicting postoperative LOS for each antibiotic. In order to do this, we included the surgical variables that were considered in the previous study [55].

2. Materials and Statistical Tools

SS and subsequently the modeling phase were used to implement the HTA methodology. In detail, deploying the DMAIC cycle, characteristic of SS, means developing five phases:

The Define phase identifies the customers and the objectives to be reached will be established [27] allowing a team to identify the problem

The Measure phase defines the main characteristics of the process and the parameters that will lead to improvement [56]

The Analyze phase is used to understand the influence of the collected variables on the CTQ or to evaluate the data collected in the previous phases of the study using various analytical tools available such as regression analysis, fishbone diagram, tree diagrams, and brainstorming

The Improve phase employs all the previous analyses to design changes in a process and to improve the performance, i.e., introducing a new antibiotic protocol

The Control phase is employed to monitor the whole process and, in this research, to compare the performance of the drugs

SS led the way for the development of the modeling phase, providing us with information about all the variables. Modeling allowed us to enrich the univariate analysis with a multivariate one and to implement a tool able to predict the postoperative LOS for each patient. These models will be very useful for both ward management and hospital management. Predicting the LOS of a patient determines a more efficient hospital bed organization, a better management of nurses and doctors on duty, and lastly, a cost reduction for hospitals. Thus, combining SS and modeling could be considered a valuable tool for HTA methodology.

In conclusion, the purpose of this paper is to assess the performance of two antibiotics, cefazolin plus clindamycin [57, 58] and ceftriaxone [59], through an HTA by using SS and modeling as a tool in the framework of oral cavity cancer surgery on bone tissues.

2.1. The Clinical Case Study

In this study, two groups of patients with oral cancer starting from the bone were analyzed: the first one was treated with ceftriaxone between 2006 and 2011, while the second one was treated with cefazolin and clindamycin between 2011 and 2019. The cefazolin group consisted of 54 patients, while the other by 51 patients. Oral cancer is the sixth most common cancer in the world [60] but the ones starting from the jaws are rare. The majority of the oral cancers affecting the bone derives from the epithelial quote of the oral mucosa, but there are also cancers that originally start from the bones, which are rare. Sarcomas are very rare tumors in the head and neck district, osteosarcoma being the most common of them [61]. They represent 1% of all the malignancies affecting the head and neck [62]. The incidence of sarcomas starting from the mandibles ranges from 4% to 10% [63]. In this study, we decided to analyze also those patients affected by ameloblastomas, which is not actually a malignant neoplasm. This choice is due to the fact that in the case of big ameloblastomas affecting the jaws, a big removal of tissue and reconstruction with the same surgical techniques used for patients affected by oral bone cancers are often required. The data was taken from printed medical records. Statistical tests, useful for analyses, were carried out with IBM SPSS.

For the collection of data, some inclusion and exclusion criteria were taken into consideration:

All patients were included without exclusion due to medical history (gender, age, cardiovascular diseases, diabetes, oral hygiene, American Society of Anaesthesiologists (ASA) Score)

Patients with cancers starting from the bones or starting from the oral mucosa and then affecting the bone were included. We also included patients with ameloblastomas because of their osteolytic patterns

Patients with a change of the antibiotic therapy during their recovery, because no evidence of efficacy, were not included in the analysis, but their number was recorded as it is a qualitative indicator of treatment failure

As regards the Unit of Maxillofacial Surgery, the ward consists of 9 rooms with 22 beds for the patients and some more rooms for surgeons and nurses. The Operatory Block of the Department disposes of two operating rooms.

Oncological maxillofacial surgery is a branch of maxillofacial surgery which deals with the surgical approach to head and neck malignancies and the reconstruction of the lost tissues [64].

When no allergy was described, from 2006 to 2011, a postoperative antibiotic protocol with ceftriaxone was used. Since 2011, there has been a shift to the use of the association of cefazolin plus clindamycin as postoperative antibiotic prophylaxis.

2.2. The Development of the Six Sigma: The Define Phase

The purpose of the “Define” phase is to define a multidisciplinary workgroup and to divide the tasks for the analysis. The team consists of clinicians from the Maxillofacial Department of the University Hospital “Federico II” of Naples, an economist, and biomedical engineers with experience in health management. The team was responsible for collecting and analyzing data of patients with oral cavity cancer considering the influence of some variables. The sample and the leader supervised and coordinated the study and interpretation of the data. A project diagram was created to define the problem to be solved:

• (i) Project Title. Health Technology Assessment between two antibiotics in the context of Maxillofacial Surgery
• (ii) Question. Investigation of the best antibiotics in the analyzed context
• (iii) Critical to Quality. Postoperative LOS
• (iv) Target. Realize corrective measures to reduce the CTQs
• (v) Deliverables. The performance of cefazolin/clindamycin and ceftriaxone, the outcome of patients, reducing postoperative LOS, and the related costs
• (vi) Timeline:

2.3. Dataset Description: The Measure Phase

The data collected from the medical records at the Department of Maxillofacial Surgery were selected according to the inclusion and exclusion criteria. After applying the inclusion and exclusion criteria, the first sample of data concerned patients treated with ceftriaxone from 2006 to 2011 (45 patients), and the other sample of data (48 patients) was referred to patients treated with cefazolin and clindamycin from 2011 to 2019. The variables used to compare the two antibiotics were

Other variables were analyzed through univariate analysis in a previous study [55]; thus, they were included only in the modeling phase. Descriptive characteristics of the dataset were carried out for the postoperative LOS variables: the results for cefazolin/clindamycin were, respectively, an average of 16.51 days and a variance of 62.21. Instead, the results for ceftriaxone were an average of 9.75 days and a variance of 66.81.

We drew a histogram ( Figure 1 ) showing the mean postoperative LOS of patients, measured in days, submitted to the administration of cefazolin/clindamycin according to each variable. The highest average LOS is for patients with a high ASA score, while the lowest is for patients with a low ASA score.

Mean postoperative LOS for each mode of variables regarding cefazolin/clindamycin.

Figure 2 shows the distribution of mean postoperative LOS of patients who used ceftriaxone. Patients below the age of 51 have the highest mean LOS, whereas those without cardiovascular disease have the lowest mean LOS.

Mean postoperative LOS for each mode of variables regarding ceftriaxone.

2.4. Statistical Analysis: The Analyze Phase

In Figure 3 , patients' pathway is shown from the arrival at the hospital to the discharge. They arrived at the hospital; then, if they receive a previous prehospitalization, they undergo surgery directly; otherwise, they are subjected to preoperative activities before surgery. Finally, if there are complications after the surgery, the patient undergoes postoperative activities; otherwise, they will be discharged after fewer days.

The flowchart of the hospitalization process for patients undergoing oncologic surgery at the Maxillofacial Department of the University Hospital of Naples “Federico II.”

A Kolmogorov–Smirnov test showed a p value lower than 0.0001. In order to understand the variables that could influence the postoperative LOS in the ceftriaxone group, nonparametric tests were employed: Mann–Whitney and Kruskal–Wallis (only for age). In this case, some significant p values were found for age and ASA score while the p value of cardiovascular disease was almost significant (p value = 0.066) ( Table 1 ).

Table 1

The analysis of potential factors influencing postoperative LOS for the “ceftriaxone” group.

A box diagram was developed and is shown in Figure 4 , which clearly highlights the decrease in the ceftriaxone group of LOS, measured in days.

Boxplot of the mean postoperative LOS for “cefazolin/clindamycin” and “ceftriaxone” groups.

The Control phase allowed us to monitor and guarantee the sustainability of the long-term continuous improvement of the performance. Thus, the team identified the following actions:

After analyzing the data according to the DMAIC cycle, the modeling phase started by implementing the multiple linear regression. It is also known simply as multiple regression and is a statistical technique that uses several explanatory variables to predict the outcome of a response variable. The goal of multiple linear regression is to model the linear relationship between the explanatory (independent) variables and response (dependent) variables. In other words, multiple regression is the extension of ordinary least-squares (OLS) regression that involves more than one explanatory variable.

In this study, it was used to obtain a model capable of predicting the postoperative LOS for each patient undergoing oral cavity cancer surgery on the bone. In order to obtain the best models, we considered also the surgical variables that were studied in a previous research on the same topic [55]. Therefore, the considered variables in order to implement the model were 11: gender, age, ASA score, the quality of oral hygiene, diabetes, cardiovascular diseases, tracheotomy, lymphadenectomy, infections, dehiscence, and flap.

3. Results

3.1. Statistical Analysis for Cefazolin plus Clindamycin

The Kolmogorov–Smirnov test was applied to investigate the distribution of the postoperative LOS data regarding cefazolin/clindamycin; a p value of 0.200 indicated a normality distribution. Thus, to investigate the variables potentially influencing postoperative LOS, t-test and ANOVA were employed. The results are represented in Table 2 . No significance was found in the tests, but the difference between postoperative LOS in each category gave insights about a potential influence in many of the variables; the ASA score was almost significant.

Table 2

The analysis of potential factors influencing postoperative LOS for the “cefazolin/clindamycin” group.