Annals of Thoracic Medicine Official publication of the Saudi Thoracic Society, affiliated to King Saud University
 
Search Ahead of print Current Issue Archives Instructions Subscribe e-Alerts Login 
Home Email this article link Print this article Bookmark this page Decrease font size Default font size Increase font size


 
ORIGINAL ARTICLE
Year : 2009  |  Volume : 4  |  Issue : 3  |  Page : 128-132
Evaluation of three scales of dyspnea in chronic obstructive pulmonary disease


Department of Cardiorespiratory Physiology, Clinical Research Centre, Vallabhbhai Patel Chest Institute, University of Delhi, Delhi, India

Date of Submission27-Dec-2008
Date of Acceptance20-Mar-2009
Date of Web Publication10-Jul-2009

Correspondence Address:
S K Chhabra
Department of Cardiorespiratory Physiology, Vallabhbhai Patel Chest Institute, University of Delhi, Delhi - 110 007
India
Login to access the Email id


DOI: 10.4103/1817-1737.53351

PMID: 19641643

Rights and Permissions

   Abstract 

Background: The Modified Medical Research Council (MMRC) scale, baseline dyspnea index (BDI) and the oxygen cost diagram (OCD) are widely used tools for evaluation of limitation of activities due to dyspnea in patients with chronic obstructive pulmonary disease (COPD). There is, however, limited information on how these relate with each other and with multiple parameters of physiological impairment.
Objectives:
To study the interrelationships between MMRC, BDI and OCD scales of dyspnea and their correlation with multiple measures of physiological impairment.
Materials and Methods: A retrospective analysis of pooled data of 88 male patients with COPD (GOLD stages II, III and IV) was carried out. Dyspnea was evaluated using the MMRC, BDI and OCD scales. Physiological impairment was assessed by spirometry (FVC % predicted and FEV 1 % predicted), arterial blood gas (ABG) analysis and measurement of the 6-min walk distance (6MWD).
Results: The interrelationships between MMRC, BDI and OCD scales were moderately strong. The BDI and OCD scores had strong correlations with ABG abnormalities, weak correlations with spirometric parameters but none with 6MWD. MMRC grades were significantly associated with BDI and OCD scores but did not show clear associations with spirometric parameters, ABG abnormalities and 6MWD.
Conclusions: The MMRC grades of dyspnea and the BDI and OCD scales are moderately interrelated. While the BDI and OCD scales have significant associations with parameters of physiological impairment, the MMRC scale does not.


Keywords: Baseline dyspnoea index, chronic obstructive pulmonary disease, Modified Medical Research Council scale, oxygen cost diagram


How to cite this article:
Chhabra S K, Gupta A K, Khuma M Z. Evaluation of three scales of dyspnea in chronic obstructive pulmonary disease. Ann Thorac Med 2009;4:128-32

How to cite this URL:
Chhabra S K, Gupta A K, Khuma M Z. Evaluation of three scales of dyspnea in chronic obstructive pulmonary disease. Ann Thorac Med [serial online] 2009 [cited 2019 Jul 17];4:128-32. Available from: http://www.thoracicmedicine.org/text.asp?2009/4/3/128/53351


The American Thoracic Society (ATS) has defined breathlessness, or dyspnea, as a "subjective experience of breathing discomfort that is comprised of qualitatively distinct sensations that vary in intensity." [1] It is a complex symptom of cardiopulmonary diseases, including chronic obstructive pulmonary disease (COPD), and affects several dimensions of a patient's life, reducing activity and functional capacity besides causing substantial distress and discomfort. The GOLD (Global Initiative for Chronic Obstructive Lung Disease) guidelines for grading the severity of COPD have recommended the use of a staging system based on post-bronchodilator FEV 1 . [2] Physiological assessments, however, provide only limited information about the impact of disease, and therefore a multidimensional approach has been recommended for evaluation. [3],[4] Assessment of dyspnea is critical in patient evaluation and management as it is the major limiting factor for activities of daily living and is also a superior predictor of mortality than spirometry. [5] It is included in the body mass index, airflow obstruction, dyspnea, and exercise capacity (BODE) index, a multidimensional scoring system that predicts mortality [6] and risk of hospitalization. [7]

Among the more commonly used validated scales to evaluate dyspnea in COPD [8] are the Modified Medical Research Council (MMRC) [9] and the closely related ATS78 scale, [10] the baseline dyspnea index (BDI), [11] and the oxygen cost diagram (OCD). [12] Although differing widely in content and scoring, these scales were found to be significantly interrelated. [13] Studies using the technique of factor analysis have also found that these scales club in the same domain. [14],[15] Pulmonary function tests, arterial blood gas analysis and measurement of exercise capacity such as the 6-min walk distance (6MWD) are commonly used clinical tools to evaluate the physiological and functional impairment caused by COPD. In separate evaluations, these scales of dyspnea have been found to have weak or no correlation with lung function parameters. [13],[14],[15] As these scales have wide differences in content, their construct validity against physiological parameters is likely to differ. We therefore studied the relationship of these scales with multiple parameters of physiological impairment.


   Materials and Methods Top


We carried out a retrospective analysis of pooled data of two recent studies on COPD [16],[17] done in our Institute. These had common outcomes, including dyspnea, spirometry, arterial blood gases and 6MWD measured with standardized methodology. The studies were approved by the Institutional Ethics Committee. Informed consent was taken.

Eighty-eight ambulatory stable male patients that were all former or current smokers with at least 10 pack-years of smoking and diagnosed with moderate or severe COPD (stages II, III and IV) on the basis of GOLD recommendations [2] were investigated. Only males were included as smoking-induced COPD is uncommon in the female population in India due to their substantially lower habit of smoking. The number of female patients reporting to our practice is therefore limited. Patients were excluded if there was a history suggestive of asthma, an acute exacerbation of COPD in the previous 4 weeks, and evidence of any other concurrent respiratory disorder or any systemic disease such as ischemic heart disease, musculoskeletal disorders, peripheral vascular diseases or other disabling conditions that would interfere with the tests.

Study design

The study was carried out in the outpatient setting between 10 a.m. and 12 noon. Bronchodilators were withdrawn 12 hours before the investigation. Inhaled corticosteroids were however allowed unchanged. After a detailed history and examination, the patients had an evaluation of dyspnea using the three instruments in a random order, followed by an arterial blood gas analysis and the 6-min walk test. After the patient had rested for some time, spirometry before and after administration of bronchodilator was carried out.

Methods

Dyspnea assessment was carried out using the MMRC, BDI and the OCD scales. The MMRC is an ordinal five-point scale (grades I to V) based on degrees of various physical activities that precipitate dyspnea. [9] Grade V represents the most severe category. The BDI has a scale of five grades, scored 0 to 4, for each of the following categories: Functional impairment, magnitude of task and magnitude of effort. [11] The ratings for each of the three categories were added to give the total score (range, 0-12). Lower the score, greater is the dyspnea. The OCD is a visual analog scale that corresponds to oxygen requirements at different activity levels, which are represented as a value ranging from zero to 100mm with the highest score indicating no impairment. [12] The patient is asked to indicate the level of activity at which he/she begins to experience dyspnea. The OCD score is measured in 100mm. The shorter the distance, the greater is the breathlessness.

The 6-min walk test was carried out as recommended by the American Thoracic Society. [18] The patients were instructed to walk in a corridor on level at their own pace attempting to cover as much ground as possible in 6 minutes. They were constantly encouraged using standard phrases to put in their maximal effort but were permitted to slow down or stop and restart during the test. The distance traveled within 6 minutes (6MWD) was noted. Supplemental oxygen was not used during the test. The test was carried out only if oxygen saturation on pulse oximetry was greater than 88%. Only 1 patient was excluded by this criterion.

Spirometry was performed on the dry, rolling-seal spirometer of the Benchmark model lung function machine (P.K. Morgan, Kent, UK) as per the recommendations of the ATS. [19] Three acceptable and at least two reproducible maximal expiratory flow volume curves were obtained. The highest values of FEV 1 (forced expiratory volume in 1 s) and FVC (forced vital capacity) obtained were recorded. Post-bronchodilator values were obtained by repeating the spirometry 20 minutes after inhalation of 200 µg salbutamol from a metered-dose inhaler. Reference equations for north Indian adults were used to calculate the predicted values in terms of percentage. [20] Arterial blood gas (ABG) analysis was carried out on radial artery blood sample measured in instrumentation laboratory machine (model 1312). The ABG analysis was obtained after a 15-minute rest in supine position breathing room air.

Statistical analysis

Statistical analysis was carried out with the help of SPSS 14.0 software for Windows and GraphPad Prism 4.02 for Windows. Multiple group comparisons of quantitative variables were made using analysis of variance (ANOVA), and inter-group comparisons were carried out using the Bonferroni's test to identify the significantly different groups. Coefficients of correlation for bivariate relationships were obtained using the Spearman rank correlation test. The relationship between the MMRC grade and BDI/OCD scores was evaluated by computing the polyserial correlation. This was done as the MMRC is an ordinal variable and Spearman's or Pearson's correlation analysis is not an appropriate test to evaluate relationships between ordinal and continuous variables like BDI or OCD scores. Polyserial correlation was computed using Lisrel 8.80 student edition software. Proportions of patients with different MMRC grades across GOLD stages were analyzed by chi-square test. A P value of <0.05 was considered statistically significant.


   Results Top


[Table 1] shows the demographic and clinical characteristics according to GOLD stages. [Table 2] shows the inter-correlations (spearman's rho) among spirometric, blood gas parameters, 6MWD, BDI and OCD. The OCD and BDI were moderately strongly correlated [Figure 1]. The correlations of these measures of dyspnea with FEV 1 %predicted were weak although statistically significant, while there was no correlation with 6MWD. The correlations of BDI and OCD scores with arterial blood gas parameters were moderately strong. Polyserial correlation coefficients computed between MMRC grades and BDI and OCD were −0.56 ( P <.05) and −0.54 (.05 > p <.1), respectively.

As the MMRC scale is an ordinal variable, we compared the different study variables across its different grades [Table 3]. The PaO 2 , PaCO 2 , post-bronchodilator FEV 1 % predicted, OCD and BDI scores were significantly different between these categories, while the 6MWD and post-bronchodilator FVC % predicted were not. The PaO 2 , PaCO 2 and post-bronchodilator FEV 1 % predicted were significantly different between patients with MMRC grade 2 and 4, while grade 3 patients did not differ significantly for these variables with grade 2 and grade 4 patients. However, the OCD and BDI scores were significantly different for all the pairs of between-group comparisons, with scores indicating progressively increasing dyspnea with increasing MMRC grades.


   Discussion Top


The present study shows that the OCD and BDI scales of dyspnea measurement have a moderately strong correlation with each other in patients with COPD. Each of these scales had weak and moderately strong relationships with FEV 1 % predicted and arterial blood gas abnormalities, respectively. The MMRC grade of dyspnea had a moderately strong correlation with BDI scores and somewhat weaker correlation with OCD scores. The BDI and OCD scores were significantly different between patients grouped according to MMRC grades. However, the categories of MMRC were not sufficiently discriminatory for spirometric and arterial blood gas abnormalities. None of the three measures of dyspnea was found to have a significant relationship with the 6MWD.

Previous studies have shown that the BDI and the OCD scores have significant although modest correlations with FEV 1 . [11] We observed weak correlations between these scales and FEV 1 % predicted and none between these scales and FVC%predicted. In addition, these scales were also observed to have a moderately strong correlation with arterial blood gas abnormalities. This contrasts with the MMRC scale, which did not show any association with any of the measures of physiological impairment used in the present study. Bestall et al. [21] have also shown that FEV 1 is not associated with grades of MMRC. The lack of relationship of MMRC grade with physiological parameters has also been shown in other studies, where it was however shown to be correlated more strongly with health-related quality of life and with indices of anxiety and depression than with spirometric values. [22],[23] Hajiro et al [24] have shown that these three scales have similarly strong correlations with several indices of health-related quality of life. Thus, the BDI and the OCD scales appear to have a better construct validity. In a recent comparison of different scales by Eakin et al, [25] the BDI was found to have the highest levels of reliability and validity. Our results support this observation.

As these scales have been designed to capture the limitations imposed on daily activities by dyspnea, the significant inter-relationships among these are not surprising. In studies using factor analysis, these scales were found to group in the same factor, suggesting that these provide information about the same dimension of the disease. [14],[15] Another study showed that the BDI scores correlated well with the MMRC grading and the OCD scores. [13]

Each of these scales has merits and limitations. In clinical practice, the MMRC grade remains the commonest scale used because of its simplicity, ease of administration and established validation as a useful marker in COPD. It predicts the likelihood of survival of patients with COPD, [5] correlates well with the health status scores [11],[23],[24] and is also a component of the BODE index. [6] Its limitations include a lack of significant association with any parameter of physiological and functional impairment. Further, it is primarily a discriminative instrument and is not very responsive, especially over short terms. Thus, its utility to evaluate response to treatment is limited. The BDI and the transition dyspnea index (TDI) [11] provide a method of assessing dyspnea at baseline (the BDI) and measuring subsequent changes over time following interventions (the TDI). It is more elaborate than the MMRC grade and is multidimensional. In addition to the magnitude of task required to induce dyspnea, it also rates functional impairment and the magnitude of effort needed to carry out a task. Thus, it is also more time consuming and complex. Being quantitative, these scales are amenable to mathematical operations. These have good discriminative and evaluative properties. The oxygen cost diagram (OCD) is a vertical scale designed to rate activities on a continuum according to the number of calories expended in the performance of the activity. [12] It is simple and easy to administer. However, not all patients engage in all the activities depicted along the scale, and some need repeated instructions for marking the appropriate response. [1] It is a unidimensional scale focusing on the magnitude of task.

The study has a few limitations. First, it is a retrospective analysis of pooled data from two studies. Although standardized methods were used, inter-observer differences may have influenced results. Second, only male patients were included for reasons explained earlier. This limits generality of results to female subjects. Gender differences in perception of dyspnea may modify the results.


   Conclusion Top


The inter-relationships among commonly used scales to measure dyspnea - the MMRC grading, BDI and OCD - are moderately strong in patients with COPD. However, the BDI and OCD scales are significantly associated with some of the measures of physiological impairment, while the MMRC grade is not.


   Acknowledgment Top


The technical assistance provided by Mr. HK Vatsa, Mr. GC Chandrashekharan, Mr. Rajesh Mehta and Mr. Ramveer Singh is acknowledged with thanks.[Table 1]

 
   References Top

1.Dyspnea. mechanisms, assessment, and management: A consensus statement. American Thoracic Society. Am J Respir Crit Care Med 1999;159:321-34.   Back to cited text no. 1    
2. NHLBI/WHO Global Initiative for Chronic Obstructive Lung Disease (GOLD) Workshop: Global strategy for the diagnosis, management, and prevention of Chronic Obstructive Pulmonary Disease. 2006 update. Available from: http://www.goldcopd.com. accessed on 26 th Dec 2008.  Back to cited text no. 2    
3.Yusen RD. What outcomes should be measured in patients with COPD? Chest 2001;119:327-8.  Back to cited text no. 3    
4.Chhabra SK. Assessment of response to treatment in COPD. Chest (India) 2004;5:309-11.  Back to cited text no. 4    
5.Nishimura K, Izumi T, Tsukino M, Oga T. Dyspnea is a better predictor of 5-year survival than airway obstruction in patients with COPD. Chest 2002;121:1434-40.  Back to cited text no. 5  [PUBMED]  [FULLTEXT]
6.Celli BR, Cote CG, Marin JM, Casanova C, de Oca MM, Mendez RA, et al The body mass index, airflow obstruction, dyspnea, and exercise capacity index in chronic obstructive pulmonary disease. N Engl J Med 2004; 350:1005-12.  Back to cited text no. 6    
7.Ong KC, Earnest A, Lu SJ. A multidimensional grading system (BODE index) as predictor of hospitalization for COPD. Chest 2005;128:3810-6.  Back to cited text no. 7  [PUBMED]  [FULLTEXT]
8.Bausewein C, Farquhar M, Booth S, Gysels M, Higginson IJ. Measurement of breathlessness in advanced disease: A systematic review. Respir Med 2007;101: 399-410.  Back to cited text no. 8  [PUBMED]  [FULLTEXT]
9.Schilling RSF, Hughes JPW, Dingwall-Fordyce I. Disagreement between observers in an epidemiological study of respiratory disease. Br Med J 1955;1:65-8.  Back to cited text no. 9    
10.Ferris BG. Epidemiology Standardization Project (American Thoracic Society). Am Rev Respir Dis 1978;118:1-120.  Back to cited text no. 10  [PUBMED]  
11.Mahler DA, Weinberg DH, Wells CK, Feinstein AR. The measurement of dyspnea: Contents, interobserver agreement, and physiologic correlates of two new clinical indexes. Chest 1984;85:751-8.  Back to cited text no. 11  [PUBMED]  [FULLTEXT]
12.McGavin CR, Artvinli M, Naoe H, McHardy GJ. Dyspnea, disability, and distance walked: Comparison of estimates of exercise performance in respiratory disease. Br Med J 1978;2:241-3.  Back to cited text no. 12  [PUBMED]  [FULLTEXT]
13.Mahler DA, Wells CK. Evaluation of clinical methods for rating dyspnoea. Chest 1988;93:580-6.   Back to cited text no. 13  [PUBMED]  [FULLTEXT]
14.Wegner RE, Jφrres RA, Kirsten DK, Magnussen H. Factor analysis of exercise capacity, dyspnoea ratings and lung function in patients with severe COPD. Eur Respir J 1994;7:725-29.  Back to cited text no. 14    
15.Hajiro T, Nishimura K, Tsukino M, Ikeda A, Koyama H, Izumi T. Analysis of clinical methods used to evaluate dyspnea in patients with chronic obstructive pulmonary disease. Am J Respir Crit Care Med 1998;158:1185-9.  Back to cited text no. 15  [PUBMED]  [FULLTEXT]
16.Gupta A. Assessment of health-related quality of life in chronic obstructive pulmonary disease and its relationship with dyspnoea, exercise performance, lung function and arterial blood gas abnormalities. MD (Pulmonary Medicine) Thesis, University of Delhi, 2002.  Back to cited text no. 16    
17.Khuma MZ. Assessment of outcome measures for treatment in chronic obstructive pulmonary disease. MD (Pulmonary Medicine) Thesis, University of Delhi, 2006   Back to cited text no. 17    
18. American Thoracic Society. ATS Statement: Guidelines for the Six-Minute Walk Test. Am J Respir Crit Care Med 2002;166: 111-7.   Back to cited text no. 18    
19.American Thoracic Society. 1994 Update. Standardization of spirometry. Am J Respir Crit Care Med 1995;152:1107-36.  Back to cited text no. 19    
20.Jain SK, Gupta CK. Age, height and body weight as determinants of ventilatory norms in healthy men above forty years age. Ind J Med Res 1967;55:599-606.  Back to cited text no. 20    
21.Bestall JC, Paul EA, Garrod R, Garnham R, Jones PW, Wedzicha JA. Usefulness of the Medical Research Council (MRC) dyspnoea scale as a measure of disability in patients with chronic obstructive pulmonary disease. Thorax 1999;54:581-6.  Back to cited text no. 21  [PUBMED]  [FULLTEXT]
22.Schlecht NF, Schwartzman K, Bourbeau J. Dyspnea as clinical indicator in patients with chronic obstructive pulmonary disease. Chr Respir Dis 2005;2:183-91.  Back to cited text no. 22    
23.Hajiro T, Nishimura K, Tsukino M, Ikeda A, Oga T, Izumi T. A comparison of the level of dyspnea vs disease severity in indicating the health-related quality of life of patients with COPD. Chest 1999;116:1632-7.  Back to cited text no. 23  [PUBMED]  [FULLTEXT]
24.Hajiro T, Nishimura K, Tsukino M, Ikeda A, Koyama H, Izumi T. Comparison of discriminative properties among disease-specific questionnaires for measuring health-related quality of life in patients with chronic obstructive pulmonary disease. Am J Respir Crit Care Med 1998;157:785-90.  Back to cited text no. 24  [PUBMED]  [FULLTEXT]
25.Eakin EG, Sassi-Dambron DE, R ies AL, Kaplan RM. Reliability and validity of dyspnoea measures in patients with obstructive lung disease. Int J Behav Med 1995;2:118-34.  Back to cited text no. 25    


    Figures

  [Figure 1]
 
 
    Tables

  [Table 1], [Table 2], [Table 3]

This article has been cited by
1 Buspirone for management of dyspnea in cancer patients receiving chemotherapy: a randomized placebo-controlled URCC CCOP study
Anita R. Peoples,Peter W. Bushunow,Sheila N. Garland,Charles E. Heckler,Joseph A. Roscoe,Luke L. Peppone,Deborah J. Dudgeon,Jeffrey J. Kirshner,Tarit K. Banerjee,Judith O. Hopkins,Shaker R. Dakhil,Marie A. Flannery,Gary R. Morrow
Supportive Care in Cancer. 2016; 24(3): 1339
[Pubmed] | [DOI]
2 Evaluation of Quality of Life with the Chronic Obstructive Pulmonary Disease Assessment Test in Chronic Obstructive Pulmonary Disease and the Effect of Dyspnea on Disease-Specific Quality of Life in These Patients
Oguzhan Okutan,Dilaver Tas,Ersin Demirer,Zafer Kartaloglu
Yonsei Medical Journal. 2013; 54(5): 1214
[Pubmed] | [DOI]
3 The relation of the subjective dyspnoea perception with objective dyspnoea indicators, quality of life and functional capacity in patients with COPD
Ayse Cil Akinci,Rukiye Pinar,Tuncalp Demir
Journal of Clinical Nursing. 2013; 22(7-8): 969
[Pubmed] | [DOI]
4 Melatonin reduces lung oxidative stress in patients with chronic obstructive pulmonary disease: a randomized, double-blind, placebo-controlled study : Melatonin reduces oxidative stress in COPD
Antonio George de Matos Cavalcante, Pedro Felipe Carvalhedo de Bruin, Veralice Meireles Sales de Bruin, Deuzilane Muniz Nunes, Eanes Delgado Barros Pereira, Marina Medeiros Cavalcante, Geanne Matos Andrade
Journal of Pineal Research. 2012; : no
[VIEW] | [DOI]
5 Weighing up the evidence–a systematic review of measures used for the sensation of breathlessness in obesity
Y Gerlach, M T Williams, A M Coates
International Journal of Obesity. 2012;
[VIEW] | [DOI]
6 Airflow limitation or static hyperinflation: which is more closely related to dyspnea with activities of daily living in patients with COPD?
Koichi Nishimura, Maya Yasui, Takashi Nishimura, Toru Oga
Respiratory Research. 2011; 12(1): 135
[VIEW] | [DOI]
7 Factors associated with mortality in patients with exacerbation of chronic obstructive pulmonary disease hospitalized in General Medicine departments
Internal and Emergency Medicine. 2011; 6(1): 47
[VIEW] | [DOI]
8 Measures of systemic sclerosis (scleroderma): Health Assessment Questionnaire (HAQ) and Scleroderma HAQ (SHAQ), Physician- and Patient-Rated Global Assessments, Symptom Burden Index (SBI), University of California, Los Angeles, Scleroderma Clinical Trials
Janet Pope
Arthritis Care & Research. 2011; 63(s11): S98
[VIEW] | [DOI]
9 Scale of dyspnea in COPD: User friendly
Wiwanitkit, V.
Annals of Thoracic Medicine. 2010; 5(1): 55
[Pubmed]
10 Using the Oxygen-Cost Diagram in Ramp-slope Selection for Dyspneic Patients
Chih-Hsin Lee, Ming-Lung Chuang, I-Feng Lin
Internal Medicine. 2010; 49(14): 1325
[VIEW] | [DOI]
11 Photodynamic Laser Therapy for Lesions in the Airway
Douglas J. Minnich,Ayesha S. Bryant,Adam Dooley,Robert J. Cerfolio
The Annals of Thoracic Surgery. 2010; 89(6): 1744
[Pubmed] | [DOI]
12 Photodynamic Laser Therapy for Lesions in the Airway
Minnich, D.J., Bryant, A.S., Dooley, A., Cerfolio, R.J.
Annals of Thoracic Surgery. 2010; 89(6): 1744-1749
[Pubmed]



 

Top
Print this article  Email this article
 
  Search
 
   Next article
   Previous article 
   Table of Contents
  
    Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
    Article in PDF (388 KB)
    Citation Manager
    Access Statistics
    Reader Comments
    Email Alert *
    Add to My List *
* Registration required (free)  


    Abstract
    Materials and Me...
    Results
    Discussion
    Conclusion
    Acknowledgment
    References
    Article Figures
    Article Tables

 Article Access Statistics
    Viewed8839    
    Printed283    
    Emailed4    
    PDF Downloaded1220    
    Comments [Add]    
    Cited by others 12    

Recommend this journal