Assessing the academic achievement of United States orthopaedic departments

lNTRODUCTlON

The evaluation of academic orthopaedic surgery programs based on scholarly contribution is difficult to assess.Faculty are often measured by bibliometric variables that represent their academic productivity such as citation indices,number of publications and amount of research funding[1-3].Our study aimed to measure the current scholarly productivity of orthopaedic departments in the United States.The results of this analysis are an assessment of orthopaedic programs based on the academic contributions of their faculty.In addition to recognizing highly academic departments,our study aims to provide orthopaedic departments with a tool that can be continually utilized to monitor academic productivity and,thus,identify areas for improvement.

There are many difficulties associated with evaluating the academic productivity of orthopaedic surgery programs.The subjective nature of certain metrics,such as national reputation and faculty satisfaction,used in rankings likeor[4-7]often make standardizing academic achievement difficult.Furthermore,current productivity is not always accurately reflected,as the productivity of alumni for the preceding 15 years are included in these rankings[4].Another difficulty in evaluating academic achievement is the lack of consensus as to the weight that different objective bibliometric measures should have when determining overall academic contribution.

6. A talent worth having: Usually the woman does not have the skills her parent boasts of. Sometimes she cannot spin flax and can only spin gold which is actually an undesirable trait in these versions of the story. In these versions gold is worthless in comparison to the ability to create much needed clothing. Othertimes she cannot spin the gold, only flax and the gold is preferred as by the king in this story. Othertimes she cannot or will not spin at all, being lazy and a burden to her family that just wants to be rid of her. Return to place in story.

38. No ill shall befall you: This promise provides a sense of foreboding. Hansel and Gretel s good fortune seems too good to be true and it is.Return to place in story.

Efforts to quantify academic achievement have gained popularity amongst various specialties over the past decade.Publicly available metrics such as National Institutes of Health(NIH)funding,faculty Hirsch-indices(h-index)and number of publications have been used in plastic surgery,ophthalmology,dermatology,urology and a variety of other medical specialties to provide a measurement of an institution’s academic prowess[8-15].The h-index is a well validated tool to accurately measure academic output and has been lauded in the orthopaedic community[16-18].The h-index is defined as the number of publications()an individual has that receive at leastcitations,with each other publication having ＜citations[19].Therefore,the h-index can never exceed the number of publications a faculty member has and considers both quality and quantity of a faculty member’s publications.NIH funding has also been validated as an accurate measure of scholarly impact across different specialties[9,20].

Previous studies in other specialties have utilized algorithms that weigh metrics of academic achievement to ultimately rank programs based on their academic productivity[12,21,22].Our current study uses data from 2014-2018 to provide a five-year updated,enhanced analysis of our previous study[23],and continue to assess orthopaedic programs based on the academic output of their faculty.Cumulative statistics as well ascapita statistics,which help to highlight both programs with a large volume of academic output as well as smaller programs with a high academic yield,were used in this study.Furthermore,in order to acknowledge programs that have improved their scholarly productivity,we quantified the change in cumulative score from our previous paper that used data from 2013 to our current study that uses data from 2014-2018.As scholarly productivity is often linked with academic promotion,the ability to attract talented faculty and other important factors[3],standardized methods are necessary to accurately assess the academic achievement of orthopaedic surgery programs.The authors believe that the establishment of a consistent and representative algorithm of program achievement can be used as a tool to continually monitor progress over time and,importantly,provide guidance to individual programs on target areas to enhance overall scholarly productivity.

Based on per-capita measurements of academic achievement that accounts for the number of full-time faculty in each program,the most academically productive orthopaedic surgery programs were:Mayo Clinic(Rochester),Washington University in St.Louis,Rush University,VCU,MGH/Brigham and Women’s/Harvard,and Duke University(Figure 3).

MATERlALS AND METHODS

Our study retrospectively analyzed a cohort of orthopaedic faculty at United States-based academic orthopaedic programs.Variables included for analysis were National Institutes of Health funding(2014-2018),leadership positions in orthopaedic societies(2018),editorial board positions of top orthopaedic journals(2018),total number of publications and Hirsch-index.A weighted algorithm was used to calculate a cumulative score for each academic program.

Our current study includes the same bibliometrics as our prior study[23]to quantify academic productivity for each faculty member.While some of these metrics are cumulative over a faculty member’s career,including total number of publications and h-index,other metrics such as NIH funding from 2014-2018,leadership society membership for 2018 and journal editorial board membership for 2018 provide a more current evaluation of academic productivity.As NIH funding can fluctuate dramatically,five years of NIH funding was analyzed.The bibliometrics of each faculty member within an orthopaedic surgery department were cumulated.A weighted algorithm was subsequently used to compute a score for each academic institution to assess their scholarly contribution.Change in cumulative score for each program from 2013 to 2018 was then calculated.

The weighted score was calculated as follows.For each of the five categories,each academic program was assigned a score from zero to one,with the program with the highest score in an individual category assigned a one.The category-specific score for each academic institution was calculated by dividing the value of a specific outcome measure attained by an institution by the value of that outcome measure attained by the highest achieving institution.That score was then either multiplied by 2.0 for the category of NIH funding,1.0 for the categories of number of publications and h-index or 0.5 for the categories of leadership society membership and journal editorial board membership.Thus,NIH funding accounted for 40% of the total score,number of publications and h-index each accounted for 20% and society leadership and journal editorial board membership each accounted for 10%(Figure 1).For example,the faculty from the University of Iowa accounted for 2789 total publications.This was divided by the highest number of publications(13494)achieved by any institution(the Hospital for Special Surgery,Cornell)and multiplied by 1.0(weighing factor for number of publications).A score of 0.207 was then given to the University of Iowa for the “number of publications” category.The same computation was then repeated with each bibliometric being divided by the number of full-time faculty within a department to calculate thecapita measurement.

Assessing academic productivity allows academic departments to identify the strengths of their scholarly contribution and provides an opportunity to evaluate areas for improvement.

Our analysis also included the two largest general orthopaedic societies in the United States:The Orthopaedic Research Society(ORS)and the American Academy of Orthopaedic Surgeons(AAOS)as well as the preeminent society from each orthopaedic subspecialty to stay consistent with our prior study.These societies were chosen to give an equal representation to all orthopaedic subspecialties.In an effort to decrease inherent bias,no societies that are nomination-dependent were included for analysis.Faculty on editorial boards of American orthopaedic journals with an impact factor over 2.5 were included for analysis.These journals included all journals from our previous study as well as theand

RESULTS

Of 181 academic orthopaedic programs with an accredited residency program were included for data analysis.Three institutions were excluded due to the lack of a list of faculty members on departmental websites and limited contacts to find this information.36 out of the remaining 178 programs responded to the authors’ email for a response rate of 20.2%.

Be comforted, said the old woman. I will help you. Here is a golden comb for you. Wait until the full moon has risen, then go to the millpond, sit down on the bank and comb your long black hair with this comb. When you are finished set it down on the bank, and you will see what will happen.

The total number of publications for full-time faculty of an orthopaedic department ranged from 12 to 13494 at the Hospital for Special Surgery(Cornell).The five institutions with the most publications following the Hospital for Special Surgery were SKMC at Thomas Jefferson University(9259),Mayo Clinic(Rochester)(8735),Washington University in St.Louis(6616),Massachusetts General Hospital(MGH)/Brigham and Women's/Harvard(6421),and Rush University(5661)(Table 2).

Towards the evening robbers came; they killed three of herpeople and also the yard-dog, and attached Mrs. Meta to the kennelby the chain, while they themselves made good cheer in the hall anddrank the wine and the good ale out of her cellar. Mrs. Meta was nowon the chain, she could not even bark.

The Hospital for Special Surgery had the highest cumulative h-index with 3318,followed by SKMC at Thomas Jefferson University which had a h-index of 1988(Table 3).

Fifty-two programs had faculty members holding at least one leadership position in orthopaedic surgery societies in the United States,representing 29.2% of the 178 programs evaluated.Full-time faculty at MGH/Brigham and Women’s/Harvard garnered the most leadership positions with seven,followed by Duke University with six.Four leadership positions were held by faculty at the Hospital for Special Surgery,Johns Hopkins University,Mayo Clinic(Rochester),Rush University,and SKMC at Thomas Jefferson University(Table 4).

Full-time faculty at the Hospital for Special Surgery held the most editorial board positions at top orthopaedic and subspecialty journals with 20 positions,followed by MGH/Brigham and Women’s/Harvard with 19 positions and Washington University in St.Louis with 18 editorial board positions(Table 5).

All 178 programs were evaluated using the comprehensive weighted algorithm.Based on this algorithm,Washington University in St.Louis was shown to be the most academically productive orthopaedic surgery program in the United States.The following five most academically productive orthopaedic programs were:The Hospital for Special Surgery,SKMC at Thomas Jefferson University,the UCSF,MGH/Brigham and Women’s/Harvard and Mayo Clinic(Rochester)(Figure 2).

The prince accepted gladly, and they went back to the palace, where the bald-headed king s daughter, who was still more beautiful than the other princess, welcomed them at the door and led the way into a large hall and to a table covered with silver dishes

One afternoon, a little boy had lost his puppy1. He looked under his bed. He looked all over his house. But still there was no puppy. Finally, he looked for his puppy in the garden. After a few hours, he still could not find the puppy.

Programs received cumulative NIH grant funding between 2014-2018 ranging from $31.9 million[University of California,San Francisco(UCSF)]to no NIH funding.Washington University in St.Louis($29.3 million),Virginia Commonwealth University(VCU)($28.6 million),University of Rochester($23.0 million),Brown University($22.1 million)and Sidney Kimmel Medical College(SKMC)at Thomas Jefferson University($18.2 million)represent the next five institutions with the highest NIH funding during this period(Table 1).

As an overseas student, I have been looking forward to working in Australian enterprises2 in order to learn how the Australian working system and management goes

DlSCUSSlON

This study aims to assess the scholarly contribution of orthopaedic departments using objective bibliometrics from 2014 to 2018.With so many metrics available to assess academic achievement,this is admittedly both difficult and controversial.In light of the financial,reputational and academic pressures surrounding academic productivity,our goal was to(1)Acknowledge academic institutions for their scientific contribution;(2)Allow academic departments to communicate best practices to one another;and(3)Provide a method to longitudinally monitor academic improvement to facilitate a discussion asto the definition of academic success.

It is imperative to consider that the mission of orthopaedic programs and faculty is not always rooted in academic achievement,but rather is based on operative and clinical capability,outreach to underserved populations,teaching and mentorship,and/or technological innovation.Undoubtedly,there are metrics other than academic productivity that define a program’s “success.” Although many of the results of this study are organized numerically,the findings are not intended for comparison against one another.The purpose of this study was not to “rank” orthopaedic departments,but rather to establish a tool that programs may use to assess their own academic productivity against their respective baseline values established in this study.The conclusions reached in this study only pertain to academic productivity as related to the specific bibliometrics analyzed.Additionally,departments inherently differ in size and maturity of research infrastructure.Nonetheless,in a culture that isconstantly interested in evaluations,analyzing academic productivity using objective metrics remains an important factor to appraise orthopaedic departments.

There are several limitations of this study to consider.One important limitation to our study is the subjective nature by which the weighted algorithm was formulated.The authors believe that our previous study[23]did not place enough emphasis on the effect that basic science research has on academic productivity.Although other sources of basic science funding such as the Department of Defense and the Orthopaedic Research and Education Foundation exist,the NIH is the largest public funder of biomedical research worldwide[27].Therefore,as basic science is a large portion of the NIH portfolio[27],NIH funding was given additional weight(40% of cumulative score)relative to other bibliometrics included.As the distribution of points in our current study slightly differs from our previous paper,our calculation of score improvement from 2013 to 2018 is subject to limitations.Given that any choice of variables for a weighted algorithm will have an element of subjectivity to it,the authors accept these limitations.The authors also acknowledge that it may be difficult to identify parttime,co-appointed or emeritus faculty based solely on departmental websites.Furthermore,there are undoubtedly changes in faculty lists over from 2013 to 2018.However,universities and academic centers have different criteria for “joint appointments.” As such,in an effort to decrease inherent bias,part-time or co-appointed faculty were not included.All 178 programs were also contacted in an attempt to confirm faculty lists.

Further limitations of this study lie in the actual bibliometrics used.The h-index has been validated both within and outside of the orthopaedic community[16-18],however it is not without its criticisms.The h-index does not proportionally reflect the impact of authors who have published a small number of highly cited studies,nor does it proportionally reflect the impact of authors who have published a large number of scarcely cited studies.The h-index and the total number of publications also do not take into account the order that authors are listed and thus,the impact that each author had[28,29].Augmentations of the h-index have been proposed[28-30],however until they are widely accepted and publicly available,the authors believe that the h-index remains the best metric.Furthermore,the authors only reported journal editorial board members and leaders in orthopaedic academic societies for the 2018 calendar year as most journals and societies do not have this data for prior years publicly available.This excluded either of these metrics for years prior.In an effort to minimize this limitation,editorial board and society leadership each only accounted for 10% of the overall score.While the authors believe a snapshot of recent academic productivity is important when evaluating recent academic achievement,it is imperative to understand how the availability of prior data would affect these results.

VCU had the largest improvement in their score from 2013 with a 1.62 point change.SKMC at Thomas Jefferson University(1.40),UCSF(1.39),MGH/Brigham and Women’s/Harvard(1.31),Brown University(1.23)and Icahn School of Medicine at Mount Sinai(0.92)were the next five institutions with the largest improvement in cumulative score since 2013(Table 6).

Based on this algorithm,Washington University in St.Louis,the Hospital for Special Surgery,SKMC at Thomas Jefferson University,the UCSF and MGH/Brigham and Women’s/Harvard are currently the five most cumulatively academically productive orthopaedic surgery programs.The Mayo Clinic(Rochester),Washington University in St.Louis,Rush University,VCU and MGH/Brigham and Women’s/Harvard are currently the five most academically productive orthopaedic surgery programscapita.The five academic programs that had the largest improvement in cumulative score from 2013 to 2018 were:VCU,SKMC at Thomas Jefferson University,UCSF,MGH/Brigham and Women’s/Harvard,and Brown University.

CONCLUSlON

This algorithm is easily reproducible and provides a metric that departments can use to track their academic productivity over time as well as identify areas for improvement.These reported bibliometrics can continually be updated in upcoming years as a measure of changing scholarly contribution.Programs that have shown dramatic improvement in scholarly contribution since our 2013 study can be seen as model programs.Programs striving for similar improvement would have other programs identified to serve as roadmaps,opening up an avenue for communication.Furthermore,as factors affecting academic promotion are often difficult to assess,this standardized algorithm may,importantly,aid academic medical centers to determine promotion.This is not a list of the “best”orthopaedic surgery institutions as clinical care metrics were not included in the analysis.This analysis did not attempt to take into account the quality of clinical care provided,or the clinical education provided to medical students,residents or fellows,therefore the authors would like to reiterate that this algorithm was not used to rank orthopaedic departments.

ARTlCLE HlGHLlGHTS

Research background

Orthopaedic surgery faculty are often measured by bibliometric variables that represent their academic productivity such as citation indices,number of publications and amount of research funding.

Research motivation

The validated Scopus database was used to determine the total number of publications and h-index for each individual faculty member[25,26].Scopus was chosen to analyze the total number of publications as Scopus only includes peer-reviewed literature and has the broadest coverage of any database[25].Our analysis of total number of publications will therefore include all types of publications from an individual’s career.The NIH Research Portfolio Online Reporting Tool was used to obtain NIH funding from 2014-2018[27].

Research objectives

To provide objective benchmarks for departments seeking to enhance academic productivity and identify those with improvement in recent past.

Research methods

Programs included for analysis were identified through a search of the Accreditation Council for Graduate Medical Education website[24].Faculty included for analysis were identified using faculty lists on individual departmental websites.An email was subsequently sent to program directors and coordinators for each institution to verify that the list of faculty on the department website was accurate and up to date.In an effort to standardize faculty lists,only faculty with a full-time appointment in the respective department of orthopaedic surgery were included for analysis.This included research faculty but did not include surgeons from other specialties,house staff,co-appointed faculty,part-time faculty or emeritus faculty as depicted on individual departmental websites.All inputted data and calculations were reviewed by multiple authors independently to ensure accuracy.

Research results

The five institutions with the highest cumulative score,in decreasing order,were:Washington University in St.Louis,the Hospital for Special Surgery,Sidney Kimmel Medical College(SKMC)at Thomas Jefferson University,the University of California,San Francisco(UCSF)and Massachusetts General Hospital(MGH)/Brigham and Women’s/Harvard.The five institutions with the highest score per capita,in decreasing order,were:Mayo Clinic(Rochester),Washington University in St.Louis,Rush University,Virginia Commonwealth University(VCU)and MGH/Brigham and Women’s/Harvard.The five academic programs that had the largest improvement in cumulative score from 2013 to 2018,in decreasing order,were:VCU,SKMC at Thomas Jefferson University,UCSF,MGH/Brigham and Women’s/Harvard,and Brown University.

Then, and while he was all unaware96, she put forth her hand to a stick fashioned like a snake; she said some words over it and struck him so sharply on the shoulder that he cried out; then he made a pirouette and found that he was a deer

Research conclusions

This algorithm can provide orthopaedic departments a means to assess academic productivity,monitor progress,and identify areas for improvement as they seek to expand their academic contributions to the orthopaedic community.

Research perspectives

The authors would like to reiterate that this is in no way a ranking system as there are many unique challenges that institutions face.We hope that this provides a tool that programs may use to assess and improve their own academic productivity,while simultaneously providing an opportunity to praise the growth and achievement of institutions on a cumulative as well ascapita basis.

FOOTNOTES

The above authors were each included in substantial contributions to research design,or the acquisition,analysis or interpretation of data,drafting the paper or revising it critically,approval of the submitted and final versions.

All authors have no financial disclosures or conflicts of interest to report for this work.

In Robin McKinley s novelization of the story, Deerskin confronts her father at his wedding ceremony to a young princess, exposing his crime of rape86 against her.Return to place in story.

The authors have read the PRISMA 2009 Checklist,and the manuscript was prepared and revised according to the PRISMA 2009 Checklist.

21.Made a present of these to the king: Gift giving has long been a method of gaining favor, such as with a king. Gifts are often presented to sovereigns, dignitaries, and diplomats in hopes of gaining their favor while showing honor to them. Gift giving can also be used to show one s wealth through the costliness of the gift.

This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers.It is distributed in accordance with the Creative Commons Attribution NonCommercial(CC BYNC 4.0)license,which permits others to distribute,remix,adapt,build upon this work non-commercially,and license their derivative works on different terms,provided the original work is properly cited and the use is noncommercial.See:https://creativecommons.org/Licenses/by-nc/4.0/

The fact that she did not live with the family, preferring her tiny cottage and solitude3 to the comfortable but rather noisy household where we were brought up-added to the respectful fear in which she was held

United States

Rishi Trikha 0000-0003-4193-5446;Thomas E Olson 0000-0003-3960-156X;Ameen Chaudry 0000-0002-1702-0468;Chad R Ishmael 0000-0003-3204-9354;Cristina Villalpando 0000-0001-7240-618X;Clark J Chen 0000-0002-3544-7625;Kellyn R Hori 0000-0001-8146-8518;Nicholas M Bernthal 0000-0003-3338-5878.

Fan JR

A

Fan JR