Many surgical research projects commence with ideas, technical modifications, and individual observations rather than specific hypotheses, which creates methodologic difficulties in evaluating surgical complexity by established clinical research designs. This has been known for years, but remains a matter of controversy and debate.

In a series of three papers, the Balliol Group highlighted current shortcomings and summarized the archetype of surgical clinical research in the IDEAL statement (Idea, Development, Exploration, Assessment, Long-term Surveillance).1-3 While IDEAL gives a detailed overview of the past and reflects the present, it neither offers forward-looking solutions nor addresses the specific issues of orthopaedic surgery.

The latter is intrinsically tied to the development of new equipment and implants, creating complex interactions between demographic and biological variables, preferences, manual skills of the surgeon, and hardware properties, all of which may contribute to the patient’s clinical progress. Biological principles (eg, open reduction and internal fixation, intramedullary nailing, locked plating), together with the expertise of the surgeon and the process quality of the institution, have a much higher impact on outcomes than individual implants.

Device Approval Process

Because of diminishing marginal values, novel products with similar biomechanical features are unlikely to alter patient outcomes. Ceiling effects prevent the demonstration of marked improvements in endpoints such as function, health-related quality of life, and radiologic surrogates (eg, fracture healing, restoration of the mechanical limb axis) with a new as compared to an established implant. Approval regulations such as the FDA’s 510(k) process or the European CE certificate still accept equivalence to available devices as entry criterion to the market, whereas manufacturers later solicit the same products as superior to their competitors.

Recent changes in device approval regulations demand proof of effectiveness of the experimental intervention. According to the first philosophical bridge principle coined by Hans Albert ("Ought implies Can"), the marginal expected gains with a new intramedullary nail compared to an established one represent a discussion stopper. Whereas an uncontrolled flood of implants and the approval of useless devices must be averted by regulatory and legal measures, there remains a delicate balance between avoiding extra costs to health care budgets and abolishing any commercial or scientific incentives for developers.

This is very similar to the type I (alpha) error: Accepting the 5% threshold exposes 1 in 20 patients to the risk of receiving an ineffective treatment, but lowering alpha to 0% means that no patient will have the chance to benefit from innovative therapies in the future.

It is noteworthy that neither regulatory bodies nor manufacturers currently have convincing strategies to define the additional value and utility of new devices and implants compared to virtually similar products already available on the market.

Understandably, consumers, health care professionals, payers, and manufacturers are concerned about current device approval regulations, monitoring of safety issues, methods to demonstrate clinical benefits, and so on.

Regardless of local or global financial crises, innovation in health care is socially and politically demanded. In addition, manufacturers are under pressure to release new health care products, whether drugs or devices, on a regular basis to stay competitive in the market. Health Technology Assessment (HTA) must not only consider the outcomes (including safety) observed with a new treatment, but the underlying theoretical construct and development efforts as well. The way to the product may be as valuable as the product itself, and should be accredited.

Investigating a New Approach

In addition to IDEAL, there is a need for a new approach to assess the utility and value of orthopaedic devices and related surgical procedures to distinguish between innovations that are likely to change the practice of care, interesting novelties that satisfy surgeons’ needs, and technical modifications that may overcome minor weaknesses in established technology and techniques.

In a multi-institutional, international project (METEOR, Matrix to Evaluate Technology in Orthopaedics) led by the Unfallkrankenhaus in Berlin, Germany, and McMaster University, Hamilton, Ontario, Canada, the authors set out to generate a tool that may:

  1. Allow the classification of the innovative potential of a new orthopaedic concept and/or implant
  2. Ease communication between various stakeholders
  3. Add to established Levels of Evidence (LoE) and Grades of Recommendation (GoR) in HTA

These methods should evolve from within the clinical community of orthopaedic and trauma surgeons to ensure acceptance. They also must be developed with professional methodological support to avoid scientific separatism.

The first step of the project was a thorough literature search to gather various definitions about utility and value in different disciplines, settings, populations and countries.

A key finding was the following simple but ubiquitous equation of value:

V = %I / %C

where V = value of a product or process, %I = relative importance of the new product or process (compared to an established one), and %C = its relative costs.

Another result was that potential innovations may simply be classified as:

  1. Radical (new biological principle)
  2. Substantial (improvement of established concepts)
  3. Incremental (technical modification)

Table 1 summarizes the dimensions, categories, and classifications of the preliminary METEOR matrix.

Table 1. Preliminary METEOR matrix to assess the value and utility of presumed innovations in orthopaedics


Need for a change in established principles

Grade of Innovation














Implantation technique










Key Points

  1. Innovation in surgery is a process that has recently been analyzed and formally described by the Balliol Group in the IDEAL statement (Innovation, Development Exploration, Assesment and Long-term Monitoring).
  2. The IDEAL statement may have limited application in bringing implants or devices to market.
  3. We propose METEOR (Matrix to Evaluate Technology in Orthopedics) as an adjunct to IDEAL to assist orthopaedic surgeons, patients, manufacturers, and payers in appraising the need for a change in established principles, grade of innovation, value and safety of novel products and procedures.


  1. Barkun J.S., Aronson J.K., Feldman L.S. et al. Evaluation and stages of surgical innovations. Lancet 2009;374(9695):1089-1096.
  2. Ergina P.L., Cook J.A., Blazeby J.M. et al. Challenges in evaluating surgical innovation. Lancet 2009;374(9695):1097-1104.
  3. McCulloch P., Altman D.G., Campbell W.B. et al. No surgical innovation without evaluation: the IDEAL recommendations. Lancet 2009;374(9695):1105-1112.
  4. Albert H. Traktat über kritische Vernunft. 1991. Stuttgart, UTB.
  5. Coccia M. Classifications of Innovations. Survey and Future Directions. Ceris-Cnr Workpackage 2006.

Reprinted with permission from the Fall 2011 issue of COA Bulletin