How do you license a black box?



Amidst all the recent hype and hope surrounding the use of machine learning and AI in medicine it is good to know someone is also asking how the heck we going to figure out whether the proposed algorithm or application is doing what it is supposed to be doing.

Enter Health Canada’s Digital Health Division (you will be pleased to know we have one), created a year ago to do just that (among other things).

At the recent CADTH symposium in Edmonton, Dr. Tyler Dumouchel (PhD), a senior scientific evaluator in that division gave a fascinating overview of regulatory challenges currently facing Health Canada as applications utilizing machine learning and AI are already being submitted (and receiving approval).

While noting that AI has the potential to revolutionize health care, Dumouchel said there is currently no established regulatory framework for assessing AI in medical devices. Hence the challenges, he enumerated which include:

  • How to balance encouraging innovation and facilitating market access with safety and effectiveness.
  • What pre-market authorization should be required from the manufacturer and should we be regulating manufacturing processes rather than the finished product?
  • How reliable and representative are the training data used to develop the particular application (a biggie)? Dumouchel noted such data need to reflect the Canadian patient population, be accurate, done on multi-centre basis, and include accurate disease prevalence data.
  • How do we know the algorithm used is generalizable and transferable and what is the best approach to use to ensure the algorithms used generate correct and predictable results?
  • What are the best metrics to use to assess the performance of an AI algorithm?
  • How can we ensure that AI is integrated appropriately into its intended environment without unintended consequences?
  • When and how do you assess to continuous learning algorithms where results can vary in time?
  • What sort of post-market regulation framework is appropriate for AI applications?
  • Who is accountable if the application makes a mistake?

These are all obviously important issues and it is gratifying that the federal government has acknowledged them and is taking them into consideration.

But there is some sense of urgency here. As Dumouchel points out, Health Canada doesn’t want to be left behind internationally when setting standards for medical devices that use AI algorithms.


Disrupting a futuristic health care system near you – patients #CADTHsymp


Artificial intelligence, genetic engineering, targeted wonder drugs? The opening plenary on the topic of disruptive technologies in healthcare transformation at arguably Canada’s top health technology policy conference held the promise of all of these.

But patient partnering and involving patients?


Perhaps it should be no surprise that discussion at one of the first Canadian healthcare organizations to wholeheartedly embrace patient involvement, the Canadian Agency for Drugs and Technologies in Health (CADTH), should settle on patient participation as the biggest potential disrupters in health care in the years to come.

Opening presenter at the CADTH symposium in Edmonton, Dr. Irfan Dhalla, VP of Evidence Development and Standards at Health Quality Ontario, set the stage by stating that biggest change to health care in next few decades may be partnering with patients for not just their own care but in how we organize health care and run health care organizations.

Dr. Dhalla said he came to this conclusion after crowdsourcing input through Twitter on the most disruptive technology to come and found that patient input was by far the most popular response. During the session when he delved into more technical issues of how reassessment of health technology assessments should be done, he stated it must be done “with patients at the table.”

Other speakers on the plenary panel also touched on this theme. Carole McMahon a patient, caregiver and patient advocate stated that in future patients should have new roles such as in helping design clinical trials or in designing patient surveys.

At the end of the session when asked what changes the panelists would make to improve health care, involving patients was among their answers.

Despite being a strong leader in patient partnering, CADTH was not spared criticisms from the patient community at this year’s conference.

A conference session on patient engagement that was listed with four speakers and no obvious patient presence was the subject of a critical tweet prior to the main conference that generated widespread concern from the patient community. “I find this unrelentingly frustrating,” was one response. “These kinds of presentations have the potential to actually ‘model’ what they’re allegedly talking about.”

Fear not though, the patient presence remains strong on the second day of the conference tomorrow beginning at a breakfast session at which Zal Press (@PatientCommando) will be hosting a session on the “thorny issue” of patient compensation in research and health care.

And lest people fear, the debate between patients might become too intense as it did at times at least year’s Halifax meeting, CADTH now has a meeting code of conduct that notes “harassment or disrespectful conduct do not belong at any CADTH event” including “sustained disruption of a speaker” and “grouping or isolating.”

Patient portals – windows to health care


Patient portals that offer the ability to access information from a hospital or physician provide a valuable window to at least part of a person’s health and care. But such portals remain unavailable to many and often fall short of being comprehensive.

Those were conclusions that could be drawn from a lively chat on Twitter that I helped coordinate on behalf of Health Quality Ontario (HQO) on March 20. Hosted by HQO Interim President and CEO Anna Greenberg (@AnnaGreenbergON) and respected patient advisor Alies Maybee (@amaybee), at #HQOchat the discussion saw more than 500 tweets from almost 100 participants over the course of an hour.

However, that’s just part of the story as tweets on the topic continue to be seen on the @HQOntario Twitter account days after the chat. The topic of configuring patient portals also promoted a lively exchange between a largely Ontario-based group of participants on Twitter earlier this year.

Greenberg and Maybee set the stage for the chat in a background blog that also framed the debate by asking about people’s experiences with portals, the scope of portals and their functionality. The portal discussion inevitably overlapped a discussion on electronic access to one’s health record even though this is often not a function of hospital-based portals.

The chat involved many patients, physicians, representatives from one of largest patient portals in the province (@MyUHNPatientPortal), Canada Health Infoway (@Infoway), other senior HQO staff, and others including representatives of quality care organizations and the @MyOpenNotes initiative.

As Ontario currently has no province-wide patient portal initiative (unlike Quebec, Nova Scotia and Alberta) it was not surprising that patients described a wide range of experiences with the portals. While some people talked of having up to six different portals, others said they had no access to such a portal.

Many expressed thanks for having access to what a patient portal has to offer. One of the most positively received tweets came from a specialist at the hepatology clinic at UHN who stated:

Our staff have embraced it (clinicians and support staff): fewer calls: patients happier: less printing of results for patients: patients pointing out errors. Overall very positive

Others felt the experience fell short of expectations. To quote one comment:

I have had access to a patient portal and while the concept is very exciting, I have been disappointed with the reality. The patient portal I have used has a limited # of parts of my record and lacks notes and referrals etc. so it is not a complete picture 

And Maybee commented:

I found that the hospital needs to update appointments right away. I went for an appointment listed on the portal, waited 45 mins until someone told me “Oh, that was cancelled but we are behind a lot on updating the portal.” Way to discourage us! 

The chat closed with a discussion of whether patients should be able to access their own laboratory or other test results before these have been discussed with their physician and the response was – with a few dissenters and caveats – an overwhelming ‘yes’. The same was true on the issue of whether patients should have access to their physician’s notes.

Dr. Irfan Dhalla (@IrfanDhalla), a general internist and VP at HQO commented:

Good news is that overwhelming majority of physicians are supportive of patients having access to their own notes. Some express nervousness, but usually this goes away after some experience.

It is clear that recent efforts to improve the availability and functionality of patient portals are paying dividends. But while they are useful tools, these portals managed by hospitals, clinics or medical record vendors are no answer for patients who seek a complete record of their health and care that they control.

And as another Canadian-lead discussion on Twitter a week ago makes quite clear, the numerous barriers facing patients who want to easily correct errors in their own medical record show how far the culture and environment need to change to truly make the new digital health environment patient-centred.

As Maybee commented:

We need to see the great stuff in portals all in one place — perhaps an aggregator?

To which another participant commented:

Bingo – I’m going to go one step further (sorry) but what I really want is one record, regardless of hospital, organization or discipline. If I go to 6 hospitals, I want to have a portal with the complete picture and full record. Call me a dreamer! 

And two other comments which summarize this point:

Allow patients to control their own data – we’ve spoken to thousands that want to share and export to their circle of care. System level change takes time re: connectivity, but empowering patients to share can be done sooner. Patient control = patient centred. 

While most of Canada have no access, some of us here in the know have several portals. That’s great! And it’s confusing. With @access2022 & @infoway talking to vendors now about creating the #digitalhealth highway for HC data in Canada we need #interoperability #HQOChat



#AI in dermatology: Imag(in)ing a future


“…to promote the rapid and accurate reading of … skin lesions.”

That was part of the answer Dr. Eric Topol gave in a recent New York Times interview when asked where Artificial Intelligence (AI) shows the most promise in medicine.

Dr. Topol expands on this in the first part of his new book Deep Medicine dealing with AI in medicine where he describes dermatologists, along with radiologists and pathologists, as “doctors with patterns” or those who rely heavily on images in their work. He said such doctors could potentially benefit from AI algorithmic support.

“Many,” he writes “were frankly surprised by what deep learning could accomplish: studies that claim AI’s ability to diagnose some types of skin cancer as well or perhaps even better than board-certified dermatologists”. However, he also noted that the work done with deep learning and AI to diagnose melanomas represent computer-based validation rather than using AI support in a real clinical environment.

The complexities involved in doing just that and the hope it held for dermatologists who fear losing out to technology formed the backbone of two standing-room only sessions at the recent American Academy of Dermatology(AAD) meeting in Washington DC in early March where AI was formally on the agenda for the first time. While the ability of algorithms to identify melanomas as well if not better than dermatologists has been grabbing headlines for a couple of years, the subject matter was still novel enough at the world’s premier gathering of dermatologists that a portion of each session was needed to discuss the basics of exactly what was being talked about.

Dermatologists were also chastised by one of their own for doing a lousy job of what is needed to fundamentally support the use of AI to support a pattern-based specialty – namely taking and a categorizing images of skin lesions.

Dr. Allan Halperin, is head of the dermatology service at Memorial Sloan Kettering Cancer Centre at president of the International Skin Imaging Collaboration, a group that has pioneered the collection of dermoscopy images to develop algorithms to diagnose melanoma. “Frankly its embarrassing that, as a specialty, we are far behind what can be done already,” he said, commenting on the image-gathering capabilities of dermatologists.  “We’re not taking images of enough things in a standardized enough way to make them available in a pipeline to the computer scientists to build an AI.”

Dr. Roger Ho, assistant professor of dermatology at NYU Langone Health in New York, agreed. In a news release he noted “hundreds of thousands of photos that have been confirmed as benign or malignant are used to teach the technology to recognize skin cancer, but all of these images were captured in optimal conditions – they’re not just any old photos snapped with a smartphone.”

To address these issues, Dr. Curiel said community-based dermatologists need to be involved in building the databases that will inform machine learning for melanoma. It was also noted that how the databases are compiled will depend to a degree on who will be making use of the data – patients, primary care physicians or dermatologists.

Dr. Ho and others – including the incoming president of the AAD Dr. George Hruza feel that even if the use of deep learning and AI is optimized for use with skin lesions, dermatologists will always be needed to properly interpret the findings and counsel patients. In his address to the meeting, Dr. Hruza said he was excited to imagine that when it comes to AI, dermatologists will have a “seat at the table and will not be on the menu.”

“The future of augmented dermatology is an opportunity not a threat,” said Dr. Clara Curiel, director of the dermatology program at the University of Arizona.  “AI should reinforce clinical skills.”

It remains to be seen whether this almost universally expressed optimism holds true for a specialty already facing inroads from numerous other pretenders in the art and science of diagnosing and managing skin conditions.

(Photo: Patterns of light and dark at AAD meeting, Washington DC)



“Are You The F**king Doctor?” A review


Farrell   “Cometh the hour, cometh the man.”

It would be the very height of pretentiousness to apply this phrase to Dr. Liam Farrell, an author and former family physician from Rostrevor, Co. Down, Ireland and I am sure he would be the last person to do so.

But at a time when family medicine seems to be at its lowest ebb, if not globally then very much here in Canada, there is much to be said for having a physician who can so eloquently write about both the rigors and the joys of general practice – and using erudite quotations and references to do so.

I would argue that Dr. Farrell does this in “Are You The F**king Doctor? Tales from the Bleeding Edge of Medicine” – a collection of his columns, blogs and short stories published over more than 20 years in publications such as the BMJ.

This is not necessarily an easy book to read, filled as it is with a juxtaposition of not just blood but all manner of unpleasant bodily fluids ,and anecdotes which are usually brief – sometimes too brief to absorb either the story or the message contained therein. Dr. Farrell combines these descriptions with a degree of learning and knowledge that one takes for granted with a well-educated physician.

Take for instance this description of his office (aka ‘surgery’)

“The surgery had the spiritual ambience of a monastery chapel, and if C.S. Lewis and J.R.R. Tolkien had walked in twittering gently on about morality and suspiciously androgynous fairies, I wouldn’t have been surprised.”

All of this, mind you, in a short piece describing a woman bringing him a sample of one of her husband’s bowel movements.

If some of this book is not easy to read or comprehend, that challenge surely begins in the very first chapter where Dr. Farrell writes with grim realism about his own battle with morphine addiction. Once past this however, one finds oneself immersed in stories which are often very funny yet which deal with  issues of life and death that family doctors must face every day.

Dr. Farrell talks about family medicine in a manner that would resonate with those Ontario family doctors currently tweeting in despair about the future of the profession:

“… it is not a specialty, but rather a generality, with all the burdens and privileges that this role confers, demanding skills that reach across the divide, not just of a dissonant phalanx of fragmented medical specialties, but of that deep romantic chasm between science and art …


“Real medicine is the process of making decisions among diverse influences. It involves discussion, calm deliberation and the capacity to balance valid but competing interests.

In this era of Choosing Wisely Dr. Farrell also writes from the trenches about patients who insistent on antibiotics to treat colds or x-rays for sprains.

Whether we are dealing with truth or Dr. Farrell’s feverish imagination is often hard to distinguish.

As he writes in a piece defending a much-maligned column involving trying to drive over a cat and instead hitting a young fox, Dr. Farrell states “I write by stream of consciousness; the caprice, the knights-move thought, the harsh imagery, something to grab the reader and want them to read on and assimilate the message.”

I have had the pleasure of meeting Dr. Farrell in person (the photo accompanying this article in no way reflects our interactions) and there are many others who know him as the gracious host of #Irishmed, one of the most successful regular tweet chats still dedicated to matters medical.


As a non-physician I can recommend this book to anyone who enjoys a good turn of phrase. But I would particularly recommend it to the family physicians who feel nobody understands them or the value of what they do – it’s not CME but the book may prompt more than a chuckle or two.



Canada Health Infoway (@Infoway): Back to the Future again


How you greeted the news last week of Canada Health Infoway’s rebranding and new campaign to mobilize support for digital health will probably depend very much on how you have viewed the success of this pan-Canadian federally funded health organization to date.

Over its 17 years of existence, Infoway has had more than its share of detractors for either failing to do enough with hundreds of millions of dollars it was charged with co-coordinating to bring digital health to Canada, or for focusing on the wrong things at the wrong times. Or take electronic prescribing: That Infoway’s PrescribeIT initiative has now spread to three provinces will be viewed as a major success by supporters of the organization. However, detractors will point out Canadian is many years behind other countries in ePrescribing, PrescribeIT involves few physician practices,  and at least one major national pharmacy chain has evidently failed to endorse Infoway’s funding model.

Part of the conflicting views of Infoway can perhaps be traced to the way it has radically reinvented itself over the years.  In his address to the Infoway Partnership Conference in Montreal last week, Infoway President and CEO Michael Green acknowledged that over the course of its existence, Infoway’s role has radically shifted focus; from building the infrastructure to support digital health, to providing clinician tools, and most recently to driving access for Canadians to digital health.

The latest iteration of the driving force behind Infoway – ACCESS 2022 sees the organization firmly positioning itself as an advocate for building a coalition committed to promoting “a future where all Canadians have access to their health information through the availability and use of digital health tools and services, which will empower patients and improve health outcomes.”

“ACCESS 2022 will bring together the collective expertise of an agile technology sector, the knowledge base of health system experts, and the insights and experiences of patients and caregivers, to ultimately meet and exceed the demands of Canadians in the 21st century…We are asking all Canadians to join us in this movement,” Green said.

Unveiled with touques for all and the music of Stompin Tom Connors at the Montreal conference, it was all very archetypically Canadian – and once again was doubtless viewed with either enthusiasm or cynicism depending on how you view Infoway.

We actually have a credible objective assessment of the organization presented in Fit for Purpose, a report published in March by two highly respected health policy experts Drs. Danielle Martin and Pierre-Gerlier Forest.  At the request of the federal health minister, Martin and Forest assessed all 8 federal pan-Canadian health agencies including Infoway.

They noted that Infoway has had several signature successes, including:

  • Driving the digital health agenda in Canada by creating pan-Canadian leadership
  • Parnering with all jurisdictions to achieve “close to” the 2004 federal/provincial/territorial health accord goal of 100% availability of electronic health records thereby resulting in $19.2 billion in cost savings and efficiencies since 2007
  • Initiating a patient engagement framework that “provides Canadians with access to their health information and digital health solutions that empower them to be more active members of their care team.”

However, while not criticizing Infoway directly the report goes on to identify significant shortfalls in how Canada is doing with digital health:

Data currently exists that could be used to improve care in hospitals, primary care environments, community settings, and health regions across the country. Unfortunately, the underlying architecture to support this meaningful use and enable continuous improvement across health systems does not yet exist in Canada. Infrastructure has been built and electronic health record systems purchased across the country, but two key outcomes are still missing: an inter-operable set of electronic systems and a “single” accessible electronic record for every Canadian patient as a critical means to achieving coordinated, integrated care.

How successful Infoway is in addressing this – and it is clearly committed to doing so  – will very much determine how it is viewed in future years.

(Image from website)

MyHealthNS – A perplexing patient portal poser


“Most QI (Quality Improvement) programs don’t incrementally ascend, but often fail. Many QI projects never work out at all and of those that do, almost all have real failures, along the way. But failure is not bad and contains within it the seeds for future improvement.”

Dr. Joshua Tepper, former president and CEO of Health                                                             Quality Ontario and now CEO of North York General Hospital

Those committed to bringing a patient portal to the residents of Nova Scotia should take heart from these recent remarks by Dr. Tepper. While he was talking about quality improvement initiatives, the comment speaks directly to the Nova Scotia situation and is probably more relevant than people would acknowledge to all aspects of digital health and health technology innovation.

Surely, nothing seemed better set up for success than the MyHealthNS portal intended to allow patients to communicate with their physician virtually, receive laboratory results, book appointments, and allow physicians access to Econsultations and Ereferrals. Canadians have overwhelmingly indicated that this – in some ways more than anything else – is what they want from digital health.

Yet despite government backing and strong early support, MyHealthNS has enrolled only a small number of Nova Scotians. And while the initiative is by no means dead it must, to date, surely be judged a failure.

The reasons behind this are informative and speak volumes about the importance of local variables and the challenges of bringing change to a complex system such as health care.

The MyHealthNS story was told recently by Dr. Stewart Cameron, a family physician and advocate of the portal, at the Canada Health Infoway (@Infoway) 2018 partnership conference in Montreal. A very, very abridged version of that follows:

MyHealthNS was piloted in 2012 with about 30 family physicians and 6,000 patients. Results were overwhelmingly positive and a full provincial roll-out of the portal was initiated in 2016 (and again in 2018 and again in?).

Two important points:

  • Enrollment in MyHealthNS is done through family physician offices which made it easy to authenticate patient identities but has had negative consequences leading people to question whether this is the right model.
  • Physicians involved in the pilot project noted that the portal must be integrated with their EMRs to be successful and that a payment model was needed to remunerate physicians for using the portal. In fact, one physician involved in the pilot saw their income drop as a result of participation.

Fast forward to this year.

In February 2018, only 13,782 patients were registered with the portal out of a provincial population of approximately 1 million people (that figure is now up to about 20,000, according to Dr. Cameron). Of these only 10% had accessed the portal at all in the month prior the evaluation in February. Additionally, only 250 practitioners had registered with the portal and of these only 35% were using the portal actively.

The reasons behind these poor numbers can be traced to two key reasons:

  • Payment for physicians for enrolling patients or using the portal is still not part of the master fee agreement although physicians can now receive a stipend for involvement.
  • While the portal is now integrated with most EMRs in the province, one major system is not integrated directly. Further, many physicians are currently migrating to different EMRs and are reluctant to take on the extra burden of enrolling patients in the portal during this time.

Of course, with physicians acting as the gatekeeper for patient enrollment it is obvious their reluctance to be support this initiative actively is a huge barrier to success. At a time when family physicians are under increasing stress and many suffer from burnout, embarking on such an initiative must produce tangible, immediate benefits to attract them.

Not only are family physicians reluctant to enrol their patients (which must be done face to face or by phone) but many residents of the province currently lack a family physician and so have nobody to sign them up.

What must have been a source of frustration for Nova Scotia delegates to the Infoway meeting was that the same meeting heard how Quebec has already signed up 300,000 people to their portal in a short time using a direct enrollment model.

MyHealthNS has failed to date but that is no indicator of future failure as long as those involved with the project learn from their experiences to date. In the words of Dr. Cameron: “This is going to happen. It’s inevitable.”

(Image courtesy, the Government of Nova Scotia)