#HLG2016 Metrics made practical

As promised here are my slides from #HLG216 in Scarborough.

A few thoughts follow to build on the content in the slides for those not around at the time / to make them a bit more helpful generally.

As my talk progressed I became very aware of how the metrics work linked closely to my day to day. We operate in a functional structure and I am part of a team responsible for Partnership & Liaison. This translates for me into wanting to have lots of meaningful conversations with people.  Metrics are a means to that end. I didn’t mention it during my talk but this year I produced Annual Action Plan style reports (a la York) for the NHS Trusts I work with. These proved much more engaging than my old annual report. They featured a small number of carefully picked metrics with explanations of what I thought they might mean.

The presentation builds through the various models / methods we considered as we researched the use of metrics.  It was good to tap into experience in the room of why our measures can be unconvincing, hard to share and obscure. I was really pleased to find by chance (picked up from the weeding trolley at work) a 1990 text by some of the greats “Quality assurance in libraries: the healthcare sector” which strongly affirmed the areas people were focusing on and some of the approaches under way. Not much changes in the end.

We ended up with four overarching principles for good metrics

Meaningful – the core of this is that the metric must be something people (other than you) care about. It should be aligned to organisational objectives and be readily understood by stakeholders.  You need to talk them about it! An extension of this is to remember that framing metrics as a target should be approached with caution. There is the potential to set targets for the sake of it that will lack meaning for stakeholders. Tell them how you are performing and then discuss whether this is more or less than they need. Metrics need to be kept under regular review to reflect changing priorities and remain meaningful.

Actionable – for a metric to be useful for us it needs to be in an area we can influence. If we cannot make it move one way or the other it then we don’t want to be held accountable for it. A good metric will drive changes of behaviour and service development.  We also need to remember that the metric is only an indicator and we need to carry out appropriate research to back up what we suspect the figures are telling us.

No numbers without stories – no stories without numbers

Reproducible – this principle contains quite a lot.  It starts from the position that tracking a metric is a piece of research. Accordingly we need to be transparent about our methods and we need to be so before we collect the data. We should use the best data available to us.  Replication implies that we should get consistent results if two people examine the same thing at the same time.  Finally we also want our metric data collection to not be excessively burdensome. If it takes two solid months to crunch the data then it probably isn’t reproducible (or you would really have to get an awful lot from it).

Comparable – finally we want metrics that allow us to see change over time.  Often we will need to recognise that this can only be internally. We may be able to look to benchmark externally but we should be realistic. Even if we are transparent it will remain difficult to establish consistent data and there are frequently influencing factors that we may or may not be aware of. For example – what is the impact of being in a Trust three times bigger? or with three sites? or thirty? what kind of staffing model is in place? how is the service funded and delivered?

All this is a fair bit to keep in mind so the Metrics task and finish group prepared a Quality Metrics Template. This is designed to support people in creating, documenting and sharing their metrics.  The slides include a worked example and others were distributed in the room for the final group work section were people had a chance to start drafting some out or just discuss the principles.

In discussion the potential was seen to use completed templates as the basis of a process of refinement seeking best of breed metrics around particular questions.  Hopefully a tool will be available to collect them in the first instance and then an approriate group might be assembled. There was some concern that metrics might be imposed but this strikes me as unlikely.  The diversity of services and the needs of local stakeholders mean that one size will definitely not fit all.  There was discussion of the NHS national statistics return and the importance of considering these in the light of the principles.

I hope people will find the principles and template useful. It was great to talk to such an enthusiastic audience. More conversations please!

#HLG2016 cutting through the fog

A few thoughts while #HLG2016 remains fresh in the mind.  Hopefully there will be a good number of reflections shared in the coming days (Abi Alayo has been quick off the mark with her thorough posts for the first and second day.  I am likely to be less thorough!

The conference had a packed agenda but it is the window to talk to so many colleagues that really makes it.  This started ahead of time on twitter with the depth of adoption of this channel continuing to grow (NodeXL analysis of patterns of use, language and so on). The journey up offered time for initial conversations and the world was partially set to rights with Ben Skinner on a later than hoped for train from York. One of the topics was around the challenge of liberating the data that we hold and need to use (more on this in another post).

Safely arrived at the Royal there was time to help some guy push his slush puppy cart into an arcade and to buy three pints and half a coke for £7.70 (Seven pounds seventy pence London pub drinkers vidiprinter) down on the harbour side.

The next morning started with the pleasure of finally meeting Michael Cook after years of being in contact online.  Running along the sea front was a great way to open the day and to get a feel for the fog. This was one of at least a couple of semi organised runs by delegates and it would be a nice thing to continue at future events.

The conference venue was the Scarborough Spa which had a slightly faded glamour but met our needs overall (the wifi worked!). The shifted date was less successful with the Higher Education contingent clearly reduced due to student inductions already being underway or looming.

There was a strong Knowledge for Healthcare theme throughout the two days which may have been off putting for some.  This is balanced to some extent by the extent to which the products of KfH workstreams are publicly available and often widely applicable.  The volume and quality of work going on is impressive with tool kits in all directions.  Work on increasing the role of centralised procurement rang some alarm bells for me – we have moved from £2mill spent centrally to £4mill but the view is that some £12mill could be spent this way.  That money is unlikely to be new money!  Efforts to look to the future of staffing are also welcome with another leadership programme and a development path for senior managers on the way.  I suspect there are non NHS health folk on the current leadership programme and I really hope so as it is important to get a wider perspective where possible.

I forget how long it is since the introduction of “Do once and share” but duplication of effort remains stubbornly persistent.  The work on Current Awareness illustrated this clearly with over 700 bulletins under production just from the people who responded to a survey on this. Approaches around consolidating these while establishing best of breed models feels overdue.  The guideline on good bulletin production will be one to watch for.

My own session on metrics drew a larger crowd than I had hoped for with pretty much a full room. It brought home to me the need for additional efforts to explain how the principles we developed for good metrics can be applied. In a similar way to the CAS bulletins I could see the germ of a plan to develop best of breed metrics based around shared templates. There was some confusion over whether this was an additional piece of work or a replacement for national statistics returns.  In essence I hope the principles will be used as part of the national statistics review to inform any changes.  What I hope I expressed strongly during the talk was that the interest in metrics is mostly  the extent to which they can drive useful conversations – with our stakeholders and with each other.  Through out the conference we were reminded of the importance of being active in the boardroom as well as at the bedside. Having something concrete to talk about that responds to the priorities of senior stakeholders must be a good thing. I will make some tweaked slides for the web and blog them in the next few days.

After the rush of presenting we then had a fun conference dinner from which I may have escaped with slightly too much CILIP HLG rock!

HLG Rock(s)

A pair of keynotes kicked us off well on the second day with Nick Poole running through progress at CILIP and recognising the impact of health library and information professionals work. I hope the new model for subscriptions and improving offer will have the desired effect to widen participation and membership. This was followed by a moving talk from Lynn Daniel on the Expert Patient Programme. While I am sure I was not alone in wondering about the evidence base for some of the interventions proposed it was clear that her work has considerable impact on peoples lives with access to information at the heart of it.

The expert work of Judy Wright in supporting research proposals was fascinating and highly relevant to some of the thinking I am doing about search support for my own organisation. This along with a number of discussions I had made we wonder a little as to how up to scratch our skills are in these areas.  While there are some seriously well equipped librarians out there I certainly feel that I know less about systematic review and other advanced searching than I would like.  As we shift to delivering more highly specialist work and automation advances we need to ensure that we can maintain credibility. More thoughts on this to come in another blog post.

Other useful talks were Jo Milton from Cambridge on UX work (experience sharing planned for the future) and Andrew Brown looking at RoI.  The RoI work confirmed how hard this is to do well and the risks associated with starting to move into the realm of putting a price on all things.

HLG committee were keen to hear about ideas for what else we might do. The potential for holding HLG Conference annually was suggested. I wonder whether we might run something like the UKSG one day event which combines a trade fair with a selection of talks? A notable difference is that this is a free event to attend. We could look to make the overall cost lower (no lunch unless a sponsor covers?) to allow this. I would also like to see HLG campaign with members to increase uptake of revalidation. The concept of regular revalidation is understood and undertaken by many of those we work closely with. HLG can lead the way on raising use of this method and normalising it across the profession.

We emerged from the fog as the train whisked us off home. This felt like an important conference and reminded me how much I love working in the health information community. There is no doubt that significant progress is being made across many areas of work. There is also no doubt that financial pressure is going to be intense for most of us. The call to engage with NHS Sustainability and Transformation Plans, with the patient information agenda and with making the future we want to be part of has to be heeded.

 

A trip to the coast

Slightly ahead of the crowd once again I went to Brighton last week to visit the library services at Royal Sussex County Hospital.

I was keen to visit this service as they operate across the NHS and Higher Education so there are similarities to my responsibilities.  They are also something of a powerhouse in terms of clinical librarianship and their work on current awareness.

A major difference is that the medical school library service is run from within the NHS Trust set up.  The Head of Service reports to the Director of Education and Knowledge (a multiprofessional structure has been created) who was previously the Director of Medical Education.

Brighton and Sussex Medical School is a partnership between the two universities and Brighton and Sussex Universty Hospitals NHS Trust.  The student cohort is much smaller than our medical school which sounds great in terms of student experience.  They have found some elegant ways to smooth the issues of needing to have multiple network logins with harmonized usernames for the universities and a free standing BSMS email system. An open source discovery layer is used to allow searching across the three LMS in use.

Resource purchasing is conducted in partnership and Trust OpenAthens is used for authentication from the outset.

Wifi appears to be challenging in similar ways to at many NHS organisations. The availability of Eduroam simplifies some things but it is not widespread in the Trust buildings.

Training is conducted jointly with two thirds of attendees undergraduates.  The shared nature of resources supports this.  A very popular course they have added recently is around reflective writing for nurses preparing for revalidation.  This looks like something we should investigate offering with others in the room having to add multiple sessions of this.

It was great to hear from Tom Roper about the work of the Clinical Librarian team.  I was interested to learn that an early experiment with this had been conducted by Jean Farmer as the Will’s Librarian at Guy’s – I am looking into this history further as I suspect it is little remembered locally.  The take away for me was quite how slow it is to get these services off the ground and embedded.  Any move in this direction must be prepared to be patient.

I had heard a fair bit about KnowledgeShare at various other events but it was good to hear more about it in use and see the various ins and outs.  I would love to have something of this kind for our users but the impact of the additional work around registration would be significant.

Always fun to go to the seaside.

 

CILIP Update June 2016 and July August 2015

June 2016 Update (plus July / August 2015)

Thought I would have a change from reading about the chaos being unleashed by the EURef and flip through the latest from CILIP (and from them a year back).

June 2016 first.

Bit surprised that the consultation on the new membership model is given just a small news item. This seems a pretty big deal to me.  The new lower price is likely to be welcomed by many.  I wonder how strong the maths is on the number of people likely to take up the “leaders” option and how convincing the package of extra benefits is?  There will definitely need to be some strong recruitment of members.

Positive to hear how the Library at Ferguson supported the community following the shooting of Michael Brown.

Even more positive is the column by Dawn Finch as President on the importance of ethical principles in the profession.  I have been disappointed again by the decision of HLG to suggest conference sponsors should influence the content of the event.  For me this falls foul of the 7th principle

Impartiality, and avoidance of inappropriate bias, in acquiring and evaluating information and in mediating it to other information users.

Meanwhile back in 2015 when we had some semblance of order in UK politics.. Michael Gove was scrapping the book ban in prisons.

I really should have checked out the Impact Toolkit (launched in this issue) by now.  Glad to see Mary Dunne was involved – she spoke really well at HLG 2014 in Oxford.

Over the page is David Gurteen on Knowledge Cafes.  I attended one a few years back at the BDA but found it a bit overwhelming with numbers involved that day.  The focus on conversation is really helpful as we work on driving engagement at work.  A reference is lacking for the Zeldin book on Conversation which I recommend to all as a quick but worthwhile read.  I am very keen on the idea of conversation needing to take place in the physical world – definitely more benefit for me.  The more I can talk to people the better things go.

Interesting to read of efforts at Plymouth to provide core reading material as e-textbooks for undergrads.  It would be interesting to know how strong the actual usage was – a drop in paper circ for purchased titles was mentioned which you would expect.  The preceding article on ebooks was a handy reminder of a few issues and options as I look at what I can do for my NHS users.

 

Journal clubbing – strategic engagement: new models of relationship management for academic librarians

My team journal club discussed the following paper this week

Strategic engagement: New models of relationship management for academic librarians

Jeanette Eldridge , Katie Fraser , Tony Simmonds , Neil Smyth . (2016) New Review of Academic Librarianship.

Very much hot off the press having been published earlier this month!

We had a highly productive discussion reflecting the paper echoing our own ongoing work on development of new models around liaison and engagement.  I found the concept of “bridging conversations” helpful as an alternative way to present what I have always thought of as the translation service I operate between HE and the NHS. The focus on senior academics and professional services colleagues was greater than in our approach and we were left wondering who does carry out the liaison that is no longer covered by the team in the article?

It would be good to have seen more robust research around the extent of engagement roles in Russell Group (and beyond) institutions.  Investigation focused on information from websites when a conversation might have been both more straightforward and useful. We do not use engagement as a term in either our team name or job titles so may have slipped through the net.

The importance of resolving ambiguities within the library about the new model chimed and prompted discussion of how we might need to continue this work locally.  Also interesting was that the model appeared to have been compromised by the subject related needs of some disciplines.  I am very aware of some of the tensions around this in health with specialist knowledge around systematic review valued by those we work with.

There would be definite value in meeting with the team at University of Nottingham to share experience.

 

IMG_0415

May 2016 Cilip Update (plus September 2015)

Time for a quick read of the latest issue and one from the pile.

May leads on Shakespeare which is understandable given current celebrations (there are just a few days left to see the King’s College London “By Me William Shakespeare” with collaboration by the National Archives and the London Shakespeare Centre.

An exhibition with a few more days to run is “Scholar, courtier, magician: the lost library of John Dee”. It was great to read such a clear explanation of how this had been put together and the pleasures / pains of it all. @girlinthe had already been giving us a good flavour and this is a good example of how social media can get people involved in something well ahead of time.  I need to be sure to go before it closes in July.

The other main articles of note for me were about collaboration in health.  The work of Macmillan with Glasgow Public Libraries demonstrates the importance of specialist information support within a more general service. Meanwhile work in the Wirral links an NHS library with public libraries to support reminiscence therapy provides shows a new service being created with powerful effects.  Both of these illustrate the debate around how NHS LKS should work with patients and the public.  I understand the unwillingness of NHS services to open entirely to the public – the services required are very different and difficult conversations need the right preparation / conditions. I need to make the connections into the relevant services locally to see what we can do.

Nothing massively spoke to me in the September issue – these things happen.  Glad to see the progress of works at the Glasgow School of Art.