If you look me up on Linkedin ( https://www.linkedin.com/in/philbutton/ ) you'll know that I worked in IT for 36 years - that's around half of the time that IT has existed, if you think about it. Computing, as it was in the seventies, wasn't something I ever really enjoyed, but, as with anything to which you commit more than half your life, I have a sometimes-useful mine of experience from those days. What I find frustrating is that lessons have clearly not been fully learned from the mistakes my colleagues and I made maybe 20 or 30 years ago.
Invoicing cock-up
One story from a role I had many years ago relates to invoicing. At the time I was working on a fulfillment system, which interfaced to invoicing - because, of course, once an order is fulfilled you need to invoice the customer.
The invoicing team told me they would be running some tests at the weekend, which didn't really affect me. I was just not to tinker with any data that they had set up specifically to check some new function or other.
On the Monday all was tickety boo, the invoicing team were happy, and the change was able to be implemented in the live system.
But on about the Thursday the invoicing lead was summoned to a meeting, and we developers were all sent an email (or an equivalent, I think it was before what we now know as email existed) telling us we were not allowed to do any testing until further notice.
We soon found out what had happened. Invoicing, at that organisation, interfaced directly with a mailing system, so that once an invoice was printed it was automatically put into an envelope and sent off in the post - and the invoicing team hadn't stopped that from happening in their testing! Real customer names and addresses had been used, so invoices addressed to them had been printed and sent out. Some had already paid them! It was possible to unpick what had happened, and all affected customers were credited, but, as they say in business-speak, there was some 'reputational damage'.
After this episode, at that business we were only allowed to move data from live to test if the names of the customers were altered - 'anonymised' was the technical term - and I tried to follow that rule in all my subsequent roles.
However, testing with 'live data' was still a regular event for all my time in IT, and I'm not sure if anonymisation was always employed. I do remember at one business an analyst was despatched to the print centre at the time testing was undertaken to intercept any letters that might arise from testing, and ensure they weren't sent out. I think many businesses do this sort of thing, but I'm not sure they all do. I am also not convinced that all members of test teams involved in this sort of thing really understand the possible effects of any oversight on their part.
A letter arrives
On last Monday morning Viv and I were getting ready for a few days away. The taxi was due to call for us at 10am, and, an hour and a half earlier we were having breakfast when the postman brought us a few letters.
One got me worried.
Viv has a number of related health conditions, including epilepsy (on medication for this, but she had not had a major seizure for three years until a one a couple of months previously), a brain tumour, poor memory and cognitive issues. As a consequence she doesn't use online banking or anything like it. This letter, from her credit card company, said:
'Action required - please update your email address in Online Banking.
Thank you for choosing to receive online statements. As you've opted for this service we need your email address.'
It went on to ask her to update her email address online, and, if she had any questions, to log on to online banking and send the bank a message.
Viv hadn't done any of the things the letter said she had.
Worried that someone was trying to intercept her bank statements - often done to facilitate identity theft - I called the number on her credit card statement.
Unproductive conversation
My phone call was answered promptly. The agent gave his name, in an asian accent so thick I could not understand him, and continued, asking us questions that we had to ask him to repeat. He asked for digits from Viv's telephone passcode number; she once had one of these, but seldom phones the bank (for much the same reasons as she doesn't use the internet); most of her banking is done in branch, or by me, online, using the power of attorney we have set up. We explained what the problem was, but the agent could not help us until he had identity information from Viv. He asked more questions, clearly following a script. We asked him questions relating to the letter but he did not answer them. Viv became very upset; with hindsight we realise we should have asked to speak to a supervisor. Eventually he asked her full name, date of birth, credit card number and expiry date, and the credit limit on her account. We provided that information correctly; whether he recorded it so I am not sure, for he told us he could do no more to help. I told him that we'd visit a branch to explain the problem, hung up, and then thought about completing our breakfast.
'I'm not hungry after that phone call,' Viv said, 'that bloke was no help whatsoever.' I wasn't hungry either. We tidied up our breakfast things and, perhaps less nourished than we should have been, got ready for the taxi and our trip away.
Branch staff helpful
The following day we found ourselves outside the bank in Sudbury, Suffolk, a couple of minutes before it opened. We joined the queue and waited our turn for the cashier.
When she was free Viv and I walked forward, handed her the letter Viv had received, and explained that Viv didn't ever do online banking or want online statements.
'Oh,' she said, 'we've seen a couple of these letters. Let me check your account.'
She typed something into her computer and waited for it to respond, then typed something more.
'No,' she continued, 'you don't have internet banking, and you haven't requested online statements. There's nothing to worry about.'
We thanked her and went on our way.
'Using live data in test', I said to Viv as we walked out into the sunshine. 'We've seen this before, with some other bill or statement. I bet they've loaded data from live to test, tweaked it to test a change they're making, but haven't remembered to intercept interfaces to the letter system, or the letters themselves before they're sent out.'
I wasn't surprised, indeed people come to expect 'computer cock-ups' from time to time. The IT staff involved probably get a bit of a telling off, and are reminded about 'reputational damage' - but, since all the banks have IT failures at some point, the issue may be shrugged off as somewhat inevitable, and the people who have the power to stop them happening may feel a little remorseful but believe that they have done no real harm.
That may not be the case. I need to explain what had happened on the Monday evening.
A trip to the supermarket
After our conversation with the call centre agent on the Monday morning and our non-breakfast, we'd got ourselves ready to go away. The taxi arrived, took us to the station and we travelled down to Suffolk where we were due to meet Viv's aunt for afternoon tea at 4.30.
Afternoon tea was a success, although the food was perhaps not quite in line with Viv's usual afternoon menu (so she didn't eat very much, and drank a couple of americanos rather than tea which she doesn't drink very much). Viv's aunt stayed with us until 7pm, she had lots to talk about and, not having had much social contact due to lockdown, seemed to greatly enjoy being out of her home.
When she left I said to Viv that we needed some bits and pieces for our planned walk the next day, so we should go to a shop or supermarket and stock up. Viv seemed a little out of sorts, but got herself together and we left the hotel together in search of a shop. I'd prepared her tablets which she normally took by 5pm, but thought it would be ok to pop out quickly and Viv could take them later.
There was no shop nearby; the town centre being half a mile or so away we headed there. I was walking briskly, and Viv keeping up very well; two thirds of the way in she started to speak a little oddly. She began to say the word 'yes' in response to everything I said. I even asked her questions requiring a descriptive answer, but all that came back was 'yes'. She was still able to walk well, but I was worried, for I had seen this before in her a couple of months before, when she'd next had a horrible seizure, and been taken to A&E.
I didn't know what to do. I didn't have any of her medication with me, I could either carry on and go to the shop, or lead her back to the hotel. I opted to go on, hoping that she would snap out of the speech issue (as she has done with minor seizures in the past).
We bought some water at the shop, where I looked around for a bench where we might perhaps rest, but there was none (probably down to Covid). We headed back to the hotel.
Two hundred yards up the road my fears came true. Viv suddenly stopped walking, groaned, and began to move her head as if to look to the right. We were beside a garden wall, I dropped the shopping bag and put my right arm across behind her to support her and help her to lean against the wall. She began to convulse; I kept my right arm where it was and also supported her with my left arm and shoulder.
A passer by came and asked if he could help; as best as I could I explained what was happening and said that she sometimes comes out of these (although she hadn't done when she had the similar one two months before). A few minutes later two police officers came by, and kept an eye on things for a short while before asking whether they should call an ambulance.
'How long have you been here - about ten minutes?', I asked the passer by. If Viv had been convulsing for more than that sort of time we would need an ambulance, in line with advice from doctors.
'Ha, no, more like half an hour!', he replied.
'Time flies when you're having fun', I remarked. 'Yes please', I said to the policeman, 'can you call an ambulance?'. Between us we laid Viv down on the pavement in the recovery position as best we could, head on her handbag, and I held her steady while we waited for the ambulance.
Next, to Bury St Edmunds
It took a further quarter of an hour for the ambulance to arrive. There was a competent crew of three, and Viv was in the back of the ambulance within a couple of minutes. Space was made for me, the policeman handed me our shopping and before I had even had chance to thank the passer by or the police officers we were on our way to the West Suffolk Hospital. Two of the crew were in the back, one holding and monitoring Viv, the other administering drugs to try to settle her; the first shot of diazepam didn't work, but a second, intraveneously, did calm her down and she settled into what looked like a deep sleep.
The staff at the Emergency Department at the West Suffolk were ready for us, she was in to resus straight away, where I tried to give them all her medical details. She slowly came round, seeming very sleepy, until she started to get cramps in her legs.
'That's because of the seizure, it'll have drained salts from her body. Maybe some painkiller, then saline, possibly IV magnesium', one of the doctors said. The sister plumbed in 500ml of paracetomol solution and they found some antiepileptic medication for her to take, because it was now way past the time when she should have taken it.
Viv's legs continued to spasm. 'OOOWWWW', she cried, causing staff passing by to look over, concerned.
'Massage her feet', the sister suggested. 'I get cramps sometimes and that seems to work for me.'
I tried this but the cramps continued, Viv's screaming unsettling staff, patients and carers alike.
'Have you any cold spray?', I asked. 'That always works for her.'
They didn't have spray, but sent a healthcare assistant to find a cold pack, while Viv was taken away for a CT scan.
'OOOWWWWW', came the sound from the scan room while Viv was set up. 'IT BLOODY HURTS SO MUCH!'
Strangely I was rather cheered by this, for I hadn't heard her string a sentence of any sort together since soon after we'd left the hotel. She was still in great pain though.
She came out of the scan room and was taken back to a booth in A&E, where the HCA was waiting, shaking a chemical cold pack. He handed it to the sister who put it in a pillow case and asked me to administer it.
The cold of the pack, gently wiped across the base of her feet and along her calves, seemed to do the trick with the cramps. Viv was unplugged from the now-empty paracetomol and into a litre of saline.
'Some of her clothes were cut off, have you got shoes and things for her to wear?', the sister asked me. 'If the scan and blood tests we've taken come back negative, and she settles and is able to walk a little, she'll be able to go once she's had that fluid.'
I said she could wear my fleece, we had her shoes and jeans, and we could get a taxi back to the hotel. It took an hour or so for the fluid to all go in, the doctor then confimed the scan and bloods were clear, and suggested that he thought her seizure might have been down to the stress of the day, and the delay in her having her medication. Feeling chastised for rushing Viv out of the hotel rather than taking her tablets, I waited for her to be ready to leave.
It was 1.30am when the sister said we could go; I pushed Viv in a wheelchair to the exit, we found a taxi and were soon on our way back to our hotel where we arrived about 2am. Our shopping trip had taken seven hours, our planned walk the next day was out of the question, we needed a good night's sleep and a restful day to follow.
In the morning, and for a few days after, Viv was much better, but still a little down on her usual self. Seizures, especially long ones like the one she suffered, can and indeed do cause lasting damage which might affect memory, speech and general cognition, in addition to the bruising and grazes that arise from the convulsions. Viv can do without such further injury.
Agglomeration of stresses
It's not ever possible to pin one definite cause on one of Viv's seizures. Prior to an operation in 2018 she would have them quite often, and they would appear to be caused by a combination of factors such as dehydration, insufficuient food, hot weather, tiredness and suchlike, all in line with the sorts of things they list on epilepsy-related websites.
Seizures tend to arise when a stress threshold is exceeded by the sum of all stresses the affected individual is experiencing. Everyone has a different seizure threshold, and stresses will affect one person more than another.
On this occasion the sorts of things that might have increased Viv's risk of seizure could have been:
- She hadn't eaten much
- She'd drunk more strong coffee than usual
- She had a covid jab 16 days before
- It had been a warm day
- She had been a couple of hours late in taking her medication
The first of these - not eaten very much - was at least in part down to the stress caused by the letter from the credit card company, and the issue not being resolved by the call centre agent. It is quite possible, but cannot be proven, that, had the letter not arrived, or had the agent resolved the matter, Viv would have had a better breakfast and, even with the other 'stresses', would not have suffered the seizure.
To put it another way, if the credit card company had done its job better, Viv might have been spared a horrible seizure and consequent damage. Two police officers, three paramedics and an ED team at the West Suffolk Hospital would have had capacity to attend to other challenges sooner.
I would like to think that anyone involved in, and particularly the managers responsible for, testing IT systems that generate letters to customers are well aware of the potential impact of any kind of error in those letters. Seizures aren't the only potential medical emergency that might arise, and, it may be fair to say, NHS staff have plenty to attend to other than resolving the downstream impacts of IT errors.
Recommendations
Thirty-six years in IT, and having the right to use the letters MBCS after my name, makes me feel somewhat qualified to suggest steps that should be taken to avoid or minimise the damage this sort of problem can cause.
- Any use of 'live data' for testing applications that interface to letter generation systems should be very carefully controlled. Names should be anonymised, all customer facing staff should be aware of such testing (so they may reassure customers in the event of any 'test' letters being mailed out), and consideration should be given to physical interception of such letters. (All of these options incur costs, but these costs should be balanced against the true damage that can be caused by mailing an incorrect letter, not just 'reputational damage'.)
- If banks are going to close branches, expecting customers to deal with call centre staff by telephone whenever they experience an issue (including for potential security issues), they MUST ensure that the call centre staff are able to communicate with any of their customers (by that I mean speak clearly, listen, understand what the customer is saying and iniitiate action as needed). Call centre staff must be able to reassure a customer in the same way that a cashier in branch will do - and that (from our experience) does not require many requests for date of birth or passcode numbers.
- Anyone involved in IT systems development, especially testing, must understand the potential repercussions of errors on their part. Their managers should also support them and ensure that their work is adequately resourced; thoughts that will be in their minds should include how they might feel should a loved one of theirs suffer after receiving an incorrectly sent letter.
- Work on systems that send other forms of automated communication - such as emails or text messages - should also be undertaken only with full consideration of the potential impact on any recipients of false messages.
Comments
Post a Comment