Data-Driven Bad & why we need to raise our game

The term ‘data-driven’ is today used almost exclusively in a positive tone; perhaps because it implies logic or decisions that are somehow based on impartial facts (nb. even though data != facts nor is it impartial). But there’s a substantially more sombre reality to all of it – one of data-driven bad.

You may not think of it, but you are a victim of data-driven bad fairly constantly; that completely irrelevant ‘targeted’ ad on Facebook or wherever? Data-driven bad. Amazon or your store loyalty program sending either naïve suggestions or pushing products you’d never dream of buying? Data-driven bad.

But it gets much worse.

From policy decisions to business strategies, being ‘data-driven’ can go wrong in many ways.

The most obvious situation is when someone – the government comes to mind pretty often – claims some decision is data-driven when in fact it is not. The data is either non-existent or made up. Why do they bother? Because it works; as noted by Joe Romm in his great book Language Intelligence, “many studies find that repeated exposure to a statement increases its acceptance as true.”

In cases when there some actual data is present, one of the most common pitfalls is selection bias, i.e. you pay attention to just the data or the results that best fit your preconceptions. As I wrote earlier, this tendency to ignore undesirable data can result in entire organisations acting irrationally.

Good data, bad data – but can you even tell the difference?

Data in and of itself is often not particularly useful. It needs to be analysed one way or another to make use of it effectively, or to uncover insights. The results of your data-driven endeavour depend on many things, but let’s look at the two obvious ones: quality of the data and quality of the analysis.


Seems straightforward enough, right? Just do good analysis on good data and it’s all good, right?

Well, kind of.

Problem is most data is not of particularly high quality. And even when you think it is, it may not be.

This is well illustrated by a recent discovery of a 15-year-old bug in software used to analyse functional MRI (fMRI) images; the cool brain activity scans in those “how the brain works” articles? That’s fMRI.

And the bug? It caused false positive rates as high as 70%.

How bad can it be, you may ask? Surely it’s just a matter of minor re-calibration of some results.

Unfortunately no. As The Register put it, “that’s not a gentle nudge that some results might be overstated: it’s more like making a bonfire of thousands of scientific papers.” The validity of some 40,000 fMRI studies spanning more than a decade are now in question.


When much of a field that prided itself on being data-driven and using state-of-the-art equipment to acquire the said data is now under the shadow of data-driven-bad, just how confident are you in your organisation’s capability to ensure the quality of data?

Actually, before you answer that, you should keep in mind that everything is broken – bugs, mistakes and errors leading to unexpected behaviour are everywhere.

Good analysis, bad analysis

Even when you do have good data, it’s not enough. Just like the standard financial results disclaimer of “past performance is no guarantee of future results” falls on deaf ears, so does the #1 principle of statistics, “correlation does not imply causation”.

Both so obviously true, and yet usually ignored – because the alternative is hard. When there is a compelling story to be extracted out of good data and a clear correlation, why bother with the analysis bit?

Not to worry, Big Data is here to make it worse….wait, what?

Close to a decade ago, Wired’s editor-in-chief Chris Anderson embraced this line of thinking, stating “with enough data, the numbers speak for themselves”.

He neglected to mention that if we let the numbers speak for themselves, even good data can lie through its non-existent teeth.

One major problem is that Big Data makes the discovery of spurious correlations so much easier. As noted by one paper on the topic, “too much information tends to behave like very little information”.

Hand on your hearts now, data scientists – how often do you really dig into the data and to verify that your correlation is, in fact, causation?

I’ll venture a guess; exceedingly rarely.

Demand more of ourselves

I’m not against data, quite the opposite. I’m not even against Big Data. But am I vehemently against using it just because, or to somehow replace insight or theory.

Luckily there is, if not a solution, at least a way to improve the situation.

As Nate Silver puts it in “The Signal and the Noise“;

Data-driven predictions can succeed – and they can fail. It is when we deny our role in the process that the odds of failure rise. Before we demand more of our data, we need to demand more of ourselves.

In reality, the world often does the opposite – we use data to make life easier for us, to demand less of ourselves. We let the data speak for itself, while doing anything from fabricating the data in the first place to neglecting to check whether what it says makes any sense whatsoever.

But when we lose understanding and forget the theory, we find it becomes very easy to mistake those correlations for causation and trust the ‘data’ – no matter how misguided it may be.

It’s time to demand more of ourselves; only then we can demand more of the data.

Posted in ICT-stuff | Tagged , , , , , | Leave a comment

Reality Check on Autonomous Cars

Not a week goes by without someone extolling the imminent virtues of autonomous vehicles. It’s gotten to the point where some see them solving pretty much everything. Which, obviously, they’re not going to do – so let’s take a quick look of what impacts we can really expect, and when.


What, by the way, is a train image doing on a post about autonomous cars? It’s because trains already provide the user experience that people are likely to most rave about when they get their autonomous car. We’ll return to this later.

Some, like RAND, provide good, balanced reports on autonomous cars. Others, like Accenture, focus on the eccentric (“Autonomous vehicles can expand consumers’ access to banking, using the car to pay for fuel and tolls” – wot?! :)) while yet others go for the utopia-model: this Investopedia article is a good example: autonomous vehicles will supposedly “dramatically reduce the number of cars” and make for “greener urban areas“, among other impacts.

Let me begin by saying I love the concept of autonomous cars, and I believe they will be a net benefit to society and will change a number of things – some dramatically, some less so.

Will they reduce vehicle ownership? Maybe, but it’ll be a reduction, not elimination, and the shift is likely to be generational in timeframe. While fewer young people now own a car, those predicting significant shifts in attitudes to ownership ought to keep in mind humans aren’t exactly logical creatures, and ownership of “stuff” is rarely driven by rational reasons alone. I can also see the reverse trend being possible, such as affluent people buying cars for their kids – kids too young to drive themselves – to get around with.

Will they reduce accidents? Very likely yes, and this is the single biggest human benefit of them. I fully expect that in another 50 years, driving yourself on normal roads is likely to be either illegal or ridiculously expensive because of insurance.

Will they reduce emissions? No. Of course, their driving style might be more efficient, but that alone is not a huge impact – especially when you factor in the increased miles driven (see below). Even if we assume they’ll all be electric, that won’t automatically have a positive impact – because depending on the way your electricity is produced, EVs can pollute more than traditional combustion engine-equipped cars. This is the case here in Victoria for example. Thanks, coal.

Will they reduce miles driven? Almost certainly not. This is even contradictory to the other often expected impact of reducing vehicle ownership.

How so? Barring substantial changes in human behaviour (e.g. dramatically reduced mobility), the only way to reduce the number of vehicles is to use them more efficiently – share them. Given their (then autonomous) movement from one passenger in need of them to the next, they will actually increase miles driven – and as such, also increase emissions, and congestion.

Even if we assume a more conservative “no sharing” deployment of autonomous cars, they’ll be driven more and will worsen congestion. Imagine: you drive – well, are driven – to work. Do you tell the car then to park in the busy CBD area at a cost of, say, $50 per day – or tell it to go find a cheaper spot further away? As long as it’s there to pick you up when you get off work, chances are you’ll send it somewhere cheaper. Or when you need to stop somewhere for 10 minutes and there are no parking spots? Just tell the car to drive around until it’s time to pick you up.

Not to mention that when driving becomes more convenient, people are likely to do it more. For example: why fly from Melbourne to Sydney, when you can just head out in the evening in your comfortable, self-driving vehicle that provides a lie-flat bed for you to sleep in? On the less extreme end of the scale, they will likely make longer commutes more feasible – again leading to more miles driven.

Will they eliminate traffic jams and ease congestion? No, as above. But they will make traffic jams more bearable so I guess that’s something.

What about the user experience; what will that be like? Calling for your car like KITT was summoned in Knight Rider will be neat alright, but the most meaningful impact is elsewhere: we all know driving in traffic jams, in technical terms, sucks. It’s stressful, and for some, road rage ensues. Having an autonomous car will improve that experience significantly – it’ll reduce stress as the “driver” can concentrate on other things: relaxing, working, reading, even sleeping.

Which brings us back to the trains. I suspect something along the lines of “my commutes are SO much better with my self-driving car” will be the most common thing the owners of autonomous vehicles will be raving about.

I find that rather ironic, because that “hey I can read a book or sleep during my commute!“-experience is exactly what users of public transport, where available and done right, have enjoyed for decades. Everything old will be new again 🙂

Over time, Level 4 autonomous vehicles will allow a complete re-design of the car interior which has already led to some exciting concepts. But again: if you make car travel something really enjoyable, it’s likely to lead more of it being done.


But most of all, none of this will happen overnight. The current commercially sold state-of-the-art vehicles are Teslas, which are “only” Level 3 autonomous vehicles. Level 3, while providing added convenience and safety, allows for none of the significant societal changes to take place yet – cars aren’t really truly autonomous until they reach Level 4, which is still some way off.

A number of manufacturers have stated they will have autonomous models on the market by 2020. Chances are the majority will be Level 3 autonomous. Taking an extremely optimistic assumption, let’s say that Level 4 autonomous capabilities – for all roads and all conditions – are perfected by 2025 (which I think is overly optimistic), and that they will be in every single vehicle sold in 2030.

There is little reason to believe that vehicle replacement cycles have the potential to dramatically quicken – and there would be production bottlenecks to overcome, too. That means that by 2040, “only” approximately 50% of the vehicles will be autonomous and we’d approach 90% penetration only by 2050. Some benefits of autonomous cars, such as increasing road capacity (through smaller distances between cars and potentially higher speeds) won’t reach full potential until the penetration is dominant.

If 2050 seems like “too long”, consider that the average age of vehicles is about 10 years, give or take a couple years (11.5yrs in the USA). And there’s a long tail – did you know there are 14 million vehicles over 25 years old on the road in the USA, and a total of 58 million cars over 16 years old? (58 million is over 20% of the light vehicle “install base” of 258M in the US).

2050 doesn’t sound like an overnight revolution to utopia, now does it? And remember, this is under extremely optimistic assumptions.

In other words, remember Amara’s Law:

We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.

While realism doesn’t sell papers, urban planners, take note: autonomous cars will not solve your infrastructure woes. Neither will Uber or taxi drivers all become imminently unemployed – although eventually they will, so I wouldn’t recommend it as a long-term career goal for kids.

The bottom line? As fast-moving and exciting as the autonomous cars thing is, it won’t happen overnight – and the impacts are not so clear-cut-positive as many would have you believe.

Posted in Culture, Technology | Tagged , , , | Leave a comment

The New Media Diet

It’s news to nobody that the news and media industry has gone through, and continues to go through, some tough times. Whether as a result or a cause of it, most content today – excuse my French – sucks. The quality of your average news or other media/content outlet today is shockingly bad.

Sturgeon was such an optimist.

The current situation would easily have you believe that journalism is a lost skill, as there are precious few examples of good journalism around. I cherish the remaining ones that there are, but by and large… *sigh* Among the exaggerating and fear-mongering tabloids to the ostensibly neutral but agenda-driven national newspapers and television to the click-baiting headlines online, I sometimes feel the media industry today deserves every little bit of disruption it’s getting.

Great Guidance – Ignored

It’s not for the lack of good guidance – for example, there’s a good book called Elements of Journalism which outlines the essential principles and practices of good journalism. They are nicely recapped here by the American Press Institute. Go through the guidelines and reflect how well your average media outlet performs – or doesn’t, as it quickly becomes painfully clear most media is essentially failing their own criteria.

The guidelines start off with what ought to be a given; Journalism’s first obligation is to the truth. Except when you spend even a few minutes looking into an average story, you’re likely to find more than your fair share of errors, lies and half truths. Mostly, that criteria is a fail today.

Next up is Its first loyalty is to citizens. One doesn’t need to look very closely at the driving forces of media industry today to see that principle is mostly out the window, too. Governments globally de-funding institutions that are supposed to be unbiased doesn’t exactly help either. So, fail.

It doesn’t really get any better further down the list. Or what do you think of the performance of your average media outlet against principles of Its essence is a discipline of verification, It must serve as an independent monitor of power, It must strive to keep the significant interesting and relevant or It must keep the news comprehensive and proportional. Fail, fail, fail and fail.

These are, if you ask me, all great principles. Unfortunately, the vast majority of what bills itself as journalism today neglects to adhere to most of them. What gives? Do they need a refresher on their own guidelines? Acknowledging that people and organizations are never truly free of all bias, I would like to add one point to the list:

State Your Bias Explicitly

It’s shocking how few media outlets do this. Often the bias is clearly visible to any discerning reader, and yet they media organization claims to be neutral. Newsflash: such bullshit is very transparent. Admit your bias and spell out your values.

As a positive example, I offer The Economist: they are abundantly clear about what they believe in, where their biases lie and how they see the world. I applaud them for that, and wish all media had the ounce of introspective capability to produce a similar statement.

Stating one’s bias alone would be a huge improvement; not only would it show that the outlet acknowledges its limits in being “unbiased”, but it would also allow the reader to filter any information in the proper light.

The New Media Diet

So, media is broken. What to do? Throw hands up in the air, give up and tune into Fox News? With the old guard increasingly letting us down, where does one go to get a balanced view of the world?

A range of places. As Jimmy Wales, the founder of Wikipedia, once said, “You shouldn’t use anything as the sole source for anything, in my view.” One has to develop a diverse list of sources; let’s call it a New Media Diet.

But how? I don’t know, really, but here’s what I currently do in my continually developing quest for the best possible situational awareness:

  • Skip the “breaking news”. Life is better when you pay no attention to the “news news”. The vast majority of it is a) negative, b) meaningless noise and worse, c) bad-quality noise. Skipping all that also saves you a lot of time from focusing on irrelevant things . Besides, if/when there is a 9/11-scale event, you will hear about it anyway, and if your train is late, well, see below.
  • Use custom app alerts. Now, some “breaking news” are useful. I don’t want to hear if there’s a curiosity delay on a freeway I never take, but I do want to know if my train is delayed. I also don’t necessarily want to know about a bushfire 500km away, but I do want to know about one that could get close to home. Luckily, one can usually get an app (or construct an IFTTT rule) for the majority of such situations without getting all the irrelevant notifications.
  • Read quality papers. There are still a few; The Guardian does good work regularly, and New York Times is among the better ones also. Of the weeklies, I would single out The Economist and of monthlies, The Atlantic.
  • Read Books. Pick carefully, and try to find at least two books on any given topic – ideally ones that take contrary views. Good books are invaluable in providing ample context into complex issues and deepening your knowledge on various topics. I prefer physical books, but YMMV.
  • Use social media wisely. Twitter and other social media tools can provide interesting glimpses into how some people think, what they believe in and even some of those more useful breaking news – all very valuable. There is scope for having interesting conversations, too, but a word of warning is in order: it all depends on who you follow, so be very, very careful there – and don’t get dragged into incessant, pointless arguments which all social media is riddled with. If you can’t resist the temptation of being this guy in the comic, you’re better off not being on social media – or online forums – at all.
  • Read blogs and online sources – selectively. Blogs and other online sources are a good way to find out lots of interesting stuff. I subscribe to several hundred RSS feeds, spanning from expert blogs on technical issues and research organizations to more generalist thinkers and think tanks, to some blogs more focused on cultural observations, some on patents and so on and on. As with any other source, filtering is needed. As a tool to manage this, Feedly works well – so well that I’m kind of happy now that Google Reader was discontinued as it forced me to switch (I wasn’t at the time though).
  • Dive into research, statistics and open data. Scientific journals are good methods of finding out about interesting developments early on, particularly on the advancements of technologies underlying many future products or services. Governments also have bureaus of statistics as well as, as do organizations like the UN and World Bank, open data sources that can be very useful to do some deep dives into.
  • Talk to people. As different and diverse set of people you can find. It can be an eye-opening experience to notice how differently different people see the world. Few things beat a good face-to-face chat.

Keeping tabs of all of that is a lot of work, for sure. But it does, in my opinion, provide a much better view of the world than reading any one newspaper or other source would provide – and a much, much better view than reading one of those “good old” newspapers 20 years ago could ever have offered.

In that sense, maybe media isn’t broken. It’s just that good content is more distributed than before, and for the New Media to be useful, you need to do the mixing yourself.

You may also ask what developing a good situational awareness is good for – but the answer to that is a long list, so maybe a topic for another time.

Thoughts? Additions? I’d welcome any other useful sources and tips that people have.

Posted in Books, Building, Management, Personal, Psychology | Leave a comment

The Real Reason Why Customer Experience Matters

You’ve heard all about how customer experience or CX is critically important. But if you look around you, bad CX (and its cousin, bad UX) are so ubiquitous it begs the question whether it’s really that important. I mean what’s the worst that can happen if your customer experience is not up to scratch?

It’s not just about losing a customer to a competitor; it’s not even about your company going out of business, although both can certainly happen. It’s bigger. If your industry collectively sucks at customer experience, entire industries can falter – or to use the tired buzzword for greater-than-average change, become disrupted.


Really. Let’s see how that could happen / is happening. Most of us should have no difficulties in coming up with several entire industries that collectively don’t seem to give a damn about customer experience.

Little Love Lost for Utilities

Take energy companies. Exaggerating only slightly, all their customers hate them. For many consumers, they are the epitome of greedy business with powerful lobby groups, vested interests, price-gouging behaviour, limited competition and more. At best, they’re considered a necessary evil.

Whether they really are all that evil (mostly they aren’t) is however irrelevant, because it’s the perception that matters. And the perception is firmly that there is little love lost between utilities and their customers.

That’s a problem.

Now, many of us know the energy sector is facing a number of interesting and potentially quite severe challenges over the coming years. In Australia, some of the key drivers include the incessant rise in solar PV installations and an equally relentless decline in total electricity demand – all the while maintaining the brief peak demand levels.

Add to this already-uncomfortable mix the rapidly improving solutions for household-scale energy storage, and you can see why the energy sector is nervous. It’s slowly starting to dawn on the customers that maybe they are not an absolutely necessary evil after all; that maybe there’s a way to get rid of them.

The ultimate threat scenario is that eventually even suburban households will go off-grid in meaningful numbers, setting off a death spiral. In isolated areas with a high peak/off-peak charge delta, grid-connected household storage is already financially viable. As time goes by, it will only become increasingly so, and viability of off-grid scenarios will follow some years later.

The industry seems to take solace from the fact that for the vast majority of their customers, storage – let alone going off-grid – does not currently offer a good ROI, and won’t do so for some time to come.

They’re right.

But also so, so wrong.

Didn’t We Bury The Rational Consumer Model Already?

They’re right in that going off-grid in areas with existing grid coverage makes, on average, currently little financial sense. And it won’t for years.

They’re wrong in assuming that the economics of it would be the primary driver.

Assuming a standard, neo-classical consumer model is dangerous to begin with, given it has been shown to be dead-wrong time and again. It’s especially dangerous if your customers have a proven, strongly negative perception of your company.

If Your Customer Hates You, ROI Is Secondary

Anecdotally, once “affordable” systems come to market, my #1 motivation of going off-grid would not be based on ROI.

It would be the fact that my distributor is a moron.

While obviously setup affordability is an issue, the primary desire for going off-grid has little to do with financial viability. It’s simply because I would prefer to have nothing to do with my distributor.

(nb. yes yes, personifying a company by calling them morons is not “rational” and maybe not even be fair given they have many hard-working employees. Then again, maybe it is. Also, notably, I have nothing against my retailer, an entirely different company thanks to how the industry is structured here.)

Given the public perception of utilities, I would imagine I am not the only one.

Now, the energy companies could respond to the gathering of dark strategic clouds in an intelligent manner; first, they should urgently fix the customer relationship. Second, they could revamp the connection pricing structure to be based on maximum flux. Third, they could encourage increased usage (How, you ask? Encouraging adoption of electric vehicles is one sure-fire way of massively increasing usage, but it’s not the only one) and demand shifting to make disconnection harder in a positive rather than negative way, while also selling those batteries.

Yet, their main response so far has been substantially less intelligent, and chances are you’ve seen it in your electricity bill (fixed charges up). Long-term, this will just make their problems worse.

It’s Already Happening Elsewhere

If the energy sector scenario above seems far-fetched to you, consider that the same principle of bad customer experience driving big changes has already progressed further elsewhere.

Take taxis and Uber for example – the latter isn’t popular purely or even primarily because of often cheaper pricing, although it certainly helps. People choose Uber over taxis because their customer experience is better.

Or how about telecommunication companies and the so-called over-the-top players? As telcos failed to deliver good new customer / user experiences (anyone remember walled gardens, WAP content, the promise of RCS etc?), others took over a large share of the customer’s attention – and money.

If You Don’t Care, Someone Else Will

Don’t be the kind of company – or industry – that doesn’t care about customer experience. For your sake.

Because if you don’t care, sooner or later someone else will.

After all, all the over-use and debate about the term disruption doesn’t mean it can’t happen to you.

Posted in Business | Tagged , , , , , , | Leave a comment