인공지능의 투명성에 대해서

얼마전 Human-Computer Interaction쪽의 최대 학회인 CHI를 다녀왔습니다.  그 곳에서 알게 된 한 지인께서 페북에 재미있는 글을 올리셨고, 제가 반론을 제기했던 일이 있었습니다. 간략하게 이야기하자면, AI전문가 외의 사람들(언론, 일반인, 타 분야 학자 등)이 꾸준히 제기하는 AI의 불투명성에 대한 문제점이 논점을 벗어났다는 게 지인의 주장이었습니다.  제 반론은 투명성Transparency에 대한 더 엄밀한 정의가 필요하다. 아마도 predictability of the model's behavior to unseen events정도가 아닐까 였습니다. 

지인의 주장 

Here's my response to "ML algorithms are okay, but models are unpredictable and not transparent, thus the problem.": Models are abstractions, "all models are wrong, only some are useful." *For ML, data is the model.* Unpredictable is actually not what happens with ML models, their prediction just does not sometimes align with what "humans expect". These are two different problems. Humans are so stubborn that we believe what we expect (or accept as social norms) is the ground truth. But how do we know? Do we fully understand how our brain works? Don't we also use models to explain how our brains function? Are these brain models more predictable than ML counterparts? (If you think so, 4th paragraph) I want to believe we are, but the rise of artificial intelligence imposes some really interesting questions for ourselves to find out.

ML models are in fact as transparent as the algorithms used to train them. Probabilistic does not mean unpredictable, it means predictable under certain assumptions. These assumptions are inevitable in any kind of models, because they're, well, models. But, efforts exist from the ML community (while sadly none exists from HCI). For example, we have visualization tools to show how activation in NN works on each node with different parameters what that means in terms of the output results. For people who "blindly use" such algorithms might not understand them. (I think that's a problem - there's this field called usable ML to mitigate this problem) Knife is a system built using "sharp edges", if you don't understand how "sharp edges" work, you might not be the best person to design the system. (Of course, democratizing "sharp edges" so people can learn to design with it is an extremely important effort, and I think that's one of places where HCI community should stand.) We also have social systems that prevent people from using "knives" or "sharp edges" in malicious ways, social scientists are leading the discussions (Why not us?)

Systems using ML algorithms or models CAN BE made usable. But, do you understand how every function in Photoshop works under the hood? But why doesn't anyone complain? Because users are okay as long as it does the job. But with ML, even if it does the job, people complain "it's not transparent" (which is false). Why do some people do that? Could it be just a difference in mental models? Amount of user control? Designing systems with ML can be and should be exactly the same as designing any other systems.

Do we understand how humans make decisions? Why were we okay with human judges making biased decisions so far, but now upset about algorithms doing the same?? This very funny, because we're actually using previous human judgments to feed into ML systems, and now people think ML is biased. No, WE are biased! We were only able to realize we made mistakes because we can finally examine the phenomena in a large scale data-driven way. How is the ML system, in this case, the villain? If our brain was so transparent and "predictable", why did we even make biased decisions in the first place? Why are we okay with babies throwing a cup on the floor, and not okay when a robot arm drops it?

Do you see what I mean? We need deeper discussions on how we view ML technology and I thought CHI was the perfect place for it! I don't think ML is the solution to all problems, I'm not an advocate of DL, I just think the two communities are blaming each other, where we should really be collaborating.

I think there's an enormous opportunity to really investigate how humans make decisions, interact with each other, interact with technology, novel ways to do tasks previously unsupported because of the large-scale probabilistic computational capabilities. And as HCI researchers, we should be extremely excited!!

What really upsets me is the tone and the attitude of the HCI community I observed. What I experience at CHI was no different from journalists who repeatedly advertise false premises like "AI kills people", "AI destroys jobs". We should've done much better than that. Without any effort in understanding what's really going on, CHI was just bashing another field, misusing the terms to try to make its ground stronger.

I love discussions like this. I wish more people raised questions instead of pointing fingers. Also, I wish someone more knowledgeable had explained it, I'm sure they would do a much better job, and be able to point out what I and others misunderstood.

제 반론.

What a nice opportunity for debating. After reading the response, I came to realize that we need to clarify some concepts here.

First of all, let’s distinguish ML algorithms from models. The below is the process of developing ML applications that I just made up.

 Dataset -> (Algorithm) -> Model -> (Product Development) -> Application

From a given dataset, an algorithm extracts knowledge, and returns a trained model. Then the model goes through product development to create a useful application. All ML algorithms are transparent as their instructions are - which means their low-level behaviors (e.g. how back propagation algorithm updates a specific node for a specific iteration) are predictable (or at least deterministic). However, being able to read the source code does not allow human (i.e. ML scientist) to predict high-level behavior of learning such as “how quickly overfitting would occur”, “what is the ceiling of F-score", "what bias the model contains," or "how much of the problem space the model covers". Thus it is a common practice of ML scientists to try out a wide range of parameters to get the best model. Being transparent does not mean it’s predictable - especially when it interacts with unseen noisy training data. A related topic is DIKW pyramid (https://en.wikipedia.org/wiki/DIKW_pyramid). ML scientists (as users of learning algorithm), UX designers (as users of ML models), end-users (when ML applications break down) need knowledge or wisdom. However, raw algorithms and models are raw data or at most information.

Let’s talk about models. Same as learning algorithms, ML models are completely transparent. I can manually execute a model on paper like other low-level behaviors. Nevertheless, being transparent at data level does not automatically give answers for high-level questions, which require knowledge or wisdom. For instance,

Q. Can ML scientists predict if a ImageNet model would correctly tag an image out of its dataset? 
A. No. When the task is out of the training dataset, the model’s outcome is unpredictable. If there’s a way to explain the boundary of unseen problems that a model can accurately answer, that would be an amazing breakthrough.

Q. Can ML scientists explain why a certain image is tagged incorrectly? 
A. To my knowledge, this is an open-ended research question that infoviz researchers are trying to solve. Highlighting activated nodes is as useless as showing dump data of runtime errors. In his golden rules, Prof. Ben Shneiderman has explained why understandable and actionable feedback is important (https://faculty.washington.edu/…/schneidermanGoldenRules.ht…)

Q. Can ML scientists fix a model to perform correctly? 
A. Technically yes. But practically no. They can do duct-taping (directly modifying the model) to fix the individual case, but it can potentially break the entire model. Even worse, duct-taping may not create consistent results if the model is trained again with additional data.

I guess most ML researchers would agree with the above limitations - since they are trying to solve them. Even though ML models are largely unpredictable, unexplainable, and hard to fix, they have amazing power to automate a wide range of tedious tasks. What really bugs people (UX designers, and end-users) is the fuzzy boundary of tasks that people can truly delegate to ML models. Here I would say "ML is hard to understand" in comparison with traditional imperative logic where couple if-else conditionals are sufficient. But, in what context, should we compare ML models with human brains? Without question, ML is much simpler than tiny insect's brain. So what? We have no plan to put our lives on any kind of brain.

People use Photoshops without complaining thanks to so many design iterations. Most automated outcomes are within the boundary of user’s expectation - or at least users can learn in a few trials and errors. How many features in Photoshop do return unexpected outcomes? Only a few artistic filters that no artists use for serious work. At Adobe, there are many projects going on, trying to add ML techniques on Photoshop, but so far very few have passed the high standard that target users have. We are still trying to find the right place where human and AI can work together.

In the end, I completely agree with you that “AI is not transparent” is very misleading. If they got transparent AI, they would complain "This is too complicated! If we make ML systems working, they would simply use it without even wondering how it works. " As Jobs said, people don’t know what they need.

Besides, I guess we can learn a lot from the phenomenon - people crying for transparent AI as if they know how their cars running. Why are they so curious of how AI works? Will they stop complaining if it just works? I am not sure. I cannot help wondering why my Passat makes high-frequency noise using my fairly-limited mechanical knowledge. Uber drivers socially discuss how the job-assigning AI works, not because the AI does not work but to manipulate the AI in their favor (https://www.cs.cmu.edu/…/2015-CHI_algorithmic_management.pdf)

재반론. 

1) BTW, I'm not offended at all! 

2) "transparent is enough.” + “ML is fine because human brains are even more unpredictable.” = You misinterpreted me for these two claims. That's not what I meant. Transparency and interpretability are two different things, and transparency and usability are also two different things. If people claimed "ML applications should be made more interpretable and usable" instead of "AI will kill us all", I would actually be very happy and wouldn't have started all this. 
What I'm really trying to say is that It's not okay to bash ML for these reasons, because then we become hypocrites for not bashing humans/users. As a matter of fact, the ML community is doing a much better job on the important issues like interpretability, explainability, usability and ethics than we are, while HCI is just whining "don't forget us!", "you guys kill people with AI". 
Higher level questions have always been difficult for humans to answer regardless of the field or subject. For example, how quickly does a person learn to bias decisions? What's my potential for becoming a top engineer? What prejudice do I have against a certain population? How much do I know about calculus? 
It's not the fault of the subject (ie. ML), but it's just our incapability to understand high dimensional data, process large scale information at the same time, etc. So this really IS an HCI problem. 
So the narrative should be "why we have different expectations for machine-assisted systems, what are these expectations, and how do we design and implement so they're usable and useful to our users?". That makes me much more comfortable. 
In HCI, we have this enormous opportunity to learn from rich and vast user interaction data and they're all 100% computationally represented, and we have a good track record of being able to understand and look beyond 0's and 1's. 
So, I believe ML scientists CAN predict if an image labeling model would correctly tag an image, CAN explain why a certain image is tagged incorrectly, CAN fix a model to perform correctly IF they are given the right tools. I'm beginning to see this happen elsewhere, just not in HCI. 

---

On that last remark on algorithmic decision making, and why people are so keen to know how ML works and not their cars (lol!!), I think what we're really panicking about is ourselves being pathetic biased and racist decision makers hahaha. We know where we want to be, but ML as of now doesn't allow us to become more capable of getting there (or at least such developments don't get covered much in media), but rather is showing just mirror images of ourselves (maybe...) I just feel like we're exposed and naked as a society, and we're being defensive hahaha I hope someone figures it out one day.