Data Analytic Thinking (2)
데이터사이언스개론
| 목차
I. 데이터 사이언스의 의의
1. 데이터 사이언스란 무엇인가?
2. 데이터 사이언티스트란 누구인가?
3. 데이터 사이언스 현황과 예 소개
II. 데이터 사이언스의 성질
1. 데이터 사이언스와 데이터 마이닝의 비교
2. 데이터 사이언스와 데이터 엔지니어링의 비교
III. 데이터 사이언스의 적용
1. 데이터 사이언스와 빅데이터
2. 데이터 기반 의사결정
3. 데이터 분석적 사고와 활용
-
지난 글에 이어 II-3. 데이터 분석적 사고와 활용부터 Data science tasks and process (1) 까지 이론 정리를 시작합니다. 한국어는 강의안 필기이며 번역문이 아닙니다.
[summary]
*Data science
-To extract useful information and knowledge from large volumes of data, in order to support data-driven decision-making.
-Comprised of a set of fundamental concepts and principles.
*Thinking data-analytically
-You should be able to apply the fundamental concepts to real problems.
-Important not only for data scientists, but for anyone working with them.
-Aided by a systemic process or framework with well-defined stages.
*Data science depends on big data technologies and engineering.
-However, its principles are separate from them.
3. 데이터 분석적 사고와 활용
Data-Analytic Thinking
: It's a kind of solution that you should approach "data-analytical" when faced with a real-world problem. In this process, whether and how data improve performance, you should asssess datasets.
Structured by a systematic framework, also Data-analytic thinking is facilitated by a set of fundamental concepts and principles.
and It is taken advantage of to help see data-oriented competitive threats when you face the situation to interact competitively with others and lead to improving data-driven decision-making. So, If you do, you will have a better relationship with that.
And Most of current industries and companies employ Data scientist to increase revenue and decrease costs, using data mining as a key strategic component.
Even if we are not a Data scientist, much of business industry need to employ managers to be able to oversee data science teams and projects, marketers to understand data-driven campaigns, and business strategists to devise plans that exploit data.
Therefore, we must know it to understand fundamental concepts of data science in our business.
and Let's start to know the examples of fundamental concepts.
First, Follow 1) a systematic process that provides a framework to structure your thinking, to extract useful knowledge from data. For instance, most of the process can perform a careful analysis of the problem first, and then, modeling, and careful evaluation.
Systematic process 는 문제를 자세하게 분석해보고, 모델링, 평가 단계를 거치는 것이다.
And 2) Find informative attributes that give us the needed information on entities of interest from a large mass of data. For example, you can ask "which of the following attributes are correlated with churn?"
Attributes of the database can give an account of a particular person. From the database's perspective, the term "attribute" is used, but Machine learning called it to feature. In the view of the program, each one is "변수"
속성은 a라는 사람에 대해 설명해주고 있는 것이다. 데이터 베이스 관점에서 속성이란 표현을 쓰고, 머신러닝에서는 feature을 쓴다. 프로그램 관점에서 보면 이 하나하나가 변수라고 생각할 수 있다.
You can think of which of these many attributes is more important.
이 많은 속성들 중 어느 속성이 더 중요할까, 를 생각할 수 있다.
Which of the following attributes are correlated with churn?
And 3) Avoid overfitting that means a model is too closely fit to a particular set of data so it cannot generalize beyond this data. In other words, It results that overfitting state cannot apply new data to your program.
데이터 관점에서 학습을 시키는 것이 fitting이다. 과적합, over fitting은 어떤 모델이 너무 자세히 fitting을 해서 새로운 것에 대한 일반화가 안 된다.
Before fitting
(1) Good fitting (2) Over fitting x1 is age, and 2 is salary. The simplest way of finding the model is "직선의 방정식" like 'y=-x'. but if you do this, it's the wrong answer because it's less fitting(Under fitting).
가장 간단한 방식은 y=-x라는 직선의 방정식을 하나 찾아서 기울기와 절편을 구하면 직선의 방정식이 나온다. 만약에 직선의 방정식을 model이라고 하면 이것은 틀린 것이 꽤 된다. 직선의 방정식 그냥 있는 것은 under fitting 과소적합, 직선의 방정식 하나로 충분하지 않다.
You can assume this is "곡선의 방정식" in the direction of reducing errors. This kind of process is called fitting. however, if you do this so perfectly(100%) through using our traning(given) data, it leads to overfitting.
직선이 아니라 곡선으로 해 본다면 오류를 좀 더 줄일 수 있다. 100% 되는 f(x)를 찾아서, particular set에 대해 필요 이상으로 fitting하는 것을 over fitting한다고 말한다.
With the overfitting model to new data(that's originally our first purpose.), using it does not fit our purpose because it fails to utilize.
우리가 과적합된 모델을 가지고 새로운 데이터를 적용하면, traning data를 완전히 맞추더라도 새로운 데이터를 활용해야 하므로 일반화에 실패하게 된다. +딥러닝의 큰 챕터가 regulation인데 이는 over fitting을 피할 수 있는 규제기도 함. 그만큼 overfitting을 피하는 것이 중요함.
Last, 4) Evaluate the results of mining carefully and objectively. You should evaluate how much your results are better, for instance, 'How can I show that (1) Good fitting is better than (2) Overfitting?'
평가도 평가 기준이 있으니까 배워서 적용해보는 것이다. 많은 속성이 있지만 정말 그게 다 필요한지 확인해보는 것이 중요하다.
Now, design analytically to fit this concept!
Engineering Side of Data Science
Recently, data scientists should have two types of capabilities being good at both sides of Science, the ability to apply theoretical concepts and principles to practical situations, and Technology, the ability to use popular programming languages and tools.
Science is a kind of logistic regression, support vector, machines, and matrix factorization. and Techonlogy is a kind of scikitlearn, tensorflow, hadoop, mongoDB, and spark etc.
A data scientist requires proficiency with certain sorts of software tools. In the view of engineering, you show the result with programing to use popular tools.
