Veri Ön İşleme (Data PreProcessing)

Transkript

1 Veri Ö İşleme (Data PreProcessig) Şadi Evre ŞEKER Youtube : Bilgisayar Kavramlar 02/11/16 Data Miig: Cocepts ad Techiques 1

2 Veri Ö İşleme (Data Preprocessig) Data Preprocessig: Giriş Veri Kalitesi Veri Ö işlemedeki aa işlemler Veri Temizleme (Data Cleaig) Veri Uyumu (Data Itegratio) Veri Küçültme (Data Reductio) Veri Döüştürme ve Verii Ayrıklaştırılması (Data Trasformatio ad Data Discretizatio) 2

3 Data Warehouse: A Multi-Tiered Architecture Other sources Operatioal DBs Metadata Extract Trasform Load Refresh Moitor & Itegrator Data Warehouse (Veri Ambarı) OLAP Server Serve Aalysis Query Reports Data miig Data Marts Veri Kayakları Data Storage OLAP Egie Frot-Ed Tools

4 Veri Kalitesi ( Data Quality) Çok boyutlu olarak veri kalitesi kriterleri : Nede Ö işlem yapılır? Kesilik (Accuracy) doğru ve yalış veriler Tamamlık (Completeess) : kaydedilmemiş veya ulaşılamaya veriler Tutarlılık (Cosistecy) verileri bir kısmıı gücel olmaması, sallatıda veriler (daglig) Gücellik (Timeliess) İadırıcılık (Believability) Yorumlaabilirlik (Iterpretability): Verii e kadar kolay alaşılacağı 4

5 Veri Ö İşleme İşlemleri Veri Temizleme (Data cleaig) Eksik verileri doldurulması, gürültülü verileri düzeltilmesi, aykırı verileri (outlier) temizlemesi, uyuşmazlıkları (icosistecies) çözümlemesi Veri Etegrasyou (Data itegratio) Farklı veri kayaklarıı, Veri Küplerii veya Dosyaları etegre olması Verii Küçültülmesi (Data reductio) Boyut Küçültme (Dimesioality reductio) Sayısal Küçültme (Numerosity reductio) Verii Sıkıştırılması (Data compressio) Verii Döüştürülmesi ve Ayrıklaştırılması (Data trasformatio ad data discretizatio) Normalleştirme (Normalizatio ) Kavram Hiyerarşisi (Cocept hierarchy geeratio) 5

6 Veri Ö İşleme (Data Preprocessig) Data Preprocessig: Giriş Veri Kalitesi Veri Ö işlemedeki aa işlemler Veri Temizleme (Data Cleaig) Veri Uyumu (Data Itegratio) Veri Küçültme (Data Reductio) Veri Döüştürme ve Verii Ayrıklaştırılması (Data Trasformatio ad Data Discretizatio) 6

7 Veri Temizleme (Data Cleaig) Gerçek hayattaki veriler kirlidir: Çok sayıda makie, isa veya bilgisayar hataları, iletim bozulmaları yaşaabilir. Eksik Veri (icomplete) bazı özellikleri eksik olması (missig data), sadece birleşik verii (aggregate) buluması ör., Meslek= (girilmemiş) Gülrültülü Veri (oisy): Gürültü, hata veya aykırı veriler buluması ör., Maaş= 10 (hata) Tutarsız Veri (icosistet): farklı kayaklarda farklı veriler gelmesi Yaş= 42, Doğum Tarihi= 03/07/2010 Eski otlama 1, 2, 3, yei otlama A, B, C Tekrarlı kayıtlarda uyuşmazlık Kasıtlı Problemler (Itetioal) Doğum tarihi bilimeye herkese 1 Ocak yazılması 7

8 Eksik Veriler (Icomplete (Missig) Data) Veriye her zama erişilmesi mümkü değildir Ör., bazı kayıtları alı(a)mamış olması. Satış sırasıda müşterileri gelir düzeyii yazılmamış olması. Eksik veriler geelde aşağıdaki durumlarda olur: Doaımsal bozukluklarda Uyuşmazlık yüzüde silie veriler Alaşılamaya verileri girilmemiş olması Veri girişi sırasıda veriye öem verilmemiş olması Verideki değişiklikleri kaydedilmemiş olması Eksik verileri çözülmesi gerekir 8

9 Eksik veriler asıl çözülür? İhmal etme: Eksik veriler işleme alımaz, yokmuş gibi davraılır. Kullaıla VM yötemie göre souca etkileri bilimelidir. Eksik verileri elle doldurulması: her zama mümkü değildir ve baza çok uzu ve maliyetli olabilir Otomatik olarak doldurulması Bütü eksik veriler içi yei bir sııf oluşturulması ( bilimiyor gibi) Ortalamaı yazılması Sııf bazıda ortalamaları yazılması Bayesia formül ve karar ağacı uygulaması 9

10 Gürültülü Veri (Noisy Data) Gürültü (Noise): ölçümdeki rasgele oluşa değerler Yalış özellik değerleri aşağıdaki durumlarda oluşabilir: Veri toplama araçlarıdaki hatalar Veri giriş problemleri Veri iletim problemleri Tekoloji sıırları İsimledirmedeki tutarsızlıklar Veri temizlemesii gerektire diğer durumlar Tekrarlı kayıtlar Eksik veriler Tutarsız veriler 10

11 Gürültülü Veri Nasıl Çözülür? Paketleme (Biig) Veri sıralaır ve eşit frekaslarda paketlere bölüür. Eksik veriler farklı yötemlerle doldurulur: Mea Media Boudary Regrezisyo (Regressio) Regrezisyo foksiyolarıa tabi tutularak eksik verileri girilmesi Bölütleme (Kümeleme, Clusterig) Aykırı verileri buluması ve temizlemesi Bilgisayar ve isa bilgisii ortaklaşa kullaılması detect suspicious values ad check by huma (e.g., deal with possible outliers) 11

12 Veri Temizleme Süreci Verideki farklılıkları yakalaması Üst verii (metadata) kullaılması (ör., veri alaı (domai, rage), bağlılık (depedecy), dağılım (distributio) Aşırı yüklü alalar (Field Overloadig) Veri üzeride kural kotrolleri (uique, cosecutive, ull) Ticari yazılımları kullaılması Bilgi Ovalaması (Data scrubbig): Basit ala bilgileri kurallarla kotrol etmek (e.g., postal code, spell-check) Veri Deetimi (Data auditig): veriler üzeride kural çıkarımı ve kurallara uymayaları buluması (ör., correlatio veya clusterig ile aykırıları (outliers) buluması) Veri Göçü ve Etegrasyou (Data migratio ad itegratio) Data migratio Araçları: Verii döüştürülmesie izi verir ETL (Extractio/Trasformatio/Loadig) Araçları: Geelde grafik arayüzü ile döüşümü yöetme imkaı verir İki farklı işi etegre yürütülmesi Iterative / iteractive (Ör.., Potter s Wheels) 12

13 Aşırı Yüklü Alalar (Overloaded Fields) Aşırı Yüklü Alaları Temizlemesi Zicirleme (Chaiig) Birleştirme (Couplig) Çok Amaçlılık (Multipurpose) 13

14 Örekler Vijayshakar Rama ad Joseph M. Hellerstei, Potter s Wheel: A Iteractive Data Cleaig System berkeley 14

15 Chapter 3: Data Preprocessig Data Preprocessig: A Overview Data Quality Major Tasks i Data Preprocessig Data Cleaig Data Itegratio Data Reductio Data Trasformatio ad Data Discretizatio Summary 15

16 Data Itegratio Data itegratio: Combies data from multiple sources ito a coheret store Schema itegratio: e.g., A.cust-id B.cust-# Itegrate metadata from differet sources Etity idetificatio problem: Idetify real world etities from multiple data sources, e.g., Bill Clito = William Clito Detectig ad resolvig data value coflicts For the same real world etity, attribute values from differet sources are differet Possible reasos: differet represetatios, differet scales, e.g., metric vs. British uits 16

17 Hadlig Redudacy i Data Itegratio Redudat data occur ofte whe itegratio of multiple databases Object idetificatio: The same attribute or object may have differet ames i differet databases Derivable data: Oe attribute may be a derived attribute i aother table, e.g., aual reveue Redudat attributes may be able to be detected by correlatio aalysis ad covariace aalysis Careful itegratio of the data from multiple sources may help reduce/avoid redudacies ad icosistecies ad improve miig speed ad quality 17

18 Correlatio Aalysis (Nomial Data) Χ 2 (chi-square) test The larger the Χ 2 value, the more likely the variables are related The cells that cotribute the most to the Χ 2 value are those whose actual cout is very differet from the expected cout Correlatio does ot imply causality χ = ( Observed Expected Expected 2 2 ) # of hospitals ad # of car-theft i a city are correlated Both are causally liked to the third variable: populatio 18

19 Chi-Square Calculatio: A Example Play chess Not play chess Sum (row) Like sciece fictio 250(90) 200(360) 450 Not like sciece fictio 50(210) 1000(840) 1050 Sum(col.) Χ 2 (chi-square) calculatio (umbers i parethesis are expected couts calculated based o the data distributio i the two categories) 2 χ (250 90) = 90 2 (50 210) ( ) It shows that like_sciece_fictio ad play_chess are correlated i the group 2 ( ) =

20 Correlatio Aalysis (Numeric Data) Correlatio coefficiet (also called Pearso s product momet coefficiet) r A, B ( a i i A)( bi B) ( a i ibi ) AB = 1 = 1 = = ( 1) σ σ ( 1) σ σ A B A B where is the umber of tuples, A ad are the respective meas of A ad B, σ A ad σ B are the respective stadard deviatio of A ad B, ad Σ(a i b i ) is the sum of the AB cross-product. If r A,B > 0, A ad B are positively correlated (A s values icrease as B s). The higher, the stroger correlatio. r A,B = 0: idepedet; r AB < 0: egatively correlated B 20

21 Visually Evaluatig Correlatio Scatter plots showig the similarity from 1 to 1. 21

22 Correlatio (viewed as liear relatioship) Correlatio measures the liear relatioship betwee objects To compute correlatio, we stadardize data objects, A ad B, ad the take their dot product a' k = ( a k mea( A))/ std ( A) b' k = ( b k mea( B))/ std ( B) correlatio ( A, B) = A' B' 22

23 Covariace (Numeric Data) Covariace is similar to correlatio Correlatio coefficiet: where is the umber of tuples, A ad B are the respective mea or expected values of A ad B, σ A ad σ B are the respective stadard deviatio of A ad B. Positive covariace: If Cov A,B > 0, the A ad B both ted to be larger tha their expected values. Negative covariace: If Cov A,B < 0 the if A is larger tha its expected value, B is likely to be smaller tha its expected value. Idepedece: Cov A,B = 0 but the coverse is ot true: Some pairs of radom variables may have a covariace of 0 but are ot idepedet. Oly uder some additioal assumptios (e.g., the data follow multivariate ormal distributios) does a covariace of 0 imply idepedece 23

24 Co-Variace: A Example It ca be simplified i computatio as Suppose two stocks A ad B have the followig values i oe week: (2, 5), (3, 8), (5, 10), (4, 11), (6, 14). Questio: If the stocks are affected by the same idustry treds, will their prices rise or fall together? E(A) = ( )/ 5 = 20/5 = 4 E(B) = ( ) /5 = 48/5 = 9.6 Cov(A,B) = ( )/ = 4 Thus, A ad B rise together sice Cov(A, B) > 0.

25 Chapter 3: Data Preprocessig Data Preprocessig: A Overview Data Quality Major Tasks i Data Preprocessig Data Cleaig Data Itegratio Data Reductio Data Trasformatio ad Data Discretizatio Summary 25

26 Data Reductio Strategies Data reductio: Obtai a reduced represetatio of the data set that is much smaller i volume but yet produces the same (or almost the same) aalytical results Why data reductio? A database/data warehouse may store terabytes of data. Complex data aalysis may take a very log time to ru o the complete data set. Data reductio strategies Dimesioality reductio, e.g., remove uimportat attributes Wavelet trasforms Pricipal Compoets Aalysis (PCA) Feature subset selectio, feature creatio Numerosity reductio (some simply call it: Data Reductio) Regressio ad Log-Liear Models Histograms, clusterig, samplig Data cube aggregatio Data compressio 26

27 Data Reductio 1: Dimesioality Reductio Curse of dimesioality Whe dimesioality icreases, data becomes icreasigly sparse Desity ad distace betwee poits, which is critical to clusterig, outlier aalysis, becomes less meaigful The possible combiatios of subspaces will grow expoetially Dimesioality reductio Avoid the curse of dimesioality Help elimiate irrelevat features ad reduce oise Reduce time ad space required i data miig Allow easier visualizatio Dimesioality reductio techiques Wavelet trasforms Pricipal Compoet Aalysis Supervised ad oliear techiques (e.g., feature selectio) 27

28 Mappig Data to a New Space Fourier trasform Wavelet trasform Two Sie Waves Two Sie Waves + Noise Frequecy 28

29 What Is Wavelet Trasform? Decomposes a sigal ito differet frequecy subbads Applicable to - dimesioal sigals Data are trasformed to preserve relative distace betwee objects at differet levels of resolutio Allow atural clusters to become more distiguishable Used for image compressio 29

30 Wavelet Trasformatio Discrete wavelet trasform (DWT) for liear sigal processig, multi-resolutio aalysis Compressed approximatio: store oly a small fractio of the strogest of the wavelet coefficiets Similar to discrete Fourier trasform (DFT), but better lossy compressio, localized i space Method: Legth, L, must be a iteger power of 2 (paddig with 0 s, whe ecessary) Each trasform has 2 fuctios: smoothig, differece Applies to pairs of data, resultig i two set of data of legth L/2 Haar2 Daubechie4 Applies two fuctios recursively, util reaches the desired legth 30

31 Wavelet Decompositio Wavelets: A math tool for space-efficiet hierarchical decompositio of fuctios S = [2, 2, 0, 2, 3, 5, 4, 4] ca be trasformed to S^ = [2 3 / 4, -1 1 / 4, 1 / 2, 0, 0, -1, -1, 0] Compressio: may small detail coefficiets ca be replaced by 0 s, ad oly the sigificat coefficiets are retaied 31

32 Haar Wavelet Coefficiets Hierarchical decompositio structure (a.k.a. error tree ) Coefficiet Supports Origial frequecy distributio

33 Why Wavelet Trasform? Use hat-shape filters Emphasize regio where poits cluster Suppress weaker iformatio i their boudaries Effective removal of outliers Isesitive to oise, isesitive to iput order Multi-resolutio Detect arbitrary shaped clusters at differet scales Efficiet Complexity O(N) Oly applicable to low dimesioal data 33

34 Pricipal Compoet Aalysis (PCA) Fid a projectio that captures the largest amout of variatio i data The origial data are projected oto a much smaller space, resultig i dimesioality reductio. We fid the eigevectors of the covariace matrix, ad these eigevectors defie the ew space x 2 e x 1 34

35 Pricipal Compoet Aalysis (Steps) Give N data vectors from -dimesios, fid k orthogoal vectors (pricipal compoets) that ca be best used to represet data Normalize iput data: Each attribute falls withi the same rage Compute k orthoormal (uit) vectors, i.e., pricipal compoets Each iput data (vector) is a liear combiatio of the k pricipal compoet vectors The pricipal compoets are sorted i order of decreasig sigificace or stregth Sice the compoets are sorted, the size of the data ca be reduced by elimiatig the weak compoets, i.e., those with low variace (i.e., usig the strogest pricipal compoets, it is possible to recostruct a good approximatio of the origial data) Works for umeric data oly 35

36 Attribute Subset Selectio Aother way to reduce dimesioality of data Redudat attributes Duplicate much or all of the iformatio cotaied i oe or more other attributes E.g., purchase price of a product ad the amout of sales tax paid Irrelevat attributes Cotai o iformatio that is useful for the data miig task at had E.g., studets' ID is ofte irrelevat to the task of predictig studets' GPA 36

37 Heuristic Search i Attribute Selectio There are 2 d possible attribute combiatios of d attributes Typical heuristic attribute selectio methods: Best sigle attribute uder the attribute idepedece assumptio: choose by sigificace tests Best step-wise feature selectio: The best sigle-attribute is picked first The ext best attribute coditio to the first,... Step-wise attribute elimiatio: Repeatedly elimiate the worst attribute Best combied attribute selectio ad elimiatio Optimal brach ad boud: Use attribute elimiatio ad backtrackig 37

38 Attribute Creatio (Feature Geeratio) Create ew attributes (features) that ca capture the importat iformatio i a data set more effectively tha the origial oes Three geeral methodologies Attribute extractio Domai-specific Mappig data to ew space (see: data reductio) E.g., Fourier trasformatio, wavelet trasformatio, maifold approaches (ot covered) Attribute costructio Combiig features (see: discrimiative frequet patters i Chapter 7) Data discretizatio 38

39 Data Reductio 2: Numerosity Reductio Reduce data volume by choosig alterative, smaller forms of data represetatio Parametric methods (e.g., regressio) Assume the data fits some model, estimate model parameters, store oly the parameters, ad discard the data (except possible outliers) Ex.: Log-liear models obtai value at a poit i m- D space as the product o appropriate margial subspaces No-parametric methods Do ot assume models Major families: histograms, clusterig, samplig, 39

40 Parametric Data Reductio: Regressio ad Log-Liear Models Liear regressio Data modeled to fit a straight lie Ofte uses the least-square method to fit the lie Multiple regressio Allows a respose variable Y to be modeled as a liear fuctio of multidimesioal feature vector Log-liear model Approximates discrete multidimesioal probability distributios 40

41 Regressio Aalysis Regressio aalysis: A collective ame for techiques for the modelig ad aalysis of umerical data cosistig of values of a depedet variable (also called respose variable or measuremet) ad of oe or more idepedet variables (aka. explaatory variables or predictors) The parameters are estimated so as to give a "best fit" of the data Most commoly the best fit is evaluated by usig the least squares method, but other criteria have also bee used y Y1 Y1 X1 y = x + 1 Used for predictio (icludig forecastig of time-series data), iferece, hypothesis testig, ad modelig of causal relatioships x 41

42 Regress Aalysis ad Log-Liear Models Liear regressio: Y = w X + b Two regressio coefficiets, w ad b, specify the lie ad are to be estimated by usig the data at had Usig the least squares criterio to the kow values of Y 1, Y 2,, X 1, X 2,. Multiple regressio: Y = b 0 + b 1 X 1 + b 2 X 2 May oliear fuctios ca be trasformed ito the above Log-liear models: Approximate discrete multidimesioal probability distributios Estimate the probability of each poit (tuple) i a multi-dimesioal space for a set of discretized attributes, based o a smaller subset of dimesioal combiatios Useful for dimesioality reductio ad data smoothig 42

43 Histogram Aalysis Divide data ito buckets ad store average (sum) for each bucket Partitioig rules: Equal-width: equal bucket rage Equal-frequecy (or equaldepth)

44 Clusterig Partitio data set ito clusters based o similarity, ad store cluster represetatio (e.g., cetroid ad diameter) oly Ca be very effective if data is clustered but ot if data is smeared Ca have hierarchical clusterig ad be stored i multidimesioal idex tree structures There are may choices of clusterig defiitios ad clusterig algorithms Cluster aalysis will be studied i depth i Chapter 10 44

45 Samplig Samplig: obtaiig a small sample s to represet the whole data set N Allow a miig algorithm to ru i complexity that is potetially sub-liear to the size of the data Key priciple: Choose a represetative subset of the data Simple radom samplig may have very poor performace i the presece of skew Develop adaptive samplig methods, e.g., stratified samplig: Note: Samplig may ot reduce database I/Os (page at a time) 45

46 Types of Samplig Simple radom samplig There is a equal probability of selectig ay particular item Samplig without replacemet Oce a object is selected, it is removed from the populatio Samplig with replacemet A selected object is ot removed from the populatio Stratified samplig: Partitio the data set, ad draw samples from each partitio (proportioally, i.e., approximately the same percetage of the data) Used i cojuctio with skewed data 46

47 Samplig: With or without Replacemet SRSWOR (simple radom sample without replacemet) SRSWR Raw Data 47

48 Samplig: Cluster or Stratified Samplig Raw Data Cluster/Stratified Sample 48

49 Data Cube Aggregatio The lowest level of a data cube (base cuboid) The aggregated data for a idividual etity of iterest E.g., a customer i a phoe callig data warehouse Multiple levels of aggregatio i data cubes Further reduce the size of data to deal with Referece appropriate levels Use the smallest represetatio which is eough to solve the task Queries regardig aggregated iformatio should be aswered usig data cube, whe possible 49

50 Data Reductio 3: Data Compressio Strig compressio There are extesive theories ad well-tued algorithms Typically lossless, but oly limited maipulatio is possible without expasio Audio/video compressio Typically lossy compressio, with progressive refiemet Sometimes small fragmets of sigal ca be recostructed without recostructig the whole Time sequece is ot audio Typically short ad vary slowly with time Dimesioality ad umerosity reductio may also be cosidered as forms of data compressio 50

51 Data Compressio Origial Data lossless Compressed Data Origial Data Approximated 51

52 Chapter 3: Data Preprocessig Data Preprocessig: A Overview Data Quality Major Tasks i Data Preprocessig Data Cleaig Data Itegratio Data Reductio Data Trasformatio ad Data Discretizatio Summary 52

53 Data Trasformatio A fuctio that maps the etire set of values of a give attribute to a ew set of replacemet values s.t. each old value ca be idetified with oe of the ew values Methods Smoothig: Remove oise from data Attribute/feature costructio New attributes costructed from the give oes Aggregatio: Summarizatio, data cube costructio Normalizatio: Scaled to fall withi a smaller, specified rage mi-max ormalizatio z-score ormalizatio ormalizatio by decimal scalig Discretizatio: Cocept hierarchy climbig 53

54 Normalizatio Mi-max ormalizatio: to [ew_mi A, ew_max A ] Ex. Let icome rage $12,000 to $98,000 ormalized to [0.0, 1.0]. The $73,000 is mapped to Z-score ormalizatio (µ: mea, σ: stadard deviatio): v v mi maxa mi A ' = ( ew_ maxa ew_ mia) + v' = v µ A σa Ex. Let µ = 54,000, σ = 16,000. The Normalizatio by decimal scalig A ew_ mi 73,600 12,000 (1.0 0) + 0 = ,000 12,000 73,600 54,000 = ,000 v v'= Where j is the smallest iteger such that Max( ν ) < 1 j 10 A 54

55 Discretizatio Three types of attributes Nomial values from a uordered set, e.g., color, professio Ordial values from a ordered set, e.g., military or academic rak Numeric real umbers, e.g., iteger or real umbers Discretizatio: Divide the rage of a cotiuous attribute ito itervals Iterval labels ca the be used to replace actual data values Reduce data size by discretizatio Supervised vs. usupervised Split (top-dow) vs. merge (bottom-up) Discretizatio ca be performed recursively o a attribute Prepare for further aalysis, e.g., classificatio 55

56 Data Discretizatio Methods Typical methods: All the methods ca be applied recursively Biig Top-dow split, usupervised Histogram aalysis Top-dow split, usupervised Clusterig aalysis (usupervised, top-dow split or bottom-up merge) Decisio-tree aalysis (supervised, top-dow split) Correlatio (e.g., χ 2 ) aalysis (usupervised, bottom-up merge) 56

57 Simple Discretizatio: Biig Equal-width (distace) partitioig Divides the rage ito N itervals of equal size: uiform grid if A ad B are the lowest ad highest values of the attribute, the width of itervals will be: W = (B A)/N. The most straightforward, but outliers may domiate presetatio Skewed data is ot hadled well Equal-depth (frequecy) partitioig Divides the rage ito N itervals, each cotaiig approximately same umber of samples Good data scalig Maagig categorical attributes ca be tricky 57

58 Biig Methods for Data Smoothig q Sorted data for price (i dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34 * Partitio ito equal-frequecy (equi-depth) bis: - Bi 1: 4, 8, 9, 15 - Bi 2: 21, 21, 24, 25 - Bi 3: 26, 28, 29, 34 * Smoothig by bi meas: - Bi 1: 9, 9, 9, 9 - Bi 2: 23, 23, 23, 23 - Bi 3: 29, 29, 29, 29 * Smoothig by bi boudaries: - Bi 1: 4, 4, 4, 15 - Bi 2: 21, 21, 25, 25 - Bi 3: 26, 26, 26, 34 58

59 Discretizatio Without Usig Class Labels (Biig vs. Clusterig) Data Equal iterval width (biig) Equal frequecy (biig) K-meas clusterig leads to better results 59

60 Discretizatio by Classificatio & Correlatio Aalysis Classificatio (e.g., decisio tree aalysis) Supervised: Give class labels, e.g., cacerous vs. beig Usig etropy to determie split poit (discretizatio poit) Top-dow, recursive split Details to be covered i Chapter 7 Correlatio aalysis (e.g., Chi-merge: χ 2 -based discretizatio) Supervised: use class iformatio Bottom-up merge: fid the best eighborig itervals (those havig similar distributios of classes, i.e., low χ 2 values) to merge Merge performed recursively, util a predefied stoppig coditio 60

61 Cocept Hierarchy Geeratio Cocept hierarchy orgaizes cocepts (i.e., attribute values) hierarchically ad is usually associated with each dimesio i a data warehouse Cocept hierarchies facilitate drillig ad rollig i data warehouses to view data i multiple graularity Cocept hierarchy formatio: Recursively reduce the data by collectig ad replacig low level cocepts (such as umeric values for age) by higher level cocepts (such as youth, adult, or seior) Cocept hierarchies ca be explicitly specified by domai experts ad/ or data warehouse desigers Cocept hierarchy ca be automatically formed for both umeric ad omial data. For umeric data, use discretizatio methods show. 61

62 Cocept Hierarchy Geeratio for Nomial Data Specificatio of a partial/total orderig of attributes explicitly at the schema level by users or experts street < city < state < coutry Specificatio of a hierarchy for a set of values by explicit data groupig {Urbaa, Champaig, Chicago} < Illiois Specificatio of oly a partial set of attributes E.g., oly street < city, ot others Automatic geeratio of hierarchies (or attribute levels) by the aalysis of the umber of distict values E.g., for a set of attributes: {street, city, state, coutry} 62

63 Automatic Cocept Hierarchy Geeratio Some hierarchies ca be automa0cally geerated based o the aalysis of the umber of dis0ct values per a8ribute i the data set The a8ribute with the most dis0ct values is placed at the lowest level of the hierarchy Excep0os, e.g., weekday, moth, quarter, year coutry provice_or_ state city street 15 distict values 365 distict values 3567 distict values 674,339 distict values 63

64 Chapter 3: Data Preprocessig Data Preprocessig: A Overview Data Quality Major Tasks i Data Preprocessig Data Cleaig Data Itegratio Data Reductio Data Trasformatio ad Data Discretizatio Summary 64

65 Summary Data quality: accuracy, completeess, cosistecy, timeliess, believability, iterpretability Data cleaig: e.g. missig/oisy values, outliers Data itegratio from multiple sources: Etity idetificatio problem Remove redudacies Detect icosistecies Data reductio Dimesioality reductio Numerosity reductio Data compressio Data trasformatio ad data discretizatio Normalizatio Cocept hierarchy geeratio 65

66 Kayaklar Data Miig: Cocepts ad Techiques, Third Editio (The Morga Kaufma Series i Data Maagemet Systems) 3rd Editio by Jiawei Ha (Author), Michelie Kamber (Author), Jia Pei (Author)

67 Refereces D. P. Ballou ad G. K. Tayi. Ehacig data quality i data warehouse eviromets. Comm. of ACM, 42:73-78, 1999 A. Bruce, D. Dooho, ad H.-Y. Gao. Wavelet aalysis. IEEE Spectrum, Oct 1996 T. Dasu ad T. Johso. Exploratory Data Miig ad Data Cleaig. Joh Wiley, 2003 J. Devore ad R. Peck. Sta-s-cs: The Explora-o ad Aalysis of Data. Duxbury Press, H. Galhardas, D. Florescu, D. Shasha, E. Simo, ad C.-A. Saita. Declara0ve data cleaig: Laguage, model, ad algorithms. VLDB'01 M. Hua ad J. Pei. Cleaig disguised missig data: A heuris0c approach. KDD'07 H. V. Jagadish, et al., Special Issue o Data Reduc0o Techiques. Bulle0 of the Techical Commi8ee o Data Egieerig, 20(4), Dec H. Liu ad H. Motoda (eds.). Feature Extrac-o, Costruc-o, ad Selec-o: A Data Miig Perspec-ve. Kluwer Academic, 1998 J. E. Olso. Data Quality: The Accuracy Dimesio. Morga Kaufma, 2003 D. Pyle. Data Prepara0o for Data Miig. Morga Kaufma, 1999 V. Rama ad J. Hellerstei. Po8ers Wheel: A Iterac0ve Framework for Data Cleaig ad Trasforma0o, VLDB 2001 T. Redma. Data Quality: The Field Guide. Digital Press (Elsevier), 2001 R. Wag, V. Storey, ad C. Firth. A framework for aalysis of data quality research. IEEE Tras. Kowledge ad Data Egieerig, 7: ,