Data Preprocessing (preprocess
)¶
Preprocessing module contains data processing utilities like data discretization, continuization, imputation and transformation.
Impute¶
Imputation replaces missing values with new values (or omits such features).
from Orange.data import Table
from Orange.preprocess import Impute
data = Table("heart-disease.tab")
imputer = Impute()
impute_heart = imputer(data)
There are several imputation methods one can use.
from Orange.data import Table
from Orange.preprocess import Impute, Average
data = Table("heart_disease.tab")
imputer = Impute(method=Average())
impute_heart = imputer(data)
Discretization¶
Discretization replaces continuous features with the corresponding categorical features:
import Orange
iris = Orange.data.Table("iris.tab")
disc = Orange.preprocess.Discretize()
disc.method = Orange.preprocess.discretize.EqualFreq(n=3)
d_iris = disc(iris)
print("Original dataset:")
for e in iris[:3]:
print(e)
print("Discretized dataset:")
for e in d_iris[:3]:
print(e)
The variable in the new data table indicate the bins to which the original values belong.
Original dataset:
[5.1, 3.5, 1.4, 0.2 | Iris-setosa]
[4.9, 3.0, 1.4, 0.2 | Iris-setosa]
[4.7, 3.2, 1.3, 0.2 | Iris-setosa]
Discretized dataset:
[<5.5, >=3.2, <2.5, <0.8 | Iris-setosa]
[<5.5, [2.8, 3.2), <2.5, <0.8 | Iris-setosa]
[<5.5, >=3.2, <2.5, <0.8 | Iris-setosa]
Default discretization method (four bins with approximatelly equal number of data instances) can be replaced with other methods.
iris = Orange.data.Table("iris.tab")
disc = Orange.preprocess.Discretize()
disc.method = Orange.preprocess.discretize.EqualFreq(n=2)
Discretization Algorithms¶
- class Orange.preprocess.discretize.EqualWidth(n=4)[source]¶
Discretization into a fixed number of bins with equal widths.
- n¶
Number of bins (default: 4).
- class Orange.preprocess.discretize.EqualFreq(n=4)[source]¶
Discretization into bins with approximately equal number of data instances.
- n¶
Number of bins (default: 4). The actual number may be lower if the variable has less than n distinct values.
- class Orange.preprocess.discretize.EntropyMDL(force=False)[source]¶
Discretization into bins inferred by recursively splitting the values to minimize the class-entropy. The procedure stops when further splits would decrease the entropy for less than the corresponding increase of minimal description length (MDL). [FayyadIrani93].
If there are no suitable cut-off points, the procedure returns a single bin, which means that the new feature is constant and can be removed.
- force¶
Induce at least one cut-off point, even when its information gain is lower than MDL (default: False).
To add a new discretization, derive it from Discretization
.
Continuization¶
- class Orange.preprocess.Continuize¶
Given a data table, return a new table in which the discretize attributes are replaced with continuous or removed.
binary variables are transformed into 0.0/1.0 or -1.0/1.0 indicator variables, depending upon the argument
zero_based
.multinomial variables are treated according to the argument
multinomial_treatment
.discrete attribute with only one possible value are removed;
import Orange titanic = Orange.data.Table("titanic") continuizer = Orange.preprocess.Continuize() titanic1 = continuizer(titanic)
The class has a number of attributes that can be set either in constructor or, later, as attributes.
- zero_based¶
Determines the value used as the “low” value of the variable. When binary variables are transformed into continuous or when multivalued variable is transformed into multiple variables, the transformed variable can either have values 0.0 and 1.0 (default,
zero_based=True
) or -1.0 and 1.0 (zero_based=False
).
- multinomial_treatment¶
Defines the treatment of multinomial variables.
Continuize.Indicators
The variable is replaced by indicator variables, each corresponding to one value of the original variable. For each value of the original attribute, only the corresponding new attribute will have a value of one and others will be zero. This is the default behaviour.
Note that these variables are not independent, so they cannot be used (directly) in, for instance, linear or logistic regression.
For example, dataset “titanic” has feature “status” with values “crew”, “first”, “second” and “third”, in that order. Its value for the 15th row is “first”. Continuization replaces the variable with variables “status=crew”, “status=first”, “status=second” and “status=third”. After
continuizer = Orange.preprocess.Continuize() titanic1 = continuizer(titanic)
we have
>>> titanic.domain [status, age, sex | survived] >>> titanic1.domain [status=crew, status=first, status=second, status=third, age=adult, age=child, sex=female, sex=male | survived]
For the 15th row, the variable “status=first” has value 1 and the values of the other three variables are 0:
>>> print(titanic[15]) [first, adult, male | yes] >>> print(titanic1[15]) [0.000, 1.000, 0.000, 0.000, 1.000, 0.000, 0.000, 1.000 | yes]
Continuize.FirstAsBase
Similar to the above, except that it creates indicators for all values except the first one, according to the order in the variable’s
values
attribute. If all indicators in the transformed data instance are 0, the original instance had the first value of the corresponding variable.Continuizing the variable “status” with this setting gives variables “status=first”, “status=second” and “status=third”. If all of them were 0, the status of the original data instance was “crew”.
>>> continuizer.multinomial_treatment = continuizer.FirstAsBase >>> continuizer(titanic).domain [status=first, status=second, status=third, age=child, sex=male | survived]
Continuize.FrequentAsBase
Like above, except that the most frequent value is used as the base. If there are multiple most frequent values, the one with the lowest index in
values
is used. The frequency of values is extracted from data, so this option does not work if only the domain is given.Continuizing the Titanic data in this way differs from the above by the attributes sex: instead of “sex=male” it constructs “sex=female” since there were more females than males on Titanic.
>>> continuizer.multinomial_treatment = continuizer.FrequentAsBase >>> continuizer(titanic).domain [status=first, status=second, status=third, age=child, sex=female | survived]
Continuize.Remove
Discrete variables are removed.
>>> continuizer.multinomial_treatment = continuizer.Remove >>> continuizer(titanic).domain [ | survived]
Continuize.RemoveMultinomial
Discrete variables with more than two values are removed. Binary variables are treated the same as in FirstAsBase.
>>> continuizer.multinomial_treatment = continuizer.RemoveMultinomial >>> continuizer(titanic).domain [age=child, sex=male | survived]
Continuize.ReportError
Raise an error if there are any multinomial variables in the data.
Continuize.AsOrdinal
Multinomial variables are treated as ordinal and replaced by continuous variables with indices within
values
, e.g. 0, 1, 2, 3…>>> continuizer.multinomial_treatment = continuizer.AsOrdinal >>> titanic1 = continuizer(titanic) >>> titanic[700] [third, adult, male | no] >>> titanic1[700] [3.000, 0.000, 1.000 | no]
Continuize.AsNormalizedOrdinal
As above, except that the resulting continuous value will be from range 0 to 1, e.g. 0, 0.333, 0.667, 1 for a four-valued variable:
>>> continuizer.multinomial_treatment = continuizer.AsNormalizedOrdinal >>> titanic1 = continuizer(titanic) >>> titanic1[700] [1.000, 0.000, 1.000 | no] >>> titanic1[15] [0.333, 0.000, 1.000 | yes]
- transform_class¶
If
True
the class is replaced by continuous attributes or normalized as well. Multiclass problems are thus transformed to multitarget ones. (Default:False
)
- class Orange.preprocess.DomainContinuizer¶
Construct a domain in which discrete attributes are replaced by continuous.
domain_continuizer = Orange.preprocess.DomainContinuizer() domain1 = domain_continuizer(titanic)
Orange.preprocess.Continuize
calls DomainContinuizer to construct the domain.Domain continuizers can be given either a dataset or a domain, and return a new domain. When given only the domain, use the most frequent value as the base value.
By default, the class does not change continuous and class attributes, discrete attributes are replaced with N attributes (
Indicators
) with values 0 and 1.
Normalization¶
- class Orange.preprocess.Normalize(zero_based=True, norm_type=Normalize.NormalizeBySD, transform_class=False, center=True, normalize_datetime=False)[source]¶
Construct a preprocessor for normalization of features. Given a data table, preprocessor returns a new table in which the continuous attributes are normalized.
- Parameters
zero_based (bool (default=True)) –
Only used when norm_type=NormalizeBySpan.
Determines the value used as the “low” value of the variable. It determines the interval for normalized continuous variables (either [-1, 1] or [0, 1]).
norm_type (NormTypes (default: Normalize.NormalizeBySD)) –
Normalization type. If Normalize.NormalizeBySD, the values are replaced with standardized values by subtracting the average value and dividing by the standard deviation. Attribute zero_based has no effect on this standardization.
If Normalize.NormalizeBySpan, the values are replaced with normalized values by subtracting min value of the data and dividing by span (max - min).
transform_class (bool (default=False)) – If True the class is normalized as well.
center (bool (default=True)) –
Only used when norm_type=NormalizeBySD.
Whether or not to center the data so it has mean zero.
normalize_datetime (bool (default=False)) –
Examples
>>> from Orange.data import Table >>> from Orange.preprocess import Normalize >>> data = Table("iris") >>> normalizer = Normalize(norm_type=Normalize.NormalizeBySpan) >>> normalized_data = normalizer(data)
Randomization¶
- class Orange.preprocess.Randomize(rand_type=Randomize.RandomizeClasses, rand_seed=None)[source]¶
Construct a preprocessor for randomization of classes, attributes and/or metas. Given a data table, preprocessor returns a new table in which the data is shuffled.
- Parameters
rand_type (RandTypes (default: Randomize.RandomizeClasses)) – Randomization type. If Randomize.RandomizeClasses, classes are shuffled. If Randomize.RandomizeAttributes, attributes are shuffled. If Randomize.RandomizeMetas, metas are shuffled.
rand_seed (int (optional)) – Random seed
Examples
>>> from Orange.data import Table >>> from Orange.preprocess import Randomize >>> data = Table("iris") >>> randomizer = Randomize(Randomize.RandomizeClasses) >>> randomized_data = randomizer(data)
Remove¶
- class Orange.preprocess.Remove(attr_flags=0, class_flags=0, meta_flags=0)[source]¶
Construct a preprocessor for removing constant features/classes and unused values. Given a data table, preprocessor returns a new table and a list of results. In the new table, the constant features/classes and unused values are removed. The list of results consists of two dictionaries. The first one contains numbers of ‘removed’, ‘reduced’ and ‘sorted’ features. The second one contains numbers of ‘removed’, ‘reduced’ and ‘sorted’ features.
- Parameters
attr_flags (int (default: 0)) – If SortValues, values of discrete attributes are sorted. If RemoveConstant, unused attributes are removed. If RemoveUnusedValues, unused values are removed from discrete attributes. It is possible to merge operations in one by summing several types.
class_flags (int (default: 0)) – If SortValues, values of discrete class attributes are sorted. If RemoveConstant, unused class attributes are removed. If RemoveUnusedValues, unused values are removed from discrete class attributes. It is possible to merge operations in one by summing several types.
Examples
>>> from Orange.data import Table >>> from Orange.preprocess import Remove >>> data = Table("zoo")[:10] >>> flags = sum([Remove.SortValues, Remove.RemoveConstant, Remove.RemoveUnusedValues]) >>> remover = Remove(attr_flags=flags, class_flags=flags) >>> new_data = remover(data) >>> attr_results, class_results = remover.attr_results, remover.class_results
Feature selection¶
Feature scoring¶
Feature scoring is an assessment of the usefulness of features for prediction of the dependant (class) variable. Orange provides classes that compute the common feature scores for classification and regression.
The code below computes the information gain of feature “tear_rate” in the Lenses dataset:
>>> data = Orange.data.Table("lenses")
>>> Orange.preprocess.score.InfoGain(data, "tear_rate")
0.54879494069539858
An alternative way of invoking the scorers is to construct the scoring object and calculate the scores for all the features at once, like in the following example:
>>> gain = Orange.preprocess.score.InfoGain()
>>> scores = gain(data)
>>> for attr, score in zip(data.domain.attributes, scores):
... print('%.3f' % score, attr.name)
0.039 age
0.040 prescription
0.377 astigmatic
0.549 tear_rate
Feature scoring methods work on different feature types (continuous or discrete) and different types of target variables (i.e. in classification or regression problems). Refer to method’s feature_type and class_type attributes for intended type or employ preprocessing methods (e.g. discretization) for conversion between data types.
- class Orange.preprocess.score.ANOVA[source]¶
A wrapper for sklearn.feature_selection._univariate_selection.f_classif. The following is the documentation from scikit-learn.
Compute the ANOVA F-value for the provided sample.
Read more in the User Guide.
- feature_type¶
- class_type¶
- class Orange.preprocess.score.Chi2[source]¶
A wrapper for sklearn.feature_selection._univariate_selection.chi2. The following is the documentation from scikit-learn.
Compute chi-squared stats between each non-negative feature and class.
This score can be used to select the n_features features with the highest values for the test chi-squared statistic from X, which must contain only non-negative features such as booleans or frequencies (e.g., term counts in document classification), relative to the classes.
Recall that the chi-square test measures dependence between stochastic variables, so using this function “weeds out” the features that are the most likely to be independent of class and therefore irrelevant for classification.
Read more in the User Guide.
- feature_type¶
- class_type¶
- class Orange.preprocess.score.GainRatio[source]¶
Information gain ratio is the ratio between information gain and the entropy of the feature’s value distribution. The score was introduced in [Quinlan1986] to alleviate overestimation for multi-valued features. See Wikipedia entry on gain ratio.
- Quinlan1986
J R Quinlan: Induction of Decision Trees, Machine Learning, 1986.
- feature_type¶
- class_type¶
- class Orange.preprocess.score.Gini[source]¶
Gini impurity is the probability that two randomly chosen instances will have different classes. See Wikipedia entry on Gini impurity.
- feature_type¶
- class_type¶
- class Orange.preprocess.score.InfoGain[source]¶
Information gain is the expected decrease of entropy. See Wikipedia entry on information gain.
- feature_type¶
- class_type¶
- class Orange.preprocess.score.UnivariateLinearRegression[source]¶
A wrapper for sklearn.feature_selection._univariate_selection.f_regression. The following is the documentation from scikit-learn.
Univariate linear regression tests returning F-statistic and p-values.
Quick linear model for testing the effect of a single regressor, sequentially for many regressors.
This is done in 2 steps:
The cross correlation between each regressor and the target is computed, that is, ((X[:, i] - mean(X[:, i])) * (y - mean_y)) / (std(X[:, i]) * std(y)) using r_regression function.
It is converted to an F score and then to a p-value.
f_regression()
is derived fromr_regression()
and will rank features in the same order if all the features are positively correlated with the target.Note however that contrary to
f_regression()
,r_regression()
values lie in [-1, 1] and can thus be negative.f_regression()
is therefore recommended as a feature selection criterion to identify potentially predictive feature for a downstream classifier, irrespective of the sign of the association with the target variable.Furthermore
f_regression()
returns p-values whiler_regression()
does not.Read more in the User Guide.
- feature_type¶
- class_type¶
- class Orange.preprocess.score.FCBF[source]¶
Fast Correlation-Based Filter. Described in:
Yu, L., Liu, H., Feature selection for high-dimensional data: A fast correlation-based filter solution. 2003. http://www.aaai.org/Papers/ICML/2003/ICML03-111.pdf
- feature_type¶
- class_type¶
- class Orange.preprocess.score.ReliefF(n_iterations=50, k_nearest=10, random_state=None)[source]¶
ReliefF algorithm. Contrary to most other scorers, Relief family of algorithms is not as myoptic but tends to give unreliable results with datasets with lots (hundreds) of features.
Robnik-Šikonja, M., Kononenko, I. Theoretical and empirical analysis of ReliefF and RReliefF. 2003. http://lkm.fri.uni-lj.si/rmarko/papers/robnik03-mlj.pdf
- feature_type¶
alias of
Orange.data.variable.Variable
- class_type¶
- class Orange.preprocess.score.RReliefF(n_iterations=50, k_nearest=50, random_state=None)[source]¶
- feature_type¶
alias of
Orange.data.variable.Variable
- class_type¶
Additionally, you can use the score_data()
method of some learners (Orange.classification.LinearRegressionLearner
,
Orange.regression.LogisticRegressionLearner
,
Orange.classification.RandomForestLearner
, and
Orange.regression.RandomForestRegressionLearner
)
to obtain the feature scores as calculated by these learners. For example:
>>> learner = Orange.classification.LogisticRegressionLearner()
>>> learner.score_data(data)
[0.31571299907366146,
0.28286199971877485,
0.67496525667835794,
0.99930286901257692]
Feature selection¶
We can use feature selection to limit the analysis to only the most relevant or informative features in the dataset.
Feature selection with a scoring method that works on continuous features will retain all discrete features and vice versa.
The code below constructs a new dataset consisting of two best features according to the ANOVA method:
>>> data = Orange.data.Table("wine")
>>> anova = Orange.preprocess.score.ANOVA()
>>> selector = Orange.preprocess.SelectBestFeatures(method=anova, k=2)
>>> data2 = selector(data)
>>> data2.domain
[Flavanoids, Proline | Wine]
- class Orange.preprocess.SelectBestFeatures(method=None, k=None, threshold=None, decreasing=True)[source]¶
A feature selector that builds a new dataset consisting of either the top k features (if k is an int) or a proportion (if k is a float between 0.0 and 1.0), or all those that exceed a given threshold. Features are scored using the provided feature scoring method. By default it is assumed that feature importance decreases with decreasing scores.
If both k and threshold are set, only features satisfying both conditions will be selected.
If method is not set, it is automatically selected when presented with the dataset. Datasets with both continuous and discrete features are scored using a method suitable for the majority of features.
- Parameters
method (Orange.preprocess.score.ClassificationScorer, Orange.preprocess.score.SklScorer) – Univariate feature scoring method.
k (int or float) – The number or propotion of top features to select.
threshold (float) – A threshold that a feature should meet according to the provided method.
decreasing (boolean) – The order of feature importance when sorted from the most to the least important feature.