Data Mining and Knowledge Discovery
This task includes constructive data preparation operations such as the production of derived attributes, entire new records or transformed values for existing attributes.
Activities Derived attributes - Decide if any attribute should be normalized (e.g. when using a clustering algorithm with age and income in lire, the income will dominate). - Consider adding new information on the relevant importance of attributes by adding new attributes (for example, attribute weights, weighted normalization). - How can missing attributes be constructed or imputed? [Decide type of construction (e.g., aggregate, average, induction)]. - Add new attributes to the accessed data. Single-attribute transformations - Specify necessary transformation steps in terms of available transformation facilities (for example. change a binning of a numeric attribute). - Perform transformation steps. Check for available techniques if needed (e.g., mechanisms prototypes for each segment of segmented data).
Outputs Derived attributes Derived attributes are new attributes that are constructed from one or more existing attributes in the same record. Examples: area = length * width. Generated records Describe the creation of completely new records. Example: create records for customers who made no purchase during the past year. There was no reason to have such records in the raw data, but for modeling purposes it might make sense to explicitly represent the fact that certain customers made zero purchases.
These are methods whereby information is combined from multiple tables or records to create new records or values.
Activities -Check integration facilities if they are able to integrate the input sources as required. - Integrate sources and store result. - Reconsider Data Selection Criteria (See Task 2.1) in light of experiences of data integration (i.e. may wish include/exclude other sets of data).
Output Merged data Merging tables refers to joining together two or more tables that have different information about the same objects. Example: a retail chain has one table with information about each store’s general characteristics (e.g., floor space, type of mall), another table with summarized sales data (e.g., profit, percent change in sales from previous year) and another with information about the demographics of the surrounding area. Each of these tables contains one record for each store. These tables can be merged together into a new table with one record for each store, combining fields from the source tables. Merged data also covers aggregations. Aggregation refers to operations where new values are computed by summarizing together information from multiple records and/or tables. For example, converting a table of customer purchases where there is one record for each purchase into a new table where there is one record for each customer, with fields such as number of purchases, average purchase amount, percent of orders charged to credit card, percent of items under promotion, etc.
Formatting transformations refer to primarily syntactic modifications made to the data that do not change its meaning, but might be required by the modeling tool.
Activities Rearranging attributes Some tools have requirements on the order of the attributes, such as the first field being a unique identifier for each record or the last field being the outcome field the model is to predict. Reordering records It might be important to change the order of the records in the dataset. Perhaps the modeling tool requires that the records be sorted according to the value of the outcome attribute. Reformatted within-value - These are purely syntactic changes made to satisfy the requirements of the specific modeling tool. - Reconsider Data Selection Criteria (See Task 2.1) in light of experiences of data cleaning (i.e. may wish include/exclude other sets of data).
Reformatted data Some tools have requirements on the order of the attributes, such as the first field being a unique identifier for each record or the last field being the outcome field the model is to predict. It might be important to change the order of the records in the dataset. Perhaps the modeling tool requires that the records be sorted according to the value of the outcome attribute. A common situation is that the records of the dataset are initially ordered in some way but the modeling algorithm needs them to be in a fairly random order. For example, when using neural networks it is generally best for the records to be presented in a random order although some tools handle this automatically without explicit user intervention.
In this phase, various modeling techniques are selected and applied and their parameters are calibrated to optimal values. Typically, there are several techniques for the same data mining problem type. Some techniques have specific requirements on the form of data. Therefore, stepping back to the data preparation phase is often necessary.
Background Collate the information that is known about the organization’s business situation at the start of the project. These details not only serve to more closely identify the business goals that will be solved, but also serve to identify resources, both human and material, that may be used or needed during the course of the project.
As the first step in modeling, select the actual modeling technique that is to be used. Whereas you possibly already selected a tool in business understanding, this task refers to the specific modeling technique, e.g., decision tree building with C4.5 or neural network generation with back propagation. If multiple techniques are applied, perform this task for each technique separately.
Outputs Modeling technique Document the actual modeling technique that is to be used. Modeling assumptions Many modeling techniques make specific assumptions on the data, e.g., all attributes have uniform distributions, no missing values allowed, class attribute must be symbolic etc. Record any such assumptions made.
Before we actually build a model, we need to generate a procedure or mechanism to test the model’s quality and validity. For example, in supervised data mining tasks such as classification, it is common to use error rates as quality measures for data mining models. Therefore, we typically separate the dataset into train and test set, build the model on the train set and estimate its quality on the separate test set.
Output Test design Describe the intended plan for training, testing and evaluating the models. A primary component of the plan is to decide how to divide the available dataset into training data, test data and validation datasets.
Run the modeling tool on the prepared dataset to create one or more models.
Output Parameter settings With any modeling tool, there are often a large number of parameters that can be adjusted. List the parameters and their chosen value, along with the rationale for the choice of parameter settings. Models These are the actual models produced by the modeling tool, not a report. Model description Describe the resultant model. Report on the interpretation of the models and document any difficulties encountered with their meanings.
The data mining engineer interprets the models according to his domain knowledge, the data mining success criteria and the desired test design. This task interferes with the subsequent evaluation phase. Whereas the data mining engineer judges the success of the application of modeling and discovery techniques more technically, he contacts business analysts and domain experts later in order to discuss the data mining results in the business context. Moreover, this task only considers models whereas the evaluation phase also takes into account all other results that were produced in the course of the project. The data mining engineer tries to rank the models. He assesses the models according to the evaluation criteria. As far as possible he also takes into account business objectives and business success criteria. In most data mining projects, the data mining engineer applies a single technique more than once or generates data mining results with different alternative techniques. In this task, he also compares all results according to the evaluation criteria.
Outputs Model assessment Summarize results of this task, list qualities of generated models (e.g., in terms of accuracy) and rank their quality in relation to each other. Revised parameter settings According to the model assessment, revise parameter settings and tune them for the next run in the Build Model task. Iterate model building and assessment until you strongly believe that you found the best model(s). Document all such revisions and assessments.
At this stage in the project you have built a model (or models) that appears to have high quality from a data analysis perspective. Before proceeding to final deployment of the model, it is important to more thoroughly evaluate the model and review the steps executed to construct the model to be certain it properly achieves the business objectives. A key objective is to determine if there is some important business issue that has not been sufficiently considered. At the end of this phase, a decision on the use of the data mining results should be reached.
Previous evaluation steps dealt with factors such as the accuracy and generality of the model. This step assesses the degree to which the model meets the business objectives and seeks to determine if there is some business reason why this model is deficient. Another option of evaluation is to test the model(s) on test applications in the real application if time and budget constraints permit. Moreover, evaluation also assesses other data mining results generated. Data mining results cover models which are necessarily related to the original business objectives and all other findings which are not necessarily related to the original business objectives but might also unveil additional challenges, information or hints for future directions.
Output Assessment of data mining results with respect to business success criteria Summarize assessment results in terms of business success criteria including a final statement whether the project already meets the initial business objectives. Approved models After model assessment with respect to business success criteria, the generated models that meet the selected criteria become approved models.
At this point the resultant model hopefully appears to be satisfactory and to satisfy business needs. It is now appropriate to do a more thorough review of the data mining engagement in order to determine if there is any important factor or task that has somehow been overlooked. This review also covers quality assurance issues, e.g., did we correctly build the model? Did we only use attributes that we are allowed to use and that are available for future analyses?
Activities - Give an overview of the data mining process used - Analyze data mining process For each stage of the process: - Was it necessary in retrospect? - Was it executed optimally? - In what ways could it be improved? - Identify failures. - Identify misleading steps. - Identify possible alternative actions, unexpected paths in the process. - Review data mining results with respect to business success criteria.
Output Review of process Summarize the process review and highlight activities that have been missed and/or should be repeated.
According to the assessment results and the process review, the project decides how to proceed at this stage. The project needs to decide whether to finish this project and move on to deployment if appropriate or whether to initiate further iterations or set up new data mining projects. This task includes analyses of remaining resources and budget that influences the decisions.
Outputs List of possible actions List the potential further actions along with the reasons for and against each option. Decision Describe the decision as to how to proceed along with the rationale.
Creation of the model is generally not the end of the project. Even if the purpose of the model is to increase knowledge of the data, the knowledge gained will need to be organized and presented in a way that the customer can use it. It often involves applying “live” models within an organization’s decision making processes, for example in real-time personalization of Web pages or repeated scoring of marketing databases. However, depending on the requirements, the deployment phase can be as simple as generating a report or as complex as implementing a repeatable data mining process across the enterprise. In many cases it is the customer, not the data analyst, who carries out the deployment steps. However, even if the analyst will not carry out the deployment effort it is important for the customer to understand up front what actions need to be carried out in order to actually make use of the created models.
In order to deploy the data mining result(s) into the business, this task takes the evaluation results and concludes a strategy for deployment. If a general procedure has been identified to create the relevant model(s), this procedure is documented here for later deployment.
Activities - Summarize deployable results. - Develop and evaluate alternative plans for deployment. - Decide for each distinct knowledge or information result. - How will the knowledge or information be propagated to its users? - How will the use of the result be monitored or its benefits measured (where applicable)? - Decide for each deployable model or software result. - How will the model or software result be deployed within the organization’s systems? - How will its use be monitored and its benefits measured (where applicable)? - Identify possible problems when deploying the data mining results (pitfalls of the deployment).
Output Deployment plan Summarize deployment strategy including necessary steps and how to perform them.
Monitoring and maintenance are important issues if the data mining result becomes part of the day-to-day business and its environment. A careful preparation of a maintenance strategy helps to avoid unnecessarily long periods of incorrect usage of data mining results. In order to monitor the deployment of the data mining result(s), the project needs a detailed plan on the monitoring process. This plan takes into account the specific type of deployment.
Activities - Check for dynamic aspects (i.e. what things could change in the environment?). - How will accuracy be monitored? - When should the data mining result or model not be used any more? Identify criteria (validity, threshold of accuracy, new data, change in the application domain, etc.)? What should happen if the model or result could no longer be used? (Update model, set up new data mining project, etc.). - Will the business objectives of the use of the model change over time? Fully document the initial problem the model was attempting to solve. - Develop monitoring and maintenance plan.
Output Monitoring and maintenance plan Summarize monitoring and maintenance strategy including necessary steps and how to perform them.
At the end of the project, the project leader and his team write up a final report. Depending on the deployment plan, this report may be only a summary of the project and its experiences (if they have not already been documented as an ongoing activity) or it may be a final and comprehensive presentation of the data mining result(s).
output Final report This is the final written report of the data mining engagement. It includes all of the previous deliverables and summarizes and organizes the results. Final presentation There will also often be a meeting at the conclusion of the project where the results are verbally presented to the customer.
Assess what went right and what went wrong, what was done well and what needs to be improved.
Activities - Interview all significant people involved in the project and ask them about their experiences during the project. - If end users in the business work with the data mining result(s), interview them: are they satisfied? What could have been done better? Do they need additional support? - Summarize feedback and write the experience documentation - Analyze the process (things that worked well, mistakes made, lessons learned, etc.). - Document the specific data mining process (How can the results and the experience of applying the model be fed back into the process?). - Abstract from details to make the experience useful for future projects.
Output Experience documentation Summarize important experiences made during the project. For example, pitfalls, misleading approaches or hints for selecting the best suited data mining techniques in similar situations could be part of this documentation. In ideal projects, experience documentation covers also any reports that have been written by individual project members during the project phases and their tasks.
The results of the Business Understanding phase can be summarized in one report. We suggest the following sections: Background The Background provides a basic overview of the project context. This lists what area the project is working in, what problems have been identified and why data mining appears to provide a solution. Business objectives and success criteria Business Objectives describe what the goals of the project are in business terms. For each objective, Business Success Criteria, i.e. explicit measures for determining whether or not the project succeeded in its objectives, should be provided. This section should also list objectives that were considered but rejected. The rationale of the selection of objectives should be given. Inventory of resources The Inventory of Resources aims to identify personnel, data sources, technical facilities and other resources that may be useful in carrying out the project. Requirements, assumptions and constraints This output lists general requirements about how the project is executed, type of project results, assumptions made about the nature of the problem and the data being used and constraints imposed on the project. Risks and contingencies This output identifies problems that may occur in the project, describes the consequences and states what action can be taken to minimize the effect. Terminology The Terminology allows people unfamiliar with the problems being addressed by the project to become more familiar with them. Costs and benefits This describes the costs of the project and predicted business benefits if the project is successful (e.g. return on investment). Other less tangible benefits (e.g. customer satisfaction) should also be highlighted. Data mining goals and success criteria The data mining goals state the results of the project that enable the achievement of the business objectives. As well as listing the probable data mining approaches, the success criteria for the results should also be listed in data mining terms. Project plan This lists the stages to be executed in the project, together with duration, resources required, inputs, outputs and dependencies. Where possible it should make explicit the large-scale iterations in the data mining process, for example repetitions of the modeling and evaluation phases. Initial assessment of tools and techniques This section gives an initial view of what tools and techniques are likely to be used and how. It describes the requirements for tools and techniques, lists available tools and techniques and matches them to requirements.
Business objectives Describe the customer’s primary objective, from a business perspective, in the data mining project. In addition to the primary business objective, there are typically a large number of related business questions that the customer would like to address. For example, the primary business goal might be to keep current customers by predicting when they are prone to move to a competitor, while secondary business objectives might be to determine whether lower fees affect only one particular segment of customers.
Activities Informally describe the problem which is supposed to be solved with data mining. - Specify all business questions as precisely as possible. - Specify any other business requirements (e.g., the business does not want to lose any customers). - Specify expected benefits in business terms.
Describe the customer’s primary objective, from a business perspective, in the data mining project. In addition to the primary business objective, there are typically a large number of related business questions that the customer would like to address. For example, the primary business goal might be to keep current customers by predicting when they are prone to move to a competitor, while secondary business objectives might be to determine whether lower fees affect only one particular segment of customers. Describe the criteria for a successful or useful outcome to the project from the business point of view. This might be quite specific and readily measurable, such as reduction of customer churn to a certain level or general and subjective such as “give useful insights into the relationships.” In the latter case it should be indicated who would make the subjective judgment.
application success criteria description
business objectives and success criteria information
List the resources available to the project, including: personnel (business and data experts, technical support, data mining personnel), data (fixed extracts, access to live warehoused or operational data), computing resources (hardware platforms), software (data mining tools, other relevant software).
List all requirements of the project including schedule of completion, comprehensibility and quality of results and security as well as legal issues. As part of this output, make sure that you are allowed to use the data. List the assumptions made by the project. These may be assumptions about the data, which can be checked during data mining, but may also include non-checkable assumptions about the business upon which the project rests. It is particularly important to list the latter if they form conditions on the validity of the results. List the constraints made on the project. These constraints might involve lack of resources to carry out some of the tasks in the project within the timescale required or there may be legal or ethical constraints on the use of the data or the solution needed to carry out the data mining task.
requirements & assumptions & constraints list
List the risks, that is, events that might occur, impacting schedule, cost or result. List the corresponding contingency plans; what action will be taken to avoid or minimize the impact or recover from the occurrence of the foreseen risks.
Compile a glossary of terminology relevant to the project. This should include at least two components: (1) A glossary of relevant business terminology, which forms part of the business understanding available to the project. (2) A glossary of data mining terminology, illustrated with examples relevant to the business problem in question.
Prepare a cost-benefit analysis for the project, comparing the costs of the project with the potential benefit to the business if it is successful.
Describe the intended outputs of the project that enable the achievement of the business objectives. Note that these are normally technical outputs. Define the criteria for a successful outcome to the project in technical terms, for example a certain level of predictive accuracy or a propensity to purchase profile with a given degree of “lift.” As with business success criteria, it may be necessary to describe these in subjective terms, in which case the person or persons making the subjective judgment should be identified.
Define the criteria for a successful outcome to the project in technical terms, for example a certain level of predictive accuracy or a propensity to purchase profile with a given degree of “lift.” As with business success criteria, it may be necessary to describe these in subjective terms, in which case the person or persons making the subjective judgment should be identified.
data mining success criteria description
tools & tecniques assesement description
This report describes how the different data sources identified in the inventory were captured and extracted. Topics to be covered: - Background of data. - List of data sources with broad area of required data covered by each. - For each data source, method of acquisition or extraction. - Problems encountered in data acquisition or extraction.
Each dataset acquired is described. Topics to be covered: - Each data source described in detail. - List of tables (may be only one) or other database objects. - Description of each field including units, codes used, etc.
Describes the data exploration and its results. Topics to be covered: - Background including broad goals of data exploration. For each area of exploration undertaken: - Expected regularities or patterns. - Method of detection. - Regularities or patterns found, expected and unexpected. - Any other surprises. - Conclusions for data transformation, data cleaning and any other pre-processing. - Conclusions related to data mining goals or business objectives. - Summary of conclusions.
This initial phase focuses on understanding the project objectives and requirements from a business perspective, then converting this knowledge into a data mining problem definition and a preliminary plan designed to achieve the objectives.
The first objective of the data analyst is to thoroughly understand, from a business perspective, what the client really wants to accomplish. Often the client has many competing objectives and constraints that must be properly balanced. The analyst’s goal is to uncover important factors, at the beginning, that can influence the outcome of the project. A possible consequence of neglecting this step is to expend a great deal of effort producing the right answers to the wrong questions.
Outputs: Background Record the information that is known about the organization’s business situation at the beginning of the project. Business objectives Describe the customer’s primary objective, from a business perspective. In addition to the primary business objective, there are typically other related business questions that the customer would like to address. For example, the primary business goal might be to keep current customers by predicting when they are prone to move to a competitor. Examples of related business questions are “How does the primary channel (e.g., ATM, visit branch, internet) a bank customer uses affect whether they stay or go?” or “Will lower ATM fees significantly reduce the number of high-value customers who leave?” Business success criteria Describe the criteria for a successful or useful outcome to the project from the business point of view. This might be quite specific and able to be measured objectively, such as reduction of customer churn to a certain level or general and subjective such as “give useful insights into the relationships.” In the latter case it should be indicated who makes the subjective judgment.
This task involves more detailed fact-finding about all of the resources, constraints, assumptions and other factors that should be considered in determining the data analysis goal and project plan. In the previous task, your objective is to quickly get to the crux of the situation. Here, you want to flesh out the details.
Outputs: Inventory of resources List the resources available to the project, including: personnel (business experts, data experts, technical support, data mining personnel), data (fixed extracts, access to live warehoused or operational data), computing resources (hardware platforms) and software (data mining tools, other relevant software). Requirements, assumptions and constraints List all requirements of the project including schedule of completion, comprehensibility and quality of results and security as well as legal issues. As part of this output, make sure that you are allowed to use the data. List the assumptions made by the project. These may be assumptions about the data that can be checked during data mining, but may also include non-checkable assumptions about the business upon which the project rests. It is particularly important to list the latter if they form conditions on the validity of the results. List the constraints on the project. These may be constraints on the availability of resources, but may also include technological constraints such as the size of data that it is practical to use for modeling. Risks and contingencies List the risks or events that might occur to delay the project or cause it to fail. List the corresponding contingency plans; what action will be taken if the risks happen. Terminology Compile a glossary of terminology relevant to the project. This may include two components: (1) A glossary of relevant business terminology, which forms part of the business understanding available to the project. Constructing this glossary is a useful “knowledge elicitation” and education exercise. (2) A glossary of data mining terminology, illustrated with examples relevant to the business problem in question. Costs and benefits Construct a cost-benefit analysis for the project, which compares the costs of the project with the potential benefit to the business if it is successful. The comparison should be as specific as possible, for example using monetary measures in a commercial situation.
A business goal states objectives in business terminology. A data mining goal states project objectives in technical terms. For example, the business goal might be “Increase catalog sales to existing customers.” A data mining goal might be “Predict how many widgets a customer will buy, given their purchases over the past three years, demographic information (age, salary, city, etc.) and the price of the item.”
Outputs: Data mining goals Describe the intended outputs of the project that enables the achievement of the business objectives. Data mining success criteria Define the criteria for a successful outcome to the project in technical terms, for example a certain level of predictive accuracy or a propensity to purchase profile with a given degree of “lift.” As with business success criteria, it may be necessary to describe these in subjective terms, in which case the person or persons making the subjective judgment should be identified.
Describe the intended plan for achieving the data mining goals and thereby achieving the business goals. The plan should specify the anticipated set of steps to be performed during the rest of the project including an initial selection of tools and techniques.
Outputs: Project plan List the stages to be executed in the project, together with duration, resources required, inputs, outputs and dependencies. Where possible make explicit the large-scale iterations in the data mining process, for example repetitions of the modeling and evaluation phases. As part of the project plan, it is also important to analyze dependencies between time schedule and risks. Mark results of these analyses explicitly in the project plan, ideally with actions and recommendations if the risks appear. Note: the project plan contains detailed plans for each phase. For example, decide at this point which evaluation strategy will be used in the evaluation phase. The project plan is a dynamic document in the sense that at the end of each phase a review of progress and achievements is necessary and an update of the project plan accordingly is recommended. Specific review points for these reviews are part of the project plan, too. Initial assessment of tools and techniques At the end of the first phase, the project also performs an initial assessment of tools and techniques. Here, you select a data mining tool that supports various methods for different stages of the process, for example. It is important to assess tools and techniques early in the process since the selection of tools and techniques possibly influences the entire project.
The data understanding phase starts with an initial data collection and proceeds with activities in order to get familiar with the data, to identify data quality problems, to discover first insights into the data or to detect interesting subsets to form hypotheses for hidden information.
Examine the “gross” or “surface” properties of the acquired data and report on the results.
Output Data description report Describe the data which has been acquired, including: the format of the data, the quantity of data, for example number of records and fields in each table, the identities of the fields and any other surface features of the data which have been discovered. Does the data acquired satisfy the relevant requirements?
This task tackles the data mining questions, which can be addressed using querying, visualization and reporting. These include: distribution of key attributes, for example the target attribute of a prediction task; relations between pairs or small numbers of attributes; results of simple aggregations; properties of significant sub-populations; simple statistical analyses. These analyses may address directly the data mining goals; they may also contribute to or refine the data description and quality reports and feed into the transformation and other data preparation needed for further analysis.
Output Data exploration report Describe results of this task including first findings or initial hypothesis and their impact on the remainder of the project. If appropriate, include graphs and plots, which indicate data characteristics or lead to interesting data subsets for further examination.
Examine the quality of the data, addressing questions such as: is the data complete (does it cover all the cases required)? Is it correct or does it contain errors and if there are errors how common are they? Are there missing values in the data? If so how are they represented, where do they occur and how common are they?
Output Data quality report List the results of the data quality verification; if quality problems exist, list possible solutions. Solutions to data quality problems generally depend heavily on both data and business knowledge. 22
The data preparation phase covers all activities to construct the final dataset (data that will be fed into the modeling tool(s)) from the initial raw data. Data preparation tasks are likely to be performed multiple times and not in any prescribed order. Tasks include table, record and attribute selection as well as transformation and cleaning of data for modeling tools.
Outputs Dataset This is the dataset (or datasets) produced by the data preparation phase, which will be used for modeling or the major analysis work of the project. Dataset description Describe the dataset (or datasets) that will be used for the modeling or the major analysis work of the project.
Decide on the data to be used for analysis. Criteria include relevance to the data mining goals, quality and technical constraints such as limits on data volume or data types. Note that data selection covers selection of attributes (columns) as well as selection of records (rows) in a table.
Output Rationale for inclusion/exclusion List the data to be included/excluded and the reasons for these decisions.
Raise the data quality to the level required by the selected analysis techniques. This may involve selection of clean subsets of the data, the insertion of suitable defaults or more ambitious techniques such as the estimation of missing data by modeling.
Output Data cleaning report Describe what decisions and actions were taken to address the data quality problems reported during the verify data quality task of the data understanding phase. Transformations of the data for cleaning purposes and the possible impact on the analysis results should be considered.
This report describes the completeness and accuracy of the data. Topics to be covered: - Background including broad expectations about data quality. For each dataset: - Approach taken to assess data quality. - Results of data quality assessment. - Summary of data quality conclusions.
This provides a description of the dataset (after pre-processing) and the process by which it was produced. Topics to be covered: - Background including broad goals and plan for pre-processing. - Rationale for inclusion/exclusion of datasets. For each included dataset: - Description of the pre-processing, including the actions that were necessary to address any data quality issues. - Detailed description of the resultant dataset, table by table and field by field. - Rationale for inclusion/exclusion of attributes. - Discoveries made during pre-processing and any implications for further work. - Summary and conclusions.
Describe the decisions and actions that were taken to address the data quality problems reported during the Verify Data Quality Task. The report should also address what data quality issues are still outstanding if the data is to be used in the data mining exercise and what possible affects that could have on the results.
Activities - Reconsider how to deal with observed type of noise. - Correct, remove or ignore noise. - Decide how to deal with special values and their meaning. The area of special values can give rise to many strange results and should be carefully examined. Examples of special values could arise through taking results of a survey where some questions were not asked or nor answered. This might result in a value of ‘99’ for unknown data. For example, 99 for marital status or political affiliation. Special values could also arise when data is truncated – e.g. ‘00’ for 100 year old people or all cars with 100,000 km on the clock. - Reconsider Data Selection Criteria (See Task 2.1) in light of experiences of data cleaning (i.e. one may wish include/exclude other sets of data). Good idea! Remember that some fields may be irrelevant to the data mining goals and therefore noise in those fields has no significance. However, if noise is ignored for these reasons, it should be fully documented as the circumstances may change later!
The outputs produced during the Modeling phase can be combined into one report. We suggest the following sections: Modeling assumption This section defines explicitly any assumptions made about the data and any assumptions that are implicit in the modeling technique to be used. Test design This section describes how the models are built, tested and evaluated. Topics to be covered: - Background – outlines the modeling undertaken and its relation to the data mining goals. For each modeling task: - Broad description of the type of model and the training data to be used. - Explanation of how the model will be tested or assessed. - Description of any data required for testing. - Plan for production of test data if any. - Description of any planned examination of models by domain or data experts. - Summary of test plan. Model description This report describes the delivered models and overviews the process by which they were produced. Topics to be covered: - Overview of models produced. For each model: - Type of model and relation to data mining goals. - Parameter settings used to produce the model. - Detailed description of the model and any special features. For example: - For rule-based models, list the rules produced plus any assessment of per-rule or overall model accuracy and coverage. - For opaque models, list any technical information about the model (such as neural network topology) and any behavioral descriptions produced by the modeling process (such as accuracy or sensitivity). - Description of Model’s behavior and interpretation. - Conclusions regarding patterns in the data (if any); sometimes the model will reveal important facts about the data without a separate assessment process (e.g. that the output or conclusion is duplicated in one of the inputs). - Summary of conclusions. Model assessment This section describes the results of testing the models according to the test design. Topics to be covered: - Overview of assessment process and results including any deviations from the test plan. For each model: - Detailed assessment of model including measurements such as accuracy and interpretation of behavior. - Any comments on models by domain or data experts. - Summary assessment of model. - Insights into why a certain modeling technique and certain parameter settings led to good/bad results. - Summary assessment of complete model set.
Modeling technique Record the actual modeling technique that is used. Activities Decide on appropriate technique for exercise bearing in mind the tool selected.
With any modeling tool, there are often a large number of parameters that can be adjusted. List the parameters and their chosen values, along with the rationale for the choice.
Describe the resultant model and assess its expected accuracy, robustness and possible shortcomings. Report on the interpretation of the models and any difficulties encountered.
Summarize results of this task, list qualities of generated models (e.g. in terms of accuracy) and rank their quality in relation to each other.
At the end of the project, the project leader and his team write up a final report. It depends on the deployment plan, if this report is only a summary of the project and its experiences or if this report is a final presentation of the data mining result(s).
Activities - Identify what reports are needed (slide presentation, management summary, detailed findings, explanation of models, etc.). - Analyze how well initial data mining goals have been met. - Identify target groups for report. - Outline structure and contents of report(s). - Select findings to be included in the reports. - Write a report. - Decide on target group for final presentation (will they already have received final report?). - Select which items from final report should be included in final presentation.
Output Final report At the end of the project, there will be (at least one) final report where all the threads are brought together. As well as identifying the results obtained, the report should also describe the process, show which costs have been incurred, define any deviations from the original plan, describe implementation plans and make any recommendations for future work. The actual detailed content of the report depends very much on the audience for the particular report. Output Final presentation As well as a Final Report, it may be necessary to make a Final Presentation to summarize the project – maybe to the management sponsor, for example. The Presentation normally contains a subset of the information contained in the Final Report, but structured in a different way.
- Describe any characteristics of the current model that may be useful for the future. - Give a detailed description of the model and any special features. - For rule-based models, list the rules produced plus any assessment of per-rule or overall model accuracy and coverage. - For opaque models, list any technical information about the model (such as neural network topology) and any behavioral descriptions produced by the modeling process (such as accuracy or sensitivity). - Describe the model’s behavior and interpretation. - State conclusions regarding patterns in the data (if any); sometimes the model reveals important facts about the data without a separate assessment process (e.g. that the output or conclusion is duplicated in one of the inputs).
- Analyze potentials for deployment of each result.
Derived attributes are new attributes that are constructed from one or more existing attributes in the same record. An example might be area = length * width. Why should we need to construct derived attributes during the course of a data mining investigation? It should not be thought that only data from databases or other sources is the only type of data that should be used in constructing a model. Derived attributes might be constructed because: - Background knowledge convinces us that some fact is important and ought to be represented although we have no attribute currently to represent it. - The modeling algorithm in use handles only certain types of data, for example we are using linear regression and we suspect that there are certain non-linearities that will be not be included in the model. - The outcome of the modeling phase may suggest that certain facts are not being covered.
assesement of data mining results description
- If there is a verbal description of the generated model (e.g. via rules), assess the rules; are they logical, are they feasible, are there too many or too few, do they offend common sense?
Evaluate result with respect to evaluation criteria
evaluate model with respect to evaluation criteria
Volumetric analysis of data - Identify data and method of capture. - Access data sources. - Use statistical analyses if appropriate. - Report tables and their relations. - Check data volume, number of multiples, complexity. - Does the data contain free text entries?
Specify the test design and decide on necessery steps (number of iterations, number of folds etc)
- Compare evaluation results and interpretation.
compare evaluation results and interpretation
- Get insights into why a certain modeling technique and certain parameter settings lead to good/bad results.
- Overview of results deployment and indication of which results may require updating (and why). For each deployed result: - Description of how updating will be triggered (regular updates, trigger event, performance monitoring). - Description of how updating will be performed. - Summary of the results updating process.
monitoring and maintenance plan description
check existing test designs for each data mining goal se[parately
- Get comments on models by domain or data experts.
Attribute types and values - Check accessibility and availability of attributes. - Check attribute types (numeric, symbolic, taxonomy etc.). - Check attribute value ranges. - Analyze attribute correlations. - Understand the meaning of each attribute and attribute value in business terms. - For each attribute, compute basic statistics (e.g., compute distribution, average, max, min, standard deviation, variance, mode, skewness, etc.). - Analyze basic statistics and relate the results to their meaning in business terms. - Is the attribute relevant for the specific data mining goal? - Is the attribute meaning used consistently? - Interview domain expert on his opinion of attribute relevance. - Is it necessary to balance the data (Depending on the modeling technique used)?
assess attribute datatypes and values action
plan monitoring and maintenance protocol
- Test result according to a test strategy (e.g.: Train and Test, Crossvalidation, bootstrapping etc.).
Describe the intended plan for training, testing and evaluating the models. A primary component of the plan is to decide how to divide the available dataset into training data, test data and validation test sets.
- Define any built-in assumptions made by the technique about the data (e.g. quality, format, distribution).
Keys - Analyze key relations. - Check amount of overlaps of key attribute values across tables. Review assumptions/goals - Update list of assumptions if necessary.
- Create a list of selection criteria for tools and techniques (or use an existing one if available) - Choose potential tools and techniques - Evaluate appropriateness of techniques - Review and prioritize applicable techniques according to the evaluation of alternative solutions
Summarize important experiences made during the project. For example, pitfalls, misleading approaches or hints for selecting the best-suited data mining techniques in similar situations could be part of this documentation. In ideal projects, experience documentation covers also any reports that have been written by individual project members during the project phases and their tasks.
Define the initial process plan and discuss the feasibility with all involved personnel. - Put all identified goals and selected techniques together into a coherent procedure that solves the business questions and meets the business success criteria. - Estimate effort and resources needed to achieve and deploy the solution (It is useful to consider other peoples experience when estimating timescales for data mining projects. For example, it is often postulated that 50-70 percent of the time and effort in a data mining project is used in the Data Preparation Phase and 20-30 percent in the Data Understanding Phase, while only 10-20 percent is spent in each of the Modeling, Evaluation and Business Understanding Phases and 5-10 percent in the Deployment Phase.). - Identify critical steps. - Mark decision points. - Mark review points. - Identify major iterations
The final report is used to summarize the project and its results. Contents: - Summary of Business Understanding: background, objectives and success criteria. - Summary of data mining process. - Summary of data mining results. - Summary of results evaluation. - Summary of deployment and maintenance plans. - Cost/benefit analysis. - Conclusions for the business. - Conclusions for future data mining.
- Set initial parameters. - Document reasons for choosing those values.
- Record parameter settings used to produce the model.
- Post-process data mining results (e.g. editing rules, display trees).
monitoring and maintenance plan specification
- Create ranking of results with respect to success and evaluation criteria
Assessment of data mining results with respect to business success criteria This report compares the data mining results with the business objectives and the business success criteria. Topics to be covered: - Review of Business Objectives and Business Success Criteria (which may havechanged during and/or as a result of data mining). For each Business Success Criterion: - Detailed comparison between success criterion and data mining results. - Conclusions about achievability of success criterion and suitability of data mining process. - Review of Project Success; has the project achieved the original Business Objectives? - Are there new business objectives to be addressed later in the project or in new projects? - Conclusions for future data mining projects. Review of process This section assesses the effectiveness of the project and identifies any factors that may have been overlooked that should be taken into consideration if the project is repeated. List of possible actions This section makes recommendations regarding the next steps in the project.
This section specifies the deployment of the data mining results. Topics to be covered: - Summary of deployable results (derived from Next Steps report). - Description of deployment plan.
- Run the selected technique on the input dataset to produce the model.
decide on appropriate technique for exercise bearing in mind the tool selected
Modeling assumptions Many modeling techniques make specific assumptions about the data, data quality or the data format.
- Interpret results in business terms (as far as possible at this stage).
- Decide how to deal with special values and their meaning. The area of special values can give rise to many strange results and should be carefully examined. Examples of special values could arise through taking results of a survey where some questions were not asked or nor answered. This might result in a value of ‘99’ for unknown data. For example, 99 for marital status or political affiliation. Special values could also arise when data is truncated – e.g. ‘00’ for 100 year old people or all cars with 100,000 km on the clock.
Merging tables refers to joining together two or more tables that have different information about the same objects. At this stage it may also be advisable to generate new records. It may also be recommended to generate aggregate values. Aggregation refers to operations where new values are computed by summarizing information from multiple records and/or tables.
- Compare these assumptions with those in the Data Description Report.
Generated records are completely new records, which add new knowledge or represent new data that is not otherwise represented, e.g., having segmented the data, it may be useful to generate a record to represent the prototypical member of each segment for further processing.
List the stages to be executed in the project, together with duration, resources required, inputs, outputs and dependencies. Wherever possible make explicit the large-scale iterations in the data mining process, for example, repetitions of the modeling and evaluation phases. As part of the project plan, it is also important to analyze dependencies between time schedule and risks. Mark results of these analyses explicitly in the project plan, ideally with actions and recommendations for actions if the risks appear. Remember that, although this is the only task in which the Project Plan is directly named, nevertheless it should be continually consulted and reviewed throughout the project. It should at least be consulted whenever a new Task is started or a further iteration of a task or activity is begun.
Activities - Collect appropriate additional data (from different sources – in-house as well as externally). - Perform significance and correlation tests to decide if fields should be included. - Reconsider Data Selection Criteria (See Task 2.1) in light of experiences of data quality, data exploration, (i.e. may wish include/exclude other sets of data). - Reconsider Data Selection Criteria (See Task 2.1) in light of experience of modeling (i.e. model assessment may show that other datasets are needed).
Activities - Check prior availability of glossaries, otherwise begin to draft glossaries. - Talk to domain experts to understand their terminology. - Become familiar with the business terminology.
- Estimate costs for data collection. - Estimate costs of developing and implementing a solution. - Identify benefits when a solution is deployed (e.g. improved customer satisfaction, ROI and increase in revenue). - Estimate operating costs.
- Specify criteria for model assessment (e.g., model accuracy, performance and complexity). - Define benchmarks for evaluation criteria. - Specify criteria which address subjective assessment criteria (e.g. model explain ability and data and marketing insight provided by the model).
specify data mining success criteria action
data mining goal identification protocol
-Translate the business questions to data mining goals (e.g., a marketing campaign requires segmentation of customers in order to decide whom to approach in this campaign; the level/size of the segments should be specified). - Specify data mining problem type (e.g., classification, description, prediction and clustering).
business objectives identification protocol
- Document the rationale for inclusion/exclusion.
Form suppositions for future analysis - Consider and evaluate information and findings in the data descriptions report. - Form hypothesis and identify actions. - Transform hypothesis into a data mining goal if possible. - Clarify data mining goals or make them more precise. Blind search is not necessarily useless but a more directed search towards business objectives is preferable. - Perform basic analysis to verify the hypothesis.
- Select different data subsets (e.g., different attributes, only data which meet certain conditions).
Noise and inconsistencies between sources. - Check consistencies and redundancies between different sources. - Plan how to deal with noise. - Detect type of noise and which attributes are affected. Good idea! Remember that it may be necessary to exclude some data since they do not exhibit either positive or negative behavior (e.g. to check on customers loan behavior, exclude all those who have never loaned, do not finance a home mortgage, those whose mortgage is nearing maturity, etc. Review all assumptions whether they are valid or not given the current information on data and knowledge.
assess noise and inconsistencies in data sources
Data requirements planning - Plan which information is needed (e.g. only given attributes, additional information). - Check if all the information needed (to solve the data mining goals) is actually available.
Data Quality in flat files - If the data is stored in flat files, check which delimiter is used and if it is used consistently within all attributes. - If the data is stored in flat files, check number of fields in each record. Do they coincide?
- Check available techniques for sampling data. Good idea! Based on Data Selection Criteria, decide if one or more attributes are more important than others and weight the attributes accordingly. Decide, based on the context (i.e. application, tool, etc.), how to handle the weighting.
- Check model against given knowledge base to see if the discovered information is novel and useful.
- Consider use of sampling techniques (e.g. a quick solution may involve the reduction of the size of the test dataset or the tool may not be able to handle the full dataset, splitting test and training datasets). It may also be useful to have weighted samples to give different importance to different attributes or different values of the same attribute.
Activities - Reconsider how to deal with observed type of noise.
- Identify special values and catalogue their meaning Review keys, attributes - Check coverage (e.g. are all possibly values represented). - Check keys. - Do the meanings of attributes and contained values fit together? - Identify missing attributes and blank fields. - Meaning of missing data. - Check for attributes with different values that have similar meanings (e.g., low fat, diet). - Check spelling of values (e.g., same value but sometime beginning with a lower case letter, sometimes with an upper case letter). - Check for deviations, decide whether a deviation is noise or may indicate an interesting phenomenon. - Check for plausibility of values, e.g. all fields have the same or nearly the same values. Good idea! Make a review of any attributes that may give answers which conflict with common sense (e.g. teenagers with high income) Use visualization plots, histograms, etc. to show up inconsistencies in the data
- Specify selection criteria (e.g., which attributes are necessary for the specified data mining goals? Which attributes have been identified as being irrelevant? How many attributes can we handle with the chosen techniques?). - Select tables/files of interest. - Select data within a table/file. - Think about how long history one should use even if available (e.g. even if 18 months data is available, maybe only 12 months is needed for the exercise). Beware! Be aware that data collected from different sources may give rise to quality problems when merged (e.g. address files merged with own customer base may show up inconsistencies of format, invalidity of data, etc.).
Insertion of data - If the data contains free text entries, do we need to encode them for modeling or do we want to group specific entries? - How can missing attributes be acquired? - Describe how to extract the data.
Data exploration - Analyze properties of interesting attributes in detail (e.g. basic statistics interesting sub-populations). - Identify characteristics of sub-populations.
Sources of data and knowledge - Identify data sources. - Identify type of data sources (on-line sources, experts, written documentation, etc.). - Identify knowledge sources. - Identify type of knowledge sources (online sources, experts, written documentation, etc.). - Check available tools and techniques. - Describe the relevant background knowledge (informally or formally).
specify the expected benefits in business terms
Develop contingency plans - Determine conditions under which each risk may occur. - Develop contingency plans.
Constraints - Check general constraints (e.g., legal issues, budget, timescales, and resources) - Check access rights to data sources (e.g., access restrictions, password required) - Check technical accessibility of data (operating systems, data management system, file or database format) - Check whether relevant knowledge is accessible - Check budget constraints (fixed costs, implementation costs, etc.)
Identify risks - Identify business risks (e.g., competitor comes up with better results first). - Identify organizational risks (e.g., department requesting project not having funding for project). - Identify financial risks (e.g., further funding depends on initial data mining results). - Identify technical risks. - Identify risks that depend on data and data sources (e.g. poor quality and coverage).
Requirements - Specify target group profile. - Capture all requirements on scheduling. - Capture requirements on comprehensibility, accuracy, deploy ability, maintainability and repeatability of the data mining project and the resulting model(s). - Capture requirements on security, legal restrictions, privacy, reporting and project schedule.
Organization Develop organizational charts identifying divisions, departments and project groups. The chart should also identify managers’ names and responsibilities. - Identify key persons in the business and their roles. - Identify an internal sponsor (financial sponsor and primary user/ domain expert). - Is there a steering committee and who are the members? - Identify the business units which are impacted by the data mining project (e.g., Marketing, Sales, Finance).
specify the information about organization
Current solution - Describe any solution currently in use for the problem. - Describe the advantages and disadvantages of the current solution and the level to which it is accepted by the users.
specify the information about the current solution
Personnel sources - Identify project sponsor - Identify system administrator, database administrator and technical support staff for further questions. - Identify market analysts, data mining experts and statisticians and check their availability. - Check availability of domain experts for later phases.
Assumptions - Clarify all assumptions (including implicit ones) and make them explicit (e.g., To address the business question, a minimum number of customers with age above 50 is necessary). - List assumptions on data quality (e.g., accuracy, availability). - List assumptions on external factors (e.g., economic issues, competitive products, technical advances). - Clarify assumptions that lead to any of the estimates (e.g., the price of a specific tool is assumed to be lower than $1000). - List all assumptions on whether it is necessary to understand and describe or explain the model. (E.g., How should the model and results be presented to senior management/sponsor.).
Problem area - Identify the problem area (e.g., Marketing, Customer Care, Business Development, etc.). - Describe the problem in general terms. - Check the current status of the project (e.g., Check if it is already clear within the business unit that we are performing a data mining project or do we need to advertise data mining as a key technology in the business?). - Clarify prerequisites of the project (e.g., what is the motivation of the project? Does the business already use data mining?). - If necessary, prepare presentations and present data mining to the business. - Identify target groups for the project result (e.g., Do we expect a written report for top management or do we expect a running system that is used by naive end users?). - Identify the users’ needs and expectations.
specify the information about the problem area
Hardware resources - Identify the base hardware. - Establish the availability of the base hardware for the data mining project. - Check if the hardware maintenance schedule conflicts with the availability of the hardware for the data mining project. - Identify the hardware available for the data mining tool to be used (if the tool is known at this stage).
identify who asesses the success criteria
Annotation property imported from OBI to denote terms that have been imported from other ontologies
journal article
data item
A textual entity that expresses the results of reasoning about a problem, for instance as typically found towards the end of scientific papers.
conclusion textual entity
denotes
A document that has been accepted by a publisher
publication
publication about an investigation
a planned process of capturing information in an enduring form with the intent to communicate this information
a planned process that consists of parts: planning, study design execution, documentation and which produce conclusion(s).
is a process to encode an information entity into a digital document
a protocol is a plan specification which has sufficient level of detail and quantitative information to communicate it between domain experts, so that different domain experts will reliably be able to independently reproduce the process.
protocol
Is a relationship between a generically dependent continuant and a specifically dependent continuant. A generically dependent continuant may inhere in more than one entity. It does so by virtue of the fact that there is, for each entity that it inheres, a specifically dependent *concretization* of the generically dependent continuant that is specifically dependent. For instance, consider a story, which is an information artifact that inheres in some number of books. Each book bears some quality that carries the story. The relation between this quality and the generically dependent continuant is that the former is the concretization of the latter.
Is a relationship between a specifically dependent continuant and a generically dependent continuant. A generically dependent continuant may inhere in more than one entity. It does so by virtue of the fact that there is, for each entity that it inheres, a specifically dependent *concretization* of the generically dependent continuant that is specifically dependent. For instance, consider a story, which is an information artifact that inheres in some number of books. Each book bears some quality that carries the story. The relation between this quality and the generically dependent continuant is that the former is the concretization of the latter.
Relation between a realizable and a process. Reciprocal relation of realizes [GOC:cjm]
Relation between a process and a function, where the unfolding of the process requires the execution of the function. Class level: P realizes F iff: given any p that instantiates P, there exists some f, t such that f instantiates F at t and p *realizes* f. Here, *realizes* is the primitive instance level relation [GOC:cjm]
a process of creating or modifying a plan.
a planned process that realizes the concretization of a study design
study design execution
A textual entity that describes an investigation.
investigation description
A textual entity that denotes an investigation.
investigation title
a planned process with objective to check that the accuracy or the quality of a claim or prediction satisfies some criteria and which is assessed by comparing with independent results
A study design is a plan specification comprised of protocols (which may specify how and what kinds of data will be gathered) that are executed as part of an investigation and is realized during a study design execution.
study design
Also defined in CRISP-DM: Acquire within the project the data (or access to the data) listed in the project resources. This initial collection includes data loading if necessary for data understanding. For example, if you apply a specific tool for data understanding, it makes perfect sense to load your data into this tool. This effort possibly leads to initial data preparation steps. Note: if you acquire multiple data sources, integration is an additional issue, either here or in the later data preparation phase. Output Initial data collection report List the dataset (or datasets) acquired, together with their locations within the project, the methods used to acquire them and any problems encountered. Record problems encountered and any solutions achieved to aid with future replication of this project or with the execution of similar future projects.