CliniMiner devel

CliniMiner - A demonstration of undirected data-mining method for detecting unexpected relationships in large data sets.

CliniMiner is a demonstration of undirected data-mining method for detecting unexpected relationships in large data sets. This tool discovers and predicts unexpected qualitative and quantitative phenomena in large data sets by unsupervised, that is, undirected, data mining where the combinations of items to examine explodes. By "explodes" is meant that exhaustive combinations of statistical tests or directed queries could take millions, in some case zillions, of years. Such difficulties show up particularly in analysis and use of genomic and clinical data, and they represent a major bottleneck in information-based medicine. Extensive archives of records of just a mere 100 parameters or columns could, in the worst case, contain 10 to the power of 29 combinations, which could appear as significant "rules" in any data-mining output.

In contrast, if one were simply to test a hypothesis in the classical way, that is, if one suspected what was interesting in advance, it could be tested in seconds by statistical tests or tools such as DiscoveryLink; hence, CliniMiner complements these kinds of methods. Pruning heuristics and other shortcuts are possible in CliniMiner and continue to be developed. For example, "rules" are not even explored in CliniMiner if the abundance of the component items or events predicts that the "rules" would have insufficient information content. The output data are estimates, and bias is in favor of not missing a potentially useful discovery, hence classical methods must be applied in order to verify the discoveries.

This tool combines aspects of data mining, information theory, and number theory (and even ultimately quantum theory) that directly address the hot mathematical topic known as "The Theory of Expected Information," more recently referred to as "Zeta Theory" (ZT).