iKnow and DeepSee. Agenda

Agenda
What is iKnow?
Semantic Analysis.
%iKnow.Queries
Matching Dictionaries.
%iKnow.Semantics.
Newer features:
Attribute Customizations.
iFind.
Text Categorization.
iKnow features in DeepSee.
Configuring iKnow and DeepSee.

What is iKnow?
iKnow is a semantic analysis tool.
Indexes the concepts and relations within text for querying and analysis.
Uses language models rather than training data or ontologies to detect relations.
Supported languages: Dutch, English, French, German, Portuguese, Russian, Spanish, Ukrainian, Swedish*, and Japanese*.
Sources of text include: Plain text files, SQL fields, social media.
*Support added in 2016.1 release.

Semantic Analysis: Relations, Concepts, Negation
patient
suffered from
acute hypertension
chest pain
but did not mention
The patient suffered from acute hypertension but did not mention any chest pain.

Semantic Analysis Results

Importance of Language Models
iKnow indexing is subject matter neutral.
A language model applies to any text written in the language: medical, legal, scientific, business, and so on.
iKnow indexing automatically detects meaningful word groups.
Labels “acute hypertension” and “chest pain” as concepts.
Labels “but did not mention chest pain” as a negation context.
No need for ontologies or training data.

%iKnow.Queries
Includes:
GetTop() – Most frequently occurring entities across a set of sources.
GetRelated() – Entities in a relationship with the supplied entity.
GetByEntities() – All CRCs or paths containing a particular set of entities.
GetSummary() – Most relevant sentences in a source.
GetSimilar() – Entities similar to a given entity.

Matching Dictionary
User provided group of related terms.
Provides external (domain) knowledge to iKnow results.
Allows for coarser grained analysis.
Example (2001 A Space Odyssey):
hal ? hal.
hal9000 ? hal.
heuristic algorithm computer ? hal.
iKnow smart matching mechanism returns a match score.
Configurable threshold for matches.

iKnow Architect (2016.1)
Management Portal Tool for creating, configuring, and managing iKnow domains.
Domain Settings, Metadata, Data Locations, Blacklists
Compile and build domains.
Launch indexing and knowledge portal pages.
Some iKnow features not supported by Architect. Edit class definition using IDE.
Matching Dictionaries.

Demonstration

%iKnow.Semantics (2012.2+)
Introduces concept of dominant entities.
Most important entities not most common.
Algorithm revised for 2015.2 release.
Explained in documentation.
Includes queries:
GetBySource() – Dominant elements in a specific source.
BuildOverlap() – Generates dominant term overlap information for all sources in a domain.
FindMostTypicalSources() – Most typical sources.
FindBreakingSources() – Most atypical sources.

Attribute Customizations
Negation.
Augment default markers with additional markers for particular use cases.
Sentiment.
No default markers.
Supply custom sentiment markers.
Attribute markers.
Supply custom markers in User Dictionary.
iKnow performs attribute tagging during loading.

iFind
SQL feature for performing text search.
Add iFind index to columns containing text.
Include iFind index syntax in WHERE clauses of SQL queries.
Support for the following searches:
Stemming and de-compounding.
Word and word phrase search.
iKnow entity search.
iKnow semantic search using path, proximity, and dominance information.

Text Categorization
Label (categorize) source texts based on their contents (entities and relations).
Create a classifier by analyzing an existing (training) set of already labelled texts
Apply classifier to new and as yet unlabelled texts.
Wizards available for building and testing classifiers.
System Explorer ? iKnow ? Text Categorization

DeepSee and iKnow
DeepSee cubes can include iKnow indexing results and analyses:
iKnow Dimensions.
Entities (concepts and relations).
Dictionary matching results.
Use as rows, columns, and filters on pivot tables just like data and time dimensions.
Detail Listings.
iKnow summaries.
Content Analysis Plugin to allow users to perform a variety of iKnow analyses on text sources.

Demonstration

iKnow Dimensions
Entity dimension.
Single level.
Members are entities (concepts or relations).
Analyzer displays first 100 in decreasing order by spread.
Filter options contain all entities. Searchable.
Dictionary dimension.
Level 1: one member for each dictionary.
Level 2: one member for each item containing all matches for that item.
Matching dictionaries loaded as termlists.

iKnow Measure
Connects unstructured data to cube.
Purely configuration. Not visible to Analyzer.
Connects DeepSee cube to text sources and dictionaries.
Referenced by iKnow dimensions.

Content Analysis Plugin
Launch from Analyzer or Dashboard.
Select cell and click
iKnow features include:
Content Analysis.
Typical and breaking sources.
Entity Analysis.
Overview: frequency and spread for 10 most common groups.
Cell breakdown: distribution of entities selected on Overview tab.
Entities: frequency and spread for entities similar to entity selected on Cell breakdown.

Demonstration

Configuring iKnow Measure
iKnow Measure:
Source Values: Property or expression.
Aggregate: Count.
Type: iKnow.
iKnow Source: string, stream, file, or domain.
Dictionaries: select from available termlists.

Configuring iknow Dimensions
Entity Dimension.
Dimension Type: iKnow.
iKnow Type: entity.
iKnow Measure: iKnow measure name.
Dictionary Dimension
Dimension Type: iKnow.
iKnow Type: Dictionary.
iKnow measure: iKnow measure name.

iKnow Listing Features
Include iKnow summary.
$$$IKSUMMARY[iKnowMeasure, summaryLength].
Include content analysis plugin.
$$IKLINK[iKnowMeasure].
Allows users to see: summaries, dictionary matches, negation contexts, and dominant entities for selected source(s).

Suggested Reading
Using iKnow.
Advanced DeepSee Modeling Guide ? Using Unstructured Data in Cubes.