What is Multimedia Data Mining?

Multimedia mining is a subfield of data mining that is used to find interesting information of implicit knowledge from multimedia databases. Mining in multimedia is referred to as automatic annotation or annotation mining. Mining multimedia data requires two or more data types, such as text and video or text video and audio.

Multimedia data mining is an interdisciplinary field that integrates image processing and understanding, computer vision, data mining, and pattern recognition. Multimedia data mining discovers interesting patterns from multimedia databases that store and manage large collections of multimedia objects, including image data, video data, audio data, sequence data and hypertext data containing text, text markups, and linkages. Issues in multimedia data mining include content-based retrieval and similarity search, generalization and multidimensional analysis. Multimedia data cubes contain additional dimensions and measures for multimedia information.

The framework that manages different types of multimedia data stored, delivered, and utilized in different ways is known as a multimedia database management system. There are three classes of multimedia databases: static, dynamic, and dimensional media. The content of the Multimedia Database management system is as follows:

• Media data:The actual data representing an object.
• Media format data: Information such as sampling rate, resolution, encoding scheme etc., about the format of the media data after it goes through the acquisition, processing and encoding phase.
• Media keyword data:Keywords description relating to the generation of data. It is also known as content descriptive data. Example: date, time and place of recording.
• Media feature data: Content dependent data such as the distribution of colours, kinds of texture and different shapes present in data.

Types of Multimedia Applications

Types of multimedia applications based on data management characteristics are:

1. Repository applications: A Large amount of multimedia data and meta-data (Media format date, Media keyword data, Media feature data) that is stored for retrieval purposes, e.g., Repository of satellite images, engineering drawings, radiology scanned pictures.
2. Presentation applications: They involve delivering multimedia data subject to temporal constraints. Optimal viewing or listening requires DBMS to deliver data at a certain rate, offering the quality of service above a certain threshold. Here data is processed as it is delivered. Example: Annotating of video and audio data, real-time editing analysis.
3. Collaborative work using multimedia information involves executing a complex task by merging drawings and changing notifications. Example: Intelligent healthcare network.

Challenges with Multimedia Database

There are still many challenges to multimedia databases, such as:

1. Modelling: Working in this area can improve database versus information retrieval techniques; thus, documents constitute a specialized area and deserve special consideration.
2. Design:The conceptual, logical and physical design of multimedia databases has not yet been addressed fully as performance and tuning issues at each level are far more complex as they consist of a variety of formats like JPEG, GIF, PNG, MPEG, which is not easy to convert from one form to another.
3. Storage:Storage of multimedia database on any standard disk presents the problem of representation, compression, mapping to device hierarchies, archiving and buffering during input-output operation. In DBMS, a BLOB (Binary Large Object) facility allows untyped bitmaps to be stored and retrieved.
4. Performance: Physical limitations dominate an application involving video playback or audio-video synchronization. The use of parallel processing may alleviate some problems, but such techniques are not yet fully developed. Apart from this, a multimedia database consumes a lot of processing time and bandwidth.
5. Queries and retrieval: For multimedia data like images, video, and audio accessing data through query open up many issues like efficient query formulation, query execution and optimization, which need to be worked upon.

Where is Multimedia Database Applied?

Below are the following areas where a multimedia database is applied, such as:

• Documents and record management: Industries and businesses keep detailed records and various documents. For example, insurance claim records.
• Knowledge dissemination:Multimedia database is a very effective tool for knowledge dissemination in terms of providing several resources. For example, electronic books.
• Education and training:Computer-aided learning materials can be designed using multimedia sources which are nowadays very popular sources of learning. Example: Digital libraries.
• Travelling: Marketing, advertising, retailing, entertainment and travel. For example, a virtual tour of cities.
• Real-time control and monitoring: With active database technology, multimedia presentation of information can effectively monitor and control complex tasks. For example, manufacturing operation control.

Categories of Multimedia Data Mining

Multimedia mining refers to analyzing a large amount of multimedia information to extract patterns based on their statistical relationships. Multimedia data mining is classified into two broad categories: static and dynamic media. Static media contains text (digital library, creating SMS and MMS) and images (photos and medical images). Dynamic media contains Audio (music and MP3 sounds) and Video (movies). The below image shows the categories of multimedia data mining.

1. Text Mining

Text is the foremost general medium for the proper exchange of information. Text Mining evaluates a huge amount of usual language text and detects exact patterns to find useful information. Text Mining also referred to as text data mining, is used to find meaningful information from unstructured texts from various sources.

2. Image Mining

Image mining systems can discover meaningful information or image patterns from a huge collection of images. Image mining determines how low-level pixel representation consists of a raw image or image sequence that can be handled to recognize high-level spatial objects and relationships. It includes digital image processing, image understanding, database, AI, etc.

3. Video Mining

Video mining is unsubstantiated to find interesting patterns from many video data; multimedia data is video data such as text, image, metadata, visuals and audio. It is commonly used in security and surveillance, entertainment, medicine, sports and education programs. The processing is indexing, automatic segmentation, content-based retrieval, classification and detecting triggers.

4. Audio Mining

Audio mining plays an important role in multimedia applications, is a technique by which the content of an audio signal can be automatically searched, analyzed and rotten with wavelet transformation. It is generally used in automatic speech recognition, where the analysis efforts to find any speech within the audio. Band energy, frequency centroid, zero-crossing rate, pitch period and bandwidth are often used for audio processing.

Application of Multimedia Mining

There are different kinds of applications of multimedia data mining, some of which are as follows:

• Digital Library: The collection of digital data is stored and maintained in a digital library, which is essential to convert different digital data formats into text, images, video, audio, etc.
• Traffic Video Sequences: To determine important but previously unidentified knowledge from the traffic video sequences, detailed analysis and mining are to be performed based on vehicle identification, traffic flow, and queue temporal relations of the vehicle at an intersection. This provides an economic approach for regular traffic monitoring processes.
• Medical Analysis: Multimedia mining is primarily used in the medical field, particularly for analyzing medical images. Various data mining techniques are used for image classification. Examples, Automatic 3D delineation of highly aggressive brain tumours, Automatic localization and identification of vertebrae in 3D CT scans, MRI Scans, ECG and X-Ray.
• Customer Perception: It contains details about customers' opinions, products or services, customers complaints, customers preferences, and the level of customer satisfaction with products or services, which are collected together. The audio data serve as topic detection, resource assignment and evaluation of the quality of services. Many companies have call centres that receive telephone calls from customers.
• Media Making and Broadcasting: Radio stations and TV channels create broadcasting companies, and multimedia mining can be applied to monitor their content to search for more efficient approaches and improve their quality.
• Surveillance system: It consists of collecting, analyzing, summarizing audio, video or audiovisual information about specific areas like government organizations, multi-national companies, shopping malls, banks, forests, agricultural areas and, highways etc. The main use of this technology in the field of security; hence it can be utilized by military, police and private companies since they provide security services.

Process of Multimedia Data Mining

The below image shows the present architecture, which includes the types of the multimedia mining process. Data Collection is the initial stage of the learning system; Pre-processing is to extract significant features from raw data. It includes data cleaning, transformation, normalization, feature extraction, etc. Learning can be direct if informative types can be recognized at preprocessing stage. The complete process depends extremely on the nature of raw data and the difficulty field. The product of preprocessing is the training set. A learning model must be selected for the specified training set to learn from it and make the multimedia model more constant.

Converting Un-structured data to structured data: Data resides in a fixed field within a record or file is called structured data, and these data are stored in sequential form. Structured data has been easily entered, stored, queried and analyzed. Unstructured data is bitstream, for example, pixel representation for an image, audio, video and character representation for text. These files may have an internal structure, but they are still considered "unstructured" because their data does not fit neatly in a database. For example, images and videos of different objects have some similarities - each represents an interpretation of a building without a clear structure.

Current data mining tools operate on structured data, which resides in a huge volume of the relational database, while data in multimedia databases are semi-structured or unstructured. Hence, the semi-structured or unstructured multimedia data is converted into structured one, and then the current data mining tools are used to extract the knowledge. The sequence or time element is different between unstructured and structured data mining. The architecture of converting unstructured data to structured data and which is used for extracting information from the unstructured database, is shown in the above image. Then data mining tools are applied to the stored structured databases.

Architecture for Multimedia Data Mining

Multimedia mining architecture is given in the below image. The architecture has several components. Important components are Input, Multimedia Content, Spatiotemporal Segmentation, Feature Extraction, Finding similar Patterns, and Evaluation of Results.

1. The input stage comprises a multimedia database used to find the patterns and perform the data mining.
2. Multimedia Content is the data selection stage that requires the user to select the databases, subset of fields, or data for data mining.
3. Spatio-temporal segmentation is nothing but moving objects in image sequences in the videos, and it is useful for object segmentation.
4. Feature extraction is the preprocessing step that involves integrating data from various sources and making choices regarding characterizing or coding certain data fields to serve when inputs to the pattern-finding stage. Such representation of choices is required because certain fields could include data at various levels and are not considered for finding a similar pattern stage. In MDM, the preprocessing stage is significant since the unstructured nature of multimedia records.
5. Finding a similar pattern stage is the heart of the whole data mining process. The hidden patterns and trends in the data are basically uncovered in this stage. Some approaches to finding similar pattern stages contain association, classification, clustering, regression, time-series analysis and visualization.
6. Evaluation of Results is a data mining process used to evaluate the results, and this is important to determine whether the prior stage must be revisited or not. This stage consists of reporting and using the extracted knowledge to produce new actions, products, services, or marketing strategies.

Models for Multimedia Mining

The models which are used to perform multimedia data are very important in mining. Commonly four different multimedia mining models have been used. These are classification, association rule, clustering and statistical modelling.

1. Classification: Classification is a technique for multimedia data analysis that can learn from every property of a specified set of multimedia. It is divided into a predefined class label to achieve the purpose of classification. Classification is the process of constructing data into categories for its better effective and efficient use; it creates a function that well-planned data item into one of many predefined classes by inputting a training data set and building a model of the class attribute based on the rest of the attributes. Decision tree classification has a perceptive nature that the users conceptual model without loss of exactness. Hidden Markov Model is used to classify multimedia data such as images and videos as indoor-outdoor games.
2. Association Rule: Association Rule is one of the most important data mining techniques that help find relations between data items in huge databases. There are two types of associations in multimedia mining: image content and non-image content features. Mining the frequently occurring patterns between different images becomes mining the repeated patterns in a set of transactions. Multi-relational association rule mining displays multiple reports for the same image. In image classification also, multiple-level association rule techniques are used.
3. Clustering: Cluster analysis divides the data objects into multiple groups or clusters. Cluster analysis combines all objects based on their groups. In multimedia mining, the clustering technique can be applied to group similar images, objects, sounds, videos and texts. Clustering algorithms can be divided into several methods: hierarchical methods, density-based methods, grid-based methods, model-based methods, k-means algorithms, and graph-based models.
4. Statistical Modeling: Statistical mining models regulate the statistical validity of test parameters and have been used to test hypotheses, undertake correlation studies, and transform and make data for further analysis. This is used to establish links between words and partitioned image regions to form a simple co-occurrence model.

Issues in Multimedia Mining

Major Issues in multimedia data mining contains content-based retrieval, similarity search, dimensional analysis, classification, prediction analysis and mining associations in multimedia data.

1. Content-based retrieval and Similarity search

Content-based retrieval in multimedia is a stimulating problem since multimedia data is required for detailed analysis from pixel values. We considered two main families of multimedia retrieval systems, i.e. similarity search in multimedia data.

• Description-based retrieval system creates indices and object retrieval based on image descriptions, such as keywords, captions, size, and creation time.
• Content-based retrieval system supports image content retrieval, for example, colour histogram, texture, shape, objects, and wavelet transform.
• Use of content-based retrieval system: Visual features index images and promote object retrieval based on feature similarity; it is very desirable in various applications. These applications include diagnosis, weather prediction, TV production and internet search engines for pictures and e-commerce.

2. Multidimensional Analysis

To perform multidimensional analysis of large multimedia databases, multimedia data cubes may be designed and constructed similarly to traditional data cubes from relational data. A multimedia data cube has several dimensions. For example, the size of the image or video in bytes; the width and height of the frames, creating two dimensions, the date on which image or video was created or last modified, the format type of the image or video, frame sequence duration in seconds, Internet domain of pages referencing the image or video, the keywords like a colour dimension and edge orientation dimension. A multimedia data cube can have additional dimensions and measures for multimedia data, such as colour, texture, and shape.

The Multimedia data mining system prototype is MultiMediaMiner, the extension of the DBMiner system that handles multimedia data. The Image Excavator component of MultiMediaMiner uses image contextual information, like HTML tags on Web pages, to derive keywords. By navigating online directory structures, like Yahoo! directory, it is possible to build hierarchies of keywords mapped on the directories in which the image was found.

3. Classification and Prediction Analysis

Classification and predictive analysis has been used for mining multimedia data, particularly in scientific analysis like astronomy, seismology, and geoscientific analysis. Decision tree classification is an important method for reported image data mining applications. For example, consider the sky images, which astronomers have carefully classified as the training set. It can create models for recognizing galaxies, stars and further stellar objects based on properties like magnitudes, areas, intensity, image moments and orientation.

Image data mining classification and clustering are carefully connected to image analysis and scientific data mining. The image data are frequently in large volumes and need substantial processing power, such as parallel and distributed processing. Hence, many image analysis techniques and scientific data analysis methods could be applied to image data mining.

4. Mining Associations in Multimedia

Data Association rules involving multimedia objects have been mined in image and video databases. Three categories can be observed:

• Associations between image content and non-image content features
• Associations among image contents that are not related to spatial relationships
• Associations among image contents related to spatial relationships

First, an image contains multiple objects, each with various features such as colour, shape, texture, keyword, and spatial locations, so that many possible associations can be made. Second, a picture containing multiple repeated objects is essential in image analysis. The recurrence of similar objects should not be ignored in association analysis. Third, to find the associations between the spatial relationships and multimedia images can be used to discover object associations and correlations. With the associations between multimedia objects, we can treat every image as a transaction and find commonly occurring patterns among different images.