User Tools

Site Tools


projects:year4:15.2

This is an old revision of the document!


15.2 - Predicting Future Relations: Incremental and Robust Link Prediction

Project - Summary

Objectives:

The goal of this project is to develop a multi-modal retrieval system that can work with both image and textual information for question answering. The project creates a unified, graph-based representation model in both text and image domains and is capable of using visual features to search in a textual database or use textual queries to retrieve relevant images from a database.

To accomplish this, the project aims to develop a computational toolbox for automatically generating graph representation of images and text information, and an API set for interacting with a database consisting of graph-based information.

To achieve this, following action items were considered:

  • Build a multi-scale feature extraction from images and capture the spatial relationship of image features in the form of a graph.
  • Incorporate the dependency graphs obtained from textual information annotated with syntactic information and store the resulting data in the form of a graph database.
  • Develop a unifying platform for correlating images and textual information represented in terms of multi-scale image features and dependency text representation.
  • Develop retrieval tools for a question answering system that can support a question answering system between both modalities.
  • Build a proof of concept API for the system.

Project - Team

Team Member Role Email Phone Number Academic Site/IAB
Vijay Raghavan PI Not available Not available UL Lafayette
Raju Gottumukkala PI Not available Not available UL Lafayette
Ryan Benton PI Not available Not available University of South Alabama
Murali K. Pusala Graduate Student Not available Not available UL Lafayette
Amirhossein Tavanaei Graduate Student Not available Not available UL Lafayette
Narendra Sanikommu Graduate Student Not available Not available UL Lafayette
Jaya Krishna Graduate Student Not available Not available UL Lafayette

Project - Impact and Uses/Benefits

Both of the methods that we proposed are implemented using C++ for high efficiency. Although the systems are tested using standard image and question-answer datasets, the evaluation can be extended to other datasets, potentially the ones from the IAB members.

We note that, the graph matching technique that we proposed in this project can also be adapted to other data types such as sound or video. Additionally, the graph embedding into HST can also be applied on clustering tasks which might appear fields such as in online marketing or social networks.

Project - Deep Dive

Project - Documents

[n/a: No match]
projects/year4/15.2.1566404019.txt.gz · Last modified: 2019/08/21 11:13 by sally.johnson