User Tools

Site Tools


15.1 - Information Retrieval on Multiple Data Sources Using Graph-Based Methods

Project - Summary


The goal of this project is to develop a multi-modal retrieval system that can work with both image and textual information for question answering. The project creates a unified, graph-based representation model in both text and image domains and is capable of using visual features to search in a textual database or use textual queries to retrieve relevant images from a database.

To accomplish this, the project aims to develop a computational toolbox for automatically generating graph representation of images and text information, and an API set for interacting with a database consisting of graph-based information.

To achieve this, following action items were considered:

  • Build a multi-scale feature extraction from images and capture the spatial relationship of image features in the form of a graph.
  • Incorporate the dependency graphs obtained from textual information annotated with syntactic information and store the resulting data in the form of a graph database.
  • Develop a unifying platform for correlating images and textual information represented in terms of multi-scale image features and dependency text representation.
  • Develop retrieval tools for a question answering system that can support a question answering system between both modalities.
  • Build a proof of concept API for the system.

Project - Team

Team Member Role Email Phone Number Academic Site/IAB
Ali Shokoufandeh PI Not available Not available Drexel University
Yusuf Osmanlioglu PhD Student Not available Not available Drexel University

Project - Impact and Uses/Benefits

Both of the methods that we proposed are implemented using C++ for high efficiency. Although the systems are tested using standard image and question-answer datasets, the evaluation can be extended to other datasets, potentially the ones from the IAB members.

We note that, the graph matching technique that we proposed in this project can also be adapted to other data types such as sound or video. Additionally, the graph embedding into HST can also be applied on clustering tasks which might appear fields such as in online marketing or social networks.

Project - Deep Dive

Project - Documents

FilenameFilesizeLast modified
15.1_year_4_presentation.pptx2.0 MiB2019/08/22 11:50
15.1_year_4_executive_summary.pdf162.5 KiB2019/08/22 11:50
15.1_year_4_quad_chart.pptx1.3 MiB2019/08/22 11:50
15.1_year_4_ip_letter_combined.pdf371.0 KiB2019/08/22 11:50
15.1_year_4_final_report.pdf899.0 KiB2019/08/22 10:33
projects/year4/15.1.txt · Last modified: 2019/08/22 10:33 by sally.johnson