Learning Graphs for Knowledge Transfer with Limited Labels
CVPR 2021
Pallabi Ghosh
Nirat Saini
Larry Davis
Abhinav Shivastava
University of Maryland, College Park

Our model learns to update the Knowledge Graph (KG), while training the task specific Graph Convolutional Network (GCN). The example above shows the initial neighbors of action class Pommel Horse based on language based embeddings in the KG. While training GCN, we update the KG, and the neighbors are updated as well. It can be observed that due to Horse, initial top 5 neighbors included Horse Riding and Horse Race, but after updates, neighbors are more meaningful and closer to gymnastics related activities such as Balance Beam and Vault.


Fixed input graphs are a mainstay in approaches that utilize Graph Convolution Networks (GCNs) for knowledge transfer. The standard paradigm is to utilize relationships in the input graph to transfer information using GCNs from training to testing nodes in the graph; for example, the semi-supervised, zero-shot, and few-shot learning setups. We propose a generalized framework for learning and improving the input graph as part of the standard GCN-based learning setup. Moreover, we use additional constraints between similar and dissimilar neighbors for each node in the graph by applying triplet loss on the intermediate layer output. We present results of semi-supervised learning on Citeseer, Cora, and Pubmed benchmarking datasets, and zero/few-shot action recognition on UCF101 and HMDB51 datasets, significantly outperforming current approaches. We also present qualitative results visualizing the graph connections that our approach learns to update.

System Overview

The input graph is passed through a single layer GCN to get the updated graph. This then passes through the normal GCN network to generate the output. Along with the task specific loss, we use triplet loss to train the graph learning layer. The triplet loss is based on positive and negative neighbors determined based on the input graph for zero/few-shot action recognition. For semi-supervised learning we use SoftTriple Loss (Qian et al.) to determine the triplet loss.


Here are some results on semi-supervised learning for Citeseer dataset and zero/few-shot action recognition on UCF101 and HMDB51 datasets. For more results please refer to the paper.


P. Ghosh, N. Saini, L. Davis, A. Shrivastava
Learning Graphs for Knowledge Transfer with Limited Labels
CVPR 2021
[Paper] [Supplementary] [Bibtex]


This work was supported by the Air Force, via Small Business Technology Transfer (STTR) Phase I (FA865019P6014) and Phase II (FA864920C0010), and Defense Advanced Research Projects Agency (DARPA) SAIL-ON program (W911NF2020009). The template code can be found here.