Learning Context-based Embeddings for Knowledge Graph Completion

doi:10.2478/jdis-2022-0009

Abstract

Abstract:

Purpose: Due to the incompleteness nature of knowledge graphs (KGs), the task of predicting missing links between entities becomes important. Many previous approaches are static, this posed a notable problem that all meanings of a polysemous entity share one embedding vector. This study aims to propose a polysemous embedding approach, named KG embedding under relational contexts (ContE for short), for missing link prediction.

Design/methodology/approach: ContE models and infers different relationship patterns by considering the context of the relationship, which is implicit in the local neighborhood of the relationship. The forward and backward impacts of the relationship in ContE are mapped to two different embedding vectors, which represent the contextual information of the relationship. Then, according to the position of the entity, the entity's polysemous representation is obtained by adding its static embedding vector to the corresponding context vector of the relationship.

Findings: ContE is a fully expressive, that is, given any ground truth over the triples, there are embedding assignments to entities and relations that can precisely separate the true triples from false ones. ContE is capable of modeling four connectivity patterns such as symmetry, antisymmetry, inversion and composition.

Research limitations: ContE needs to do a grid search to find best parameters to get best performance in practice, which is a time-consuming task. Sometimes, it requires longer entity vectors to get better performance than some other models.

Practical implications: ContE is a bilinear model, which is a quite simple model that could be applied to large-scale KGs. By considering contexts of relations, ContE can distinguish the exact meaning of an entity in different triples so that when performing compositional reasoning, it is capable to infer the connectivity patterns of relations and achieves good performance on link prediction tasks.

Originality/value: ContE considers the contexts of entities in terms of their positions in triples and the relationships they link to. It decomposes a relation vector into two vectors, namely, forward impact vector and backward impact vector in order to capture the relational contexts. ContE has the same low computational complexity as TransE. Therefore, it provides a new approach for contextualized knowledge graph embedding.

Key words: Full expressiveness, Relational contexts, Knowledge graph embedding, Relation patterns, Link prediction

Fei Pu , Zhongwei Zhang , Yan Feng , and Bailin Yang .(2022). Learning Context-based Embeddings for Knowledge Graph Completion. Journal of Data and Information Science, 7(2), 84-106. DOI: 10.2478/jdis-2022-0009

0
/ / Recommend

Add to citation manager EndNote|Ris|BibTeX

URL: http://manu47.magtech.com.cn/Jwk3_jdis/EN/10.2478/jdis-2022-0009

http://manu47.magtech.com.cn/Jwk3_jdis/EN/Y2022/V7/I2/84

This work is licensed under the Creative Commons Attribution 4.0 International License.

Figures/Tables 11

Table 1.

Table 2.

Table 3.

Table 4.

Table 5.

Table 6.

Table 7.

Table 8.

Table 9.

Table 10.

Table 11.

References 48

[1]	Ali M., Berrendorf M., Hoyt C.T., Vermue L., Sharifzadeh S., Tresp V., & Lehmann J. (2021). PyKEEN 1.0- A python library for training and evaluating knowledge graph embeddings. Journal of Machine Learning Research, 22, 1-6.
[2]	Antoine B., Nicolas U., Alberto G.D.,Weston, J & Yakhnenko, O. (2013). Translating embeddings for modeling multirelational data. Proceedings of Advances in Neural Information Processing Systems (NIPS), 2787-2795.
[3]	Chang K.W., Yih W.T., Yang B.S., & Meek C. (2014). Typed tensor decomposition of knowledge bases for relation extraction. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 1568-1579.
[4]	Das R, Dhuliawala S., Zaheer M., Vilnis L., Durugkar I., Krishnamurthy A., Smola A., & McCallum A. (2018). Go for a walk and arrive at the answer:Reasoning over paths in knowledge bases using reinforcement learning. Proceedings of 6th International Conference on Learning Representations (ICLR).
[5]	Dettmers T., Minervini P., Stenetorp P., & Riedel S. (2018). Convolutional 2D knowledge graph embeddings. Proceedings of 32nd AAAI Conference on Artificial Intelligence, 1811-1818.
[6]	Devlin J., Chang M.W., Lee K., & Toutanova K. (2019). Bert:Pre-training of deep bidirectional transformers for language understanding. Proceedings of 57th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 4171-4186.
[7]	Ding B.Y., Wang Q., Wang B., & Guo L. (2018). Improving knowledge graph embedding using simple constraints, Proceedings of 56th Annual Meeting of the Association for Computational Linguistics (ACL), 110-121.
[8]	Dong X., Gabrilovich E., Geremy H., Horn W., Lao N., Murphy K., Strohmann T., Sun S.H., & Zhang W. (2014). Knowledge vault: A web-scale approach to probabilistic knowledge fusion. Proceedings of 20th ACM SIGKDD conference on Knowledge Discovery and Data Mining (KDD), 601-610.
[9]	Galarraga L.A., Teflioudi C., Hose K., & Suchanek F.M. (2015). Fast rule mining in ontological knowledge bases with AMIE+. VLDB Journal, 24(6), 707-730. doi: 10.1007/s00778-015-0394-1
[10]	Gjergji K., Fabian M.S., Georgiana I., Maya R. & Gerhard W. (2018). Naga: Searching and ranking knowledge. Proceedings of 24th IEEE International Conference on Data Engineering (ICDE), 953-962.
[11]	Guillaume B., Sameer S., & Theo T. (2015). On approximate reasoning capabilities of low-rank vector spaces. Proceedings of AAAI Spring Syposium on Knowledge Representation and Reasoning (KRR): Integrating Symbolic and Neural Approaches, 6-9.
[12]	Guo S., Wang Q., Wang L.H., Wang B., & Guo L. (2018). Knowledge graph embedding with iterative guidance from soft rules. Proceedings of 32th AAAI Conference on Artificial Intelligence, 4816-4823.
[13]	Guo L.B., Sun Z.Q., & Hu W. (2019). Learning to exploit long-term relational dependencies in knowledge graphs. Proceedings of 36th International Conference on Machine Learning (ICML), 2505-2514.
[14]	Hao Y.C., Zhang Y.Z., Liu K., He S.Z, Liu Z.Y., Wu H., & Zhao J. (2017). An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge. Proceedings of 55th Annual Meeting of the Association for Computational Linguistics (ACL), 221-231.
[15]	Ji G.L., He S.Z., & Xu L. (2015). Knowledge graph embedding via dynamic mapping matrix, Proceedings of 53rd Annual Meeting of the Association for Computational Linguistics and 7th International Joint Conference on Natural Language Processing, 687-696.
[16]	Ji G.L., Liu K., He S.Z., & Zhao J. (2017). Distant supervision for relation extraction with sentence-level attention and entity descriptions. Proceedings of 31st AAAI Conference on Artificial Intelligence, 3060-3066.
[17]	Ji G.L., Liu K., He S.Z., & Zhao J. (2016). Knowledge graph completion with adaptive sparse transfer matrix. Proceedings of 30th Conference on Artificial Intelligence, 985-991.
[18]	Kok S., & Domingos P. (2007). Statistical predicate invention. Proceedings of the 24th International Conference on Machine Learning (ICML), 433-440.
[19]	Li H., Y. Liu, Mamoulis N., & Rosenblum D.S. (2019). Translation-based sequential recommendation for complex users on sparse data. IEEE Transactions on Knowledge and Data Engineering, 32(8), 1639-1651. doi: 10.1109/TKDE.2019.2906180
[20]	Lin X.V., Socher R., & Xiong C.M. (2018). Multi-hop knowledge graph reasoning with reward shaping, Proceedings of 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3243-3253.
[21]	Lin Y.K., Liu Z.Y., Luan H.B., Sun M.S., Rao S.W., & Liu S. (2015). Modeling relation paths for representation learning of knowledge bases. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 705-714.
[22]	Liu H.X., Wu Y.X., & Yang Y.M. (2017). Analogical inference for multi-relational embeddings. Proceedings of 34th International Conference on Machine Learning (ICML), 2168-2178.
[23]	Liu W.J., Zhou P., Zhao Z., Wang Z.R, Ju Q., Deng H.T., & Wang P. (2020). K-BERT:Enabling language representation with knowledge graph, Proceedings of 34th AAAI Conference on Artificial Intelligence, 1112-1119.
[24]	Liu Y., Li H., & Alberto G.D. (2019). MMKG: Multi-modal knowledge graphs, European Semantic Web Conference, LNCS, 11503, 459-474.
[25]	Mikolov T., Sutskever I., Chen K., Corrado G.S., & Dean J. (2013). Distributed representations of words and phrases and their compositionality. Proceedings of Advances in Neural Information Processing Systems (NIPS), 3111-3119.
[26]	Minervini P., Costabello L., Munoz E., Novacek V., & Vandenbussche P.Y. (2017). Regularizing knowledge graph embeddings via equivalence and inversion axioms. Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML-KDD), 668-683.
[27]	Nathani D., Chauhan J., Sharma C., & Kaul M. (2019). Learning attention-based embeddings for relation prediction in knowledge graphs. Proceedings of 57th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 4710-4723.
[28]	Nguyen D.Q., Nguyen T.D., Nguyen D.Q., & Phung D. (2018). A novel embedding model for knowledge base completion based on convolutional neural network. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT), 327-333.
[29]	Nguyen D.Q., Vu T., Nguyen T.D., Nguyen D.Q., & Phung D. (2019). A capsule network-based embedding model for knowledge graph completion and search personalization. Proceedings of 57th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2180-2189.
[30]	Peng Y.H., & Zhang J. (2020). LineaRE: Simple but Powerful Knowledge Graph Embedding for Link Prediction. Proceedings of 20th IEEE International Conference on Data Mining (ICDM), 422-431.
[31]	Pu F., Yang B.L., Ying J.C, You L.Z., & Xu C.O. (2020). A Contextualized Entity Representation for Knowledge Graph Completion. Proceedings of 13th International Conference on Knowledge Science, Engineering and Management (KSEM), 77-85.
[32]	Rocktaschel T., & Riedel S. (2017). End-to-end differentiable proving. Proceedings of Advances in Neural Information Processing Systems (NIPS), 3791-3803.
[33]	Schlichtkrull M.S., Thomas N.K., & Bloem P. (2018). Modeling relational data with graph convolutional networks. Gangemi A. et al. (eds.) ESWC 2018, LNCS, vol. 10843, 593-607, Springer, Heidelberg.
[34]	Seyed M.K., & David P. (2018). Simple embedding for link prediction in knowledge graphs. Proceedings of 32nd Conference on Advances in Neural Information Processing Systems (NIPS), 4289-4300.
[35]	Socher R., Chen D.Q., Manning C.D., & Andrew Ng. (2013). Reasoning with neural tensor networks for knowledge base completion. Proceedings of Advances in Neural Information Processing Systems, 926-934.
[36]	Sun Y., Wang S., Li Y., Feng S.K., Chen X.Y., Zhang H., Tian X, Zhu D.X., Tian H., & Wu H., (2019). ERNIE: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223.
[37]	Sun Z.Q., Deng Z.H., Nie J.Y., & Tang J. (2019). Rotate:knowledge graph embedding by relational rotation in complex space. Proceedings of 7th International Conference on Learning Representations (ICLR).
[38]	Sun Z.Q., Vashishth S., Sanyal S., Talukdar P., & Yang Y.M. (2020). A re-evaluation of knowledge graph completion methods. Proceedings of 58th Annual Meeting of the Association for Computational Linguistics (ACL), 5516-5522.
[39]	Trouillon T., Welbl J., Riedel S., Gaussier E., & Bouchard G. (2016). Complex embeddings for simple link prediction. Proceedings of International Conference on Machine Learning (ICML), 2071-2080.
[40]	Vaswani A., Shazeer N., Parmar N, Uszkoreit J, Jones Llion J., Gomez A.N., Kaiser L., & Polosukhin I. (2017). Attention is all you need. Proceedings of Advances in Neural Information Processing Systems (NIPS), 5998-6008.
[41]	Wang Z., Zhang J.W., Feng J.L., & Chen Z. (2014). Knowledge graph embedding by translating on hyperplanes. Proceedings of 28th AAAI Conference on Artificial Intelligence, 1112-1119.
[42]	Xiong C.Y., Russell P., & Jamie C. (2017). Explicit semantic ranking for academic search via knowledge graph embedding. Proceedings of 26th International Conference on World Wide Web (WWW), 1271-1279.
[43]	Yang B.S., & Tom M. (2017). Leveraging knowledge bases in LSTMs for improving machine reading. Proceedings of 55th Annual Meeting of the Association for Computational Linguistics (ACL), 1436-1446.
[44]	Yang B.S., Yih W.T., He X.D., Gao J.F., & Deng L. (2015). Embedding entities and relations for learning and inference in knowledge bases. Proceedings of the International Conference on Learning Representations (ICLR), 1-12.
[45]	Yang F, Yang Z.L., & Cohen W.W. (2017). Differentiable learning of logical rules for knowledge base reasoning. Proceedings of Advances in Neural Information Processing Systems (NIPS), 2319-2328.
[46]	Zhang N.Y., Deng S.M., Sun Z.L., Wang G.Y., Chen X., Zhang W., & Chen H.J. (2019). Long-tail relation extraction via knowledge graph embeddings and graph convolution networks, Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (ACL), 3016-3025.
[47]	Zhang Z.Q., Cai J.Y., Zhang Y.D., & Wang J. (2020). Learning hierarchy-aware knowledge graph embeddings for link prediction. Proceedings of 34th AAAI Conference on Artificial Intelligence, 3065-3072.
[48]	Zhong H.P., Zhang J.W., Wang Z., Wan H., & Chen Z.(2015). Aligning knowledge and text embeddings by entity descriptions. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 267-272.

Model	Scoring function ψ(e₁,r,e₂)	Parameters
TransE	$\|\| v_{e_1} + v_r - v_{e_2}\|\|_p$	$v_{e_1}, v_r, v_{e_2} \in R^d$
ComplEx	$Re (<v_{e_1}, v_r, \bar{v}_{e_2}>)$	$v_{e_1}, v_r, v_{e_2} \in C^d$
SimplE	$\frac{1}{2}(<h_{e_1}, v_r, t_{e_2}> + <h_{e_2}, v_{r-1}, t_{e_1}>)$	$h_{e_1}, v_r, t_{e_2}, h_{e_2}, v_{r-1}, t_{e_1} \in R^d$
ConvE	$g(vec(g(concat(\hat{v}_{e_1}, v_r)* \omega))W). v_{e_2}$	$v_{e_1}, v_r, v_{e_2} \in R^d$
ConvKB	$concat(g([v_{e_1}, v_r, v_{e_2}]* \omega)). w$	$v_{e_1}, v_r, v_{e_2} \in C^d$
Rotate	$\|\| v_{e_1} 。 v_r - v_{e_2}\|\|$	$v_{e_1}, v_{e_2}, v_r \in C^d$
HAKE	$-\|\| v_{e1_m} 。 v_{r_m} - v_{e2_m}\|\|_2$	$v_{e1_m}, v_{e2_m} \in R^d, v_{r_m} \in R^d_+$
	$- \lambda\|\| sin(v_{e1_p} + v_{r_p} - v_{e2_p})\|\|_1$	$v_{e1_p}, r_p, v_{e2_p} \in [0,2 \pi )^d$
ContE	$<f_r + v_{e_1}, b_r+ v_{e_2}>$	$v_{e_1}, v_{e_2}, f_r, b_r \in R^d$

Model	Symmetry	Antisymmetry	Inversion	Composition
TransE (Antoine et al., 2013)	×	√	√	√
DistMult (Yang et al., 2015)	√	×	×	×
ComplEx (Trouillon et al., 2016)	√	√	√	×
SimplE (Seyed & David, 2018)	√	√	√	×
ConvE (Dettmers et al., 2018)	-	-	-	-
ConvKB (Nguyen et al., 2018)	-	-	-	-
RotatE (Sun et al., 2019)	√	√	√	√
HAKE (Zhang et al., 2020)	√	√	√	√
KGCR (Pu et al., 2020)	√	√	√	×
LineaRE (Peng & Zhang, 2020)	√	√	√	√
ContE	√	√	√	√

Models	#Parameters	Time Complexity
TransE	O(n_ed + n_rd)	O(d)
NTN	O(n_ed + n_rd²k)	O(d³)
ComplEx	O(n_ed + n_rd)	O(d)
TransR	O(n_ed + n_rdk)	O(dk)
SimplE	O(n_ed + n_rd)	O(d)
ContE	O(n_ed + 2n_rd)	O(d)

Dataset	\|E\|	\|R\|	\|training\|	\|validation\|	\|test\|
FB15K-237	14,541	237	272,115	17,535	20,466
UMLS	135	46	5,216	652	661
Nations	14	55	1,592	199	201
Countries_S1	271	2	1,111	24	24
Countries_S2	271	2	1,063	24	24
Countries_S3	271	2	985	24	24

	UMLS
	MRR	Hits@N
	MRR			1	3	10
TransE (Antoine et al., 2013)	0.7966	0.6452	0.9418	0.9841
DistMult (Yang et al., 2015)	0.868	0.821	-	0.967
ComplEx (Trouillon et al., 2016)	0.8753	0.7942	0.9531	0.9713
ConvE (Dettmers et al., 2018)	0.957	0.932	-	0.994
NeuralLP (Yang, Zhang, & Cohen, 2017)	0.778	0.643	-	0.962
NTP-λ (Rocktaschel et al., 2017)	0.912	0.843	-	1.0
MINERVA (Das et al., 2018)	0.825	0.728	-	0.968
KGRRS+ComplEx (Lin et al., 2018)	0.929	0.887	-	0.985
KGRRS+ConvE (Lin et al., 2018)	0.940	0.902	-	0.992
Rotate (Sun et al., 2019)	0.9274	0.8744	0.9788	0.9947
HAKE (Zhang et al., 2020)	0.8928	0.8366	0.9387	0.9849
LineaRE (Peng & Zhang, 2020)	0.9508	0.9145	0.9856	0.9992
ContE	0.9677	0.9501	0.9811	1.0