2026-07-22T23:31:39Zhttps://iris.cnr.it/oai/request

oai:iris.cnr.it:20.500.14243/5901432026-07-09T00:17:38Zcom_20.500.14243_46com_20.500.14243_21col_20.500.14243_47ou_ou239

Linguistic Profiling of Transformer Embedding Geometry Lucia Domenichelli Dominique Brunato Felice Dell'Orletta Domenichelli, Lucia Brunato, Dominique Dell'Orletta, Felice embedding space, neural language models, linguistic profiling, isotropy, non linear intrinsic dimensionality Transformer language models embed tokens in high-dimensional spaces, but whether geometry reflects linguistic structure remains unclear. We analyse token representations in BERT and GPT-2, selected as canonical encoder-only and decoder-only Transformer architectures, through a linguistically-grounded geometric lens. We partition tokens from the Universal Dependencies English Web treebank by surface and syntactic features (position, length, POS, head distance and arity) and examine how their representational geometry evolves across layers. We employ complementary diagnostic metrics, including isotropy, linear and nonlinear intrinsic dimensionality, to capture distinct aspects of embedding structure. Our findings reveal that BERT maintains more isotropic and higher-dimensional subspaces, whereas GPT-2 exhibits stronger anisotropy driven by a compact cluster of sentence-initial tokens. Across models, open-class words, longer tokens, and predicates with several dependents occupy more isotropic, higher-dimensional manifolds than short function words and pre-head modifiers, indicating that semantic richness and syntactic centrality play a key role in structuring embedding space. Our analysis provides a reusable framework for profiling how linguistic abstractions organize the geometry of Transformer embeddings. 2026 info:eu-repo/semantics/conferenceObject https://hdl.handle.net/20.500.14243/590143 10.18653/v1/2026.conll-main.0 eng ispartofbook:Proceedings of the 30th Conference on Computational Natural Language Learning 30th Conference on Computational Natural Language Learning (CoNLL 2026) firstpage:145 lastpage:164 numberofpages:20 info:eu-repo/semantics/openAccess Association for Computational Linguistics license:Creative commons license uri:http://creativecommons.org/licenses/by-nc-nd/4.0/