Continuous and Discrete
We can think of information existing in a continuous stream or in discrete chunks. Large Language Models (LLMs) fall under the category of continuous knowledge representation, while Knowledge Graphs belong to the discrete realm. Each approach has its merits, and understanding the implications of their differences is essential.
LLM embeddings are dense, continuous real-valued vectors existing in a high-dimensional space. Think of them as coordinates on a map: just as longitude and latitude can pinpoint a location on a two-dimensional map, embeddings guide us to rough positions in a multi-dimensional 'semantic space' made up of the connections between the words on the internet. Since the embedding vectors are continuous, they allow for an infinite range of values within a given interval, making the embeddings' coordinates 'fuzzy'.
An LLM embedding for ‘Jennifer Aniston’ will be a several-thousand-dimensional continuous vector that leads to a location in a several-billion-parameter ‘word-space’. If I add the 'TV series' embedding to this vector then I will be pulled towards the position of the 'Friends' vector. Magic! But this magic comes with a price: you can never quite trust the answers. Hallucination and creativity are two sides of the same coin.
On the other hand, Knowledge Graphs embrace a discrete representation approach, where each entity is associated with a unique URL. For example, the Wikidata URL for Jennifer Aniston is https://www.wikidata.org/wiki/Q32522 . This represents a discrete location in 'DNS + IP space'. Humans have carefully structured data that is reliable, editable, and explainable. However, the discrete nature of Knowledge Graphs also comes with its own price. There is no magical internal animation here; just static facts.
Researchers have long theorized that the human brain employs both continuous and discrete information processing. Recent research indicates that perhaps continuous input could activate discrete 'grandmother neurons'. For example, a person could see a photograph which activates a discrete neuron that is specifically associated with Jennifer Aniston.
Continuous and discrete knowledge representation approaches have distinct merits and limitations. So how could we apply this to our AI architectures? Well, recent developments allow us to map continuous LLM embedding vectors to nodes and subgraphs. The convergence of these representations presents exciting opportunities for advancing artificial intelligence!
I am on holiday right now and enjoying a digital detox, but I will answer comments upon my return in two weeks. (As a side note, the family holiday was planned by ChatGPT – let’s hope that it has not hallucinated! 😉)
⭕ Working Memory Graph: https://www.knowledge-graph-guys.com/blog/the-working-memory-graph
⭕The Jennifer Aniston Neuron: https://en.wikipedia.org/wiki/Grandmother_cell
