Nested Intelligence:
Intelligence is a fascinating concept that appears to manifest in a complex, multiscale, and interconnected architecture. It transcends the boundaries of human cognition, extending throughout the natural world in various forms. From the intricate workings of individual cells to the collective intelligence of societies, intelligence is distributed and nested across different levels of organisation.
So, can organisations embrace this multiscale architecture? Perhaps one way that organisations can adopt this model right now is by embedding an LLM Agent into each Data Product within their Data Mesh. This collaboration creates what we call "Data Agents." The LLM Agent facilitates connections to underlying systems and enables communication with Users and other Data Agents.
Each Data Agent acts as an "expert," specifically trained to address enquiries related to the datasets it publishes. The main goal of each Data Agent is to reduce the level of entropy within its datasets and to make each dataset as valuable as possible. Together the whole system operates as a sort of ‘Distributed Mixture of Experts’.
Similar to nature's nested networks, Data Agents can expose network-shaped data and distribute their datasets as fragments of a larger connected graph. This approach allows agents to form clusters within departments, gradually assembling themselves into a representation of the entire organisation. This forms a layered architecture with scale-relevant intelligence embedded into each layer.
With each layer of embedding, intelligence accumulates. Each layer represents an expansion of the "cognitive light cone", widening the spatiotemporal boundaries within which events can be measured, modelled, and influenced. In the final layer, the organisation should gain a vastly enhanced capacity to achieve its objectives and goals within its environment.
Although this architecture is still a work in progress and requires further validation, many of us are actively exploring and advancing this pattern. I invite you to contribute your thoughts, provide feedback, and share this post with your network, fostering an open dialogue on this intriguing topic.
⭕ Mind Everywhere (Michael Levin): https://www.frontiersin.org/journals/systems-neuroscience/articles/10.3389/fnsys.2022.768201/full
⭕ Semantic Router: https://www.knowledge-graph-guys.com/blog/the-semantic-router
⭕ Working Memory Graph: https://www.knowledge-graph-guys.com/blog/the-working-memory-graph
⭕ Data Mesh (Zhamak Dehghani): https://martinfowler.com/articles/data-mesh-principles.html
⭕ Embrace Complexity: https://experiencestack.co/embrace-complexity-part-1-39483f10a47f
