What is symbolic artificial intelligence?

Neural networks are almost as old as symbolic AI, but they were largely dismissed because they were inefficient and required compute resources that weren’t available at the time. In the past decade, thanks to the large availability of data and processing power, deep learning has gained popularity and has pushed past symbolic AI systems. OOP languages allow you to define classes, specify their properties, and organize them in hierarchies. You can create instances of these classes (called objects) and manipulate their properties. Class instances can also perform actions, also known as functions, methods, or procedures. Each method executes a series of rule-based instructions that might read and change the properties of the current and other objects.

On the one hand, the introduction of additional linguistic complexity makes it possible to say things that cannot be said in more restricted languages. On the other hand, the introduction of additional linguistic flexibility has adverse effects on computability. As we proceed though the material, our attention will range from the completely computable case of Propositional Logic to a variant that is not at all computable. One of the first individuals to give voice to this idea was Leibnitz. He conceived of “a universal algebra by which all knowledge, including moral and metaphysical truths, can some day be brought within a single deductive system”.

His approach highlights how symbolic reasoning can enhance the ability of generative systems to create accurate depictions, something modern LLMs still need to work on. As seen recently with Stability AI’s release of Stable Diffusion 3 Medium, the latest AI image-synthesis model has been heavily criticized online for generating anatomically incorrect images. Despite advancements in AI, these visual abominations underscore the ongoing challenges in accurately depicting human forms, a problem Cohen’s symbolic approach addressed over half a century ago. One is based on possible worlds; the other is based on symbolic manipulation of expressions. Yet, for “well-behaved” logics, it turns out that logical entailment and provability are identical – a set of premises logically entails a conclusion if and only if the conclusion is provable from the premises.

With this paradigm shift, many variants of the neural networks from the ’80s and ’90s have been rediscovered or newly introduced. Benefiting from the substantial increase in the parallel processing power of modern GPUs, and the ever-increasing amount of available data, deep learning has been steadily paving its way to completely dominate the (perceptual) ML. Discover how integrating symbolic reasoning into AI can enhance its capabilities.

The current state of symbolic AI

Particularly, we will show how to make neural networks learn directly with relational logic representations (beyond graphs and GNNs), ultimately benefiting both the symbolic and deep learning approaches to ML and AI. Most of the existing literature on symbolic reasoning has been developed using an implicitly or explicitly translational perspective. Although we do not believe that the current evidence is enough to completely dislodge this perspective, it does show that sensorimotor processing influences the capacity for symbolic reasoning in a number of interesting and surprising ways.

Here, formal structure is mirrored in the visual grouping structure created both by the spacing (b and c are multiplied, then added to a) and by the physical demarcation of the horizontal line. Instead of applying abstract mathematical rules to process such expressions, Landy and Goldstone (2007a,b see also Kirshner, 1989) propose that reasoners leverage visual grouping strategies to directly segment such equations into multi-symbol visual chunks. To test this hypothesis, they investigated the way manipulations of visual groups affect participants’ application of operator precedence rules.

Deep reinforcement learning (DRL) brings the power of deep neural networks to bear on the generic task of trial-and-error learning, and its effectiveness has been convincingly demonstrated on tasks such as Atari video games and the game of Go. However, contemporary DRL systems inherit a number of shortcomings from the current generation of deep learning techniques. For example, they require very large datasets to work effectively, entailing that they are slow to learn even when such datasets are available.

Most recently, an extension to arbitrary (irregular) graphs then became extremely popular as Graph Neural Networks (GNNs). You can foun additiona information about ai customer service and artificial intelligence and NLP. Driven heavily by the empirical success, DL then largely moved away from the original biological brain-inspired models of perceptual intelligence to “whatever works in practice” kind of engineering approach. In essence, the concept evolved into a very generic methodology of using gradient descent to optimize parameters of almost arbitrary nested functions, for which many like to rebrand the field yet again as differentiable programming. This view then made even more space for all sorts of new algorithms, tricks, and tweaks that have been introduced under various catchy names for the underlying functional blocks (still consisting mostly of various combinations of basic linear algebra operations). Historically, the two encompassing streams of symbolic and sub-symbolic stances to AI evolved in a largely separate manner, with each camp focusing on selected narrow problems of their own.

Neither pure neural networks nor pure symbolic AI alone can solve such multifaceted challenges. But together, they achieve impressive synergies not possible with either paradigm alone. Visual cues such as added spacing, lines, and circles influence the application of perceptual grouping mechanisms, influencing the capacity for symbolic reasoning. When deep learning reemerged in 2012, it was with a kind of take-no-prisoners attitude that has characterized most of the last decade.

A simple guide to gradient descent in machine learning

It had the first self-hosting compiler, meaning that the compiler itself was originally written in LISP and then ran interpretively to compile the compiler code. In summary, symbolic logic is an intellectual Swiss Army knife, carving out clarity from the potentially murky world of arguments and ideas. It’s not just about symbols; it’s about seeing the hidden structure in our thoughts, like x-ray vision for the mind.

However, when individuals engage in tasks requiring thinking and reasoning, such as executive functions, novel problem solving, mathematics, or understanding computer code, entirely different brain networks are activated—the multiple demand network. The existence of a formal language for representing information and the existence of a corresponding set of mechanical manipulation rules together have an important consequence, viz. This example is interesting in that it showcases our formal language for encoding logical information. As with algebra, we use symbols to represent relevant aspects of the world in question, and we use operators to connect these symbols in order to express information about the things those symbols represent. Although logical sentences can sometimes pinpoint a specific world from among many possible worlds, this is not always the case. Sometimes, a collection of sentences only partially constrains the world.

We use curriculum learning to guide searching over the large compositional space of images and language. Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences. Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains.

This type of logic allows more kinds of knowledge to be represented understandably, with real values allowing representation of uncertainty. Many other approaches only support simpler forms of logic like propositional logic, or Horn clauses, or only approximate the behavior of first-order logic. Semantic networks, conceptual graphs, frames, and logic are all approaches to modeling knowledge such as domain knowledge, problem-solving knowledge, and the semantic meaning of language. Ontologies model key concepts and their relationships in a domain. DOLCE is an example of an upper ontology that can be used for any domain while WordNet is a lexical resource that can also be viewed as an ontology. YAGO incorporates WordNet as part of its ontology, to align facts extracted from Wikipedia with WordNet synsets.

Our NSQA achieves state-of-the-art accuracy on two prominent KBQA datasets without the need for end-to-end dataset-specific training. Due to the explicit formal use of reasoning, NSQA can also explain how the system arrived at an answer by precisely laying out the steps of reasoning. Controversies arose from early on in symbolic AI, both within the field—e.g., between logicists (the pro-logic “neats”) and non-logicists (the anti-logic “scruffies”)—and between those who embraced AI but rejected symbolic approaches—primarily connectionists—and those outside the field.

• Perceptual Manipulations Theory (PMT) goes further than the cyborg account in emphasizing the perceptual nature of symbolic reasoning.
• This view then made even more space for all sorts of new algorithms, tricks, and tweaks that have been introduced under various catchy names for the underlying functional blocks (still consisting mostly of various combinations of basic linear algebra operations).
• No explicit series of actions is required, as is the case with imperative programming languages.
• However, the relational program input interpretations can no longer be thought of as independent values over a fixed (finite) number of propositions, but an unbound set of related facts that are true in the given world (a “least Herbrand model”).
• But the benefits of deep learning and neural networks are not without tradeoffs.

By combining deep learning neural networks with logical symbolic reasoning, AlphaGeometry charts an exciting direction for developing more human-like thinking. We have described an approach to symbolic reasoning which closely ties it to the perceptual and sensorimotor mechanisms that engage physical notations. With respect to this evidence, PMT compares favorably to traditional “translational” accounts of symbolic reasoning. The recent adaptation of deep neural network-based methods to reinforcement learning and planning domains has yielded remarkable progress on individual tasks. In pursuit of efficient and robust generalization, we introduce the Schema Network, an object-oriented generative physics simulator capable of disentangling multiple causes of events and reasoning backward through causes to achieve goals. The richly structured architecture of the Schema Network can learn the dynamics of an environment directly from data.

These old-school parallels between individual neurons and logical connectives might seem outlandish in the modern context of deep learning. However, given the aforementioned recent evolution of the neural/deep learning concept, the NSI field is now gaining more momentum than ever. The research conducted by Fedorenko and colleagues provides compelling evidence that language does not engage the brain’s symbolic reasoning functions. With these abbreviations, we can represent the essential information of this problem with the following logical sentences. The first says that p implies q, i.e. if Mary loves Pat, then Mary loves Quincy. The second says that m and r implies p or q, i.e. if it is Monday and raining, then Mary loves Pat or Mary loves Quincy.

Even in philosophy’s diverse landscape, from ethics to metaphysics, it’s a versatile tool for disentangling some exceptionally knotty problems. Note the similarity to the use of background knowledge in the Inductive Logic Programming approach to Relational ML here. Although Logic is a single field of study, there is more than one logic in this field. In the three main units of this book, we look at three different types of logic, each more sophisticated than the one before.

Logic is the study of information encoded in the form of logical sentences. Each logical sentence divides the set of all possible world into two subsets – the set of worlds in which the sentence is true and the set of worlds in which the set of sentences is false. A set of premises logically entails a conclusion if and only if the conclusion is true in every world in which all of the premises are true. Deduction is a form of symbolic reasoning that produces conclusions that are logically entailed by premises (distinguishing it from other forms of reasoning, such as induction, abduction, and analogical reasoning). A proof is a sequence of simple, more-or-less obvious deductive steps that justifies a conclusion that may not be immediately obvious from given premises. In Logic, we usually encode logical information as sentences in formal languages; and we use rules of inference appropriate to these languages.

Learn from Harold Cohen’s pioneering work and explore how modern AI systems benefit from combining symbolic logic with machine learning techniques. Deep learning and neural networks excel at exactly the tasks that symbolic AI struggles with. They have created a revolution in computer vision applications such as facial recognition and cancer detection. Deep learning has also driven advances in language-related tasks.

Explainable neural networks that simulate reasoning

Each sentence divides the set of possible worlds into two subsets, those in which the sentence is true and those in which the sentence is false, as suggested by the following figure. Believing a sentence is tantamount to believing that the world is in the first set. Looking at the worlds above, we see that all of these sentences are true in the world on the left.

Each approach—symbolic, connectionist, and behavior-based—has advantages, but has been criticized by the other approaches. Symbolic AI has been criticized as disembodied, liable to the qualification problem, and poor in handling the perceptual problems where deep learning excels. In turn, connectionist AI has been criticized as poorly suited for deliberative step-by-step problem solving, incorporating knowledge, and handling planning. Finally, Nouvelle AI excels in reactive and real-world robotics domains but has been criticized for difficulties in incorporating learning and knowledge. NSI has traditionally focused on emulating logic reasoning within neural networks, providing various perspectives into the correspondence between symbolic and sub-symbolic representations and computing. Historically, the community targeted mostly analysis of the correspondence and theoretical model expressiveness, rather than practical learning applications (which is probably why they have been marginalized by the mainstream research).

2) The two problems may overlap, and solving one could lead to solving the other, since a concept that helps explain a model will also help it recognize certain patterns in data using fewer examples. As a consequence, the Botmaster’s job is completely different when using Symbolic AI technology than with Machine Learning-based Chat GPT technology as he focuses on writing new content for the knowledge base rather than utterances of existing content. He also has full transparency on how to fine-tune the engine when it doesn’t work properly as he’s been able to understand why a specific decision has been made and has the tools to fix it.

We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analog to the human concept learning, given the parsed program, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences.

• By contrast, several of the sentences are false in the world on the right.
• Although deep learning has historical roots going back decades, neither the term “deep learning” nor the approach was popular just over five years ago, when the field was reignited by papers such as Krizhevsky, Sutskever and Hinton’s now classic (2012) deep network model of Imagenet.
• Logic is important in all of these disciplines, and it is essential in computer science.
• By integrating symbolic reasoning into AI, we build on the legacy of brilliant minds like Harold Cohen and push the boundaries of what AI systems can achieve.
• If we replace x by Toyotas and y by cars and z by made in America, we get the following line of argument, leading to a conclusion that happens to be correct.

So the main challenge, when we think about GOFAI and neural nets, is how to ground symbols, or relate them to other forms of meaning that would allow computers to map the changing raw sensations of the world to symbols and then reason about them. These dynamic models finally enable to skip the preprocessing step of turning the relational representations, such as interpretations of a relational logic program, into the fixed-size vector (tensor) format. They do so by effectively reflecting the variations in the input data structures into variations in the structure of the neural model itself, constrained by some shared parameterization (symmetry) scheme reflecting the respective what is symbolic reasoning model prior. While the aforementioned correspondence between the propositional logic formulae and neural networks has been very direct, transferring the same principle to the relational setting was a major challenge NSI researchers have been traditionally struggling with. The issue is that in the propositional setting, only the (binary) values of the existing input propositions are changing, with the structure of the logical program being fixed. And while these concepts are commonly instantiated by the computation of hidden neurons/layers in deep learning, such hierarchical abstractions are generally very common to human thinking and logical reasoning, too.

We compare Schema Networks with Asynchronous Advantage Actor-Critic and Progressive Networks on a suite of Breakout variations, reporting results on training efficiency and zero-shot generalization, consistently demonstrating faster, more robust learning and better transfer. We argue that generalizing from limited data and learning causal relationships are essential abilities on the path toward generally intelligent systems. Combining symbolic reasoning with deep neural networks and deep reinforcement learning may help us address the fundamental challenges of reasoning, hierarchical representations, transfer learning, robustness in the face of adversarial examples, and interpretability (or explanatory power). According to Wikipedia, machine learning is an application of artificial intelligence where “algorithms and statistical models are used by computer systems to perform a specific task without using explicit instructions, relying on patterns and inference instead. (…) Machine learning algorithms build a mathematical model based on sample data, known as ‘training data’, in order to make predictions or decisions without being explicitly programmed to perform the task”. In the next article, we will then explore how the sought-after relational NSI can actually be implemented with such a dynamic neural modeling approach.

Source Data Fig. 5

Kahneman describes human thinking as having two components, System 1 and System 2. System 1 is the kind used for pattern recognition while System 2 is far better suited for planning, deduction, and deliberative thinking. In this view, deep learning best models the first kind of thinking while symbolic reasoning best models the second kind and both are needed. Implementations of symbolic reasoning are called rules engines or expert systems or knowledge graphs. Google made a big one, too, which is what provides the information in the top box under your query when you search for something easy like the capital of Germany. These systems are essentially piles of nested if-then statements drawing conclusions about entities (human-readable concepts) and their relations (expressed in well understood semantics like X is-a man or X lives-in Acapulco).

In situations like this, which world should we use in answering questions? Even though a set of sentences does not determine a unique world, there are some sentences that have the same truth value in every world that satisfies the given sentences, and we can use that value in answering questions. Once we know which world is correct, we can see that some sentences must be true even though they are not included in the premises we are given. For example, in the first world we saw above, we can see that Bess likes Cody, even though we are not told this fact explicitly.

These pioneers crafted symbolic logic into the precise, finely tuned tool that it is today, comparable to a mathematician’s trusty set of formulas and equations. Note the similarity to the propositional and relational machine learning we discussed in the last article. Perhaps surprisingly, the correspondence between the neural and logical calculus has been well established throughout history, https://chat.openai.com/ due to the discussed dominance of symbolic AI in the early days. Functional Logic takes us one step further by providing a means for describing worlds with infinitely many objects. The resulting logic is much more powerful than Propositional Logic and Relational Logic. Unfortunately, as we shall see, some of the nice computational properties of the first two logics are lost as a result.

Even if you take a million pictures of your cat, you still won’t account for every possible case. A change in the lighting conditions or the background of the image will change the pixel value and cause the program to fail. Symbols play a vital role in the human thought and reasoning process. If I tell you that I saw a cat up in a tree, your mind will quickly conjure an image. Opposing Chomsky’s views that a human is born with Universal Grammar, a kind of knowledge, John Locke[1632–1704] postulated that mind is a blank slate or tabula rasa.

Its history was also influenced by Carl Hewitt’s PLANNER, an assertional database with pattern-directed invocation of methods. For more detail see the section on the origins of Prolog in the PLANNER article. Symbolic logic is a podium where our thoughts and arguments can stand to be inspected.

McCarthy’s Advice Taker can be viewed as an inspiration here, as it could incorporate new knowledge provided by a human in the form of assertions or rules. For example, experimental symbolic machine learning systems explored the ability to take high-level natural language advice and to interpret it into domain-specific actionable rules. While the particular techniques in symbolic AI varied greatly, the field was largely based on mathematical logic, which was seen as the proper (“neat”) representation formalism for most of the underlying concepts of symbol manipulation. With this formalism in mind, people used to design large knowledge bases, expert and production rule systems, and specialized programming languages for AI. A remarkable new AI system called AlphaGeometry recently solved difficult high school-level math problems that stump most humans.

Whether it’s mapping out the logic of a scientific discovery or navigating the ethics of right and wrong, symbolic logic helps us get there without getting lost. It might not have all the answers, but it sure points us in the right direction. Looking again — a bit closer — at the first proposal of a computational neuron from the 1943’s paper “A logical calculus of the ideas immanent in nervous activity” by McCulloch and Pitts [1], we can see that it was actually thought to emulate logic gates over input (binary-valued) propositions. The idea was based on the, now commonly exemplified, fact that logical connectives of conjunction and disjunction can be easily encoded by binary threshold units with weights — i.e., the perceptron, an elegant learning algorithm for which was introduced shortly.

With the rules of logic, you can move these symbols around, swap them, or combine them in different ways to explore the argument. It’s a little like following a recipe where the rules are your ingredients and steps. You end up with a clear idea of whether the argument holds up to scrutiny. Since its foundation as an academic discipline in 1955, Artificial Intelligence (AI) research field has been divided into different camps, of which symbolic AI and machine learning. While symbolic AI used to dominate in the first decades, machine learning has been very trendy lately, so let’s try to understand each of these approaches and their main differences when applied to Natural Language Processing (NLP). However, in the meantime, a new stream of neural architectures based on dynamic computational graphs became popular in modern deep learning to tackle structured data in the (non-propositional) form of various sequences, sets, and trees.

In machine learning, the algorithm learns rules as it establishes correlations between inputs and outputs. In symbolic reasoning, the rules are created through human intervention and then hard-coded into a static program. From a more practical perspective, a number of successful NSI works then utilized various forms of propositionalisation (and “tensorization”) to turn the relational problems into the convenient numeric representations to begin with [24]. However, there is a principled issue with such approaches based on fixed-size numeric vector (or tensor) representations in that these are inherently insufficient to capture the unbound structures of relational logic reasoning.

Lacking the ability to model complex real-life problems involving abstract knowledge with relational logic representations (explained in our previous article), the research in propositional neural-symbolic integration remained a small niche. The true resurgence of neural networks then started by their rapid empirical success in increasing accuracy on speech recognition tasks in 2010 [2], launching what is now mostly recognized as the modern deep learning era. Shortly afterward, neural networks started to demonstrate the same success in computer vision, too. The difficulties encountered by symbolic AI have, however, been deep, possibly unresolvable ones.

For example, OPS5, CLIPS and their successors Jess and Drools operate in this fashion. Symbolic artificial intelligence is very convenient for settings where the rules are very clear cut,  and you can easily obtain input and transform it into symbols. In fact, rule-based systems still account for most computer programs today, including those used to create deep learning applications. The key innovation underlying AlphaGeometry is its “neuro-symbolic” architecture integrating neural learning components and formal symbolic deduction engines.