The interdisciplinary work on the linear threshold unit was carried out by a seasoned psychiatrist and neurophysiologist (Warren McCulloch) and a young mathematician Walter Pitts. Their monumental work, "A Logical Calculus of the Ideas Immanent in Nervous Activity", was based on a network of simple units which are meant to capture the essential workings of the biological neuron.
The linear threshold unit computes its net input as the weighted sum of all inputs converging to it. If the net input exceeds the threshold, then the unit activates (output of 1). Otherwise, it remains inactive (output of 0). In the figure below, xi is the input signal coming from input unit i (i.e., output signal of unit i) at the other end of the incoming connection. wi is the weight of the connection between the logical threshold unit and input unit i. Notice how the linear threshold unit adheres to the general operation of the biological neuron.
The output y is computed as follows, where q is the unit’s threshold:
The McCulloch & Pitts model has gained popularity because linear threshold units can represent boolean operators such as AND and OR. By assigning suitable connection weights and thresholds, we can have AND and OR units as depicted in the figures below:
By putting together these boolean operators in a huge "nerve network", it seemed no longer far-fetched that an artificial machine could one day be doted with intelligence. This was the main thrust of the inetllectual movement in the 1940's and 1950's, called "cybernetics", which combined results from various fields like biology, psychology, engineering, mathematics, and in a way, philosophy. It may surprise young researchers that such attempts at building intelligent machines have been around even before the modern computer was born.
Indeed, much theoretical and philosophical work on computation and cybernetics have predated the modern day computers. Most notable are Turing's work on Turing Machines and the Universal Turing Machine which predated the ENIAC by a whole decade.
No doubt, the "nerve net" was a first concrete fairly sophisticated model for building artificial systems that might be able to exhibit intelligent behavior. Conspicuously missing, though, was a systematic way of finding suitable weights and threshold values. If only the linear threshold units could automatically take on suitable values for these parameters by just being shown examples of the boolean operation results, then perhaps the McCulloch and Pitts model could be more useful. Unfortunately, not much was known at that time about how neurons change synaptic efficiencies (which correspond to weights) .
By 1949, Donald Hebb published his findings and conjectures in a book, "The Organization of Behavior", which studied how "cell assemblies" in the brain might be able to make internal representations of 'concepts'. In the same book, Hebb postulates on how nearby neurons can reinforce each other and thus learn. What is now known as Hebb's postulate of learning (or simply Hebb's rule), is the following: "When an axon of cell A is near enough to excite cell B and repeatedly or persistently takes part in firing it, some growth processes or metabolic changes take place in one or both cells such that A's efficiency as one of the cells firing B, is increased".
Most of the learning rules in use today allow for connection weights (synaptic efficiencies) to also decrease. Note that Hebb's rule only talks of "increasing" connection weights. Nevertheless, all of these modern incremental learning rules find their roots in the original Hebbian postulate.
There was not enough detail in Hebb's work to be able to model the learning process in some artificial system, but Hebb’s ideas were concrete enough to pave the way for Rosenblatt [1959] to device automatic learning methods for linear threshold units. More importantly, Rosenblatt provided mathematical proofs that these methods would, under certain conditions, converge to adequate solutions in finite time. McCulloch and Pitts’ linear threshold units that are capable of learning are essentially Rosenblatt’s "perceptrons”.
