On 2026-11-16 7:22 p.m., in response to some suggestions from Bob Worcester, I wrote:
I think the 3b1b series (starting with https://www.youtube.com/watch?v=aircAruvnKk) is well worth digging into. The second link you sent (at https://youtu.be/olDwTWN9vs4?si=NYKdlvasO6nOhA0N ) is an interesting example of something, but I think is wrong to identify the iterations of Newton’s method as training runs of a learning system. And this last one (at https://www.youtube.com/watch?v=jmmW0F0biz0&t=10s )is I think “less complicated” only because it explains so much less (and overlooks the importance, explained in the second one, of NON-linear behaviour of a neuron).
On 2026-11-16 11:41 a.m., Robert Worcester wrote:
This is a less complicated explanation of neural networks: https://www.youtube.com/watch?v=jmmW0F0biz0&t=10s
On 2026-11-12 8:34 a.m., Robert Worcester wrote:
The best simple explanation I have run across so far.
https://youtu.be/olDwTWN9vs4?si=NYKdlvasO6nOhA0N
The math is (to me) scary BUT the organic structure that this machinery seek to emulate is MUCH more complicated than the math. Where does physics become biology (or vise versa)?
On Wed, 11 Nov 2026 at 07:58, Robert Worcester wrote:
Hi – In response to our discussions yesterday I was motivated to try to better “understand” how neural networks (wet or dry) actually work.
I found this helpful though I confess I don’t really understand the math of linear algebra BUT it looks potentially “understandable.”
I am wondering if a mathematician can facilitate the transmission of the neural weightings necessary for such understanding?
A Simple Model Transmission Neuron is something that accepts as input (from a set of sources which we’ll call its “dendritic synapses” or “upstream layer”) a discrete set of signal strengths (each of which can be represented by a real number), and for each such collection of inputs sends to a destination (which we’ll call its “downstream channel” or “axon”) an output signal (of the same kind as the inputs). In the case of a computer model neuron the output signal is completely determined by the input values (ie in mathematical terminology it is a “function” of them), but of course in a biological neuron the behaviour may be influenced by many things including the surrounding chemical bath (so believers in the special value of natural intelligence might take comfort in the idea that a biological neural network may well have properties that are not captured in the current computer models).
Transmission Neurons may be all that is required in order to pass signals from the brain to muscles etc. And such neurons can also be useful for various kinds of signal processing (such as identifying boundaries in a pixelated image. But in order to be useful for learning, the neuron must ALSO accept inputs that it interprets as requests to increase or decrease its output value, to which it responds with some combination of adjusting its own behaviour and/or passing on requests for changes of its other inputs. Such requests coming, either directly or indirectly, from the output channel are referred to as “back propagation”. This can be done in various ways, and the details may be hard to follow, so it is often left out of discussions of how to model a single neuron.
For example, the model of a single neuron from 1:05 to 3:15 in the video by Aviraj Ingle deals well enough with the forward process but omits any reference to the adjustment capability which is required for “back propagation” in neural networks. [This is followed by a section which oddly combines discussion of the need for nonlinearity in order for network “layers” to be useful (even though there is no discussion of multi-layer networks anywhere else in the video) with how cutoffs enhance the capabilities of a single layer, and then an example in which back propagation is actually used without ever having been properly explained and in which iteration of steps in producing the response to a single data set appears to be being confounded with the process of training with different sample data points. But despite these weaknesses the video does point in the direction of some examples that might be simple enough to explain properly, and complex enough to convey a good idea of what is going on in realistic applications of neural networks, and that is part of what has prompted me to see if I can do better.]
The most important thing we need to know at this point is that the output of a single neuron is defined by a function of its input values which depends on various parameters (often called weights, biases, and cutoffs), and that the neuron may have inputs whose only contribution to the output is by way of causing changes to the parameters.
There are many possible models, but a common form for the forward propagation through a single neuron starts by just taking some basic “bias” value $#b#$ and adding multiples of the inputs $#x_i#$ by “weights” $#w_i#$ giving result $#r=b+\Sigma w_{i} x_{i}#$. This purely linear response though cannot be sufficient because (since sums and other linear combinations of linear functions are still linear) it gives no way of modelling a non-linear response, even with many neurons. However, if we combine it with cutoffs on the output and/or restrictions on the input values for which the neuron gives a non-zero response (which in math is called the “support” of the neuron’s response) then with enough neurons the combination of linear pieces can indeed model (or at least approximate) just about anything. One type of cutoff that is common especially if considering the output as a percentage of full activation (for the purpose, say, of computing a probability or confidence level) is to give (or perhaps just go towards) $#0#$ for results $#r#$ that are smaller than some lower cutoff $#m#$, and $#1#$ for all that are larger than some upper cutoff $#M#$, and scales intermediate results between $#m#$ and $#M#$ into outputs between $#0#$ and $#1#$. [We can do this in many ways – eg with strict cutoffs and linear scaling to get $#y=(\frac{1}{2}+\frac{|r-m|-|r-M|}{2(M-m)})#$, or more smoothly with a “sigmoid” $#y=\sigma(z)=(\frac{1}{1+e^{-z}})#$ with $#z=(r-\frac{m+M}{2})(\frac{4}{M-m})#$]
Even without back-propagation, if we choose the response function directly (perhaps by specifying some weights and cutoffs as in the model type discussed above) then such a neuron does provide a model for making or predicting some simple kinds of decision. For example if someone is in a situation where they have access to a down jacket but no waterproof layer and we want to predict the probability of whether or not they will go out on the basis of the predicted probabilities of cold and wet weather, then we might assign a high negative weight to the probability of precipitation and a low negative weight to the probability of having a temperature below 10 Celsius. But if we expect that they have some desire to actually go out then we might have a lower cutoff corresponding to how strong we expect that desire to be and the probability that they actually decide to go out might then be equal to the output from such a neuron with inputs equal to the applicable probabilities of rain and cold. Or in biology, such a fixed-weight neuron might enable a simple organism to decide whether or not to move on the basis of its sensitivities to the concentrations of various offensive chemicals in its environment with a cutoff based on its need not to waste energy in unnecessary motion. But in such cases the neuron just has fixed behaviour – either evolved over many generations or put in by a human, and there is no immediate learning involved.
It may seem that the use of linear functions is unduly restrictive, but with the inclusion of cutoffs a bunch of such neurons acting in parallel can approximately model any smooth function.
For example, the function $#f(x)=1-x^2#$ for $#-1<x<1#$ can be approximated to within $#\pm0.1#$ by the piecewise linear function $#f(x)=2(x+1) for -1<x<-3/4 ; 1+2x/3 for -3/4<x<0; 1-2x/3 for 0<x<3/4; and -2(x-1) for 3/4<x<1#$ which is just the sum of the outputs of four single-input neurons. But here of course, the neurons have been given their weights and cutoffs by me and there was no learning involved.
So now let’s re-introduce the learning process.
For this we need the neurons to be able to detect where they went wrong and have some way they can arrange to do better. Such adaptation requires that the neuron have at least one additional input which can be thought of coming back along the axon (or by way of a simple neuron going in the opposite direction) and which tells our main neuron the difference between its current output and the value of some target variable. However the way this input is processed is not by being included in the weighted sum but rather by causing changes of the weights.
As an example of such learning, let’s consider something like the above example but replace the given function $#f(x)=1-x^2#$ with something else – even perhaps just a curve drawn by hand and see if we can give the neurons a way of adjusting their parameters to approximate whatever target we provide. (The neurons with adjusted parameters then form a network which has in some sense learned and remembers the curve well enough to tell an organism how to re-draw it.)
In order to find a good four step polygonal approximation, we can start with the same set of four intermediate neurons but rather than put in the actual weights and cutoffs we can start by putting in random values and let the neurons themselves figure out what to do. Their goal is to provide line segments that remain connected and minimize something like the maximum or average distance from the curve; and one way of doing this, at least approximately, is to “train” the network by providing a bunch of input values for the x-variable, and for each one in turn find the difference between the output of the network and the value of the target function and adjust the weight and bias of each neuron in a direction which reduces something like the maximum or average of all those differences plus all of the jumps. This will require each neuron to get feedback from the overall output and from its neighbours (from which it will need to know the outputs they give at the ends of their support).
For simplicity though, let’s keep the cutoffs fixed so the $#i#$th neuron outputs $#b_i+w_i x#$ for $#-1+(i-1)/2<x<-1+i/2#$ and $#0#$ otherwise, and start not randomly but just with the $#b#$s and $#w#$s all set to zero.
On the first training point $#x_1#$ the neuron in whose support it lies can reduce the output error to zero by just adjusting its bias to match the value of $#f(x_1)#$ and the others (which have no impact on the output) can reduce the jumps to zero by going to the same bias, and so the first approximation is just the constant function $#f_1(x)=f(x_1)#$ for all $#x#$.
With the second training point $#x_2#$, if it is in the same support interval as the first then the neuron needs to adjust its slope and bias to match those of the straight line between the two points and other neurons can do anything that matches values at the endpoints, and if it is in a different support interval the neuron in whose support it lies needs to match the previous result at the end closer to $#x_1#$ and it will be possible to get zero errors by having the neurons on the other side give segments that all match up with the straight line through the endpoint and the point at $#(x_2, f(x_2))#$
With more inputs it may not be possible to match each segment exactly with a straight line, but if we have chosen some measure of error we can adjust the weights and biases so as to minimize it. A common choice that’s both somewhere between the maximum and average of distances (and also can be calculated without using absolute values) is the RMS error (square root of the average of the squares of the individual errors); but whatever we use will, for any given family of inputs, be a function of the neurons’ parameters (weights, biases, and cutoffs), and as such can be minimized by a process of successive small changes in which the relative step sizes of the various parameters are chosen to give the fastest possible rate of decrease. [This “method of steepest descent” (in which the reduction of step sizes to avoid overshoot is analogous to using Newton’s method for forcing the gradient towards zero) is an iterative process for getting closer and closer to the optimal parameters for a given set of training data and the number of steps we take before stopping has nothing necessarily to do with the number of points in the training data.]
The example of training given in the video by Aviraj Ingle involves neurons with two inputs and an output with two values “Red” and “Blue”; but it is not clear what the problem to be solved is as the rule for placing the post-training test points is not specified. But I suspect that the intended problem is for the plane to be divided into two regions, one Red and one Blue, and for the network to learn by checking the colours of a number of test points how to identify whether newly placed points will be red or blue.
This uses neurons with two inputs corresponding to the coordinates of a point and an output of either “Red” or “Blue” depending on which region the point is in. A natural way to do this is for each neuron to give its estimate of the probability that the colour is red by returning a number between $#0#$ and $#1#$ which is obtained by calculating $#L=w_1x_1+w_2x_2#$, mapping positive values to $#1#$ and negatives to $#-1#$, and then re-scaling by a confidence factor which is low outside some region (corresponding to where the neuron “thinks” that $#w_1x_1+w_2x_2 =0#$ is a good approximation to part of the boundary between red and blue – with red on the +side), and finally mapping $#-1#$ to $#0#$ (for certainly not red) and $#0#$ to $#1/2#$ (for completely uncertain). So, for example we might end use something like $#p=(1+e^{-((x_1-c_1)^2+(x_2-c_2)^2}\frac{L}{|L|}/2#$
If we call the input values $#x_1#$ and $#x_2#$ and the output is initially given by $#y=w_1x_1+w_2x_2#$, then a request to increase the output can be met by adjusting the “weights” $#w_1#$ and $#w_2#$ in various ways but for fixed input values the fastest increase per unit “distance” in the $#(w_1,w_2)#$ space is when the change $#(\Delta w_1,\Delta w_2)#$ is parallel to the input vector $#(x_1,x_2)#$. [And also, if we are able to change the inputs, then for any given values of $#w_1#$ and $#w_2#$ the output can be increased by requesting an increment of the input values for which the most efficient choice is when $#(\Delta x_1,\Delta x_2)#$ is parallel to the weights vector $#(w_1,w_2)#$.]



