Introduction to Soft Computing

Question Statement        Solution with step wise marking

What are 3 major categories of neural networks?              The three most important types of neural networks are: Artificial Neural Networks (ANN); Convolution Neural Networks (CNN), and Recurrent Neural Networks (RNN).

What is neural network and its types?     Neural Networks are artificial networks used in Machine Learning that work in a similar fashion to the human nervous system. Many things are connected in various ways for a neural network to mimic and work like the human brain. Neural networks are basically used in computational models.

What is CNN and DNN?  A deep neural network (DNN) is an artificial neural network (ANN) with multiple layers between the input and output layers. They can model complex non-linear relationships. Convolutional Neural Networks (CNN) are an alternative type of DNN that allow modelling both time and space correlations in multivariate signals.

How does CNN differ from Ann? CNN is a specific kind of ANN that has one or more layers of convolutional units. The class of ANN covers several architectures including Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN) eg LSTM and GRU, Autoencoders, and Deep Belief Networks.

Why is CNN better than MLP?     Multilayer Perceptron (MLP) is great for MNIST as it is a simpler and more straight forward dataset, but it lags when it comes to real-world application in computer vision, specifically image classification as compared to CNN which is great.

Classify neural network architecture.       Neural network architecture is classified as – single layer feed forward networks, multilayer feed forward networks and recurrent networks.

What are the disadvantages of Artificial Neural Networks?            The major disadvantage is that they require large diversity of training for working in a real environment. Moreover, they are not strong enough to work in the real world.

What is directed graph? We know graph consist of vertices and edges and when orientation is assigned to edge of graph then it is called as directed graph.

What are the benefits of neural network?             Benefits are as follows- Parallel computing, faster, robust on noisy training data, mapping capabilities.

Define neural network.  Neural network is simplified central nervous system model and it is motivated by the computing which is performed by human brains and the technology built on this logic is termed as neural network.

How Artificial Neural Networks can be applied in future?               (1) Pen PC’s: PC’s where one can write on a tablet, and the writing will be recognized and translated into (ASCII) text.                                                  (2) White goods and toys: As Neural Network chips become available, the possibility of simple cheap systems which have learned to recognize simple entities (e.g. walls looming, or simple commands like Go, or Stop), may lead to their incorporation in toys and washing machines etc. Already the Japanese are using a related technology, fuzzy logic, in this way. There is considerable interest in the combination of fuzzy and neural technologies.

Are Neural Networks helpful in Medicine?             (1) Electronic noses ANNs are used experimentally to implement electronic noses. Electronic noses have several potential applications in telemedicine. Telemedicine is the practice of medicine over long distances via a  (2) Instant Physician     An application developed in the mid-1980s called the “instant physician” trained an auto-associative memory neural network to store a large number of medical records, each of which includes information on symptoms, diagnosis, and treatment for a particular case. After training, the net can be presented with input consisting of a set of symptoms; it will then find the full stored pattern that represents the “best” diagnosis and treatment.communication link. The electronic nose would identify odors in the remote surgical environment. These identified odors would then be electronically transmitted to another site where an door generation system would recreate them. Because the sense of smell can be an important sense to the surgeon, telesmell would enhance telepresent surgery.

List some commercial practical applications of Artificial Neural Networks.              Since neural networks are best at identifying patterns or trends in data, they are well suited for prediction or forecasting needs including:   sales forecasting ndustrial process control customer research data validation,  risk management, target marketing

How Artificial Neurons learns?    Associative Mapping:   Here the network produces a pattern output by working in a pattern on the given input.  This is a two paradigm process- §  Regularity Detection:  In this, units learn to respond to particular properties of the input patterns. Whereas in associative mapping the network stores the relationships among patterns, in regularity detection the response of each unit has a particular ‘meaning’. This type of learning mechanism is essential for feature discovery and knowledge representation.

What is simple Artificial Neuron?              It is simply a processor with many inputs and one output….It works in either the Training Mode or Using Mode.  In the training mode, the neuron can be trained to fire (or not), for particular input patterns. In the using mode, when a taught input pattern is detected at the input, its associated output becomes the current output. If the input pattern does not belong in the taught list of input patterns, the firing rule is used to determine whether to fire or not.

How human brain works?             It is weird at the same time amazing to know that we really do not know how we think.  Biologically, neurons in human brain receive signals from host of fine structures called as dendrites.  The neuron sends out spikes of electrical activity through a long, thin stand known as an axon, which splits into thousands of branches. At the end of each branch, a structure called a synapse converts the activity from the axon into electrical effects that inhibit or excite activity from the axon into electrical effects that inhibit or excite activity in the connected neurons. When a neuron receives excitation input that is sufficiently large compared with its inhibitory input, it sends a spike of electrical activity down its axon. Learning occurs by changing the effectiveness of the synapses so that the influence of one neuron on another changes.

How are Artificial Neural Networks different from Normal Computers?     Simple difference is that the Artificial Neural Networks can learn by examples contrary to Normal Computers who perform the task on Algorithms. Although, the examples given to Artificial Neural Networks should be carefully chosen. Once properly “taught” Artificial Neural Networks can  do on their own,,,or at least try to imitate..But that makes them so Unpredictable , which is opposite to that of algorithm based computers which we use in our daily life.

What are Neural Networks? What are the types of Neural networks?        In simple words, a neural network is a connection of many very tiny processing elements called as neurons. There are two types of neural network-

Why use Artificial Neural Networks? What are its advantages?     :  Mainly, Artificial Neural Networks OR Artificial Intelligence is designed to give robots human quality thinking. So that machines can decide “What if” and ”What if not” with precision. Some of the other advantages are:-   Adaptive learning: Ability to learn how to do tasks based on the data given for training or initial experience.  Self-Organization: An Artificial Neural Networks can create its own organization or representation of the information it receives during learning time. §  Real Time Operation: Artificial Neural Networks computations may be carried out in parallel, and special hardware devices are being designed and manufactured which take advantage of this capability. Fault Tolerance via Redundant Information Coding: Partial destruction of a network leads to the corresponding degradation of performance. However, some network capabilities may be retained even with major network damage.

Elaborate about Neural Network architecture     Neural networks represent deep learning using artificial intelligence. Certain application scenarios are too heavy or out of scope for traditional machine learning algorithms to handle. As they are commonly known, Neural Network pitches in such scenarios and fills the gap. Also, enrol in the neural networks and deep learning course and enhance your skills today. Briefly, each neuron receives a multiplied version of inputs and random weights which is then added with static bias value (unique to each neuron layer), this is then passed to an appropriate activation function which decides the final value to be given out of the neuron.

Explain about weights.   Weights are numeric values which are multiplied with inputs. In backpropagation, they are modified to reduce the loss. In simple words, weights are machine learnt values from Neural Networks. They self-adjust depending on the difference between predicted outputs vs training inputs.

Activation function          Activation Function is a mathematical formula which helps the neuron to switch ON/OFF.

Types of layers in Neural Network.           ·        Input layer represents dimensions of the input vector. Hidden layer represents the intermediary nodes that divide the input space into regions with (soft) boundaries. It takes in a set of weighted input and produces output through an activation function.     Output layer represents the output of the neural network.

Perceptron model            Perceptron model, proposed by Minsky-Papert is one of the simplest and oldest models of Neuron. It is the smallest unit of neural network that does certain computations to detect features or business intelligence in the input data. It accepts weighted inputs, and apply the activation function to obtain the output as the final result. Perceptron is also known as TLU(threshold logic unit), Perceptron is a supervised learning algorithm that classifies the data into two categories, thus it is a binary classifier.

Advantage and disadvantages of Perceptron        Advantages of Perceptron Perceptrons can implement Logic Gates like AND, OR, or NAND. Disadvantages of Perceptron Perceptrons can only learn linearly separable problems such as boolean AND problem. For non-linear problems such as the boolean XOR problem, it does not work.

Applications on Feed Forward Neural Networks  ·        Simple classification (where traditional Machine-learning based classification algorithms have limitations) ·        Face recognition [Simple straight forward image processing] ·        Computer vision [Where target classes are difficult to classify] ·        Speech Recognition, The simplest form of neural networks where input data travels in one direction only, passing through artificial neural nodes and exiting through output nodes. Where hidden layers may or may not be present, input and output layers are present there. Based on this, they can be further classified as a single-layered or multi-layered feed-forward neural network., Number of layers depends on the complexity of the function. It has uni-directional forward propagation but no backward propagation. Weights are static here. An activation function is fed by inputs which are multiplied by weights. To do so, classifying activation function or step activation function is used. For example: The neuron is activated if it is above threshold (usually 0) and the neuron produces 1 as an output. The neuron is not activated if it is below threshold (usually 0) which is considered as -1. They are fairly simple to maintain and are equipped with to deal with data which contains a lot of noise.

Advantages and disadvantages of Feed Forward Neural Networks              Advantages of Feed Forward Neural Networks: 1.       Less complex, easy to design & maintain 2.     Fast and speedy [One-way propagation] 3.     Highly responsive to noisy data Disadvantages of Feed Forward Neural Networks: 1.       Cannot be used for deep learning [due to absence of dense layers and back propagation]

Applications on Multi-Layer Perceptron  Applications on Multi-Layer Perceptron: An entry point towards complex neural nets where input data travels through various layers of artificial neurons. Every single node is connected to all neurons in the next layer which makes it a fully connected neural network. Input and output layers are present having multiple hidden Layers i.e. at least three or more layers in total. It has a bi-directional propagation i.e. forward propagation and backward propagation, Inputs are multiplied with weights and fed to the activation function and in backpropagation, they are modified to reduce the loss. In simple words, weights are machine learnt values from Neural Networks. They self-adjust depending on the difference between predicted outputs vs training inputs. Nonlinear activation functions are used followed by softmax as an output layer activation function..

How would you tune the Training Algorithm Hyperparameters to get the highest accuracy in a Neural Network?          "Learning Rate: Setting the learning rate low will cause the learning process to slow down but converge smoothly. If the learning rate is high then the learning process will speed up but it might never converge.

Momentum: Momentum helps know the direction of the next step with the knowledge of the previous step. This will help prevent oscillations. Typically, momentum is chosen between 0.5 and 0.9.

Number of epochs: Increasing the number of epochs will cause the validation accuracy to decrease while increasing the training accuracy, this is due overfitting.

Batch size: This is the number of subsamples given to the network before which the parameter update happens. Typically, batch size is kept as 32, while different numbers such as 64, 128, 256 should also be tried."

What are some advantages of using Multilayer Perceptron over a Single-layer Perceptron?               "

A multilayer perceptron is a type of neural network which has many layers of perceptron stacked on top of each other.

Mathematically, multilayer perceptron are capable of learning any mapping function and have been proven to be a universal approximation algorithm.

Single layer perceptron only learn linear patterns, while multilayer perceptron can learn complex relationships. This predictive capability comes from the multi-layered structure of the network, so that the features can be combined into higher-order features."

Advantages and disadvantages of Convolution Neural Network:  Advantages of Convolution Neural Network: 1.       Used for deep learning with few parameters, 2.     Less parameters to learn as compared to fully connected layer, Disadvantages of Convolution Neural Network: ·        Comparatively complex to design and maintain, ·        Comparatively slow [depends on the number of hidden layers]

Radial Basis Function Neural Networks    Radial Basis Function Network consists of an input vector followed by a layer of RBF neurons and an output layer with one node per category. Classification is performed by measuring the input’s similarity to data points from the training set where each neuron stores a prototype. This will be one of the examples from the training set., When a new input vector [the n-dimensional vector that you are trying to classify] needs to be classified, each neuron calculates the Euclidean distance between the input and its prototype. For example, if we have two classes i.e. class A and Class B, then the new input to be classified is more close to class A prototypes than the class B prototypes. Hence, it could be tagged or classified as class A., Each RBF neuron compares the input vector to its prototype and outputs a value ranging which is a measure of similarity from 0 to 1. As the input equals to the prototype, the output of that RBF neuron will be 1 and with the distance grows between the input and prototype the response falls off exponentially towards 0. The curve generated out of neuron’s response tends towards a typical bell curve. The output layer consists of a set of neurons [one per category].

Applications of Recurrent Neural Networks          Applications of Recurrent Neural Networks, Designed to save the output of a layer, Recurrent Neural Network is fed back to the input to help in predicting the outcome of the layer. The first layer is typically a feed forward neural network followed by recurrent neural network layer where some information it had in the previous time-step is remembered by a memory function. Forward propagation is implemented in this case. It stores information required for it’s future use. If the prediction is wrong, the learning rate is employed to make small changes. Hence, making it gradually increase towards making the right prediction during the backpropagation.

Advantages and disadvantages of Recurrent Neural Networks      Advantages of Recurrent Neural Networks Model sequential data where each sample can be assumed to be dependent on historical ones is one of the advantage.   Disadvantages of Recurrent Neural Networks:  Gradient vanishing and exploding problems, Training recurrent neural nets could be a difficult task, Difficult to process long sequential data using ReLU as an activation function.     Used with convolution layers to extend the pixel effectiveness.

Applications of Modular Neural Network               Applications of Modular Neural Network Compression of high level input data: A modular neural network has a number of different networks that function independently and perform sub-tasks. The different networks do not really interact with or signal each other during the computation process. They work independently towards achieving the output, Adaptive MNN for character recognitions, Stock market prediction systems

Advantages and disadvantages of Modular Neural Network          Advantages of Modular Neural      Efficient,   Independent training,   Robustness, Network, Disadvantages of Modular Neural Network: Moving target Problems

The pros and cons of a typical RNN architecture  The pros and cons of a typical RNN architecture are summed up in the table below:, • Weights are shared across time, • Possibility of processing input of any length, • Model size not increasing with size of input, • Computation takes into account historical information

Neural Network Architecture Types          Neural Network Architecture Types: Neural Network is having two input units and one output unit with no hidden layers. These are also known as ‘single-layer perceptrons.' These networks are similar to the feed-forward Neural Network, except radial basis function is used as these neurons' activation function. These networks use more than one hidden layer of neurons, unlike single-layer perceptron. These are also known as Deep Feedforward Neural Networks. Type of Neural Network in which hidden layer neurons have self-connections. Recurrent Neural Networks possess memory. At any instance, the hidden layer neuron receives activation from the lower layer and its previous activation value. The type of Neural Network in which memory cell is incorporated into hidden layer neurons is called LSTM network. A fully interconnected network of neurons in which each neuron is connected to every other neuron. The network is trained with input patterns by setting a value of neurons to the desired pattern. Then its weights are computed. The weights are not changed. Once trained for one or more patterns, the network will converge to the learned patterns. It is different from other Neural Networks. These networks are similar to the Hopfield network, except some neurons are input, while others are hidden in nature. The weights are initialized randomly and learn through the backpropagation algorithm. ·        Convolutional Neural Network Get a complete overview of Convolutional Neural Networks through our blog Log Analytics with Machine Learning and Deep Learning. ·        Modular Neural Network It is the combined structure of different types of neural networks like multilayer perceptron, Hopfield Network, Recurrent Neural Network, etc., which are incorporated as a single module into the network to perform independent subtask of whole complete Neural Networks. ·        Physical Neural Network In this type of Artificial Neural Network, electrically adjustable resistance material is used to emulate synapse instead of software simulations performed in the neural network.

Pattern recognition         Pattern recognition is the study of how machines can observe the environment, learn to distinguish patterns of interest from their background, and make sound and reasonable decisions about the patterns' categories. Some examples of the pattern are - fingerprint images, a handwritten word, a human face, or a speech signal. Given an input pattern, its recognition involves the following task - Supervised classification - Given the input pattern is known as the member of a predefined class, Unsupervised classification - Assign pattern is to a hitherto unknown class, So, the recognition problem here is essentially a classification or categorized task. The design of pattern recognition systems usually involves the following three aspects- Data acquisition and preprocessing,

Difference between Neural Networks and Fuzzy Logic     In a hybrid (neuro-fuzzy) model, Neural Networks Learning Algorithms are fused with the fuzzy reasoning of fuzzy logic. The neural network determines the values of parameters, while if-then rules are controlled by fuzzy logic. Neural networks have been successfully applied to the broad spectrum of data-intensive applications, Neural Networks and Fuzzy Logic: Fuzzy logic refers to the logic developed to express the degree of truthiness by assigning values between 0 and 1, unlike traditional boolean logic representing 0 and 1. Fuzzy logic and Neural networks have one thing in common. They can be used to solve pattern recognition problems and others that do not involve any mathematical model. Systems combining both fuzzy logic and neural networks are neuro-fuzzy systems. These systems (Hybrid) can combine the advantages of both neural networks and fuzzy logic to perform in a better way. Fuzzy logic and Neural Networks have been integrated to use in the following applications -

What are the Advantages of Neural Networks?    ·        A neural network can perform tasks that a linear program can not, an element of the neural network fails, its parallel nature can continue without any problem,

What are the Limitations of Neural Networks?     Limitations of Neural NetworksThe neural network needs training to operate. architecture of a neural network is different from the architecture of microprocessors. Therefore, emulation is necessary.

Face Recognition Using Artificial Neural Networks             Face Recognition Using Artificial Neural Networks: Face recognition entails comparing an image with a database of saved faces to identify the person in that input picture. It is a mechanism that involves dividing images into two parts; one containing targets (faces) and one providing the background. The associated assignment of face detection has direct relevance to the fact that images need to analyze and faces identified earlier than they can be recognized.

What is the difference between Forward Propagation and Backward Propagation?               Forward propagation is the input data that is fed in the forward direction through the network. Each hidden layer accepts the input data, processes it as per the activation function, and passes it to the successive layer. Back propagation is the practice of fine-tuning the weights of the neural network based on the error rate obtained from the previous epoch. Proper tuning of the weights ensures low error rates, making the model more reliable.

How would you prevent Overfitting when designing an Artificial Neural Network?              A model with too little capacity cannot learn the problem, but a model with too much capacity will overfit the data. To prevent overfitting the data some things that can be done are: Training the model in more data. Giving the model an infinite number of examples will plateau the model in terms of what the capacity of the network is capable of learning. Changing the network structure (number of weights). Pruning by removing the nodes in the network will counteract the overfitting. Changing the network parameters (values of weights). Larger weights cause sharp transitions in the activation functions and cause large changes in output for small changes in inputs. This method to keep weights small is called Regularization.

How would you tune the Training Algorithm Hyperparameters to get the highest accuracy in a Neural Network?          Learning Rate: Setting the learning rate low will cause the learning process to slow down but converge smoothly. If the learning rate is high then the learning process will speed up but it might never converge. Momentum: Momentum helps know the direction of the next step with the knowledge of the previous step. This will help prevent oscillations. Typically, momentum is chosen between 0.5 and 0.9. Number of epochs: Increasing the number of epochs will cause the validation accuracy to decrease while increasing the training accuracy, this is due overfitting.

How to decide Batch size              Batch size: This is the number of subsamples given to the network before which the parameter update happens. Typically, batch size is kept as 32, while different numbers such as 64, 128, 256 should also be tried. Having Machine Learning, Data Science or Python Interview? Check 👉 132 Neural Networks Interview Questions

What are some advantages of using Multilayer Perceptron over a Single-layer Perceptron?               A multilayer perceptron is a type of neural network which has many layers of perceptron stacked on top of each other. Mathematically, multilayer perceptron are capable of learning any mapping function and have been proven to be a universal approximation algorithm. Single layer perceptron only learn linear patterns, while multilayer perceptron can learn complex relationships. This predictive capability comes from the multi-layered structure of the network, so that the features can be combined into higher-order features.

What are some criticisms of Neural Networks?    Neural networks require too much data to train. A classification network may require thousands of examples in a single class for it to identify it in unseen data. Due to this, sometimes it is not feasible to create ANN models for fringe applications. Neural networks are not interpretable. The user needs to input data into the network and it outputs the required output, but the work that goes into processing the input and giving an output is not understandable to human beings. The power required to train the neural network is extremely high compared to the amount of power that a human brain uses (around 20 Watts) to do almost the same things such as image classification.

What are some differences between SVMs and Neural Networks?             SVM has a number of parameters that increases linearly with the linear increase in the size of the input. All the parameters of the SVM interacts with one another. SVMs use only a subset of data for training. They can reliably define the decision boundary solely on the basis of the support vectors. As a result, they do not require much data for training. Neural Networks get trained based on the batches of data fed into it. This means that the specific decision boundary that the neural network learns is highly dependent on the order in which the batches of data are presented to it. This, in turn, requires processing the whole training dataset; or otherwise, the network may perform extremely poorly. Neural Networks have as many layers as possible, but due to the complex interaction between the model's parameters, NN always has a higher complexity when compared with SVM with a similar number of parameters.

What are some similarities between SVMs and Neural Networks?              Parametric: SVM and neural networks are both parametric but for different reasons. For SVM the typical parameters are; soft-margin parameter (C), parameter of the kernel function (gamma). Neural networks also have parameters but it is a lot more than SVM. Some NN parameters are the number of layers and their size, number of training epochs, and the learning rate. Embedding Non-Linearity: Both the methods can embed non-linear functions. SVM does this through the usage of kernel method. Neural Networks embed non-linearity using non-linear activation functions. Comparable Accuracy: If both SVM and Neural Networks are trained in the same dataset, given the same training time, and the same computation power they have comparable accuracy. If neural networks are given as much computation power and training time as possible then it outperforms SVMs.

What does 1x1 convolution mean in a Neural Network?  A 1x1 convolution simply maps an input pixel with all its channels to an output pixel, not looking at anything around itself. It is often used to reduce the number of depth channels since it is often very slow to multiply volumes with extremely large depths. input (256 depth) -> 1x1 convolution (64 depth) -> 4x4 convolution (256 depth) input (256 depth) -> 4x4 convolution (256 depth) The bottom one is about ~3.7x slower. Theoretically, the neural network can choose which input colors to look at using this, instead of brute force-multiplying everything.

What is the Vanishing Gradient Problem in Artificial Neural Networks?     The vanishing gradient problem is encountered in artificial neural networks with gradient-based learning methods and backpropagation. In these learning methods, each of the neural networks weights receives an update proportional to the partial derivative of the error function with respect to the current weight in each iteration of training. Sometimes when gradients become vanishingly small, this prevents the weight to change value. If the neural network has many hidden layers, the gradients in the earlier layers will become very low as we multiply the derivatives of each layer. As a result, learning in the earlier layers becomes very slow. This can cause the neural network to stop learning. This problem of vanishing gradient descent happens when training neural networks with many layers because the gradient diminishes dramatically as it propagates backwards through the network. Many fixes and workarounds have been proposed and investigated to fix the vanishing gradient problem, such as alternate weight initialization schemes, unsupervised pre-training, layer-wise training, and variations on gradient descent. Perhaps the most common change is the use of the rectified linear activation function that has become the new default, instead of the hyperbolic tangent activation function that was the default through the late 1990s and 2000s.

What happens when you trade the Breadth of a Neural Network for the Depth?  Networks with more layers (Depth) tend to require fewer units per layer because the composition functions created by successive layers make the neural network more powerful. The number of units in each layer of a deep network can be decreased to such an extent that when compared to a shallower network (more Breadth), the deep network has far fewer parameters even when added up over the greater number of layers. In neural networks with many layers the loss derivatives with respect to the weights in different layers of the network tend to have vastly different magnitudes, which causes challenges in properly choosing step sizes. This can lead to vanishing and exploding gradient problems.

Why does an artificial Neural Network use Backpropagation?       Backpropagation is an algorithm that computes the gradient of the loss function with respect to the weights of the network for a single input-output example. Backpropagation only refers to the algorithm used to compute the gradient, it does not include how the gradient is used. The formula for updating the weights and biases after calculating the gradient

Why should you Normalize the input for Neural Networks?           Normalizing the data to obtain a mean as close to 0 as possible is always a good thing to do to the input due to the following reasons: Standardization gives equal importance to every feature. In most of the networks, 'good' and 'bad' features can not be differentiated, so it is better to consider every feature equal by standardization. In neural networks, empirical evidence show that standardization improves the accuracy of the model. It improves the numerical stability of the model. This is because the data determines the update rate and a larger value will cause the gradient descent algorithm to overshoot the minimum and hence cause oscillations. It improves the training process. When different features have different ranges, then the learning rate depends on the feature with the largest range. When the input is standardized, the training process speeds up.

"Define artificial neural network (ANN)

"             "Artificial Neural Network is defined as information processing devices with the capability of

performing computations similar to human brain or biological neural network."

 List out the differences between artificial neural network and biological network.               "The differences between artificial neural network and biological network are

Artificial Neural Network:

i. Speed: Slow in processing information. (Processing time in the range of

nanoseconds.

ii. Processing: Sequential or step by step processing.

iii. Size and compatibility: Simple but cannot be used for complex pattern

recognition

Biological network:

i. Speed: faster in processing information.( processing time in the range of

milliseconds)

ii. Processing: Parallel processing.

iii. Size and compatibility: It can be used for complex pattern."

Define weight.   "

Weight is defined as information used by the neural net to solve a problem."

What are the classifications of activation function?           "

The classifications of activation function are

1. Identity function

2. Binary Step function 3. Sigmoidal function"

What are the types of Sigmoidal Function?           "

The types of Sigmoidal Function are

1. Binary Sigmoidal Function

2. Bipolar Sigmoidal Function"

What is the function of Synaptic gap?      "

The function Synaptic gap is used to convert the electrical signals to some

chemicals and these chemicals are again converted to electrical signals."

Define Learning.               "

Learning is defined as the process by which the free parameters of a neural network get

adapted through a process of stimulation by the environment in which the network

is embedded."

What is Back Propagation Network (BPN)?            "

It is a multi-layer forward network used extend gradient-descent waste delta

Learning rule."

What are the merits and demerits of Back Propagation Algorithm?            "

The merits and demerits of Back Propagation Algorithm are

Merits:

 

IC1403 PREPARED BY Dr. V.BALAJI, AP/EEE DHANALAKSHMI COLLEGE OF ENGG Page 3

1. The mathematical formula present here can be applied to any network and does

not require any special mention of the features of the function to be learnt.

2. The computing time is reduced if the weights chosen are small at the beginning.

Demerits:

1. The number of learning steps may be high, and also the learning phase has

intensive calculations.

2. The training may cause temporal instability to the system."

What are the four main steps in back propagation algorithm?      "

The four main steps in back propagation algorithm

1. Initialization of weights 2. Feed forward function

3. Back propagation 4. Termination"

What are feedback networks?     "

The Feedback networks, are one which can return back the output to the input, thereby

giving rise to an iteration process, are defined as feedback networks."

What is the main purpose of Hop field network? "

The main Purpose of Hop field network is able to recognize unclear pictures

correctly. However, only one picture can be stored at a time."

What is discrete Hop field net?   "

The discrete Hop field net is a fully interconnected neural net with each unit

connected to every other unit. The net has symmetric weights with no self-

connections i.e. all the diagonal elements of the weight matrix of a Hop field net are

zero"

List two difference between Hop field and iterative auto associative net. "

The two main differences between Hop field and iterative auto associative net are

that, in the Hop field net,

• Only one unit updates its activation at a time, and,

• Each unit continues to receive an external signal in addition to the signal from

the other units in the net."

What is continuous Hop field net?            "

The continuous Hop field net is a modified form of discrete Hop field net. It

uses continuous valued output functions, which can be used either for associative

memory problems or constrained optimization problems."

What is artificial neural network?              "

Artificial neural networks are non-linear information (signal) processing

devices, which are built from interconnected elementary processing devices called

neurons."

List the basic building blocks of artificial neural network. "

The basic building blocks of the artificial neural network are:

1. Network architecture.

2. Setting the weights.

3. Activation function."

Define neural network    "

The neural network is defined as an inter connection of neurons, such

that neuron outputs are connected, through weights, to all other neurons including

themselves; both lag-free and delay connections are allowed."

State the need for training the neural network    "

The need for training the neural network is to observe who would measure the cart position and angles

of pendulum at every instant is replaced by the input of the visual image of the cart and the pendulum.

Since it is essential to provide the controller with the present and most recent position and angle data,

the present and most recent images arranged in pairs have been used to train the network."

How ANN resembles brain?         "

The ANN resembles the brain in two respects:

1) Knowledge is acquired by the network through a learning process, and,

2) Inter-neuron connection strengths known as synaptic weights are used to store

the knowledge."

Learning Rules in Neural Network             "

The learning rule is a type of mathematical logic. It encourages a Neural Network to gain from the present conditions and upgrade its efficiency and performance. The learning procedure of the brain modifies its neural structure. The expanding or diminishing quality of its synaptic associations rely upon their activity. Learning rules in the Neural network:

 

Hebbian learning rule: It determines how to customize the weights of nodes of a system.

Perceptron learning rule: Network starts its learning by assigning a random value to each load.

Delta learning rule: Modification in a node's sympatric weight is equal to the multiplication of error and the input.

Correlation learning rule: It is similar to supervised learning."

What are the advantages and Disadvantages of Neural Networks?             "

The advantages of of Neural Networks are listed below:

 

A neural network can perform tasks that a linear program can not.

When an element of the neural network fails, its parallel nature can continue without any problem.

A neural network learns and reprogramming is not necessary.

It can be implemented in any application.

It can be performed without any problem.

The disadvantages of of Neural Networks are described below:

 

The neural network needs training to operate.

The architecture of a neural network is different from the architecture of microprocessors. Therefore, emulation is necessary.

Requires high processing time for large neural networks."

Classify neural network architecture.       Neural network architecture is classified as – single layer feed forward networks, multilayer feed forward networks and recurrent networks.

What are the benefits of neural network?             "Benefits are as follows-

 

Parallel computing, faster, robust on noisy training data, mapping capabilities."

List some commercial practical applications of ANN?        "

sales forecasting

industrial process control

customer research

data validation

risk management

target marketing"

What is a Multilayer Perceptron (MLP)?  "

 

A perceptron (a single neuron model) is always feedforward, that is, all the arrows are going in the direction of the output. In addition, it is assumed that in a perceptron, all the arrows are going from layer ‘i’ to layer ‘i+1’, and it is also usual (to start with having) that all the arcs from layer ‘i’ to ‘i+1’ are present.

 

Finally, having multiple layers means more than two layers, that is, you have hidden layers. It consists of 3 layers of nodes: a. Input, b. Hidden c. and output.

 

A perceptron is a network with two layers, one input and one output whereas a multilayered network means that you have at least one hidden layer (we call all the layers between the input and output layers hidden)."

What is an Input layer in a Neural Network?         "

Input layers are the training observations that are fed through the neurons."

What does a neuron compute?   "

It computes a linear function (z = Wx+b) followed by an activation function."

Describe the hidden layer in a Neural network?   "

These are the intermediate layers between input and output which help the neural networks to learn the complicated relationships involved in data."

What is a sigmoid function?         "

The sigmoid function is a logistic function bounded by 0 and. This is an activation function that is continuous and differentiable. It is nonlinear in nature that helps to increase the performance making sure that small changes in the weights and bias causes small changes in the output of the neuron."

Why RNN (recurrent neural networks) are used for machine translation, say English to French? "

It can be trained as a supervised learning problem

It is applicable when the input/output is a sequence. (i.e. a sequence of words)"

Why do we normalize the input x?           "

Normalizing the input x makes the count function faster to optimize."

What is Data Normalization and why do we need it?        "

Data normalization is very important preprocessing step, used to rescale values to fit in a specific range to assure better convergence during backpropagation. In general, it boils down to subtracting the mean of each data point, dividing by its standard deviation."

What is a representation learning algorithm?       "

Neural network converts data in such a form that it would be better to solve the desired problem and hence, called representation learning."

What is the role of Activation Functions in neural network?           "

The goal of an activation function is to introduce nonlinearity/a non-linear decision boundary via non-linear combinations of the weighted inputs into the neural network so that it can learn more complex function i.e. converts the processed input into an output called the activation value. Without it, the neural network would be only able to learn function which is a linear combination of its input data.

 

Some of the examples are – Step Function, ReLU, Tanh, Softmax."

What is a cost function? "

 It is a measure to evaluate how good your model's performance is. Also referred to as ‘ Loss/Error’

It is used to compute the error of the output layer during back propagation.

Mean squared error or Sum of squared errors is an example of the popular cost function.

C = ½ (y – y_hat)^2

 

Where c = cost function, y = Original output & y_hat = Predicted output."

What is Gradient Descent?           "

Gradient descent is an optimization algorithm to minimize the cost function in order to maximize the performance of the model. It aims to find the local or the global minima of the function. So, if your cost is a function of K variables, then the gradient is the length-K vector that defines the direction in which the cost is increasing most rapidly and you follow the negative of the gradient to the point where the cost is a minimum."

What are the steps involved in the backpropagation algorithm?   "

 

 

 Forward propagation of the training data through the network in order to generate the output.

 Compute the error derivative using the target value and the output value with respect to output activations.

 Then, we backpropagate to compute the derivative of the error with respect to the output activations in the previous layer.

 We continue the same for all the hidden layers.

 Use the previously calculated derivatives for output and all the hidden layers, to calculate the derivative of the error w.r.t weights.

 Update the weights."

What is the difference between Feedforward Neural Network and Recurrent Neural Network?            "

 

 

In feed forward neural networks –

 

The signal travels in one direction, from input to the output.

 

There are No feedbacks or loops

 

It considers only the current input

 

It cannot memorize the previous inputs. (eg. CNNs)

 

In Recurrent neural networks –

 

The signal travels in both the directions i.e. they can pass the information onto themselves.

 

It considers the current input along with the previously received inputs to generate the output of a layer."

What are Softmax and ReLU functions?   "

Softmax Function-

Softmax function calculates the probabilities distribution of the event over ‘n’ different events.

 

 

 

In general way of saying, this function will calculate the probabilities of each target class over all possible target classes. Later the calculated probabilities will be helpful for determining the target class for the given inputs.

 

So, softmax function squashes the outputs of each unit to be between 0 and 1, just like a sigmoid function. But it also divides each output such that the total sum of the outputs is equal to 1 (check it on the figure above) where an output is equivalent to a categorical probability distribution.

 

Mathematically the softmax function is shown to the right, where z is a vector of the inputs to the output layer (if you have 10 output units, then there are 10 elements in z). And again, j indexes the output units, so j = 1, 2, ..., K."

What are hyperparameters?        "

 

Hyperparams can’t learn from the data, they are set before the training phase such as –

 

A number of epochs – In the context of training the model, epoch is a term used to refer to one iteration where model sees the whole training set to update its-weights.

Batch Size – refers to the number of training examples in one forward/backward pass.

Learning rate – The learning rate, often noted as ‘alpha’, indicates at which pace the weight gets updated. It can be fixed or adaptively changed. The current most popular method is called ‘Adam’, a method that adapts the learning rate.  "

What is Vanishing gradient problem?       "

The problem of the vanishing gradient was first discovered by Sepp (Joseph) Hochreiterback in 1991.

 

Now that we know, information travels through time in RNNs, which means that information from previous time points is used as input for the next time points. Secondly, you can calculate the cost function, or your error, at each time point.

Basically, during the training, your cost function compares your outcomes to your desired output.

You’ve calculated the cost function, and now you want to propagate your cost function back through the network because you need to update the weights, meaning that you’ve to propagate all the way back through the time to these neurons.

This is where the problem lies, for any activation function, abs(dW) will get smaller and smaller as we go back with every layer during back propagation.

Since the weight update is minor and results in slower convergence. This makes the optimization of the loss function slow. In the worst case, this may completely stop the neural network from training further or may as well return half of the network not trained."

What are CNNs?              "

Convolutional Neural Networks are very similar to ordinary Neural Networks. They are made up of neurons that have learnable weights and biases. Each neuron receives some inputs, performs a dot product and optionally follows it with a non-linearity. The whole network still expresses a single differentiable score function: from the raw image pixels on one end to class scores at the other. And they also have a loss function (e.g. SVM/Softmax) on the last (fully-connected) layer.

 

The change is that ConvNet architectures make the explicit assumption that the inputs are images, which allows us to encode certain properties into the architecture. These then make the forward function more efficient to implement and vastly reduce the number of parameters in the network."

What are the different layers in CNN?     "Convolutional Layer – A layer that performs the convolutional operation on 2 functions to produce a third function that expresses how the shape of the one is modified by the another.

ReLU Layer – It brings non-linearity to the network and converts all the negative pixels to the zero, where your output is a rectified feature map.

Pooling layer – A down sampling operation that reduces the dimensionality of the feature map."

What are the 9 major types of neural networks? "The nine types of neural networks are:

 

Perceptron

Feed Forward Neural Network

Multilayer Perceptron

Convolutional Neural Network

Radial Basis Functional Neural Network

Recurrent Neural Network

LSTM – Long Short-Term Memory

Sequence to Sequence Models

Modular Neural Network"

What is neural network and its types?     Neural Networks are artificial networks used in Machine Learning that work in a similar fashion to the human nervous system. Many things are connected in various ways for a neural network to mimic and work like the human brain. Neural networks are basically used in computational models.

What is CNN and DNN?  "

 

A deep neural network (DNN) is an artificial neural network (ANN) with multiple layers between the input and output layers. They can model complex non-linear relationships. Convolutional Neural Networks (CNN) are an alternative type of DNN that allow modelling both time and space correlations in multivariate signals."

How does CNN differ from Ann? CNN is a specific kind of ANN that has one or more layers of convolutional units. The class of ANN covers several architectures including Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN) eg LSTM and GRU, Autoencoders, and Deep Belief Networks.

Why is CNN better than MLP?     Multilayer Perceptron (MLP) is great for MNIST as it is a simpler and more straight forward dataset, but it lags when it comes to real-world application in computer vision, specifically image classification as compared to CNN which is great.

How can Neural Networks be Unsupervised?        "

Neural Networks are used in unsupervised learning to learn better representations of the input data.

Neural networks can learn a mapping from document to real-valued vector in such a way that resulting vectors are similar for documents with similar content. This can be achieved using autoencoders which is a model that is trained to reconstruct the original vector from a smaller representation with reconstruction error as a cost function.

There are neural networks that are specifically designed for clustering as well. The most widely known is the self-organizing maps (SOM)."

What is an Activation Function?  "

An activation function applies a step rule (convert the numerical output into +1 or -1) to check if the output of the weighting function is greater than zero or not.

An activation function of a node defines the output of the node given an input or set of inputs to the node.

Activation functions can be divided into three categories:

Ridge functions

Radial functions, and

Fold functions.

A type of ridge function called Rectified Linear Function (ReLU) is shown below:"

What is the difference between Deep Learning and Artificial Neural Networks?    "

When researchers started to create large artificial neural networks, they started to use the word deep to refer to them.

As the term deep learning started to be used, it is generally understood that it stands for artificial neural networks which are deep as opposed to shallow artificial neural networks.

Deep Artificial Neural Networks and Deep Learning are generally the same thing and mostly used interchangeably."

What is the difference between Forward Propagation and Backward Propagation?               "Forward propagation is the input data that is fed in the forward direction through the network. Each hidden layer accepts the input data, processes it as per the activation function, and passes it to the successive layer.

Back propagation is the practice of fine-tuning the weights of the neural network based on the error rate obtained from the previous epoch. Proper tuning of the weights ensures low error rates, making the model more reliable."

 Explain why the Initialization process of weights and bias is important for NN?    "The initialization step can be critical to the model's performance, and it requires the right method.

 

Initializing the weights to zero leads the network to learn zero output which makes the network not learn anything.

Initializing the weights to be too large causes the network to experience exploding gradients.

Initializing the weights to be too small causes the network to experience vanishing gradients.

To find the perfect initialization, there are a few rules of thumb to follow:

 

The mean of activations should be zero.

The variance of activations should stay the same across every layer."

How to choose the features for a Neural Network?           "

A very strong correlation between the new feature and an existing feature is a fairly good sign that the new feature provides little new information.

A low correlation between the new feature and existing features is likely preferable.

 

A strong linear correlation between the new feature and the predicted variable is a good sign that a new feature will be valuable, but the absence of a high correlation is not necessarily a sign of a poor feature, because neural networks are not restricted to linear combinations of variables.

 

If the new feature was manually constructed from a combination of existing features, consider leaving it out. The beauty of neural networks is that little feature engineering and preprocessing are required - features are instead learned by intermediate layers.

 

Whenever possible, prefer learning features to engineering them."

How would you tune the Training Algorithm Hyperparameters to get the highest accuracy in a Neural Network?          "Learning Rate: Setting the learning rate low will cause the learning process to slow down but converge smoothly. If the learning rate is high then the learning process will speed up but it might never converge.

Momentum: Momentum helps know the direction of the next step with the knowledge of the previous step. This will help prevent oscillations. Typically, momentum is chosen between 0.5 and 0.9.

Number of epochs: Increasing the number of epochs will cause the validation accuracy to decrease while increasing the training accuracy, this is due overfitting.

Batch size: This is the number of subsamples given to the network before which the parameter update happens. Typically, batch size is kept as 32, while different numbers such as 64, 128, 256 should also be tried."

What are some advantages of using Multilayer Perceptron over a Single-layer Perceptron?               "

A multilayer perceptron is a type of neural network which has many layers of perceptron stacked on top of each other.

Mathematically, multilayer perceptron are capable of learning any mapping function and have been proven to be a universal approximation algorithm.

Single layer perceptron only learn linear patterns, while multilayer perceptron can learn complex relationships. This predictive capability comes from the multi-layered structure of the network, so that the features can be combined into higher-order features."