ó þÍSc@sédZdZddlZddlZddlZddlZddlZddlZddlZddl j Z ddl m Z mZdefd„ƒYZdefd„ƒYZd d d d d ddd„ZedkråeƒndS(sƒ This tutorial introduces the multilayer perceptron using Theano. A multilayer perceptron is a logistic regressor where instead of feeding the input to the logistic regression you insert a intermediate layer, called the hidden layer, that has a nonlinear activation function (usually tanh or sigmoid) . One can use many such hidden layers making the architecture deep. The tutorial will also tackle the problem of MNIST digit classification. .. math:: f(x) = G( b^{(2)} + W^{(2)}( s( b^{(1)} + W^{(1)} x))), References: - textbooks: "Pattern Recognition and Machine Learning" - Christopher M. Bishop, section 5 srestructedtext eniÿÿÿÿN(tLogisticRegressiont load_datat HiddenLayercBseZddejd„ZRS(c Csg||_|d kr´tj|jdtjd||ƒ dtjd||ƒd||fƒdtjjƒ}|tj j j kr“|d9}ntj d|dd d t ƒ}n|d krÿtj|fdtjjƒ} tj d| dd d t ƒ}n||_||_tj||jƒ|j} |d kr?| n || ƒ|_|j|jg|_d S( sX Typical hidden layer of a MLP: units are fully-connected and have sigmoidal activation function. Weight matrix W is of shape (n_in,n_out) and the bias vector b is of shape (n_out,). NOTE : The nonlinearity used here is tanh Hidden unit activation is given by: tanh(dot(input,W) + b) :type rng: numpy.random.RandomState :param rng: a random number generator used to initialize weights :type input: theano.tensor.dmatrix :param input: a symbolic tensor of shape (n_examples, n_in) :type n_in: int :param n_in: dimensionality of input :type n_out: int :param n_out: number of hidden units :type activation: theano.Op or function :param activation: Non linearity to be applied in the hidden layer tlowg@thightsizetdtypeitvaluetnametWtborrowtbN(tinputtNonetnumpytasarraytuniformtsqrtttheanotconfigtfloatXttensortnnettsigmoidtsharedtTruetzerosR R tTtdottoutputtparams( tselftrngR tn_intn_outR R t activationtW_valuestb_valuest lin_output((sB/net/nas-lsis-3/SABIOD/METHODES/NICOLAS/CNN2D_LIFEBIRD/CODE/mlp.pyt__init__(s$   ! !  N(t__name__t __module__R RttanhR'(((sB/net/nas-lsis-3/SABIOD/METHODES/NICOLAS/CNN2D_LIFEBIRD/CODE/mlp.pyR'stMLPcBseZdZd„ZRS(s¢Multi-Layer Perceptron Class A multilayer perceptron is a feedforward artificial neural network model that has one layer or more of hidden units and nonlinear activations. Intermediate layers usually have as activation function tanh or the sigmoid function (defined here by a ``HiddenLayer`` class) while the top layer is a softamx layer (defined here by a ``LogisticRegression`` class). c Csêtd|d|d|d|dtjƒ|_td|jjd|d|ƒ|_t|jjƒj ƒt|jjƒj ƒ|_ |jjdj ƒ|jjdj ƒ|_ |jj |_ |jj |_ |jj|jj|_dS(s§Initialize the parameters for the multilayer perceptron :type rng: numpy.random.RandomState :param rng: a random number generator used to initialize weights :type input: theano.tensor.TensorType :param input: symbolic variable that describes the input of the architecture (one minibatch) :type n_in: int :param n_in: number of input units, the dimension of the space in which the datapoints lie :type n_hidden: int :param n_hidden: number of hidden units :type n_out: int :param n_out: number of output units, the dimension of the space in which the labels lie R R R!R"R#iN(RRR*t hiddenLayerRRtlogRegressionLayertabsR tsumtL1tL2_sqrtnegative_log_likelihoodterrorsR(RR R R!tn_hiddenR"((sB/net/nas-lsis-3/SABIOD/METHODES/NICOLAS/CNN2D_LIFEBIRD/CODE/mlp.pyR'ts   (R(R)t__doc__R'(((sB/net/nas-lsis-3/SABIOD/METHODES/NICOLAS/CNN2D_LIFEBIRD/CODE/mlp.pyR+is g{®Gáz„?gg-Cëâ6?iès mnist.pkl.gziiôc1 CsÙt|ƒ}|d\}} |d\} } |d\} } |jdtƒjd|}| jdtƒjd|}| jdtƒjd|}dGHtjƒ}tjdƒ}tjdƒ}tj j dƒ}t d |d |d d d |ddƒ}|j |ƒ||j ||j}tjd|gd|j|ƒdi| |||d|!|6| |||d|!|6ƒ}tjd|gd|j|ƒdi| |||d|!|6| |||d|!|6ƒ}g}x0|jD]%}tj||ƒ}|j|ƒqìWg}x;t|j|ƒD]'\}}|j||||fƒq.Wtjd|gd|d|di||||d|!|6| |||d|!|6ƒ}dGHd}d}d} t||dƒ}!d}"tj}#d}$d}%tjƒ}&d}'t}(xf|'|kry|( ry|'d}'xBt|ƒD]4})||)ƒ}*|'d||)}+|+d|!dkr\gt|ƒD]},||,ƒ^qƒ}-tj|-ƒ}.d|'|)d||.dfGH|.|#kr\|.|#| krùt||+|ƒ}n|.}#|+}$gt|ƒD]},||,ƒ^q}/tj|/ƒ}%d|'|)d||%dfGHq\n||+kr>t}(Pq>q>WqWtjƒ}0d|#d|$d|%dfGHtj dt!j"j#t$ƒdd|0|&dIJdS(!sÕ Demonstrate stochastic gradient descent optimization for a multilayer perceptron This is demonstrated on MNIST. :type learning_rate: float :param learning_rate: learning rate used (factor for the stochastic gradient :type L1_reg: float :param L1_reg: L1-norm's weight when added to the cost (see regularization) :type L2_reg: float :param L2_reg: L2-norm's weight when added to the cost (see regularization) :type n_epochs: int :param n_epochs: maximal number of epochs to run the optimizer :type dataset: string :param dataset: the path of the MNIST dataset file from http://www.iro.umontreal.ca/~lisa/deep/data/mnist/mnist.pkl.gz iiiR s... building the modeltxtyiÒR R R!iR4R"i tinputstoutputstgivenstupdatess ... trainingi'g×£p= ×ï?gs1epoch %i, minibatch %i/%i, validation error %f %%gY@s> epoch %i, minibatch %i/%i, test error of best model %f %%skOptimization complete. Best validation score of %f %% obtained at iteration %i, with test performance %f %%sThe code for file s ran for %.2fmgN@Ni(%Rt get_valueRtshapeRtlscalartmatrixtivectorRtrandomt RandomStateR+R2R0R1RtfunctionR3RtgradtappendtziptminR tinfttimetclocktFalsetxrangetmeantmaxtsyststderrtostpathtsplitt__file__(1t learning_ratetL1_regtL2_regtn_epochstdatasett batch_sizeR4tdatasetst train_set_xt train_set_yt valid_set_xt valid_set_yt test_set_xt test_set_ytn_train_batchestn_valid_batchestn_test_batchestindexR6R7R t classifiertcostt test_modeltvalidate_modeltgparamstparamtgparamR;t train_modeltpatiencetpatience_increasetimprovement_thresholdtvalidation_frequencyt best_paramstbest_validation_losst best_itert test_scoret start_timetepocht done_loopingtminibatch_indextminibatch_avg_costtitertitvalidation_lossestthis_validation_losst test_lossestend_time((sB/net/nas-lsis-3/SABIOD/METHODES/NICOLAS/CNN2D_LIFEBIRD/CODE/mlp.pyttest_mlp°s¤  %       "    "    t__main__(R5t __docformat__tcPickletgzipRQRORIRRt theano.tensorRRt logistic_sgdRRtobjectRR+RR((((sB/net/nas-lsis-3/SABIOD/METHODES/NICOLAS/CNN2D_LIFEBIRD/CODE/mlp.pyts        BG ´