ó ýÍSc@s˝dZdZddlZddlZddlZddlZddlZddlZddlZddl j Z de fd„ƒYZ d„Zddd d d „Zed kršeƒndS( s This tutorial introduces logistic regression using Theano and stochastic gradient descent. Logistic regression is a probabilistic, linear classifier. It is parametrized by a weight matrix :math:`W` and a bias vector :math:`b`. Classification is done by projecting data points onto a set of hyperplanes, the distance to which is used to determine a class membership probability. Mathematically, this can be written as: .. math:: P(Y=i|x, W,b) &= softmax_i(W x + b) \ &= rac {e^{W_i x + b_i}} {\sum_j e^{W_j x + b_j}} The output of the model or prediction is then done by taking the argmax of the vector whose i'th element is P(Y=i|x). .. math:: y_{pred} = argmax_i P(Y=i|x,W,b) This tutorial presents a stochastic gradient descent optimization method suitable for large datasets, and a conjugate gradient optimization method that is suitable for smaller datasets. References: - textbooks: "Pattern Recognition and Machine Learning" - Christopher M. Bishop, section 4.3.2 srestructedtext eni˙˙˙˙NtLogisticRegressioncBs)eZdZd„Zd„Zd„ZRS(s5Multi-class Logistic Regression Class The logistic regression is fully described by a weight matrix :math:`W` and bias vector :math:`b`. Classification is done by projecting data points onto a set of hyperplanes, the distance to which is used to determine a class membership probability. cCsÔtjdtj||fdtjjƒdddtƒ|_tjdtj|fdtjjƒdddtƒ|_t j j t j ||jƒ|jƒ|_ t j|j ddƒ|_|j|jg|_d S( s  Initialize the parameters of the logistic regression :type input: theano.tensor.TensorType :param input: symbolic variable that describes the input of the architecture (one minibatch) :type n_in: int :param n_in: number of input units, the dimension of the space in which the datapoints lie :type n_out: int :param n_out: number of output units, the dimension of the space in which the labels lie tvaluetdtypetnametWtborrowtbtaxisiN(ttheanotsharedtnumpytzerostconfigtfloatXtTrueRRtTtnnettsoftmaxtdott p_y_given_xtargmaxty_predtparams(tselftinputtn_intn_out((sK/net/nas-lsis-3/SABIOD/METHODES/NICOLAS/CNN2D_LIFEBIRD/CODE/logistic_sgd.pyt__init__;s+cCs4tjtj|jƒtj|jdƒ|fƒ S(s‰Return the mean of the negative log-likelihood of the prediction of this model under a given target distribution. .. math:: rac{1}{|\mathcal{D}|} \mathcal{L} ( heta=\{W,b\}, \mathcal{D}) = rac{1}{|\mathcal{D}|} \sum_{i=0}^{|\mathcal{D}|} \log(P(Y=y^{(i)}|x^{(i)}, W,b)) \ \ell ( heta=\{W,b\}, \mathcal{D}) :type y: theano.tensor.TensorType :param y: corresponds to a vector that gives for each example the correct label Note: we use the mean instead of the sum so that the learning rate is less dependent on the batch size i(RtmeantlogRtarangetshape(Rty((sK/net/nas-lsis-3/SABIOD/METHODES/NICOLAS/CNN2D_LIFEBIRD/CODE/logistic_sgd.pytnegative_log_likelihood_scCsw|j|jjkr<tddtjd|jjfƒ‚n|jjdƒrjtjtj |j|ƒƒSt ƒ‚dS(sQReturn a float representing the number of errors in the minibatch over the total number of examples of the minibatch ; zero one loss over the size of the minibatch :type y: theano.tensor.TensorType :param y: corresponds to a vector that gives for each example the correct label s+y should have the same shape as self.y_predR RtintN( tndimRt TypeErrorttargetttypeRt startswithRRtneqtNotImplementedError(RR ((sK/net/nas-lsis-3/SABIOD/METHODES/NICOLAS/CNN2D_LIFEBIRD/CODE/logistic_sgd.pyterrors|s !(t__name__t __module__t__doc__RR!R*(((sK/net/nas-lsis-3/SABIOD/METHODES/NICOLAS/CNN2D_LIFEBIRD/CODE/logistic_sgd.pyR2s $ cCsytjj|ƒ\}}|dkrŒtjj|ƒ rŒtjjtjjtƒddd|ƒ}tjj|ƒs€|dkrŒ|}qŒntjj|ƒ rŮ|dkrŮddl}d}d |GH|j||ƒnd GHtj |d ƒ}t j |ƒ\}}} |j ƒt d „} | | ƒ\} } | |ƒ\} }| |ƒ\}}||f| |f| | fg}|S( sk Loads the dataset :type dataset: string :param dataset: the path to the dataset (here MNIST) tis..tdatas mnist.pkl.gzi˙˙˙˙Ns>http://www.iro.umontreal.ca/~lisa/deep/data/mnist/mnist.pkl.gzsDownloading data from %ss... loading datatrbcSsv|\}}tjtj|dtjjƒd|ƒ}tjtj|dtjjƒd|ƒ}|tj|dƒfS(sŠ Function that loads the dataset into shared variables The reason we store our dataset in shared variables is to allow Theano to copy it into the GPU memory (when code is run on GPU). Since copying data into the GPU is slow, copying a minibatch everytime is needed (the default behaviour if the data is not in a shared variable) would lead to a large decrease in performance. RRtint32(RR R tasarrayR R Rtcast(tdata_xyRtdata_xtdata_ytshared_xtshared_y((sK/net/nas-lsis-3/SABIOD/METHODES/NICOLAS/CNN2D_LIFEBIRD/CODE/logistic_sgd.pytshared_datasetšs   (tostpathtsplittisfiletjoint__file__turllibt urlretrievetgziptopentcPickletloadtcloseR(tdatasettdata_dirt data_filetnew_pathR@torigintft train_sett valid_setttest_setR9t test_set_xt test_set_yt valid_set_xt valid_set_yt train_set_xt train_set_ytrval((sK/net/nas-lsis-3/SABIOD/METHODES/NICOLAS/CNN2D_LIFEBIRD/CODE/logistic_sgd.pyt load_data“s* +     g¤p= ףŔ?ičs mnist.pkl.gziXc+ Cs–t|ƒ}|d\}}|d\}}|d\} } |jdtƒjd|} |jdtƒjd|} | jdtƒjd|} dGHtjƒ}tjdƒ}tjdƒ}td|d d d d ƒ}|j |ƒ}t j d |gd|j |ƒdi| |||d|!|6| |||d|!|6ƒ}t j d |gd|j |ƒdi||||d|!|6||||d|!|6ƒ}tj d|d|jƒ}tj d|d|jƒ}|j|j||f|j|j||fg}t j d |gd|d|di||||d|!|6||||d|!|6ƒ}dGHd}d}d}t| |dƒ}d}tj}d}tjƒ} t}!d}"x`|"|kr&|! r&|"d}"x<t| ƒD].}#||#ƒ}$|"d| |#}%|%d|dkr gt| ƒD]}&||&ƒ^q6}'tj|'ƒ}(d|"|#d| |(dfGH|(|kr |(||krŹt||%|ƒ}n|(}gt| ƒD]}&||&ƒ^qż})tj|)ƒ}d|"|#d| |dfGHq n||%krńt}!PqńqńWqÇWtjƒ}*d|d|dfGHd|"d|"|*| fGHtjdtjjtƒdd|*| IJdS(!s  Demonstrate stochastic gradient descent optimization of a log-linear model This is demonstrated on MNIST. :type learning_rate: float :param learning_rate: learning rate used (factor for the stochastic gradient) :type n_epochs: int :param n_epochs: maximal number of epochs to run the optimizer :type dataset: string :param dataset: the path of the MNIST dataset file from http://www.iro.umontreal.ca/~lisa/deep/data/mnist/mnist.pkl.gz iiiRs... building the modeltxR RRiRi tinputstoutputstgivenstcosttwrttupdatess... training the modeliˆgףp= ×ď?gs1epoch %i, minibatch %i/%i, validation error %f %%gY@s> epoch %i, minibatch %i/%i, test error of best model %f %%sUOptimization complete with best validation score of %f %%,with test performance %f %%s.The code run for %d epochs, with %f epochs/secgđ?sThe code for file s ran for %.1fsNi( RWt get_valueRRRtlscalartmatrixtivectorRR!RtfunctionR*tgradRRtmintNoneR tinfttimetclocktFalsetxrangeRtmaxtsyststderrR:R;R<R?(+t learning_ratetn_epochsRGt batch_sizetdatasetsRTRURRRSRPRQtn_train_batchestn_valid_batchestn_test_batchestindexRXR t classifierR\t test_modeltvalidate_modeltg_Wtg_bR^t train_modeltpatiencetpatience_increasetimprovement_thresholdtvalidation_frequencyt best_paramstbest_validation_losst test_scoret start_timet done_loopingtepochtminibatch_indextminibatch_avg_costtitertitvalidation_lossestthis_validation_losst test_lossestend_time((sK/net/nas-lsis-3/SABIOD/METHODES/NICOLAS/CNN2D_LIFEBIRD/CODE/logistic_sgd.pytsgd_optimization_mnistŰsœ         "    "    t__main__(R-t __docformat__RDRBR:RmRhR Rt theano.tensorttensorRtobjectRRWRR+(((sK/net/nas-lsis-3/SABIOD/METHODES/NICOLAS/CNN2D_LIFEBIRD/CODE/logistic_sgd.pyt#s        a H œ