+7

mini_proj/report/references.bib

··· 1 1 + @misc{openData, 2 2 + title={Open Database License (ODbL) v1.0}, 3 3 + url={https://opendatacommons.org/licenses/odbl/1.0/}, 4 4 + journal={Open Data Commons}, 5 5 + year={2018}, 6 6 + month={Feb} 7 7 + } 1 8 @techreport{knn, 2 9 title={Discriminatory analysis-nonparametric discrimination: consistency properties}, 3 10 author={Fix, Evelyn and Hodges Jr, Joseph L},

+95 -45

mini_proj/report/waldo.tex

··· 6 6 \usepackage[justification=centering]{caption} % Used for captions 7 7 \captionsetup[figure]{font=small} % Makes captions small 8 8 \newcommand\tab[1][0.5cm]{\hspace*{#1}} % Defines a new command to use 'tab' in text 9 9 - % Math package 10 10 - \usepackage{amsmath} 9 9 + \usepackage[comma, numbers]{natbib} % Used for the bibliography 10 10 + \usepackage{amsmath} % Math package 11 11 % Enable that parameters of \cref{}, \ref{}, \cite{}, ... are linked so that a reader can click on the number an jump to the target in the document 12 12 \usepackage{hyperref} 13 13 %enable \cref{...} and \Cref{...} instead of \ref: Type of reference included in the link ··· 99 99 architectures, as this method is currently the most used for image 100 100 classification. 101 101 102 102 + \textbf{ 103 103 + \\A couple of papers that may be useful (if needed): 104 104 + - LeNet: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf 105 105 + - AlexNet: http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks 106 106 + - General comparison of LeNet and AlexNet: 107 107 + "On the Performance of GoogLeNet and AlexNet Applied to Sketches", Pedro Ballester and Ricardo Matsumura Araujo 108 108 + - Deep NN Architecture: 109 109 + https://www-sciencedirect-com.ezproxy.lib.monash.edu.au/science/article/pii/S0925231216315533 110 110 + } 111 111 + 102 112 \subsection{Classical Machine Learning Methods} 103 113 104 114 The following paragraphs will give only brief descriptions of the different ··· 130 140 131 141 \subsection{Neural Network Architectures} 132 142 \todo{Did we only do the three in the end? (Alexnet?)} 143 143 + Yeah, we implemented the LeNet architecture, then improved on it for a fairly standar convolutional neural network (CNN) that was deeper, extracted more features, and condensed that image information more. Then we implemented a more fully convolutional network (FCN) which contained only one dense layer for the final binary classification step. The FCN added an extra convolutional layer, meaning the before classifying each image, the network abstracted the data more than the other two. 144 144 + \begin{itemize} 145 145 + \item LeNet 146 146 + \item CNN 147 147 + \item FCN 148 148 + \end{itemize} 133 149 134 150 \paragraph{Convolutional Neural Networks} 135 151 ··· 139 155 140 156 141 157 \section{Method} \label{sec:method} 158 158 + \tab 159 159 + In order to effectively utilize the aforementioned modelling and classification techniques, a key consideration is the data they are acting on. 160 160 + A dataset containing Waldo and non-Waldo images was obtained from an Open Database\footnote{``The Open Database License (ODbL) is a license agreement intended to allow users to freely share, modify, and use [a] Database while maintaining [the] same freedom for others"\cite{openData}}hosted on the predictive modelling and analytics competition framework, Kaggle. 161 161 + The distinction between images containing Waldo, and those that do not, was providied by the separation of the images in different sub-directories. 162 162 + It was therefore necessary to preprocess these images before they could be utilised by the proposed machine learning algorithms. 163 163 + 164 164 + \subsection{Image Processing} 165 165 + \tab 166 166 + The Waldo image database consists of images of size 64$\times$64, 128$\times$128, and 256$\times$256 pixels obtained by dividing complete Where's Waldo? puzzles. 167 167 + Within each set of images, those containing Waldo are located in a folder called `waldo', and those not containing Waldo, in a folder called `not\_waldo'. 168 168 + Since Where's Waldo? puzzles are usually densely populated and contain fine details, the 64$\times$64 pixel set of images were selected to train and evaluate the machine learning models. 169 169 + These images provide the added benefit of containing the most individual images of the three size groups. 170 170 + \\ 171 171 + \par 172 172 + Each of the 64$\times$64 pixel images were inserted into a Numpy 173 173 + \footnote{Numpy is a popular Python programming library for scientific computing} 174 174 + array of images, and a binary value was inserted into a seperate list at the same index. 175 175 + These binary values form the labels for each image (waldo or not waldo). 176 176 + Colour normalisation was performed on each so that artefacts in an image's colour profile correspond to meaningful features of the image (rather than photographic method). 177 177 + \\ 178 178 + \par 179 179 + Each original puzzle is broken down into many images, and only contains one Waldo. Although Waldo might span multiple 64$\times$64 pixel squares, this means that the non-Waldo data far outnumbers the Waldo data. 180 180 + To combat the bias introduced by the skewed data, all Waldo images were artificially augmented by performing random rotations, reflections, and introducing random noise in the image to produce news images. 181 181 + In this way, each original Waldo image was used to produce an additional 10 variations of the image, inserted into the image array. 182 182 + This provided more variation in the true positives of the data set and assists in the development of more robust methods by exposing each technique to variations of the image during the training phase. 183 183 + \\ 184 184 + \par 185 185 + Despite the additional data, there were still over ten times as many non-Waldo images than Waldo images. 186 186 + Therefore, it was necessary to cull the no-Waldo data, so that there was an even split of Waldo and non-Waldo images, improving the representation of true positives in the image data set. 187 187 + \\ 142 188 143 143 - % Kelvin Start 144 144 - \subsection{Benchmarking}\label{benchmarking} 189 189 + % Kelvin Start 190 190 + \subsection{Benchmarking}\label{benchmarking} 145 191 146 146 - In order to benchmark the Neural Networks, the performance of these 147 147 - algorithms are evaluated against other Machine Learning algorithms. We 148 148 - use Support Vector Machines, K-Nearest Neighbours (\(K=5\)), Gaussian 149 149 - Naive Bayes and Random Forest classifiers, as provided in Scikit-Learn. 192 192 + In order to benchmark the Neural Networks, the performance of these 193 193 + algorithms are evaluated against other Machine Learning algorithms. We 194 194 + use Support Vector Machines, K-Nearest Neighbours (\(K=5\)), Gaussian 195 195 + Naive Bayes and Random Forest classifiers, as provided in Scikit-Learn. 150 196 151 151 - \subsection{Performance Metrics}\label{performance-metrics} 197 197 + \subsection{Performance Metrics}\label{performance-metrics} 152 198 153 153 - To evaluate the performance of the models, we record the time taken by 154 154 - each model to train, based on the training data and statistics about the 155 155 - predictions the models make on the test data. These prediction 156 156 - statistics include: 199 199 + To evaluate the performance of the models, we record the time taken by 200 200 + each model to train, based on the training data and statistics about the 201 201 + predictions the models make on the test data. These prediction 202 202 + statistics include: 157 203 158 158 - \begin{itemize} 159 159 - \item 160 160 - \textbf{Accuracy:} 161 161 - \[a = \dfrac{|correct\ predictions|}{|predictions|} = \dfrac{tp + tn}{tp + tn + fp + fn}\] 162 162 - \item 163 163 - \textbf{Precision:} 164 164 - \[p = \dfrac{|Waldo\ predicted\ as\ Waldo|}{|predicted\ as\ Waldo|} = \dfrac{tp}{tp + fp}\] 165 165 - \item 166 166 - \textbf{Recall:} 167 167 - \[r = \dfrac{|Waldo\ predicted\ as\ Waldo|}{|actually\ Waldo|} = \dfrac{tp}{tp + fn}\] 168 168 - \item 169 169 - \textbf{F1 Measure:} \[f1 = \dfrac{2pr}{p + r}\] where \(tp\) is the 170 170 - number of true positives, \(tn\) is the number of true negatives, 171 171 - \(fp\) is the number of false positives, and \(tp\) is the number of 172 172 - false negatives. 173 173 - \end{itemize} 204 204 + \begin{itemize} 205 205 + \item 206 206 + \textbf{Accuracy:} 207 207 + \[a = \dfrac{|correct\ predictions|}{|predictions|} = \dfrac{tp + tn}{tp + tn + fp + fn}\] 208 208 + \item 209 209 + \textbf{Precision:} 210 210 + \[p = \dfrac{|Waldo\ predicted\ as\ Waldo|}{|predicted\ as\ Waldo|} = \dfrac{tp}{tp + fp}\] 211 211 + \item 212 212 + \textbf{Recall:} 213 213 + \[r = \dfrac{|Waldo\ predicted\ as\ Waldo|}{|actually\ Waldo|} = \dfrac{tp}{tp + fn}\] 214 214 + \item 215 215 + \textbf{F1 Measure:} \[f1 = \dfrac{2pr}{p + r}\] where \(tp\) is the 216 216 + number of true positives, \(tn\) is the number of true negatives, 217 217 + \(fp\) is the number of false positives, and \(tp\) is the number of 218 218 + false negatives. 219 219 + \end{itemize} 174 220 175 175 - Accuracy is a common performance metric used in Machine Learning, 176 176 - however in classification problems where the training data is heavily 177 177 - biased toward one category, sometimes a model will learn to optimize its 178 178 - accuracy by classifying all instances as one category. I.e. the 179 179 - classifier will classify all images that do not contain Waldo as not 180 180 - containing Waldo, but will also classify all images containing Waldo as 181 181 - not containing Waldo. Thus we use, other metrics to measure performance 182 182 - as well. 221 221 + Accuracy is a common performance metric used in Machine Learning, 222 222 + however in classification problems where the training data is heavily 223 223 + biased toward one category, sometimes a model will learn to optimize its 224 224 + accuracy by classifying all instances as one category. I.e. the 225 225 + classifier will classify all images that do not contain Waldo as not 226 226 + containing Waldo, but will also classify all images containing Waldo as 227 227 + not containing Waldo. Thus we use, other metrics to measure performance 228 228 + as well. 183 229 184 184 - \emph{Precision} returns the percentage of classifications of Waldo that 185 185 - are actually Waldo. \emph{Recall} returns the percentage of Waldos that 186 186 - were actually predicted as Waldo. In the case of a classifier that 187 187 - classifies all things as Waldo, the recall would be 0. \emph{F1-Measure} 188 188 - returns a combination of precision and recall that heavily penalises 189 189 - classifiers that perform poorly in either precision or recall. 190 190 - % Kelvin End 230 230 + \emph{Precision} returns the percentage of classifications of Waldo that 231 231 + are actually Waldo. \emph{Recall} returns the percentage of Waldos that 232 232 + were actually predicted as Waldo. In the case of a classifier that 233 233 + classifies all things as Waldo, the recall would be 0. \emph{F1-Measure} 234 234 + returns a combination of precision and recall that heavily penalises 235 235 + classifiers that perform poorly in either precision or recall. 236 236 + % Kelvin End 191 237 192 238 \section{Results} \label{sec:results} 193 239 194 240 \section{Conclusion} \label{sec:conclusion} 195 241 242 242 + \clearpage % Ensures that the references are on a seperate page 243 243 + \pagebreak 244 244 + % References 245 245 + \section{References} 246 246 + \renewcommand{\refname}{} 196 247 \bibliographystyle{alpha} 197 248 \bibliography{references} 198 198 - 199 249 \end{document}

+73 -4

mini_proj/waldo_model.py

··· 25 25 ''' 26 26 Model definition define the network structure 27 27 ''' 28 28 - def FCN(): 28 28 + def CNN(): 29 29 ## List of model layers 30 30 inputs = Input((3, 64, 64)) 31 31 ··· 33 33 m_pool1 = MaxPooling2D(pool_size=(2, 2))(conv1) 34 34 35 35 conv2 = Conv2D(32, (3, 3), activation='relu', padding='same')(m_pool1) 36 36 - #drop1 = Dropout(0.2)(conv2) # Drop some portion of features to prevent overfitting 37 36 m_pool2 = MaxPooling2D(pool_size=(2, 2))(conv2) 38 37 39 38 conv3 = Conv2D(32, (3, 3), activation='relu', padding='same')(m_pool2) ··· 47 46 drop3 = Dropout(0.2)(dense) 48 47 classif = Dense(2, activation='sigmoid')(drop3) # Final layer to classify 49 48 50 50 - ## Define the model structure 49 49 + ## Define the model start and end 50 50 + model = Model(inputs=inputs, outputs=classif) 51 51 + # Optimizer recommended Adadelta values (lr=0.01) 52 52 + model.compile(optimizer=Adam(), loss='binary_crossentropy', metrics=['accuracy', f1]) 53 53 + 54 54 + return model 55 55 + 56 56 + ''' 57 57 + Model definition for a fully convolutional (no dense layers) network structure 58 58 + ''' 59 59 + def FCN(): 60 60 + ## List of model layers 61 61 + inputs = Input((3, 64, 64)) 62 62 + 63 63 + conv1 = Conv2D(16, (3, 3), activation='relu', padding='same', input_shape=(64, 64, 3))(inputs) 64 64 + m_pool1 = MaxPooling2D(pool_size=(2, 2))(conv1) 65 65 + 66 66 + conv2 = Conv2D(32, (3, 3), activation='relu', padding='same')(m_pool1) 67 67 + m_pool2 = MaxPooling2D(pool_size=(2, 2))(conv2) 68 68 + 69 69 + conv3 = Conv2D(32, (3, 3), activation='relu', padding='same')(m_pool2) 70 70 + drop2 = Dropout(0.2)(conv3) # Drop some portion of features to prevent overfitting 71 71 + m_pool2 = MaxPooling2D(pool_size=(2, 2))(drop2) 72 72 + 73 73 + conv4 = Conv2D(64, (2, 2), activation='relu', padding='same')(m_pool2) 74 74 + 75 75 + flat = Flatten()(conv4) # Makes data 1D 76 76 + drop3 = Dropout(0.2)(flat) 77 77 + classif = Dense(2, activation='sigmoid')(drop3) # Final layer to classify 78 78 + 79 79 + ## Define the model start and end 51 80 model = Model(inputs=inputs, outputs=classif) 52 81 # Optimizer recommended Adadelta values (lr=0.01) 53 82 model.compile(optimizer=Adam(), loss='binary_crossentropy', metrics=['accuracy', f1]) 54 83 55 84 return model 56 85 86 86 + 87 87 + ''' 88 88 + Model definition for the network structure of LeNet 89 89 + Note: LeNet was designed to classify into 10 classes, but we are only performing binary classification 90 90 + ''' 91 91 + def LeNet(): 92 92 + ## List of model layers 93 93 + inputs = Input((3, 64, 64)) 94 94 + 95 95 + conv1 = Conv2D(6, (5, 5), activation='relu', padding='valid', input_shape=(64, 64, 3))(inputs) 96 96 + m_pool1 = MaxPooling2D(pool_size=(2, 2))(conv1) 97 97 + 98 98 + conv2 = Conv2D(16, (5, 5), activation='relu', padding='valid')(m_pool1) 99 99 + m_pool2 = MaxPooling2D(pool_size=(2, 2))(conv2) 100 100 + 101 101 + flat = Flatten()(m_pool2) # Makes data 1D 102 102 + 103 103 + dense1 = Dense(120, activation='relu')(flat) # Fully connected layer 104 104 + dense2 = Dense(84, activation='relu')(dense1) # Fully connected layer 105 105 + drop3 = Dropout(0.2)(dense2) 106 106 + classif = Dense(2, activation='sigmoid')(drop3) # Final layer to classify 107 107 + 108 108 + ## Define the model start and end 109 109 + model = Model(inputs=inputs, outputs=classif) 110 110 + model.compile(optimizer=Adam(), loss='binary_crossentropy', metrics=['accuracy', f1]) 111 111 + 112 112 + return model 113 113 + 114 114 + ''' 115 115 + AlexNet architecture 116 116 + ''' 117 117 + def AlexNet(): 118 118 + inputs = Input(shape=(3, 64, 64)) 119 119 + 120 120 + 121 121 + return model 122 122 + 123 123 + 57 124 def f1(y_true, y_pred): 58 125 def recall(y_true, y_pred): 59 126 """Recall metric. ··· 110 177 lbl_test = to_categorical(lbl_test) 111 178 112 179 ## Define model 180 180 + #model = CNN() 113 181 model = FCN() 182 182 + #model = LeNet() 114 183 # svm_iclf = ImageClassifier(svm.SVC) 115 184 # tree_iclf = ImageClassifier(tree.DecisionTreeClassifier) 116 185 # naive_bayes_iclf = ImageClassifier(naive_bayes.GaussianNBd) 117 186 # ensemble_iclf = ImageClassifier(ensemble.RandomForestClassifier) 118 187 119 188 ## Define training parameters 120 120 - epochs = 10 # an epoch is one forward pass and back propogation of all training data 189 189 + epochs = 25 # an epoch is one forward pass and back propogation of all training data 121 190 batch_size = 150 # batch size - number of training example used in one forward/backward pass 122 191 # (higher batch size uses more memory, smaller batch size takes more time) 123 192 #lrate = 0.01 # Learning rate of the model - controls magnitude of weight changes in training the NN