Multi tool use

Updated: reshape each row data into a (x, 1) array

Recently I was reading neural network and deep learning by Michael Nielsen (link) and wanted to test the neural network on loan default data. However after quite a few tries I still did not manage to transform my csv format data into the required matrix format by the script.

The csv file contains 769 variables and 1 boolean default entry. looks like this:

. v1 v2 v3 ... v770;
1. 1 2 3 ... 0;
2. 2 1 2 ... 1;
...


This is how I do my import:

import numpy as np
tr_input = [np.reshape(genfromtxt('training.csv', delimiter=','), (769,10000))]
tr_res = np.reshape(genfromtxt('training2.csv', delimiter=','),(1, 10000))
tr_test = [np.reshape(genfromtxt('testing.csv', delimiter=','), (769,2000))]
tr_test2 = np.reshape(genfromtxt('testing2.csv', delimiter=','), (1, 2000))
test_data = list(zip(tr_test, tr_test2))
training_data = list(zip(tr_input, tr_res))


However it returns

Traceback (most recent call last):

File "<ipython-input-8-de046f78e8ed>", line 3, in <module>
net.SGD(training_data, 30, 300, 3.0, test_data = test_data)

File "/Users/Neal/Documents/Sources/network.py", line 71, in SGD
self.update_mini_batch(mini_batch, eta)

File "/Users/Neal/Documents/Sources/network.py", line 85, in update_mini_batch
delta_nabla_b, delta_nabla_w = self.backprop(x, y)

File "/Users/Neal/Documents/Sources/network.py", line 105, in backprop
z = np.dot(w, activation)+b

ValueError: shapes (30,769) and (10000,) not aligned: 769 (dim 1) != 10000 (dim 0)


Updated:

Don't know why genfromtxt gives the first entry NAN but pandas works just fine.

After studying what the original tutorial data, I think I might need to reshape each row data into a (769, 1) array but I don't know how to.

Attached are the links to download the neural network and my data:

Neural network: https://github.com/MichalDanielDobrzanski/DeepLearningPython35

Data: https://drive.google.com/drive/folders/1bQEqgb1o9kKNyv8_IBPlNRci5cfSYwFL?usp=sharing
(testing and training are variables and testing2 and training2 are default information booleans, 0 for no default and 1 for default).

genfromtxt()

2 Answers
2

You can use either Numpy or Pandas for reading such CSV files - I prefer Pandas:

import pandas as pd

X_train = pd.read_csv(r'D:downloadtraining.csv', header=None, dtype='float64')


result:

In [18]: X_train.shape
Out[18]: (10000, 769)


if you want to transpose it, so it has a shape: 769 x 10000

In [19]: X_train = X_train.T

In [20]: X_train.shape
Out[20]: (769, 10000)


In your code you have some errors:

One spotted in the comments is genfromtxt , this function already does read in the numpy array object so based on your import statement you didn't actually referring to the module, here is a cleaner way to read the data and reshape. Here is the numpy documentation for the reading txt files and reshape : numpy.genfromtxt and numpy.reshape

Here is the more "readable" way of reading the data and reshaping (I hate to get lost into so many parentheses and is not so easy to read and need to be carefull about the orders), so as you see this is the proper way of reading data as genfromtxt you need ...

So will "translate" the steps:

for the tr_input variable: read the raw data, based on the file (10000, 769), and perform a reshape so the data to look (769,10000)

for the tr_res variable: read the raw data, based on the file (10000,), and perform a reshape so the data to look (1, 10000)

for the tr_test variable: read the raw data, based on the file (2000, 769), and perform a reshape so the data to look (769, 2000)

for the tr_test2 variable: read the raw data, based on the file (2000,), and perform a reshape so the data to look (1, 2000)

Note: As can be see the errors are solved, one was using the packages
module and your error pointed int he question is the shape of the data
... you can't reshape an array of (2000,) into (769,10000) ... your testing.csv is of the format (2000,)

# readable format prefered by me, not a standard.

# https://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html
# https://docs.scipy.org/doc/numpy/reference/generated/numpy.reshape.html

import numpy as np
tr_input = np.genfromtxt('/Users/sb0709/Desktop/dqn-pytorch/data_loan/training.csv', delimiter=',').reshape(769,10000)
tr_res = np.genfromtxt('/Users/sb0709/Desktop/dqn-pytorch/data_loan/training2.csv', delimiter=',').reshape(1, 10000)
tr_test = np.genfromtxt('/Users/sb0709/Desktop/dqn-pytorch/data_loan/testing.csv', delimiter=',').reshape(769,2000)
tr_test2 = np.genfromtxt('/Users/sb0709/Desktop/dqn-pytorch/data_loan/testing2.csv', delimiter=',').reshape(1, 2000)

# creating the lists
test_data = list(zip(tr_test, tr_test2))
training_data = list(zip(tr_input, tr_res))


Also here listing the output:

Note: if the data has header/variable names, than pass skip_header=1 to skip first row when reading it and the data shape also will chage to -1.

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.