Combine fireTS library with neupy library for NARX network based on Levenberg Marquardt

[TobiasEl]

Hi.
I want to create a NARX (Nonlinear Autoregressive with exogenous variables) model based on LM (Levenberg Marquardt) method.

Since this two method are not implemented in keras, I search for the library fireTs (for NARX) and neupy (for LM).

I'm using the sample code for both libraries:

fireTS (NARX): fireTS
neupy (LM): Neupy

and I combine them:

from fireTS.models import NARX
from sklearn.ensemble import RandomForestRegressor
import numpy as np
from neupy import algorithms
from neupy.layers import *

x = np.array([[1, 2], [3, 4]])
y = np.array([[1], [0]])

#y = np.ravel(y) <-just to avoid a warning (the error is the same without comment)

network = Input(2) >> Sigmoid(3) >> Sigmoid(1)
optimizer = algorithms.LevenbergMarquardt(network)

mdl1 = NARX(
    optimizer, #I change random forest for LevenbergMarquardt
    auto_order=2,
    exog_order=[2, 2],
    exog_delay=[1, 1])

mdl1.fit(x, y)

ypred1 = mdl1.predict(x, y)

ypred1

But I'm having this error in .fit method:

ZeroDivisionError Traceback (most recent call last)

in ()
18 exog_delay=[1, 1])
19
---> 20 mdl1.fit(x, y)
21
22 ypred1 = mdl1.predict(x, y)

5 frames

/usr/local/lib/python3.7/dist-packages/neupy/utils/iters.py in
count_minibatches(inputs, batch_size)
22
23 def count_minibatches(inputs, batch_size):
---> 24 return int(math.ceil(count_samples(inputs) / batch_size))
25
26

ZeroDivisionError: division by zero

Any solution?

[itdxer]

Thank you for reporting the issue. I looks like a bug, at least Levenberg-Marquardt algorithm doesn't work with mini-batches.

[itdxer]

as a temporary solution you can either try to run

mdl1.fit(x, y, batch_size=None)

or modify events attribute before you do the training

from neupy.algorithms.base import Events

optimizer = algorithms.LevenbergMarquardt(network)
optimizer.events = Events(network=optimizer, signals=[])

[TobiasEl]

Hi.
Thanks for answer the question.
I have tried both options, but neither works. The error remains the same: division by zero

[TobiasEl]

I also write to fireTS library:
https://github.com/jxx123/fireTS/issues/13
It something about LM batch_size. I put it to 1: mdl1.fit(x, y, batch_size=1)
And it pass the fit method. But the same error happens in predict ypred1 = mdl1.predict(x, y):

/usr/local/lib/python3.7/dist-packages/fireTS/models.py in predict(self, X, y, step)
98
99 for k in range(step):
--> 100 yhat = self._predictNA(features)
101 if k == step - 1:
102 break

/usr/local/lib/python3.7/dist-packages/fireTS/core.py in _predictNA(self, Xdata)
131 mask = np.isnan(Xdata).any(axis=1)
132 X2pred = Xdata[~mask]
--> 133 ypred[~mask] = self.base_estimator.predict(X2pred)
134 return ypred
135

/usr/local/lib/python3.7/dist-packages/neupy/algorithms/gd/base.py in predict(self, *X, **kwargs)
274 raise TypeError("Unknown arguments: {}".format(kwargs))
275
--> 276 return self.network.predict(*self.format_input(X), **predict_kwargs)
277
278 def train(self, X_train, y_train, X_test=None, y_test=None,

/usr/local/lib/python3.7/dist-packages/neupy/layers/graph.py in predict(self, *inputs, **kwargs)
528 inputs=inputs,
529 batch_size=batch_size,
--> 530 show_progressbar=verbose,
531 )
532 return np.concatenate(outputs, axis=0)

/usr/local/lib/python3.7/dist-packages/neupy/utils/iters.py in apply_batches(function, inputs, batch_size, show_progressbar, show_output, average_outputs)
170 batch_size = n_samples if batch_size is None else batch_size
171
--> 172 n_batches = count_minibatches(inputs, batch_size)
173 bar = progressbar.NullBar()
174

/usr/local/lib/python3.7/dist-packages/neupy/utils/iters.py in count_minibatches(inputs, batch_size)
22
23 def count_minibatches(inputs, batch_size):
---> 24 return int(math.ceil(count_samples(inputs) / batch_size))
25
26

ZeroDivisionError: division by zero

[itdxer]

I have tried both options, but neither works. The error remains the same: division by zero

I'm quite surprised that the second option didn't work. Do you get the same exception as before? I mean even if the same function fails I wonder if it fails in exactly the same place or if it happens elsewhere

[TobiasEl]

This is the code:

from fireTS.models import NARX
import numpy as np
from neupy import algorithms
from neupy.layers import *
from neupy.algorithms.base import Events

# Random training data
x = np.array([[1, 2], [3, 4]])
y = np.array([[1], [0]])

#y = np.ravel(y)

network = Input(2) >> Sigmoid(3) >> Sigmoid(1)

optimizer = algorithms.LevenbergMarquardt(network)
optimizer.events = Events(network=optimizer, signals=[])

mdl1 = NARX(
    optimizer,
    auto_order=2,
    exog_order=[2, 2],
    exog_delay=[1, 1])

mdl1.fit(x, y, batch_size=None)

ypred1 = mdl1.predict(x, y)

ypred1

And here the error:

---

ZeroDivisionError Traceback (most recent call last)

in ()
22 exog_delay=[1, 1])
23
---> 24 mdl1.fit(x, y, batch_size=None)
25
26 ypred1 = mdl1.predict(x, y)

5 frames

/usr/local/lib/python3.7/dist-packages/neupy/utils/iters.py in count_minibatches(inputs, batch_size)
22
23 def count_minibatches(inputs, batch_size):
---> 24 return int(math.ceil(count_samples(inputs) / batch_size))
25
26

ZeroDivisionError: division by zero