Efficient algebraic operators overloading

[evgueni-ovtchinnikov]

Efficient algebraic operators overloading

In C++, algebraic operators for objects of a user-defined class, say DataContainer, can be overloaded by functions with the prototype of the kind

DataContainer operator+(const DataContainer& x, const DataContainer& y);

However, this would create temporary DataContainer objects when computing algebraic expression involving more than one operation. For exapmle, computing a + b*(c - d) creates temporary objects for storing c - d and b*(c - d).

The creation of such temporary objects can be avoided by delaying the computation until the result gets assigned or passed to a function. One way to arrange such a delay is to derive DataContainer class from a class for handling algebraic expressions, say Expression, and overload by functions with the prototype of the kind

Expression operator+(const Expression& x, const Expression& y);

This function task is to recursively record the data pointers ({&c, &d, &b, &a}) and the sequence of operations ({-, *, +}) to be performed when the final result is eventually computed based on the data pointers and operations recorded in x and y.

Actual computation is performed either by the overloaded assignment operator

DataContainer& operator=(const Expression& ex);

or the constructor

DataContainer(const Expression& ex);

Python/Matlab?

A good question is: once the suggested C++ implementation of overloaded algebraic operators is available, can it be interfaced into Python/Matlab (where assignment cannot be overloaded) without having to edit all existing scripts that use algebraic operators (e.g. introducing functions similar to dask.delayed/compute)?

[paskino]

A similar concept is done, for instance in Dask with the object Delayed.
In that case a graph of the operations to be carried out is generated, and then the computation is done in parallel.

[evgueni-ovtchinnikov]

Dask approach has too much boilerplate IMHO, we could simply do something like

x = compute(a + b*(c - d))

but both approaches are not backward compatible - all existing scripts that use DataContainer algebra would have to be edited.

[epapoutsellis]

Personally, I would start by doing experiments on something that is already available and in my opinion very successful and very popular. Unless, there is no significant reason to reinvent the wheel, I would first use Dask with SIRF. I have already done some experiments with CIL, but it is work in progress.

Dask is a parallel and distributed library that lives in python and communicates with many known libraries living in the python ecosystem.

In terms of SIRF and Dask, I realised something interesting. SIRF does not depend on python, its algebra is at the C level with python/numpy wrappers. The same applies with the standard numpy library. So my guess is that SIRF and Dask will play well. Therefore, a simple example is to use the triplet from the dask.distributed framework which is:

• client
• worker
• Scheduler

In practice for a quick test on the synergy of SIRF+DASK, we could use dask.scatter useful for large data, and decorate our direct method from the AcquisitionModel with the delayed decorator. We can do a time comparison SIRF vs SIRF+DASK or monitor the memory usage, with the Dask jupyter Dashboard.

Now, if you are interested in a distributed framework at the C level, implementing client, workers and schedulers, I suggest to read distributed-dask-cpp-workers and the issue in the dask github repository, Cross Language Client/Workers. Many people are interested, there is an implemention in Julia but probably not maintained.