How long takes TornadoVM JIT compiler to transform Java bytecode into OpenCL

[MLobatoR]

Hello :)

How long should I wait for TornadoVM JIT compiler to transform Java bytecode into OpenCL? Or if is taking too long should I stop the process? I have no exceptions so far but the process started 5h ago.

[stratika]

Hi @MLobatoR, the compilation time depends on the actual architecture you target.

A) In case, you compile OpenCL for CPUs or GPUs, this should not take more than few milliseconds.

B) In case, you compile for FPGAs, it is common to compile for several hours. To be fair the actual compilation of TornadoVM that translates Java bytecodes to OpenCL will still take a few milliseconds. However, a second compilation takes place from the FPGA HLS which is transparently performed by TornadoVM. We have seen such cases, in which the compilation can take up to 3 hours.

Which one is your case?

[MLobatoR]

Hi @jjfumero and @stratika,

It is working now. Thank you so much for your assistance.

I have one last question, this one related with the new version.
What should be considered as an object to lock in Memory, to transfer for device and transfer for host? What is the criteria I should follow to know which one is which?

Thank you,
ML

[jjfumero]

Thanks for the question. We are preparing documentation to be released for 0.15 that explains all these new functions. So please, do not hesitate to ask. We understand some of them can be confusing. Having said that, we are also open to feedback.

Short answer

So the new function lockObjectsInMemory is a way in TornadoVM to lock a device buffer and avoid alloc/dealloc every time a task-graph is executed (taskGraph.execute()).

Long answer

We have changed the way TornadoVM handles memory. Currently, with the latest APIs, TornadoVM will alloc and dealloc device buffers. If a task-graph is going to be executed just once, then this should be the way to go, and it is the default mode in Tornado right now.

However, if a task-graph is going to be executed multiple times, we can "optimize" buffer allocation by reusing device buffers as much as possible. That's the purpose of the lockObjectsInMemory function. This means that the first time the task-graph is executed, TornadoVM will do a malloc for all device buffers. When the execute finishes, TornadoVM will not release device memory resources. From the second time the task-graph is executed, TornadoVM will directly reuse all device buffers already created. This has a huge impact on performance.

The reason we do not have documented this function yet is that we are adding a TornadoVM-Executor. What we want to achieve is a separation between the task-graph definition and task-graph execution plan and optimization. We are planning to move this function (lockObjectsInMemory) to the executor once we have it.

[MLobatoR]

@jjfumero thank for your clarification, I think I understand now.

Hope this is the last question :) but if I want to run the same code in my while cicle and the only thing that changes are the input variables is there a way to not be always generating graph?

Thank you very much for all your help and assistance,
ML

[jjfumero]

Yes, there are a few ways:

1. If your input/output arrays do not change in length over time, then you can use the following template:

// pseudocode 
TaskGraph tg = new TaskGraph("sample")
            .lockObjectsInMemory(arr1, arr2, output1, output2, .... )      // This is to optimize data allocation between calls
            .transferToDevice(DataTransferMode.EVERY_EXECUTION, arr1, arr2, ...)  // This is to stream-In data from Host -> Device 
            .task( "tk1", ... )
            .task( "tk2", ... ) 
            .transferToHost(output1, output2, ...) ;      // Stream-Out data Device -> Host 

while (condition) {
    tg.execute();

    checkCondition();
}

2. If your Input/Output arrays change length over time, you need to use another method called replaceParameter every time there is a new resize. This is because TornadoVM, when it compiles the code from Java to {OpenCL | SPIR-V | PTX }, it also performs partial evaluation, and it propagates loop-bounds and some other primitive as constants.

// pseudocode 
TaskGraph tg = new TaskGraph("sample")
            .transferToDevice(DataTransferMode.EVERY_EXECUTION, arr1, arr2, ...)
            .task( "tk1", ... )
            .task( "tk2", ... ) 
            .transferToHost(output1, output2, ...) ;

tg.execute();

while (condition) {

    tg.execute();
    tg.replaceParameter(oldArray1, newArray1);
    tg.replaceParameter(oldArray2, newArray2); 
    ... 

    checkCondition();
    checkReferences(oldArrays, newArrays);
}

You can see some examples here: https://github.com/beehive-lab/TornadoVM/blob/master/tornado-unittests/src/main/java/uk/ac/manchester/tornado/unittests/dynsize/Resize.java#L67

[MLobatoR]

@jjfumero that a great help 👍

Thank you again for all your help,
ML