Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Child processes taking more time than parent for JSON processing #97

Open
ishitach opened this issue Jul 22, 2019 · 0 comments
Open

Child processes taking more time than parent for JSON processing #97

ishitach opened this issue Jul 22, 2019 · 0 comments

Comments

@ishitach
Copy link

ishitach commented Jul 22, 2019

I am trying JSON stringify of very large datasets. I am creating child processes to perform the operations and doing equivalent operations in the parent process.

The child processes should execute the results faster as they are executing the process in parallel but I am observing opposite results in my test JSON.
For running this code please replace the url's with the actual JSON(unable to paste them due to character limit).
The code for main.js is as follows:

let workerFarm = require("worker-farm")
    , workers    = workerFarm(require.resolve('./child'))
    , ret        = 0
    ,final_out =0;

let start1 = new Date().getTime();
for (let i = 0; i < 100; i++) {
    workers('#' + i + ' FOO', function (err, outp) {
        if (++ret == 100){
            console.log('Doing it the fast (multi-process) way...');
            let end1 = new Date().getTime();
            console.log('fast took',  (end1 - start1), 'milliseconds')
            workerFarm.end(workers)
        }
    })
}

let start2 = new Date().getTime();

for (let i = 0; i < 100; i++) {

    let json= JSON.stringify([https://pkgstore.datahub.io/datahq/1mb-test/1mb-test_json/data/ca5fd34861cc68b4f519b1c1e15c510e/1mb-test_json.json](url));

        if (i == 99){
        
        console.log('Doing it the slow (single-process) way...')
        console.log('slow took', new Date().getTime() - start2, 'milliseconds')
    }
}

The code for child.js file is as follows:

'use strict'

module.exports = function (inp, callback) {
    let srt= new Date().getTime();
    let json= JSON.stringify([https://pkgstore.datahub.io/datahq/1mb-test/1mb-test_json/data/ca5fd34861cc68b4f519b1c1e15c510e/1mb-test_json.json](url));

    let end = new Date().getTime();
    return callback(null, end-srt);
}

Kindly help me identify the issue as the results of this iteration are as follows:

Doing it the slow (single-process) way...
slow took 967 milliseconds
Doing it the fast (multi-process) way...
fast took 1079 milliseconds

The child processes are taking more time. I have tried this for less as well as more iterations but the results are quite similar.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant