Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Files sometimes aren't correctly downloaded #494

Open
Aeris1One opened this issue Jan 13, 2024 · 1 comment
Open

Files sometimes aren't correctly downloaded #494

Aeris1One opened this issue Jan 13, 2024 · 1 comment

Comments

@Aeris1One
Copy link

Aeris1One commented Jan 13, 2024 •
edited
Loading

What ?

Synchronisation doesn't download files, at least for employees bulletins, likely to be broken also for employers though.
Can reproduce both locally and on Cozy Cloud.

Logs

Here are the logs from a standalone run (note that employer name, bulletins number, my cesu number and EUR amounts were manually redacted and were correct in the original logs) :

1️⃣ First run :

Click to expand 
yarn run v1.22.21
$ cozy-konnector-standalone
BaseKonnector: debug : Preparing konnector...
BaseKonnector: debug : Running konnector main...
cozy-konnector-libs: info : Authenticating...
cozy-konnector-libs: info : Correctly logged in
cozy-konnector-libs: info : Cesu number found in page
cozy-client-js-stub: debug : Checking if /EMPLOYER_NAME exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
cozy-client-js-stub: debug : Checking if /EMPLOYER_NAME exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
cozy-client-js-stub: debug : Checking if /EMPLOYER_NAME exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
cozy-client-js-stub: debug : Checking if / exists
cozy-client-js-stub: debug : Checking if / exists
cozy-client-js-stub: debug : Checking if / exists
cozy-client-js-stub: debug : Creating new directory EMPLOYER_NAME
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
cozy-client-js-stub: debug : Creating new directory EMPLOYER_NAME
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
cozy-client-js-stub: debug : Creating new directory EMPLOYER_NAME
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
saveFiles: debug : Checking existence of BULLETINNO1
saveFiles: debug : Checking existence of BULLETINNO2
saveFiles: debug : Checking existence of BULLETINNO3
saveFiles: debug : not found
saveFiles: debug : not found
saveFiles: debug : not found
saveFiles: debug : Checking existence of /EMPLOYER_NAME/2023-12_XXXEUR.pdf
cozy-client-js-stub: debug : Checking if /EMPLOYER_NAME/2023-12_XXXEUR.pdf exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME/2023-12_XXXEUR.pdf
saveFiles: debug : Checking existence of /EMPLOYER_NAME/2023-11_XXXEUR.pdf
cozy-client-js-stub: debug : Checking if /EMPLOYER_NAME/2023-11_XXXEUR.pdf exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME/2023-11_XXXEUR.pdf
saveFiles: debug : Checking existence of /EMPLOYER_NAME/2023-10_XXXEUR.pdf
cozy-client-js-stub: debug : Checking if /EMPLOYER_NAME/2023-10_XXXEUR.pdf exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME/2023-10_XXXEUR.pdf
saveFiles: debug : /EMPLOYER_NAME/2023-12_XXXEUR.pdf does not exist
saveFiles: debug : /EMPLOYER_NAME/2023-11_XXXEUR.pdf does not exist
saveFiles: debug : /EMPLOYER_NAME/2023-10_XXXEUR.pdf does not exist
saveFiles: debug : {
  fileurl: 'https://www.cesu.urssaf.fr/cesuwebdec/salaries/MYCESUNUMBER/editions/bulletinSalairePE?refDoc=BULLETINNO1',
  filename: '2023-12_XXXEUR.pdf',
  amount: xxx,
  isRefund: true,
  date: 2023-12-31T00:00:00.000Z,
  vendorRef: 'BULLETINNO1',
  subPath: 'EMPLOYER_NAME',
  fileAttributes: {
    metadata: {
      contentAuthor: 'cesu.urssaf.fr',
      issueDate: 2024-01-13T22:57:21.517Z,
      carbonCopy: true,
      qualification: [Qualification]
    }
  },
  requestOptions: { jar: RequestJar { _jar: [CookieJar] } },
  vendor: 'cesu',
  matchingCriterias: { labelRegex: '.*' }
}
saveFiles: debug : File /EMPLOYER_NAME/2023-12_XXXEUR.pdf does not exist yet or is not valid
cozy-client-js-stub: debug : Checking if //EMPLOYER_NAME exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
saveFiles: debug : {
  fileurl: 'https://www.cesu.urssaf.fr/cesuwebdec/salaries/MYCESUNUMBER/editions/bulletinSalairePE?refDoc=BULLETINNO2',
  filename: '2023-11_XXXEUR.pdf',
  amount: xxx,
  isRefund: true,
  date: 2023-11-30T00:00:00.000Z,
  vendorRef: 'BULLETINNO2',
  subPath: 'EMPLOYER_NAME',
  fileAttributes: {
    metadata: {
      contentAuthor: 'cesu.urssaf.fr',
      issueDate: 2024-01-13T22:57:21.518Z,
      carbonCopy: true,
      qualification: [Qualification]
    }
  },
  requestOptions: { jar: RequestJar { _jar: [CookieJar] } },
  vendor: 'cesu',
  matchingCriterias: { labelRegex: '.*' }
}
saveFiles: debug : File /EMPLOYER_NAME/2023-11_XXXEUR.pdf does not exist yet or is not valid
cozy-client-js-stub: debug : Checking if //EMPLOYER_NAME exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
saveFiles: debug : {
  fileurl: 'https://www.cesu.urssaf.fr/cesuwebdec/salaries/MYCESUNUMBER/editions/bulletinSalairePE?refDoc=BULLETINNO3',
  filename: '2023-10_XXXEUR.pdf',
  amount: xxx,
  isRefund: true,
  date: 2023-10-31T00:00:00.000Z,
  vendorRef: 'BULLETINNO3',
  subPath: 'EMPLOYER_NAME',
  fileAttributes: {
    metadata: {
      contentAuthor: 'cesu.urssaf.fr',
      issueDate: 2024-01-13T22:57:21.518Z,
      carbonCopy: true,
      qualification: [Qualification]
    }
  },
  requestOptions: { jar: RequestJar { _jar: [CookieJar] } },
  vendor: 'cesu',
  matchingCriterias: { labelRegex: '.*' }
}
saveFiles: debug : File /EMPLOYER_NAME/2023-10_XXXEUR.pdf does not exist yet or is not valid
cozy-client-js-stub: debug : Checking if //EMPLOYER_NAME exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
cozy-client-js-stub: debug : Creating new file 2023-12_XXXEUR.pdf
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME/2023-12_XXXEUR.pdf
cozy-client-js-stub: debug : Creating new file 2023-11_XXXEUR.pdf
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME/2023-11_XXXEUR.pdf
cozy-client-js-stub: debug : Creating new file 2023-10_XXXEUR.pdf
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME/2023-10_XXXEUR.pdf
saveFiles: warn : File download error Error: read ECONNRESET
saveFiles: warn : File download error Error: read ECONNRESET
saveFiles: warn : File download error Error: read ECONNRESET
Done in 1.34s.

🔢 Subsequent run (without cleaning output folder) :

Click to expand 
yarn run v1.22.21
$ cozy-konnector-standalone
BaseKonnector: debug : Preparing konnector...
BaseKonnector: debug : Running konnector main...
cozy-konnector-libs: info : Authenticating...
cozy-konnector-libs: info : Correctly logged in
cozy-konnector-libs: info : Cesu number found in page
cozy-client-js-stub: debug : Checking if /EMPLOYER_NAME exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
cozy-client-js-stub: debug : Checking if /EMPLOYER_NAME exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
cozy-client-js-stub: debug : Checking if /EMPLOYER_NAME exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
saveFiles: debug : Checking existence of BULLETINNO1
saveFiles: debug : Checking existence of BULLETINNO2
saveFiles: debug : Checking existence of BULLETINNO3
saveFiles: debug : not found
saveFiles: debug : not found
saveFiles: debug : not found
saveFiles: debug : Checking existence of /EMPLOYER_NAME/2023-12_XXXEUR.pdf
cozy-client-js-stub: debug : Checking if /EMPLOYER_NAME/2023-12_XXXEUR.pdf exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME/2023-12_XXXEUR.pdf
saveFiles: debug : Checking existence of /EMPLOYER_NAME/2023-11_XXXEUR.pdf
cozy-client-js-stub: debug : Checking if /EMPLOYER_NAME/2023-11_XXXEUR.pdf exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME/2023-11_XXXEUR.pdf
saveFiles: debug : Checking existence of /EMPLOYER_NAME/2023-10_XXXEUR.pdf
cozy-client-js-stub: debug : Checking if /EMPLOYER_NAME/2023-10_XXXEUR.pdf exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME/2023-10_XXXEUR.pdf
saveFiles: warn : /EMPLOYER_NAME/2023-12_XXXEUR.pdf is empty
saveFiles: warn : BAD_FILE_SIZE
saveFiles: warn : /EMPLOYER_NAME/2023-11_XXXEUR.pdf is empty
saveFiles: warn : BAD_FILE_SIZE
saveFiles: warn : /EMPLOYER_NAME/2023-10_XXXEUR.pdf is empty
saveFiles: warn : BAD_FILE_SIZE
saveFiles: error : Could not get a file name for the entry
saveFiles: warn : false is invalid
saveFiles: error : Could not get a file name for the entry
saveFiles: warn : false is invalid
saveFiles: error : Could not get a file name for the entry
saveFiles: warn : false is invalid
saveFiles: info : Error in shouldReplaceFile : BAD_DOWNLOADED_FILE
saveFiles: debug : Will replace /EMPLOYER_NAME/EMPLOYER_NAME/2023-12_XXXEUR.pdf...
saveFiles: debug : {
  fileurl: 'https://www.cesu.urssaf.fr/cesuwebdec/salaries/MYCESUNUMBER/editions/bulletinSalairePE?refDoc=BULLETINNO1',
  filename: '2023-12_XXXEUR.pdf',
  amount: XXX,
  isRefund: true,
  date: 2023-12-31T00:00:00.000Z,
  vendorRef: 'BULLETINNO1',
  subPath: 'EMPLOYER_NAME',
  fileAttributes: {
    metadata: {
      contentAuthor: 'cesu.urssaf.fr',
      issueDate: 2024-01-13T22:43:43.995Z,
      carbonCopy: true,
      qualification: [Qualification]
    }
  },
  requestOptions: { jar: RequestJar { _jar: [CookieJar] } },
  vendor: 'cesu',
  matchingCriterias: { labelRegex: '.*' }
}
saveFiles: debug : File /EMPLOYER_NAME/2023-12_XXXEUR.pdf does not exist yet or is not valid
cozy-client-js-stub: debug : Checking if //EMPLOYER_NAME exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
saveFiles: info : Error in shouldReplaceFile : BAD_DOWNLOADED_FILE
saveFiles: debug : Will replace /EMPLOYER_NAME/EMPLOYER_NAME/2023-11_XXXEUR.pdf...
saveFiles: debug : {
  fileurl: 'https://www.cesu.urssaf.fr/cesuwebdec/salaries/MYCESUNUMBER/editions/bulletinSalairePE?refDoc=BULLETINNO2',
  filename: '2023-11_XXXEUR.pdf',
  amount: XXX,
  isRefund: true,
  date: 2023-11-30T00:00:00.000Z,
  vendorRef: 'BULLETINNO2',
  subPath: 'EMPLOYER_NAME',
  fileAttributes: {
    metadata: {
      contentAuthor: 'cesu.urssaf.fr',
      issueDate: 2024-01-13T22:43:43.995Z,
      carbonCopy: true,
      qualification: [Qualification]
    }
  },
  requestOptions: { jar: RequestJar { _jar: [CookieJar] } },
  vendor: 'cesu',
  matchingCriterias: { labelRegex: '.*' }
}
saveFiles: debug : File /EMPLOYER_NAME/2023-11_XXXEUR.pdf does not exist yet or is not valid
cozy-client-js-stub: debug : Checking if //EMPLOYER_NAME exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
saveFiles: info : Error in shouldReplaceFile : BAD_DOWNLOADED_FILE
saveFiles: debug : Will replace /EMPLOYER_NAME/EMPLOYER_NAME/2023-10_XXXEUR.pdf...
saveFiles: debug : {
  fileurl: 'https://www.cesu.urssaf.fr/cesuwebdec/salaries/MYCESUNUMBER/editions/bulletinSalairePE?refDoc=BULLETINNO3',
  filename: '2023-10_XXXEUR.pdf',
  amount: XXX,
  isRefund: true,
  date: 2023-10-31T00:00:00.000Z,
  vendorRef: 'BULLETINNO3',
  subPath: 'EMPLOYER_NAME',
  fileAttributes: {
    metadata: {
      contentAuthor: 'cesu.urssaf.fr',
      issueDate: 2024-01-13T22:43:43.995Z,
      carbonCopy: true,
      qualification: [Qualification]
    }
  },
  requestOptions: { jar: RequestJar { _jar: [CookieJar] } },
  vendor: 'cesu',
  matchingCriterias: { labelRegex: '.*' }
}
saveFiles: debug : File /EMPLOYER_NAME/2023-10_XXXEUR.pdf does not exist yet or is not valid
cozy-client-js-stub: debug : Checking if //EMPLOYER_NAME exists
cozy-client-js-stub: debug : Real path : /var/home/charlesp/Documents/cozy-konnector-cesu/data/EMPLOYER_NAME
saveFiles: debug : replacing file for 2023-12_XXXEUR.pdf
saveFiles: debug : replacing file for 2023-11_XXXEUR.pdf
saveFiles: debug : replacing file for 2023-10_XXXEUR.pdf
saveFiles: warn : unknown file download error: Cannot read properties of undefined (reading 'dir_id')
saveFiles: warn : unknown file download error: Cannot read properties of undefined (reading 'dir_id')
saveFiles: warn : unknown file download error: Cannot read properties of undefined (reading 'dir_id')
saveFiles: info : saveFiles created 3 files for 3 entries
hydrateAndFilter: debug : 0 items before hydrateAndFilter
hydrateAndFilter: debug : 0 items after hydrateAndFilter
saveFiles: warn : File download error Error: read ECONNRESET
saveFiles: warn : File download error Error: read ECONNRESET
saveFiles: warn : File download error Error: read ECONNRESET
saveFiles: warn : No file to download
saveFiles: info : saveFiles created 0 files for 0 entries
BaseKonnector: debug : The connector has been run
Done in 1.70s.
  • On Cozy Cloud, EMPLOYER_NAME folder is created but empty (are files created but cleaned up after connector run? are files not being cleaned up on standalone due to differences between client-stub and client?)
  • On standalone run it is created and populated with 0-bytes empty files with the correct name (YYYY-MM_XXXEUR.pdf), sometimes with bigger, corrupt files

Will investigate more thoroughly in the coming week. Can't reproduce any error through manual API calls, requests made by the connector seems to be correct. The only ECONNRESET error I can reproduce is when not sending any User-Agent header, which is not the case here, as the connector correctly sends one.
@Aeris1One Aeris1One changed the title Files aren't correctly downloaded Files sometimes aren't correctly downloaded Jan 14, 2024
@Aeris1One
Copy link
Author

Worked today, was broken on my Cozy Cloud for approx. a week. Broken on standalone yesterday, yet it works today on both. Might have been an issue on CESU side, but what's weird is that their web interface uses the exact same API calls and wasn't broken.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Aeris1One