-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KAT error #353
Comments
Hi, First of all, thanks for your detailed explanations and providing the necessary files. We have been able to reproduce the errors in the scenarios you provided on our end, though we do not have a definite solution at the moment. Our initial theory for the cause of the problem was the use of the We will continue looking into this problem. In the meanwhile, I'm hoping that you are not stuck in your work, and can deploy your network on the hardware thanks to your workarounds. |
@alicangok thank you for providing the update. At the moment, I am able to deploy my end application network on the hardware. Thanks to the hardware's capabilities, I have been curious about the energy consumption for variations of my network. In addition, I have a few related questions.
I can measure energy through the PMON and an external device for input data dimensions, |
Hello again @eshankn, Regarding your first question, while there is no direct method to measure the energy consumed by each layer, you may use the As for your second question, I am not aware of such an explicit constraint. However, with very small & fast networks, the hardware may hang if the inference finishes before the main code had a chance to enter sleep mode. I would suggest you to try the P.S. We will let you know once we have better understanding of your earlier issue. We have been continuing our investigation in different scenarios and both MAX78000 and MAX78002 hardware. |
@alicangok thank you for your response. Using Regarding the KAT error, I have another scenario. According to Limitations of MAX78000 Networks, the maximum dimension (number of rows or columns) for input or output data is 1023. In theory, a 1D-shaped input data of size, Additionally, an input data of size, ep_demo_input_size.zip contains the necessary files to replicate the above scenario. |
This issue has been marked stale because it has been open for over 30 days with no activity. It will be closed automatically in 10 days unless a comment is added or the "Stale" label is removed. |
Thanks for reporting this issue. In this case, the network fails not due to the convolution at the initial layer but the maxpool operation at the second layer. For the maxpool, the input length + kernel length should be smaller than 1026 but until the input length of 1016, the input length of the second layer is greater than 1017, which should be the max input length for a pooling layer with 8-length kernel. We will update the documentation accordingly and/or put proper assertions to the synthesize code. |
Hello, for my application I have been able to generate the C code using the synthesizer tool. But while testing the code on the hardware, the KAT fails with the
Data mismatch
error.The sample input provided is 1D-shaped data of size,
1 x 768
and the model is as followsI am unable to interpret the error but I believe the KAT failing is due to the processor mapping in the YAML file. After multiple trial and error with the processor configurations, the code could pass the KAT. Though I have a fair understanding of creating the YAML file from the provided documentation, I am unable to understand certain configurations when compared with
energy_profiling_kat_pass.yaml
.processors
(as the previous layers) fails the KAT.processors
(as layer 3) also fails the KAT.output_processors
for the last layer 5 fails the KAT.output_processors
fails the KAT.output_processors: 0x0006.0000.0000.0000
oroutput_processors: 0x0009.0000.0000.0000
oroutput_processors: 0x0011.0000.0000.0000
for layer 5 also fails the KAT.ep_demo.zip contains the necessary files to replicate the above specific scenarios.
The C code was generated using
python ai8xize.py --verbose --test-dir demos --prefix ep_demo --checkpoint-file ep_demo_qat_best-q.pth.tar --device MAX78000 --softmax --compact-data --sample-input sample_ep_demo.npy --config-file energy_profiling_kat_pass.yaml --energy
I was also unable to use
--stop-after
to debug the problematic layer.EDIT: Using
out_channels=2
orout_channels=4
after the first layer while training and then generating the code passes the KAT for the first four above scenarios. The fifth scenario still fails the KAT.The text was updated successfully, but these errors were encountered: