mkvalidator mkvalidator.c gcd function Floating point exception

[giantbranch]

Credit: giantbranch of NSFOCUS Security Team

What's the problem?

Floating point exception was found in gcd function of mkvalidator.c in mkvalidator v0.5.2.

ASAN reports:

$ ./mkvalidator ./tests_82.mkv
WRN080: Unknown element [30][57][41] at 59 size 18
WRN080: Unknown element [2A][D7][B1] at 81 size 2
WRN080: Unknown element [44][89] at 87 size 4
ERR202: Unique element 'DisplayHeight' in Video at 177 found more than once at 196
ERR202: Unique element 'DisplayHeight' in Video at 177 found more than once at 196
ERR0E7: Video track #1 at 99 has a null display height
AddressSanitizer:DEADLYSIGNAL
=================================================================
==12833==ERROR: AddressSanitizer: FPE on unknown address 0x0000004ca475 (pc 0x0000004ca475 bp 0x7ffe9d9d8380 sp 0x7ffe9d9d3820 T0)
    #0 0x4ca475 in gcd /root/debug-fuzz-reslut/mkvalidator/foundation-source/mkvalidator/mkvalidator.c:209:23
    #1 0x4ca475 in CheckVideoTrack /root/debug-fuzz-reslut/mkvalidator/foundation-source/mkvalidator/mkvalidator.c:265:31
    #2 0x4ca475 in main /root/debug-fuzz-reslut/mkvalidator/foundation-source/mkvalidator/mkvalidator.c:1178:19
    #3 0x7f330cf9a83f in __libc_start_main /build/glibc-e6zv40/glibc-2.23/csu/../csu/libc-start.c:291
    #4 0x41bf58 in _start (/root/reproduce/mkvalidator+0x41bf58)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: FPE /root/debug-fuzz-reslut/mkvalidator/foundation-source/mkvalidator/mkvalidator.c:209:23 in gcd
==12833==ABORTING

location: foundation-source/libebml2/ebmlcrc.c:244

How can we reproduce the issue?

Compile command I use:

clang corec/tools/coremake/coremake.c -o coremake && ./coremake gcc_linux_x64

make -C mkvalidator -e CC="clang -g -O2 -fsanitize=address -fsanitize-address-use-after-scope" -e CXX="clang++ -g -O2 -fsanitize=address -fsanitize-address-use-after-scope" -e STRIP=""

reproduce the issue

unzip tests_xx.zip
mkvalidator ./tests_xx.mkv

poc:
tests_82.zip

the details about my environment.

• mkvalidator version:

$ ./mkvalidator --version
mkvalidator v0.5.2, Copyright (c) 2010-2015 Matroska Foundation
	file "--version"

• Host Operating System and version: Ubuntu 16.04.3 LTS
• Host CPU architecture: x86_64
• clang: clang version 11.0.0