Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ViacomCBS] Samsung fatal stall on preroll to first content period. #2620

Closed
dsparacio opened this issue Jun 5, 2020 · 53 comments
Closed

[ViacomCBS] Samsung fatal stall on preroll to first content period. #2620

dsparacio opened this issue Jun 5, 2020 · 53 comments
Assignees
Labels
component: ads The issue involves the Shaka Player ads API or the use of other ad SDKs platform: TV/STB Issues affecting smart TV or set-top box platforms status: archived Archived and locked; will not be updated type: bug Something isn't working correctly
Milestone

Comments

@dsparacio
Copy link
Contributor

Have you read the FAQ and checked for duplicate open issues?
Yes completely.

What version of Shaka Player are you using?
Tested in Shaka 2.5.12, 2.5.11, 2.5.5 and 2.5.1 - Same results on all

Can you reproduce the issue with our latest release version?
Yes

Can you reproduce the issue with the latest code from master?
Yes

Are you using the demo app or your own custom app?
Demo raw simple setup. Page urls included in support email with content and la urls.

If custom app, can you reproduce the issue using our demo app?
yes

What browser and OS are you using?
This issue is only being reported and reproduced by our team for SOME Samsung Tizen based Smart TV mainly years 2017 and 2018. 2019 and 2020 year seems unaffected. Some 2017 and 2018 are also working as expected!!

For embedded devices (smart TVs, etc.), what model and firmware version are you using?

  • 2017 UN50MU6070 Fails 100%
  • 2018 UN55NU710D Fails 100%
  • 2018 UN32N5300AF PASSES
  • 2018 UN49NU8000 PASSES

What are the manifest and license server URIs?
Sent over on shaka-player-issues@google.com

What did you do?

  • Play from 0 start time and let preroll finish, on the first transition to a content period player stalls.
  • When a samsung model year has the issue, it happens 100%
  • Does not happen with the exact same encode of the stream that is not passed though DAI.
  • Does not happen with the exact same encode pass to DAI but no DRM.
  • In logs. DRM keysystem, when we fail, both seem to init fine. I tested WV and PR here.
  • Non zero test are interesting.
    • If you skip the pre-roll and start past that second in stream... all mid-roll DAI period transition thereafter works as expected.
    • Stream is fine on all TVs we tested.
  • Dash.js does play the same DAI/DRM steam without issue on all the tested devices.
    • Shaka is much better overall quality of experience so it is preferred on Samsung

I am also including logs in the support email with content. These logs will be from both fail and pass captures in a few different versions of Shaka mentioned above.

What did you expect to happen?
The first content period initializes and plays out.

What actually happened?
The content period fails to init and play properly.

@dsparacio
Copy link
Contributor Author

One update here
2017 UN50MU6070 Fails 100% This fails differently than the 2018 tv. Same point in time. Logs have different recovery and loop to them. And sometimes another ad period starts to play. We never get into content periods.

2018 UN55NU710D Fails 100% Fails always at first period break... and stays there until page app is closed on TV. never picks up.

@chrisfillmore
Copy link
Contributor

chrisfillmore commented Jun 8, 2020

There are several issues related to ad playback on Tizen platforms, you can take a look at #1094 and follow the thread.

Ultimately, even with fixes in the JS/app layer, we still found we had playback problems on Tizen devices if we started playing clear content before unencrypted content. We attributed this to bug(s) in the Tizen media layer. So, our solution was to force playback to start during protected content. Are you able to skip your prerolls? I'll bet this would solve the problem.

@dsparacio
Copy link
Contributor Author

@chrisfillmore Thanks for the added info and point me to that ticket. Same experience.

if you skip the pre-roll and start past that second in stream... all mid-roll DAI period transition thereafter works as expected.

Totally get that this is a Tizen EME spec/implementation issue.
Side note, all the Issues we are having with Dash.js are different but equally annoying. I was a main contributor for years on Dash.js and understand the logic in the lib Most likely not something wrong in that application either. Tizen has so many variations in model years and the shipped firmware and other factors that make this so hard to debug.

Again, thanks so much for the feedback. Anything helps right now.

@dsparacio
Copy link
Contributor Author

@chrisfillmore... forgot to mention..... Dash.js does not have an issue with DAI unexcrypted pre-roll period into ContentProtected CBS period. The lib initilizes DRM without issue. Same TV, same content. Dash.js then fails to stream or be stable thereafter where Shaka is 100% stable post pre-roll.

My point to bring this up is not to say one is better....clearly shaka is! Rather to ask, with 1094, although I have not read fully yet, seems to make it sound like it is not possible without adding Content Protectionin the DAI period?

@TheModMaker
Copy link
Contributor

Dash.js resets the entire media pipeline on Period transitions. This allows it to switch between audio+video and video-only content. We keep the media pipeline initialized and only transition between compatible Periods. The browser may have trouble transitioning between clear and encrypted content and Dash.js is resetting the pipeline so it plays.

IIRC, there was a similar issue before where we couldn't change to fully encrypted content later. Do you add Shaka configuration for encryption (i.e. license server) all the time? After playback starts, do you see video.mediaKeys as null when starting in clear content? When you start with clear content, is a protected Period available in that manifest or does a protected Period get added in a manifest update? If we don't see a protected Period at all, we may not initialize EME and you sometimes can't add EME once playback has started.

@joeyparrish joeyparrish added platform: TV/STB Issues affecting smart TV or set-top box platforms component: ads The issue involves the Shaka Player ads API or the use of other ad SDKs labels Jun 8, 2020
@dsparacio
Copy link
Contributor Author

@TheModMaker -

Do you add Shaka configuration for encryption (i.e. license server) all the time?

Yes, we config Shaka with DAI media url with LA urls for both Widevine and Playready.

After playback starts, do you see video.mediaKeys as null when starting in clear content?

Ill have to run this again and test that. Here is the top part of the log and Ill attach on log when working vs one when failing. Same content, both Tizen, different models. Not even years just models

indexed_db.js:36 IndexedDB.install
input_event.js:35 InputEvent.install
mathround.js:41 mathRound.install
mediasource.js:36 MediaSource.install
video_play_promise.js:35 VideoPlayPromise.install
vttcue.js:36 Using native VTTCue.
player.js:811 Starting attach...
player.js:1009 Starting load of https://dai.google.com/ondemand/dash/content/2497752/vid/19A7D434-8AC8-3835…54E32CBA735B/DLS/streams/330caf08-26eb-4bf5-aaa4-1beec7dc52bb/manifest.mpd...
player.js:1993 Found variant with audio and video content, so filtering out audio-only content in all periods.
drm_engine.js:929 Created MediaKeys object for key system com.microsoft.playready
player.js:2056 codecs avc1-mp4a avg bandwidth 2205296.1666666665
player.js:4138 onChooseStreams_ Object
player.js:4177 Choosing new streams after period changed
streaming_engine.js:458 init: completed initial Stream setup
drm_engine.js:628 Ignoring duplicate init data.
drm_engine.js:628 Ignoring duplicate init data.
drm_engine.js:628 Ignoring duplicate init data.
drm_engine.js:628 Ignoring duplicate init data.

When you start with clear content, is a protected Period available in that manifest or does a protected Period get added in a manifest update?

All of our content periods have both PlayReady and WV ContentProtection.
We do not xlink update anything in the manifest nor are we doing any updates to manifest post first request. So all periods have all the content info already at start.

Dash.js resets the entire media pipeline on Period transitions.

Correct and when you tear down and reinit source buffers like Dash.js does, currerntTime sets back to 0 on video element for an instance. This is a split second time change causes much trouble with our internal AD rules.

@dsparacio
Copy link
Contributor Author

@TheModMaker

Was thinking more about this and just had this come across my mind.

  1. We know Dash.js successfully transitions from “clear to protected” periods on all Samsung models
  2. We know Dash.js forces teardown and recreation of source buffers (all periods),
  3. We know Shaka does not do this for good reason, to have cleaner Multiperiod transitions.
  4. And it seems based on DRM setup should be implied by DRM config, to better support unencrypted ads #1094 that there is a clear known issue on some models of Samsung that will not init DRM properly when a clear period is played first in the SAME source buffer.

So I am wondering, could the Shaka team add a config to set the player in a mode to tear down only the first source buffer when when the first clear to protection period transition happens? then keep it solid the rest of the presentation?

@joeyparrish
Copy link
Member

I'm not sure how we could tear that down cleanly, or how we would make a special case for it. It would be a big change from what we do now.

I'll be trying out your content shortly in Shaka Player v3.0, in which the way we parse and represent multi-period DASH has completely changed. It's possible that this will make a difference.

@joeyparrish joeyparrish self-assigned this Jun 11, 2020
@joeyparrish
Copy link
Member

I get 404 errors for the mp4 segments in v3.0, but not in v2.5.12. I'm testing on ChromeOS as a baseline.

So that means we have a unique issue on v3.0 that may be unrelated.

@dsparacio
Copy link
Contributor Author

@joeyparrish Thank you so much for your help. Let me know how I can help. If you need more device testing on a 3.0 version etc, we have many models available.

@joeyparrish
Copy link
Member

joeyparrish commented Jun 16, 2020

We have fixed a bug in our DASH parser that was present in v3.0.0. Would you be willing/able to test the latest master branch code in your environment to see if this Tizen+DRM+ads issue is improved in master?

@dsparacio
Copy link
Contributor Author

Not at all, @joeyparrish I will run these test tomorrow and get back to you asap.

@dsparacio
Copy link
Contributor Author

dsparacio commented Jun 17, 2020

@joeyparrish Tested both branches listed below both with no change, or same result, freezing, same spot, same log results.

v3.0.x branch version reported in console
shaka.Player.version
"v3.0.0-25-gdaa625d8-debug"

Master branch version reported in console
shaka.Player.version
"v2.5.13-master-15-g34337ca2-debug"

I wanted to make sure I have the right version, master's current version is 2.5.13.

Models Tested
2017 UN50MU6070
2018 UN55NU710D

@dsparacio
Copy link
Contributor Author

All - we've run into the same issue on the XBOX platform with the Shaka player. Identical behavior as samsung with inability to transition from unencrypted ad content to encrypted main content at the preroll only.

Reproduced on these devices:

Sandbox ID: XDKS.1
OS version: 10.0.19041.3068 (rs_xbox_release_2006.200610-0000)
OS edition: June 2020
Xbox Live device ID: FD00A956242E1869
Console ID: 74f233ae.f9265a03.c4fee763.851eb008.01
Serial number: 210633693016
Console type: Xbox One S
Model: 1681
Devkit type: Universal Windows App Devkit
Sandbox ID: XDKS.1
OS version: 10.0.19041.3068 (rs_xbox_release_2006.200610-0000)
OS edition: June 2020
Console ID: A7DCC4D8.6B218CCF.694B68FC.0D3DB308.01
Model: 1681
Console type: Xbox One S
Devkit type: Universal Windows App Devkit
Sandbox ID: XDKS.1
OS version: 10.0.19041.3068 (rs_xbox_release_2006.200610-0000)
OS edition: June 2020
Xbox Live device ID: FD00812AA7534FE0
Console ID: 4c279407.7e461c98.d56bb52f.dfb64007.01
Serial number: 051235192517
Console type: Xbox One X
Devkit type: Universal Windows App Devkit

@joeyparrish
Copy link
Member

Thank you for the extra info. We also have an Xbox One in the lab, but we've never had any automation or remote access for that platform, and we're all still working from home.

The Tizen device in our lab, however, will be accessible again soon. I've been working on repairing the infrastructure that will allow us to access it remotely in automated tests. It won't be as good as interactive debugging, but hopefully, with that and some concentrated effort on reproducing the issue through automation, it might be enough for us to make progress.

We apologize for the inconvenience and the time it has taken us to help with this, and we thank you again for your patience.

@dsparacio
Copy link
Contributor Author

dsparacio commented Jul 9, 2020

@joeyparrish Thank you for the quick response. Totally get it, we are suffering from the same issue here due to access to device labs and remote working. Hopefully this sheds a bit more light on this issue since not confined to Samsung implementation. As always, let me know how our team can help. Thank you for even looking into this for us. Very much appreciated and I know how this goes....

@joeyparrish
Copy link
Member

We now have access to our Tizen device again, at least as far as automated tests in Karma/Jasmine are concerned. I'll try to focus on reproducing this situation in an automated test. After that, we may be able to use that as a platform to debug and make progress on a workaround.

@joeyparrish
Copy link
Member

joeyparrish commented Jul 10, 2020

I'm now able to attempt to play your content using our automated tests. Since this is a long and obscure process, I'm going to try to record all the steps I'm taking in as much detail as I can.

We have a generic, automated playback test that we use on our demo content. For VOD, it plays the first and last 30 seconds the first 30 seconds and last 15 seconds of the content and then checks that there were no errors and that video.ended is true.

build/test.py has arguments for running custom assets through that playback test. It looks roughly like this:

./build/test.py \
  --test-custom-asset MANIFEST_URL \
  --test-custom-license-server com.widevine.alpha=WIDEVINE_SERVER_URL \
  --test-custom-license-server com.microsoft.playready=PLAYREADY_SERVER_URL

There was a bug in the test which prevented --test-custom-license-server from working, but a fix for that is now in review.

Since the MPD URL was not constant, I took the instructions you sent privately for generating the MPD URL, and I built a shell script that outputs just the URL, so I could easily run the test repeatedly.

./build/test.py \
  --test-custom-asset "$(./get-2620-url.sh)" \
  --test-custom-license-server "com.widevine.alpha=$(./get-2620-widevine-server.sh)" \
  --test-custom-license-server "com.microsoft.playready=$(./get-2620-playready-server.sh")

I got an EME failure on IE11 (MSPR_E_INVALID_CDMSESSION, same as #1689), and the test was skipped on Safari (no PlayReady or Widevine), but the content was playable on every other desktop platform, plus Chromecast and Android.

On Tizen, when the test seeks to 30 15 seconds from the end, playback stalls until the test times out. So I'll call that a repro!

@joeyparrish joeyparrish added type: bug Something isn't working correctly and removed needs triage labels Jul 10, 2020
@joeyparrish
Copy link
Member

Test output:

Safari 3.0.0 (Tizen 3.0.0) Player plays DEMO_CUSTOM / custom : .../manifest.mpd FAILED
        Error: Timeout waiting for movement from 1490.662 to 1491.662
            at _class.waitUntilGeneric_ (test/test/util/waiter.js:166:19 <- test/test/util/waiter.js:203:19)
            at _class.waitUntilPlayheadReaches (test/test/util/waiter.js:90:17 <- test/test/util/waiter.js:111:19)
            at _class.waitForMovement (test/test/util/waiter.js:59:17 <- test/test/util/waiter.js:76:19)
            at Function.waitForMovementOrFailOnTimeout (test/test/util/util.js:423:19 <- test/test/util/util.js:682:21)
            at _callee3$ (test/player_external.js:149:24 <- test/player_external.js:213:29)
            at tryCatch (node_modules/babel-polyfill/dist/polyfill.js:6900:40)
            at GeneratorFunctionPrototype.invoke [as _invoke] (node_modules/babel-polyfill/dist/polyfill.js:7138:22)
            at <Jasmine>
            at step (test/player_external.js:3:191)
            at test/player_external.js:3:361
Safari 3.0.0 (Tizen 3.0.0): Executed 1 of 1 (1 FAILED) (57.734 secs / 54.374 secs)

@joeyparrish
Copy link
Member

I added --uncompiled to run tests using the uncompiled sources and --enable-logging=debug to get log output from the device during the test run.

After seeking to close to the end of the stream, we wait for the playhead to advance by 1 second before checking additional expectations. When this happens, the logs show a time of 1490.67. Then the stall detector starts firing, each time with the playhead within a few ms of time 1489.104. Somehow, it went backward between those two events.

When the test times out, the logs show that the ready state is 4 (HAVE_ENOUGH_DATA), we're not in a paused or buffering state, and that the buffered range (1483.114 - 1501.103) includes the current time. The buffered range makes sense in combination with the stall detector firing, because a "stall" is defined there as any time the playhead is inside a buffered range, not paused, and not moving.

The streaming.stallEnabled config defaults to true, except on WebOS. If I expand that so that it also defaults to false on Tizen (lib/util/player_configuration.js), there is no noticeable change except that the stall detector logs go away.

Logs:

LOG LOG: 'Waiting for movement from 1490.67 to 1491.67'
LOG LOG: '(audio:1)', 'clear: handling right now'
LOG LOG: '(audio:1)', 'clearing buffer'
LOG LOG: '(video:7)', 'clear: handling right now'
LOG LOG: '(video:7)', 'clearing buffer'
LOG LOG: '(audio:1)', 'cleared buffer'
LOG LOG: '(video:7)', 'cleared buffer'
WARN LOG: 'No variants met the ABR restrictions. Choosing a variant by lowest bandwidth.'
LOG LOG: 'Calling switch_(), bandwidth=3958 kbps'
LOG LOG: 'switch_'
LOG LOG: 'Stall detected at 1489.104 for 1.002000093460083 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.204'
LOG LOG: 'Stall detected at 1489.106 for 1.001000165939331 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.206'
LOG LOG: 'Stall detected at 1489.104 for 1.001999855041504 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.204'
LOG LOG: 'Stall detected at 1489.104 for 1.001000165939331 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.204'
LOG LOG: 'Stall detected at 1489.106 for 1.000999927520752 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.206'
LOG LOG: 'Stall detected at 1489.107 for 1.002000093460083 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.206999'
LOG LOG: 'Stall detected at 1489.106 for 1.001999855041504 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.206'
LOG LOG: 'Stall detected at 1489.105 for 1.001000165939331 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.205'
LOG LOG: 'Stall detected at 1489.104 for 1.000999927520752 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.204'
LOG LOG: 'Stall detected at 1489.104 for 1.002000093460083 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.204'
LOG LOG: 'Stall detected at 1489.104 for 1.002000093460083 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.204'
LOG LOG: 'Stall detected at 1489.104 for 1.000999927520752 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.204'
LOG LOG: 'Stall detected at 1489.104 for 1.002000093460083 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.204'
LOG LOG: 'Stall detected at 1489.105 for 1.000999927520752 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.205'
LOG LOG: 'Stall detected at 1489.104 for 1.002000093460083 seconds. Seeking forward 0.1 seconds.'
LOG LOG: '(all): seeked: buffered seek: presentationTime=1489.204'
ERROR LOG: 'Timeout waiting for movement from 1490.67 to 1491.67', 'current time', 1489.104, 'duration', 1505.67, 'ready state', 4, 'playback rate', 1, 'paused', false, 'buffered', [Object{start: 1483.114, end: 1501.103}]

@shaka-bot shaka-bot added this to the v3.1 milestone Jul 10, 2020
shaka-bot pushed a commit that referenced this issue Jul 10, 2020
At some point, a refactor of the demo app broke the use of
--test-custom-license-server, which is rarely used, but is being used
now in debugging #2620.

Change-Id: I6376bbf7a0ef9ce650e846cf8df5a6b3bebbfcf8
@joeyparrish
Copy link
Member

Interestingly, none of the seeks triggered by the stall detector had any effect on the playhead position. That may be an indication of the media pipeline being hung in some fundamental way under the hood of the browser.

The content has a ~30-second ad at the beginning, followed by a 30 second encrypted content period. The final period is encrypted main content, so seeking to duration - 15 definitely puts us in an encrypted segment.

I modified the test to pass a start time to load(), to start playback in encrypted main content instead of the first ad. The first main content period has 6-second segments, so I'm starting playback at time 40, which should be in the second segment of the second period, even if there's a rounding error somewhere. This should ensure that the first init segment fed to MediaSource is one that indicates encryption. Setting up EME should be enough, but Tizen TVs are full of bugs, so who knows.

Starting playback at time 40 didn't change the results.

I modified the test again to start at time 1490, close to the end. This way, we will only seek once, only play one period, and never play an ad period.

Playback continued successfully until the end of the content at 1505.

To determine if the difference is the ad period or the extra seek, I started playback toward the beginning of the final period, which is just over 7 minutes long. The start position was 1145.

The content played successfully for 30 seconds from 1145, then the test seeked to 1490 (15 seconds from the end). From there, it stalled again, without ever touching an ad period or any second content period.

(Of course, I'm assuming it didn't touch an ad period, based on the timestamps I fed it, but I haven't verified by watching the URLs it fetched.)

I tried this all with and without the stall detector. I also tried making the stall detector less "spammy" by having it wait for 5 seconds of stall instead of 1, and I tried making it more forceful by seeking 1.0 seconds instead of 0.1 when it detects a stall. None of that made a difference. Seeking by the stall detector was effectively ignored by the platform, and the playhead stayed where it was.

shaka-bot pushed a commit that referenced this issue Jul 23, 2020
The high playbackRate test seems to trigger a Tizen 3 platform bug in
which the decoders fail when the playback rate is above 2x.  We can't
control this as far as I can tell, so this change skips the test on
Tizen.

Found while investigating #2620.

Change-Id: I5b72005c9539c3945dad2cdd7eced8649d7a9cb6
@joeyparrish
Copy link
Member

  1. Ask @dsparacio and @piyush2dev to test the fix in their own applications.

The fix is out in the master branch. Please test in your applications!

These fixes can be backported to v3.0.x and potentially to v2.5.x as well.

@dsparacio
Copy link
Contributor Author

@joeyparrish !! Very nice, we will test this out first thing in morning then figure out which version etc. I have a fork for the other issue going so Ill review the PR and see if I can do a port etc. We are looking to upgrade to 3.0 as well. Keep you informed on this.

@dsparacio
Copy link
Contributor Author

At first round of testing resolved on my system 100%. New issue on other model mentioned in ticket creation, but we are debugging this today to see what it really is. 500 errors on irdeto side so not a shaka issue. We just threw master at 3.1 into our player that was on 2.5.11 with no attention to details so maybe that is the issue or something else. We are slammed with another device issue so we will get info back to you EOD, but this seems to resolve it @joeyparrish thank you so much!

@joeyparrish
Copy link
Member

Happy to help!

joeyparrish added a commit that referenced this issue Jul 27, 2020
At some point, a refactor of the demo app broke the use of
--test-custom-license-server, which is rarely used, but is being used
now in debugging #2620.

Change-Id: I6376bbf7a0ef9ce650e846cf8df5a6b3bebbfcf8
joeyparrish added a commit that referenced this issue Jul 27, 2020
src= playback on Tizen 3 was stalled at time 0, causing automated test
failures.  These went unnoticed while our Tizen device in the lab was
offline.

The only workaround I was able to find was to call load(), which we
recently stopped doing for src= playback.  This adds it back, but only
for Tizen.

Found while investigating #2620.

Change-Id: I4e071a6f1ccb18b9891dfc646a22dbb8c1711eb4
joeyparrish added a commit that referenced this issue Jul 27, 2020
Tizen 3 has the NetworkInformation API, but not the downlink
attribute.  This led to NaN as a bandwidth estimate.  Now we screen
for the presence of this attribute as well as the API overall.

Found while investigating #2620.

Backported to v3.0.x

Change-Id: Iaa1486545825ceee536ecbe5ea617f92de4fbc2d
joeyparrish added a commit that referenced this issue Jul 27, 2020
The high playbackRate test seems to trigger a Tizen 3 platform bug in
which the decoders fail when the playback rate is above 2x.  We can't
control this as far as I can tell, so this change skips the test on
Tizen.

Found while investigating #2620.

Change-Id: I5b72005c9539c3945dad2cdd7eced8649d7a9cb6
joeyparrish added a commit that referenced this issue Jul 27, 2020
Playback stalls on Tizen can be corrected by pausing and playing
again.  For Tizen and similar platforms, this is superior to the old
way of seeking to unstall playback, because seeking is expensive on
those platforms.

To facilitate pausing and playing again instead of seeking, a
stallSkip value of 0 will trigger the pause/play workaround instead of
the seeking workaround.  Further, we will make this the default on
Tizen, WebOS, and Chromecast.

An internal detail which is changing is that GapJumpingController will
now call StallDetector first and inhibit gap jumping when the stall
detector has something to do, rather than the other way around.  This
allows the StallDetector to be in charge of deciding what is buffered,
rather than having to pass GapJumpingController's interpretation of
buffered ranges first.

Fixes #2620

Change-Id: I5ad7aad42aa185b2837c830970e54218b4e55a21
joeyparrish added a commit that referenced this issue Jul 27, 2020
Tizen 3 has the NetworkInformation API, but not the downlink
attribute.  This led to NaN as a bandwidth estimate.  Now we screen
for the presence of this attribute as well as the API overall.

Found while investigating #2620.

Backported to v2.5.x

Change-Id: Iaa1486545825ceee536ecbe5ea617f92de4fbc2d
joeyparrish added a commit that referenced this issue Jul 27, 2020
src= playback on Tizen 3 was stalled at time 0, causing automated test
failures.  These went unnoticed while our Tizen device in the lab was
offline.

The only workaround I was able to find was to call load(), which we
recently stopped doing for src= playback.  This adds it back, but only
for Tizen.

Found while investigating #2620.

Backported to v2.5.x

Change-Id: I4e071a6f1ccb18b9891dfc646a22dbb8c1711eb4
joeyparrish added a commit that referenced this issue Jul 27, 2020
The high playbackRate test seems to trigger a Tizen 3 platform bug in
which the decoders fail when the playback rate is above 2x.  We can't
control this as far as I can tell, so this change skips the test on
Tizen.

Found while investigating #2620.

Backported to v2.5.x

Change-Id: I5b72005c9539c3945dad2cdd7eced8649d7a9cb6
joeyparrish added a commit that referenced this issue Jul 27, 2020
Playback stalls on Tizen can be corrected by pausing and playing
again.  For Tizen and similar platforms, this is superior to the old
way of seeking to unstall playback, because seeking is expensive on
those platforms.

To facilitate pausing and playing again instead of seeking, a
stallSkip value of 0 will trigger the pause/play workaround instead of
the seeking workaround.  Further, we will make this the default on
Tizen, WebOS, and Chromecast.

An internal detail which is changing is that GapJumpingController will
now call StallDetector first and inhibit gap jumping when the stall
detector has something to do, rather than the other way around.  This
allows the StallDetector to be in charge of deciding what is buffered,
rather than having to pass GapJumpingController's interpretation of
buffered ranges first.

Fixes #2620

Backported to v2.5.x

Change-Id: I5ad7aad42aa185b2837c830970e54218b4e55a21
@dsparacio
Copy link
Contributor Author

dsparacio commented Jul 28, 2020

@joeyparrish :( So It is fixed on our models regarding Samsung except 2017 UN50MU6070 Fails 100% still.

However, XBOX it is failing 100% on all boxes. We tested both master version and 2.5.10, both fail exactly the same way as samsung did before you made changes.

Platform Info
XBOX Win/10 Spartan (2014–2019 Edge) non chromium build of edge which is what XBOX dev offers at this time.

I am starting to gather logs for XBOX. I am not sure the one Samsung failure is our main focus at this point. Get back to here soon.

I tried to apply some of your Tizen fixs I saw in master for Edge as well to just test on XBOX. For instance this block

 // on one of these platforms.  Instead, default stallSkip to 0 to force the
    // stall detector to pause and play instead.
    if (shaka.util.Platform.isWebOS() ||
        shaka.util.Platform.isTizen() ||
        shaka.util.Platform.isEdge() ||
        shaka.util.Platform.isChromecast()) {
      streaming.stallSkip = 0;
    }

and the Endianness changes applied.... All of which made no difference for XBOX.

@joeyparrish
Copy link
Member

So It is fixed on our models regarding Samsung except 2017 UN50MU6070 Fails 100% still.
... We tested both master version and 2.5.10 ...

I'm not sure if you made a typo in that version number, but please make sure to test with v2.5.14, or v3.0.2, which have the fix. v2.5.10 does not have the fix. Thanks!

Also, I just checked, and our Tizen device is model UN40MU6300. Hopefully this is representative enough of your UN50MU6070, which appears to be the same family and year.

As for XBox, we have an XBox One in our test lab, but we have no remote access to it at this time. Getting remote access to Tizen took some time, but I was essentially just revamping and repairing existing remote testing infrastructure for Tizen.

We have never had remote access to XB1, unfortunately. I have an idea for how it might work, but I'll have to be physically in the office again for an extended period to get started on that, and I don't think I will get to work on it any time soon.

If v2.5.14 doesn't work on XBox (please double-check!), is there an older version of Shaka Player that plays this content on XB1? Perhaps the previous release branch, such as v2.4.7? If so, we may be able to narrow down the cause somewhat without having to build the testing infrastructure first.

@dsparacio
Copy link
Contributor Author

@joeyparrish sorry not clear, what I mean to say is we tested XB1 with that same master branch built locally as instructed, with the Tizen changes, to see if it helped with XBOX.

We also tested on XBOX with our unaltered production version where we first found this issue 20 days ago.
#2620 (comment)

The reason I tested our current prod version again, is I added an Endianness change (mentioned the other day) that I added for comcast to see if it helped.

If v2.5.14 doesn't work on XBox (please double-check!), is there an older version of Shaka Player that plays this content on XB1? Perhaps the previous release branch, such as v2.4.7? If so, we may be able to narrow down the cause somewhat without having to build the testing infrastructure first.

We will certainly run more test with 2.5.14 version on XBOX and we can certainly do more testing with older version to help find if it works ther. I will get back to you on all of that.

In the meanwhile, should we re-open this ticket for XBOX or should I create a new one?

@joeyparrish
Copy link
Member

Because most of the comments on this issue are focused on Tizen, and because we may not be able to work on XBox at this time, I think opening a separate ticket for XBox would make sense. It may help us keep track of things that way.

@joeyparrish
Copy link
Member

But please let us know if v2.5.14 works on your 2017 Tizen device. It was playing in our tests of our own 2017 device.

@dsparacio
Copy link
Contributor Author

Testing 2.5.14 on Samsung 2017 model and on XBOX for fun as well.

Then we will do XBOX on versions back to 2.4.7, 2.3.9, 2.2.10 and maybe 2.0.1.... all make sense. Let me know if going lower than 2.4.7 does not make sense in your mind for XB1?

I will now great a new ticket with all the XBOX info and from this ticket and add the results for XB1 in that ticket. I will add the result to Samsung 2.5.14 testing in this ticket.

Thanks for your help.

@joeyparrish
Copy link
Member

If 2.3 or under is working on XB1, that's good to know. Finding the relevant change that far back will be much more challenging, but I'd rather know than not know. More information can't hurt, especially when gathering it firsthand is not immediately possible. Thanks!

@dsparacio
Copy link
Contributor Author

We tested 2.5.14 on this 2017 Samsung and get the same negative results.

This model UN50MU6070 , Samsung classifies it as a premium tier 2017 model, but it is the only model in our fleet that we must add "use strict" in the JS code for let and const etc. It is the only TV that would exhibit a slightly different result than the rest of our TVs, both before your Tizen fixes and after, which is shown in this image:

20200724_112920

Most the failing TV before you issued your fixes, would just freeze on last frame of the Advert, no rendering issues displayed.

Another interesting point
(We need to confirm but I believe it is same on Samsung as it is for sure on XBOX)
If you leave the player alone after this issue, and wait long enough. It seems that currentTime is marching on and we reach a mid roll and it PLAYS properly. Then, when we get back to encrypted content, same green decode issue.

By the way, This is also the exact same result we get on XBOX all the time for all versions.
The first frame of content is displayed, but not decoded correctly etc. I will add all the xbox info in a new ticket that I am putting together now.

Log from the Samsung model mentioned above. We are going to spread out the Samsung testing to see if it is ONLY this TV. We just need to coordinate across other teams to do this and will take a bit of time.

cvp.min.js:2 cv-player; Ver. 0.11.0  2020 Jul 28 11:21:17 PM UTC
cvp.min.js:2 [wfh]  00:00:00:022 >  Model prepared
cvp.min.js:2 [wfh]  00:00:00:060 >  Player DOM created.
cvp.min.js:2 [wfh]  00:00:00:065 >  Timer mediator registered
cvp.min.js:2 [wfh]  00:00:00:067 >  PrepView: Skipping UI creation.
cvp.min.js:2 [wfh]  00:00:00:076 >  View prepared
cvp.min.js:2 [wfh]  00:00:00:080 >  Plugins Prepared
cvp.min.js:2 [wfh]  00:00:01:262 >  Html5VideoSurface created
cvp.min.js:2 [wfh]  00:00:01:614 >  Loading shaka external library version: 2.5.14
cvp.min.js:2 [wfh]  00:00:01:618 >  ShakaAdapter created
indexed_db.js:36 IndexedDB.install
input_event.js:35 InputEvent.install
mathround.js:41 mathRound.install
mediasource.js:36 MediaSource.install
video_play_promise.js:35 VideoPlayPromise.install
vttcue.js:36 Using native VTTCue.
index.js:52 EmeEncryptionSchemePolyfill: Waiting to detect encryptionScheme support.
index.js:241 McEncryptionSchemePolyfill: MediaCapabilities not found
cvp.min.js:2 [wfh]  00:00:01:639 >  Shaka version: v2.5.14-debug
player.js:833 Starting attach...
player.js:1031 Starting load of https://dai.google.com/ondemand/dash/content/2497752/vid/dPq7tu1k_gHJVhTSV3…VW76YdW2ss3x/DLS/streams/61d2a4f6-2ad0-4299-ac72-e321e8a02d29/manifest.mpd...
player.js:2183 Found variant with audio and video content, so filtering out audio-only content in all periods.
index.js:96 EmeEncryptionSchemePolyfill: No native encryptionScheme support found. Patching encryptionScheme support.
drm_engine.js:943 Created MediaKeys object for key system com.microsoft.playready
stream_utils.js:98 codecs avc1-mp4a avg bandwidth 2254462.4166666665
player.js:4443 onChooseStreams_ Object {startTime: 0, textStreams: Array[0], variants: Array[6]}
player.js:4482 Choosing new streams after period changed
streaming_engine.js:461 init: completed initial Stream setup
60drm_engine.js:626 Ignoring duplicate init data.
player.js:1792 No preferred audio language set.  We will choose an arbitrary language initially
60drm_engine.js:626 Ignoring duplicate init data.
cvp.min.js:2 [wfh]  00:00:03:285 >  Quality Info:  Object {mode: "auto", quality: null, qualities: Array[6]}
streaming_engine.js:1615 (audio:1) looking up segment: presentationTime=0 currentPeriod.startTime=0
streaming_engine.js:1615 (video:3) looking up segment: presentationTime=0 currentPeriod.startTime=0
60drm_engine.js:626 Ignoring duplicate init data.
drm_engine.js:1360 PlayReady request is already unwrapped.
streaming_engine.js:2160 (audio:1) startup complete
streaming_engine.js:1150 (all) setting up Period 0
streaming_engine.js:1144 (all) Period 0 is being or has been set up
streaming_engine.js:1150 (all) setting up Period 1
streaming_engine.js:1150 (all) setting up Period 2
streaming_engine.js:1150 (all) setting up Period 3
streaming_engine.js:1150 (all) setting up Period 4
streaming_engine.js:1150 (all) setting up Period 5
streaming_engine.js:1150 (all) setting up Period 6
streaming_engine.js:1150 (all) setting up Period 7
streaming_engine.js:1150 (all) setting up Period 8
streaming_engine.js:1150 (all) setting up Period 9
streaming_engine.js:1150 (all) setting up Period 10
streaming_engine.js:1150 (all) setting up Period 11
streaming_engine.js:1150 (all) setting up Period 12
streaming_engine.js:1150 (all) setting up Period 13
streaming_engine.js:1150 (all) setting up Period 14
streaming_engine.js:1150 (all) setting up Period 15
streaming_engine.js:1150 (all) setting up Period 16
streaming_engine.js:1150 (all) setting up Period 17
streaming_engine.js:1150 (all) setting up Period 18
streaming_engine.js:1150 (all) setting up Period 19
streaming_engine.js:1150 (all) setting up Period 20
streaming_engine.js:1150 (all) setting up Period 21
streaming_engine.js:1150 (all) setting up Period 22
streaming_engine.js:1150 (all) setting up Period 23
streaming_engine.js:1150 (all) setting up Period 24
streaming_engine.js:1150 (all) setting up Period 25
streaming_engine.js:1150 (all) setting up Period 26
streaming_engine.js:1150 (all) setting up Period 27
streaming_engine.js:1150 (all) setting up Period 28
streaming_engine.js:1150 (all) setting up Period 29
streaming_engine.js:1218 (all) Stream 1 is being or has been set up
streaming_engine.js:1218 (all) Stream 3 is being or has been set up
player.js:4588 canSwitch_
simple_abr_manager.js:254 Calling switch_(), bandwidth=2841 kbps
player.js:4652 switch_
streaming_engine.js:807 switch: switching to Stream (video:4)
streaming_engine.js:1615 (video:4) looking up segment: presentationTime=5.005 currentPeriod.startTime=0
streaming_engine.js:790 switch: Stream (audio:1) already active
streaming_engine.js:1615 (video:4) looking up segment: presentationTime=10.01 currentPeriod.startTime=0
cvp.min.js:2 [wfh]  00:00:05:269 >  metadata received - id: urn:google:dai:2018 msg google_4427488288096268164
streaming_engine.js:1435 (audio:1) need Period 1 presentationTime=0.76 timeNeeded=30.016 currentPeriodIndex=0
streaming_engine.js:2240 (audio:1) not all MediaStates need Period 1
streaming_engine.js:1435 (video:4) need Period 1 presentationTime=5.48 timeNeeded=30.071 currentPeriodIndex=0
streaming_engine.js:2255 (video:4) all need Period 1
streaming_engine.js:1144 (all) Period 1 is being or has been set up
player.js:4443 onChooseStreams_ Object {startTime: 30.071, textStreams: Array[1], variants: Array[6]}
player.js:4482 Choosing new streams after period changed
streaming_engine.js:807 switch: switching to Stream (audio:21)
streaming_engine.js:807 switch: switching to Stream (video:19)
player.js:4588 canSwitch_
cvp.min.js:2 [wfh]  00:00:12:792 >  metadata received - id: urn:google:dai:2018 msg google_2630202043314982722
streaming_engine.js:1615 (audio:21) looking up segment: presentationTime=30.016 currentPeriod.startTime=30.071
streaming_engine.js:1615 (video:19) looking up segment: presentationTime=30.071 currentPeriod.startTime=30.071
cvp.min.js:2 [wfh]  00:00:19:831 >  metadata received - id: urn:google:dai:2018 msg google_7062123705125723286
simple_abr_manager.js:254 Calling switch_(), bandwidth=18865 kbps
player.js:4652 switch_
streaming_engine.js:790 switch: Stream (video:19) already active
streaming_engine.js:790 switch: Stream (audio:21) already active
cvp.min.js:2 [wfh]  00:00:27:591 >  metadata received - id: urn:google:dai:2018 msg google_0088992214810277424
simple_abr_manager.js:254 Calling switch_(), bandwidth=26582 kbps
player.js:4652 switch_
streaming_engine.js:790 switch: Stream (video:19) already active
streaming_engine.js:790 switch: Stream (audio:21) already active
cvp.min.js:2 [wfh]  00:00:34:101 >  metadata received - id: urn:google:dai:2018 msg google_5115618755624134587
gap_jumping_controller.js:246 Jumping forward 0.46799999999999997 seconds because of gap starting at 30.015 and ending at 30.071
streaming_engine.js:964 (all): seeked: buffered seek: presentationTime=30.071
60drm_engine.js:626 Ignoring duplicate init data.
streaming_engine.js:730 Assertion failed: switch: expected mediaState to existshaka.media.StreamingEngine.switchInternal_ @ streaming_engine.js:730shaka.media.StreamingEngine.switchTextStream @ streaming_engine.js:691shaka.Player.switchTextStream_ @ player.js:4085shaka.Player.selectTextTrack @ player.js:3013Object.defineProperty.set @ cvp.min.js:2A.updateTextTracks @ cvp.min.js:2A.onTracksChanged @ cvp.min.js:2shaka.util.FakeEventTarget.dispatchEvent @ fake_event_target.js:120(anonymous function) @ player.js:5050$jscomp.generator.Engine_.nextStep_ @ generator_engine] :795$jscomp.generator.Engine_.next_ @ generator_engine] :699next @ generator_engine] :833b @ execute_async_generator] :48
simple_abr_manager.js:254 Calling switch_(), bandwidth=28317 kbps
player.js:4652 switch_
streaming_engine.js:790 switch: Stream (video:19) already active
streaming_engine.js:790 switch: Stream (audio:21) already active
simple_abr_manager.js:254 Calling switch_(), bandwidth=25216 kbps
player.js:4652 switch_
streaming_engine.js:790 switch: Stream (video:19) already active
streaming_engine.js:790 switch: Stream (audio:21) already active
simple_abr_manager.js:254 Calling switch_(), bandwidth=20040 kbps
player.js:4652 switch_
streaming_engine.js:790 switch: Stream (video:19) already active
streaming_engine.js:790 switch: Stream (audio:21) already active
simple_abr_manager.js:254 Calling switch_(), bandwidth=20931 kbps
player.js:4652 switch_
streaming_engine.js:790 switch: Stream (video:19) already active
streaming_engine.js:790 switch: Stream (audio:21) already active
simple_abr_manager.js:254 Calling switch_(), bandwidth=22566 kbps
player.js:4652 switch_
streaming_engine.js:790 switch: Stream (video:19) already active
streaming_engine.js:790 switch: Stream (audio:21) already active

@joeyparrish
Copy link
Member

The garbled content with big green blocks appears to be a decryption or decode issue. I don't believe it has any relation to the stall we were working on in this issue, so I would prefer not to reopen this one. But I think I have a lead on that, and will respond on #2759 with details.

@dsparacio
Copy link
Contributor Author

@joeyparrish Thank you, totally agree on not reopening this, just read the other comment for XBOX.

@shaka-project shaka-project locked and limited conversation to collaborators Sep 21, 2020
@shaka-bot shaka-bot added the status: archived Archived and locked; will not be updated label Apr 15, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
component: ads The issue involves the Shaka Player ads API or the use of other ad SDKs platform: TV/STB Issues affecting smart TV or set-top box platforms status: archived Archived and locked; will not be updated type: bug Something isn't working correctly
Projects
None yet
Development

No branches or pull requests

6 participants