-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ViacomCBS] Samsung fatal stall on preroll to first content period. #2620
Comments
One update here 2018 UN55NU710D Fails 100% Fails always at first period break... and stays there until page app is closed on TV. never picks up. |
There are several issues related to ad playback on Tizen platforms, you can take a look at #1094 and follow the thread. Ultimately, even with fixes in the JS/app layer, we still found we had playback problems on Tizen devices if we started playing clear content before unencrypted content. We attributed this to bug(s) in the Tizen media layer. So, our solution was to force playback to start during protected content. Are you able to skip your prerolls? I'll bet this would solve the problem. |
@chrisfillmore Thanks for the added info and point me to that ticket. Same experience.
Totally get that this is a Tizen EME spec/implementation issue. Again, thanks so much for the feedback. Anything helps right now. |
@chrisfillmore... forgot to mention..... Dash.js does not have an issue with DAI unexcrypted pre-roll period into ContentProtected CBS period. The lib initilizes DRM without issue. Same TV, same content. Dash.js then fails to stream or be stable thereafter where Shaka is 100% stable post pre-roll. My point to bring this up is not to say one is better....clearly shaka is! Rather to ask, with 1094, although I have not read fully yet, seems to make it sound like it is not possible without adding Content Protectionin the DAI period? |
Dash.js resets the entire media pipeline on Period transitions. This allows it to switch between audio+video and video-only content. We keep the media pipeline initialized and only transition between compatible Periods. The browser may have trouble transitioning between clear and encrypted content and Dash.js is resetting the pipeline so it plays. IIRC, there was a similar issue before where we couldn't change to fully encrypted content later. Do you add Shaka configuration for encryption (i.e. license server) all the time? After playback starts, do you see |
Yes, we config Shaka with DAI media url with LA urls for both Widevine and Playready.
Ill have to run this again and test that. Here is the top part of the log and Ill attach on log when working vs one when failing. Same content, both Tizen, different models. Not even years just models
All of our content periods have both PlayReady and WV ContentProtection.
Correct and when you tear down and reinit source buffers like Dash.js does, currerntTime sets back to 0 on video element for an instance. This is a split second time change causes much trouble with our internal AD rules. |
Was thinking more about this and just had this come across my mind.
So I am wondering, could the Shaka team add a config to set the player in a mode to tear down only the first source buffer when when the first clear to protection period transition happens? then keep it solid the rest of the presentation? |
I'm not sure how we could tear that down cleanly, or how we would make a special case for it. It would be a big change from what we do now. I'll be trying out your content shortly in Shaka Player v3.0, in which the way we parse and represent multi-period DASH has completely changed. It's possible that this will make a difference. |
I get 404 errors for the mp4 segments in v3.0, but not in v2.5.12. I'm testing on ChromeOS as a baseline. So that means we have a unique issue on v3.0 that may be unrelated. |
@joeyparrish Thank you so much for your help. Let me know how I can help. If you need more device testing on a 3.0 version etc, we have many models available. |
We have fixed a bug in our DASH parser that was present in v3.0.0. Would you be willing/able to test the latest |
Not at all, @joeyparrish I will run these test tomorrow and get back to you asap. |
@joeyparrish Tested both branches listed below both with no change, or same result, freezing, same spot, same log results. v3.0.x branch version reported in console Master branch version reported in console I wanted to make sure I have the right version, master's current version is 2.5.13. Models Tested |
All - we've run into the same issue on the XBOX platform with the Shaka player. Identical behavior as samsung with inability to transition from unencrypted ad content to encrypted main content at the preroll only. Reproduced on these devices:
|
Thank you for the extra info. We also have an Xbox One in the lab, but we've never had any automation or remote access for that platform, and we're all still working from home. The Tizen device in our lab, however, will be accessible again soon. I've been working on repairing the infrastructure that will allow us to access it remotely in automated tests. It won't be as good as interactive debugging, but hopefully, with that and some concentrated effort on reproducing the issue through automation, it might be enough for us to make progress. We apologize for the inconvenience and the time it has taken us to help with this, and we thank you again for your patience. |
@joeyparrish Thank you for the quick response. Totally get it, we are suffering from the same issue here due to access to device labs and remote working. Hopefully this sheds a bit more light on this issue since not confined to Samsung implementation. As always, let me know how our team can help. Thank you for even looking into this for us. Very much appreciated and I know how this goes.... |
We now have access to our Tizen device again, at least as far as automated tests in Karma/Jasmine are concerned. I'll try to focus on reproducing this situation in an automated test. After that, we may be able to use that as a platform to debug and make progress on a workaround. |
I'm now able to attempt to play your content using our automated tests. Since this is a long and obscure process, I'm going to try to record all the steps I'm taking in as much detail as I can. We have a generic, automated playback test that we use on our demo content. For VOD, it plays
./build/test.py \
--test-custom-asset MANIFEST_URL \
--test-custom-license-server com.widevine.alpha=WIDEVINE_SERVER_URL \
--test-custom-license-server com.microsoft.playready=PLAYREADY_SERVER_URL There was a bug in the test which prevented Since the MPD URL was not constant, I took the instructions you sent privately for generating the MPD URL, and I built a shell script that outputs just the URL, so I could easily run the test repeatedly. ./build/test.py \
--test-custom-asset "$(./get-2620-url.sh)" \
--test-custom-license-server "com.widevine.alpha=$(./get-2620-widevine-server.sh)" \
--test-custom-license-server "com.microsoft.playready=$(./get-2620-playready-server.sh") I got an EME failure on IE11 (MSPR_E_INVALID_CDMSESSION, same as #1689), and the test was skipped on Safari (no PlayReady or Widevine), but the content was playable on every other desktop platform, plus Chromecast and Android. On Tizen, when the test seeks to |
Test output:
|
I added After seeking to close to the end of the stream, we wait for the playhead to advance by 1 second before checking additional expectations. When this happens, the logs show a time of When the test times out, the logs show that the ready state is 4 ( The Logs:
|
At some point, a refactor of the demo app broke the use of --test-custom-license-server, which is rarely used, but is being used now in debugging #2620. Change-Id: I6376bbf7a0ef9ce650e846cf8df5a6b3bebbfcf8
Interestingly, none of the seeks triggered by the stall detector had any effect on the playhead position. That may be an indication of the media pipeline being hung in some fundamental way under the hood of the browser. The content has a ~30-second ad at the beginning, followed by a 30 second encrypted content period. The final period is encrypted main content, so seeking to I modified the test to pass a start time to Starting playback at time 40 didn't change the results. I modified the test again to start at time 1490, close to the end. This way, we will only seek once, only play one period, and never play an ad period. Playback continued successfully until the end of the content at 1505. To determine if the difference is the ad period or the extra seek, I started playback toward the beginning of the final period, which is just over 7 minutes long. The start position was 1145. The content played successfully for 30 seconds from 1145, then the test seeked to 1490 (15 seconds from the end). From there, it stalled again, without ever touching an ad period or any second content period. (Of course, I'm assuming it didn't touch an ad period, based on the timestamps I fed it, but I haven't verified by watching the URLs it fetched.) I tried this all with and without the stall detector. I also tried making the stall detector less "spammy" by having it wait for 5 seconds of stall instead of 1, and I tried making it more forceful by seeking 1.0 seconds instead of 0.1 when it detects a stall. None of that made a difference. Seeking by the stall detector was effectively ignored by the platform, and the playhead stayed where it was. |
The high playbackRate test seems to trigger a Tizen 3 platform bug in which the decoders fail when the playback rate is above 2x. We can't control this as far as I can tell, so this change skips the test on Tizen. Found while investigating #2620. Change-Id: I5b72005c9539c3945dad2cdd7eced8649d7a9cb6
The fix is out in the master branch. Please test in your applications! These fixes can be backported to v3.0.x and potentially to v2.5.x as well. |
@joeyparrish !! Very nice, we will test this out first thing in morning then figure out which version etc. I have a fork for the other issue going so Ill review the PR and see if I can do a port etc. We are looking to upgrade to 3.0 as well. Keep you informed on this. |
At first round of testing resolved on my system 100%. New issue on other model mentioned in ticket creation, but we are debugging this today to see what it really is. 500 errors on irdeto side so not a shaka issue. We just threw master at 3.1 into our player that was on 2.5.11 with no attention to details so maybe that is the issue or something else. We are slammed with another device issue so we will get info back to you EOD, but this seems to resolve it @joeyparrish thank you so much! |
Happy to help! |
At some point, a refactor of the demo app broke the use of --test-custom-license-server, which is rarely used, but is being used now in debugging #2620. Change-Id: I6376bbf7a0ef9ce650e846cf8df5a6b3bebbfcf8
src= playback on Tizen 3 was stalled at time 0, causing automated test failures. These went unnoticed while our Tizen device in the lab was offline. The only workaround I was able to find was to call load(), which we recently stopped doing for src= playback. This adds it back, but only for Tizen. Found while investigating #2620. Change-Id: I4e071a6f1ccb18b9891dfc646a22dbb8c1711eb4
Tizen 3 has the NetworkInformation API, but not the downlink attribute. This led to NaN as a bandwidth estimate. Now we screen for the presence of this attribute as well as the API overall. Found while investigating #2620. Backported to v3.0.x Change-Id: Iaa1486545825ceee536ecbe5ea617f92de4fbc2d
The high playbackRate test seems to trigger a Tizen 3 platform bug in which the decoders fail when the playback rate is above 2x. We can't control this as far as I can tell, so this change skips the test on Tizen. Found while investigating #2620. Change-Id: I5b72005c9539c3945dad2cdd7eced8649d7a9cb6
Playback stalls on Tizen can be corrected by pausing and playing again. For Tizen and similar platforms, this is superior to the old way of seeking to unstall playback, because seeking is expensive on those platforms. To facilitate pausing and playing again instead of seeking, a stallSkip value of 0 will trigger the pause/play workaround instead of the seeking workaround. Further, we will make this the default on Tizen, WebOS, and Chromecast. An internal detail which is changing is that GapJumpingController will now call StallDetector first and inhibit gap jumping when the stall detector has something to do, rather than the other way around. This allows the StallDetector to be in charge of deciding what is buffered, rather than having to pass GapJumpingController's interpretation of buffered ranges first. Fixes #2620 Change-Id: I5ad7aad42aa185b2837c830970e54218b4e55a21
Tizen 3 has the NetworkInformation API, but not the downlink attribute. This led to NaN as a bandwidth estimate. Now we screen for the presence of this attribute as well as the API overall. Found while investigating #2620. Backported to v2.5.x Change-Id: Iaa1486545825ceee536ecbe5ea617f92de4fbc2d
src= playback on Tizen 3 was stalled at time 0, causing automated test failures. These went unnoticed while our Tizen device in the lab was offline. The only workaround I was able to find was to call load(), which we recently stopped doing for src= playback. This adds it back, but only for Tizen. Found while investigating #2620. Backported to v2.5.x Change-Id: I4e071a6f1ccb18b9891dfc646a22dbb8c1711eb4
The high playbackRate test seems to trigger a Tizen 3 platform bug in which the decoders fail when the playback rate is above 2x. We can't control this as far as I can tell, so this change skips the test on Tizen. Found while investigating #2620. Backported to v2.5.x Change-Id: I5b72005c9539c3945dad2cdd7eced8649d7a9cb6
Playback stalls on Tizen can be corrected by pausing and playing again. For Tizen and similar platforms, this is superior to the old way of seeking to unstall playback, because seeking is expensive on those platforms. To facilitate pausing and playing again instead of seeking, a stallSkip value of 0 will trigger the pause/play workaround instead of the seeking workaround. Further, we will make this the default on Tizen, WebOS, and Chromecast. An internal detail which is changing is that GapJumpingController will now call StallDetector first and inhibit gap jumping when the stall detector has something to do, rather than the other way around. This allows the StallDetector to be in charge of deciding what is buffered, rather than having to pass GapJumpingController's interpretation of buffered ranges first. Fixes #2620 Backported to v2.5.x Change-Id: I5ad7aad42aa185b2837c830970e54218b4e55a21
@joeyparrish :( So It is fixed on our models regarding Samsung except 2017 UN50MU6070 Fails 100% still. However, XBOX it is failing 100% on all boxes. We tested both master version and 2.5.10, both fail exactly the same way as samsung did before you made changes. Platform Info I am starting to gather logs for XBOX. I am not sure the one Samsung failure is our main focus at this point. Get back to here soon. I tried to apply some of your Tizen fixs I saw in master for Edge as well to just test on XBOX. For instance this block
and the Endianness changes applied.... All of which made no difference for XBOX. |
I'm not sure if you made a typo in that version number, but please make sure to test with v2.5.14, or v3.0.2, which have the fix. v2.5.10 does not have the fix. Thanks! Also, I just checked, and our Tizen device is model UN40MU6300. Hopefully this is representative enough of your UN50MU6070, which appears to be the same family and year. As for XBox, we have an XBox One in our test lab, but we have no remote access to it at this time. Getting remote access to Tizen took some time, but I was essentially just revamping and repairing existing remote testing infrastructure for Tizen. We have never had remote access to XB1, unfortunately. I have an idea for how it might work, but I'll have to be physically in the office again for an extended period to get started on that, and I don't think I will get to work on it any time soon. If v2.5.14 doesn't work on XBox (please double-check!), is there an older version of Shaka Player that plays this content on XB1? Perhaps the previous release branch, such as v2.4.7? If so, we may be able to narrow down the cause somewhat without having to build the testing infrastructure first. |
@joeyparrish sorry not clear, what I mean to say is we tested XB1 with that same master branch built locally as instructed, with the Tizen changes, to see if it helped with XBOX. We also tested on XBOX with our unaltered production version where we first found this issue 20 days ago. The reason I tested our current prod version again, is I added an Endianness change (mentioned the other day) that I added for comcast to see if it helped.
We will certainly run more test with 2.5.14 version on XBOX and we can certainly do more testing with older version to help find if it works ther. I will get back to you on all of that. In the meanwhile, should we re-open this ticket for XBOX or should I create a new one? |
Because most of the comments on this issue are focused on Tizen, and because we may not be able to work on XBox at this time, I think opening a separate ticket for XBox would make sense. It may help us keep track of things that way. |
But please let us know if v2.5.14 works on your 2017 Tizen device. It was playing in our tests of our own 2017 device. |
Testing 2.5.14 on Samsung 2017 model and on XBOX for fun as well. Then we will do XBOX on versions back to 2.4.7, 2.3.9, 2.2.10 and maybe 2.0.1.... all make sense. Let me know if going lower than 2.4.7 does not make sense in your mind for XB1? I will now great a new ticket with all the XBOX info and from this ticket and add the results for XB1 in that ticket. I will add the result to Samsung 2.5.14 testing in this ticket. Thanks for your help. |
If 2.3 or under is working on XB1, that's good to know. Finding the relevant change that far back will be much more challenging, but I'd rather know than not know. More information can't hurt, especially when gathering it firsthand is not immediately possible. Thanks! |
We tested 2.5.14 on this 2017 Samsung and get the same negative results. This model UN50MU6070 , Samsung classifies it as a premium tier 2017 model, but it is the only model in our fleet that we must add "use strict" in the JS code for let and const etc. It is the only TV that would exhibit a slightly different result than the rest of our TVs, both before your Tizen fixes and after, which is shown in this image: Most the failing TV before you issued your fixes, would just freeze on last frame of the Advert, no rendering issues displayed. Another interesting point By the way, This is also the exact same result we get on XBOX all the time for all versions. Log from the Samsung model mentioned above. We are going to spread out the Samsung testing to see if it is ONLY this TV. We just need to coordinate across other teams to do this and will take a bit of time.
|
The garbled content with big green blocks appears to be a decryption or decode issue. I don't believe it has any relation to the stall we were working on in this issue, so I would prefer not to reopen this one. But I think I have a lead on that, and will respond on #2759 with details. |
@joeyparrish Thank you, totally agree on not reopening this, just read the other comment for XBOX. |
Have you read the FAQ and checked for duplicate open issues?
Yes completely.
What version of Shaka Player are you using?
Tested in Shaka 2.5.12, 2.5.11, 2.5.5 and 2.5.1 - Same results on all
Can you reproduce the issue with our latest release version?
Yes
Can you reproduce the issue with the latest code from
master
?Yes
Are you using the demo app or your own custom app?
Demo raw simple setup. Page urls included in support email with content and la urls.
If custom app, can you reproduce the issue using our demo app?
yes
What browser and OS are you using?
This issue is only being reported and reproduced by our team for SOME Samsung Tizen based Smart TV mainly years 2017 and 2018. 2019 and 2020 year seems unaffected. Some 2017 and 2018 are also working as expected!!
For embedded devices (smart TVs, etc.), what model and firmware version are you using?
What are the manifest and license server URIs?
Sent over on shaka-player-issues@google.com
What did you do?
I am also including logs in the support email with content. These logs will be from both fail and pass captures in a few different versions of Shaka mentioned above.
What did you expect to happen?
The first content period initializes and plays out.
What actually happened?
The content period fails to init and play properly.
The text was updated successfully, but these errors were encountered: