Missing file type identifyer for images

[SWalter1]

Describe the solution you'd like

I'm interested in accessing images and noticed that the URLs included in tweets$entities$urls linking to images are missing a file type identifier (e.g. .jpg). Instead, the link leads back to the tweet in which the image is included. It would be great if this could be fixed in the future, thanks!

Anything else?

No response

[chainsawriot]

@SWalter1 I want to let you know that the image URLs are there. But

1. You need to set context_annotations to TRUE
2. It is quite convoluted to extract the URLs. You can keep those ends with "orig".

require(academictwitteR)
#> Loading required package: academictwitteR
x <- get_all_tweets("from:ScienceMagazine", 
                    start_tweets = "2021-11-01T00:00:00Z", end_tweets = "2021-11-03T00:00:00Z",
                    n = 15,
                    context_annotations = TRUE)
#> Warning: Recommended to specify a data path in order to mitigate data loss when
#> ingesting large amounts of data.
#> Warning: Tweets will not be stored as JSONs or as a .rds file and will only be
#> available in local memory if assigned to an object.
#> page_n is limited to 100 due to the restriction imposed by Twitter API
#> query:  from:ScienceMagazine 
#> Total pages queried: 1 (tweets captured this page: 27).
#> Total tweets captured now reach 15 : finishing collection.
purrr::keep(x$entities$urls, ~"images" %in% colnames(.)) %>% purrr::map_dfr(~.) -> images
images$images %>% purrr::discard(is.null) %>% purrr::map_dfr(~.) 
#>                                                                                   url
#> 1    https://pbs.twimg.com/news_img/1456392373404057608/RDlEY3M9?format=jpg&name=orig
#> 2 https://pbs.twimg.com/news_img/1456392373404057608/RDlEY3M9?format=jpg&name=150x150
#> 3    https://pbs.twimg.com/news_img/1456362079045263371/O6OpYWZY?format=jpg&name=orig
#> 4 https://pbs.twimg.com/news_img/1456362079045263371/O6OpYWZY?format=jpg&name=150x150
#>   width height
#> 1   715    550
#> 2   150    150
#> 3   715    550
#> 4   150    150

^{Created on 2021-11-10 by the reprex package (v2.0.0)}

[cjbarrie]

Thanks for this @chainsawriot i was struggling to work out where the image urls were lurking as well

[IsaacBravo]

I am having problems accessing the image, because when I connect to the url I get a 404 error. I am using this same approach, a few months ago it worked perfectly, but now working on the same data, it is not possible to access the image. Is it an update of the API or the package? @chainsawriot. Thank you very much for any help!

[cjbarrie]

Please provide us with a reproducible example of what you're trying to do.

Without that, it's hard for us to tell what the issue is or whether it belongs on this Issue thread

[chainsawriot]

@IsaacBravo It is highly possible that this is a problem at the Twitter side.

https://twittercommunity.com/t/entity-image-urls-from-tweets-lookup-api-returns-404/166948

I've tried with my Science Mag example and indeed some URLs won't return an image. I guess you would get a better answer at the Twitter developer portal.

[IsaacBravo]

@cjbarrie I did the same example that did @chainsawriot, but with a different data, and when I want to download the image, I get this error:

Thank you for your quick response @cjbarrie @chainsawriot! I will ask through the Twitter developer portal.

[david-schweizer]

Hi @IsaacBravo,

did you find a workaround? I am having the same problem.

It seems as if the URL (esp. this part .../news_img/...) is the problem. Looks like Twitter changed the format to .../media/...

Using the academictwitteR package, the URL to the images is stored as a list in the raw format under "tweet.entities.urls$images (which for some reason gives: .../news_img/...). Currently, the raw output has only Tweet ID and media key under "tweet.attachments.media_keys" (I think here should be the right URL if queried).

I found that you can get the right URL via these parameters:

params <- list(
  expansions = "attachments.media_keys",
  "media.fields" = "url")

For now, I can check which Tweets have a media key, select them, and get the right URL by making a call for each Tweet to the APIv2. However, I run quickly into problems with the rate limit, since I want to access over 100.000 Tweets.

Therefore, I wanted to ask @cjbarrie, @chainsawriot: Do you still update the package? If yes, it would be great if the expansion with the right URL could be added :)

Best,
David