Ld Json Schema Scraper #50

shaniqwa · 2021-08-15T12:51:26Z

adding support for a huge variety of recipe websites worldwide, this scraper is looking for the google recipe JSON schema and populates the recipe object.

If the domain is not found in the scraper factory, the default is to use the new JSON scraper. if it fails to find the schema, the default error will return ('No recipe found in page').
Added structured instruction, so that each step can include optional image and title if found. this is added in addition to the existing instructions array to not insert a breaking change.
Fixed broken tests, some scrapers now use the JSON scraper.
Added support and some tests for major recipe websites requested in issues: Big recipe sites missing! #32 Add german recipe sites #36 New website for scrap https://www.vegrecipesofindia.com/recipes/ #48 I want to make a recipe app for me and my gf. Can you please add "nosalty.hu" on the list? #40
Also added tests for some major local (my local language is Hebrew) recipe websites
I wrote most of the logic in the Base Scraper in order to share this logic with existing scrapers.

Happy to hear your thoughts and add improvements
Cheers, Shani

add logs

…since server response is 403

fix tests: mels kitchen now using ld json scraper add expected results to all ld json scraper test cases fix bug/edge case handling in description and servings

add Json+ld scraper

…ue (pav-bhaji-recipe-mumbai the longest recipe ever!!)

add support for HowToSection. add test for a website requested in iss…

Json+ld

… image & description meta data

* master:

…ult ld json

shaniqwa added 30 commits April 28, 2021 15:06

support jamie oliver scraping

6fdf15a

fix some tests. add defaultSetDescription to BaseScraper

ecbe2bd

fix more tests

cda11c4

rename

8669a34

fix SimplyRecipes tests

e451674

fix yummly. update puppeteer to latest version

a005ae9

fix thespruceeats tests

b39a9f6

theSpruceEast - add recipe description

4db4c1d

fix woolworths

55d8d9f

closetCooking add description

27e38f9

try to change travis node version

00ba239

revert travis node version to stable

6b217c2

add logs

more logs

caa38c9

logs

baf8a87

log

f6ff471

log

affc05a

log

70314e3

skip should fetch the expected recipe test if server is not responding

f04e3a4

add ambitiouskitchen description

20583c3

add recipetineats description

4736af1

add GimmeDelicious description

b47e379

add tastOfHome description

1e07117

add SeriousEats description

71e1029

add therecipecritic description

7045020

add myrecipes description

f426a0f

bonappetit add descriptions and tags

7a7e1ae

tastesbetterfromscratch add description and tags

5821206

kitchen stories - add description and tags

dd2f66a

foodandwine add description

972b953

NomNomPaleo add description

60c7803

shaniqwa added 30 commits August 9, 2021 17:50

woolworths should use the ld json schema, but it's still not working …

e651c01

…since server response is 403

remove woolworths test

d776a87

fix handling unsupported / invalid urls

f4533ac

html decode - add regex for removing specific short cods

fbad66d

fix tests: mels kitchen now using ld json scraper add expected results to all ld json scraper test cases fix bug/edge case handling in description and servings

fix test

ee6379c

Merge pull request #2 from shaniqwa/JSON+LD

daa4b0a

add Json+ld scraper

add support for HowToSection. add test for a website requested in iss…

ba11390

…ue (pav-bhaji-recipe-mumbai the longest recipe ever!!)

Merge pull request #3 from shaniqwa/JSON+LD

2ca3b61

add support for HowToSection. add test for a website requested in iss…

fix typo

00ae10b

add logs

36be742

Throw errors

7485a33

update puppeteer to latest version

bd18d28

don't use puppeteer in json scraper

89e26fb

remove logs, fix test

1a85858

fix tests

d89b6d9

Merge pull request #4 from shaniqwa/JSON+LD

48c8413

Json+ld

if no recipe schema was found, do not throw error. return page title,…

d53f5cc

… image & description meta data

Merge branch 'master' into JSON+LD

97afad1

* master:

fix all tests

bf64b78

fix 404 handling

0aac237

fix test. make expected test to match 80%. more scrapers now use defa…

caa800f

…ult ld json

more fixes

5e7223e

change to a proper module. update dependencies

e0c1e34

remove puppeteer

28ddf8a

fix main path

8e2de13

fix files

72286a3

fix file path import

fc39ded

fix

c357254

remove console.logs

d6e7250

bump version

d4acc04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ld Json Schema Scraper #50

Ld Json Schema Scraper #50

shaniqwa commented Aug 15, 2021 •

edited

Loading

Ld Json Schema Scraper #50

Are you sure you want to change the base?

Ld Json Schema Scraper #50

Conversation

shaniqwa commented Aug 15, 2021 • edited Loading

shaniqwa commented Aug 15, 2021 •

edited

Loading