Skip to content

Commit

Permalink
Update data
Browse files Browse the repository at this point in the history
  • Loading branch information
@actions-user
actions-user committed Jul 1, 2023
1 parent 83ceac1 commit 79662b0
Show file tree
Hide file tree
Showing 9 changed files with 326 additions and 153 deletions.
8 changes: 4 additions & 4 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ Combining lists of UK Government organisations to make useful CSV files. Automat

Data on UK Government organisations is published in a couple of places under the Open Government Licence:

* [gov.uk/government/organisations](https://www.gov.uk/government/organisations) - this is an HTML page listing <!--GB-GOVUK-->602<!--END GB-GOVUK--> departments, agencies and public bodies. It isn't clear how up-to-date this source is. It has a note if the organisation has a separate website (separate to gov.uk/government/organisations) although these separate website URLs can only be found by following the links and extracting the link from the subsequent webpage. These "real" URLs are saved in [gov.uk_government_organisations-replacement-urls.csv](gov.uk_government_organisations-replacement-urls).
* [gov.uk/api/organisations](https://www.gov.uk/api/organisations) - this is a paged JSON endpoint that provides more data on <!--GB-GOVUKAPI-TOTAL-->1179<!--END GB-GOVUKAPI-TOTAL--> organisations including some which are no longer operating. It provides an ID that seems to match [GB-GOR](http://org-id.guide/list/GB-GOR) as well as useful metadata about update/closed dates.
* [gov.uk/government/organisations](https://www.gov.uk/government/organisations) - this is an HTML page listing <!--GB-GOVUK-->603<!--END GB-GOVUK--> departments, agencies and public bodies. It isn't clear how up-to-date this source is. It has a note if the organisation has a separate website (separate to gov.uk/government/organisations) although these separate website URLs can only be found by following the links and extracting the link from the subsequent webpage. These "real" URLs are saved in [gov.uk_government_organisations-replacement-urls.csv](gov.uk_government_organisations-replacement-urls).
* [gov.uk/api/organisations](https://www.gov.uk/api/organisations) - this is a paged JSON endpoint that provides more data on <!--GB-GOVUKAPI-TOTAL-->1181<!--END GB-GOVUKAPI-TOTAL--> organisations including some which are no longer operating. It provides an ID that seems to match [GB-GOR](http://org-id.guide/list/GB-GOR) as well as useful metadata about update/closed dates.

It isn't clear if these use the same source or are independently maintained.

Expand All @@ -16,7 +16,7 @@ It isn't clear if these use the same source or are independently maintained.
The [get.pl](get.pl) perl script compiles the data from the two sources (HTML/JSON) into several output files. The [gov.uk_government_organisations-replacement-urls.csv](gov.uk_government_organisations-replacement-urls) file can be manually updated to add replacement URLs if gov.uk/government/organisations doesn't know them. The output files are:

* [gov.uk_api_organisations.json](gov.uk_api_organisations.json) - a JSON blob that contains the results from paging through gov.uk/api/organisations.
* [gov.uk_api_organisations-augmented.csv](gov.uk_api_organisations-augmented.csv) & [gov.uk_api_organisations-augmented.tsv](gov.uk_api_organisations-augmented.tsv) - CSV/TSV versions of the results from gov.uk/api/organisations augmented with the "real" URL from gov.uk/government/organisations. It doesn't include any organisations with `closed_at` set so has <!--GB-GOVUKAPI-CURRENT-->1067<!--END GB-GOVUKAPI-CURRENT--> organisations.
* [gov.uk_api_organisations-augmented.csv](gov.uk_api_organisations-augmented.csv) & [gov.uk_api_organisations-augmented.tsv](gov.uk_api_organisations-augmented.tsv) - CSV/TSV versions of the results from gov.uk/api/organisations augmented with the "real" URL from gov.uk/government/organisations. It doesn't include any organisations with `closed_at` set so has <!--GB-GOVUKAPI-CURRENT-->1069<!--END GB-GOVUKAPI-CURRENT--> organisations.
* [gov.uk_government_organisations.html](gov.uk_government_organisations.html) - the HTML page from gov.uk/government/organisations.
* [gov.uk_government_organisations.csv](gov.uk_government_organisations.csv) - a CSV version of gov.uk/government/organisations with the unique URL slug used as an ID along with the title, gov.uk URL, and the "real" URL (which may be identical).
* [gov.uk_government_organisations-augmented.csv](gov.uk_government_organisations-augmented.csv) & [gov.uk_government_organisations-augmented.tsv](gov.uk_government_organisations-augmented.tsv) - CSV/TSV versions of gov.uk/government/organisations that has had IDs, type, and updated date added. It has <!--GB-GOVUK-->602<!--END GB-GOVUK--> organisations listed.
* [gov.uk_government_organisations-augmented.csv](gov.uk_government_organisations-augmented.csv) & [gov.uk_government_organisations-augmented.tsv](gov.uk_government_organisations-augmented.tsv) - CSV/TSV versions of gov.uk/government/organisations that has had IDs, type, and updated date added. It has <!--GB-GOVUK-->603<!--END GB-GOVUK--> organisations listed.
162 changes: 120 additions & 42 deletions docs/gov.uk_api_organisations.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1564,6 +1564,34 @@
"updated_at" : "2015-08-07T14:43:49.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/british-pharmacopoeia"
},
{
"analytics_identifier" : "OT1395",
"child_organisations" : [],
"details" : {
"abbreviation" : null,
"closed_at" : null,
"content_id" : "1145bbed-a220-43d5-9379-f3d32863ba4f",
"govuk_closed_status" : null,
"govuk_status" : "exempt",
"logo_formatted_name" : "British Technology Investments Ltd",
"organisation_brand_colour_class_name" : "department-for-science-innovation-and-technology",
"organisation_logo_type_class_name" : "single-identity",
"slug" : "british-technology-investments-ltd"
},
"format" : "Other",
"id" : "https://www.gov.uk/api/organisations/british-technology-investments-ltd",
"parent_organisations" : [
{
"id" : "https://www.gov.uk/api/organisations/department-for-science-innovation-and-technology",
"web_url" : "https://www.gov.uk/government/organisations/department-for-science-innovation-and-technology"
}
],
"superseded_organisations" : [],
"superseding_organisations" : [],
"title" : "British Technology Investments Ltd",
"updated_at" : "2023-06-30T11:26:14.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/british-technology-investments-ltd"
},
{
"analytics_identifier" : "PB461",
"child_organisations" : [],
Expand Down Expand Up @@ -8379,6 +8407,39 @@
"updated_at" : "2014-10-15T15:37:29.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/gas-and-electricity-consumer-council-energywatch"
},
{
"analytics_identifier" : "PB577",
"child_organisations" : [],
"details" : {
"abbreviation" : "GACS",
"closed_at" : null,
"content_id" : "7c75bfa9-f13b-4d1e-b083-1fb75d5f7a76",
"govuk_closed_status" : "replaced",
"govuk_status" : "closed",
"logo_formatted_name" : "General Advisory \r\nCommittee on Science",
"organisation_brand_colour_class_name" : null,
"organisation_logo_type_class_name" : "single-identity",
"slug" : "general-advisory-committee-on-science"
},
"format" : "Advisory non-departmental public body",
"id" : "https://www.gov.uk/api/organisations/general-advisory-committee-on-science",
"parent_organisations" : [
{
"id" : "https://www.gov.uk/api/organisations/food-standards-agency",
"web_url" : "https://www.gov.uk/government/organisations/food-standards-agency"
}
],
"superseded_organisations" : [],
"superseding_organisations" : [
{
"id" : "https://www.gov.uk/api/organisations/science-advisory-committees",
"web_url" : "https://www.gov.uk/government/organisations/science-advisory-committees"
}
],
"title" : "General Advisory Committee on Science",
"updated_at" : "2023-06-13T11:13:03.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/general-advisory-committee-on-science"
},
{
"analytics_identifier" : "OT809",
"child_organisations" : [],
Expand Down Expand Up @@ -17540,7 +17601,7 @@
"superseded_organisations" : [],
"superseding_organisations" : [],
"title" : "Committee on Radioactive Waste Management",
"updated_at" : "2022-07-21T08:14:02.000+01:00",
"updated_at" : "2023-06-09T14:59:52.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/committee-on-radioactive-waste-management"
},
{
Expand Down Expand Up @@ -18707,10 +18768,6 @@
"id" : "https://www.gov.uk/api/organisations/competition-and-markets-authority",
"web_url" : "https://www.gov.uk/government/organisations/competition-and-markets-authority"
},
{
"id" : "https://www.gov.uk/api/organisations/land-registry",
"web_url" : "https://www.gov.uk/government/organisations/land-registry"
},
{
"id" : "https://www.gov.uk/api/organisations/acas",
"web_url" : "https://www.gov.uk/government/organisations/acas"
Expand Down Expand Up @@ -19274,6 +19331,10 @@
{
"id" : "https://www.gov.uk/api/organisations/electricity-settlements-company",
"web_url" : "https://www.gov.uk/government/organisations/electricity-settlements-company"
},
{
"id" : "https://www.gov.uk/api/organisations/office-for-zero-emission-vehicles",
"web_url" : "https://www.gov.uk/government/organisations/office-for-zero-emission-vehicles"
}
],
"details" : {
Expand All @@ -19290,7 +19351,12 @@
"format" : "Ministerial department",
"id" : "https://www.gov.uk/api/organisations/department-for-energy-security-and-net-zero",
"parent_organisations" : [],
"superseded_organisations" : [],
"superseded_organisations" : [
{
"id" : "https://www.gov.uk/api/organisations/department-for-business-energy-and-industrial-strategy",
"web_url" : "https://www.gov.uk/government/organisations/department-for-business-energy-and-industrial-strategy"
}
],
"superseding_organisations" : [],
"title" : "Department for Energy Security and Net Zero",
"updated_at" : "2023-05-03T08:44:44.000+01:00",
Expand Down Expand Up @@ -22306,34 +22372,6 @@
"updated_at" : "2019-09-12T11:19:17.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/gangmasters-licensing-appeals"
},
{
"analytics_identifier" : "PB577",
"child_organisations" : [],
"details" : {
"abbreviation" : "GACS",
"closed_at" : null,
"content_id" : "7c75bfa9-f13b-4d1e-b083-1fb75d5f7a76",
"govuk_closed_status" : null,
"govuk_status" : "exempt",
"logo_formatted_name" : "General Advisory \r\nCommittee on Science",
"organisation_brand_colour_class_name" : null,
"organisation_logo_type_class_name" : "single-identity",
"slug" : "general-advisory-committee-on-science"
},
"format" : "Advisory non-departmental public body",
"id" : "https://www.gov.uk/api/organisations/general-advisory-committee-on-science",
"parent_organisations" : [
{
"id" : "https://www.gov.uk/api/organisations/food-standards-agency",
"web_url" : "https://www.gov.uk/government/organisations/food-standards-agency"
}
],
"superseded_organisations" : [],
"superseding_organisations" : [],
"title" : "General Advisory Committee on Science",
"updated_at" : "2014-10-15T15:36:53.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/general-advisory-committee-on-science"
},
{
"analytics_identifier" : "OT1271",
"child_organisations" : [
Expand Down Expand Up @@ -24632,6 +24670,10 @@
{
"id" : "https://www.gov.uk/api/organisations/accelerated-capability-environment",
"web_url" : "https://www.gov.uk/government/organisations/accelerated-capability-environment"
},
{
"id" : "https://www.gov.uk/api/organisations/office-for-the-independent-examiner-of-complaints",
"web_url" : "https://www.gov.uk/government/organisations/office-for-the-independent-examiner-of-complaints"
}
],
"details" : {
Expand Down Expand Up @@ -27184,6 +27226,10 @@
{
"id" : "https://www.gov.uk/api/organisations/sheffield-forgemasters-international-ltd",
"web_url" : "https://www.gov.uk/government/organisations/sheffield-forgemasters-international-ltd"
},
{
"id" : "https://www.gov.uk/api/organisations/pension-compensation-and-welfare-support-for-armed-forces-and-veterans",
"web_url" : "https://www.gov.uk/government/organisations/pension-compensation-and-welfare-support-for-armed-forces-and-veterans"
}
],
"details" : {
Expand All @@ -27208,7 +27254,7 @@
],
"superseding_organisations" : [],
"title" : "Ministry of Defence",
"updated_at" : "2022-10-14T13:56:32.000+01:00",
"updated_at" : "2023-06-09T15:49:12.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/ministry-of-defence"
},
{
Expand Down Expand Up @@ -29215,7 +29261,7 @@
"analytics_identifier" : "OT649",
"child_organisations" : [],
"details" : {
"abbreviation" : null,
"abbreviation" : "NIPB",
"closed_at" : null,
"content_id" : "9d5e9caf-1a1f-4911-9209-8fd084fd683c",
"govuk_closed_status" : null,
Expand All @@ -29236,7 +29282,7 @@
"superseded_organisations" : [],
"superseding_organisations" : [],
"title" : "Northern Ireland Policing Board",
"updated_at" : "2018-10-16T12:13:19.000+01:00",
"updated_at" : "2023-06-26T17:24:43.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/northern-ireland-policing-board"
},
{
Expand Down Expand Up @@ -30171,7 +30217,7 @@
"superseded_organisations" : [],
"superseding_organisations" : [],
"title" : "Office of the Advocate General for Scotland",
"updated_at" : "2021-01-14T16:20:38.000+00:00",
"updated_at" : "2023-06-15T13:54:31.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/office-of-the-advocate-general-for-scotland"
},
{
Expand Down Expand Up @@ -31222,12 +31268,16 @@
{
"id" : "https://www.gov.uk/api/organisations/civil-service",
"web_url" : "https://www.gov.uk/government/organisations/civil-service"
},
{
"id" : "https://www.gov.uk/api/organisations/cabinet-office",
"web_url" : "https://www.gov.uk/government/organisations/cabinet-office"
}
],
"superseded_organisations" : [],
"superseding_organisations" : [],
"title" : "Policy Profession",
"updated_at" : "2014-10-15T15:38:42.000+01:00",
"updated_at" : "2023-06-16T09:51:23.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/civil-service-policy-profession"
},
{
Expand Down Expand Up @@ -32462,6 +32512,34 @@
"updated_at" : "2021-08-06T10:59:39.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/school-teachers-review-body"
},
{
"analytics_identifier" : "PB1394",
"child_organisations" : [],
"details" : {
"abbreviation" : "SACs",
"closed_at" : null,
"content_id" : "3882b19a-2f5a-478a-b69c-4686b84a1414",
"govuk_closed_status" : null,
"govuk_status" : "exempt",
"logo_formatted_name" : "Science Advisory Committees",
"organisation_brand_colour_class_name" : "department-for-environment-food-rural-affairs",
"organisation_logo_type_class_name" : "no-identity",
"slug" : "science-advisory-committees"
},
"format" : "Executive non-departmental public body",
"id" : "https://www.gov.uk/api/organisations/science-advisory-committees",
"parent_organisations" : [
{
"id" : "https://www.gov.uk/api/organisations/department-for-environment-food-rural-affairs",
"web_url" : "https://www.gov.uk/government/organisations/department-for-environment-food-rural-affairs"
}
],
"superseded_organisations" : [],
"superseding_organisations" : [],
"title" : "Science Advisory Committees",
"updated_at" : "2023-06-13T11:11:48.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/science-advisory-committees"
},
{
"analytics_identifier" : "PB135",
"child_organisations" : [],
Expand Down Expand Up @@ -32650,7 +32728,7 @@
"superseded_organisations" : [],
"superseding_organisations" : [],
"title" : "Security Industry Authority",
"updated_at" : "2022-06-27T09:36:12.000+01:00",
"updated_at" : "2023-06-28T17:13:38.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/security-industry-authority"
},
{
Expand Down Expand Up @@ -33390,7 +33468,7 @@
"superseded_organisations" : [],
"superseding_organisations" : [],
"title" : "Student Loans Company",
"updated_at" : "2023-04-26T12:15:07.000+01:00",
"updated_at" : "2023-06-23T10:03:02.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/student-loans-company"
},
{
Expand Down Expand Up @@ -34899,7 +34977,7 @@
],
"superseding_organisations" : [],
"title" : "UK Health Security Agency",
"updated_at" : "2023-05-17T09:32:49.000+01:00",
"updated_at" : "2023-06-26T14:49:46.000+01:00",
"web_url" : "https://www.gov.uk/government/organisations/uk-health-security-agency"
},
{
Expand Down
Loading

0 comments on commit 79662b0

Please sign in to comment.