payload: {"action":"created","comment":{"url":"https://api.github.com/repos/alphagov/search-api/pulls/comments/343658164","pull_request_review_id":313348665,"id":343658164,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDM0MzY1ODE2NA==","diff_hunk":"@@ -0,0 +1,35 @@\n+require \"learn_to_rank/feature_sets\"\n+require \"learn_to_rank/ranker\"\n+\n+module LearnToRank\n+  class Reranker\n+    # Reranker re-orders elasticsearch results using a pre-trained model\n+    def rerank(es_results: [], count: DEFAULT_COUNT)\n+      feature_sets = FeatureSets.new.call(es_results)\n+      new_scores   = Ranker.new(feature_sets).ranks\n+\n+      reorder_results(es_results, new_scores)\n+    end\n+\n+  private\n+\n+    DEFAULT_COUNT = 20\n+    MAX_MODEL_BOOST = 5\n+\n+    def reorder_results(search_results, new_scores)\n+      search_results\n+        .map\n+        .with_index { |result, index|","path":"lib/learn_to_rank/reranker.rb","position":22,"original_position":22,"commit_id":"0bf768759f35f7ce7c690d535d10d9fc44f03940","original_commit_id":"0bf768759f35f7ce7c690d535d10d9fc44f03940","user":{"login":"barrucadu","id":75235,"node_id":"MDQ6VXNlcjc1MjM1","avatar_url":"https://avatars0.githubusercontent.com/u/75235?v=4","gravatar_id":"","url":"https://api.github.com/users/barrucadu","html_url":"https://github.com/barrucadu","followers_url":"https://api.github.com/users/barrucadu/followers","following_url":"https://api.github.com/users/barrucadu/following{/other_user}","gists_url":"https://api.github.com/users/barrucadu/gists{/gist_id}","starred_url":"https://api.github.com/users/barrucadu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/barrucadu/subscriptions","organizations_url":"https://api.github.com/users/barrucadu/orgs","repos_url":"https://api.github.com/users/barrucadu/repos","events_url":"https://api.github.com/users/barrucadu/events{/privacy}","received_events_url":"https://api.github.com/users/barrucadu/received_events","type":"User","site_admin":false},"body":"As a minor point, I think `begin` / `end` is clearer when the block is multiple lines.","created_at":"2019-11-07T13:48:29Z","updated_at":"2019-11-07T13:48:29Z","html_url":"https://github.com/alphagov/search-api/pull/1768#discussion_r343658164","pull_request_url":"https://api.github.com/repos/alphagov/search-api/pulls/1768","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/alphagov/search-api/pulls/comments/343658164"},"html":{"href":"https://github.com/alphagov/search-api/pull/1768#discussion_r343658164"},"pull_request":{"href":"https://api.github.com/repos/alphagov/search-api/pulls/1768"}}},"pull_request":{"url":"https://api.github.com/repos/alphagov/search-api/pulls/1768","id":336399555,"node_id":"MDExOlB1bGxSZXF1ZXN0MzM2Mzk5NTU1","html_url":"https://github.com/alphagov/search-api/pull/1768","diff_url":"https://github.com/alphagov/search-api/pull/1768.diff","patch_url":"https://github.com/alphagov/search-api/pull/1768.patch","issue_url":"https://api.github.com/repos/alphagov/search-api/issues/1768","number":1768,"state":"open","locked":false,"title":"[spike] Learning To Rank","user":{"login":"bilbof","id":8124374,"node_id":"MDQ6VXNlcjgxMjQzNzQ=","avatar_url":"https://avatars1.githubusercontent.com/u/8124374?v=4","gravatar_id":"","url":"https://api.github.com/users/bilbof","html_url":"https://github.com/bilbof","followers_url":"https://api.github.com/users/bilbof/followers","following_url":"https://api.github.com/users/bilbof/following{/other_user}","gists_url":"https://api.github.com/users/bilbof/gists{/gist_id}","starred_url":"https://api.github.com/users/bilbof/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bilbof/subscriptions","organizations_url":"https://api.github.com/users/bilbof/orgs","repos_url":"https://api.github.com/users/bilbof/repos","events_url":"https://api.github.com/users/bilbof/events{/privacy}","received_events_url":"https://api.github.com/users/bilbof/received_events","type":"User","site_admin":false},"body":"This spike demonstrates how we can implement [Learning To Rank](https://en.wikipedia.org/wiki/Learning_to_rank) using Tensorflow’s [ranking module](https://github.com/tensorflow/ranking).\r\n\r\nThe spike is functional, but not ready for merge.\r\n\r\nContents:\r\n\r\n1. What is Learning To Rank?\r\n2. Implementation\r\n3. Results\r\n4. Next steps\r\n\r\n## What is Learning To Rank?\r\n\r\nLearning To Rank (LTR) for our use case is the application of machine learning to create a ranking model that can be used to improve search relevance.\r\n\r\nThis is a common approach for advanced search applications.\r\n\r\nThere is a good explanation on what LTR is here:\r\n\r\n- https://elasticsearch-learning-to-rank.readthedocs.io/en/latest/core-concepts.html\r\n- https://opensourceconnections.com/blog/2017/02/24/what-is-learning-to-rank/\r\n\r\nIt's easiest to explain with an example:\r\n\r\nSay you searched for 'Harry Potter'. Search API might return the following results:\r\n\r\n1. HMRC Sign in\r\n2. Harry Potter World\r\n3. Pottery barn\r\n4. Harry Potter\r\n\r\nThese are not good results.\r\n\r\n### How do we improve search results currently?\r\n\r\nOur approach now would be to manually rank these results, giving them a score of between 0-3 (0 being irrelevant, 3 being completely relevant):\r\n\r\n```\r\nquery         rating    document\r\nharry potter  0         HMRC Sign in\r\nharry potter  2         Harry Potter World\r\nharry potter  0         Pottery barn\r\nharry potter  3         Harry Potter\r\n```\r\n\r\nThese judgements are an expression of the ideal ordering.\r\n\r\nWe can then use nDCG to tell us how good the results are. We might get a score of 0.3 for this result (from 0 to 1).\r\n\r\nWe could then investigate why the results are bad. Perhaps there is a best bet. Or popularity might be pushing HMRC docs. We can fix these things, then re-run nDCG, and we might get 0.7 for this query. Good stuff.\r\n\r\nThis works, but doesn't scale well. Eventually this process becomes a game of whack-a-mole. If we downgrade popularity boosting, then this query might have better results, but other queries might have worse results.\r\n\r\nHand-tuning relevance doesn't work very well because field boosts are complex, and difficult to maintain.\r\n\r\nLearning to Rank makes tuning relevance:\r\n\r\n- automatable\r\n- scalable\r\n- more successful\r\n- more maintainable (arguable)\r\n\r\nLTR won't make relevance tuning easier. But it could make search results more relevant.\r\n\r\n### How do we improve search results with Learn To Rank?\r\n\r\nThe end goal is to make search results more relevant, without manually increasing the boost of a field.\r\n\r\nWe would go through the same process of rating the search results:\r\n\r\n```\r\nharry potter  0         Pottery barn\r\nharry potter  3         Harry Potter\r\n...\r\n```\r\n\r\nHowever, after doing this, we would train a model with this data to rank the results better. The model would optimise for nDCG.\r\n\r\nWe would provide the model features, such as the fields we boost on.\r\n\r\nFor example, take the SVM dataset below for the query 'harry potter' (qid:1)\r\n\r\n```\r\n0 qid:1 popularity:0.3 title_score:0.2 description_score:0.3 recency:0.9 pagerank:5 ...\r\n3 qid:1 popularity:0.1 title_score:0.6 description_score:0.7 recency:0.5 pagerank:2 ...\r\n```\r\n\r\nThis data, in addition to many other graded query:document pairs and their features can be used to train a model.\r\n\r\nThe model might learn that title and description scores are slightly more useful that popularity and recency, and how much, in certain circumstances.\r\n\r\n## Implementation\r\n\r\nI chose to use [TensorFlow's ranking library](https://github.com/tensorflow/ranking) for this.\r\n\r\nTensorFlow is well supported, and the ranking library is under active development.\r\n\r\nI wanted to try https://github.com/o19s/elasticsearch-learning-to-rank, but as we are using AWS Elasticsearch, we are not permitted to install custom plugins.\r\n\r\n### Architecture:\r\n\r\nUser searches\r\nSearch API pulls results out of Elasticsearch\r\nSearch API send results to LTR model served by TF Serving\r\nLTR model returns results re-ranked\r\nSearch API returns results to user\r\n\r\nThe model is trained on implicit and explicit relevancy judgements. Ratings are provided by manual raters and derived from user click data.\r\n\r\nFeatures can added and improved on.\r\n\r\n### Notes on the code in this spike:\r\n\r\nWithin this spike there is a new module `LearnToRank` which contains various utilities for training and using the ranking model.\r\n\r\nWe needed to rewrite NDCG outside of Elasticsearch because the re-ranking happens outside of ES.\r\n\r\n`LearnToRank::Reranker` is called after the results have been extracted from Elasticsearch, it calls the separate process serving the model which re-ranks the results.\r\n\r\nThere is also a directory of scripts for training and serving the model.\r\n\r\n## Results\r\n\r\nThis spike contains a working MVP of a ranking model that can be trained with relevancy judgements to re-rank results at query-time.\r\n\r\n### Relevancy impact\r\n\r\nThe results are positive. For our existing dataset (explicit relevancy judgements) the model gets a higher score (9.2) compared to the original (0.89). This was done on local data, we might see different results in other envs.\r\n\r\nFor the combined dataset (explicit and implicit sourced from GA) we get a score of 0.88 with the model and the same score without. The model has a very slightly higher score (0.88265 vs 0.88011).\r\n\r\nI think this demonstrates that with some more work (data/feature engineering), the model can outperform our existing search algorithm.\r\n\r\nThe model uses ndcg as one of the measures by which it trains itself, so it's not surprising that it can get a better score.\r\n\r\nWe were able to beat our existing search's ndcg score within a few days. This doesn't mean that search is better, just better by this metric with these relevancy judgements available.\r\n\r\n### Query performance impact\r\n\r\nIn Bloomberg's talk on their project, they devote half of the talk to discussion of how they made query performance better after implementing LTR. I was worried that our search performance might also be hit.\r\n\r\nThe re-ranking model is very quick. TF Serving is production ready; I think any performance bottlenecks will be in ruby and Elasticsearch.\r\n\r\nI didn't notice a performance hit, but I haven't done proper testing of this.\r\n\r\n### Architectural / code complexity impact\r\n\r\nThe code is not much more complex. Architecturally the application is more complex. It requires a separate process to be running. It requires some knowledge of machine learning concepts and Learn To Rank from developers and product managers.\r\n\r\n### Pros and cons\r\n\r\nIt's harder to do, but will lead to better results. I am unclear whether this lead to a harder to maintain application.\r\n\r\nPros:\r\n\r\n- Potentially significant improvements to relevancy\r\n- This is a well trodden path, as many orgs use LTR (Airbnb, Wikimedia,\r\n  commercial search engines, Bloomberg, Yelp, etc.)\r\n\r\nCons:\r\n\r\n- Added architectural complexity\r\n- Harder to build and debug in contrast to hand-tuning\r\n\r\n## Next steps\r\n\r\nAside from making this more robust so that it is production ready, I see two main areas for further development:\r\n\r\n- Further research into how we should implement Learn To Rank\r\n- Feature engineering\r\n- Acquiring more/better relevancy judgements\r\n\r\nThe tasks would be aimed at making the model give more precise relevancy predictions, should we decide to proceed with LTR.\r\n\r\nIf we choose not to implement LTR yet, we could work on a click model for generating relevancy judgements. This would help us in the short term (providing ratings for nDCG) and for any LTR project we might do. This was something Mat from the previous search team worked on: https://github.com/MatMoore/accelerator.","created_at":"2019-11-04T17:42:32Z","updated_at":"2019-11-07T13:48:29Z","closed_at":null,"merged_at":null,"merge_commit_sha":"635c6056091f56c0b9af38026032fbc27a7207e7","assignee":null,"assignees":[],"requested_reviewers":[{"login":"sihugh","id":773037,"node_id":"MDQ6VXNlcjc3MzAzNw==","avatar_url":"https://avatars2.githubusercontent.com/u/773037?v=4","gravatar_id":"","url":"https://api.github.com/users/sihugh","html_url":"https://github.com/sihugh","followers_url":"https://api.github.com/users/sihugh/followers","following_url":"https://api.github.com/users/sihugh/following{/other_user}","gists_url":"https://api.github.com/users/sihugh/gists{/gist_id}","starred_url":"https://api.github.com/users/sihugh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sihugh/subscriptions","organizations_url":"https://api.github.com/users/sihugh/orgs","repos_url":"https://api.github.com/users/sihugh/repos","events_url":"https://api.github.com/users/sihugh/events{/privacy}","received_events_url":"https://api.github.com/users/sihugh/received_events","type":"User","site_admin":false},{"login":"laurentqro","id":3141541,"node_id":"MDQ6VXNlcjMxNDE1NDE=","avatar_url":"https://avatars3.githubusercontent.com/u/3141541?v=4","gravatar_id":"","url":"https://api.github.com/users/laurentqro","html_url":"https://github.com/laurentqro","followers_url":"https://api.github.com/users/laurentqro/followers","following_url":"https://api.github.com/users/laurentqro/following{/other_user}","gists_url":"https://api.github.com/users/laurentqro/gists{/gist_id}","starred_url":"https://api.github.com/users/laurentqro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laurentqro/subscriptions","organizations_url":"https://api.github.com/users/laurentqro/orgs","repos_url":"https://api.github.com/users/laurentqro/repos","events_url":"https://api.github.com/users/laurentqro/events{/privacy}","received_events_url":"https://api.github.com/users/laurentqro/received_events","type":"User","site_admin":false},{"login":"hannako","id":17908089,"node_id":"MDQ6VXNlcjE3OTA4MDg5","avatar_url":"https://avatars3.githubusercontent.com/u/17908089?v=4","gravatar_id":"","url":"https://api.github.com/users/hannako","html_url":"https://github.com/hannako","followers_url":"https://api.github.com/users/hannako/followers","following_url":"https://api.github.com/users/hannako/following{/other_user}","gists_url":"https://api.github.com/users/hannako/gists{/gist_id}","starred_url":"https://api.github.com/users/hannako/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hannako/subscriptions","organizations_url":"https://api.github.com/users/hannako/orgs","repos_url":"https://api.github.com/users/hannako/repos","events_url":"https://api.github.com/users/hannako/events{/privacy}","received_events_url":"https://api.github.com/users/hannako/received_events","type":"User","site_admin":false},{"login":"SamJamCul","id":25597009,"node_id":"MDQ6VXNlcjI1NTk3MDA5","avatar_url":"https://avatars2.githubusercontent.com/u/25597009?v=4","gravatar_id":"","url":"https://api.github.com/users/SamJamCul","html_url":"https://github.com/SamJamCul","followers_url":"https://api.github.com/users/SamJamCul/followers","following_url":"https://api.github.com/users/SamJamCul/following{/other_user}","gists_url":"https://api.github.com/users/SamJamCul/gists{/gist_id}","starred_url":"https://api.github.com/users/SamJamCul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SamJamCul/subscriptions","organizations_url":"https://api.github.com/users/SamJamCul/orgs","repos_url":"https://api.github.com/users/SamJamCul/repos","events_url":"https://api.github.com/users/SamJamCul/events{/privacy}","received_events_url":"https://api.github.com/users/SamJamCul/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/alphagov/search-api/pulls/1768/commits","review_comments_url":"https://api.github.com/repos/alphagov/search-api/pulls/1768/comments","review_comment_url":"https://api.github.com/repos/alphagov/search-api/pulls/comments{/number}","comments_url":"https://api.github.com/repos/alphagov/search-api/issues/1768/comments","statuses_url":"https://api.github.com/repos/alphagov/search-api/statuses/0bf768759f35f7ce7c690d535d10d9fc44f03940","head":{"label":"alphagov:learn-to-rank-spike","ref":"learn-to-rank-spike","sha":"0bf768759f35f7ce7c690d535d10d9fc44f03940","user":{"login":"alphagov","id":596977,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU5Njk3Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/596977?v=4","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"repo":{"id":2686767,"node_id":"MDEwOlJlcG9zaXRvcnkyNjg2NzY3","name":"search-api","full_name":"alphagov/search-api","private":false,"owner":{"login":"alphagov","id":596977,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU5Njk3Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/596977?v=4","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/alphagov/search-api","description":"Search API for GOV.UK","fork":false,"url":"https://api.github.com/repos/alphagov/search-api","forks_url":"https://api.github.com/repos/alphagov/search-api/forks","keys_url":"https://api.github.com/repos/alphagov/search-api/keys{/key_id}","collaborators_url":"https://api.github.com/repos/alphagov/search-api/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/alphagov/search-api/teams","hooks_url":"https://api.github.com/repos/alphagov/search-api/hooks","issue_events_url":"https://api.github.com/repos/alphagov/search-api/issues/events{/number}","events_url":"https://api.github.com/repos/alphagov/search-api/events","assignees_url":"https://api.github.com/repos/alphagov/search-api/assignees{/user}","branches_url":"https://api.github.com/repos/alphagov/search-api/branches{/branch}","tags_url":"https://api.github.com/repos/alphagov/search-api/tags","blobs_url":"https://api.github.com/repos/alphagov/search-api/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/alphagov/search-api/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/alphagov/search-api/git/refs{/sha}","trees_url":"https://api.github.com/repos/alphagov/search-api/git/trees{/sha}","statuses_url":"https://api.github.com/repos/alphagov/search-api/statuses/{sha}","languages_url":"https://api.github.com/repos/alphagov/search-api/languages","stargazers_url":"https://api.github.com/repos/alphagov/search-api/stargazers","contributors_url":"https://api.github.com/repos/alphagov/search-api/contributors","subscribers_url":"https://api.github.com/repos/alphagov/search-api/subscribers","subscription_url":"https://api.github.com/repos/alphagov/search-api/subscription","commits_url":"https://api.github.com/repos/alphagov/search-api/commits{/sha}","git_commits_url":"https://api.github.com/repos/alphagov/search-api/git/commits{/sha}","comments_url":"https://api.github.com/repos/alphagov/search-api/comments{/number}","issue_comment_url":"https://api.github.com/repos/alphagov/search-api/issues/comments{/number}","contents_url":"https://api.github.com/repos/alphagov/search-api/contents/{+path}","compare_url":"https://api.github.com/repos/alphagov/search-api/compare/{base}...{head}","merges_url":"https://api.github.com/repos/alphagov/search-api/merges","archive_url":"https://api.github.com/repos/alphagov/search-api/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/alphagov/search-api/downloads","issues_url":"https://api.github.com/repos/alphagov/search-api/issues{/number}","pulls_url":"https://api.github.com/repos/alphagov/search-api/pulls{/number}","milestones_url":"https://api.github.com/repos/alphagov/search-api/milestones{/number}","notifications_url":"https://api.github.com/repos/alphagov/search-api/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/alphagov/search-api/labels{/name}","releases_url":"https://api.github.com/repos/alphagov/search-api/releases{/id}","deployments_url":"https://api.github.com/repos/alphagov/search-api/deployments","created_at":"2011-11-01T10:12:45Z","updated_at":"2019-11-07T11:42:22Z","pushed_at":"2019-11-07T12:08:22Z","git_url":"git://github.com/alphagov/search-api.git","ssh_url":"git@github.com:alphagov/search-api.git","clone_url":"https://github.com/alphagov/search-api.git","svn_url":"https://github.com/alphagov/search-api","homepage":"https://docs.publishing.service.gov.uk/apps/search-api.html","size":11718,"stargazers_count":1,"watchers_count":1,"language":"Ruby","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"archived":false,"disabled":false,"open_issues_count":18,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit","node_id":"MDc6TGljZW5zZTEz"},"forks":3,"open_issues":18,"watchers":1,"default_branch":"master"}},"base":{"label":"alphagov:master","ref":"master","sha":"778fb0b66a78882615efa8e873c91c4d6559f4ba","user":{"login":"alphagov","id":596977,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU5Njk3Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/596977?v=4","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"repo":{"id":2686767,"node_id":"MDEwOlJlcG9zaXRvcnkyNjg2NzY3","name":"search-api","full_name":"alphagov/search-api","private":false,"owner":{"login":"alphagov","id":596977,"node_id":"MDEyOk9yZ2FuaXphdGlvbjU5Njk3Nw==","avatar_url":"https://avatars0.githubusercontent.com/u/596977?v=4","gravatar_id":"","url":"https://api.github.com/users/alphagov","html_url":"https://github.com/alphagov","followers_url":"https://api.github.com/users/alphagov/followers","following_url":"https://api.github.com/users/alphagov/following{/other_user}","gists_url":"https://api.github.com/users/alphagov/gists{/gist_id}","starred_url":"https://api.github.com/users/alphagov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alphagov/subscriptions","organizations_url":"https://api.github.com/users/alphagov/orgs","repos_url":"https://api.github.com/users/alphagov/repos","events_url":"https://api.github.com/users/alphagov/events{/privacy}","received_events_url":"https://api.github.com/users/alphagov/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/alphagov/search-api","description":"Search API for GOV.UK","fork":false,"url":"https://api.github.com/repos/alphagov/search-api","forks_url":"https://api.github.com/repos/alphagov/search-api/forks","keys_url":"https://api.github.com/repos/alphagov/search-api/keys{/key_id}","collaborators_url":"https://api.github.com/repos/alphagov/search-api/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/alphagov/search-api/teams","hooks_url":"https://api.github.com/repos/alphagov/search-api/hooks","issue_events_url":"https://api.github.com/repos/alphagov/search-api/issues/events{/number}","events_url":"https://api.github.com/repos/alphagov/search-api/events","assignees_url":"https://api.github.com/repos/alphagov/search-api/assignees{/user}","branches_url":"https://api.github.com/repos/alphagov/search-api/branches{/branch}","tags_url":"https://api.github.com/repos/alphagov/search-api/tags","blobs_url":"https://api.github.com/repos/alphagov/search-api/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/alphagov/search-api/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/alphagov/search-api/git/refs{/sha}","trees_url":"https://api.github.com/repos/alphagov/search-api/git/trees{/sha}","statuses_url":"https://api.github.com/repos/alphagov/search-api/statuses/{sha}","languages_url":"https://api.github.com/repos/alphagov/search-api/languages","stargazers_url":"https://api.github.com/repos/alphagov/search-api/stargazers","contributors_url":"https://api.github.com/repos/alphagov/search-api/contributors","subscribers_url":"https://api.github.com/repos/alphagov/search-api/subscribers","subscription_url":"https://api.github.com/repos/alphagov/search-api/subscription","commits_url":"https://api.github.com/repos/alphagov/search-api/commits{/sha}","git_commits_url":"https://api.github.com/repos/alphagov/search-api/git/commits{/sha}","comments_url":"https://api.github.com/repos/alphagov/search-api/comments{/number}","issue_comment_url":"https://api.github.com/repos/alphagov/search-api/issues/comments{/number}","contents_url":"https://api.github.com/repos/alphagov/search-api/contents/{+path}","compare_url":"https://api.github.com/repos/alphagov/search-api/compare/{base}...{head}","merges_url":"https://api.github.com/repos/alphagov/search-api/merges","archive_url":"https://api.github.com/repos/alphagov/search-api/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/alphagov/search-api/downloads","issues_url":"https://api.github.com/repos/alphagov/search-api/issues{/number}","pulls_url":"https://api.github.com/repos/alphagov/search-api/pulls{/number}","milestones_url":"https://api.github.com/repos/alphagov/search-api/milestones{/number}","notifications_url":"https://api.github.com/repos/alphagov/search-api/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/alphagov/search-api/labels{/name}","releases_url":"https://api.github.com/repos/alphagov/search-api/releases{/id}","deployments_url":"https://api.github.com/repos/alphagov/search-api/deployments","created_at":"2011-11-01T10:12:45Z","updated_at":"2019-11-07T11:42:22Z","pushed_at":"2019-11-07T12:08:22Z","git_url":"git://github.com/alphagov/search-api.git","ssh_url":"git@github.com:alphagov/search-api.git","clone_url":"https://github.com/alphagov/search-api.git","svn_url":"https://github.com/alphagov/search-api","homepage":"https://docs.publishing.service.gov.uk/apps/search-api.html","size":11718,"stargazers_count":1,"watchers_count":1,"language":"Ruby","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"archived":false,"disabled":false,"open_issues_count":18,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit","node_id":"MDc6TGljZW5zZTEz"},"forks":3,"open_issues":18,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/alphagov/search-api/pulls/1768"},"html":{"href":"https://github.com/alphagov/search-api/pull/1768"},"issue":{"href":"https://api.github.com/repos/alphagov/search-api/issues/1768"},"comments":{"href":"https://api.github.com/repos/alphagov/search-api/issues/1768/comments"},"review_comments":{"href":"https://api.github.com/repos/alphagov/search-api/pulls/1768/comments"},"review_comment":{"href":"https://api.github.com/repos/alphagov/search-api/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/alphagov/search-api/pulls/1768/commits"},"statuses":{"href":"https://api.github.com/repos/alphagov/search-api/statuses/0bf768759f35f7ce7c690d535d10d9fc44f03940"}},"author_association":"CONTRIBUTOR"}}
other: {"actor":{"display_login":"barrucadu"}}