Reading/Web/Projects/Related pages
Read more is a feature that is currently stable on almost all Wikipedia mobile websites. It aims to drive page views by engaging users by directing them to related content. The rationale is, if readers are offered suggestions that are similar to the topic they are reading about, this will further engage their reading session time, it will further educate them about the topic they are looking for, and supports a richer reading experience for those who are just randomly browsing topics. The concept already exists on apps, where you can check the performance report.
The Problem
[edit]If a reader has reached to the bottom of the article, they might be looking to read more about the topic and surfacing articles that are similar might be exactly what they are looking for.
The How
[edit]Using the Extension:RelatedArticles extension will show suggested articles at the bottom of pages encouraging the user to read another page.
A user who gets to the bottom of an article on either mobile or desktop web is shown a list of other articles that are related to the one they just finished. The notion is that if the reader has read to the end of the article, they might be looking to read more about the topic and surfacing articles that are similar might be exactly what they are looking for.
The Effect
[edit]This has been released on apps and resulted in a 16% click-through rate (For users who saw it). Additionally, 25% of the users who read more results, clicked through at least 1x in a 1-day period (data).
Moving to stable plan and staging
[edit]After 4 months of running in beta on both mobile and desktop, a proposal was moved to stable on mobile.
FAQ
[edit]- Where do article suggestions come from?
A more like query which will programmatically choose related pages. Here is an overview of the algorithm the function is based on: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html.
The more-like query service ranks articles in order to show the top 3 and uses a score that computes the similarity between documents, this can be fine-tuned as documented on the wiki page for it: Help:CirrusSearch#Morelike. The defaults in code can be seen on github here.
Given the concerns expressed by community members here and elsewhere about popular results making little room for relevance or serendipity, changes were made to the algorithm in August after testing and communication. As of August 30th, 2016, boostlinks (a measure of importance) and boosttemplates (usually a measure of quality) were removed from the scoring, as they caused popular, less relevant pages to appear with high frequency. The ticket for this work is captured here.
Results can be overridden by editors using the {{#relatedarticle:}}
magic word.
- Do editors have any control or are they given any preview of the suggested articles?
Though this feature is not considered part of the article, editors can change the suggested articles given by adding up to 3 manually curated examples to this part of the page navigation.
- {{#related:new page title1}}
- {{#related:new page title2}}
- {{#related:new page title3}}
For example:
On Korur language the related pages have been over-ridden to:
- {{#related:Western Oceanic languages}}
- {{#related: New Guinea}}
- {{#related: Mbula language}}
User:Jkatz (WMF) thinks that making them editable is a problem because:
- the results are not automatically updated
- it means that improvements we make to the algorithm/selection will be lost to pages where an editor has overridden the automated selection.
- right now the manual selection option only applies to the web (not the apps), which is misleading
- some editors have requested that we do automated refreshes every time you load the page--this would not be possible if the above keywords are used.
- How is related articles different from the See also section, navigation boxes and the category system?
They have pretty pictures, are limited to 3, for greater simplicity (unlike see also, are only intended for users who have gotten to the bottom of the article without finding another link more appealing to visit). It is thought that the images and simplicity increase the click-through rate. Which is the metric of success for this feature.
Initial Community Feedback
[edit]The following is an attempt to summarize the feedback collected on the talk page for this feature (as of Feb 25, 2016), along with responses to each one.
Disambiguation pages are pretty bad. See the holocaust disambiguation page as an example: w:Holocaust (disambiguation)
Response:
- Luckily, the feature has no real utility on those pages. There is now a ticket to remove this: T127068 prior to any further roll-out.
FIXED On English Wikipedia, the use of non-free (fair use) article images for navigation goes against En Wikipedia policy.
Response:
This is being remedied by this task: T124225 and is considered a blocker
The click-through-rate metric is not air-tight and reader value is not certain.
Response:
- It is true that click-through rates can be gamed. I (JKatz) submit that sustained click-through rates are indicative that readers are finding value, and placement at the bottom of the page, as well as some analysis of raw pagelogs above, suggest that cannibalization of blue links is not occurring. However, enough others disagree that some additional work is necessary, at least prior desktop roll-out. Here are some ideas:
- Roll-out on some small % of anon-traffic. It is not entirely fair to test only with logged-in users a feature that is intended primarily for users who are not logged in.
- Then, a/b test and look at overall pageviews (for those of you who find session depth to be a bad metric: please suggest a better one)
- The, use the quick-survey feature to ask users who click on 'related pages' if they are satisfied with the results or to get feedback
This is the same as see-also and does not add additional value
Response:
- The sustained high click through rate on mobile suggests that users are finding it valuable enough to warrant promotion. On desktop this is not yet the case--though the click-rate still exceeds any given link. (see above about obtaining better measures of reader value).
- The best thing to do would be to combine with see also, but this is thorny and a lot of technical work. I think we should be assessing whether this is a net benefit to Wikipedia or net negative, not whether or not is a perfect solution.
Sometimes pages return anywhere from suboptimal to damaging results
Response:
- for 'damaging' results, which are rare, editors can over-ride results. I was against mixing algorithm and editing, but the value here is now clear.
- for 'suboptimal results', for now I think we should live with it because the readers seem to be deriving value (see below on underlined portion)
Algorithms are non-NPOV and go against our ethos OR algorithms should be adapted to community norms
Response:
- In the FAQ, we have pasted put a link to the code somewhere and describe the variables/logic? Ultimately, a solution might be to allow each community to tweak the weighting to their needs, but I think that is not warranted by the value of this feature at this point. How do you feel about transparency with regard to the algorithm logic as a solution?
- we might want to denote that they are machine-driven somewhere to clarify. Is this a blocker?
In German and maybe Russian, the tool may violate a principle against 'Themenring' policy
Response:
- I see no reason why we should push on these wikis if they don't want it, but have asked community members to poll their wikis and find out if the concerns are representative
The community was not consulted first and we would have helped you avoid major pitfalls.
Response:
- Unfortunately, yes. We thought since this worked on apps it was a no-brainer, but we clearly misjudged. Given that we obviously made a mistake here, and are making an attempt to improve moving forward both with this feature and others, we hope that you can help us move forward together.
Success criteria
[edit]- %CTR (click through rate, clicks/views) is higher than 10%.
- Ideally, we will be able to ensure that these clicks are not-cannibalistic (increasing overall clicks as opposed to taking clicks from blue links)
Roadmap
[edit]Prototyping
[edit]This was tested in beta first.
MVP
[edit]A user reaching the bottom of an article is shown the title and lead image for 3 articles that relate to the article they just finished.
We were able to measure the engagement clicks/impressions with this feature.
User Stories
[edit]A user reaching the bottom of an article is shown the title and lead image for 3 articles that relate to the article they just finished so that they can continue reading about that topic. Someone editing a Wikipedia article can manually change the article suggestions using wikitext so that they can correct any erroneous or sub-optimal suggestions. A project stakeholder (such as PM or data analysis) is able to measure the engagement clicks/impressions with this feature so they can determine if it is adding user value.
Metrics Implementation
[edit]We wanted to track:
- Impressions of read more suggestions (the lump--not each item suggested)
- Clicks to read a suggested article
- The position of article (1,2,3). Ideally: whether or not article was edited manually
- Ideally: a)overall referrals from a page with read more, b)overall referrals from same page without read more
Timeline Estimate
[edit](this has been updated on 2/24/16 to reflect the actual order to-date)
- build read more according to mobile web specs
Done
- launch on mobile web beta
Done
- launch on desktop web beta
Done
- measure impact using event logs CTR (referral data won't be helpful at these numbers)
Done
- discuss with community
- launch on mobile
- launch on desktop (open discussion about whether we launch in beta first and/or do progressive rollout)
- Delivery Estimate: end of Q3 on mobile across wikis...desktop is too much of a question mark as of February 24, 2016.
Metrics analysis
[edit]Behavior in Stable
[edit]The related pages feature was deployed to 90% of all users on the mobile version of English Wikipedia on March 30, 2017. We collected data in the timespan between March 30 and May 15, 2017. The purpose of this test was to determine the impact of the feature over time. The main metrics we focused on throughout this analysis were clickthrough rate and variations in total events recorded between the test group and the control group.
Summary
[edit]On enwiki, clickthrough rates for related pages links were especially high (6%) when users see the related pages section at the bottom of the article. The overall clickthrough rate on related pages links for mobile was 1.9%. No significant changes in these rates were observed over time, indicating that users will continue using related pages with a similar frequency. When compared to the control group, no significant impact on pageviews was observed hinting at the conclusion that while the feature's content curation is preferred among users, it does not significantly affect user behavior - if a user was interested in continuing to read, she will select an article suggested by related pages with a higher probability than another article on a page. However, the feature has little to no effect on users who were not interested in accessing more articles.
Details
[edit]Clickthrough Rates
[edit]On mobile enwiki, clickthrough rates (ratio of pages clicked/pages seen) for the feature average d 1.91% (ranging between 1.85% and 1.97% with 95% convidence) over the period 03/30/2017 - 05/15/2017. When users view the related pages (by scrolling to the bottom of the article), the clickthrough rate is 6.0% (range 5.89% - 6.14% with 95% confidence). No significant changes in this trend were observed over time.
Total Events Comparison
[edit]On mobile enwiki, we observed negligible differences between the expected pageviews for the test (related pages on) and the control (related pages off), with the ratio of expected events for the test group to expected events for the control group being 1.003 (range 0.998 to 1.010 with 95% confidence), suggesting that the change has little effect on total pageviews
When comparing the ratio of unique events per total events per session for the test and control groups, similar results were observed, with the control group having 0.9986 unique events per session, and the control group having 0.9994 unique events per session. No significant changes to these trends were observed over time.
Queries
[edit]SELECT count(*) AS total_Events, event_eventName, left(TIMESTAMP, 8) AS mmddyyyy
FROM `RelatedArticles_16352530`
WHERE wiki = 'enwiki' and event_skin = 'minerva-stable'
GROUP BY mmddyyyy, event_eventName;
SELECT count (DISTINCT event_userSessionToken) AS total_Events, event_eventName, LEFT(TIMESTAMP, 8) AS ddmmyyyy
FROM RelatedArticles_16352530
WHERE wiki = 'enwiki' and event_skin = 'minerva-stable'
GROUP BY ddmmyyyy, event_eventName;
Event logging
[edit]As measured for the first two weeks of 2016 on a sampled basis, related articles items were frequently tapped on mobile web beta Wikipedias (the skin "minerva-beta" in the table below) - 19.7% of the time when seen. The relatively high tap rate on the mobile web beta is believed to be attributable to the interesting content, as well as in part automatically collapsed sections in its phone form factor layout yielding a higher impact visually to the related articles at the end of the article.
Interestingly, while users on the desktop web Wikipedias (see "vector" in data below) were likely to see the related articles panel (35% ready-to-seen event ratio versus mobile web's 27.3% ratio), desktop users evidently were far less likely to click on a related article - they apparently clicked only 3.4% of the time when seeing the related articles panel.
select event_skin, event_eventName, count(*)
from RelatedArticles_14496900
where timestamp > '2016' and timestamp < '20160115' group by event_skin, event_eventName;
event_skin event_eventName count(*) cologneblue ready 3 minerva-beta clicked 217 minerva-beta ready 4025 minerva-beta seen 1099 modern ready 114 modern seen 62 monobook clicked 1 monobook ready 270 monobook seen 82 vector clicked 48 vector ready 4053 vector seen 1429
The relatively lower click rate on the desktop web is believed to be attributable to the richly laid out expanded sections yielding a relatively lower visual impact for the related articles. It may also be related to the mechanics of beta opt-in: on the mobile web beta opt-in enables all beta features, whereas on the desktop web users are able to opt-in to specific beta features. It should be noted that some users on desktop web also enable the feature to auto-enroll on all beta features.
Tablet devices by default are presented the mobile website, which commonly expands sections, but still has a relatively lighter weight layout. But users have the ability to change to desktop mode, where again sections are expanded but there's a relatively heavier weight layout.
Note that for a confined set of tablet devices, although it's an imperfect analysis and there are a variety of potentially impacting factors that could muddle the analysis, the data suggest that we can at least say with some level of confidence that
- mobile web beta treatment dramatically outperformed the desktop treatment
- although users with a confined set of tablets on mobile web beta had not insignificant engagement with related articles, their rate of engagement was somewhat lower than the fuller mobile web beta population (confined tablet population's engagement at 16.7% versus full mobile web beta population's engagement of 19.7% for a two week period and 13.2% versus 18.4% for the full period since it was introduced on mobile web beta) - but this was still quite promising on mobile web beta
select event_skin, event_eventName, count(*)
from RelatedArticles_14496900
where timestamp > '2016' and timestamp < '20160115'
and (userAgent like '%Nexus 7%' or userAgent like '%Nexus 9%' or userAgent like '%iPad%')
group by event_skin, event_eventName;
minerva-beta clicked 11 minerva-beta ready 262 minerva-beta seen 66 monobook ready 1 monobook seen 1 vector ready 37 vector seen 9
All events and skins since related articles available in beta for confined set of tablets...
select event_skin, event_eventName, count(*)
from RelatedArticles_14496900
where userAgent like '%Nexus 7%' or userAgent like '%Nexus 9%' or userAgent like '%iPad%'
group by event_skin, event_eventName;
minerva-beta clicked 39 minerva-beta ready 1485 minerva-beta seen 295 monobook ready 2 monobook seen 1 vector ready 70 vector seen 21
All events and skins since related articles available in beta for all UAs...
select event_skin, event_eventName, count(*)
from RelatedArticles_14496900
group by event_skin, event_eventName;
cologneblue ready 9 cologneblue seen 2 minerva-beta clicked 768 minerva-beta ready 19009 minerva-beta seen 4170 modern ready 185 modern seen 105 monobook clicked 4 monobook ready 535 monobook seen 171 vector clicked 110 vector ready 8005 vector seen 2848
Do note that a CTA experiment to heighten the general enrollment in mobile web beta was running in the latter part of 2015 but was disabled, hence relatively lower event counts for the two week period observed at the start of January 2016. Engagement with the related articles was relatively high with both the CTA in force and has continued to be relatively high (albeit slightly lower) since the CTA was removed. See the data above. Note also that a small experiment with collapsed sections has been running, but should have a negligible impact on the analysis in this page.
HTTP Referer header analysis
[edit]One additional signal in the environment suggesting the efficacy of the mobile web beta related articles feature is the share of internally referred pageviews as a percentage of total pagevews. Again, caveats could apply, but it appears that prior to the related articles feature, mobile web beta internally referred pageviews were in the low 50s percentagewise, whereas with the feature they are now in the high 60s percentagewise (the stable channel where the feature was not implemented stayed relatively consistent around 25%). However, this is only a signal; partial rollout in the stable channel of the mobile web would be more telling.
What follows are relative internally referred pageviews rates on the mobile web for a set of Thursdays - two Thursdays in the first two weeks of January, and two Thursdays prior to the related articles feature being released in beta on the mobile web and desktop web. Dates coinciding with holidays or known high rate fundraising campaign banners are not included, as they would complicate the analysis.
Thursday, January 14, 2016
[edit]Note on this query: feature running in mobile web beta Wikipedias.
select referer_class, x_analytics_map['mf-m'], count(1)
from wmf.webrequest where year = 2016 and month = 1 and day = 14
and access_method = 'mobile web'
and agent_type = 'user'
and is_pageview = true
and normalized_host.project_class = 'wikipedia'
and pageview_info['page_title'] <> '-' and pageview_info['page_title'] <> 'Special:Search'
group by referer_class, x_analytics_map['mf-m'];
beta: 48450/(48450+14958+8465) = 67.4% stable: 58400028/(58400028+56362895+113239899) = 25.6% external NULL 113239899 external b 8465 unknown NULL 56362895 unknown b 14958 internal NULL 58400028 internal b 48450
Thursday, January 7, 2016
[edit]Note on this query: feature running in mobile web beta Wikipedias.
select referer_class, x_analytics_map['mf-m'], count(1)
from wmf.webrequest where year = 2016 and month = 1 and day = 7
and access_method = 'mobile web'
and agent_type = 'user'
and is_pageview = true
and normalized_host.project_class = 'wikipedia'
and pageview_info['page_title'] <> '-' and pageview_info['page_title'] <> 'Special:Search'
group by referer_class, x_analytics_map['mf-m'];
beta: 51732/(51732+14540+8137) = 69.5% stable: 59881357/(59881357+55286651+109964317) = 26.6% external NULL 109964317 external b 8137 unknown NULL 55286651 unknown b 14540 internal NULL 59881357 internal b 51732
Thursday, December 3, 2015
[edit]Note on this query: feature was not yet running in mobile web beta Wikipedias.
select referer_class, x_analytics_map['mf-m'], count(1)
from wmf.webrequest where year = 2015 and month = 12 and day = 3
and access_method = 'mobile web'
and agent_type = 'user'
and is_pageview = true
and normalized_host.project_class = 'wikipedia'
and pageview_info['page_title'] <> '-' and pageview_info['page_title'] <> 'Special:Search'
group by referer_class, x_analytics_map['mf-m'];
beta: 184067/(184067+55814+119778) = 51.1% stable: 50273872/(50273872+51372508+102363512) = 24.6% external NULL 102363512 external b 119778 unknown NULL 51372508 unknown b 55814 internal NULL 50273872 internal b 184067
Thursday, November 19, 2015
[edit]Note on this query: feature was not yet running in mobile web beta Wikipedias.
select referer_class, x_analytics_map['mf-m'], count(1)
from wmf.webrequest where year = 2015 and month = 11 and day = 19
and access_method = 'mobile web'
and agent_type = 'user'
and is_pageview = true
and normalized_host.project_class = 'wikipedia'
and pageview_info['page_title'] <> '-' and pageview_info['page_title'] <> 'Special:Search'
group by referer_class, x_analytics_map['mf-m'];
beta = 189359/(189359+58549+121871) = 51.2% stable = 51119529/(51119529+54209700+105205731) = 24.3% external NULL 105205731 external b 121871 unknown NULL 54209700 unknown b 58549 internal NULL 51119529 internal b 189359
Proposal for moving forward
[edit]In light of the feedback noted above, I would like to propose the following next steps for this feature:
- Publish variables and code that impact article selection (see FAQ)
Done
- Fix non-free image issue via: T124225
Done
- Work on T127068 to remove from disambiguation pages
- Rollout on mobile web on select wikis (it seems like stakeholders are less concerned with the experience here, and the 'performance' is much higher)
- Rollout small traffic a/b test on desktop on a few wikis and either measure session depth change or use quick survey (the latter is more difficult)
- Use results of 3 to inform possible desktop rollout (Hear back from De and Ru Wikipedias as to whether or not a rollout is desired, given 'Themenring' policy)
Any thoughts, suggestions? I will post on the talk page as well here: Topic:Sz30fah2s3enrql4
On June 15th 2016 21:00 local time the German speaking Wikipedia will most likely reject this feature with a large mayority. Sorry for all the money spend. --Eingangskontrolle (talk) 16:12, 4 June 2016 (UTC)
Improvements/Feature requests
[edit]- enable close button on individual items to alert editors (or tweak algorithm) when a particularly recommendation is not helpful
- merge with see-also
- Recast as a "Find Similar Articles" button with a long list of search-style results