Page MenuHomePhabricator

Analyze usage of desktop text size beta feature
Closed, ResolvedPublic

Description

Background

We would like to learn how users so far have been using the new text size beta feature by answering a set of questions about the feature. This analysis will help us identify the next steps for the feature and confirm the deployment plan.

Acceptance criteria

Answer the following questions:

  • What percentage of users who have opted into the beta feature have changed their text?
    • What were they most likely to change it towards?
  • Of all clicks to the text section of the new menu, what percentage is for each text option?
  • Were there any other interesting observations around the usage of the text menu?

Instrumentation notes:

font sizefont value in schemaevent name of click on it
Small (default)0skin-client-pref-vector-feature-custom-font-size-value-0
Standard1skin-client-pref-vector-feature-custom-font-size-value-1
Large2skin-client-pref-vector-feature-custom-font-size-value-2

Event Timeline

ovasileva updated the task description. (Show Details)

Here is the font size stats on desktop web by skin version. A few questions based on the data

  1. What is the default font value on vector-2022 ?
  2. What does the value disabled stand for on vector-2022?
  3. What is the default font value on vector ? Should we include vector skin in this analysis?
skinfontsessions
vector-2022NA48
vector-20220132970648
vector-2022113299
vector-2022disabled47
vector-202222285
vectorNA46
vectorlarge15
vectorregular35
vectorsmall730
vector053258754

data timeframe: 2024-03-01 ~ 2024-03-29
data source: desktopwebuiactionstracking schema

cc: @ovasileva , @KSarabia-WMF

What is the default font value on vector-2022 ? Regular
What does the value disabled stand for on vector-2022? I believe this is what is returned when it checks whether the user preference is enabled or not. @Jdlrobson Do you mind confirming?
What is the default font value on vector ? Regular Should we include vector skin in this analysis? *I'm not sure. @ovasileva Is it necessary?*

What is the default font value on vector-2022 ? Regular

Hi @KSarabia-WMF, thanks for taking a look. Something is not right. Currently, there are 3 font preference settings: small, standard, large. Can you confirm whether the mapping table below of font settings and even.font values in desktopwebuiactionstracking schema is correct?
If Regular is the default, assume i.e, standard, we should have largest number of sessions with event.font=1, correct? But the data shows most of sessions have event.font=0.

font preference settingevent.font in desktopwebuiactionstracking schema
Small0
Standard1
Large2

image.png (310×406 px, 18 KB)

query

SELECT event.skin, event.font,
COUNT(DISTINCT event.token) AS sessions
FROM event.desktopwebuiactionstracking 
WHERE year = 2024  AND month=3 
GROUP BY event.skin, event.font

@jwang Small is the default currently so we would expect most sessions to be event.font=0 so this is correct.

No need to include vector skin in analysis we are not changing font size there.

The value disabled is not expected. Please filter out from results. I am not sure what's causing this - it could be a on-wiki gadget override or a historic artifact but it should be ignored.

@ovasileva, here are the analysis result. The answer to the third questions is a very rough estimate. Let me know if you disagree with any of the assumptions.

Of all clicks to the text section of the new menu, what percentage is for each text option?

  • In March 2024, there were 4140 clicks on the font menu across wikis.
  • Among them there are more clicks to select the standard font size, followed by clicks for a smaller size. Clicks for the large font size are fewer. The trend remains consistent on a daily basis.
  • A detailed breakdown is shown below:
    • 30.2% were for small font size
    • 46.7% were for standard font size
    • 23.1% were for large font size

What were they most likely to change it towards?

  • The 4140 clicks were made by 1279 unique sessions in March 2024.
  • 55% of unique sessions were opt for a non-default font size, either standard or large. Nine sessions are excluded because they have multiple most recent events with different font size settings but the same timestamp. We cannot determine their current font setting status.
  • A detailed breakdown is shown below:
    • 44.9% of sessions eventually opt for small (default: font=0) font size
    • 43.6% of sessions eventually opt for standard (font=1) font size.
    • 11.5% of sessions eventually opt for large (font=2) font size.

What percentage of users who have opted into the beta feature have changed their text?

We don’t have an accurate answer for these questions. We have click data on a session basis and beta feature enrollment status on a user basis. However, we cannot connect sessions with users. Below is a very rough estimate.

23959 users have enabled beta feature preference cross wikis since the deployment (to confirm whether it is in December).
If we assume that the 1279 sessions which clicked in March represent 1279 users, and that the monthly number of clicked users has remained consistent since the deployment, from December 2023 to March 2024, then we estimate that 5116 (1279*4) users have clicked font menu since deployment. The overall click rate is estimated to be 21.4% (5116/23959). Given that 55% of unique sessions opted for a non-default font size in March, we estimate that 11.7% (21.4% * 55%) of users who enabled the beta feature opted for a non-default font size. However, it's important to note that this is an overestimate, as one user might have multiple sessions.

Code:link