Language packs: Meta’s mobile localization solution

Language Pack: Meta’s mobile localization solution

More than 3 billion people around the world rely on our services every month. On mobile, around 57% of people on Facebook for Android and 49% of people on Facebook for iOS use the app in a language other than English. Offering the best experience to these people, in their own language, is an important step in helping people around the world connect. This means giving people an interface in their local language and making sure it’s accurate, down to grammar, gender nouns, plurals, etc. With dozens of languages ​​(hundreds for some interfaces) to fit every UI interface of our app, it’s a big challenge. To address this, we have developed a new mobile localization infrastructure, which includes downloadable language packs, a download-on-demand translation delivery solution for Android and iOS. Language Packs enable us to provide a localized, high-quality user experience for people around the world. With this new infrastructure, engineers have the ability to create and deliver gender translations in a simple and intuitive way. This infrastructure has already been adopted by Facebook and Workplace and we are exploring integration into our other Meta apps. With language packs, we’ve reduced Facebook’s IPA size for iOS by 16.6MB, which helps it stay fast and responsive for people using it.

Why do we need language packs?

Traditional localization frameworks offered by native Android and iOS platforms have two key scaling problems.

Problem 1: Translation accuracy

Native Android and iOS location frameworks only support plain plain text and pluralized text, making it difficult to create gender text without boilerplate code. To provide billions of people around the world with a quality localized experience, we have developed our API and our string framework: ours is called FBT. FBT supports PHP, Hack, Javascript and React Native. This API allows us to define and ship grammatically correct translated texts, including both the viewer and the subject gender text, in a simple and intuitive way.

FBT API

fbt(
   'Write on ' +
     fbt.pronoun('possessive', gender) +
     ' timeline...',
   'Placeholder text for inline composer',
 )

Android

 <string name="title_male" description="Placeholder text for inline composer">Write on his timeline...</string>
 <string name="title_female" description="Placeholder text for inline composer">Write on her timeline...</string>
 <string name="title_unknown" description="Placeholder text for inline composer">Write on their timeline...</string>

IOS

if (gender == Male) {
  FBLocalizedString("Write on his timeline...", "Placeholder text for inline composer");
} else if (gender == Female) {
  FBLocalizedString("Write on her timeline...", "Placeholder text for inline composer");
} else {
  FBLocalizedString("Write on their timeline...", "Placeholder text for inline composer");
}

Problem 2: language support and app size

All native floating strings and native reaction strings must be translated into the appropriate formats and integrated into the app bundle before being sent to the app store. However, as we support multiple languages ​​and features in the app, its size increases rapidly. People may be reluctant to upgrade to newer versions of the app due to device bandwidth and storage issues. As a result, new features and improvements would then not be provided to people’s devices and prevent them from receiving important security updates.

In 2020, Facebook for iPhone supported 35 different languages ​​for people around the world. We ran an app size test for the Facebook app for iOS and found that by removing all translation files from the package, we could have saved up to 16.6MB download size. Since most people only use one language on their device, the rest of the translation files in the package would still be a wasted resource, taking up unnecessary space.

With these issues in mind, we have developed a solution called Downloadable Language Packs for Android and iOS platforms. To support the FBT API and ship accurate translations to mobile devices, each language pack file includes all translation variants. To simultaneously support multiple languages ​​and limit app size, we now provide an on-demand download solution for language packs, where devices download only the language pack file to use. Since enabling downloadable language packs, we’ve added nearly a dozen more languages, including Burmese, Georgian, Latvian and Sinhala to Facebook on Android with no impact on the size of the app.

End-to-end flow

There are two major phases of the language pack framework: one phase occurs before the release of the mobile build and the other occurs after it. As we build the app binaries, we also build language packs for the FBT strings used in the build. To adopt new strings and recent updates for translations, each release build must be associated with its language pack in our cloud storage before submitting it to the app store. When the release build is published in the App Store, the language pack is downloaded from the cloud storage or loaded from a disk during localization initialization as part of the app launch. The client side will access the language pack file and load the translation data into memory. Thereafter, each time the string API is called, we look for its translation from the parsed data.

Building language packages

Each day, a set of native strings are created or modified in the codebase of each Meta app. We have created an automated pipeline to extract strings, collect translations from the database and group them. For each mobile build released, we have a build phase to create language packs for all supported localizations based on the strings in the latest release version and upload the language packs to our cloud storage.

For each mobile build released, we have a build phase to create language packs for all supported localizations based on the strings in the latest release version and upload the language packs to our cloud storage.

Creation of FBT strings

FBT is an open source localization framework that provides a more efficient way to define content for flexible, high-quality localization. This is a simple function wrapper of what would otherwise be English text. The framework is designed to be easy for engineers to use.

Example: plain text

fbt('Hello, World', 'a simple example', {project:"foo"})

// Translation: "Hello, World"

Example: complex string

fbt("View " +
     fbt.name('user', shortName, gender) +
       "'s Timeline - " +
       fbt.plural('follower', count, {many: 'followers', showCount: 'yes'}),
     'In user composer, gives details about the person to who the post is directed.',
   )

// Translation: "View Lu's timeline - 1 follower"
// Translation: "View Lu's timeline - 5 follower"

Extraction

Our pipeline collects fbt() from codebase and transforms it into the abstract extracted object. In the extracted object, two types of information will be used to build the language packs

  • id – hash key determined by text + description + relevant metadata
  • text_or_table – A multilevel lookup table, with lookup values ​​defined at each level based on FBT callsite

Example: extracted object of a complex string in the previous section

[
  {
     "description":"In user composer, gives details about the person to who the post is directed.",
     "id":"4sIjkwerw",
     "text_or_table":{
        "UNKNOWN":{
           "ONE":"View {user}'s Timeline - 1 follower",
           "OTHER":"View {user}'s Timeline - {number} followers"
        }
     },
            "variations": [
                {
                    "type": "GENDER",
                    "token": "user"
                },
                {
                    "type": "NUMBER",
                    "token": "number"
                }
            ],
            "tokens": {
                "name": "%1$@"
                "number": "%2$ld"
            }

  },
]

Structure of the language package

Once all the extracted objects have been collected, the language pack creation phase will retrieve the translations from the database and encode them in a binary format. The high-level structure for each language pack file is a hash map table.

Language_pack

|

|____ resource_id_1 : nested_fbt_resource_data

|

|____ resource_id_2 : nested_fbt_resource_data

|

|...
  • resource_id – as “id” in the extracted object. This unique key will be used on the client side to find translations at run time.
  • nested_fbt_resource_data – a multilevel lookup table, with lookup values ​​defined at each level based on the FBT token
  • Each token has a type and possible variations (e.g. gender, plural)
  • Number of levels and search values ​​determined by the FBT API callsite
  • The values ​​of the FBT tokens are known only at runtime, so the dictionary lookup occurs at runtime

Example: nested_fbt_resource_data complex string in the Greek language pack

{
   "4sIjkwerw":{
      "male":{
         "one":"Δείτε το Χρονολόγιο του {user} - 1 ακόλουθος",
         "other":"Δείτε το Χρονολόγιο του {user} - {number} ακόλουθοι"
      },
      "female":{
         "one":"Δείτε το Χρονολόγιο της {user} - 1 ακόλουθος",
         "other":"Δείτε το Χρονολόγιο της {user} - {number} ακόλουθοι"
      },
      "default":{
         "one":"Δείτε το Χρονολόγιο του χρήστη {user} - 1 ακόλουθος",
         "other":"Δείτε το Χρονολόγιο του χρήστη {user} - {number} ακόλουθοι"
      }
   }
}

Using language packs in the app

Once the client side has adopted the downloadable language pack infrastructure, the translations are loaded from the downloaded on-demand language pack binary rather than as a bundled string resource file. To provide translations to all UI views, the localization initialization step must be completed before rendering the first UI view. Once the network setup is complete, you can immediately start the location setup. The locale setting is an asynchronous step on startup. In this step, we will check if the language pack file for a specific local and client version is on disk. If this file exists, it will be decompressed, parsed and loaded into memory. However, if this file cannot be found in memory, we will initiate a network request to download the corresponding language pack binary file. If the localization setup is completed before the first UI view is rendered, the user experience will be perfect. Conversely, if the download request is still being processed or the file cannot be processed correctly, users must retry the download or return to the previous locale.

Localization configuration improvements

For people with poor quality or unpredictable network access who are unable to download language packs at boot time, we have made two changes on the client side.

  • Preload is initiated on existing client builds before the new client version is released. A background task is scheduled to download the language pack file based on the new version and language of the app.
  • Fallback: Most translations don’t change from version to version. In the localization initialization phase, we load an outdated language pack file if the target language pack version is not available. In this session, a background task is scheduled to retrieve the target language pack version.

For both improvements, people using the app have a better chance of getting the latest translations in the next session without interrupting the user experience.

Performance on the Facebook app

Downloading language packs on demand is an infra-reliable solution that supports multiple languages ​​and savings on app size at the same time; users will only download the language pack file they will use. Let’s check out some data points that show infrastructure efficiency and end user experience based on the Facebook app. The download size of each language pack varies from 600KB to 2MB depending on the language. For Facebook on iOS, the upload success rate of this infra is over 99.99%. Over 99.8% of people load language packs from disk and the average load time is around 80ms. For Facebook for Android, the upload success rate is over 99.7%. The average time to load a language pack file from a disk is approximately 780 ms across all device classes. Based on our mid-year measurements, the download of language packs did not show a negative impact on Facebook app launch and overall critical metrics.

Meta-scale language pack integration

While this solution has worked well for Facebook and Workplace, the language pack framework is not a one size fits all solution. For example, for apps that don’t require a lot of translations or that are sensitive to startup performance, bundled language packs might be a better option. To tackle our next challenge, integrating this localization framework across the enterprise, we’ll need to consider which solutions will best fit each of our apps.

We continue to explore new ways to deliver high-quality localized experiences to enable people in different countries and regions to connect with each other. We have opened the GitHub FBT Javascript repository for public access at facebook.github.io/fbt/ so anyone can know about FBT.

Thanks to everyone who contributed to the FBT framework and language packs.

Leave a Comment

Your email address will not be published.