article banner (priority)

Effective Kotlin Item 49: Use caching when possible

This is a chapter from the book Effective Kotlin. You can find it on LeanPub or Amazon.
If there is one performance optimization you should know, it is certainly caching. It is used in many different ways on many different levels. Your computer has a CPU cache and a disk cache. Your browser has a cache for web pages. Our network providers have caches for different types of data. JVM has many different caches that are used to speed up our applications. We define our own caches in both backend and Android applications. Caching is everywhere because it’s a very powerful technique. I would even say that caching is the most powerful way to speed up applications.
The idea behind caching is simple. A cache contains a redundant copy of data that is stored such that we can access it quickly. Let me show you a typical example. Imagine that your application needs to fetch users by their id from a web service. You can implement this in the following way:
class WebUserRepository(
    val userClient: UserClient
) : UserRepository {
    override suspend fun getUser(id: Int): User =
        userClient.fetchUser(id)
}
The problem is that for each user we want to get, we need to send a network request and wait for the response. One way to improve this is by storing the results of previous requests so we don't need to send a network request every time we want to get a user with a certain id. We can do this using a map:
class CachedWebUserRepository(
    val userClient: UserClient
) : UserRepository {
    private val users = ConcurrentHashMap<Int, User>()
    
    override suspend fun getUser(id: Int): User =
        users.getOrPut(id) { userClient.fetchUser(id) }
}
This is a simple implementation of a cache. Our map represents redundant memory because if we cleared it, no data would really be lost and our repository would just need to fetch it again. We can also see that we can access data from a cache quickly as we don't need to send a network request. However, there are two problems with caches. Firstly, if we cache data that changes, our cache can become stale. For instance, if we cache users and a user’s name changes, the data in our cache will be outdated. The standard solution to this problem is to use a caching library (like Caffeine or Ehcache) that allows us to specify how long a certain record should be considered valid. For instance, you can specify that our user record should be considered valid for one minute. After that, we will need to fetch it again. Such a solution makes a lot of sense on backend applications where users often fetch the same data repeatedly but the data rarely change. This is a typical example of using a cache on the backend:
class CachedWebUserRepository(
    val userClient: UserClient
) : UserRepository {

    private val users = Caffeine.newBuilder()
        .expireAfterWrite(1, TimeUnit.MINUTES)
        .buildSuspending<Int, User>()

    override suspend fun getUser(id: Int): User =
        users.get(id) { userClient.fetchUser(id) }
}
Using time-based expiration is far less popular on Android, where we tend to use caching for data that does not change, like configurations or database connections, or data that are only changed in an application, where we can update cached objects when they change.
private val connections =
    ConcurrentHashMap<String, Connection>()

fun getConnection(host: String) =
    connections.getOrPut(host) { createConnection(host) }
The second problem with caching is that it essentially means we are buying performance in exchange for memory. If we cache too much data, we can run out of memory, but there are a couple of tricks that can help us with this problem. One is to expire cache entries that are used less often (expire after write) or to set a cache size limit. However, the most powerful technique is making your cache use soft or even weak references. Let me explain this.
In Kotlin, when a variable references a value, it is a strong reference, so the existence of this reference prevents the garbage collector from cleaning up the value. However, JVM also offers two other kinds of references:
  • A weak reference does not prevent the Garbage Collector from cleaning up a value. So, if no other reference is using this value, it will be cleaned up.
  • A soft reference does not guarantee that a value won’t be cleaned up by the GC either, but in most JVM implementations this value won’t be cleaned up unless memory is needed.
If you are concerned about memory usage, the simplest way is to use a soft reference cache as this will not be limited when there is enough memory, but it will be cleaned up when memory is needed.
class CachedWebUserRepository(
    val userClient: UserClient
) : UserRepository {

   private val users = Caffeine.newBuilder()
      .maximumSize(10_000)
      // When size is reached, less used entries are removed
      .expireAfterAccess(10, TimeUnit.MINUTES)
      //When entry is not used for 10 minutes, it is removed
      .softValues() // Using soft references
      .buildSuspending<Int, User>()
   
   override suspend fun getUser(id: Int): User =
       users.get(id) { userClient.fetchUser(id) }
}

Summary

  • Use caching to speed up data access and reduce the number of heavy requests (like web and file system requests).
  • Use a caching library to avoid common pitfalls and to get more features.
  • Use time-based expiration to avoid stale data and to limit the size of the cache.
  • Use a cache size limit and soft references to avoid running out of memory.