Skip to content

Memcache

Jump to bottom Edit New page
iwhurtafly edited this page Oct 28, 2012 · 1 revision

Memcache is an in-memory transient key/value store typically used for caching. It's a key technology for building scalable web apps.

Local Memcache setup

You will need to run a local memcached process to test against. Install it with Homebrew on Mac OS X:

:::term
$ brew install memcached

Or on Ubuntu:

:::term
$ sudo apt-get install memcached

Or follow these instructions for Windows.

You can also download the source from the memcached site.

Now run the memcached daemon:

:::term
$ memcached -vv
slab class   1: chunk size     88 perslab 11915
...
<15 server listening

Using Memcache from Ruby

If you are already using memcache, you won't need to rewrite your code for it to run on Heroku. Only the gem dependency declaration and the memcache client object initialization (described below) need to change.

Gem installation

If you're using dalli but not Bundler, be sure to add it to your `.gems` manifest.

First remove any memcache client gems you may already be using from your application, such as memcache-client.

You must use a memcache gem that supports SASL. There are two available:

Dalli (recommended): We recommend using dalli. Dalli supports Rails 2 & 3 and Ruby 1.8.7 & 1.9.2. It is compatible with all of our deployment stacks, and is the only gem you can use if you wish to use memcache for your sessions store in Rails 2 or 3. Full setup instructions for using dalli are available at the Dalli README. Additionally, this article details using memcache and Dalli to cache static assets in Rails 3.1+. Please note that Dalli does not support older caching libraries such as cache-money or cache_fu.

Rails 2.x is no longer supported in Dalli v1.1.0 and higher. Use the latest 1.0 version ([v1.0.5](https://github.com/mperham/dalli/tree/v1.0.5) at the time of this writing) for Rails 2.x compatibility. In your Gemfile add: `gem 'dalli', '~>1.0.5'`.

**Mac OS 10.5 Users** A bug in port / Mac OS 10.5 causes an error when trying to build the memcached-northscale gem. Snow Leopard (10.6) users are unaffected. To resolve this, uninstall the cyrus-sasl2 library and then build the memcached-northscale gem. Once the memcached-northscale gem is built, you can reinstall the cyrus-sasl2 package if you require it.

memcached-northscale is a Ruby interface to the libmemcached C client. Memcached-northscale should be used only if you require compatibility with old caching libraries like cache-money.

Add whichever gem you choose to your Gemfile and run bundle install.

Deploying to Heroku

To use memcache on Heroku, install the memcache add-on:

:::term
$ heroku addons:add memcache

If you're using the memcached gem as described above, your app will use memcache on Heroku without any further code changes.

You will also have direct access to your memcache configuration via config vars:

:::term
$ heroku config
...
MEMCACHE_PASSWORD    => xxxxxxxxxxxx
MEMCACHE_SERVERS     => instance.hostname.net
MEMCACHE_USERNAME    => xxxxxxxxxxxx
...

Rails setup

Rails 2.3.3 and later abstracts caching mechanisms through Rails.cache. You must be using Rails 2.3.3 or newer to use memcache on Heroku.

Please see our documentation for a comprehensive look at Rails caching strategies on Heroku.

Dalli: Rails 3

In your Gemfile:

:::ruby
gem 'dalli'

In config/environments/production.rb:

:::ruby
config.cache_store = :dalli_store

Dalli: Rails 2.3.3 – 2.3.x

In config/environment.rb:

:::ruby
config.gem 'dalli'

In config/environments/production.rb:

:::ruby
# Object cache
require 'active_support/cache/dalli_store23'
config.cache_store = :dalli_store

In config/initializers/session_store.rb:

:::ruby
# Session cache
ActionController::Base.session = {
  :namespace   => 'sessions',
  :expire_after => 20.minutes.to_i,
  :memcache_server => ['server-1:11211', 'server-2:11211'],
  :key         => ...,
  :secret      => ...
}

require 'action_controller/session/dalli_store'
ActionController::Base.session_store = :dalli_store

Memcached-northscale: Rails 2.3.3 – 2.3.x

Add the gem via your Gemfile:

:::ruby
gem "memcached-northscale"

In config/environment.rb:

:::ruby
config.gem 'memcached-northscale', :lib => 'memcached'
require 'memcached'

In config/environment/production.rb:

:::ruby
config.cache_store = :mem_cache_store, Memcached::Rails.new

Development environment

In your development environment, Rails.cache will default to an in-memory store that doesn't require a running memcached.

Testing

You can test writing to the local memory cache in a console, or via heroku console:

:::term
$ script/console
>> Rails.cache.write('color', 'red')

And fetching the data back:

:::term
>> Rails.cache.read('color')
=> "red"

Using Memcache with Sinatra

With Dalli and Sinatra, it's as easy as adding a :cache setting:

:::ruby
set :cache, Dalli::Client.new

get '/' do
  settings.cache.set('color', 'blue')
  settings.cache.get('color')
  ...
end

Getting stats on usage

Dalli gem gives you a handy stats method.

:::term
$ heroku console
Ruby console for myapp.heroku.com
>> Rails.cache.stats
=> {"mc3.ec2.northscale.net:11211"=>{"pid"=>"10414", "uptime"=>"59892", "time"=>"1301152323", "version"=>"1.4.4_188_g0117a2c", "pointer_size"=>"64", "rusage_user"=>"418.600000", "rusage_system"=>"545.170000", "daemon_connections"=>"10", "curr_connections"=>"1246", "total_connections"=>"666019", "connection_structures"=>"1271", "cmd_get"=>"5736", "cmd_set"=>"1028", "cmd_flush"=>"0", "auth_cmds"=>"3", "get_hits"=>"4852", "get_misses"=>"884", "delete_misses"=>"0", "delete_hits"=>"0", "incr_misses"=>"0", "incr_hits"=>"0", "decr_misses"=>"0", "decr_hits"=>"0", "cas_misses"=>"0", "cas_hits"=>"0", "cas_badval"=>"0", "bytes_read"=>"11886315", "bytes_written"=>"58402597", "limit_maxbytes"=>"67108864", "rejected_conns"=>"0", "threads"=>"4", "conn_yields"=>"0", "evictions"=>"0", "curr_items"=>"81", "total_items"=>"1028", "bytes"=>"213275", "engine_maxbytes"=>"104857600"}}

Further reading

Using Memcache from Java

Spymemcached is a popular Java Memcache client. In order to use Spymemcached in your project you have to declare the dependency in your build and initialize the client from the environment variables that Heroku provides to your application.

{.note} Sample code for the Java Memcache demo application is available on GitHub.

Add Spymemcached to your pom.xml

Add the following repository and dependency to your pom.xml in order to use Spymemcached to connect to Memcache:

<repository>
  <id>spy</id>
  <name>Spy Repository</name>
  <layout>default</layout>
  <url>http://files.couchbase.com/maven2/</url>
  <snapshots>
    <enabled>false</enabled>
  </snapshots>
</repository>

...

<dependency>
    <groupId>spy</groupId>
    <artifactId>spymemcached</artifactId>
    <version>2.7.3</version>
</dependency>

Use Spymemcached in your application

:::java
AuthDescriptor ad = new AuthDescriptor(new String[]{"PLAIN"}, new PlainCallbackHandler(System.getenv("MEMCACHE_USERNAME"), System.getenv("MEMCACHE_PASSWORD")));
ConnectionFactoryBuilder factoryBuilder = new ConnectionFactoryBuilder();
ConnectionFactory cf = factoryBuilder.setProtocol(Protocol.BINARY).setAuthDescriptor(ad).build();

MemcachedClient memcachedClient = new MemcachedClient(cf, Collections.singletonList(new InetSocketAddress(System.getenv("MEMCACHE_SERVERS"), 11211)));
memcachedClient.add("test", 0, "testData");

Using Spymemcached with Spring

Use the following Java configuration class to set up a MemcachedClient object as a singleton Spring bean:

:::java
@Configuration
public class SpringConfig {
    @Bean
    public PlainCallbackHandler getPlainCallbackHandler() {
        return new PlainCallbackHandler(System.getenv("MEMCACHE_USERNAME"), System.getenv("MEMCACHE_PASSWORD"));
    }
    
    @Bean
    public AuthDescriptor getAuthDescriptor() {
        return new AuthDescriptor(new String[]{"PLAIN"},
                getPlainCallbackHandler());
    }
    
    @Bean
    public ConnectionFactory getConnectionFactory() {
        ConnectionFactoryBuilder factoryBuilder = new ConnectionFactoryBuilder();
        return factoryBuilder.setProtocol(Protocol.BINARY).setAuthDescriptor(getAuthDescriptor()).build();        
    }
    
    @Bean
    public InetSocketAddress getServerAddress() {
        return new InetSocketAddress(System.getenv("MEMCACHE_SERVERS"), 11211);
    }
    
    @Bean 
    public MemcachedClient getMemcachedClient() throws IOException{
        MemcachedClient memcachedClient = 
                new MemcachedClient(
                        getConnectionFactory(), 
                        Collections.singletonList(getServerAddress()));
        return memcachedClient;
    }
}

or the following XML configuration file:

:::xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:context="http://www.springframework.org/schema/context"
  xsi:schemaLocation="http://www.springframework.org/schema/beans
      http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
      http://www.springframework.org/schema/context
      http://www.springframework.org/schema/context/spring-context-3.0.xsd">

  <context:annotation-config/>
  <context:property-placeholder/>
  
  <bean id="plainCallbackHandler" class="net.spy.memcached.auth.PlainCallbackHandler">
       <constructor-arg index="0" value="${MEMCACHE_USERNAME}"/>
       <constructor-arg index="1" value="${MEMCACHE_PASSWORD}"/> 
  </bean>
  
  <bean id="authDescriptor" class="net.spy.memcached.auth.AuthDescriptor">
       <constructor-arg index="0">
         <array><value>PLAIN</value></array>
       </constructor-arg>
       <constructor-arg index="1" ref="plainCallbackHandler"/> 
  </bean>
  
  <bean id="memcachedClient" class="net.spy.memcached.spring.MemcachedClientFactoryBean">
    <property name="servers" value="${MEMCACHE_SERVERS}:11211"/>
    <property name="protocol" value="BINARY"/>
    <property name="authDescriptor" ref="authDescriptor"/>
  </bean>

</beans>

Sample code

To see a complete, working example, check out the sample code in github. The readme explains more about the example.

Using Memcache from Python

The memcache add-on requires SASL authentication. The de-facto Python memcache module, pylibmc, supports this protocol.

Module installation

To connect to your local memcached instance, you can use the pylibmc module. You can install it easily with pip:

You will need [libmemcached](http://libmemcached.org/libMemcached.html) available to install the `pylibmc` module. 
:::term
$ pip install pylibmc

Once pylibmc is installed, you can connect to memcache easily:

:::python
import pylibmc

# Connect to memcache.
mc = pylibmc.Client(servers=['127.0.0.1'])

Deploying to Heroku

To use memcache on Heroku, install the memcache add-on:

:::term
$ heroku addons:add memcache

Once the add-on is enabled, there will be a few environment variables available that contain the memcache connection details:

:::term
$ heroku config
MEMCACHE_PASSWORD    => xxxxxxxxxxxx
MEMCACHE_SERVERS     => instance.hostname.net
MEMCACHE_USERNAME    => xxxxxxxxxxxx
...

Next, add pylibmc to your application's requirements.txt file:

pylibmc==1.2.2

Sample code

In your application, you can interface with the provided memcache server with only a few lines of code:

:::python
import os
import pylibmc

# Connect to memcache with config from environment variables.
mc = pylibmc.Client(
    servers=[os.environ.get('MEMCACHE_SERVERS')],
    username=os.environ.get('MEMCACHE_USERNAME'),
    password=os.environ.get('MEMCACHE_PASSWORD'),
    binary=True
)

Now, you have direct access to the memcache. See the pylibmc docs for more information.

Django support

Unfortunately, Django's built-in pylibmc caching support doesn't support authentication yet. Luckily, there's a SASL-compatible backend available.

To use it, add the following to your requirements.txt file:

pylibmc==1.2.2
django-pylibmc-sasl==0.2.4

Then, add it as the configured backend in your application settings:

CACHES = {
    'default': {
        'BACKEND': 'django_pylibmc.memcached.PyLibMCCache'
    }
}

This will automatically configure Django to connect to the Memcached instance. No further configuration is required.

discordbot: python discordbot.py

Add a custom sidebar
Clone this wiki locally