1
 
 
Account
In your account you can view the status of your application, save incomplete applications and view current news and events
LoginRegister
March 19, 2015

Varnish Developer Day

Operations
by
0 Comments
 

What is the article about?

On March 6th, we hosted the Varnish Developer Day VDD at our Loft 6. We are proud to give the community a place to meet and to support the open source project Varnish. 

Why do we do this?


Varnish is one of the main software products we have integrated into otto.de to make our shop fast and to achieve our business goals. Keeping in mind that we have a 'shared nothing software architecture we need at least one central system where everything comes together. How should each of our independent applications receive HTTP requests? A reverse proxy fulfills this role naturally.

We have to decide which vertical system is responsible for which requests. And even then, when we switched from Intershop to our own custom solution, we had very many deprecated links in bookmarks or other websites which should not end in 404's. So we use Varnish and its powerful Varnish Configuration Language VCL to route requests based on URL path components. It's very easy in VCL to implement routing logic based on URLs, for example

# route any otto.de/tesla requests to the tesla backend
if (req.url ~ "^//?tesla(/|$)") {
  set req.backend_hint = TESLA.backend();
}

Header Rewriting


Because VCL makes it possible to run regex comparisions against every header in the request and backend response, you have full control of your HTTP traffic. We have agreed with our vertical systems to convert cookies into distinct HTTP headers when the vertical systems have to react in a special way. To avoid complexity, our vertical systems do not have to read and parse cookie headers - it's enough to handle HTTP header fields. You can easily separate a cookie into a new header in VCL:

import re;
sub vcl_init {
  awesome_regex = new re.regex("\b(awesome_cookie\s*=\s*\w+)");
}
sub convert_awesome_cookie_to_header {
  if (awesome_regex.match(req.http.Cookie)) {
    set req.http.X-Awesome = re.backref(1, "NOMATCH");
  }
}
sub vcl_recv {
  call convert_awesome_cookie_to_header;
  ...
}

VMODs


In the example above, we use a VMOD re to handle backreferences with a syntax that is simpler than in standard VCL. VMODs are extension modules for VCL which can be developed by third parties outside of the Varnish distribution. With VMODs you can add capabilities to VCL which might be missing for a specific use case; you can find many VMOD solutions in the Varnish VMOD directory.

Central Logic


When the independent vertical systems have a common requirement, we are often asked to implement it in VCL. This becomes part of our contract between the operations team and the software development teams. We run otto.de with privacy and data protection for our customers, so we deliver everything over ssl/tls. We redirect all direct accesses via http to https within Varnish using VCL code like this:

sub http_to_https_redirect {
  if (req.http.X-Https != "true") {
    set req.http.X-Redirect = "https://" + req.http.host + req.url;
    return (synth(601, req.http.X-Redirect));
  }
}
sub vcl_recv {
  call http_to_https_redirect;
  ...
}
sub vcl_synth {
  if (resp.status == 601) {
    set resp.http.Location = resp.reason;
    set resp.status = 301;
    set resp.reason = "Moved Permanently";
    return (deliver);
  }
}

ESI


Otto.de uses ESI for page composition ESI is a means to compose HTML pages with fragments from other HTTP responses. At our website, there is a navigation header included in nearly all of our pages. When our store front is requested, the response looks something like this to Varnish (as highly simplified HTML source:

<html>
  <head><title>otto.de</title></head>
  <body>
    <esi:include src="/navigation/header" />
    <p>this is a storefront</p>
  </body>
</html>

When Varnish parses this response, it recognizes the esi:include tag, and generates a new HTTP request to get the content of the navigation header. That content replaces the tag before the store front is delivered. Of course there are other technologies such as AJAX to replace fragments in the DOM - but using ESI avoids the round trip necessary for AJAX, and the response is fully constructed before the browser has to parse it.

Caching


To reduce load on the app servers, we cache HTTP responses where possible. That's the next major benefit of Varnish - it's a blazingly (!!!) fast cache. With the capabilities of VCL, you can implement your own rules for caching. Because you have full control of the HTTP headers before you deliver content to a client (mostly browsers), you can decide how to cache.

Together with the use of ESI, it is possible to minimize the number of requests that are sent to the vertical systems. Varnish caches ESI fragments and can add them to HTML pages without sending a new request to a backend.

We follow HTTP to make caching decisions -- the vertical systems set the Cache-Control headers to decide if and how long Varnish should cache a response.

Tracking


We have released the Varnish tracking log reader which reads the Varnish log and parses it for data that we may store in our tracking system. The Varnish logging API makes it easy to extend the capabilities of Varnish with our own components.

Limitations & Conclusion


So, what is really missing in Varnish? Software developers often feel highly constricted by a domain-specific language like VCL. VCL is very easy, but many familiar concepts are missing. You can't create objects in standard VCL (although this can be done with VMODs), variables and loops are missing, and the flow of control is strictly determined by the Varnish state machine. But compared to the static configuration languages of other products such as apache, squid or nginx, VCL is much more powerful. It is better suited for a proxy's flow of control, and you can even write inline-C code (which we do not recommend), or extend VCL with VMODs (which we do). You can write VCL in a readable code style, and VCL is compiled. All of this means that we can do much more with a request in VCL than in a static configuration such as apache.conf.

0No comments yet.

Write a comment
Answer to: Reply directly to the topic

Written by

Similar Articles

We want to improve out content with your feedback.

How interesting is this blogpost?

We have received your feedback.

Close

Allow cookies?

OTTO and three partners need your consent (click on "OK") for individual data uses in order to store and/or retrieve information on your device (IP address, user ID, browser information).
Data is used for personalized ads and content, ad and content measurement, and to gain insights about target groups and product development. More information on consent can be found here at any time. You can refuse your consent at any time by clicking on the link "refuse cookies".

Data uses

OTTO works with partners who also process data retrieved from your end device (tracking data) for their own purposes (e.g. profiling) / for the purposes of third parties. Against this background, not only the collection of tracking data, but also its further processing by these providers requires consent. The tracking data will only be collected when you click on the "OK" button in the banner on otto.de. The partners are the following companies:
Google Ireland Limited, Meta Platforms Ireland Limited, LinkedIn Ireland Unlimited Company
For more information on the data processing by these partners, please see the privacy policy at otto.de/jobs. The information can also be accessed via a link in the banner.

more information