VGTech is a blog where the developers and devops of Norways most visited website share code and tricks of the trade… Read more



Are you brilliant? We're hiring. Read more

Safari on iOS 5 Randomly Switches Images

DevOps

Since the release of iOS 5 we’ve received several bug reports about images randomly displaying the wrong image(s) on our front page for smartphones. It seems to be completely random and could affect any of the images, anywhere and anyhow. During one week we would receive anywhere between two and ten reports about this error. In the same time period we would do about 4.5 million page impressions(2.9 million of them from iPhone) on the front alone.

Safari has replaced the LIVE-icon with an image of Mancini

Our front page is manually edited with a tool called Dr. Front, so we started off with eliminating possible sources like bad HTML markup, duplicate filenames and human error. No errors could be found in the markup and there were no colliding filenames. Any of these sources would also mean that this problem would occur in a more consistent manner across different browsers and OS.

The common denominator for all reports were iOS 5, and most of them with slow and lossy internet access.

HTTP Pipelining

We recently discovered that iOS 5 enabled HTTP Pipelining by default. In short terms HTTP Pipelining enables the browser to send several HTTP requests on the same connection before receiving the responses. It can even send new requests while still receiving an earlier response. Pipelining is still pretty new; Opera was the only browser that enabled it by default, Firefox have support that’s disabled and Chrome will support it in Chrome 17.

Firefox’ implementation seems to never send a new request while receiving a response while Opera and Safari do. Even with extensive testing with both Firefox and Opera we have never seen the images get replaced.

PS! We also found a small gotcha in Firefox, you MUST send the Server-header. Firefox refuses to use pipelining against a server it does not know.

Our infrastructure

When sending a request to http://touch.vg.no you will first hit our F5 Big-IP load balancer. The Big-IP terminates the connections and use layer 7 load balancing to determine where the request should be forwared.  In this case that is a Varnish cluster consisting of 6 servers. These talk to different Apache servers, depending on the hostname. Static elements are served from static.vg.no will go through the same load balancer, the same Varnish cluster but end with a Lighttpd server.

What happens?

So far we’ve only been able to reproduce the bug two times while sniffing the traffic with Wireshark. When Safari switches the image(s) it’s always images in the same TCP stream and in both recorded cases when it sends a request before the previous response is complete.

What Safari does is that all the image(s) that are requested in the middle of a response end up containing the image currently being transferred. It also uses the cache headers of the same image thats being transferred, so if the cache headers (Expires/Etag) allows it, Safari will keep this image (with the wrong content), but with the right filename. Therefore a “soft” refresh without emptying the cache will still display the wrong image.

Tcpdumps from Wireshark:

In this dump the problem occurs in stream 8 when requesting “/drfront/images/2011-12/12/86-a17ea747-830df041.jpeg“. This image is not in the cache and gets downloaded. During the transfer  Safari requests “/gfx/snippetMinmote590.png?20091130“. Even if this is a 304 Not Modified response the result is that the local version of “snippetMinmote590.png” contains and will be displayed with the image content of “86-a17ea747-830df041.jpeg”.

In this dump it is stream 9 and the exact same behaviour, just with different elements. “/drfront/images/2011-12/12/86-e951cc7b-4ad1fd2a.jpeg” results in new download and in the middle “/gfx/ikon_vgtv.png” is requested. This will also result in a 304 Not Modified but contain “86-e951cc7b-4ad1fd2a.jpeg”.

The solution?

We don’t know yet. We can’t simply turn off pipelining as the only requirement is the “Connection: Keep-alive” header in the response. If the browser discovers keep-alive and supports pipelining it is going to use pipelining. We can increase the amount of hosts we deliver our images from, in an attempt to reduce the usefulness of pipelines, but that is really not a good solution.

Developer at VG. @androa


25 comments

  • Little Steve

    Have you talked to Apple?


  • Jason

    We are experiencing the same issue. It always seems to be that a CSS background image gets swapped in for a content image. I wonder if having a separate domain for static content would help with this issue.
    I also would like to know if Apple is aware of this issue.


  • Noah

    I have been encountering this issue since I'm using Opera for a year and since upgraded my iOS to 5.0 recently. After disabling pipeline in Opera it solved the problem immediately. I hope that Apple would follow suit.


  • Chris

    I was having this problem as well. Adding "Header set Connection: Close" to my htaccess file seemed to have solved it.


  • André Roaldseth

    Chris: Yeah, that has been one of our suggestions for workaround. But that also breaks keep-alive for every browser and that will slow down everyone else.

    Our current workaround is to distribute all our images over 10 different hostnames thus reducing the chance that Safari is able to use it's pipelining. Bad, but it seems to be helping.


  • ay

    It seems like the problem goes away when we switch from Layer 7 Loadbalancing (terminating HTTP - Normal) to Layer 4 Loadbalancing (Forwarding IP) in the BigIP setip. Still trying to figure out why.


  • Audun Ytterdal

    Update.
    We got the bug on layer 4 as well. we also know that http://www.dagbladet.no also has the bug.
    We recently have seen the bug on firefox 9 with pipelining enabled as well...


  • Patrick McManus

    I'm working on a new more aggressive and more nuanced pipeline implementation for Firefox. Please contact me through a more efficient channel than this comment stream so we can dig into this together.

    Take a harder look at your second capture - the one with stream 9 called out. The request for "ikon" is sent with "If-Modified-Since: Mon, 12 Dec 2011 13:28:11 GMT" . So the client is comparing it to something in its cache with a Last-Modified of 12/12/2011-13:28:11. The matching response is a 304 with a a Last-Modified of Nov-18-2009.

    While that's technically right (2009 is not after 2011), that's exceptionally weird. LM/IMS are basically opaque tokens - to get a 304 with a LM that doesn't match the IMS is very odd.

    Furthermore that 12-12-2011/13:28 date is BOTH what ikon is using as its conditional validator)


  • Patrick McManus

    [continued]

    Furthermore that 12-12-2011/13:28 date is BOTH what ikon is using as its conditional validator) and what the jpg uses for the Last-Modified in its own 200 response.

    That strongly suggests that the screwup has already previously happened - it isn't in this trace at all. All you're seeing is the cache revalidation of a inconsistent cache.

    Where in the network are these traces taken from (client, server, etc.)


  • André Roaldseth

    Yeah, I've noticed the deviations in LM/IMS, but not been able to connect any dots there. Some of the times images get swapped around(200 OK , followed by 304 Not Modified), the 200 OK _usually_ get the Last-Modified header of the 304 Not Modified as well. But this is not always the case.

    I've joined #imagebug on irc.freenode.net if you want to have a bit more dynamic discussion around this :)


  • Safari on iOS 5 randomly switches images, part 2 – VG Tech

    [...] post is a follow-up to Safari on iOS 5 randomly switches images. In our case almost all of the images on the front page has unique height * width combinations. [...]


  • Håkon

    Didn't read all, just wanted to tell that I use Android and the standard web browser, and I still got a "Beklager, feil bilde" message overlay on one of the pics (but it looked like the right picture, singing obama).


    • ay

      That was unfortunately a bug on our side. It's since been fixed.

  • Christian Wig

    This problem is not limited to iOS, as it also happens with the VG smartphone site on my Google Nexus One phone.

    Both Safari and the Nexus One browser are based on the Webkit browser engine, so I would investigate differencies between versions and implementations of Webkit.


  • Christian Wig

    Before you start investigating my 'Webkit theory' you need to check whether there is a bug in the program logic aimed st detecting random image changes!

    For the article "Brøt ut i sang på talerstolen", the message "Beklaget, feil bilde" is reported even if no image change takes place. And it happens consistently, even after clearing the browser cache, for various browsers (default browser, Opera Mini, Opera Mobile, Firefox) on both Google Nexus One and Samsung Galaxy II phones.


  • André Roaldseth

    Christian: Yes, it seems to affect all kind of browsers, but as far as I know, HTTP handling is not a part of Webkit, but implemented by every browser. But Safari on iOS is prone to get this error way more than any other browser out there. We are working with that around 0.5% - 1% of all page views on Safari Mobile misplaces a image.

    With regards to the logic for finding misplaced images, you are also right. But the problem is not with the detection, but our CMS that sometimes create the image a few pixels away from the targeted container. That gives a few false positives (about 3-4 a day) and I'm just not sure on how to handle it yet.


  • TGOS

    Sorry, just had to comment on that one: "Pipelining is still pretty new"??? HTTP Pipelining is part of the HTTP 1.1 standard, that means it's an official, open standard since 1997. Firefox has at least support for it since 2001, that's when the option to enable it has been first checked into the source, even though it was never enabled by default up to today.


    • ay

      The standard is old. And some have old implementations. But it's new in the sense that it hasn't really been turned on and used in the wild by anyone until recently...

  • Mark Nottingham

    Have you tried it without the F5 hardware?

    Patrick - FYI, last I checked Safari will generate an IMS off of the Date header if LM isn't there (barf).


    • ay

      Yes. I recently triggered the bug pretty consistent without BigIP as well. Directly at a sigle varnish with a public ip. The only scenario where we can't seem to trigger it is when traffic goes through our SSL-VPN (openvpn)

  • tore

    i got the same problem using opera on a samsung galaxy


  • Android, Opera Mobile, HTC Desire Z

    Jeg får samme problem, men jeg har Android OS.


  • Safari on iOS 5 Randomly Switches Images, Part 3 – VG Tech

    [...] are still digging deeper into the imagebug problem we’ve mentioned in part 1 and part [...]


  • Casper Kandelsdorff

    Did IOS6 fix this problem?


  • Matt Kuzior

    Http Pipelining requires that the resonses are returned in the same order the client requested the resources on a given TCP connection. If your load balancer changes the response order because it pulls the resources from different backends, a pipelining client cannot tell that the swap happened and assigns the first reponse with the first request and the second response with the second request.


Leave your comment