4 years ago by simonw
This article touches on "Request Coalescing" which is a super important concept - I've also seen this called "dog-pile prevention" in the past.
Varnish has this built in - good to see it's easy to configure with NGINX too.
One of my favourite caching proxy tricks is to run a cache with a very short timeout, but with dog-pile prevention baked in.
This can be amazing for protecting against sudden unexpected traffic spikes. Even a cache timeout of 5 seconds will provide robust protection against tens of thousands of hits per second, because request coalescing/dog-pile prevention will ensure that your CDN host only sends a request to the origin a maximum of once ever five seconds.
I've used this on high traffic sites and seen it robustly absorb any amount of unauthenticated (hence no variety on a per-cookie basis) traffic.
4 years ago by sleepy_keita
Back when I was just getting started, we were doing a lot of WordPress stuff. A client contacted us, "oh yeah, later today we're probably going to have 1000x the traffic because of a popular promotion". I had no idea what to do so I thought, I'll just set the varnish cache to 1 second, that way WordPress will only get a maximum of 60 requests per second. It worked pretty much flawlessly, and taught me a lot about the importance of request coalescing and how caches work.
4 years ago by sciurus
I'll echo what Simon said; we share some experiences here. There's a potential footgun, though, anyone getting started with this should know about-
Request coalescing can be incredibly beneficial for cacheable content, but for uncacheable content you need to turn it off! Otherwise you'll cause your cache server to serialize requests to your backend for it. Let's imagine a piece of uncacheable content takes one second for your backend to generate. What happens if your users request it at a rate of twice a second? Those requests are going to start piling up, breaking page loads for your users while your backend servers sit idle.
If you are using Varnish, the hit-for-miss concept addresses this. However, it's easy to implement wrong when you start writing your own VCL. Be sure to read https://info.varnish-software.com/blog/hit-for-miss-and-why-... and related posts. My general answer to getting your VCL correct is writing tests, but this is a tricky behavior to validate.
I'm unsure how nginx's caching handles this, which would make me nervous using the proxy_cache_lock directive for locations with a mix of cacheable and uncacheable content.
4 years ago by endymi0n
And to add the last big one from the trifecta:
Know how to deal with cacheable data. Know how to deal with uncacheable data. But by all means, know how to keep them apart.
Accidentally caching uncacheable data has lead so some of the most ugly and avoidable data leaks and compromises in recent times.
If you go down the "route everything through a CDN route (that can be as easy as ticking a box in the Google Cloud Platform backend), make extra sure to flag authenticated data as cache-control: private / no-cache.
4 years ago by arghwhat
no-cache does not mean content must not be cached - in fact, it specifies the opposite!
no-cache means that the response may be stored in any cache, but cached content MUST be revalidated before use.
public means that the response may be cached in any cache even if the response was not normally cacheable, while private restricts this to only the user agent's cache.
no-store specifies that this response must not be stored in any cache. Note that this does not invalidate previous cached responses from being used.
max-age=0 can added to no-store to also invalidate old cached responses should one have accidentally sent a cacheable response for this resource. No other directives have any effect when using no-store.
4 years ago by Akronymus
Speaking of non-cacheable data:
https://arstechnica.com/gaming/2015/12/valve-explains-ddos-i...
Caching is HARD.
4 years ago by mnutt
In varnish, if you have some requirements flexibility you can enable grace mode in order to serve stale responses but update from the origin, and avoid long requests every [5] seconds.
Not quite the same layer, but in node.js Iâm a fan of the memoize(fn)->promise pattern where you wrap a promise-returning function to return the _same_ promise for any callers passing the same arguments. Itâs a fairly simple caching mechanism that coalesces requests and the promise resolves/rejects for all callers at once.
4 years ago by skunkworker
I've implemented this manually in some golang web applications I've written. It really helps when you have an expensive cache-miss operation, as it can stack the specific requests so that once the original request is served, all of the stacked requests are served with the cached copy.
4 years ago by jabo
Love the level of detail that Fly's articles usually go into.
We have a distributed CDN-like feature in the hosted version of our open source search engine [1] - we call it our "Search Delivery Network". It works on the same principles, with the added nuance of also needing to replicate data over high-latency networks between data centers as far apart as Sao Paulo and Mumbai for eg. Brings with it another fun set of challenges to deal with! Hoping to write about it when bandwidth allows.
4 years ago by mrkurt
I'd love to read about it.
4 years ago by amirhirsch
This is cool and informative and Kurt's writing is great:
The briny deeps are filled with undersea cables, crying out constantly to nearby ships: "drive through me"! Land isn't much better, as the old networkers shanty goes: "backhoe, backhoe, digging deep â make the backbone go to sleep".
4 years ago by tptacek
We can't take credit for the backhoe thing; that really is an old networking shanty.
4 years ago by babelfish
fly.io has a fantastic engineering blog. Has anyone used them as a customer (enterprise or otherwise) and have any thoughts?
4 years ago by joshuakelly
Yes, I'm using it. I deploy a TypeScript project that runs in a pretty straightforward node Dockerfile. The build just works - and it's smart too. If I don't have a Docker daemon locally, it creates a remote one and does some WireGuard magic. We don't have customers on this yet, but I'm actively sending demos and rely on it.
Hopefully I'll get to keep working on projects that can make use of it because it feels like a polished 2021 version of Heroku era dev experience to me. Also, full disclosure, Kurt tried to get me to use it in YC W20 - but I didn't listen really until over a year later.
4 years ago by jbarham
One of my side projects is a DNS hosting service, SlickDNS (https://www.slickdns.com/).
I moved my authoritative DNS name servers over to Fly a few months ago. After some initial teething issues with Fly's UDP support (which were quickly resolved) it's been smooth sailing.
The Fly UX via the flyctl command-line app is excellent, very Heroku-like. Only downside is it makes me mad when I have to fight the horrendous AWS tooling in my day job.
4 years ago by mike_d
I run my own worldwide anycast network and still end up deploying stuff to Fly because it is so much easier.
The folks who actually run the network for them are super clueful and basically the best in the industry.
4 years ago by cgarvis
just started to use them for an elixir/phoenix project. multi region with distributed nodes just works. feels almost magically after all the aws work I've done the past few years.
4 years ago by tiffanyh
Whatâs magically?
I was under the impression that fly.io today (though they are working on it) doesnât do anything unique to make hosting elixir/Phoenix app easier.
See this comment by the fly.io team.
4 years ago by tptacek
I still wouldn't say we do any magic Elixir stuff; rather, our platform just happens to have a combination of features (particularly edge delivery for stuff like LiveView and zero-config private networking for clustering) that make Elixir apps sing.
But we've got full-time people working on Elixir now, too; we'll see where that goes. We've still got Elixir limerence here. :)
4 years ago by mcintyre1994
They're not doing anything special to make Elixir specifically better yet, but their private networking is already amazing for it - you can cluster across arbitrary regions completely trivially. It's a really good fit for Elixir clustering as-is even without anything specially built for it. I have no idea how you'd do multi-region clustering in AWS but I'm certain it'd be a lot harder.
4 years ago by vmception
>The term "CDN" ("content delivery network") conjures Google-scale companies managing huge racks of hardware, wrangling hundreds of gigabits per second. But CDNs are just web applications. That's not how we tend to think of them, but that's all they are. You can build a functional CDN on an 8-year-old laptop while you're sitting at a coffee shop.
huh yeah never thought about it
I blame how CDNs are advertised for the visual disconnect
4 years ago by lupire
It's misleading.
CDN software might be simple in the basic happy case, but you still need a Network of nodes to Deliver the Content.
4 years ago by mrkurt
Well it's a self serving article! It's easy to turn up a network of nodes on Fly.io. It's a little harder, but not impossible, to do the same elsewhere.
4 years ago by daniel_iversen
Years ago I was involved with some high performance delivery of a bunch of newspapers, and we used Squid[1] quite well. One nice thing you could do as well (but it's probably a bit hacky and old school these days) was to "open up" only parts of the web page to be dynamic while the rest was cached (or have different cache rules for different page components)[2]. With some legacy apps (like some CMS') this can hugely improve performance while not sacrificing the dynamic and "fresh looking" parts of the website.
[1] http://www.squid-cache.org/ [2] https://en.wikipedia.org/wiki/Edge_Side_Includes
4 years ago by chrisweekly
This is so great. See also https://fly.io/blog/ssh-and-user-mode-ip-wireguard/
4 years ago by 3np
As someone whoâs mostly clueless about BGP but have a fair grasp of all the other layers mentioned, Iâd love to see posts like this going more in depth on it for folks like myself.
Daily digest email
Get a daily email with the the top stories from Hacker News. No spam, unsubscribe at any time.