Thanks, I remember seeing that one a few months ago.
I hope this doesn't come off too harshly but that demo feels like it's meant to showcase high requests per second, not be a representation of how a real application would perform. It's broadcasting a counter over a websocket connection. It feels very similar to the old 2 million websocket connection demo. In other words I can't use your demo to make a meaningful conclusion on how LV would perform in my app. Also according to dev tools your demo page takes 300ms for me to load it from NY but I get 10-15ms response times on many other sites in the north east region of the US. A 300ms page response is usually what I would get hitting a server in Australia. The initial page load feels very delayed to load, although the counter increment was lightning fast.
I like the idea of using Livebook as a test case because it's an actual application, it's something relatable to what you might build instead of something that feels like it was created in a vacuum to promote high numbers. Sort of like how you could benchmark nginx with a hello world response. It's cool to see the high numbers but it doesn't help folks understand how things would work in a real app.
Having a breakdown of the average and p95/p99 response times at 100, 500, 1000, 2000, 10000, 25000 and 100000 connections all sending messages at varying rates along with seeing CPU and memory usage on a certain tier of VPS would be amazingly useful. You could take that data and factor in how your app might deal with similar conditions.
It's not too harsh at all! The demo is really meant to show liveview interaction latency, we didn't really ever think it would be useful for load.
That 300ms load time is actually because it queries Australia when you first load the page. Each region keeps its own counter and it grabs the current value from all of them at load time. It's probably worth optimizing that. :)
I hope this doesn't come off too harshly but that demo feels like it's meant to showcase high requests per second, not be a representation of how a real application would perform. It's broadcasting a counter over a websocket connection. It feels very similar to the old 2 million websocket connection demo. In other words I can't use your demo to make a meaningful conclusion on how LV would perform in my app. Also according to dev tools your demo page takes 300ms for me to load it from NY but I get 10-15ms response times on many other sites in the north east region of the US. A 300ms page response is usually what I would get hitting a server in Australia. The initial page load feels very delayed to load, although the counter increment was lightning fast.
I like the idea of using Livebook as a test case because it's an actual application, it's something relatable to what you might build instead of something that feels like it was created in a vacuum to promote high numbers. Sort of like how you could benchmark nginx with a hello world response. It's cool to see the high numbers but it doesn't help folks understand how things would work in a real app.
Having a breakdown of the average and p95/p99 response times at 100, 500, 1000, 2000, 10000, 25000 and 100000 connections all sending messages at varying rates along with seeing CPU and memory usage on a certain tier of VPS would be amazingly useful. You could take that data and factor in how your app might deal with similar conditions.