Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Benchmarking Ruby 2.6 to 3.2 (gettalong.org)
127 points by ksec on Dec 26, 2022 | hide | past | favorite | 68 comments


I’m surprised to see how little speed-up have happened over the years. Especially when you compare it to something like PHP which has massive gains.

Seems like most of the Ruby gains are related to memory usage.


I think these charts are more informative, especially for Rails apps: https://speed.yjit.org/

Ruby 3.2 and production-ready YJIT feel like a big deal as someone who has been in the Rails and Ruby ecosystem for 15 years.

Rails in particular has seemed to be very difficult to optimize for performance in Ruby, and YJIT seems to have done it.


> Seems like most of the Ruby gains are related to memory usage.

That would be a welcome addition, especially with Rails where you typically end up running your web app, background worker and action cable as separate processes. That's (3) running services each booting up your application to some degree.

Combine that with most apps using very little CPU and more and more memory, it's nice to be able to reduce memory when you can to optimize the machines you host things on. It always feels dirty having a server using 60% memory and 5% CPU.


Can those processes fork and save memory on duplicate pages? (I’m not a rails person)


Yes, COW was implemented a few versions ago and the garbage collector also supports it to avoid unnecessary copying.


I don’t have a ton of love for Python now that I’ve moved on to static typing but it’s hard for me to see myself choosing Ruby over Python for anything.

Python has gotten significantly faster, has static typing built in with an ecosystem of static typecheckers, and async is an ergonomic first class language feature - sure async can have rough edges but the 99.9% case is dead simple with no surprises.

Obviously packaging for Python is still a pain but I don’t see Ruby being better


Python improvements are on par with ruby, and ruby has a JIT, which means that upside fir real world improvement is much higher.

Python has no static typing. It has type annotations, which a few 3rd party static analysers such as mypy use. Ruby has had the same since 3.0 with rbs. The ecosystem may be less mature, but it exists, and Python's isn't necessarily perfect.

Python async is terrible. You have to colorize functions, opt out from a significant part of the ecosystem which either does not support async, or is very unstable, and completely relearn how to do debugging / performance measurement. Ruby has smth better since ruby 3.0, via the fiber scheduler, which supports running network related code asynchronously with *no changes or special function annotations*.

Ruby packaging has been figured out since 2007, and has been influencial for other ecosystems, being the first mainstream language adopting lockfiles.

If you prefer using Python, you are entitled to your opinion and choice. But those arguments are all FUD.


> Python has gotten significantly faster

All the synthetic benchmarks I've seen claim that Ruby is at least on par with Python's speed if not faster. Like this https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

But then speed in a dynamic language is often very context dependent. If I want to do Rails I won't choose Python. If I want to do machine learning, Python is the obvious choice.

> Obviously packaging for Python is still a pain but I don’t see Ruby being better

Ruby packaging just works whereas Python's is a pain. Can't see how you don't see it being better.


Those benchmarks are with Python 3.10 - the big recent speedups are in 3.11, with more to come

> The Python 3.11 release announcement cites 10~60% improvements over Python 3.10 and a 1.22x speed-up for its standard benchmark suite.



Yeah, the Debian benchmarks are independent.

The recent Python benchmarks all seem to use the same benchmark suite (which is incredibly complex to the point that it's unclear what is actually being measured) or hype the Microsoft involvement.

Even if you believe them, it is just 10-60% speedup for a very slow language. It won't show up in web applications.


> it’s hard for me to see myself choosing Python over Ruby for anything.

Did you mean this the other way around?


yep fixed


I've only ever known Javascript/ Typescript. I've been thinking of picking up ruby to learn "practical" functional programming and maybe see what the infatuation with rails is, but I'm starting to think it might be more "valuable" to learn python.


I wouldn’t recommend python or ruby if you’re trying to learn more about functional programming… you may as well stick with JS in that case (imo, at least). I’d choose a language that has at least had native immutability. Something like Elixir/Erlang, OCaml, Clojure, or F#.


I’m seeing a pretty significant amount of Elixir job listings these days even in areas where you used to see all Ruby or Python


I've actually been having a tough time finding another job in Elixir. Sooooo many companies who use it are in crypto and I just have no interest.


Purely my opinion after learning and professionally doing both Ruby and Python. If elegance and consistency in APIs matter, Ruby is by far the preference. Generally speaking (again from my own experience which should be taken with a grain of salt) the Ruby community tends to value readability/maintainability of code a lot higher, so working in the codebases is typically a lot more pleasant.

IIWM I'd go with Ruby hands down.


Python and Ruby are both only really as functional as you choose to use them. You’ll see more imperative code in the wild for both, though the big data stuff in Python leans more functional.

Rails and Django are both still super popular from scanning job listings, so e-commerce employment wise they’re both great.

FastAPI is also growing a lot.

Big Data leans heavily Python


Python would be the most popular option, but, for an OO language, Ruby feels clean and intuitive to me in a way that other languages, especially Python, don't. It has a performance cost, though. The two languages' performance may be roughly equivalent, but Python has more native-code libraries for performance-critical tasks. You don't see many ML projects in Ruby. :) There's crystal-lang, but that has a much smaller community behind it.

Julia, though, if you're looking for a language that isn't OO in addition to functional.


If you already know TypeScript, just use a functional approach to your programs, and/or use libraries like fp-ts and io-ts. Ruby is definitely not more FP based than TS.



Every time I try to get a project going with truffleruby, I get all kinds of dependency problems. I haven't had the patience to resolve all of the library problems yet, so I give up.


Pure speculation—Ruby has a deeper OOP story (everything is an object), and that comes with a builtin performance impact. I also feel like performance was never a top priority with the language (ergonomics instead), so certain aspects of the core design may not lend themselves to optimization as easily.


PHP had the benefit of Facebook pouring money into it. Same with JS and Google. No company has done something comparable for Ruby.


YJIT, Ruby's new JIT compiler, is from Shopify.


Let’s not forget that GitHub is a giant Rails app that Microsoft runs.

Microsoft is also one of the best language/compiler company in the world.


Microsoft didn't start pouring bucks into improving ruby yet. Their focus has recently been on JS, C# and python.


Really wish Microsoft would contribute/donate to ruby/rails platform even more than Github does already. They can afford to.


That’s not exactly accurate. FB went off and did their own thing.

PHP gains were independent of FB.


This is surprising to hear, considering how big Shopify has gotten and how much resources they've put into Ruby.



I believe Stripe use Ruby. Perhaps they'll fund its improvement...


Stripe has done a lot of work on a ruby type system (https://sorbet.org) and has been working on a ruby compiler as well (https://sorbet.org/blog/2021/07/30/open-sourcing-sorbet-comp...)


shopify is putting up a lot of money and resources into ruby.


If I'm reading those results right, it seems that Ruby was getting slower between 2.6.10 and 3.1.3 with yjit, and can still outperform Ruby 3+ versions without yjit. Does anyone know why this is?


Yes, there was a bit of a drop off in performance but only when YJIT is not used. With YJIT there are performance gain. Also note that YJIT in 3.1 was still experimental and more proof-of-concept. Only with the 3.2 release was it now deemed production ready.

And YJIT's creator, Shopify, is already using it in production with Ruby 3.2 for all their store fronts!


Small nitpick: Maxime Chevalier-Boisvert is the creator of YJIT and the BBV optimization technique that underpins it. Shopify is the company which employs her and funds its further improvement.


Yes, I know but thought it would be simpler to just say Shopify. Sorry for the inaccuracy.


I'd like to know this from a more informed person as well. My guess is that yjit is only now production ready, they've been building it for a while and enabled behind an experimental flag


YJIT was disabled by default before 3.2


Here is another benchmark from running Discourse:

https://twitter.com/rafael_falco/status/1607395317661458432


Hey, that's me!

Numbers were generated using https://github.com/discourse/discourse/blob/main/script/benc... if anyone wants to try it out too.

We already have 3.2 working internally and plan to roll it out to everyone soon.


It's interesting that Ruby now pulls Rust as a dependency, since YJIT is written in Rust [https://github.com/ruby/ruby/tree/master/yjit].


It seems well designed, as it's an optional dependency as of now.

> By default, YJIT does not get compiled and cargo/rustc is not required. If YJIT is built in dev mode, then cargo is used to fetch development dependencies, but when building in release, cargo is not required, only rustc. At the moment YJIT requires Rust 1.60.0 or newer.

https://github.com/ruby/ruby/pull/5826


I am interested in programming language theory and development but I am a beginner in both.

Do JIT compilers suffer from boxing pointer chasing primitives since everything is an object and not a memory location value type?

Can hot loops over arrays be efficient if every item in the array is a pointer?


I am by no means an expert on the topic, but I don’t think that “array of pointers”-chasing is the primary performance bottle neck of most managed language. It happens all too often with C programs as well (or even worse, linked lists) and if the pointed to objects are close to each other in memory/are in cache (as they might be in case of a generational GC) than it is not a big performance hit. Also, most apps don’t spend most of their time in a hot loop iterating over arrays, ordinary programs seldom have huge arrays.

Dynamic properties, non-uniform object shapes may be more responsible for this generic slowness of very dynamic languages, and from what I gathered assuming a shape for these objects constitutes a kind of advanced optimization (done by JS engines, truffleruby, not sure about these engines), even if its done depending on language idioms it may not give that big of a boost. Ruby is a very dynamic language with e.g. hitting not implememted methods and dynamically handling, etc.


Thank you for your explanation.

My understanding is that determining jump target at compile time is a big source of performance gain at runtime. If you know the function to jump to at compile time and not wait to runtime to decide where to jump.

I've looked into arrays of structs and structs or arrays for performance. But I am curious how to create a managed language that is fast. I think we look Java and C# for guidance.


> If you know the function to jump to at compile time and not wait to runtime to decide where to jump

Well, I don’t know about more dynamic languages, but single dispatch can be very well optimized away, Java for example can determine how many possible implementation of a given method is loaded, and simply JIT compile or even inline the function if there is only a single implementation. Later, it can deoptimize that call, e.g. when a new class is loaded at runtime that also implements that method. This is one area where JIT is definitely in a better position than AOT compilers.


Would be nice to see performance stats for running unicorn web app... I am pretty sure Shopify published some stats during the 3.1 and possibly 3.2 preview phase...


I thought Ruby 3 was supposed to be 3x faster than version 2?


3x increase when 3.0 is compared to 2.0[1]. I think it was largely achieved.

[1] https://blog.heroku.com/ruby-3-by-3



That would have been almost impossible.

Ruby was never designed as a “fast” language, but 3x is a huge number for anything this mature.


Looking at the benchmark game, it seems 3x current performance would put it in the company of racket, which is still noticably slower than node js. While right now it is much closer to python.

All benchmarks are misleading but it shows 10x slower than java, lisp, and pascal. Is there a reason that isn't actually representative of idiomatic use?


The benchmarks in the benchmark game are typically CPU bound, which heavily favors compiled languages for obvious reasons.

With regards to what "idiomatic use" is for Ruby, we could argue that most ruby apps are web backends and most of those spend the majority of their waiting for DB and/or API calls and don't spend all that much doing work in actual Ruby.


The same could be said for python with calls into numpy and pytorch consuming much of the time, but no one calls it designed for speed.


It is for many workloads. Just not Rails.


Oh look Ruby is still slow.



How many concurrent requests is 3.2. able to handle on an average consumer PC 5 years old, 4 cores? Just plain hello world.

Go(lang) about 5-6k.


I did 65k concurrent requests back in 2018 on a single core in Ruby 2.4/2.5 or so (see https://wjwh.eu/posts/2018-10-29-double-hijack.html) as long as each request doesn't do much. Having 6k open connections is not that much. Did you actually mean request/seq?

EDIT: I was curious on how many req/s modern Ruby would actually do for simple hello world, so I coded up the smallest Hello world example from https://github.com/rack/rack and ran it with the (default) Puma webserver, limited to 4 threads (and thus cores) with `-t 4`. According to `ab`, it averages around 8500 requests per second over a sample size of 100k requests. I didn't find out how to run rack apps with YJIT yet, but it seems reasonable to expect that to speed it up a bit further.


Use the ENV flag `export RUBY_YJIT_ENABLE=1` and run it again.


10445.25 req/s average over 100k requests, so about 2k req/s extra.


Concurrent requests. 5-6k without returning errors with Go.

wrk -c 6000

I doubt you had 65k concurrent.

Rps around 200k


grab ruby, run a test and show us.


Idk why the fuck you Ruby people are acting so butthurt


How many of us need 5-6k concurrent requests?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: