Depends, some libraries like ActiveRecord for Ruby makes it hard to drop down to SQL (you lose data type conversion, etc).

Also see what the others said about the ORM influencing your database schema.