Fixing a Segfault in Ruby Apps
Last updated 31 May 2016
Table of Contents
A segmentation fault can happen when software tries to issue invalid memory commands to the computer it is running on. If you are getting a segfault in your application you will see a very verbose error that may look something like this:
2016-05-10T03:04:11.572681+00:00 app[web.1]: -- Control frame information ----------------------------------------------- 2016-05-10T03:04:11.582158+00:00 app[web.1]: 975 /app/vendor/bundle/ruby/2.2.0/gems/sass-3.4.22/lib/sass/supports.rb 2016-05-10T03:04:11.572682+00:00 app[web.1]: c:0001 p:---- s:0002 e:000001 TOP [FINISH] 2016-05-10T03:04:11.582159+00:00 app[web.1]: 976 /app/vendor/bundle/ruby/2.2.0/gems/sass-3.4.22/lib/sass/engine.rb 2016-05-10T03:04:11.582160+00:00 app[web.1]: 977 /app/vendor/bundle/ruby/2.2.0/gems/sass-3.4.22/lib/sass/railtie.rb 2016-05-10T03:04:11.572683+00:00 app[web.1]: 2016-05-10T03:04:11.582162+00:00 app[web.1]: 978 /app/vendor/bundle/ruby/2.2.0/gems/sass-3.4.22/lib/sass/features.rb 2016-05-10T03:04:11.582162+00:00 app[web.1]: 979 /app/vendor/bundle/ruby/2.2.0/gems/sass-3.4.22/lib/sass.rb 2016-05-10T03:04:11.572684+00:00 app[web.1]: 2016-05-10T03:04:11.582163+00:00 app[web.1]: 980 /app/vendor/bundle/ruby/2.2.0/gems/sprockets-3.6.0/lib/sprockets/sass_functions.rb 2016-05-10T03:04:11.572684+00:00 app[web.1]: -- C level backtrace information ------------------------------------------- 2016-05-10T03:04:11.580053+00:00 app[web.1]: /app/vendor/ruby-2.2.2/bin/ruby(rb_vm_bugreport+0x51f) [0x7fc9c783dfbf] vm_dump.c:693 2016-05-10T03:04:11.580095+00:00 app[web.1]: /app/vendor/ruby-2.2.2/bin/ruby(rb_bug+0xca) [0x7fc9c78af95a] error.c:409 2016-05-10T03:04:11.580156+00:00 app[web.1]: /app/vendor/ruby-2.2.2/bin/ruby(newobj_of+0x1be) [0x7fc9c770932e] gc.c:1635 2016-05-10T03:04:11.580241+00:00 app[web.1]: /app/vendor/ruby-2.2.2/bin/ruby(rb_str_dup+0x1b) [0x7fc9c77c3fcb] string.c:552 2016-05-10T03:04:11.582165+00:00 app[web.1]: 981 /app/vendor/bundle/ruby/2.2.0/gems/sass-rails-5.0.4/lib/sass/rails/helpers.rb 2016-05-10T03:04:11.580402+00:00 app[web.1]: /app/vendor/ruby-2.2.2/bin/ruby(rb_class_path+0x83) [0x7fc9c77fea03] variable.c:259 2016-05-10T03:04:11.580460+00:00 app[web.1]: /app/vendor/ruby-2.2.2/bin/ruby(thread_start_func_2+0x1e1) [0x7fc9c784c5c1] thread.c:2717 2016-05-10T03:04:11.580543+00:00 app[web.1]: /app/vendor/ruby-2.2.2/bin/ruby(thread_start_func_1+0xbb) [0x7fc9c784d0eb] thread_pthread.c:846 2016-05-10T03:04:11.580604+00:00 app[web.1]: /lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xc2) [0x7fc9c7285182] 2016-05-10T03:04:11.580654+00:00 app[web.1]: /lib/x86_64-linux-gnu/libc.so.6(__clone+0x6d) [0x7fc9c686f47d] 2016-05-10T03:04:11.580656+00:00 app[web.1]:
A segfault can occur in the MRI Ruby VM (YARV), which is written in C. However, this is very unlikely: the VM for Ruby 2+ has been very stable and raising a segfault by writing pure Ruby code is rare. Usually a segfault is caused by a C-extension.
A C-extension is a Rubygem that has C code that will be compiled and interfaced by Ruby. There are many reasons to use or write a C-extension. You can write one to gain speed for a particularly performance sensitive piece of code. You can also use a C-extension to utilize libraries on Heroku directly by importing their headers and directly using their APIs.
If you are getting a segfault in your application you will want to find out what libraries you have that contain C-extensions. You can find that information by running this command in IRB:
$ irb
irb(main):001:0> puts `bundle show --paths`.split("\n").select{ |dep| File.directory?("#{dep}/ext") }.map{ |dep| dep.split('/').last }.join("\n")
# =>
bcrypt-3.1.11
bluecloth-2.2.0
debug_inspector-0.0.2
ffi-1.9.10
nio4r-1.2.1
nokogiri-1.6.7.2
pg-0.18.3
puma-3.0.0.rc1
rb-fsevent-0.9.7
rusage-0.2.0
scout_apm-2.0.0.pre
thread_safe-0.3.5
websocket-driver-0.6.3
unf_ext-0.0.7.2
This list was generated from the open source application CodeTriage.
Fixing a segfault
Once you have a list of C-extensions you can investigate upgrading them, as well as using the latest released version of Ruby. You can see a list on the Ruby support page. We always recommend running the latest available version.
If upgrading C-extensions and Ruby version does not work, you can use the Ruby backtrace in the segfault for more information. Here is an example that I’ve abbreviated:
2016-05-10T03:04:11.580659+00:00 app[web.1]: -- Other runtime information ----------------------------------------------- 2016-05-10T03:04:11.580659+00:00 app[web.1]: 2016-05-10T03:04:11.580661+00:00 app[web.1]: * Loaded script: puma: cluster worker 1: 27 [app] 2016-05-10T03:04:11.580662+00:00 app[web.1]: 2016-05-10T03:04:11.580663+00:00 app[web.1]: * Loaded features: 2016-05-10T03:04:11.580664+00:00 app[web.1]: 2016-05-10T03:04:11.580667+00:00 app[web.1]: 0 enumerator.so 2016-05-10T03:04:11.580668+00:00 app[web.1]: 1 rational.so 2016-05-10T03:04:11.580668+00:00 app[web.1]: 2 complex.so 2016-05-10T03:04:11.580671+00:00 app[web.1]: 3 /app/vendor/ruby-2.2.2/lib/ruby/2.2.0/x86_64-linux/enc/encdb.so 2016-05-10T03:04:11.580671+00:00 app[web.1]: 4 /app/vendor/ruby-2.2.2/lib/ruby/2.2.0/x86_64-linux/enc/trans/transdb.so 2016-05-10T03:04:11.580672+00:00 app[web.1]: 5 /app/vendor/ruby-2.2.2/lib/ruby/2.2.0/unicode_normalize.rb 2016-05-10T03:04:11.580673+00:00 app[web.1]: 6 /app/vendor/ruby-2.2.2/lib/ruby/2.2.0/x86_64-linux/rbconfig.rb 2016-05-10T03:04:11.580675+00:00 app[web.1]: 7 thread.rb 2016-05-10T03:04:11.580676+00:00 app[web.1]: 8 /app/vendor/ruby-2.2.2/lib/ruby/2.2.0/x86_64-linux/thread.so 2016-05-10T03:04:11.580676+00:00 app[web.1]: 9 /app/vendor/ruby-2.2.2/lib/ruby/2.2.0/rubygems/compatibility.rb 2016-05-10T03:04:11.580677+00:00 app[web.1]: 10 /app/vendor/ruby-2.2.2/lib/ruby/2.2.0/rubygems/defaults.rb 2016-05-10T03:04:11.582166+00:00 app[web.1]: 982 /app/vendor/bundle/ruby/2.2.0/gems/sprockets-3.6.0/lib/sprockets/sass_importer.rb 2016-05-10T03:04:11.582166+00:00 app[web.1]: 983 /app/vendor/bundle/ruby/2.2.0/gems/tilt-2.0.2/lib/tilt/mapping.rb 2016-05-10T03:04:11.582168+00:00 app[web.1]: 984 /app/vendor/bundle/ruby/2.2.0/gems/tilt-2.0.2/lib/tilt/template.rb 2016-05-10T03:04:11.582169+00:00 app[web.1]: 985 /app/vendor/bundle/ruby/2.2.0/gems/tilt-2.0.2/lib/tilt.rb 2016-05-10T03:04:11.582169+00:00 app[web.1]: 986 /app/vendor/bundle/ruby/2.2.0/gems/sass-rails-5.0.4/lib/sass/rails/importer.rb 2016-05-10T03:04:11.582172+00:00 app[web.1]: 987 /app/vendor/bundle/ruby/2.2.0/gems/sass-rails-5.0.4/lib/sass/rails/cache_store.rb 2016-05-10T03:04:11.582172+00:00 app[web.1]: 988 /app/vendor/bundle/ruby/2.2.0/gems/sass-rails-5.0.4/lib/sass/rails/template.rb 2016-05-10T03:04:11.582173+00:00 app[web.1]: 989 /app/vendor/bundle/ruby/2.2.0/gems/sass-rails-5.0.4/lib/sass/rails/logger.rb 2016-05-10T03:04:11.582175+00:00 app[web.1]: 990 /app/vendor/bundle/ruby/2.2.0/gems/sass-rails-5.0.4/lib/sass/rails/railtie.rb 2016-05-10T03:04:11.582175+00:00 app[web.1]: 991 /app/vendor/bundle/ruby/2.2.0/gems/sass-rails-5.0.4/lib/sass/rails.rb 2016-05-10T03:04:11.582176+00:00 app[web.1]: 992 /app/vendor/bundle/ruby/2.2.0/gems/sass-rails-5.0.4/lib/sass-rails.rb 2016-05-10T03:04:11.582178+00:00 app[web.1]: 993 /app/vendor/bundle/ruby/2.2.0/gems/execjs-2.6.0/lib/execjs/version.rb 2016-05-10T03:04:11.582178+00:00 app[web.1]: 994 /app/vendor/bundle/ruby/2.2.0/gems/execjs-2.6.0/lib/execjs/module.rb 2016-05-10T03:04:11.582179+00:00 app[web.1]: 995 /app/vendor/bundle/ruby/2.2.0/gems/execjs-2.6.0/lib/execjs/encoding.rb 2016-05-10T03:04:11.582182+00:00 app[web.1]: 996 /app/vendor/bundle/ruby/2.2.0/gems/execjs-2.6.0/lib/execjs/runtime.rb 2016-05-10T03:04:11.582182+00:00 app[web.1]: 997 /app/vendor/bundle/ruby/2.2.0/gems/execjs-2.6.0/lib/execjs/disabled_runtime.rb 2016-05-10T03:04:11.582183+00:00 app[web.1]: 998 /app/vendor/bundle/ruby/2.2.0/gems/execjs-2.6.0/lib/execjs/duktape_runtime.rb 2016-05-10T03:04:11.582185+00:00 app[web.1]: 999 /app/vendor/bundle/ruby/2.2.0/gems/execjs-2.6.0/lib/execjs/external_runtime.rb 2016-05-10T03:04:11.582186+00:00 app[web.1]: 1000 /app/vendor/bundle/ruby/2.2.0/gems/execjs-2.6.0/lib/execjs/ruby_racer_runtime.rb 2016-05-10T03:04:11.582187+00:00 app[web.1]: 1001 /app/vendor/bundle/ruby/2.2.0/gems/execjs-2.6.0/lib/execjs/ruby_rhino_runtime.rb 2016-05-10T03:04:11.582187+00:00 app[web.1]: 1002 /app/vendor/bundle/ruby/2.2.0/gems/execjs-2.6.0/lib/execjs/runtimes.rb 2016-05-10T03:04:11.582190+00:00 app[web.1]: 1003 /app/vendor/bundle/ruby/2.2.0/gems/execjs-2.6.0/lib/execjs.rb 2016-05-10T03:04:11.582190+00:00 app[web.1]: 1004 /app/vendor/bundle/ruby/2.2.0/gems/uglifier-3.0.0/lib/uglifier/version.rb 2016-05-10T03:04:11.582191+00:00 app[web.1]: 1005 /app/vendor/bundle/ruby/2.2.0/gems/uglifier-3.0.0/lib/uglifier.rb 2016-05-10T03:04:11.582194+00:00 app[web.1]: 1006 /app/vendor/bundle/ruby/2.2.0/gems/coffee-script-source-1.10.0/lib/coffee_script/source.rb 2016-05-10T03:04:11.582194+00:00 app[web.1]: 1007 /app/vendor/bundle/ruby/2.2.0/gems/coffee-script-2.4.1/lib/coffee_script.rb
Now that you know what C-extensions your application uses, any you have a backtrace, you can start working backwards until you find the cause. Note that the gem name might be different from the gem on the system. In this case we can see the unf_ext way later in the backtrace:
2016-05-10T03:04:11.590668+00:00 app[web.1]: 1346 /app/vendor/bundle/ruby/2.2.0/gems/unf-0.1.4/lib/unf/version.rb 2016-05-10T03:04:11.581498+00:00 app[web.1]: 517 /app/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.4/lib/active_support/core_ext/string/access.rb 2016-05-10T03:04:11.590669+00:00 app[web.1]: 1347 /app/vendor/bundle/ruby/2.2.0/gems/unf-0.1.4/lib/unf.rb
This is just an example: this case was not caused by the unf_ext gem. But as it shows up in the backtrace it’s worth looking into.
If you do determine that a gem is causing your segfault please open up an issue with the gem maintainer. Provide the full segfault error output, and whenever possible include a small example application that reproduces the segfault along with instructions on how to reproduce.