Last updated at Tue, 25 Apr 2023 23:18:04 GMT

In Metasploit 4.10, we converted Metasploit Framework (and prosvc in Metasploit Commercial Editions) to be a full-fledged Rails::Application.  You may be wondering why Metasploit Framework and prosvc, should be Rails applications when they aren't serving up web pages.  It all has to do with not reinventing the wheel and very useful parts of Rails, Rails::Railtie and Rails::Engine.

Rails 3.0 infrastructure

Since Rails 3.0, Rails has been broken into multiple gems that didn't require each other and could be used or not used.

  • actionmailer
  • actionpack
  • activerecord
  • activeresource
  • activesupport

Rails::Railtie

To tie all these gems together, and allow other gems too be used in their place, a gem, railties, sits at the bottom of rails and supplies the infrastructure to connect different Rails::Railties together.  For our purposes we care about two features of Rails::Railtie

  1. Allow rake tasks defined in other gems to be called.
  2. Allow initializers to be defined in a set order.

Rails::Engine

On top of Rails::Railtie is built Rails::Engine, which is where a lot of the conventions that are thought of as Rails Conventions reside:

  1. standardized paths like lib and app/models.
  2. Automatic code loading (when we follow standard naming conventions that make directories correspond to namespace and Classes be files).

Rails::Application

Finally, on top of Rails::Engine sits a single Rails::Application, which is in charge of running all the initializers from the Rails::Railties and Rails::Engines. Without a Rails::Application the initializers won't automatically run and functionality from Rails ecosystem gems is lost.

Working around not having a Rails::Application before 4.10

In Metasploit Framework 4.9 we had to work-around not having access to (Railtie 1) by manually loading the yard rake task from metasploit_data_models.  We used the yard.rake defined in metasploit_data_models so we didn't have to duplicate code.  Eliminating duplicate code is a long term goal of my work on Metasploit Framework as a smaller code base is easier to maintain (because it's faster to test and easier to developers to understand) and we don't run the risk of fixing bugs in one copy of the code, but not the others.

Additionally, we had to port (i.e copy and slightly change which mean we won't be able to just grep for duplicates) portions of activerecord's databases.rake to support all the rake db* commands used to test against the database with rspec for Metasploit Framework 4.7.

We couldn't take advantage of (Railtie 2) because initializers are only run by Rails::Application#initialize! and Metasploit Framework didn't have a Metasploit::Framework::Application, which meant we couldn't take advantage of Rails community helpers like factory_girl_rails to automatically load factories in tests or rspec-rails to manage transactions for tests.  Not having transactions has led to lot of hard to track down bugs with database records from one test interfering with another test.

In Metasploit Commercial Editions 4.9, we were already using a Rails::Engine in MetasploitDataModels::Engine to automatically load the Mdm models in the UI, but since Metasploit Framework 4.9 and prosvc didn't support Rails::Engines, we had to duplicate the code logic .  Duplicate code means both code paths need to be tested and since this code path was only exercised in Metasploit Framework and prosvc, any changes to loaded code needed to be tested all the way up the stack through Metasploit Framework and prosvc. All of these complexities added up to slower development on metasploit_data_models.

Impetus for adopting Rails::Application

As you can see, there was a lot of duplicate code cruft building up to support the non-Rails::Application environments in Metasploit Framework and prosvc, which came to a breaking point when we decided to make metasploit-credential.

metasploit-credential sits on top of metasploit_data_models and defines credentials models in the Metasploit::Credential namespace.  These models aren't part of metasploit_data_modelsfor the following reason:

  1. Better understandability because developers don't have to know which parts of metasploit_data_models to ignore when dealing with credentials.
  2. Faster tests because metasploit-credential can be tested independently of metasploit_data_models.
  3. Cleaner API and versioning because metasploit-credential will only change if there's a change to credential's API and not any change in database API as would be the case with metasploit_data_models containing the credentials code.

Due to ActiveRecord associations being declared on each side of the relationship, metasploit-credential would need to be able to monkey patch inverse associations onto metasploit_data_models like Mdm::Service to associate it back to Metasploit::Credential::Login.  In Ruby, monkey patching can be accomplished a number of ways

  1. Reopen the class
  2. class_eval
  3. extend a Module
  4. include a Module

(1) and (2) have a downside that it's very hard to keep track of the monkey patches since Ruby doesn't keep track where a Class is defined (or redefined).  (The closest it can do is give the source location of a method.) So, (1) and (2) aren't good solutions as they drive anyone that works on the code later crazy.  (3) and (4) are good, but the problem is how to ensure the extend and include calls happen dynamically when the class to be monkey patched is loaded.

It turned out we already had a specific solution in Metasploit Commercial Editions for loading modules from pro/ui/lib/mdm to add Metasploit Commercial Editions monkey patches to Mdm models, but the problem was that it depended on running Rails initializer, which wouldn't work in metasploit-framework and only worked in prosvc because it manually loaded the initializer and ran it as a script at the appropriate point in the prosvc startup.  So, we were left with either implementing both an elegant Rails solution using its initializers and a port to work for Metasploit Framework and prosvc to support metasploit-credential extensions to metasploit_data_models or making Metasploit Framework and prosvc be Rails::Applications so they could run initializers like Metasploit Commercial Editions' UI.  I chose making Rails::Applications.

Converting to Rails::Applications

Converting to Rails::Applications allowed to me to do one of the best things you can do in a commit: delete more code than you add.

We were able to completely remove lib/tasks/database.rake and lib/tasks/rails.rake, which means we no longer have to watch out for changes to those in upstream activerecord.  We're using them from active_record/railtie now!  Other rake tasks are now defined by *-rails gems, like rake spec from rspec-rails, which also gave us transactional fixtures, which means that specs will automatically start and rollback database transactions after each example.  factory_girl_rails automatically loads the factories from metasploit-credential, and metasploit-model, metasploit_data_models.  yard.rake is automatically loaded from metapsloit_data_models just because it's under lib/tasks!

Having Metasploit::Framework::Application also gave all these other side benefits: rails console can be used to access the Mdm and Metasploit::Credential models without having to boot msfconsole and then running irb.  You can access the SQL prompt for the database directly with rails db.

What does all this have to do with Kali?

So, when a Rails::Application boots, it needs to opt-in to requiring the railties from the core gems, including active_record/railtie.  While 4.10 was in development, the Metasploit Applications team developed with active_record/railtie always being loaded as it ensured that the database was connected and ActiveRecord::Base.logger was setup, which made triage and debugging easier; however, we knew eventually that requiring active_record/railtie would have to be optional based on whether the db Gemfile group was installed and whether the -n (disable database) flag was passed to msfconsole.  Since I was handling users that didn't install database support at all or turned it off with -n, I thought we were covered for the 4.10 release and this is the state of the boot process that shipped.

It turned out that although Kali documents to start postgresql and setup a ~/.msf4/database.yml, many users hadn't found that documentation and so were starting msfconsole without a database.yml and/or without postgresql running, but without the -n flag.  Not having a database.yml led to the ENOENT error everyone was seeing since Metasploit::Framework::Application tried to read the non-existent database.yml when the database wasn't disabled.  James "egypt" Lee and I worked to fix this and egypt came up with checking if the file exists before trying to load active_record/railtie.  This fixed the issue of no database.yml, but not having a database.yml and postgresqlnot started.  Egypt and I discussed various ways around this, but it came down to adding a lot of error detection and making a direct connection using the low-level PGConn from the pg gem so we could test the database.yml could connect to the database OR never requiring active_record/railtie. We went with not requiring active_record/railtie.  Now, in 4.10.1, the database connection is just made by msfconsole how it had been in 4.9. We lost ActiveRecord::Base.logger setup in rails console and msfconsole, but those were new additions for Metasploit Framework 4.10 anyway, so it was an acceptable loss to get Kali's msfconsole to boot under excepted configuration scenarios and led to a more robust boot process.  If we want to restore the logger for the next release we can look into moving the error handling out of msfconsole into a place that the Rails boot can use it.

Goodies for the future

To leave on a brighter note, here some cool features of having access to Rails.application we hope to use in the future or that anyone in the community can use.

Since a Rails::Application can iterate the Rails::Engines plugged into it, we made migrations automatically load from any Rails::Engine we add to the metasploit ecosystem. Meterpreter extensions are also copied from Rails::Engines .  Finally module paths from Rails::Engines are automatically added to framework.modules

That's right, you make a gem with a Rails::Engine and put it in your Gemfile.local and metasploit-framework will treat it just like the normal metasploit-framework/modules.  We hope to use this ability in the future to break up the modules into target-specific gems so as the number of modules grow users can choose which loadouts they want for a given engagement.  This will make it easier to keep running metasploit-framework on small form factor devices with limited space.