The good thing of having a new API compatible with Readline is that now people can use RawLine in their Readline-powered scripts, with very minor modifications.
Let’s have a look at some examples (they are also shipped with Rawline v0.3.1):
h3. Rush
Rush is an excellent gem which provides a cross-platform shell environment, entirely written in Ruby.
Being a shell, it obviously uses Readline for tab completion, and that does the job on Linux. On Windows though, things aren’t that easy:
RawLine doesn’t have these problems (that’s the very reason why I created it), so here’s a simple script which launches a Rawline-enabled Rush shell:
1require 'rubygems' 2require 'rush' 3require 'rawline' 4 5class RawlineRush < Rush::Shell 6 7 def initialize 8 Rawline.basic_word_break_characters = "" 9 Rawline.completion_proc = completion_proc 10 super 11 end 12 13 def run 14 loop do 15 cmd = Rawline.readline('rawline_rush> ') 16 finish if cmd.nil? or cmd == 'exit' 17 next if cmd == "" 18 Rawline::HISTORY.push(cmd) 19 execute(cmd) 20 end 21 end 22end 23 24shell = RawlineRush.new.run
What happens here? Nothing much really, all I had to do was:
Rawline.basic_word_break_characters to the same value used in the original Rush codeRawline.completion_proc to the same completion Proc used in the original Rush coderun replacing Readline with RawlineAnd it works as it was intended to, i.e. typing root['b<TAB> will expand to root['bin/, etc.
Note that I didn’t write the completion Proc from scratch: it was already there.
After trying out Rush, the next logical step was trying IRB itself: I could never use it properly on Windows, and that was really frustrating.
After a few minutes trying to figure out how to start IRB programmatically, I quickly came up with a similar example:
1require 'irb' 2require 'irb/completion' 3require 'rubygems' 4require 'rawline' 5 6Rawline.basic_word_break_characters= " \t\n\"\\'`><;|&{(" 7Rawline.completion_append_character = nil 8Rawline.completion_proc = IRB::InputCompletor::CompletionProc 9 10class RawlineInputMethod < IRB::ReadlineInputMethod 11 include Rawline 12 def gets 13 if l = readline(@prompt, false) 14 HISTORY.push(l) if !l.empty? 15 @line[@line_no += 1] = l + "\n" 16 else 17 @eof = true 18 l 19 end 20 end 21end 22 23module IRB 24 @CONF[:SCRIPT] = RawlineInputMethod.new 25end 26IRB.start
In this case, Rawline is included in the RawlineInputMethod class, derived from the original ReadlineInputMethod class, i.e. the class IRB uses to define (guess…) how to input characters.
Again, all I had to do was set a few Rawline variables to match the ones used in Readline, and then redefine the function used to get characters. All done.
It works as expected (only with inline completion, of course): typing "test".ma<TAB> will give you "test".map, "test".match, etc.
You also get all Rawline key mappings for free (CTRL-K to clear the line, CTRL-U and CTRL-R to undo and redo, etc.), and you can define your own.
]]>Some of you asked for Readline compatibility/emulation and that was actually not too difficult to implement: all the bricks were already there, I just had to put them together in the right place.
The RawLine module (you can spell it “Rawline” as well, if you wish) now behaves like Readline. This means that you can now use RawLine like this (taken from examples/readline_emulation.rb):
Basically you get a readline method, a HISTORY constant like the one exposed by Readline (Rawline’s is a RawLine::HistoryBuffer object though — much more manageable), and a FILENAME_COMPLETION_PROC constant, which provides basic filename completion. Here it is:
1def filename_completion_proc 2 lambda do |word| 3 dirs = @line.text.split('/') 4 path = @line.text.match(/^\/|[a-zA-Z]:\//) ? "/" : Dir.pwd+"/" 5 if dirs.length == 0 then # starting directory 6 dir = path 7 else 8 dirs.delete(dirs.last) unless File.directory?(path+dirs.join('/')) 9 dir = path+dirs.join('/') 10 end 11 Dir.entries(dir).select { |e| (e =~ /^\./ && @match_hidden_files && word == '') || (e =~ /^#{word}/ && e !~ /^\./) } 12 end 13 end
You can find this function as part of the RawLine::Editor class. The result is not exactly the same Readline, because completion matches are not displayed underneath the line but inline and can be cycled through — which is one of Readline’s completion modes anyway.
A few methods of the RawLine::Editor class can now be accessed directly from the RawLine module, like with Readline:
Rawline.completion_proc — the Proc object used for TAB completion (defaults to FILENAME_COMPLETION_PROC).Rawline.completion_matches — an array of completion matches.Rawline.completion_append_char — a character to append after a successful completion.Rawline.basic_word_break_characters — a String listing all the characters used as word separators.Rawline.completer_word_break_characters — same as above.Rawline.library_version — the current version of the Rawline library.Rawline.clear_history — to clear the current history.Rawline.match_hidden_files — whether FILENAME_COMPLETION_PROC matches hidden files and folders or not.I bet you didn’t know these methods were even in the Readline wrapper, did you? Probably because of lack of documentation.
Anyhow, another very important difference beween Rawline and Readline is Rawline.editor, i.e. the default instance of RawLine::Editor used by the Rawline module itself.
This makes things easier if you want more control over the line which is being edited and the previously-edited lines. Sure, Readline#completion_proc exposes the current word being typed before hitting tab, and so does Rawline#completion_proc the difference is that if you access Rawline.editor.line you get a RawLine::Line object with all the information you could possibly need about the current line: the position of the cursor, the text, the order the characters were entered, etc. etc.
Now you can imagine why it took me a few minutes to write the filename_completion_proc method (and why it will take you even less time to write your own similar method if you wanna do something different): you can access not only the last word being typed but also the current and previous lines (through Rawline.editor.history or just Rawline::HISTORY)!
It must be said, as usual, that Rawline is not a complete replacement for the Readline library yet (and it will probably never be, as Readline is huge!), but it’s a good cross-platform, more Ruby-esque alternative to what’s currently available by the Readline wrapper for Ruby.
It’s not as fast, of course, especially when completing long words, but it’s quite usable. The following libraries are not required but recommended:
win32console (on Windows)termios (on *nix)They basically make Rawline faster. If you don’t use them, Rawline will fall back on its pure-Ruby implementation to move left and right (i.e. printing backspaces and spaces character codes instead of ASCII escape codes).
Unfortunately, there’s no vi_editing_mode or emacs_editing_mode yet (for time constraints: they can be implemented!) but patches are very welcome. Also, if you need more features, all you have to do is ask :-)
P.S.: Check out the new Project Page and especially its Resources section!
]]>*Raw*Line is the new name for InLine, in case you didn’t guess. The name was changed to avoid name collision problems with the RubyInline project.
Here’s what’s new:
In particular, I decided to provide an “optimized implementation” for the left and right arrows using escape sequences rather than shameful hacks. This is now possible because the Win32Console gem now enables ANSI escape sequences on Windows as well (weehee!).
So:
The new implementation is significantly faster than the old one, on Windows at least, and the cursor now blinks properly when left or right arrows are pressed.
I re-emplemented only cursor movement because I’m still having some problems in getting the delete/insert escapes to work properly (or better: how I want them to work!).
]]>This makes sense, and I’m ready to change the name of my project, although I’m not that good at choosing original and smart names, so well, any suggestion is more than welcome!
I was thinking of something like:
I personally think that RawLine is probably the best option, but please, if have any better idea just speak up!
P.S.: “RedLine” is taken, unfortunately, otherwise it would have been my first choice since the beginning.
]]>More specifically, I liked the fact that I could use the GNU Readline library with Ruby seamlessly on both Windows and Linux.
Readline offers quite a lot of features which are useful for those people like me who enjoy creating command-line scripts, in a nutshell, it provides:
Basically it makes your command-line interface fast and powerful, and that’s not an overstatement. Ruby’s own IRB can be enhanced by enabling readline and completion, and it works great — at least on *nix systems.
For some weird reason, some people had problems with Readline on Windows: in particular, things get nasty when you start editing long lines. Text gets garbled, the cursor goes up one or two lines and doesn’t come back, and other similar leprechaun’s tricks, which are not that funny after a while.
Apparently there’s no alternative to Readline in the Ruby world. If you wan’t tab completion that’s it, you’re stuck. Would it be difficult to implement some of Readline functionality natively in Ruby? Maybe, but the problem is that for some reason the Ruby Standard Library doesn’t have low level methods to operate on keystrokes…
…but luckily, the HighLine gem does! James Edward Gray II keeps pointing out here and here that HighLine’s own get_character method does just that: it returns the corresponding character code(s) right when a key is pressed, unlike IO#gets() which waits for the user to press ENTER.
Believe it or not, that tiny method can do wonders…h2. Reverse-engineering escape codes
So here’s a little script which uses get_character() in an endless loop, diligently printing the character codes corresponding to a keystroke:
1#!/usr/local/bin/ruby -w 2 3require 'rubygems' 4require 'highline/system_extensions' 5 6include HighLine::SystemExtensions 7 8puts "Press a key to view the corresponding ASCII code(s) (or CTRL-X to exit)." 9 10loop do 11 12 print "=> " 13 char = get_character 14 case char 15 when ?\C-x: print "Exiting..."; exit; 16 else puts "#{char.chr} [#{char}] (hex: #{char.to_s(16)})"; 17 end 18 19end
A pretty harmless little thing. Try to run it and press some keys, and see what you get:
Press a key to view the corresponding ASCII code(s) (or CTRL-X to exit).
=> a 96 (hex: 61)
=> 1 49 (hex: 31)
=> Q 81 (hex: 51)
=> α 224 (hex: e0)
=> K 75 (hex: 4b)
Hang on, what are the last two codes? A left arrow key on Windows, apparently.
Welcome to the wonderful world of input escape sequences!
To cut a long story short, both Windows and *nix system “terminals” translate special keystrokes into sequences of two or more codes. This applies to things like DEL, INSERT, arrows, etc. etc.
For some ideas, check out:
Let’s now assume that we’re smart and we can write a program which can parse keystroke properly, including handling different input escape sequences according to the OS, what can it be used for?
Well:
get_character doesn’t print anything, it “steals” the keystroke)We could setup TAB to auto-complete the current word according to an array of matches, or bind the up arrow to load the last line typed in by the user, for example, that’s basically something Readline does, right?
I created a small project on RubyForge called RawLine (not to be confused with RubyInline, a completely different thing altogether, sorry about that) to play around with the possibilities offered by the get_character method. The library is just a preview of things which can be done, but it’s already usable, provided that you’re brave enough to try it out, that is.
The basic idea behind RawLine is to be able to parse keystrokes properly on different platforms and re-bind them to a set of predefined, cross-platform actions or a user-defined code block.
The first challenge was to re-invent the wheel, i.e. re-bind keystrokes to their typical actions: a left arrow moves the cursor left, a backspace deletes the character at the left of the cursor and so on. Yes, because get_characters gives you the right character codes at the price of cancelling their normal effects, which is a great thing, as you’ll soon find out.
Printing a character on the screen was one of the easiest tasks (at first). IO#putc does the job pretty well: it prints a character out.
What about moving left? Easy: print a non-descructive backspace (\b) and hope it is really not destructive. I did some tests and it seems to do as it’s told and move the cursor back by one position.
Moving right was a little trickier: the easiest thing I found was to re-print the character under the cursor, which will then move the cursor forward (as naive as it may seem, it does the job!). If there’s nothing under the cursor, then we must be at the end of the line and it shouldn’t move anywhere, so there we go.
What if I move left a bit and then start typing normal characters? Well, everything is rewritten of course: this will be our “character replace mode”. Unfortunately users don’t like this behavior that much, so what I did was this:
Again, a primitive solution which works seamlessly on all platforms, and yes, it’s fast enough that you don’t notice the difference.
As you may have guessed, this of course means that I always had to keep track of:
Backspace and delete were implemented in a similar way, you can figure it out yourself or look at the source code: I won’t bore you any further!
The next step was to implement a history for both the characters inputted by the user (to allow undoing and redoing operations) and for the whole lines. This was just an ordinary programming exercise: a simple buffer with some extra controls here and there, nothing too scary.
So every “modification” to the current line being typed is saved in a line history buffer and all the lines entered are saved in another history buffer. All is left is to allow users to navigate through these buffers back and forth.
Nothing impossible: all I had to do was keeping track of the current element of the history being retrieved and then overwrite the current line with a new line stored in the buffer? How’s this line overwriting done? Same old:
Easy and naive, as usual. But again, it works well enough.
The other challange was word completion. The current implementation can be summarized as follows:
completion_proc method which returns an array and show the first element of the array (in this case I actually used a cyclic RawLine::HistoryBuffer, not an array)Obviously this means that:
word_separator, which can obviously modified at runtime, with care.completion_proc, typically you may want to return only the elements matching the word which is currently being written, so that’s given as default parameter for your proc. Exactly like with ReadLine, the only difference is that you can access other things like the whole line and the whole history in real time, which can be really handy at times!Here’s a simple example:
All these pretty things are obviously bound to some keystrokes. If the key corresponds to only one code, everything is fine, but because special keys typically aren’t so it was necessary to implement a mechanism to track an escape key (e.g. 0xE0 and 0 on Windows and \e on Linux) and listen to further characters, in case a known sequence is found. Anyhow, the final result of the method used for character binding is the following:
bind(key, &block)
Where key can be:
Fixnum corresponding to a single character codeArray of one or more character codesString corresponding to an escape sequenceSymbol corresponding to a known escape sequence or keyHash to define a new key or escape sequencesSo, in the end you can do things like this:
Which, for Rubyists, it’s far sexier and more flexible than editing an .inputrc file.
A code example is better than a thousand words, right? So here you are:
1#!/usr/local/bin/ruby -w 2 3require 'rubygems' 4require 'rawline' 5 6puts "*** Inline Editor Test Shell ***" 7puts " * Press CTRL+X to exit" 8puts " * Press CTRL+C to clear command history" 9puts " * Press CTRL+D for line-related information" 10puts " * Press CTRL+E to view command history" 11 12editor = RawLine::Editor.new 13 14editor.bind(:ctrl_c) { editor.clear_history } 15editor.bind(:ctrl_d) { editor.debug_line } 16editor.bind(:ctrl_e) { editor.show_history } 17editor.bind(:ctrl_x) { puts; puts "Exiting..."; exit } 18 19editor.completion_proc = lambda do |word| 20 if word 21 ['select', 'update', 'delete', 'debug', 'destroy'].find_all { |e| e.match(/^#{Regexp.escape(word)}/) } 22 end 23end 24 25loop do 26 puts "You typed: [#{editor.read("=> ").chomp!}]" 27end
This example can be found in examples/rawline_shell.rb within the RawLine source code or gem package.
I currently released RawLine 0.1.0 on SourceForge, and it can be installed via:
gem install -r rawline
The RDoc documentation is available here.
Feel free to try it out. First of all try the rawline_shell.rb example, and see if it works on your machine. If it doesn’t than maybe you try re-binding some keys (use key_tester.rb to “reverse-engineer” your terminal’s input escape sequences), and let me know!
Status information and limitations:
get_character won’t parse characters correctly if you press and hold a key, and that, trust me, is a real mess!).For any ideas on where to go from here, comments and feedback, just reply below or send an email to my usual email address.
]]>