H3RALD - Tag 'rawline' (RSS Feed)

Real-world Rawline usage

h3rald@h3rald.com — Fri, 06 Mar 2009 20:54:00 -0700

So I finally decided to update RawLine last week, and I added a more Readline-like API. When I first started the project, I was determined not to do that, because the current Readline wrapper shipped with Ruby is not very Ruby-ish: it’s a wrapper, after all!

The good thing of having a new API compatible with Readline is that now people can use RawLine in their Readline-powered scripts, with very minor modifications.

Let’s have a look at some examples (they are also shipped with Rawline v0.3.1):
h3. Rush

Rush is an excellent gem which provides a cross-platform shell environment, entirely written in Ruby.
Being a shell, it obviously uses Readline for tab completion, and that does the job on Linux. On Windows though, things aren’t that easy:

text gets garbled if you write long lines
you can’t type certain characters if they use some key modifiers like , etc.

RawLine doesn’t have these problems (that’s the very reason why I created it), so here’s a simple script which launches a Rawline-enabled Rush shell:

 1require 'rubygems'
 2require 'rush'
 3require 'rawline'
 4
 5class RawlineRush < Rush::Shell
 6
 7  def initialize
 8    Rawline.basic_word_break_characters = "" 
 9    Rawline.completion_proc = completion_proc
10    super
11  end
12
13  def run
14    loop do
15      cmd = Rawline.readline('rawline_rush> ')
16      finish if cmd.nil? or cmd == 'exit'
17      next if cmd == ""
18      Rawline::HISTORY.push(cmd)
19      execute(cmd)
20    end
21  end
22end
23
24shell = RawlineRush.new.run

What happens here? Nothing much really, all I had to do was:

Derive a new class from Rush::Shell
Set Rawline.basic_word_break_characters to the same value used in the original Rush code
Set Rawline.completion_proc to the same completion Proc used in the original Rush code
Rewrite the original run replacing Readline with Rawline

And it works as it was intended to, i.e. typing root['b<TAB> will expand to root['bin/, etc.
Note that I didn’t write the completion Proc from scratch: it was already there.

IRB

After trying out Rush, the next logical step was trying IRB itself: I could never use it properly on Windows, and that was really frustrating.
After a few minutes trying to figure out how to start IRB programmatically, I quickly came up with a similar example:

 1require 'irb'
 2require 'irb/completion'
 3require 'rubygems'
 4require 'rawline'
 5
 6Rawline.basic_word_break_characters= " \t\n\"\\'`><;|&{(" 
 7Rawline.completion_append_character = nil
 8Rawline.completion_proc = IRB::InputCompletor::CompletionProc
 9
10class RawlineInputMethod < IRB::ReadlineInputMethod
11  include Rawline
12  def gets
13    if l = readline(@prompt, false)
14      HISTORY.push(l) if !l.empty?
15      @line[@line_no += 1] = l + "\n"
16    else
17      @eof = true
18      l
19    end
20  end
21end
22
23module IRB
24  @CONF[:SCRIPT] = RawlineInputMethod.new
25end
26IRB.start

In this case, Rawline is included in the RawlineInputMethod class, derived from the original ReadlineInputMethod class, i.e. the class IRB uses to define (guess…) how to input characters.
Again, all I had to do was set a few Rawline variables to match the ones used in Readline, and then redefine the function used to get characters. All done.

It works as expected (only with inline completion, of course): typing "test".ma<TAB> will give you "test".map, "test".match, etc.

You also get all Rawline key mappings for free (CTRL-K to clear the line, CTRL-U and CTRL-R to undo and redo, etc.), and you can define your own.

RawLine 0.3.0 released — now with Readline emulation

h3rald@h3rald.com — Sat, 28 Feb 2009 23:47:00 -0700

RawLine 0.3.0 has been released. This new milestones fixes some minor bugs and adds some new functionalities, must notably:

Ruby 1.9 support
A filename completion function
A new API very similar to the one exposed by the Ruby wrapper for GNU Readline

Some of you asked for Readline compatibility/emulation and that was actually not too difficult to implement: all the bricks were already there, I just had to put them together in the right place.

The RawLine module (you can spell it “Rawline” as well, if you wish) now behaves like Readline. This means that you can now use RawLine like this (taken from examples/readline_emulation.rb):

 1include Rawline
 2
 3puts "*** Readline Emulation Test Shell ***"
 4puts " * Press CTRL+X to exit"
 5puts " * Press <TAB> for file completion"
 6
 7Rawline.editor.bind(:ctrl_x) { puts; puts "Exiting..."; exit }
 8
 9Dir.chdir '..'
10
11loop do
12  puts "You typed: [#{readline("=> ", true).chomp!}]"
13end

Basically you get a readline method, a HISTORY constant like the one exposed by Readline (Rawline’s is a RawLine::HistoryBuffer object though — much more manageable), and a FILENAME_COMPLETION_PROC constant, which provides basic filename completion. Here it is:

 1def filename_completion_proc
 2      lambda do |word|
 3        dirs = @line.text.split('/')
 4          path = @line.text.match(/^\/|[a-zA-Z]:\//) ? "/" : Dir.pwd+"/"
 5        if dirs.length == 0 then # starting directory
 6          dir = path
 7        else
 8          dirs.delete(dirs.last) unless File.directory?(path+dirs.join('/'))
 9          dir = path+dirs.join('/')
10        end
11        Dir.entries(dir).select { |e| (e =~ /^\./ && @match_hidden_files && word == '') || (e =~ /^#{word}/ && e !~ /^\./) }
12      end
13    end

You can find this function as part of the RawLine::Editor class. The result is not exactly the same Readline, because completion matches are not displayed underneath the line but inline and can be cycled through — which is one of Readline’s completion modes anyway.

A few methods of the RawLine::Editor class can now be accessed directly from the RawLine module, like with Readline:

Rawline.completion_proc — the Proc object used for TAB completion (defaults to FILENAME_COMPLETION_PROC).
Rawline.completion_matches — an array of completion matches.
Rawline.completion_append_char — a character to append after a successful completion.
Rawline.basic_word_break_characters — a String listing all the characters used as word separators.
Rawline.completer_word_break_characters — same as above.
Rawline.library_version — the current version of the Rawline library.
Rawline.clear_history — to clear the current history.
Rawline.match_hidden_files — whether FILENAME_COMPLETION_PROC matches hidden files and folders or not.

I bet you didn’t know these methods were even in the Readline wrapper, did you? Probably because of lack of documentation.
Anyhow, another very important difference beween Rawline and Readline is Rawline.editor, i.e. the default instance of RawLine::Editor used by the Rawline module itself.

This makes things easier if you want more control over the line which is being edited and the previously-edited lines. Sure, Readline#completion_proc exposes the current word being typed before hitting tab, and so does Rawline#completion_proc the difference is that if you access Rawline.editor.line you get a RawLine::Line object with all the information you could possibly need about the current line: the position of the cursor, the text, the order the characters were entered, etc. etc.
Now you can imagine why it took me a few minutes to write the filename_completion_proc method (and why it will take you even less time to write your own similar method if you wanna do something different): you can access not only the last word being typed but also the current and previous lines (through Rawline.editor.history or just Rawline::HISTORY)!

It must be said, as usual, that Rawline is not a complete replacement for the Readline library yet (and it will probably never be, as Readline is huge!), but it’s a good cross-platform, more Ruby-esque alternative to what’s currently available by the Readline wrapper for Ruby.

It’s not as fast, of course, especially when completing long words, but it’s quite usable. The following libraries are not required but recommended:

win32console (on Windows)
termios (on *nix)

They basically make Rawline faster. If you don’t use them, Rawline will fall back on its pure-Ruby implementation to move left and right (i.e. printing backspaces and spaces character codes instead of ASCII escape codes).

Unfortunately, there’s no vi_editing_mode or emacs_editing_mode yet (for time constraints: they can be implemented!) but patches are very welcome. Also, if you need more features, all you have to do is ask :-)

P.S.: Check out the new Project Page and especially its Resources section!

New Release: RawLine 0.2.0

h3rald@h3rald.com — Tue, 01 Apr 2008 21:33:00 -0600

~~InLine~~ RawLine 0.2.0 is out!

*Raw*Line is the new name for InLine, in case you didn’t guess. The name was changed to avoid name collision problems with the RubyInline project.

Here’s what’s new:

Added /examples and /test directory to gem.
Escape codes can now be used in prompt.
It is now possible to use bind(key, &block) with a String as key, even if the corresponding escape sequence is not defined.
Added Editor#write_line(string) to print a any string (and “hit return”).
Library name changed to “RawLine” to avoid name collision issues (Bug 18879).
Provided alternative implementation for left and right arrows if terminal
supports escape sequences (on Windows, it requires the Win32Console gem).

In particular, I decided to provide an “optimized implementation” for the left and right arrows using escape sequences rather than shameful hacks. This is now possible because the Win32Console gem now enables ANSI escape sequences on Windows as well (weehee!).

So:

If you’re on *nix all good, your terminal is smart and can understand escape sequences => the new implementation will be used.
If you’re on Windows and you installed Win32Console, your termnal is smart and can understand escape sequences => the new implementation will be used.
If you’re on Windows and you didn’t install Win32Console, then your terminal is stupid and it doesn’t understand escape sequences, so the old implementation will be used.

The new implementation is significantly faster than the old one, on Windows at least, and the cursor now blinks properly when left or right arrows are pressed.

I re-emplemented only cursor movement because I’m still having some problems in getting the delete/insert escapes to work properly (or better: how I want them to work!).

InLine name change: what's your opinion?

h3rald@h3rald.com — Wed, 26 Mar 2008 23:30:00 -0600

I’ve been kindly asked by the lead developer of RubyInLine to change the name of my InLine project, due to potential confusion and conflicts.

This makes sense, and I’m ready to change the name of my project, although I’m not that good at choosing original and smart names, so well, any suggestion is more than welcome!

I was thinking of something like:

RawLine
EditLine
RawInput
RubyInput
RubyLine

I personally think that RawLine is probably the best option, but please, if have any better idea just speak up!

P.S.: “RedLine” is taken, unfortunately, otherwise it would have been my first choice since the beginning.

RawLine - a 100% Ruby solution for console inline editing

h3rald@h3rald.com — Sun, 09 Mar 2008 23:59:00 -0600

One of the many things I like about Ruby is its cross-platform nature: as a general rule, Ruby code runs on everything which supports Ruby, regardless of its architecture and platform (yes, there are quite a few exceptions, but let’s accept this generalization for now).

More specifically, I liked the fact that I could use the GNU Readline library with Ruby seamlessly on both Windows and Linux.
Readline offers quite a lot of features which are useful for those people like me who enjoy creating command-line scripts, in a nutshell, it provides:

File/Word completion
History support
Custom key bindings which can be modified via .inputrc
Emacs and Vi edit modes

Basically it makes your command-line interface fast and powerful, and that’s not an overstatement. Ruby’s own IRB can be enhanced by enabling readline and completion, and it works great — at least on *nix systems.

For some weird reason, some people had problems with Readline on Windows: in particular, things get nasty when you start editing long lines. Text gets garbled, the cursor goes up one or two lines and doesn’t come back, and other similar leprechaun’s tricks, which are not that funny after a while.

Apparently there’s no alternative to Readline in the Ruby world. If you wan’t tab completion that’s it, you’re stuck. Would it be difficult to implement some of Readline functionality natively in Ruby? Maybe, but the problem is that for some reason the Ruby Standard Library doesn’t have low level methods to operate on keystrokes…

…but luckily, the HighLine gem does! James Edward Gray II keeps pointing out here and here that HighLine’s own get_character method does just that: it returns the corresponding character code(s) right when a key is pressed, unlike IO#gets() which waits for the user to press ENTER.

Believe it or not, that tiny method can do wonders…h2. Reverse-engineering escape codes

So here’s a little script which uses get_character() in an endless loop, diligently printing the character codes corresponding to a keystroke:

 1#!/usr/local/bin/ruby -w
 2
 3require 'rubygems'
 4require 'highline/system_extensions'
 5
 6include HighLine::SystemExtensions
 7
 8puts "Press a key to view the corresponding ASCII code(s) (or CTRL-X to exit)."
 9
10loop do
11
12  print "=> "
13  char = get_character
14  case char
15  when ?\C-x: print "Exiting..."; exit;
16  else puts "#{char.chr} [#{char}] (hex: #{char.to_s(16)})";
17  end
18  
19end

A pretty harmless little thing. Try to run it and press some keys, and see what you get:

Press a key to view the corresponding ASCII code(s) (or CTRL-X to exit).
=> a 96 (hex: 61)
=> 1 49 (hex: 31)
=> Q 81 (hex: 51)
=> α 224 (hex: e0)
=> K 75 (hex: 4b)

Hang on, what are the last two codes? A left arrow key on Windows, apparently.

Welcome to the wonderful world of input escape sequences!

To cut a long story short, both Windows and *nix system “terminals” translate special keystrokes into sequences of two or more codes. This applies to things like DEL, INSERT, arrows, etc. etc.
For some ideas, check out:

Windows Scancodes (Thanks Huff)
VT220 Terminal Input Sequences (Thanks James)

Let’s now assume that we’re smart and we can write a program which can parse keystroke properly, including handling different input escape sequences according to the OS, what can it be used for?
Well:

For normal characters, just print them back to the screen (get_character doesn’t print anything, it “steals” the keystroke)
For special characters, do something nice!

We could setup TAB to auto-complete the current word according to an array of matches, or bind the up arrow to load the last line typed in by the user, for example, that’s basically something Readline does, right?

RawLine: how it works and what it does

I created a small project on RubyForge called RawLine (not to be confused with RubyInline, a completely different thing altogether, sorry about that) to play around with the possibilities offered by the get_character method. The library is just a preview of things which can be done, but it’s already usable, provided that you’re brave enough to try it out, that is.

The basic idea behind RawLine is to be able to parse keystrokes properly on different platforms and re-bind them to a set of predefined, cross-platform actions or a user-defined code block.

Basic line-editing operations

The first challenge was to re-invent the wheel, i.e. re-bind keystrokes to their typical actions: a left arrow moves the cursor left, a backspace deletes the character at the left of the cursor and so on. Yes, because get_characters gives you the right character codes at the price of cancelling their normal effects, which is a great thing, as you’ll soon find out.

Printing a character on the screen was one of the easiest tasks (at first). IO#putc does the job pretty well: it prints a character out.
What about moving left? Easy: print a non-descructive backspace (\b) and hope it is really not destructive. I did some tests and it seems to do as it’s told and move the cursor back by one position.

Moving right was a little trickier: the easiest thing I found was to re-print the character under the cursor, which will then move the cursor forward (as naive as it may seem, it does the job!). If there’s nothing under the cursor, then we must be at the end of the line and it shouldn’t move anywhere, so there we go.

What if I move left a bit and then start typing normal characters? Well, everything is rewritten of course: this will be our “character replace mode”. Unfortunately users don’t like this behavior that much, so what I did was this:

Copy all characters from the one at the left of the cursor till the end of the line
Print the character to be inserted
Re-print the previously-copied characters
Move the cursor back at the right place

Again, a primitive solution which works seamlessly on all platforms, and yes, it’s fast enough that you don’t notice the difference.

As you may have guessed, this of course means that I always had to keep track of:

The cursor position within the line
The text currently printed to the screen

Backspace and delete were implemented in a similar way, you can figure it out yourself or look at the source code: I won’t bore you any further!

History management

The next step was to implement a history for both the characters inputted by the user (to allow undoing and redoing operations) and for the whole lines. This was just an ordinary programming exercise: a simple buffer with some extra controls here and there, nothing too scary.

So every “modification” to the current line being typed is saved in a line history buffer and all the lines entered are saved in another history buffer. All is left is to allow users to navigate through these buffers back and forth.
Nothing impossible: all I had to do was keeping track of the current element of the history being retrieved and then overwrite the current line with a new line stored in the buffer? How’s this line overwriting done? Same old:

Move the cursor to the beginnig of the line
Print X spaces, where X is the line length, so that the characters are no longer displayed in the console
Move the cursor back to the beginning of the line
Print the new line.

Easy and naive, as usual. But again, it works well enough.

Word completion

The other challange was word completion. The current implementation can be summarized as follows:

If TAB (or another character, if you wish) is pressed, call a user-defined completion_proc method which returns an array and show the first element of the array (in this case I actually used a cyclic RawLine::HistoryBuffer, not an array)
If the user presses TAB again, show another match, and so ad infinitum if the user keeps pressing TAB.
If the user presses another key, accept the default completion and move on.

Obviously this means that:

RawLine has to keep track of the current “word”. A word is everything separated by a user defined word_separator, which can obviously modified at runtime, with care.
Regarding the completion_proc, typically you may want to return only the elements matching the word which is currently being written, so that’s given as default parameter for your proc. Exactly like with ReadLine, the only difference is that you can access other things like the whole line and the whole history in real time, which can be really handy at times!

Here’s a simple example:

1editor.completion_proc = lambda do |word|
2  if word
3    ['select', 'update', 'delete', 'debug', 'destroy'].find_all  { |e| e.match(/^#{Regexp.escape(word)}/) }
4  end
5end

Custom key bindings

All these pretty things are obviously bound to some keystrokes. If the key corresponds to only one code, everything is fine, but because special keys typically aren’t so it was necessary to implement a mechanism to track an escape key (e.g. 0xE0 and 0 on Windows and \e on Linux) and listen to further characters, in case a known sequence is found. Anyhow, the final result of the method used for character binding is the following:

bind(key, &block)

Where key can be:

A Fixnum corresponding to a single character code
An Array of one or more character codes
A String corresponding to an escape sequence
A Symbol corresponding to a known escape sequence or key
A Hash to define a new key or escape sequences

So, in the end you can do things like this:

1editor.bind(:left_arrow) { editor.move_left }
2editor.bind("\etest") { editor.overwrite_line("Test!!") }
3editor.bind(?\C-z) { editor.undo }
4editor.bind([24]) { exit }

Which, for Rubyists, it’s far sexier and more flexible than editing an .inputrc file.

How do I use it, anyway?

A code example is better than a thousand words, right? So here you are:

 1#!/usr/local/bin/ruby -w
 2
 3require 'rubygems'
 4require 'rawline'
 5
 6puts "*** Inline Editor Test Shell ***"
 7puts " * Press CTRL+X to exit"
 8puts " * Press CTRL+C to clear command history"
 9puts " * Press CTRL+D for line-related information"
10puts " * Press CTRL+E to view command history"
11
12editor = RawLine::Editor.new
13
14editor.bind(:ctrl_c) { editor.clear_history }
15editor.bind(:ctrl_d) { editor.debug_line }
16editor.bind(:ctrl_e) { editor.show_history }
17editor.bind(:ctrl_x) { puts; puts "Exiting..."; exit }
18
19editor.completion_proc = lambda do |word|
20  if word
21    ['select', 'update', 'delete', 'debug', 'destroy'].find_all  { |e| e.match(/^#{Regexp.escape(word)}/) }
22  end
23end
24
25loop do
26  puts "You typed: [#{editor.read("=> ").chomp!}]"
27end

This example can be found in examples/rawline_shell.rb within the RawLine source code or gem package.

Current status and availability

I currently released RawLine 0.1.0 on SourceForge, and it can be installed via:

gem install -r rawline

The RDoc documentation is available here.

Feel free to try it out. First of all try the rawline_shell.rb example, and see if it works on your machine. If it doesn’t than maybe you try re-binding some keys (use key_tester.rb to “reverse-engineer” your terminal’s input escape sequences), and let me know!

Status information and limitations:

It has been tested on Windows (XP, using the usual command prompt) and on Linux (ZenWalk, using XFCE Terminal).
It can handle lines no longer than the maximum terminal width – 2. This is to ensure that the cursor never “falls down” to the next line.
On Windows, the cursor doesn’t blink immedialy when moving left, but it moves, don’t worry.
On Linux, you should really consider installing the Termios library for a faster experience (otherwise get_character won’t parse characters correctly if you press and hold a key, and that, trust me, is a real mess!).
RawLine is very far from being a complete replacement for the ReadLine library, and it is currently in alpha stage.
Release 0.1.0 has been created after 2 weeks of sporadic coding during lunch breaks and week-ends.

For any ideas on where to go from here, comments and feedback, just reply below or send an email to my usual email address.