Wednesday, June 20, 2007

Roman Numeral Conversion in Factor

I whipped up some words to convert integers to Roman numerals. The word <PRIVATE changes the IN: vocabulary to roman.private to hide the implementation from the library user. Of course you can access these private words, like all words in Factor, with a USE:.

The algorithm is simple. Going from high to low, iterate the Roman numeral values and /mod (integer division with remainder) each with the input, outputting the divisor and replacing the input with the remainder. This algorithm treats 4s and 9s as digits, just as it treats single letters as digits (i, v, x, etc). Without the 4s and 9s, you end up getting longer answers that, while logical, are wrong, e.g. 9 is "ix", not "viv". (I found this bug in the first iteration while writing unit tests.)

The words we care about, >roman and >ROMAN, are placed IN: roman because of the PRIVATE> word, which drops back to the public vocabulary roman. The > in a word's name is a convention for words that do conversions; the parentheses around the word (>roman) mean it's an implementation word; you should never have a (>roman) without also having a >roman. Picking these names is done purely by convention--the only forbidden word names are numbers and words that start with a ", which parse as strings. Everything until whitespace is a word name.

The conversion from Roman numerals back to integers and roman+, roman*, etc are in roman.factor.

USING: arrays assocs kernel math math.vectors namespaces
quotations sequences sequences.private strings ;
IN: roman

<PRIVATE

: roman-digits ( -- seq )
{ "m" "cm" "d" "cd" "c" "xc" "l" "xl" "x" "ix" "v" "iv" "i" } ;

: roman-values ( -- seq )
{ 1000 900 500 400 100 90 50 40 10 9 5 4 1 } ;

TUPLE: roman-range-error n ;

: roman-range-check ( n -- )
dup 1 3999 between? [
drop
] [
<roman-range-error> throw
] if ;

: (>roman) ( n -- )
roman-values roman-digits [
>r /mod swap r> <repetition> concat %
] 2each drop ;

PRIVATE>

: >roman ( n -- str )
dup roman-range-check [
(>roman)
] "" make ;

: >ROMAN ( n -- str ) >roman >upper ;

Friday, April 27, 2007

Windows CE SEH for ARM with GCC

Structure exception handling (SEH) is a low-level way to handle software and hardware exceptions in Microsoft Windows. Other exception handling mechanisms are built on top of SEH. Factor uses SEH in order to report stack underflow/overflow errors on Windows. Microsoft Visual Studio (VS) would usually take care of the tricky implementation details by supplying __try, __except, and __finally keywords, but Factor's compiler relies on globally assigning the datastack and retainstack to specific registers, a feature which VS does not support. Conversely, GCC does not support __try, so we are left to implement the exception handler in assembly.

The internal implementation is both OS and architecture dependent, meaning that we have to support SEH on Windows NT on x86, and Windows CE for x86 and ARM. Digging through MSDN leads to a page called SEH in RISC Environments, where RISC standing for "Reduced Instruction Set Computer". (You are just supposed to know that ARM is RISC and x86 is CISC). I recommend reading about SEH on MSDN, but it basically says that SEH on RISC uses Virtual Unwinding and that you will need to set the PDATA structure for your function in order to set up the exception handler. Virtual unwinding traverses the framestack until it finds a frame with an exception handler, at which time it will actually unwind the framestack to that point and call the exception handler.

All we need to do is define void run_toplevel(void){...} to install our exception_handler and call Factor's "main" subroutine, run.

vm/os-windows-ce.c

long exception_handler(PEXCEPTION_RECORD rec, void *frame, void *ctx, void *disp
atch)
{
memory_protection_error(
rec->ExceptionInformation[1] & 0x1ffffff, // mask off process slot
native_stack_pointer());
return -1; /* unreachable */
}


vm/os-windows-ce-arm.S

.text

.globl run_toplevel

.word exception_handler
.word 0

run_toplevel:
ldr pc, _Prun

_Prun: .word run

.section .pdata
.word run_toplevel
.word 0xc0000002 | (0xFFFFF << 8)


The C code passes the memory fault address, stored in ExceptionInformation[1], and the native stack pointer (another assembly function that simply moves ESP -> EAX) to a function that converts the fault address into a Factor stack error message, e.g. "Datastack underflow". The fault address is actually a slotized address, meaning that it is relative to some process slot which must be masked off. Annoyingly, the exceptions happen at different address ranges on the emulator and on a real mobile device. Slotized addresses in this context are not well documented on MSDN, but the Windows CE Blog Team quickly answered my email. (Thanks!)

The assembly code is mostly boilerplate. The .text directive starts us in the executable code section. We're going to be exporting void run_toplevel(void){...} in order to call it from C, so we declare it as a .globl and start the definition. We simply call our "main" function, void run(void){...}, which is the platform-dependent run function to start the Factor interpreter.

Hopefully, if someone else has to do this it will take much less time.

Thursday, April 12, 2007

Building Factor in Cygwin now supported

Factor has only compiled on MinGW since we made the Windows Factor port a couple of years ago. The problem with Cygwin has been its dependence on cygwin1.dll, which slows your program. Also, the Cygwin installer doesn't add it to your path, so that is one more step (and not standard). Additionally, structured exception handling (SEH) works the first time but hangs on the second time it triggers with the dll, e.g. on the second stack underflow.

However, Eric Mertens, a new Factor user, found a compiler flag to disable cygwin.dll:
-mno-cygwin

Three cheers for new users!

This flag fixed both the dll dependency and the SEH bug, allowing all unit tests to pass. The only other change I had to make for the port was to #include <wchar.h> for the compiler to find wcslen().

So now you can compile Factor with either MinGW or Cygwin, and both versions are officially supported.

Saturday, December 30, 2006

Serialization To Disk

For fun, I wrote a TinyURL clone in Factor. You can try it out at http://wee-url.com. I needed a way to save the data to a file, so I used libs/serialize to serialize the hashtable containing the links. There was yet another bug involving vector literals -- the serialization code would reuse a vector, so the first time you serialized the output is correct, but the next time it would produce garbage.
  scratchpad SYMBOL: aa H{ { 1 2 } } aa set
scratchpad aa get .
H{ { 1 2 } }
scratchpad aa get [ [ serialize ] with-serialized ] string-out .
"hp\0\0\0\u0001Dap\0\0\0\u0001Ep\0\0\0\u0001\u0001ap\0\0..."
scratchpad aa get [ [ serialize ] with-serialized ] string-out .
"op\0\0\0\u0001:"
The fix was to change a V{ } to a V{ } clone.


Serialization to Disk Code

: save-fstore ( variable path -- )
[ [ get serialize ] with-serialized ] with-stream ;

: load-fstore ( variable path -- )
dup exists? [
[ [ deserialize ] with-serialized ] with-stream
swap
] [
drop >r H{ } clone r>
] if set-global ;

: fstore-set ( variable fstore -- )
get >r [ get ] keep r> set-hash ;

: fstore-get ( default variable fstore -- )
get dupd hash* [ swap set-global drop ] [ drop set-global ] if ;


save-fstore and load-fstore save/load a hashtable to disk. The hashtable keys are variable names and the values are the variable values. fstore-set puts your variable in the hashtable, so you should call this directly before saving to disk. fstore-get loads the data from your store. If there is no fstore or if the key does not exist, it uses your default value.

The loading code is pretty simple:
: fstore-name "wee-url.fstore" ;
SYMBOL: wee-shortcuts
SYMBOL: wee-fstore
wee-fstore fstore-name load-fstore
H{ } clone wee-shortcuts wee-fstore fstore-get
The variable wee-shortcuts stores the encoded URLs, while wee-fstore is the hashtable that the above code reads/writes to disk.

Finally, to write the store:
: save-shortcuts ( -- )
wee-shortcuts wee-fstore fstore-set
wee-fstore fstore-name save-fstore ;

Notice that we add wee-shortcuts to the hashtable before saving to disk.

I'm going to rewrite the wee-url code to use Furnace instead of callback-responder. Once I get it looking nice, I'll show you how it works.

Thursday, November 23, 2006

Compiling Factor on the Intel Mac

To compile Factor for Mac you will need:
  • Installation DVDs that came with your Mac (for X11)
  • darwin-ports
  1. Run the installation disk that came with your Mac in order to install X11. Since Apple does not offer X11 as a download for OS X 10.4 for Intel computers, you have to install from the DVD.
  2. Download the darcs binary and place it in /usr/local/bin
  3. Open terminal (Finder->Applications->Utilities->Terminal) and go to your home directory. To get the Factor source code, run:
    darcs get http://factorcode.org/repos
  4. To compile, move into the directory darcs created with cd repos/Factor and run the command make macosx-x86
  5. Download the boot image at http://factorcode.org/images/latest/boot.image.pentium4
  6. ./f boot.image.pentium4
  7. make macosx.app
  8. (Optional) Create a shell script to start Factor in your terminal window. I called mine go.
    #!/bin/sh
    DIR=~/repos/Factor/Factor.app/Contents
    $DIR/MacOS/Factor $@

    Save the file and run chmod a+x go
  9. (Optional) You can instead run open Factor.app and use Console.app to view runtime output.
To start the UI, run ./go or open Factor.app; run ./f to start Factor in the shell. If you run Factor in the terminal, I suggest using rlwrap, which adds a history buffer

Now you can code anything!

Thursday, November 16, 2006

Bootstrapping, reset-modified, and F8

Factor's module system has the ability to reload changed files from the UI, in the correct order, by calling reload-modules, which is bound to F8. When you download an image someone else created, the timestamps won't match up and when you reload the modules it will try to reload all of the Factor library. You don't want this.

There are several ways to fix the problem. First, you could generate a new image USE: image "x86" make-image and bootstrap again. Another way is to call reset-modified and then save the image. However, until you bootstrap from an image that you made yourself, will have the same problem.

If you make a module, don't forget to PROVIDE: contrib/my-module ; at the very least, or else F8 will not reload the changes you make after requireing it.

Wednesday, November 15, 2006

Using .factor-boot-rc and .factor-rc

Update: I changed the syntax to work with the new directory layout.

The two resource files Factor tries to run, if they exist, are .factor-boot-rc and .factor-rc, at bootstrap and run-time respectively. You have to create these files yourself and place them in your home directory, e.g. the directory returned by the home word.

If I used Textmate on a Mac, my .factor-boot-rc would look like this:
USING: modules namespaces ;
REQUIRES: libs/textmate ;

Here is the boot file for gvim in Windows XP:
USING: modules namespaces ;
REQUIRES: libs/vim ;
USE: vim
"c:\\program files\\vim\\vim70\\gvim.exe" vim-path set-global


Now for loading libraries. Some of the modules in contrib/ require that you load your library before running. Here are three examples of a .factor-rc that loads libraries.

Loading PostgreSQL on a Mac:
USING: alien ;
"postgresql" "/opt/local/lib/postgresql81/libpq.dylib" "cdecl" add-library

PostgreSQL and SQLite on Windows:
USING: alien ;
"postgresql" "C:\\Program Files\\PostgreSQL\\8.1\\bin\\libpq.dll"
"cdecl" add-library
"sqlite" "C:\\Program Files\\sqlite\\sqlite3.dll"
"cdecl" add-library
SQLite on Linux:
USING: alien ;
"sqlite" "libsqlite3.so" "cdecl" add-library

Tuesday, November 14, 2006

Compiling Factor on Windows

To compile Factor you will need:

  • MinGW
  • MSYS
  • darcs
  • freetype6.dll
  • zlib1.dll


  1. Download MinGW and install it.


  2. Download MSYS and install it. Make sure to tell the batch script about your MinGW install.


  3. Download the latest package version of darcs and unzip it to c:\darcsdir-w32. darcs 1.0.7 is broken on Windows, so you must also grab the 1.0.8 binary and copy it into your darcs install directory over the broken version.


  4. Set your path to include darcs. Run:
    start->Control Panel->System->Advanced->Environment Variables

    In System Variables, edit PATH, and append c:\darcsdir-w32 so that your path resembles:
    c:\Program Files\sqlite;c:\darcsdir-w32

    Start cmd.exe from the Run box and try out your darcs installation:
    C:\Documents and Settings\Administrator>darcs --version
    1.0.8 (release)


  5. (Optional) To push patches to a remote server, you will need to generate an RSA key with puttygen unless you already have a public key. Run puttygen (installed with the darcs bundle), click generate, and move the mouse around to make your key. Save both your public and private keys in c:\darcsdir-w32. Adding a passphrase is optional (I didn't do it).


  6. (Optional) On your remote Linux box running sshd, edit the file ~/.ssh/authorized_keys2 and paste your public key so that it looks something like this (but all on one line):

    ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAgMPa1vvlSi4ZrS7hrk/XJNwd9trM+TyvJjhkMzf3fhks
    n8TFin7dhrRfFhVdjka0ep0+RGJrUI/ja6nDyG2SvzG1ka5Ml/Nes2PtpHWcwK7fwU9gBUZNzlvXCkOk
    UKO7X7N65LnyCPjJDZa+8LPqrBeESjbVmstG4cw7uh0= doug@windows


  7. (Optional) So that darcs can find your key, start Regedit and add a key for Pageant:
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run
    Name: Pageant
    Data: c:\darcsdir-w32\pageant.exe c:\darcsdir-w32\private.ppk

    When you reboot or run the above command, pageant will find your key and you can begin using darcs to push patches to your Linux box.


  8. To get the Factor source, run this command wherever you want your local copy, perhaps from c:\:
    darcs get http://factorcode.org/repos


  9. Additionally, put these dlls in your Factor directory (repos/Factor/ on a default checkout):
    freetype6.dll
    zlib1.dll


  10. Start MSYS (the blue M link) and change to your Factor directory. Mine is at c:\repos\Factor, so the command is:
    cd /c/repos/Factor


  11. From here you should be able to
    make windows


  12. From cmd.exe, change to the Factor directory and bootstrap with an image (x86) from factorcode.org:
    f -i=boot.image.x86


Now you can run Factor!

f

I recommend bootstrapping and running Factor from cmd.exe because MSYS messes up stderr--you will not notice for a while if Factor has an internal error because there will be no output.

As a final note, the latest version from darcs is usually the best. If the boot image of Factor is stale and bootstrap fails, contact one of the devs in #concatenative on irc.freenode.org.

Happy Windows Factoring!

Update: Troubleshooting

Q: When starting Factor, why do I get the error 'alien-invoke-error-library "freetype", alien-invoke-error-symbol "FT_Init_FreeType"'?

A: You forgot to put freetype6.dll and zlib1.dll in the Factor directory. Download these and bootstrap again.

Q: When I bootstrap, why does Factor spew "*** Data GC (5 minor, 24 cards)" and error out with 'Word not found in current vocabulary search path delegate f no-word-name "\u000c"'?

A: On bootstrap you must pass the '-i' command line argument. The correct command line is:

./f -i=boot.image.x86

Factor 0.86 released

Factor 0.86 has been released. We took about a week to test it before releasing this time, and it promises to be the most stable release ever. Factor for Windows has a few improvements too, as a long-standing bug with the native i/o has been fixed, allowing Factor to be run in the console while still supporting native i/o with the command:
f -shell=tty
Furthermore, in 0.85, Windows Factor took several minutes to slurp through a 600 MB file; the new version takes five seconds and will speed up again when continuations compile.