Volume 1 Issue 3
Date: 13-Jul-2001
Issues: prev | next

Contents

• Lit-Word!
• Rebol Puzzles and Idioms
  • Compose is your friend
  • Compose Idioms
  • Block stripper
• Parsing financial statements - sort of.
• Doing the impossible - shell access without shell access
  • Watch out
  • Beware the Spammer
  • Of scripts and sneaky tricks
  • Anatomy of an email
  • Putting it all together
  • The downside
• Iterated Fields?? (an iterator intro and iter-field howto(-fakeit))
  • Iterator Intro
  • Iterated Fields (being sneaky)
• Get-Word!

Lit-Word!

This episode is a little smaller than the prior.  That's okay, though.  No fixed sized requirement.  Rebol is small, after all.

I have to thank whole heartedly our contributing authors that continue to support this periodical with their generosity and scripty- craftiness.  I don't mind racing home from the day job to spend a big chunk of my friday night getting this out because there are other fine people helping to make the Zine exist.

The life of a zine is like the life of a community, like the life of a language.  Shared learning and shared strength.

We program so we can convince the computer of our ideas, convince the computer to behave as we would like it to, so the computer can be more useful to us than it was when we first turned it on.  Rebol helps us express our intentions to the computer, allowing for a more fluent and natural conversation.  When we're not hacking, we can study the conversation.  So that's the Zine, and I'm totally rambling... enjoy!

Rebol Puzzles and Idioms

by Jeff Kreis

Compose is your friend

A year or so ago, I implemented COMPOSE for Rebol.  Carl and I discussed what COMPOSE should do for an afternoon and then I created it using REDUCE as my guide.

COMPOSE came about from witnessing an "alarming" (smirk) proliferation of code that was attempting to create dynamic Rebol expressions using string joins, lots of DO REJOIN ["string-word" word "string"] types of things. The problem with the "DO REJOIN" approach is that it isn't very efficient, for starters, it doesn't make use of Rebol's richer data model, and it is not very context friendly (when you DO a string, it is always interpreted in the global context).  The answer was COMPOSE which allows you to build blocks with selective evaluation.

Carl once described COMPOSE as the "opposite of PARSE"— and that's really meant in light of looking at how COMPOSE's dialect is sort of related to PARSE block— various datatypes with parens that represent actions to take, whereas in COMPOSE's case, the parens are those areas of selective interpretation and replacement, parsing through Rebol data, as opposed to constructing Rebol data.

Ever since COMPOSE came along, I've been one of its greatest fans.  I practically couldn't code with out it, and the rest of RT sometimes thinks I go overboard COMPOSING my scripts together. That may be true, but golly it's fun.

Compose Idioms

There are some very useful idioms that go along with COMPOSE which come in very handy.  First notice the following:

reduce [()]
== [unset]
 
compose [()]
== []

The curious discrepancy above is an important piece of COMPOSE's functionality that I am proud to say that I argued for.  Yep, it broke some consistency and meant that UNSET values could not result from COMPOSE paren expressions.  It was also inconsistent with REDUCE, as we see, and so it may be troublesome for those keeping track of language purity metrics.

So why was this done? Many of you might already understand why: the reason is it allows you to compose expressions which "evaluate away" which "evaluate to nothing".  The result of the evaluation is to vanish entirely, to leave to the universe, to cease to be there. Ha ha! Isn't that just hilarious?

Well, so what good is that?

First, it allows you to place print statements in parens within COMPOSE blocks, and they won't leave a trace of their evaluation in the resulting block:

compose [1 2 (print "the next one is three") 3]
== [1 2 3]

Cute, but in a broad sense, this "vanishing expression" capability comes in handy when when you are metaprogramming, writing programs which write programs.  There are times when you are building up a block of Rebol data that you will later use a function body or a layout, or a parse rule, etc.. where you may want to include some code under one condition, or NOT include anything under another condition.

Here's a simple example of using COMPOSE in this fashion in constructing a layout.

t1: form now/time
bz: [button "send zine" [send z@example.com read %rzine-1-03.txt]]
view layout compose [
    h1 "Time:"
    h2 t1
    (either now/time > 21:00 button [bz][])
]

If it's after 9:00 pm, the layout will offer a button to send the zine. If not, the EITHER evaluates the empty ELSE block, which results in...

type? do []
== unset!

Which means the paren expression evaluates to nothing at all.  It's funny because this means Rebol allows you to create code which evaluates to something that's not even a datatype, let alone a piece of memory— okay I'm foaming at the mouth. (-:

The previous example is admittedly hokey, but there are plenty more applications of COMPOSE's vanishing paren.

Block stripper

Another simple, but useful idiom of COMPOSE is it's block flattening capability.  When a paren expressions evaluate to a block, the contents are inserted into the block, stripping the outer layer of of brackets, so to speak:

a: [1 2]
compose [add (a)]
== [add 1 2]

See how that's useful? You can put arguments to a function after it and then DO the block.

One other handy use, though, is when you have a function that accepts various types of arguments but you want to only operate on the data as a block.  You might do something like:

something: func [x][
    if not block? :x [x: append copy [] :x]
    parse x [
        some [
            word!   (print "I've got a word!") |
            path!   (print "I've got a path!") |
            string! (print "A string, I say!")
        ]
    ]
]

That's okay, but doesn't actually work for everything, paths for instance, because appending a path to a block breaks up the words. Compose handles this universally:

something: func [x][
    x: compose [(x)]
 
    ...
]

Now X is guaranteed to be a block, no matter what you throw at it.  If you send in a block, it is just inserted into the block.  (Of course, for efficiency's sake you might check to see if it's not a block first before COMPOSING so you avoid the additional insertion.) Consequently, this idiom is a handy helper for dialecting.

So those are some of my favorite things about COMPOSE. I hope you find them useful.

-Jeff

Parsing financial statements - sort of.

by Brett Handley

Recently I've been doing some financial statement analysis for some companies I've been thinking about investing in. For the first analysis I actually typed the numbers in. Why? It was PDF! The pain of that experience made me think a little harder.

I found that the PDF reader would allow me to copy the text, but when I pasted it into Excel it was a mess. I could not use Excel's "text to columns" function because there were no delimiters and the text was not a fixed format.

Rebol to the rescue. I reasoned that since financial statements are relatively structured I should be able to write some parse rules that could separate out the numbers so that I could paste them into Excel in an easy way.

So I set about designing some logic for parse that would do this wonderful thing for me. What I wanted was to recognize the line description (eg. "Revenues") and then recognize the numbers that follow. The problem was how could I write some code to decide where the numbers started on the line?

With a sinking feeling I decided I couldn't answer this question. But then I realized I could turn the problem around - literally.  If I reversed the characters on the line, the problem would be transformed to where do the numbers run out? This is an easier question to answer, because I could write parse rules to recognize the numbers (even if they were reversed). When I ran out of numbers the rest of it must be the line description - well close enough.

Flushed with this success, I modified the script to recognize dates as well. This was important as one company seemed to make a habit of making share issues all the time. Now I could import the date of the issue as well.

The code below uses this approach to read through lines of text and separate the numbers from the text. The result is a block of blocks - the same structure that the array function in Rebol produces. I already had a function that takes this structure and creates a tab delimited string from it. Precisely the format that Excel loves to read into nice separate cells - excellent. I don't bother with converting the values to Rebol numbers because Excel will do that for me anyway.

So now my process is: (1) copy the text from the PDF (2) run the Rebol function "edit-clipboard-fs" which parses the clipboard data and writes the tab delimited format back onto the clipboard (3) paste into Excel. I still have a bit of manual work in Excel but the tedium is pretty much banished and I've saved my fingers from a few thousand keystrokes.

I've tried the script on a number of financial statements for Australian companies and it seems to work well. If you use it let me know how it goes - I'd love to know.

The script is below. If you want to do the last step - the tab separated bit you'll need the delimited-text.r script from my site or write your own. The location of this supporting script is:

http://www.codeconscious.com/Rebol/scripts/delimited-text.r

I hope I've shown how with a slightly different perspective, backwards even, some problems can be solved that otherwise might be a little too hard.

Cheers Brett.

Rebol [
    Title: "Financial Statement Tools"
    Purpose: "Separate numbers from runs of text - ie from PDF"
    Author: "Brett Handley"
    Date: 3-Jul-2001
]
 
if not :use-script [use-script: :do]   ; Use-script is my script manager
 
use-script %delimited-text.r
 
parse-financialstatement-line: function [
    input-string [string!]
][digit revordnum revnumber parsed-line][
    digit: charset [#"0" - #"9"]
    revordnum: [3 digit any [opt "," 1 3 digit] | 1 3 digit]
    revnumber: [")" revordnum "(" | revordnum]
    revdate: [4 digit " " revmonths " " 1 2 digit]
    revmonths: reduce [
        mold head reverse copy first system/locale/months
    ]
    foreach m next system/locale/months[
        insert tail revmonths [ | ]
        insert tail revmonths head reverse copy m
    ]
    RETURN either parse/all head reverse copy input-string [
        (parsed-line: copy [])
        any [
            mark: copy item [
            revnumber "&" to end |
            "-" opt " " |
            revdate |
            revnumber opt " "
            ]
            (
                while [all[item equal? first item " "]][remove item]
                insert parsed-line head reverse item
            )
        ]
        to end
        (insert parsed-line head reverse copy mark)
    ][parsed-line][none]
]
 
parse-financial-statement: function [
    input-text [string!]
][result][
    result: copy []
    foreach line (parse/all input-text "^/") [
        insert/only tail result parse-financialstatement-line line
    ]
    RETURN result
]
 
edit-clipboard-fs: does [
    write clipboard:// form-tab-delimited
        parse-financial-statement read clipboard://
]

Doing the impossible - shell access without shell access

By Chris Page

Watch out

In this issue I describe something that is, technically, a hack. It is far from pretty in many respects, but it is the only solution I could find to a problem with sending emails from some hosters. Feel free to try this out, but please make sure you know what you're doing first - if you use this on an insecure website it may be possible for 3rd parties to get up to no good with it.

Beware the Spammer

It's 3am and you sit back after a long hack run writing your Amazing Rebol Based Web Thingie - you have tested it works on your local webserver, it can even email you when you need to sort out a problem on your site. A few minutes later and it's running on your real website.. A revolution is born, or something.

But all is not well in dotcom land (and people are having problems using your website as well). Over the next day or so you get reports of internal server errors, but how can this be? Your scripts work fine on your test setup!

Consider the SMTP server, a lowly piece of unassuming software found throughout the Internet where people gather to send emails. Rebol can talk to your SMTP server, sending your mails to the four corners of the world with a single Rebol command. But in these latter days the evil, fercilicious and compunctious Spammer has forced many hosters to remove the SMTP facility from their servers, and scripts must talk directly to the mail transfer system using sendmail when they need to send emails.

Obviously, your hoster doesn't have an SMTP server running and your scripts need to send emails. Without an SMTP server Rebol's send command is useless. To make matters worse, you want to use /core - you have no shell access so you can't invoke sendmail directly either.

Of scripts and sneaky tricks

At first glance it is impossible to use /core in this situation: you can't use send without an SMTP server and you can't invoke sendmail without shell access. There is an alternative - it's not as pretty, it has some drawbacks, it will only work on Unix or Linux servers, but it does work.

The solution is only made possible by Rebol's ability to read from a URL as well as a file. If this was not possible, it really would be impossible to solve this problem and I'd be using Perl by now. The basic idea is to split the solution into three parts: a file containing the email you want to send, a Rebol script which creates this file and then reads from a cgi script and a script in another language (I use a normal shell script) to take the file, invoke sendmail and generate a response. But before we can cover either script, you need to understand a bit about sendmail..

sendmail is a program found on most unix and linux systems. It is used to send text passed in on its stdin to an address. The address can either be specified in the command line, or in the To: header field in the text provided on stdin. This latter mode of operation is of most interest to us. Combine this with file redirection and you can see that it is possible to save the complete email - recipient address and all - to a file and then pass it into sendmail in a script. The following does exactly that:

#!/bin/sh
 
echo "Content-Type: text/plain"
echo ""
echo ""
 
/usr/sbin/sendmail -t < /path/to/cgi-bin/tempdata.dat
 
echo "Email sent successfully"

Save this script as emailer.cgi and make sure it is readable and executable and you have the first part of the solution. This gives you the ability to send the contents of a file to an address given in the file, but how do you create the file?

Anatomy of an email

Emails have a fairly simple structure (provided you ignore the various Received: fields anyway...) The header is composed of <Name>: <value>; pairs, one pair per line. The end of the header is marked by two successive newlines. Assuming you have no attachments, the rest of the email is the body.

There are a large number of possible fields in the header; you would need to trawl through several RFCs to find them all, and those are just the "official" ones. Fortunately, we only need to bother about three of them in this situation: To, From and Subject.

Creating an email is simply a matter of writing out the To:, From: and Subject: lines, two newlines and then the email body. Creating the header is as simple as

header: reduce [ {To: } dest "^/"
                 {From: } senderaddr "^/"
                 {Subject: } subject "^/"
                 {^/^/}]

The whole email can then be created using one line

write %/path/to/cgi-bin/tempdata.dat rejoin [ header bodytext ]

Now you have the file ready to be sent by the emailer.cgi script you made above, but how do you tell it to send it? Thanks to the magic of Rebol all you need to do is:

result: read http://www.yourdomain.com/cgi-bin/emailer.cgi

Putting it all together

So far I have shown you how to create an email using Rebol and then use a unix script to send the email. Now I will put the pieces together into one function which can be called from any cgi script. This function is only a basic framework, you will probably want to add error checking and other features, but it will serve as a starting point for you:

; Set this to false if you do not have smtp facilities
useInternal: false
 
; Placed in the from: header field
fromAddress: chris@starforge.co.uk
 
sendemail: func [
    "Send an email using through the abstraction layer"
    recipient [email! block!]  "address(es) of the destination(s)"
    subject   [string!] "text to put in the subject line"
    bodytext  [string!] "text of the email"
    /setfrom            "override the From:"
    newfrom   [string!] "new From: address"
][
 
    ; use Rebol's send function if SMTP is available
    either useInternal [
        ; I have no idea if it's safe to play with the From: here...
        header: make system/standard/email [
            Subject: subject
        ]
        send/header recipient bodytext header
    ][
        ; Check that the recipient is valid
        if email? recipient [recipient: reduce [recipient]]
 
        ; Now we need to compile the To: line contents
        dest: make string! 1024
        comma: false
 
        foreach addr recipient [
            if email? addr [
                either comma [
                    append dest rejoin ["," addr]
                ][
                    append dest addr
                    comma: true
                ]
            ]
        ]
 
        ; This allows the sender to be changed on the fly. Useful for
        ; when a single script could generate emails that seem to come
        ; from more than one address in your domain.
        senderaddr: fromAddress
        if setfrom [
            senderaddr: newfrom
        ]
 
        ; Build the header as shown in the text.
        header: reduce [ {To: } dest "^/"
                         {From: } senderaddr "^/"
                         {Subject: } subject "^/"
                         {^/^/}]
 
        ; "send" the email. You may want to process result in some way
        write %/path/to/cgi-bin/tempdata.dat rejoin [ header bodytext ]
        result: read http://www.yourdomain.com/cgi-bin/emailer.cgi
 
        ; Get rid of the temporary file.
        delete %/path/to/cgi-bin/tempdata.dat
 
        return result
    ]
]

The downside

I mentioned drawbacks above so I suppose you better know about them now!

The most serious problem is concurrency - it is obviously impossible to use several instances of this script in parallel in the same directory as they share the same file. This is great scope for all manner of nasty problems inherent in this solution if you have a very heavy traffic site. However, in general this shouldn't cause too many problems.

The second problem is that this will only work on unix or linux systems. There may be an alternative method for Windows - I have very little Windows programming experience so I can't be much help in that direction.

Provided you can live with the rather hackish nature of the solution, and the problems don't worry you, these scripts form an almost drop-in replacement for the Rebol send command. I hope you find them as useful as I have!

- Chris chris at starforge dot co dot uk

Iterated Fields?? (an iterator intro and iter-field howto(-fakeit))

by Sterling Newton

Iterator Intro

Iterated faces are a unique and interesting feature of Rebol.  The LIST style is an iterated face.  To use it, you define a small layout block of elements which are to be iterated in some way.  You also fill in the body of a function which is called for each iteration.  This allows you to change virtually any attribute of the faces in the iterated layout.

The most common use is to iterate a list of items using a layout block like:

[across txt 100 txt 150]

with a supply block like:

[
    if count > length? my-data [face/show?: false exit]
    face/show?: true
    face/text: my-data/:count/:index
]

All this does is set the text in each text field to an item from the my-data list.  My-data would look like this:

[["left" "right"] ["one" "two"] ["some" "stuff"]]

The supply block is a function defined like this:

func [count index] [...]

where the body ([...]) is the block you supply.  Count is the iteration number and index is the element in the layout that is being iterated.  In the example above there are two txt elements in the layout so index would alternate between one and two to tell you which one you are working with.

This piece of code can be assembled like this:

my-data: [["left" "right"] ["one" "two"] ["some" "stuff"]]
view layout [
    list [across txt 100 txt 150] supply [
        if count > length? my-data [face/show?: false exit]
        face/show?: true
        face/text: my-data/:count/:index
    ]
]

Easy.  However, not all styles can be iterated.  Fields, for instance, cannot be iterated, and this is a problem for somebody wanting to make a list of entry fields.  You can, of course, make a bunch of faces and line them all up.  One of the strengths of iterators is that they can be highly dynamic.  With an iterator, you also only have one face object that is being reused so the memory impact can be lower.

Iterated Fields (being sneaky)

So how do you make iterated fields? The quick answer is that you don't, you can't.  But why should we let that stop us? The following code is excerpted from the web-calendar script located in the REB script library under the View topic.

The solution to iterated fields is to iterate text and have a special action associated with the text face that, when clicked, will move a field face to the right location to pretend to be the same face.

The code is a bit lengthy but the important sections in the layout have comments that look like "; === comment" so they should be easy to spot.  The day-plan layout defines a layout of two text elements, just as the example at the top did, dp-from and dp-info.  These are the time of day and the entry field (the one we're faking).  Both are text styles.

Read the action associated with the dp-info carefully.  This is where the magic happens.  It makes a lot of references to dp-area which is defined at the bottom of the layout.  Many fields in the dp-area face are set here:

* dp-area/ff is set to the iterated text face it is being attached to

* dp-area/day is the day the planner is on; specific to this example

* dp-area/time is the time of day we're working with; specific to this example

* dp-area/offset is set to the same location as the iterated text field; this moves the field face on top of the text face.  This is a rather complicated calculation because the iterated face does not report the offset of the face within the entire layout but only the offset within the iterated layout.  Special calculations must be made to determine exactly where in the whole layout the field face needs to go.  This code will vary between layouts and is tricky.

* dp-area/text is filled in with any text currently in the text face so it shows up in the field

Finally the dp-area is focused and the needed faces are shown, completing the illusion.  The call to "write-text" is mostly code that stored the entered information into the calendar database but it also sets the text entered into the field back into the text face that the field was on so that it appears in the list when the field is moved.

Rebol [
    Title: "Iterated Fields"
    Notes: "Excerpt from %calendar.r"
    Base-File: "REB Script Library/view/calendar.r"
    Purpose: {How to make iterated fields which are not otherwise possible.}
]
 
cal-data: either exists? %cal-sched.r [load %cal-sched.r] [copy []]
cur-day-data: copy []
 
get-ev-item: func [list count 'word] [
    select pick list count word
]
 
write-text: does [
    if dp-area/ff [
        either tmp: find cur-day-data dp-area/time [
            change/only next tmp compose [info (dp-area/text)]
        ] [
            append cur-day-data compose/deep [
                (dp-area/time) [info (dp-area/text)]
            ]
        ]
    ]
    if all [empty? dp-area/text tmp: find cur-day-data dp-area/time] [
        remove/part tmp 2
   ]
 
    either all [cur-day-data tmp: find/tail cal-data dp-day/data] [
        change/only tmp cur-day-data
    ] [
        if not empty? cur-day-data [
            append cal-data compose/deep [(dp-area/day) [(cur-day-data)]]]
    ]
]
 
view day-plan: layout [
    styles link-styles
    backdrop
    across
    dp-al: arrow left [show-day dp-day/data - 1]
    dp-day: h1 300x30 center font-size 18
    dp-ar: arrow right [show-day dp-day/data + 1] return
    dp-hol: text 300x16 black return
; === here is the iterated face definition ===
    m1: at dp-list: list 320x450 [
        space 0x0 across
        dp-from: txt black ivory 50x20
        dp-info: txt black ivory 270x20 [
; === this is the action performed when the iterated text is clicked on ===
; === here we move the magic field into place to fake the iterated field ===
            write-text
            dp-area/ff: dp-info
            dp-area/day: dp-day/data
            dp-area/time: dp-from/text
 
            dp-area/offset: dp-list/offset +
                (0x22 * (dp-info/data - dp-list/oset))
                + 50x0 + dp-list/edge/size
            dp-area/text: dp-info/text
 
            focus dp-area
            show [dp-area dp-list cal-face]
        ] font [colors: reduce [black black]] return
        box black 320x2
; === the supply block controls each iteration of the faces ===
    ] supply [
        count: count + dp-list/oset
        if count > 48 [face/show?: false exit]
        face/show?: true
        dp-from/text: 0:30 * (count - 1)
        either tmp: find cur-day-data dp-from/text [
            dp-info/text: get-ev-item tmp 2 info
        ] [dp-info/text: none]
 
        dp-info/data: count - 1
    ] with [oset: 16 lc: to-integer 450 / 22]
    at m1 + 320x0 dp-sld: slider 16x450 [
        dp-list/oset: to-integer (48 - dp-list/lc * dp-sld/data)
        do write-text
        hide dp-area
        show dp-list
    ]
    return
    pad 100 button coal "Close" [quit]
    at m1 + (0x1 * dp-list/size / 2)
    at m1
    dp-area: area (dp-info/size - 4x0) ivory ivory
    edge [size: none]
    with [show?: false ff: day: time: none]
]
dp-sld/redrag dp-list/lc / (48 - dp-list/lc)
dp-sld/data: 16 / (48 - dp-list/lc)

And that's it.  Also note that the slider hides the field face and calls write-text when sliding.  If you don't, the field will remain in place while the iterated list scrolls behind it.

Anything is possible to a sufficiently sneaky scripter.

Sterling

Get-Word!

The Zine says: while [on][forever [loop 10000000 [CONTIBUTE!]]]

Send your articles, covering any Rebol topic to rebolzine@yahoo.com. Articles are in make-doc format which is:

=== Title
--- subsection
+++ subsubsection

Indent code samples with 4 spaces — see make-doc.r for more formatting options.  Make-doc makes it much easier to assemble and publish to web.

Other comments are welcome to rebolzine@yahoo.com as well.

We're running the zine mail through yahoo because we're impartial to Rebol Technologies, which means you can send in articles of any type and they won't be subject to the harsh authoritarian business bottom line scrutiny of an incorporated entity.  Ha ha, just a joke, but seriously— censorship is very lame so feel free to push the envelope until someone complains.

Cheers!

halt