to insert a living object of type "net.antonioshome.netbeans.MyObject".

7. Scheme Editor Options

In the last section we ended up with a SchemeEditorKit that did not work properly. We were missing some key bindings and some syntax highlighting.

In this section we'll add some options to the Scheme editor. We'll set the key bindings, the caret blink rate, the fonts, the number of spaces per tab, a default indentation engine, lots of much more options and even an useless option (just to see how things work). We'll also have to persist those options, so that the user doesn't have to reenter them when he starts again the Scheme IDE.

And, you know what? We'll do all that with a single class, a specially cooked XML file and another entry in our sandwich layer. It's so easy that I'm ashamed I haven't looked into this stuff earlier! It would have saved me lots of time!

7.1. Options and BaseOptions

The NetBeans Platform includes an Options API that handles the so called System Options. These are properties that you set in a module so that the user can modify them easily through the Options Dialog. These options are automatically persisted on disk, so you don't have to worry about storing them yourself. This is something similar the Java Preferences API, but much more powerful.

As we will see, the Options API is so powerful that you can add your custom properties to a Java Bean you're defining, and have all those properties automagically persisted for you. The fact is that the NetBeans Platform uses Java Bean Introspection to seek for those properties.

In fact the NetBeans team has been always concerned about standards, and this introspection mechanism follows the Java Beans Specification. This basically means that you can use BeanInfos to customize how your properties are edited (this is, you could add custom property editors to edit them), or to specify descriptions for those options.

I won't be covering BeanInfos at the moment (I don't think I really need them now). If you think you need them then you should probably go take a look at the Java Editor Options package, the Java Editor Options has a powerful BeanInfo you can learn from.

The NetBeans Editor Module uses the Options API to handle editor-related settings, such as the number of spaces per tab, the caret blink rate, the abbreviations, the font size, the indentation engine to use, the color of the caret, the key bindings, different colors and many, many other settings. It does so using a special class, called "BaseOptions" that is included in the Editor Module.

Of course it's important to associate at least a default set of options to an editor, otherwise you won't be able to work with it (you won't have key bindings, for instance!). That's the problem we were facing in the previous section: our SchemeEditorKit had no options at all (not even default ones) so the NetBeans Platform was confused and didn't know how to properly render our SchemeEditorKit, resulting in an unusable editor.

7.2. Do we need a new module?

In previous sections I suggested that you should be doing different things in different modules. This makes your application much more flexible. But, should we place the options in a new module of its own? Or should we just add the single class inside the Scheme Editor Module?

I could build a separate module just for options (following my self advice of making tiny modules) but, why would anyone want a Scheme Editor Module with a SchemeEditorKit that does not work? Wouldn't it be better to include the Scheme editor options stuff inside the Scheme Editor Module, so that everything works correctly?

Yes. I think it's a better idea. So let's place all the stuff inside the Scheme Editor Module. But let's try to separate it from other classes. Let's store the new classes and files inside the "net.antonioshome.scheme.editor.options" package.

7.3. SchemeEditorOptions

So let's build a new package "net.antonioshome.scheme.editor.options" and let's create a new class, call it SchemeEditorOptions.java, and make it extend the BaseOptions in the NetBeans Editor Module.

As we said before, the "BaseOptions" class is a derivative of the "SystemOptions" class (in the "Options API") that contains predefined options for editors (key bindings, caret blink rate, and many others). Base Options is a Java Bean with properties for editors. It has setters and getters. There's a "setCaretBlinkRate(int)" method, for instance, that sets the caret blink rate. If we wanted to add another property to Since BaseOptions (and SchemeEditorOptions) are JavaBeans, we can add new properties of our liking. And these will be persisted to disk and restored when the Scheme IDE starts up. This is cool!

Just for fun, and to see how things work, I've added a "MyFunnyProperty" to the SchemeEditorOptions class. I don't know yet what sort of options I'll have to add to an Scheme editor, but I now know where I should be storing them!

The "SchemeEditorOptions.java" class, that we have to create, looks something like this:

package net.antonioshome.scheme.editor.options;

import org.netbeans.module.editor.options.BaseOptions;

/**
 * SchemeEditorOptions is responsible for handling Scheme options.
 * @author $Author: $ Antonio Vieiro (antonio@antonioshome.net)
 * @version $Revision: $
 */
public class SchemeEditorOptions extends org.netbeans.modules.editor.options.BaseOptions
{
  /** A name for these options */
  public static String SCHEME_PREFIX= "scheme"; // NOI18N ①

  /** A default constructor */
  public SchemeEditorOptions()
  {
    super( SchemeEditorKit.class, SCHEME_PREFIX ); ②
    // Just for fun: modify a property in the base class ;-)
    setCaretBlinkRate( 450 );
  }

  /** A funny property */
  private int myFunnyProperty; ③

  public void setMyFunnyProperty( int aValue ) ④
  {
    myFunnyProperty = aValue;
  }

  public int getMyFunnyProperty() ⑤
  {
    return myFunnyProperty;
  }

  /**
   * This is used to retrieve display names to
   *   easy internationalization of options.
   */
  protected String getString(String key) ⑥
  {
    try
    {
      return NbBundle.getMessage(SchemeEditorOptions.class, key);
    }
    catch (MissingResourceException e)
    {
      return super.getString(key);
    }
  }
}
    

Let's review what this code does.

① Options are stored on disk in a centralized place. In order to separate options from different modules, each module uses a specific name to differentiate its options from others. That way an editor cannot override by accident options from another editor.

② In the constructor we associate the SchemeEditorKit class with the prefix we have chosen for Scheme options. That way our SchemeEditorKit (that we created in the previous section) will be initialized with all those default options for key bindings, caret blink rates and the such. The "BaseOptions" constructor will associate default key bindings, default carent blink rates, default colors and fonts, and lots of other different options. So we need no extra coding, we have all this bunch of functionality with just a single line of code. Anyway you can set your preferred values of some properties there, too. I've modified the caret blink rate, for instance, to 450 milliseconds. Just for fun ;-).

③④⑤ You can add custom properties of your liking too. These will be persisted by the options subsystem automatically for you. The fact is that NetBeans performs some Java Beans Introspection for you, and detects all the properties you add to your options. These properties will then be persisted and presented to the user in the "Options Dialog", as we'll see later. Java Beans Introspection is a quite common task in NetBeans, and provides a quite powerful mechanism in different modules. Properties can be simple types or complex types of your liking. [3]

⑥ The "getString" method is responsible for returning localized strings for the user to see. The options API will invoke this method to retrieve display names of stuff to be presented to the user in the Advanced Options Dialog. What we do in this method is to use a NbBundle and to ease internationalization. The first argument to the "getMessage" method is a class used to locate a "Bundle.properties" file, which in turn is a resource bundle for internationalization. Note that the NbBundle class is located inside the utilities library, and that you'll have to add a dependency to that.

The Bundle.properties file, that you have to place near the "SchemeEditorOptions.java" file, contains just a single entry used to give a display name to the options. The code looks like this:

# Resource Bundle file for Scheme Editor Options.
OPTIONS_scheme=Scheme Editor Options
    

The key "OPTIONS_scheme" is fixed by the options module, and is used to identify the set of options for our Scheme Editor (note that the "scheme" part is due to the SCHEME_PREFIX above).

Of course we'll have to tell the NetBeans Platform that we have created this class, and that we want it to be executed so as to get those default options in our EditorKit. We have to register our custom SchemeEditorOptions within the platform. Any idea on how to do it?

No? Really? Think of it again...

Yep. We'll be registering it with... the XML Layer File! Of course! Let's add another layer of options to our sandwich.

7.4. Registering options with the XML Layer File

Registering options with the sandwich file is performed in two steps. First we'll add an entry pointing to an external XML file, and then we'll have to write that XML file in a special format, obeying a special DTD just for options.

This may seem complicated, but it's again a piece of cake once you know what you're doing. Just think of all the hassle we're avoiding (persistence, for instance). Isn't it worth the effort? I think so!

The entry to add to the XML Layer File is a "file" entry named "Settings.settings" (this name is fixed by the platform). The "file" entry has an "url" attribute pointing to our external XML file for options. Since URLs are relative to the location of the XML Layer File, we'll set the value of the "url" attribute to "*options/*SchemeEditorOptions.settings", and will create an empty XML file named "SchemeEditorOptions.settings" under the "net.antonioshome.scheme.editors.options" package.

So to summarize, the XML layer file has a new entry (located under the "Editors/text/x-scheme" folder) that looks like Figure 30, “Registering editor options in the XML Layer”.

The external file ("SchemeEditorOptions.settings") is much more involved than the XML Layer file, and looks like this:

<?xml version="1.0"?>
<!DOCTYPE settings PUBLIC "-//NetBeans//DTD Session settings 1.0//EN"
"http://www.netbeans.org/dtds/sessionsettings-1_0.dtd">
<settings version="1.0">
  <module name="net.antonioshome.scheme.editor"/>
  <instanceof class="java.io.Externalizable"/>
  <instanceof class="org.openide.util.SharedClassObject"/>
  <instanceof class="java.beans.beancontext.BeanContextProxy"/>
  <instanceof class="java.io.Serializable"/>
  <instanceof class="org.openide.options.SystemOption"/>
  <instanceof class="org.netbeans.modules.editor.options.OptionSupport"/>
  <instanceof class="org.netbeans.modules.editor.options.BaseOptions"/>
  <instance class="net.antonioshome.scheme.editor.options.SchemeEditorOptions"/>
</settings>

This "settings" file specifies settings for the module "net.antonioshome.scheme.editor" (see the "module" element at the beginning?). And then it specifies all the class hierarchy until our "net.antonioshome.scheme.editor.SchemeEditorOptions" file. Note that this is a Serializable class (because we will be persisting data from SchemeEditorOptions to disk!)

I assume[4] that specifying the class hierarchy explicitly is needed in order to correctly deserialize the class. After all deserializing the class will require loading classes from different modules and libraries (the libraries containing org.openide.util, and org.openide.options and the editor module and the Scheme editor module). Specifying them explicitly avoids the NetBeans Platform having to load too many modules: the platform will be able to load only those modules required to deserialize our options. This, of course, is better for performance!

Once we've updated our XML Layer file we, of course, have to wash our hands. This is, validate it. Right-click on the editor and select "XML Validate" in the popup menu

And we're almost ready to run our editor. You can run the SCHEMESUITE project now, but you won't yet be able to correctly edit Scheme files. We need another little step: we need to recognize the syntax of Scheme files.

7.5. Using a plain syntax

Since I don't want to make this section too long I think it's better to introduce some plain syntax, and dedicate the whole next section (still on the frying pan ;-)) to syntax highlighting.

A plain syntax just recognices words. And words are printed on black on white. Just like plain text files. That's why it's called plain, after all!.

Adding a plain syntax is just a piece of cake. You'll have to add the following method to the SchemeEditorKit:

public Syntax createSyntax(Document document)
{
  return new PlainSyntax();
}
      

And, of course, add a dependency to the module containing the PlainSyntax class using the steps we talked about earlier at Section 6.4, “Adding dependencies to other modules”. PlainSyntax is useful to have a quick and dirty syntax for editors, or for editors that do not need to have syntax highlighting. Indeed a handy class.

7.6. Summary

In this section we had a snack with NetBeans options for editors, we introduced the XML file for settings and we added a plain syntax for Scheme files. Our editor should be working now, and we should be able to open and save Scheme files.

In the next section we'll add some color to those ugly black-on-white Scheme files, adding special colors for Scheme library procedures and constructs. That's going to be fun all the way long. Keep tuned for some colorful Scheme!

[3] If you have a complex type you'll have to provide a custom PropertyEditor to modify the property (and use a BeanInfo class).

[4] And this is just my assumption, I may be completely wrong on this

8. Colorful editors: The Old, The Good and The Young

So on last section we ended up with a working black-on-white editor. And we wanted to add some color to our Scheme code. In this section we'll take a look at he current syntax highlighting stuff in the NetBeans Platform. While doing so we'll learn about Editor Settings and module installers. Since this is a complex issue I'm afraid I'll split the stuff in two different entries.

8.1. Adding color to editors: The Old, The Good and The Young!

As far as I have seen there are at least three ways to add color to your editors in the NetBeans platform: there is the ugly, old way (the one I will be following here), the standard way (the one explained in the excellent NetBeans Syntax Highlighting Tutorial) and the young one (the one that is currently on the works).

So why all these methods? Well, these different ways to do the same thing show how NetBeans has evolved. The old method is probably not a proper way to do things nowadays (but it has its advantages, as we'll see). The "good way" is a better approach, and it's the current way to add syntax highlighting. And the "young" way is indeed young, still in its infancy, and the NetBeans Team is still polishing it.

The good news is that all these three methods are backwards compatible, so "the old way" to add coloring to text is a subset of "the good way". And "the young" is in turn a superset of the previous one.

Let's give them a quick review, let's understand how they relate to each other and see what the pros and cons of each one are.

The ugly, old way

The old way to add color to your editors is the one I will be following here. I will be doing so because this mechanism requires only the NetBeans Editor Library and, as a consequence, the method allows you to use the same technique in standalone editors. And having a standalone editor with syntax highlighting is something really interesting for me, so I can add it to my Swing applications!!

As we will see, this old way to do things is just a subset of the method described in the NetBeans Syntax Highlighting Module Tutorial ("The Good"). We can always move to "The Good" mechanism by adding some extra Java and XML files.

The good

The good way to add color to your editors in the NetBeans Platform is summarized in the NetBeans Syntax Highligthing Module Tutorial. This is basically the same as the old way, but requires some additional classes and files.

This good way is an evolution over the previous one. The main benefit of this method is that colors can be easily edited by the user using the new Options Dialog. This new Options Dialog presents some sample text to the user (sample text you can define, of course), and allows easy selection of colors for the differents parts of the text. If you are running the NetBeans IDE and choose the "Tools/Options" menu item you'll see what I mean. You have special entries for assigning colors to the editors. This is just cool.

Furthermore, "the good" way allows you to use "color schemes" that represent a combination of colors. NetBeans includes two color schemes: the NetBeans default color scheme and the "City Lights" color scheme (that uses a black background for the editor). The old mechanism does not allow these features, and makes it more difficult for the user to change colors easily.

As you can see the "good mechanism" is much more advanced than the previous one. As a consequence it depends on some more modules, and makes it more difficult to use it on a standalone editor in a Swing application.

So I will follow the old way, and I may add an extra module to "upgrade" the old mechanism into the new one.

The young

The new way to assign colors to your editors is to use some sort of grammar files, using the Lexer Module. So you can add colors to your favourite languages much more easily. This module has been added to NetBeans CVS Trunk as of 2006-10-04, after passing an exhaustive API review (remember I told you that NetBeans Team was very strict about APIs?). Although this mechanism has been added recently to NetBeans CVS Trunk, the fact is that it has been around for quite a long time. You can see examples (using an old version of the Lexer Module) at the Coyote Project.

The lexer module makes things easier for you, because you can define syntax coloring more easily using text files. Furthermore, it makes it easier to work with files that require different syntaxes (such as JSP files, that require an HTML syntax and a Java syntax). It also makes things even faster, and requires less parsing and overhead than the previous mechanisms.

The lexer module API is still considered unstable, so you are still on time to suggest changes. It's a great time to try it out, experiment with it and give feedback to the NetBeans Team.

So now that we have a high-level overview of these mechanisms, let's go take another high-level overview of what we have to do. Let's review the main classes needed to add coloring to our Scheme editor.

8.2. The old way: overview

The class diagram at Figure 34, “Classes required to colorify our Scheme editor” shows the classes that we need. Note: if you want to see source code for these, I recommend you visiting the NetBeans Syntax Highlighting Tutorial, many links below point to that tutorial.

SchemeTokenContext extends TokenContext

TokenContext objects are responsible for defining the types of tokens that appear in the text. For instance, when parsing Scheme documents I think I'll need the following tokens: whitespace, scheme standard procedures, scheme constructs, constants (strings and numbers), variables and comments (and probably some others).

Those tokens are represented by special Java classes called "TokenIDs". These "TokenIDs" contain an integer (a sort of primary key) and a display name (that is also internationalized). Display names use a prefix (such as "scheme-" or "mf-" in the tutorial) to distinguish between different languages, avoiding namespace polution.

You can see some source code from the NetBeans Syntax Highlighting Tutorial. As you can see this is an easy class: you just define what types of tokens you want in your language and create BaseTokenIDs for those.

SchemeTokenColoringInitializer extends TokenColoringInitializer

TokenColoringInitializers are responsible for assigning colors to tokens (to TokenIDs). Both for on-screen visualization and for printing output as well.

In the NetBeans Syntax Highlighting Tutorial you can see this as an inner class inside the ManifestSettingsInitializer. Source code is here.

SchemeSettingsInitializer extends SettingsInitializer

SettingsInitializers are responsible for populating options with appropriate settings. These are useful if you have complex set of options, such as those defining fonts and colors.

Building a custom SettingsInitializer is not that difficult. You can use the ManifestSettingsInitializer and use it as a template for your own stuff. That's what I've done with my SchemeSettingsInitializer.

SchemeSyntax extends Syntax

Syntax objects are responsible for recognizing text as the user types it, returning appropriate TokenIDs (those defined in the associated TokenContext).

Syntax objects are possibly the most difficult thing to build when adding colors to your editor, because there are very tough non-functional requirements for them: they must run fast and generate as little garbage as possible. Because they are run whenever the user modifies the document.

The "old way" and the "good way" to do things use state machines to keep track of the token the user is typing. That's efficient (both in time and space) and is probably the best solution around. The problem is that building them is somewhat difficult.

The "new way" to do things does this operation automatically, using language description files that are easier to build, so you don't have to build the Syntax object by hand, using state machines. I may consider using the new way as soon as it stabilizes.

You declare your custom Syntax object in the Editor Kits. So, for instance, in our SchemeEditorKit we have to return our custom SchemeSyntax, like this:

public class SchemeEditorKit extends NbEditorKit
{
  public SchemeEditorKit()
  {
    super();
  }
  public String getContentType()
  {
    return "text/x-scheme"; // NOI18N
  }
  public Syntax createSyntax(Document document)
  {
    return new SchemeSyntax();
  }
}

So, to summarize: adding colors to our Scheme text files works like this: you have to build your custom Syntax object, that translates text to TokenIDs (defined in your TokenContext). Those TokenIDs are in turn translated to colors using the TokenColoringInitializer, which is in turn inserted into the options dialog using a SettingsInitializer.

Not very difficult, but, who invokes the "SettingsInitializer"? We'd like it to be executed when our editor module is initialized, right?

But, how do we execute code when a module is initialized?

That's such an important thing that I've moved that to the next section below.

8.3. Executing Java code on module initialization

As we said before, executing code on module initialization is a problem, because code execution is time consuming, and we don't want all the modules to consume time on startup. That would make the whole application startup slow.

But in this case we'll have to add a module installer that inserts our custom editor settings (our custom colors) into the editor options.

But, wait a moment! Why can't we do this with the XML Layer file? Isn't that supposed to be used to register stuff into the NetBeans Platform based application? Why shall we do this by hand and not by adding some more tomato layers to our sandwich?

I am not sure about this, but I assume we have to do things by hand because the editor Settings stuff belongs to the NetBeans Editor Library, and this is a library that can be used outside NetBeans, in standalone Java applications, where no XML Layers exist. And that's why we need a module installer to populate our coloring settings inside our options.

8.3.1. Creating a Module installer

Each NetBeans module may contain a "module installer". This a plain Java class that extends ModuleInstall[5].

The NetBeans IDE contains a wizard that eases creating these Module Installers. The wizard creates this custom class and registers it in the XML Layer file automatically for you.

To start this wizard you choose "File/New..." in the main menu and then you select the "Module Installer" option under the "NetBeans Module Development" section, as explained graphically at Figure 35, “Creating a Module Installer, step 1”

After that you just click "Next" and "Finish" (as you can see in Figure 36, “Creating a Module Installer, step 2”).

Once created you can override different methods of the "ModuleInstall" super class: restored(), uninstalled(), close(), closing() and validate(). Let's take a quick look at these methods.

restored()

The "restored()" method is the one you'll probably use more often. The NetBeans Platform will invoke the "restored()" method during application startup. It is important to note that this method sums up into application startup time[6], so you must do things as quickly as possible. It is important to note as well that the whole application has not been yet initialized, so you are limited on what you can do. This method is the one we'll be using to inject our editor color settings.

uninstalled()

The "uninstalled()" method will be invoked by the NetBeans Platform when your module is deactivated (removed) from the running application. We won't be using this, because all the settings we're adding are not permanent.

closing()

The "closing()" method is invoked on your custom ModuleInstall when NetBeans is about to close. This gives you a chance to check if everything is ok to close. You can stop the close operation by returning "false" from this method. This is a good place to check if all your stuff is saved, and things like that (note that the NetBeans Platform takes care of unsaved buffers in editors for you, though).

close()

The "close()" method is invoked on your custom ModuleInstall when the application is being closed because all modules have returned "true" from the "closing()" method.

validate()

Finally the "validate()" method is also invoked during startup (so the application may not be initialized yet). This method allows you to verify that everything is ready for you to run. If you develop commercial applications on top of the NetBeans Platform you can use this method to check for licenses, for instance.

And that's basically it! Inserting our SchemeSettingsInitializer (which in turn populates colors with our SchemeTokenColoringInitializer) may be done in the restored() method, like this:

  package net.antonioshome.scheme.editor;

  import net.antonioshome.scheme.editor.syntax.SchemeSettingsInitializer;
  import org.netbeans.editor.Settings;
  import org.openide.modules.ModuleInstall;

  public class Installer extends ModuleInstall
  {
    public void restored()
    {
      Settings.addInitializer( new SchemeSettingsInitializer() );
    }
  }

Before finishing this section let me remind you that the good way to add colors to your NetBeans editors is explained in the NetBeans Syntax Highlighting Tutorial. Following that tutorial is probably your best option: remember that (since the old way is a subset of it) you'll also be doing things "the old way".

On the next section (on the works) I'll take a look at the "Syntax" object, and explain how I've defined it for the Scheme language.

[5] A plain Java class that can be instantiated like a JavaBean, this is, that has a public void constructor

[6] I think the NetBeans Team is looking at multithreaded application startup, so things may be faster in the future

9. Adding Syntax Support

As we saw in the previous section, there are different ways to add syntax highlighting to NetBeans. The "old" and "good" ways to do things required a Syntax object, responsible for tokenizing text. The young way (the lexer module) eases building such objects (and provides some other cool features).

In this section we'll take a look at Syntax objects. When the lexer module is stabilized I'll add a new section here about it.

9.1. Syntax - non functional requirements

The Syntax class has been around since 1999. At that time Java 2 had just been released. At that time microprocessors were somewhat slow (remember those Pentium II?), and so were the Java Virtual Machines (and Swing!). In 1999 the performance of Java based applications was a real issue.

Existing garbage collectors were not as fast as they are nowadays either[7], so memory consumption was also a concern.

So the design of the Syntax highlighting support in NetBeans was influenced by these tight constraints: it should be both time and space efficient. It should run fast, and it should consume as little memory as possible. As a penalty, defining a custom Syntax object is a little bit complex, and requires careful hand-coding.

This section is about building custom Syntax objects, that transform text into tokens.[8]

9.2. Building a custom syntax object

Let's start by creating our custom Syntax class, responsible for transforming text into TokenIDs.

9.2.1. SchemeSyntax, step I: extends Syntax

The very first step to build a custom Syntax is (apart of having ready all the classes we talked about in the previous section, such as SettingsInitializers and TokenContexts) to extend the Syntax class, which is a little bit tricky.

Extending the Syntax class requires creating a public void constructor that updates a superclass variable (and this is the tricky part), and overriding the protected "parseToken()" method that translates text into tokens. The basic skeleton for building our custom Syntax object looks then like this:

public class SchemeSyntax extends Syntax
{
  public SchemeSyntax()
  {
    // The tricky part!
    tokenContextPath = SchemeTokenContext.contextPath;
  }
  // @Override
  protected TokenID parseToken()
  {
    // Some magic here!
    return null;
  }
}

9.2.2. SchemeSyntax, step II: building a parseToken

The editor module infrastructure will make calls to our overriden "parseToken()" method for us, automatically. And it will be making calls to that method until you analyze the whole text buffer.

And what's the text buffer we have to scan, you ask? Well, that's stored in an internal protected variable in the Syntax class. In fact we have some other juicy variables there. Let's take a look at what these variables mean:

buffer

The buffer variable is the array of chars containing the text to be parsed. Note that we don't have to scan the whole array, but just a part of it. The editor infrastructure is responsible for telling us what part of the buffer we have to scan (and it performs some magic there so that we don't have to scan the whole buffer all the time!).

The part of the buffer we have to scan is stored as a set of offset and length integer variables. Let's review those:

offset (and stopOffset)

The offset variable is an integer containing the current offset inside the buffer. So "buffer[offset]" is always the character we're currently looking at.

The stopOffset variable is another integer containing the maximum offset inside the buffer that we have to check. This is so because we don't always have to parse the whole file, sometimes we only have to check a part of it. The editor module keeps track of what parts of the file may have changed, and asks us to analyze the text between offset and stopOffset.

We are responsible for advancing "offset" until we reach "stopOffset". That's why the "parseToken()" usually has a loop like this:

protected TokenID parseToken()
{
  while( offset < stopOffset )
  {
    char currentChar = buffer[offset];
    if ( this is a whole token )
      return anAppropriateTokenID;
    else
      offset ++;
  }
  return null;
}

lastBuffer

The lastBuffer variable is a boolean indicating if this is our last chance to process the buffer. If set to true then we should return whatever token we're currently parsing, even though we haven't finished yet.

Detecting the end of the buffer is important, and this is usually done after the loop above, like this:

protected TokenID parseToken()
{
  while( offset < stopOffset )
  {
    char currentChar = buffer[offset];
    if ( this is a whole token )
      return anAppropriateTokenID;
    else
      offset ++;
  }
  if ( lastBuffer )
  {
    // We may return an appropriate token
    // and reset internal state
  }
  return null;
}

Most NetBeans custom Syntax objects use the structure above to perform the lexical analysis of text. You may want to go take a look at the SQLSyntax, the JavaSyntax or at the NetBeans Syntax Highlighting Tutorial Syntax Example.

state

The state variable is an integer containing the current state of our syntax object.

States allow you to react in different ways to the same character. Of course! Imagine you're analyzing a piece of Java code, you'll react differently to the slash "/" character if it's the first time you see it (like in " 1 / 2") or if you're seeing it immediately after a previous slash character (like in " // Comment ")!.

So Syntax objects work effectively as state machines: for each iteration on the loop (for each character in the text buffer) the token to be returned depends on the character itself and your current internal state. Figure Figure 38, “Sample states in JSON syntax” explains graphically how internal states allow you to know what part of a syntax branch you are analyzing.

tokenOffset

The tokenOffset variable is an integer that represents the start of the current token. Whenever you return a token, the Syntax object keeps track of the offset, and stores it in the "tokenOffset" variable. This is an interesting variable, because you can know the whole text you're currently analyzing by using something like:

String currentToken = new String( buffer, tokenOffset, offset - tokenOffset );

This is what the SQLSyntax, does: it creates temporary Strings with the token currently under analysis, and then it compares that String against a list of keywords. This is an easy way to colorize your text: you basically create three categories of tokens: punctuation, whitespace and text. And then you compare the text with a predefined set of keywords.

9.2.3. Syntax examples

So that's it. Building a custom Syntax object is not that difficult after all: it requires some some attention.

If you want to see real-life examples of custom Syntax objects you may want to take a look at NetBeans' SQLSyntax, the JavaSyntax or at the NetBeans Syntax Highlighting Tutorial Syntax Example.

But, hold down just a moment, there're some juicy tricks you may use to ease building custom syntax objects!!

9.3. Syntax tricks

Since building custom Syntax objects seems somewhat involved let's see some basic tricks to ease things.

9.3.1. Using regular expressions

If you take a detailed look at SQLSyntax you'll see a helper method that recognizes keywords. This helper method creates a new String from the text buffer (using "new String") and that then compares that String against a constant list of SQL keywords.

Of course they're creating new things, so this is not as space-efficient at the JavaSyntax, for instance. Anyway the current garbage-collection technology in recent JVMs has improved so much that creating these short-lived objects is very, very efficient nowadays.

In fact you can even create more interesting short-lived objects: you can create "Matchers" and use regular expressions to match keywords. As you probably know, regular expressions are strongly related to finite state authomata, and so may help us build our custom Syntax state machine. You could, for instance, define the following Pattern that is used to detect Scheme constructs:

private static Pattern CONSTRUCT = Pattern.compile("and|begin|case|cond" +
"|define|define-syntax|delay|do|else|if|lambda|let|let\*|letrec" +
"|letrec-syntax|let-syntax|or|quasiquote|set!|syntax-rules" );

And then use a helper function to detect if a part of the buffer is a Scheme construct or not, like this:

public boolean isConstruct()
{
  SegmentCharSequence chars = 
    new SegmentCharSequence( buffer, tokenOffset, offset - tokenOffset );
  return CONSTRUCT.matcher( chars ).matches();
}

And we would only be creating a lightweight "SegmentCharSequence"[9] and a Matcher object per iteration. Again note we're trading space-efficiency by ease of development. And that seems fair to me.

Using regular expressions is most handy if you want to parse complex tokens, such as Scheme numbers. Scheme number syntax is somewhat involved, because you have rational numbers, complex numbers, and exact numbers, for instance. Choosing your regular expressions wisely may boost your productivity when building custom syntax objects.

Remember, again, that if you want to do things really efficiently then you may want to experiment with the lexer module: the new way to do syntax highlighting on top of NetBeans.

9.3.2. Syntax unit testing

The second trick I suggest for building custom Syntax objects is to fully exploit unit testing. Building unit tests in the NetBeans IDE is just a piece of cake (just right-click on your custom Syntax.java and choose Tools/Create unit tests), and building unit-tests for your syntax is not that difficult either.

The NetBeans Syntax Highlighting Module Tutorial contains a good example for building your Syntax unit tests.

9.4. Multi syntaxes

Java Server Pages can contain HTML code and/or Java code. Web pages contain both HTML and JavaScript code. These type of files, that include different languages, require the use of a MultiSyntax object. I won't be covering them here, though.

One of the objectives of the lexer module (the "young" way to do things) is to ease building syntax support for text files that (as in the JSP case) require different syntaxes. So if you need one of these I suggest you experimenting with the lexer module.

During the last weeks there has been some interesting movement in this new way of doing things. Geertjan has been talking about this new mechanism in his blog. If you find that building syntaxes "the old way" is too complex for you then you may want to explore this new, fancy API. I may cover it in the future, when it is considered more stable.

Meanwhile I will keep on using the old API. I may be interested in building standalone editors that use the NetBeans Editor API, like this small Scheme Editor Pane that I can use in small Swing Applications. I'll talk about those standalone editors either in my blog, or in a new section. Keep tuned for that.

[7] the guys at Sun Labs have been quite busy during these last 7 years!

[8] Note that Syntax objects work basically as incremental lexers, since they detect changes in the document and process only the part of the buffer that is affected. Normal lexers (such as JFlex) scan the whole buffer instead.

[9] SegmentCharSequence is both a CharSequence and a Segment.

public class SegmentCharSequence extends Segment
  implements CharSequence
{
  /**
   * Creates a new instance of SegmentCharSequence
   */
  public SegmentCharSequence( char[] buffer, int startOffset, int length )
  {
    super( buffer, startOffset, length );
  }

  public char charAt(int index)
  {
    if ( index < 0 || index > count )
      throw new StringIndexOutOfBoundsException( index );
    return array[ offset + index ];
  }

  public CharSequence subSequence(int start, int end)
  {
    if ( start < 0 || end > count || start > end )
      throw new StringIndexOutOfBoundsException( " start " + start + " end " + end );
      
    return new SegmentCharSequence( array, offset + start, end-start );
  }

  public int length()
  {
    return count;
  }
}

Note that Segments will be CharSequences in Java 6

10. Wrapping Scheme into our IDE (Library Wrappers)

So we have a working editor for Scheme files now. Now we want to be able to interpret and run that Scheme source code.

To do that we need to embed our favourite Scheme interpreter into our Scheme IDE. There are different Scheme interpreters that run on top of the JVM (JScheme, Kawa, Jaja, Skij and SISC, and possibly many others). I'll talk about the differences between these in a future entry (there's an outdated comparison in this MIT Thesis by Brian D. Carlstrom, with a description on how to embed Scheme into Java, in case you're interested in these topics).

I must admit I haven't used many of those: SISC is good enough for my needs (furthermore, I want to explore SISCweb in the future, to give web continuations a go). So let's see how to embed SISC into our Scheme IDE. Later on I may want to add different Scheme runtimes into the IDE, but let's keep it small at the moment.

To embed an external library into a NetBeans based application we must understand "Library Wrappers". So let's take a look at them first.

10.1. Library Wrappers

Library Wrappers are NetBeans modules that contain an external library.

Since Library Wrappers are NetBeans modules, you have a well defined Public API. As we'll see in a minute, when you define a Library Wrapper, the NetBeans IDE automatically builds a Public API that contains all classes in the library.

And you have versions, too. Your own versions, that you define when building your Library Wrapper. These "Library Wrapper versions" are independent of the library being wrapped (because these versions correspond to the wrapper, and not to the library).

Finally note that Library Wrappers are designed to contain libraries, and not projects. What I mean is that a library is something that rarely changes (projects change more often). That's why Library Wrappers do not work well if you change the library after building them. We'll also see later on how to handle these cases.

10.2. Getting started: wrapping SISC

So let's get started with Library Wrappers. Let's build a Library Wrapper around SISC.

10.2.1. Download SISC

Oh, well. Before building a Library Wrapper we need a library! So let's download SISC first, and extract the contents somewhere in our hard disk.

10.2.2. Create a new Library Wrapper

You build a Library Wrapper Module by selecting "File/New..." in the menu, and then selecting the "NetBeans Plug-In Modules" category and the "Library Wrapper Module Project" option (see Figure 40, “File/New... Library Wrapper”).

The next step asks us to select the library we want to wrap. Note that you can select multiple jar files and not just only one (I was confused by this when I first built a Library Wrapper), as can be seen Figure 41, “Selecting the jar files to wrap”).

You can also specify a license for the library. The SISC Scheme interpreter is released both under the Mozilla Public License 1.1 (quite similar to the CDDL) and under the GPL 2.0 license. SISC includes its license in the "COPYING" file, so just click the "Browse..." button and select it.

Remember that you can select multiple jar files when building a library wrapper module.

The third step in the wizard allows you to enter the project name and location. Remember to add the module to the Scheme module suite we're building.

The fourth and last step in the wizard asks for a code base name for the module. As usual this is a unique name for the module. I follow a convention similar to package names for this name. I enter my domain name (net.antonioshome) and then a name for the application I'm building (scheme) and finally a name for the module (sisc), so I have "net.antonioshome.scheme.sisc" as my code base name.

See Figure 42, “Setting the codename” for details.

10.2.3. Library Public APIs

Now that we have built our Library Wrapper module let's go take a look at it. The very first you may notice is that NetBeans has automagically generated a Public API for our library. You can take a look at this public API by right-clicking on the Library Module and selecting the "Properties" menu item in the popup menu. If you then select the "API Versioning" node you should see something similar to Figure 43, “The Public API of our library”.

The fact is that, NetBeans scans all the jar files in the library and detects the Java packages there, and then marks all of them as public.

This opens new possibilities to us. We can select a small set of packages in the library, and mark only those as public, so that the rest of modules in our application can only interact with those packages.

We could, for instance, mark the "sisc.boot" package as private, so that no other modules could access the classes under "sisc.boot".

This feature has deep consequences in the maintainability of our applications, allowing us to manage dependencies between libraries and modules in a much more powerful way. That's one of the main reasons why I like the NetBeans Platform as a way to build complex applications: you can impose big restrictions on dependencies, which is of main importance when building applications with different development teams, for instance.

So, to summarize: Library Wrappers allow us to define which parts of a library are public, giving us a powerful tool to handle dependencies in our applications.

10.2.4. Managing change

So what happens if we want to change (or update) the jar files in our library? The NetBeans IDE doesn't currently support this operation (well, after all libraries are not expected to change, right?), so we'll have to do it ourselves (or wait until this issue is solved).

I can think of different ways to have our jar files updated. Let's see them:

The first way is, of course, to delete the Library Wrapper module and rebuild it with the same code base name. This is somewhat tedious, but it's probably the safest way to do it.

If you haven't changed the packages in the jar files you can try to rewrite the old ones. When you build the Library Wrapper, NetBeans stores the jar files under the "releases/modules/ext" directory (see Figure 44, “Jar file location”). Of course you can modify the "build.xml" file to have the files automatically overwritten for you.

Finally you may want to use the versions in your Library Wrapper and create different LibraryWrappers, with different versions each, and make a dependency with your preferred version.

10.2.5. Beloved warnings!

After creating the Library Wrapper for SISC you may want, of course, to build it.

And if you do so you'll notice... warnings! (see Figure 45, “Beloved warnings!”)

How is that possible? How is it possible that we have warnings after just creating our Library Wrapper?

Well, the fact is that NetBeans is doing quite interesting things when building our module. It's "verifying the class linkage". What this actually means is that NetBeans scans the class files of our library and looks for unresolved dependencies with other libraries. This is cool because it allows you to quickly see what other libraries you may need.

In our case, NetBeans is informing us that the SISC libraries (well, exactly the sisc.tests.TestR5RS, sisc.tests.UtilTest and sisc.tests.AllTests) require the JUnit library.

We know that those libraries are only required to build SISC unit tests (but are not required to run SISC), so we may dismiss those warnings.

I think this feature of Library Wrappers is very interesting. The fact is that I find it difficult to handle dependencies myself. You know, people tend to use lots of libraries in their software, and it's difficult to see if you have all the required jar files or not. Having NetBeans make that check for you saves lots of time!

NetBeans scans the class files seeking for unresolved dependencies. This allows you to quickly see if you need some extra libraries or not, and helps you solve those problems quickly.

10.2.6. Native librariers

I haven't yet added native libraries to a NetBeans based application, but it seems this is somewhat tricky. I will have to investigate this thread in the future (well, just in case I ever use a native library! ;-) ).

If you're delivering native libraries with JNLP you may be interested in reading this interesting blog entry by Alan Burlinson about using NetBeans, native libraries, JNLP and the new compression format (pack200) with an Apache server (I also have another entry on using JNLP and pack200 with Apache).

10.2.7. Further reading

The NetBeans Developer FAQ has an interesting entry explaining when to use Library Wrapper (you can also bundle your jars inside your module). This is a good read if you're interested in using Library Wrappers in your applications.

In the next section we'll start invoking SISC and running our first Scheme code. Keep tuned for that, we'll see how to asynchronously invoke things... without SwingWorkers!

11. Invoking Scheme I (creating a Public API)

In the last section we added the SISC Scheme interpreter libraries to our NetBeans Platform based Scheme IDE.

In this section we'll see how to invoke the SISC interpreter. We will define a generic interpreter interface (publishing our first API), and in the next section we'll see how to actually implement it. By doing so we'll learn about NetBeans Platform's Lookup system (the best thing after sliced bread), NetBeans threading (Oh, my! I should have learned about this before!) and the Output Window (the Console).

And, of course, we'll be having fun all the way through. Let's go!

Let's get started.

11.1. Yet another module? No, sir: it's TWO more modules

I think I'll add another module into the Scheme Module Suite. I like making modules responsable for very specific things. We have now a module for recognizing Scheme file types, another module that adds editing capabilities and another module that wraps the SISC library. This new module will be responsible for evaluating Scheme expressions using the SISC Scheme Intepreter (using the SISC Library Wrapper).

But, hold down just a minute. What about if we want to change the current interpreter and use, say, JScheme or guile instead of SISC? Shall we go and change everything? That wouldn't be too modular, would it?

If we were using Java then this could be done easily: we would define a generic interpreter interface and, aftewards, define different implementations, allowing people to change implementations easily without the rest of classes noticing, using a Abstract Factory Pattern.

But can we do this with the NetBeans Platform? And if so, is this difficult to do?

The answer is that it is extremely easy to do, and that the solution relies on the Service Provider mechanism defined in the Jar File Specification (yes, I told you that the NetBeans Community loves following standards) and in the NetBeans Platform Lookup technology.

So instead of creating a single module responsible for invoking the SISC Scheme Interpreter we will be creating two modules: a generic module that defines what a Scheme interpreter looks like, defining a Public API (something similar to a java interface) and another that contains the actual SISC interpreter, the implementation of the interface.

To clarify the current structure of our Scheme Module Suite I've created a sort of UML package diagram (but with modules, not packages) in Figure 47, “An overview of the current modules”.

You may be thinking that I add too many modules, and developing with the NetBeans Platform makes you build tons and tons of modules. Well, the fact is that this is a Good Thing. Having modules have single responsibilities follows the DeMarco's Single Responsibility Principle.

11.2. The Scheme Interpreter Module

So let's create this Scheme Interpreter Module, responsible for defining a public API that we'll use to evaluate Scheme expressions.

Creating a module is so easy that I won't repeat all the steps here (but see Section 6.1, “Creating another module for the editor...” if you need help on creating modules).

I will call this the "Scheme Interpreter" module and I'll specify a code base name of "net.antonioshome.scheme.interpreter": "net.antonioshome" (my domain) + ".scheme" (the project) + ".interpreter" (the module). Yes, I'm following the conventions that I used in previous sections.

Once the module is created let's see what this generic Scheme API should look like, and let's then make it public.

11.2.1. Designing a generic Scheme API

So let's define an API for evaluating Scheme expressions. [10] This API should be as generic as possible (so that we can plug-in different Scheme interpreters afterwards) and, of course, should have no dependencies with the actual implementation.

Let's review some requirements first, so as to define a good API.

An asynchronous API

The very first requirement for our API is that it is asynchronous, so that we can schedule Scheme expressions for evaluation in a worker thread. This will serve us for learning how to do threading with the NetBeans Platform (which is both easy and powerful, as we'll see later on).

So the Scheme expressions will be evaluated in worker thread. Cute. But we need to know what the result of the evaluation is, so we'll need to develop some sort of callback mechanism, that informs us what the result of the evaluation is (and what the errors are, if any).

Evaluating Strings, Files and URLs

We'll also need to keep it simple. I'd like to evaluate simple expressions (stored as simple Strings) and to evaluate files containing Scheme source code. So we'll need to methods: one for evaluating Strings and other for evaluating files.

But what if our Scheme source code is stored somewhere else but on a file? What about if our Scheme source code is stored in a web server miles away?

Mmmm. Maybe it's a better idea to add a method for evaluating Readers (after all a new StringReader( aString ) builds a Reader from a String) and another for evaluating Files (setting the current working directory to the directory where the file is stored). That way we'll be able to evaluate both stuff we read from a network connection and stuff stored in our local filesystem.

What about environments?

I'd also like to evaluate an expression and, later on, evaluate another in the same environment. So if I evaluate a "(define a 10)" expression and later on I evaluate a "(display a)" expression (in another thread, possibly), the second one refers to the value of "a" defined in the prior expression.

But to make things more complicated, I won't introduce environments now, but later on.

Yes, this is not a typo! Not introducing them will make things more complicated.

Why, you ask?

Well, think of it before reading any further.

Ready?

By introducing environments later on I will be changing a Public API. And that's asking for trouble, because I will be forcing the rest of modules to adapt to my changes. I will be forced to change all modules implementing the "Scheme Interpreter" interface!

And that's the reason why making things simpler now is making things harder later: if you define a Public API think twice all the use cases. After all you don't want to propagate changes in lots of modules, right?

If you change a Public API you'll have to propagate that change all over the place, to other modules using it. That's why the NetBeans Community has specific processes for API Reviews, and builds use cases for those Public APIs. Making sure a Public API is solid and that it covers all possible use cases is saving time (and money) in the future.

11.2.2. One single interface to rule them all

So to fullfil all those requirements I propose the following API for invoking any Scheme interpreter I can think of:

public interface SchemeInterpreter
{
  public String getName();
  public void eval( SchemeCallback aCallback, java.io.Reader someExpressions );
  public void eval( SchemeCallback aCallback, java.io.File aFile );
  public interface SchemeCallback
  {
    public void onSuccess( String aResultingExpression );
    public void onFailure( String anErrorMessage );
  }
}

And this will be our Public API for Scheme Interpreters. I've added a "getName()" method to return the name of the interpreter. All the rest should be obvious: two methods for evaluating expressions in Strings and Readers; and a callback interface with two methods: one to be invoked when the expression is evaluated correctly, and another one invoked when the expression generates an error.

And, of course, we should add some more Javadoc (since this is a Public API). Public APIs are Public, and must have good Javadoc (not included for clarity).

Now that our generic Scheme Interpreter interface is ready let's see how to make it public.

11.3. Making our interface public

So we add the SchemeInterpreter interface to our module. To make the API Public we select the module (in the Projects folder), right click on it and choose Properties in the popup menu. The Properties dialog pops up.

Finally we select "API versioning" option on the left of the dialog. A list of all packages in our module is presented to us in the "Public Packages" list. We select ours (net.antonioshome.scheme.interpreter) and click on it to make it public.

I like setting a major release version (and an implementation version) whenever I define a Public API. This way I may be able to make changes in the Public API (which is painful) in case of need.

11.3.1. Next stop: implementation

Once our Scheme Interpreter Public API is ready we are ready to start building an implementation of it (using SISC).

Of course we'll have to integrate the implementation with the SISC Library Wrapper, and we'll have also to see how to do threading (and show progress) using the NetBeans Platform. Keep tuned for that.

[10] The NetBeans Community has special procedures for defining new APIs (that include some formal review processes, for instance), but I won't be following them here. After all this whole thing is just an experiment and we won't cause any harm to the rest of the Community ;-)

12. Noodle Threads (threading, progress, console)

In the last section we defined a generic Scheme interpreter API, and made it public.

We need to implement this generic Scheme Interpreter API using a Scheme interpreter (such as SISC). But before doing so, we need to learn how to do basic things with the NetBeans Platform.

We need to learn how to run a task in background (and how to cancel it), and how to show progress to the user while the task runs. And, of course, how to print out the results of those tasks in the Output Window. With those basic idioms we may start cooking some more complex recipes, such as using our favourite Scheme interpreter.

So let's get some threads chopped.

12.1. Scheduling Threads for Execution

You can schedule Threads for execution as you always have done. Of course. There's nothing new under the sun here. You just create a Thread and you "start()" it. And you're all set.

But the NetBeans Platform contains some other mechanisms for scheduling threads. The first is the so called "RequestProcessor" class, and the second is the "Progress UI API", used to show progress of your tasks to the user. Let's see how easy is to use them (and again let me complain of not having studied all this before).

12.1.1. Threading with the RequestProcessor

The RequestProcessor (included in the Utilities API) has been around probably before 2000. So it's probably seven or eight years old now.

Eight years seems to me as sound. I mean, if a piece of software has been around during almost eight years (being used daily, by a lot of people, and with lots of eyes looking for issues and bugs and improvements) then I think that piece of software must be solid as rock, right?

RequestProcessors are roughly equivalent to Java Executors and Java Executor Services, this is, they're responsible for scheduling Runnables for execution in a set of threads.

Using a RequestProcessor is extremely simple. You may use the default RequestProcessor (a sort of default thread pool) to post your Runnables, like this:

Runnable myRunnable = ...
RequestProcessor.Task myTask = RequestProcessor.getDefault().post( myRunnable );

And your task is automatically scheduled for execution in a worker thread. That simple.

Note that you get a RequestProcessor.Task in turn, which is roughly equivalent to a Java FutureTask: an object you can use to cancel the task, for instance.

You can also create your own RequestProcessor with a predefined number of threads. That's probably a good idea if you don't know how many task user may spawn.

So that's all we wanted to know about RequestProcessors. As we've seen they're quite powerful (and quite simple to use as well).

I recommend you to review RequestProcessor's javadoc, just to see for yourself what you can do with this nice piece of (reusable) code.

Interrupting tasks

To interrupt a task you just use the RequestProcessor.Task handle and invoke "cancel()" on it. And you may be able to interrupt it.

May interrupt it? How so, you ask?

Well, the fact is that not all Java threads are really interruptible. I mean, you can "interrupt()" all Java threads, but not all of them will be really interrupted.

If you thread performs I/O operations (such as reading a file, or reading data through a JDBC connection) then your thread may be interrupted. But if your task does some heavy maths (or other computations without I/O operations) then your thread may not be interruptible. (I blogged about this earlier). This is so because all I/O operations invoke the "Thread.interrupted()" method periodically, and if the method returns true then they stop operation.[11]

But if you just burn your CPU with a long math computation and you don't check if "Thread.interrupted()" for yourself, then your task may be really uninterruptible. It will keep going even you invoke RequestProcessor.Task on it.

So if you want to make your tasks interruptible remember to check for "Thread.interrupted()" periodically. Just in case.

12.1.2. Showing progress

Of course we want to show progress of our task to our users. I mean, scheduling things for execution and not telling the user is somewhat rude and frustrating to them.

The NetBeans Platform includes utilities to help you show progress to the user, all of them packed under the Progress API module. This is just a module with two public classes.

There're different use-cases you may want to explore (by reading the javadoc), but the following one is good enough for our needs:

Cancellable myCancellable = ...
ProgressHandle myProgressHandle =
ProgressHandleFactory.createHandle("Cooking noodles, please wait...",  myCancellable );
myProgressHandle.start();
myProgressHandle.finish();

This is, you create a ProgressHandle using some text (shown to the user) and a Cancellable object. And then you start() the handle or finish() it.

By default this "ProgressHandle" object is undeterminate (you'll see a progress bar that is continuosly moving). But you can make it determinate and specify the percentage of task you've already finished (see ProgressHandle javadoc for more info).

If you create your ProgressHandle with a Cancellable task (as I've done) then a small button will appear in the UI that allows the user to cancel the task. (See Figure 50, “Progress UI interface”)

A lazy progress UI

The user interface for the progress bar is automatically shown in the lower-left part of the window. As you can see in Figure 50, “Progress UI interface”.

This special component, that visually shows progress to the user, does not appear immediately. This is, if you "start" your progress handle, then the UI does not appear immediately, but a few seconds later. This is good because if your task executes very quickly then nothing is shown to the user (and nothing gets her distracted!). This is a cool behaviour, I'd say. So don't get confused if your UI does not show immediately, just take it easy (and or have some more noodles cooked in your task ;-)).

12.2. The Output Window

Of course we will want to print out the results of our Scheme computations in the NetBeans Output Window. We will also want to read user input from there. [12]

The so called "Output Window" is contained in the I/O APIs, a small package with three classes, two interfaces and a highly scalable mechanism to show very large files in your Swing based application.

To use the "Output Window" you need an "InputOutput" object, used for input (reading stuff from the user) and showing output to the user. And you get an InputOutput object like this:

InputOutput console = IOProvider.getDefault().getIO("Scheme evaluation", true );

(Now, that's what I call a simple API!).

where "Scheme evaluation" is the name of the output tab, and "true" means that we want to open a new tab in case there's none already open (you may want to go take a look at the IOProvider Javadoc for other use cases).

Once you have your "InputOutput" object you can do lots of things:

• You can select() it, so that it appears on front of other tasks in case it's hidden (you usually do this before starting to output anything to the console, so the user knows that your task is really printing some stuff).

• You can getOut().println("Hello, world"), for instance, to print out stuff in your output window.

• And you can do many other things (reading stuff, adding buttons to the console, etc.). To do that you'll need to see the InputOutput javadoc for more details.

12.3. Small summary

So in this section we've learned how to spawn tasks for execution in background, how to show progress to the user and how to print out stuff into the console window.

With these tools we're ready to start implementing our Scheme interpreter.

But that'll require another section. Let's have a rest now. After all our noodle threads are already cooked!

[11] Well, this is not really so, but it's a fairly good explanation, I think.

[12] The Output Window can hold lots of text, and do it quickly and efficiently. The amount of stuff you can hold in the Output Window is limited by the amount of memory in your hard disk, and not in RAM. So don't try to build your own "Output Window". You'll be missing lots of functionality.

13. Invoking Scheme II (Lookup)

In a previous section Section 11, “Invoking Scheme I (creating a Public API)” we defined a Scheme Interpreter Public API as a simple interface.

We also learned how to fire threads for execution, and how to print out stuff to the console window, and how to cancel tasks (see Section 12, “Noodle Threads (threading, progress, console)”).

And we also bundled our preferred Scheme interpreter in a Library Wrapper module (see Section 10, “Wrapping Scheme into our IDE (Library Wrappers)”).

So we have all the tools required to implement that generic Scheme Interpreter interface.

In this section we'll implement that Public API, and we'll learn how to find the implementations of that interface using one of the most important tools in the NetBeans Platform: the Lookup library.

13.1. Invoking Scheme (finally!!!)

Invoking the SISC Scheme Interpreter from Java is not that difficult. There're there's documentation on how to do it, so I won't cover the details here. (I may make a blog entry on that later on).

But let me clarify how I'm doing it with the NetBeans Platform, just in case you're interested in trying out yourself.

The very first step is (of course!) to build a new NetBeans module. I'll be calling it the Scheme SISC Interpreter Module. We will be needing dependencies on the SISC Library Wrapper Module (that contains the SISC libraries) and with the Scheme Interpreter API Module (the one that contains the Scheme Interpreter Interface). [13]

In this new "SISC Interpreter Module" we'll be creating an implementation of the Scheme Interpreter Interface (that was defined a while ago, see Section 11.2.2, “One single interface to rule them all”). The implementation should look something like this:

import (sisc libraries here);
public class SISCSchemeInterpreter implements SchemeInterpreter
{
  public String getName() { return "SISC"; }
  public void eval( SchemeCallback aCallback, java.io.File aFile )
  {
    // evaluate a scheme file here using sisc libraries
  }
  public void eval( SchemeCallback aCallback, java.io.Reader someExpressions )
  {
    // evaluate a Reader here using sisc libraries
  }
}

Of course I've omitted the lengthy details here (but I'll be releasing the source code really soon, in case you're interested. If you're really interested you may want to download a preliminar version).

13.2. Looking-up for a Scheme implementation

So we have a Scheme Interpreter implementation inside our module. To use this exact implementation in another module all we need to do is to add a new dependency with this implementation, then create a "new SISCSchemeInterpreter()" and start using our implementation.

Simple, right?

But, hold down just a minute.

What happens if we have another Scheme interpreter? Shall we add a dependency with that other Scheme intepreter and then invoke it directly? What if we have ten different Scheme interpreters?

What if we want to iterate over all Scheme intepreters available in the system?

The answers to all these questions are included in the NetBeans Lookup Library (the "solution to communication between components"). A great library with lots of functionalities available as a simple jar file [14] that you can reuse in your standalone applications (even in your server-side applications!).

The NetBeans Lookup Library has lots of interesting functionalities. We'll see different applications of the library in upcoming sections. But at the moment we'll concentrate in this great feature: the Lookup Library is compliant with the Service Provider Interface as defined in the Jar File Specification (yes, NetBeans people love following standards), and can be used to register and to search implementations of services in the classpath.

So let's see how we can register our Scheme Interpreter implementation and then how to look-it-up using the Lookup Library.

[13] If you need help on how to define a new module, or on how to add a dependency with an existing module, you may want to go take a look up your answers at the Section 14, “Alphabetical Index”

[14] The "platform6/lib/org-openide-util.jar" file in your NetBeans installation directory

I. Building a standalone NetBeans editor

I.1. Building a standalone NetBeans editor

As I said previously, the NetBeans Platform contains a library called "The NetBeans Editor Library". This is, as far as I know, one of the most powerful open-sourced Swing text libraries out there, fully compliant with the Swing Text API.

Once upon a time, the NetBeans Team did an effort to make this excellent piece of code available as a standalone library. And, as of today, the library can still be used in standalone Swing applications.

In this appendix I present a tiny library that I've cooked that allows me to add a text editor based on the NetBeans Editor Library (this is, a text editor with syntax highlighting, line numbers and lots of cool features) in any Swing application.

Using the NetBeans editor in standalone applications, without the benefits of the whole NetBeans Platform, is somewhat weird. It's like having eggs without bacon (or, as we say here in Spain, "a day without bread"). Since we're going to use just a small set of NetBeans libraries, we'll miss lots of good features from the NetBeans Platform, such as code completion, for instance.

So, before going any further, let me give you a piece of advice: use the standalone NetBeans editor only if you have to (because you have a legacy Swing application, for instance) or as a way to learn about the NetBeans editor internals, or to have a quick and dirty way to test your syntax objects. But I wouldn't recommend you building a Swing application from scratch without using the whole NetBeans Platform. You'll definitely miss it!

The good news is that all the code described in this appendix is highly reusable in your NetBeans Platform based applications!

So let's get our feet wet with the standalone NetBeans editor.

I.1.1. What we need

This is what we need to build a standalone NetBeans editor.

Some NetBeans jar files

First of all we'll need four NetBeans 5.5 jar files (NetBeans 5.0 jar files will do too): org-openide-util.jar, org-netbeans-modules-editor-lib.jar, org-netbeans-modules-editor-util.jar and org-netbeans-modules-editor-fold.jar.

NetBeans 5.5 is released under the CDDL, so I think you can use those files without any legal problems. NetBeans 5.0 is released under the SPL, so there may be some legal concerns if you release those files alone (but I'm not sure: I'm glad I'm not a lawyer ;-)).

Note that those files don't contain all features of the editor. You don't have code completion (org-netbeans-modules-editor-completion.jar) nor bookmarks (org-netbeans-modules-editor-bookmarks.jar), for instance. I've decided to add just as little jars as possible to make the standalone editor runnable, but you're welcome to experiment adding some more jars if you want some extra functionality. You'll end up seeing that the NetBeans Platform is a much better choice, then! ;-)

Some specific classes for your editor

Of course we'll need specific classes for the language we'll be editing: a custom Syntax object (that parses our text), a custom TokenContext object (that specifies the tokens in our language), a custom TokenColoringInitializer (that specifies the colors of our tokens) and a SettingsInitializer (that specifies different editor settings, such as visibility of line numbers, text anti-aliasing, the height of line-separation, and more).

These classes are described somewhere else (Section 8, “Colorful editors: The Old, The Good and The Young”) in this document.

Note that you can reuse all these classes in your applications based on the NetBeans Platform. In fact using this standalone editor is a good way to test your syntax objects!

I have built a small project that contains source code for a Scheme language Syntax object, as well as a SettingsInitializer for Scheme and all those files you need. You may want to use those as a basis to build your custom syntax objects, or reuse some of the excellent NetBeans Syntax objects out there (such as the SQL Syntax).

A tiny library

In order to hide all editor internals I've built a tiny library (CDDL) that eases things. This tiny library is optional: you can either use it or do those internal calls yourself, by hand.

I.1.2. Little by little: Doing things by hand

So let's see what has to be done. Let's start doing things by hand.

.1. Initializing the editor settings

The very first thing we need to do is to initialize the NetBeans Editor Library settings stuff. You can do it yourself, like this:

import org.netbeans.editor.*;
...
Settings.addInitializer( new BaseSettingsInitializer(), Settings.CORE_LEVEL );
Settings.addInitializer( new ExtSettingsInitializer(), Settings.CORE_LEVEL );
Settings.reset();

Or, as we'll see in a minute, you can let the tiny library to do it for you.

.2. Adding your custom settings

After initializing the NetBeans Editor Library settings mechanism, you can add your own settings, using a similar code:

import org.netbeans.editor.*;
...
Settings.addInitializer( new MyCustomSettingsInitializer() );
Settings.reset();

Or, if you wish, you can let the tiny library do it for you too.

Note that the "MyCustomSettingsInitializer" is a class you have to build. This "SettingsInitializer" is valid for a given mime type, so all those settings will be used for all editors of that type.

The "SettingsInitializer" is the place where you specify the settings for your editor, such as if line numbers are to be visible, or if you want antialiased text, or if you want to make the status bar visible, the height between lines in the editor and tons of many other options.

For more information on the set of settings available please see the SettingsNames class.

.3. Registering your editor kit

After adding your settings to the hierarchy of settings, you need to register your editor kit using Swing's registration mechanism (JEditorPane.registerEditorKitForContentType). Again you can do it yourself, like this:

import javax.swing.JEditorPane;
...
MyCustomEditorKit kit = new MyCustomEditorKit();
JEditorPane.registerEditorKitForContentType(
  kit.getContentType(),
  kit.getClass().getName() );

Or, of course, you can let the tiny library do this for you, as we'll see in a minute.

.4. Adding key bindings

Finally you may want to add your preferred key bindings to a huge variety of actions. You can do it yourself, like this:

JEditorPane pane = new JEditorPane();
pane.setContentType( kit.getContentType() );
pane.getInputMap().put( 
  KeyStroke.getKeyStroke("DELETE"),
  ExtKit.deleteNextCharAction );

Or, again, you can let the tiny library add lots of key bindings for you. Note that the "ExtKit" class has tons of interesting actions that you can reuse in JMenus and JButtons or wherever. This set of actions is the most powerful I've seen so far out there.

.5. To JEditorPane or not: that's the question!

If you have followed all the steps above then you should be able to edit text in your JEditorPane without problems. In fact the JEditorPane will render the different tokens with the colors that you have specified in the TokenColorizer.

So far so good. But, where's the NetBeans editor status bar? Why can't we see a bar with line numbers?

The fact is that those little features are only available if you use a custom JComponent responsible for rendering text. Well, of course! JEditorPanes don't have line numbers after all!

So we need a custom JComponent responsible for rendering the status bar and the line number bar. And such a JComponent is created like this:

JEditorPane myEditorPane = ...
JComponent myEditorComponent = 
  ExtUtilities.getExtEditorUI( myEditorPane ).getExtComponent();

Or, again, you can use the tiny library as we'll see in a minute.

The created JComponent has some interesting properties. Apart of presenting the line number bar and status bar (if you have made them visible in your custom EditorSettings object), there're some other important things to note:

No JScrollPane needed

The created JComponent already incorporates its own (fast?) scrolling mechanisms, so you don't need to wrap it around a JScrollPane.

JEditorPane XOR JComponent

The JEditorPane and the specific JComponent have a funny behaviour: if you make the JEditorPane visible then the JComponent does not work anymore, and if you make the JComponent visible then the JEditorPane stops working!

I think this is so because the JComponent replaces the JEditorPane UI, and that's why you can either use one or the other.

The important thing to remember is that you want to use a JEditorPane to set text into the editor, but you want to use the JComponent to visualize the JEditorPane content. So remember to make just the JComponent visible (by adding it to a JFrame, for instance) and remember to keep a reference to the JEditorPane to be able to set text into the editor.

I.1.3. A tiny library: Faster, simpler

So having seen the internals of this standalone editor let's see how we can do things easier. The tiny library contains a single class called "NBEditorFactory". Let's see how to use it by using the following example:

/** Creates new form NBEditorLibDemoFrame */
public NBEditorLibDemoFrame()
{
  initComponents();
  
  // Create an editor kit for Scheme
  SchemeEditorKit editorKit = new SchemeEditorKit(); ①
  
  // Initialize Scheme language support
  NBEditorFactory.addSyntax( editorKit, new SchemeSettingsInitializer() ); ②
  
  // Create a plain editor pane
  JEditorPane editorPane = new JEditorPane();
  
  // Create a renderer to replace the editor pane *visually*
  JComponent renderer = 
    NBEditorFactory.newTextRenderer( editorKit, editorPane ); ③
  
  // Set the text *in the editor pane*
  editorPane.setText("; A scheme definition\n(define i (sqrt -1 ))\n"); ④
  
  // Visualize the *renderer*
  getContentPane().add( renderer, BorderLayout.CENTER ); ⑤

Let's see what the code above does:

① Creates a new ExtEditorKit for the Scheme language (the source code for SchemeEditorKit is available in the example).

② Initializes the editor settings (if needed), and then registers the SchemeSyntax object and the Scheme editor settings. (Source code for SchemeSyntax and SchemeSettingsInitializer available in the example).

③ Creates the custom JComponent to be used instead of the JEditorPane for visualization purposes.

④ Adds some text into the JEditorPane. Note that we keep on using the JEditorPane to add text to the editor, but we will use the JComponent for visualization purposes...

⑤ Adds the JComponent to the frame, to visualize text. Note that we don't add the JEditorPane, but the JComponent instead.

I.1.4. Do it yourself!

So what are you waiting for? Do it yourself! Now you know how to do it! Don't you? Let me review the basics again for you, just in case:

Build the syntax, settings and other files for your language

Start by taking a look at the the NetBeans Syntax Highlighting Tutorial (hurry up, this is going to change with the new syntax support!). You may also go take a look at the Section 8, “Colorful editors: The Old, The Good and The Young” and to the Section 9, “Adding Syntax Support” sections.

Get the tiny library

Or build your own. You can download it from my attic. Don't forget to get the jar files from a working NetBeans 5.5 IDE.

And don't forget...

... that using the NetBeans Editor Library as a standalone editor is probably a bad idea. The NetBeans Platform has lots of good stuff in there you can't miss! Keep tuned for more news, and have fun editing!

14. Alphabetical Index

Alphabetical Index

A

Autoload module, One for all, and all for one!

B

BaseKit, NetBeans' delicious EditorKits

BaseOptions, Options and BaseOptions

C

ClassLoader

Link to further information, Don't shout: I can't hear you anyway

ClassLoaders

One per module, Don't shout: I can't hear you anyway

ClassNotFoundException

requirements to meet to avoid ClassNotFoundExceptions, Don't shout: I can't hear you anyway

close(), in ModuleInstall, Executing Java code on module initialization

closing(), in ModuleInstall, Executing Java code on module initialization

Console (see Output Window)

D

dependencies, Using... and being used: dependencies

cyclic, I need you, you need me, I need you...

declared in the Project Metadata file, Adding dependencies to other modules

how to declare one with a module, Adding dependencies to other modules

transitive, I need you, you need him, I don't know him

E

Eager module, One for all, and all for one!

editor

EditorKits, NetBeans' delicious EditorKits

registering EditorKits, ... and registering our SchemeEditorKit

standalone editor, Building a standalone NetBeans editor

ExtKit, NetBeans' delicious EditorKits

I

I/O APIs (see Output Window)

Important Files

Project Metadata, Adding dependencies to other modules

initialization

of modules, One for all, and all for one!

of modules, using ModuleInstall, Executing Java code on module initialization

L

library wrapper, Modules, Module Suites and Library Wrappers

library wrappers

definition, Library Wrappers

lookup

global lookup, La bolsa global

introduction, So, where's my object?

looking-up objectxs, Looking-up objects

Lookups.fixed, Otros tipos de bolsas

performance, Complex queries, performance and... thieves!
Lookup

used to find interface implementations, Looking-up for a Scheme implementation

M

meta-inf/services

cómo buscarlos con Lookup, La bolsa global

module suites, Modules, Module Suites and Library Wrappers

modules, Modules, Module Suites and Library Wrappers

public API, Public APIs, exported packages, documentation and the importance of APIs

versions, Embracing change

N

NbEditorKit, NetBeans' delicious EditorKits

NetBeans Modules

Public APIs

API Design and API Review Tutorials, Public APIs, exported packages, documentation and the importance of APIs

degrees of stability, Public APIs, exported packages, documentation and the importance of APIs

javadoc, Public APIs, exported packages, documentation and the importance of APIs

O

options

system options

for editors, Options and BaseOptions

Output Window, The Output Window

P

Progress UI API, Showing progress

Project Metadata

and dependencies, Adding dependencies to other modules

Public API

changes in, What about environments?

creating one, Making our interface public

R

Regular module, One for all, and all for one!

restored(), in ModuleInstall, Executing Java code on module initialization

S

Service Provider Interface (see Lookup)

SettingsInitializer, The old way: overview

suites (see module suites)

Syntax

overview, The old way: overview

syntax highlighting

creating a custom Syntax object, Syntax - non functional requirements

for multiple languages (JSPs, etc.), Multi syntaxes

overview of basic syntax highlighting, The old way: overview

plain syntax, Using a plain syntax

SettingsInitializer, The old way: overview

Syntax - overview, The old way: overview

syntax highlighting mechanisms, Adding color to editors: The Old, The Good and The Young!

TokenColoringInitializer, The old way: overview

TokenContext, The old way: overview

T

threading

and the RequestProcessor, Threading with the RequestProcessor

Showing progress to the user, Showing progress

TokenColoringInitializer, The old way: overview

TokenContext, The old way: overview

U

uninstalled(), in ModuleInstall, Executing Java code on module initialization

V

validate(), in ModuleInstall, Executing Java code on module initialization

versions, Embracing change

and Public APIs, Making our interface public

X

XML FileSystem, The internals of the XML layer file

XML Layer, Sandwiches!! (XML Layer Files)

adding Java object instances, ... and registering our SchemeEditorKit

attributes, Attributes

examples of use, What are sandwiches good for?

files, Files

folders, Folders

how they work, How sandwich files work

how to see the whole sandwich, Watching the sandwich