Edit Add a file or picture Printable page Raw Text Page change history More options

RabbitHole FAQ

 TDI FAQ

This FAQ is under perpetual construction, and many of the question are clipped out of newsgroup. Because the question might be a little out of the context, you might have a better/more general wording for it. Please let us know if that is the case or if you have other FAQ questions!

Note that most of the items are dated, although old answers may be better than none, please check both the offical TDI docs and the newsgroup for updated information...


TDI: What is it?

Q: What is TDI?

2005.02.21

A: Depending on who you ask, you will get a lot different answers on this one.  But everybody agrees on it being (at least) an edge integration tool that has proved extremely useful in Identity Integration scenarios.

TDI is a very lightweight edge integration tool that installs quickly and scales well. It lets data flow in an AssemblyLine, pulling data from one or more sources and pushing them out to one or more target systems. 'Sources' and 'targets' include a multitude of systems and protocols.

TDI consists of a server run-time environment and a graphical editor to build, test and maintain the rules that the server executes.  It's all Java, and runs on most platforms. The GUI integrates seamlessly, but independently with the server. This enables the developer to continuously test connectivity, attribute mapping and transformation logic against live sources, and to do this incrementally and with immediate visual feedback.


Q: What is the difference between TDI and other integration frameworks?

2005.02.21

A: TDI makes very little assumptions about your data.  This means that it is flexible, but that it will not solve your business problems out of the box: No assumptions about your business logic are made. 

However, it does amazingly much out of the box when it comes to edge integration:  Grabbing data from one or more sources and merging them out to new data-sources.

This means that you will usually use TDI together with something that has more business rules build in: Sometime even another integration framework such as DB2II or WBI.

Since TDI gives you a rich scripting environment and easily lets you write your own Java-classes, it can be used to solve very complex scenarios: It is a matter of how much you want to build versus what to buy.


Q: What systems can TDI Connect to?

2005.02.21

A: The short answer is 'Look in the items available under

in the official reference manual.  (The links above might be outdated).

Also, if you have TDI installed, do a Help | About Tivoli Directory Integrator Components.

As of 2005.02.21, here is the current list:

---- Connectors

  • Active Directory Changelog Connector
  • Active Directory Changelog Connector v2
  • IBM Tivoli Directory Integrator AssemblyLine Connector
  • Axis Easy Web Service Server Connector
  • BTree Connector
  • Command Line Connector
  • DSML v2 Connector
  • Domino Change Detection Connector
  • Domino Users Connector
  • Exchange Changelog Connector
  • FTP Client Connector
  • File System Connector
  • HTTP Client (version 2)
  • HTTP Server (version 2)
  • JMS Pub/Sub Connector
  • TDS/iPlanet Changelog Connector
  • ITIM Agent Connector
  • JDBC Connector
  • JMS Pub/Sub Connector
  • Generic JNDI Connector
  • IBM Tivoli Directory Integrator LDAP Connector
  • IBM Tivoli Directory Integrator LDAP Server Connector
  • MQe Password Store Connector
  • Mailbox Connector
  • IBM MemQ Connector
  • Memory Stream Connector
  • NT4 Connector
  • Lotus Domino Connector
  • HTTP Client
  • PES Connector
  • RDBMS Changelog Connector
  • SNMP Connector
  • Script Connector
  • TCP Connector
  • Timer Connector
  • URL Connector
  • Web Service Receiver Server Connector

 

------ Parsers

  • CSV Parser
  • DSML v1 Parser
  • DSML v2 Parser
  • Fixed Record Parser
  • HTTP Parser
  • LDIF Parser
  • Line Reader/Writer
  • SOAP Parser
  • Script Parser
  • Simple Parser
  • XML Parser
  • XML Sax Parser
  • XSL based XML Parser

 

---- EventHandlers

  • Active Directory Changelog EventHandler
  • Connector Event Handler
  • DSML Version 2 EventHandler
  • Exchange Changelog EventHandler
  • Generic Thread
  • HTTP Switchboard
  • IBM Directory Server LDAP EventHandler
  • JMX Event Handler
  • LDAP Event Handler
  • Server side LDAP EventHandler
  • POP/IMAP Mailbox Event Handler
  • SNMP EventHandler
  • TCP EventHandler
  • Timer Event Handler
  • IBM zOS LDAP EventHandler

 

---- Function Components

  • AssemblyLine Function Component
  • Axis Easy Web Service Invoke
  • Axis Java-to-Soap
  • Axis Soap-to-Java
  • Castor Binding Java-to-XML
  • Castor Binding XML-to-Java
  • Complex Types Generator
  • Invoke Soap Web Service
  • MemBufferQ FC
  • Parser FC
  • Scripted Function Component
  • Wrap Soap
  • z/OS TSO/E Command Line Function Component

 


TDI How to/Can I ...

Installation

Q: How much memory and how much CPU power do I need to run TDI?

2005.05.02

A: It depends.  The server itself will run on a 486 CPU, and if you have 128 MB free memory after the OS/JVM have gotten their toll, you are in business.  However, certain AssemblyLines that build up data in memory (non-SAX based XML, highly recursive solutions etc) will need more memory.


Q: Can I Install and run TDI as non-root under Unix?

A:  Running as non-root should/can/may work.

Here are a couple of things you might need to be careful about (and there might be others. We will try to collect them for later use so let us know because this is not tested):

  • The installer itself must be run as root.  However, once installed, you can set the permissions to whatever you want :-).

    Take extra care around the logfiles, global.properties/solution.properties files.  In a non-root environment, you might want to force users to use solution.properties.

  • You will of course not be able to access unix-resources that you don't have privileges for (duh).  On the top of my head, the only one I can think of is portnumbers < 1024.  I'm sure there are more ...


Q: Can I do silent/non-interactive installs of TDI 6.0?

2005.08.26

A: Yes and this should work for earlier versions.

Silent/batch installs are done by creating a response file and using it as a parameter to your install-program/script.

To actually run a silent install using a previously created response file:

  • (Unix) setup.bin -options <responseFileName> -silent
  • (Windows) setupwin32.exe -options <responseFileName> -silent

To create a response file, either (Unix syntax used):

  • setup.bin -options-record <responseFileName>:  this will record and save the settings specified to the referenced file while actually doing an install.
  • setup.bin -options-template <responseFileName>: this will create an options type file to be used later, no install is done.

For TDI 6.0 the important values in the response file are:

  • installLocation - Where to install to 
  • WinSolDirPanleInstall.SolDirButtons - a very long name to indicate where one would like their "Solutions" directory created.  The options are:
    • NoSpec - Do not specify. Use current working directory at startup time.
    • UseTdi - Use a TDI subdirectory under my home directory
    • InstDir - Use Install Directory
    • SelectDir - Select a directory to use

      (So, for example,  to set one of these it would look like WinSolDirPanelInstall.SolDirButtons=NoSpec)

    • WinSolDirPanelInstall.SolDirDirectory: if the last item above is selected (SelectDir) use this key to set the actual directory to use.


Configuration

Q:Can I/How do I use DB2 as a system store?

2005.04.11

A: Yes you can since TDI 6.0.

If you look at the global.properties file, there are some  CREATE_TABLE statements for using setting up the system store.  If you use the right syntax, you can use non-cloudscape databases as system store.  DB2 has been tested, but no other RDBMS has for the time being.

Here is the DB2 syntax:

## Location of the DB2 database (networked mode)

com.ibm.di.store.database=jdbc:db2://168.199.48.4:3700/tdidb

com.ibm.di.store.jdbc.driver=com.ibm.db2.jcc.DB2Driver

com.ibm.di.store.jdbc.urlprefix=jdbc:db2:

com.ibm.di.store.jdbc.user=db2inst1

com.ibm.di.store.jdbc.password=******

com.ibm.di.store.start.mode=automatic

com.ibm.di.store.port=3700

com.ibm.di.store.sysibm=true

# the varchar(length) for the ID columns used in system store and PES Connector tables

com.ibm.di.store.varchar.length=512

# create statements for DB2 system store tables

com.ibm.di.store.create.delta.systable=CREATE TABLE {0} (ID VARCHAR(VARCHAR_LENGTH) NOT NULL, SEQUENCEID int, VERSION int)

com.ibm.di.store.create.delta.store=CREATE TABLE {0} (ID VARCHAR(VARCHAR_LENGTH) NOT NULL, SEQUENCEID int, ENTRY LONG VARCHAR FOR BIT DATA )

com.ibm.di.store.create.property.store=CREATE TABLE {0} (ID VARCHAR(VARCHAR_LENGTH) NOT NULL, ENTRY LONG VARCHAR FOR BIT DATA )

com.ibm.di.store.create.checkpoint.store=CREATE TABLE {0} (ID VARCHAR(VARCHAR_LENGTH) NOT NULL, ALSTATE LONG VARCHAR FOR BIT DATA, ENTRY LONG VARCHAR FOR BIT DATA, TCB LONG VARCHAR FOR BIT DATA )

com.ibm.di.store.create.sandbox.store=CREATE TABLE {0} (ID VARCHAR(VARCHAR_LENGTH) NOT NULL, ENTRY LONG VARCHAR FOR BIT DATA )


#mozTocId159908 Q: I want to use my additional java-classes supplied in my own jar-files.  Where should I put them and how does the loader work?

2005.05.31

A: Generally speaking, you can simply drop the jars you want to access under the jar-directory where you have installed TDI as they will be loaded by the loader.  If you want to use existing directories, read on.

There are two loader-mechanisms in action in TDI:  

  1. TDI's class own loader 
  2. The standard JVM loader.

The com.ibm.di.loader.userjars property will add a jar file or a directory to the TDI class loader. It is set in  global/solution.properties files, see the official documentation.  The directory will be recursively searched.  

The classpath env variable is used by the JavaVM App class loader. This can be set in many ways but the most common is to do a "SET CLASSPATH=X;Y;Z;etc....".

Since the list of libraries TDI had to have in the classpath grew large,  we migrated the "set classpath=xx" to the manifest file in the IDILoader.jar file to avoid problems with windows fainting on the classpath being too large. So, you could still set the classpath independently of the way we add jars to the classpath.

Want even more details? Read the following if you care about JavaVM class loading anatomy.

On a different but related matter: the class loaders in the Java VM operate in a hierachically delegating mode. At the top are the JavaVM class loaders (bootstrap, app and extensions loader) that are always given the first shot at loading a class. If they cant find the class, they delegate the operation to the lower level class loaders according to very specific rules. Every object (instance) in the VM has an associated class which in turn is associated with the class loader that loaded it. And every class loader has a parent class loader which is always given the first shot at loading a class (if no parent exists then the bootstrap class loader gets the first shot). So, the class loader hierachy is a tree of class loaders.

    OBJECT ---- instanceof -->  CLASS ---- associated with --> ClassLoader

This is an important point since class load delegation stops at the class loader associated with the object that initially requested a class. So, if you have a class that is loaded by IDILoader then this class is associated with IDILoader. This happens when you start the CE (java com.ibm.di.loader.IDILoader ..... com.ibm.di.admin.miadmin).

  1. One of the JavaVM class loaders load IDILoader (typically app loader as we do a java -classpath idiloader.jar com etc ....)
  2. IDILoader loads com.ibm.di.admin.miadmin (VM class loaders first try and fail this operation but delegates to IDILoader since it is a class loader subclass)
  3. IDILoader loads miadmin and miadmin is now associated with IDILoader
  4. miadmin request an instance of a new Java class .. say com.ibm.di.entry.Entry
  5. Delegation causes the class load request to first let the JavaVM class loaders attempt a load (which fails cause Entry is not in the CLASSPATH nor in the lib/ext directory)
  6. Since the request came from miadmin, the VM class loader delegates the operation to miadmin's class loader which is IDILoader
  7. IDILoader searches its list of paths and load the Entry class (which is now associated with IDILoader)

The point is that in order to accomodate our own loading of jars and classes we have to bootstrap our classloader into the chain when we start the server or the CE. If you did a "java com.ibm.di.admin.miadmin" then everything would fail perfectly since neither of the VM class loaders know anything about our jars directory. By going via the IDILoader we insert ourselves neatly in the class loading delegation.

So, when you add directories and files to the userjars (e.g. IDILoader) you must have the above in mind since class loading in some circumstances can be very tricky. Consider this example:

  1. A class (associated with IDILoader) calls another class (associated with a VM class loader) providing the name of a class as a param: someclass.method("com.ibm.di.entry.Entry")
  2. The called class tries to load the class you specified in the param
  3. This operation fails (assuming the jar is in TDI's jars dir only) since the request to load the class was made by an object associated with a VM class loader and IDILoader

This actually happens every now and then. One example is the Notes connector that causes a class load request from some classes we dont load. The classes they try to load are in the "notes.jar/ncso.jar" so these files must be somewhere where the JavaVM class loaders can find them (e.g. classpath=xxx or put it in the lib/ext directory)

There are a few tricks one can do to avoid this problem. One is preloading where you load the class in an IDILoader associated object before making the call to the one you cant control (once a class has been loaded no other attempts are made to load it and it retains the loader association for the life time of the VM).

So, the bottom line:

Whenever a class is loaded by the VM bootstrap, extensions (lib/ext), application (classpath) or a URLLoader (the files specified in the manifest.mf) then further class load requests made by any of those classes does NOT involve IDILoader. This can lead to the confusing "ClassNotFoundException" even though you have the jar file in the jars directory (or in the userjars prop).


Q: Is it possible to synchronize the Connector library between several TDI servers to provide the same set of Connectors on them?

2005.10.17

Here are some suggestions: 

  • The simplest, manual approach would be to copy/paste AL/EH/connectors/Scripts/etc from one config to another with the configuration editor
  • Another simple approach would be to use config 'includes' mechanism as covered by the user's guide, Chapter 2 (ITDI concepts) -> The config -> Include/Namespace. Included components belong to another ITDI configuration, but can be used within the context of the including Config as though they were stored there.

  • You could generate or alter ITDI config files programmatically, optionally using the contents from another config file, as David Saucier and David Druker did to auto-generate their ITDI configs; this involves the ITDI server API. ITDI is running, you Connect to it (locally or remotely) and tell it to create a new config with objects in it taken from an existing file. So there is room for writing complex ALs to do this job .


Q: I have edited global.properties, and I want those changes to survive upgrades/fixpacks. Is there a way?

2005.10.19

A: For 6.0, you can not directly achieve this.  However, note that:

  1. If you use solution.properties, that file will be created only if it does not exist. So editing an existing solution.properties file will make your solution robust to new versions of global.properties
  2. The command !include can be used in global.properties to include the contents of an url to your own property file. Examples include

    Note that in TDI 6.0 this line will have to be reinserted in global.properties if it gets overwritten by an install ...

    Unless you really know what you are doing, use absolute path for these includes, current working directory is not always obvious.

  3. The command !include can also be used in solution.properties


Q: Can I start AMC2 as a Windows Service?

A: Apparently it can be done :-)

Here is an  article from the external newsgroup on that topic. We have not even tried this, so use at your own risk.  But some of the relevant parameters might be documented below.

I use a free Java Service Wrapper from:

http://wrapper.tanukisoftware.org.  Below is my wrapper.conf for AMC2 as a

service.  It took a few hours to get this working so enjoy it!


I really want to do this for Webshere Express, but that looks complex enough

to take a week.


My wrapper.conf for AMC2 is as follows:  (Edit it for your directories.


#********************************************************************

# Wrapper Properties

#********************************************************************

# Java Application

wrapper.java.command=D:\Program Files\IBM\ITDI/_jvm/jre/bin/java


# Java Main class.  This class must implement the WrapperListener interface

#  or guarantee that the WrapperManager class is initialized.  Helper

#  classes are provided to do this for you.  See the Integration section

#  of the documentation for details.

wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp


# Java Classpath (include wrapper.jar)  Add class path elements as

#  needed starting from 1

wrapper.java.classpath.1=./wrapper.jar

wrapper.java.classpath.2=D:\Progra~1\IBM\ITDI\jars\webserver.jar


# Java Library Path (location of DLL or libwrapper.so)

wrapper.java.library.path.1=D:\Progra~1\IBM\ITDI


# Java Additional Parameters

#wrapper.java.additional.1=


# Initial Java Heap Size (in MB)

#wrapper.java.initmemory=3


# Maximum Java Heap Size (in MB)

#wrapper.java.maxmemory=64


# Application parameters.  Add parameters as needed starting from 1

wrapper.app.parameter.1=com.ibm.di.amc.tomcat.EmbeddedTomcat

wrapper.app.parameter.2="D:\Program

Files\IBM\ITDI/ibm_help/eclipse/plugins/org.eclipse.tomcat_4.0.6.2" amc2

wrapper.app.parameter.3="D:\Program

Files\IBM\ITDI/ibm_help/eclipse/plugins/org.eclipse.tomcat_4.0.6.2/webapps/amc2"

"9091"

#********************************************************************

# Wrapper Logging Properties

#********************************************************************

# Format of output for the console.  (See docs for formats)

wrapper.console.format=PM


# Log Level for console output.  (See docs for log levels)

wrapper.console.loglevel=INFO


# Log file to use for wrapper output logging.

wrapper.logfile=/ITDI Logs/wrapper.log


# Format of output for the log file.  (See docs for formats)

wrapper.logfile.format=LPTM


# Log Level for log file output.  (See docs for log levels)

wrapper.logfile.loglevel=INFO


# Maximum size that the log file will be allowed to grow to before

#  the log is rolled. Size is specified in bytes.  The default value

#  of 0, disables log rolling.  May abbreviate with the 'k' (kb) or

#  'm' (mb) suffix.  For example: 10m = 10 megabytes.

wrapper.logfile.maxsize=0


# Maximum number of rolled log files which will be allowed before old

#  files are deleted.  The default value of 0 implies no limit.

wrapper.logfile.maxfiles=0


# Log Level for sys/event log output.  (See docs for log levels)

wrapper.syslog.loglevel=NONE


#********************************************************************

# Wrapper Windows Properties

#********************************************************************

# Title to use when running as a console

wrapper.console.title=AMC2

#********************************************************************

# Wrapper Windows NT/2000/XP Service Properties

#********************************************************************

# WARNING - Do not modify any of these properties when an application

#  using this configuration file has been installed as a service.

#  Please uninstall the service before modifying this section.  The

#  service can then be reinstalled.


# Name of the service

wrapper.ntservice.name=AMC2START

# Display name of the service

wrapper.ntservice.displayname=Start AMC2 Service


# Description of the service

wrapper.ntservice.description=Start AMC2 Service


# Service dependencies.  Add dependencies as needed starting from 1

wrapper.ntservice.dependency.1=


# Mode in which the service is installed.  AUTO_START or DEMAND_START

wrapper.ntservice.starttype=AUTO_START


# Allow the service to interact with the desktop.

wrapper.ntservice.interactive=false

-------------:




Starting/Invoking TDI/ALs

Q: How do I pass command line parameters from the ibmdisrv command to AssemblyLines?

The customer doesn't want to use customized a external property file to achieve this because they will have a number of TDI depolyment and they want to use single TDI definition and change TDI behavior by command line option.

A: The following code will get you command line options:

    // Only for 6.0

   task.logmsg( "Param r : " + main.commandLineParam( "r" ) );

   // For both 6.0 and 5.x

   task.logmsg( "Param r : " + main.params.get( "r" ) );

So for 5.2, you use the main.params.get() method. Now, to ensure

that you don't activate a parameter that TDI uses, try parameters

0 to 9:

   ibmdisrv -c myConfig.xml -0 param1 -1 param2

Then you can grab them from within your Config (replace r by the paramter number...)

 


Q: How do I start an AssemblyLine from another AssemblyLine/EventHandler?

2005.02.28

A: Check out the section on AssemblyLine parameter passing in the User Guide, it will show you how to use startAL().


Q: How do I start an EventHandler from another AL/EH?

2005.02.28

A:

    eh = main.startEventTrigger("EHName", <0 or 1>);

    // eh is then a Thread, which you can wait for completion by

    // doing eh.join() for example


Q: Do I need join() in order to call an AssemblyLine from another AssemblyLine?

(This answer is related to scripting TDI and from TDI starting other AssemblyLines)

The customer found that calling an AssemblyLine from another AssemblyLine aborts when caller AssemblyLine ends or dies if customer doesn't use join() when calling  Assemblylines. Does this behavior follow product design? Should we use join() to call Assemblylines from other Assemblylines not to produce this kind of result?

A: How is the main AL being started? If you start the Server with the -w command param, then the Server stops when the specified ALs are complete -- even if they start other ALs. If you do not use -w then the Server will not terminate before all threads (ALs and EHs) are completed.

Please note that when the CE (Config Editor) runs an AL, it calls the Server with the -w option. It will not do this anymore in the 6.0 release. So it could fail from the CE, but work if you start from the Command line yourself.

Normally, you do not need to use join(), so you can have multiple ALs running in parallel.


Stopping/Terminating TDI/ALs

Q: How can an AssemblyLine terminate itself?

2005.02.24

A: The clean (graceful) way to abort an AL is by running the shutdown() method of the AssemblyLine class. From the AL in question,

task.shutdown();

would do the trick. 'On shutdown request' hooks would be invoked, making it possible for user-created Java objects to be closed/terminated. Connections are properly closed, and the epilogs are executed.

A more brutal way to shutdown an AL is through the system.abortAssemblyLine("reason"); The 'on shutdown request' hooks won't get called, though. The epilog will be called, unless it is already executed.

And also, you are probably wondering how to shutdown one AL from another AL. You can grab the running AL objects through MOBJ in TDI 5.x, and through the serverAPI in TDI 6. If you need to shutdown an AL from an external process, then you would use JMX or AMC in TDI 5.x. In TDI 6, you could add the RMI-based server API to your choices.

This only works for an AL trying to kill itself, if you want to have other processes within the same JVM kill it,  see How can I terminate ALs and/or EHs from other AL/EHs?


Q: How can an EventHandler terminate itself?

2005.02.28

First, a word from a sponsor: You really want to get away from EventHandlers and start using AssemblyLines and Connectors. Of course, not all Connectors support Server mode, so that's not always an options.  However, we want you  all to check out the Server mode availability ...  But back to the answer:

A:  The EH can kill itself by using the EH action "Terminate Handler!"

This is equivalent to the script

system.abortAL();

You can also call

task.setExitRequested(true);

All of the above will terminate the current instance of the EH, e.g. for a HTTP EH, it will terminate the current session, but the main EH will still listen for incoming HTTP requests.

If you want to terminate the HTTP server thread (the one that listens to incoming requests), you can use the code

    task.getServer().setExitRequested(true);


Q: How can I terminate ALs and/or EHs from other AL/EHs?

A: See the answer of the question below that deals with killing them all, killing one is just a special case.

Q: How can I terminate my Server

2005.02.28

A: If you are running 6.0:

var session = new Packages.com.ibm.di.api.APIEngine.getLocalSession();

session.shutDownServer()

You can always terminate the Server by killing the JVM, but that is probably what you asked about....

f you are running 5.2 and you don't want to kill the process through the OS, you will need to kill all the EHs and ALs running on te server.

Answer to 'Everything you wanted to know about killing it all':

2005.02.24

A: version 6.0: Version 6.0 lets you use the server api (com.ibm.di.api.local or com.ibm.di.api.remote) and it has a set of powerful methods. Here is a short script example:

The below script kills all EHs one by one, and then the server itself.

var session = new Packages.com.ibm.di.api.APIEngine.getLocalSession();

// Stop EventHandlers

ehs = session.getEventHandlers();

task.logmsg("RUNNING EHs: " + ehs.length);

for(i = 0; i < ehs.length; i++) {

   var eh = ehs[i];

   task.logmsg("STOPPING " + eh.getName());

   eh.stop();

}

// Kill the server

session.shutDownServer()

The ALs can be killed by the exact same mechanism.

A: version 5.2 and earlier.

The below code will do the job in 5.2 for most EH and ALs.  There are some methods that will trap interrupts, so test it out on your system.

When all ALs and EHs have died, the server will too.

// Stop EventHandlers

x = task.runningEHs();

task.logmsg("RUNNING EHs: " + x.size());

for(i = 0; i < x.size(); i++) {

   var eh = x.get(i);

   // If I am an EH dont commit suicide

   if ( eh == task )

      continue;

   task.logmsg("Stopping " + eh.getName());

   eh.setExitRequested(true);

}

// Stop AssemblyLines

x = task.runningALs();

task.logmsg("RUNNING ALs: " + x.size());

for(i = 0; i < x.size(); i++) {

   var al = x.get(i);

   // If I am an AL dont commit suicide

   if ( al == task )

      continue;

   task.logmsg("STOPPING " + al.getName());

   al.shutdown();

}


Q:How do I get the error status of the AssemblyLine from an EventHandler that started the AL?

A: I know the getResult() method to get the result of AssemblyLine execution from the EventHandler.  However, for example, when an AssemblyLine calls the filesystem Connector with a CSV parser, and the CSV parser produces exeption by illegal format of data, we can't get error status of the AssemblyLine from the caller EventHandler. Is there any way to get error status of Assemblylines in the  EventHandler in cases like this?

A: Calling al.getStats().getError() gets you the java.lang.Exception object if one exists. This is the exception (error code and message) that caused the AL failure.

Logging

Q:  How can I see my TDI 6.0 AssemblyLine logs using AMC2?

A:  The logs generated by the SystemLog appender will be visible for AMC2, so you need to  add a SystemLog appender to your server or AssemblyLine.


Q: I want to be able to log *just* my messages, to create an audit log. How can I do that?

(I don't want all the TDI messages)

2005.04.19

2005.04.20

Here are a couple of suggestions:

  1. The easiest is to log on only FATAL level and set up an appender to log on FATAL level.  TDI logs almost nothing on FATAL
  2. You can also make a Scripted Component that logs to file by writing directly to  the file. You will have to open that file during initialization of the AssemblyLine of course.

  3. You could set up a Passive Connector of your choice that you invoke to log.  The advantages here is that you have full control of what you log to (file, JDBC, LDAP, JMS ....), and it is easy to change.

    This Connector would typically be in you Connector Library.  JavaScript code (probably wrapped in a function)

    var err = system.newEntry(); // Create new Entry object

    err.merge(work); // Merge in attributes in the work Entry

    // This next line sets an attribute called Error

    err.setAttribute ( "Error", "Operation failed" );

    myErrorConnector.add( err ) // Add new err Entry;

  4. You can define your own log4j appender.  Check out log4j documentation.  If you do that, your config-files would not use task.logmsg() but rather something like:

First define (in the prologe for example):

myLog = Packages.org.apache.log4j.Logger.getLogger("myLog");

and then, instead of task.logmsg() use

myLog.info(msg); myLog.error() etc ...

In addition you will need (as spelled out by Jason Williams) to define the myLog appender itself in the log4j.properties file to something like: (NOTE! The Threshold must be defined for this to work. )

log4j.debug=true

log4j.rootCategory=INFO, Default, myLog


log4j.appender.Default=org.apache.log4j.FileAppender

log4j.appender.Default.file=ibmdi.log

log4j.appender.Default.layout=org.apache.log4j.PatternLayout

log4j.appender.Default.layout.ConversionPattern=%d{ISO8601} %-5p [%c] -

%m%n

log4j.appender.Default.append=false


log4j.logger.com.ibm.di.config=WARN

log4j.logger.com.ibm.di.loader=WARN


log4j.appender.myLog=org.apache.log4j.FileAppender

log4j.appender.myLog.File=myInfo.log

log4j.appender.myLog.layout=org.apache.log4j.PatternLayout

log4j.appender.myLog.layout.ConversionPattern=%c %d{ISO8601} -- %p --

%m%n

log4j.appender.myLog.append=false

log4j.appender.myLog.Threshold=WARN


Q: What's the correct approach in TDI 5.2 for writing output from many components to one destination? Can I configure multiple ALs and EHs to use the same logging specification through the TDI GUI.

2003

That is, is it allowable to fill out the logging tabs for several ALs and EHs with exactly the same details.  So if I specifiy the same output file for each AL, and the same Appender type and format,  and the ALs execute in parallel, will they cooperate and share writing to the file, or will each AL think it owns the file and overwrite each others output?  ALso if each is an Appender of type IDIFileRoller or DailyRollingFile, when and who decides to roll the file?

My goal is to ensure that I capture all the output from an EventHandler and the ALs it calls.  I would also like to be able to get the log file to roll periodically (Daily or on a size basis).

A: When starting an AL from an EH using the Start AL Action, you have the Internal Logfile checkbox to indicate that you want to them to share log appenders (including, sharing the same files for output if appenders have been set up for file output).

You can also do this by hand, however, there are limitations in the current version. For more information on how to get around the limit in parameters in the main.startAL() function, check out my posting in this forum with the subject:

"Re: IDI 5.2 logging: relative vs absolute file names AND  consolidating "main-"ALs and "sub-"ALs"

There I describe how to use a java.util. Vector to pass any number of params to an AL you want to start, including the logging context of the calling process.


Scripting

Variables and Properties

Q: How do I get/set Java properties?

2005.04.13

A: You use java.lang.System methods:

// Get all properties:

java.lang.System.getProperties()

// get  a property

task.logmsg("com.ibm.di.store.database = " + java.lang.System.getProperty("com.ibm.di.store.database"));

// set a property

java.lang.System.setProperty("com.ibm.di.store.jdbc.driver","com.ibm.db2j.jdbc.DB2jDriver");

Note that in addition to setting through scripting, you can set java properties in the global.properties files or using the Config Editor.

Q: How do I get a variable to persist across iterator iterations when several spawned ALs are involved?

I'm running an AL, which spawns off an AL, which spawns off an AL which needs to access a variable from the second AL across every iteration.  I  can pass the variable okay via the IWE (Intital Work Entry), but I can't figure out how to  take the value from the IWE and persist it across every iteration.  Is  there some sort of global variable concept or something like work with a  broader scope?

 A: Check out the system.setPersistentObject and getPersistentObject (and deletePersistentObject) if you want this info really persisted -- these methods use the System Store. Or you can try setting Java Properties, which means you can pass stuff around in the same JVM.

For example,

java.lang.System.getProperties().put("myKey", someJavaObject)

will save someJavaObject to Java properties using "myKey" to

access it later with:

java.lang.System.getProperties().get("myKey");

This means the object is only kept in memory and your solution will perform better than persisting it to the System Store (cloudscape) database.

Of course, you can always set an attribute equal to task in the calling al and then pass it to the child AL. The task variable is a pointer to the current AL. Then you can call the task.getWork() function. As long as you keep passing in a reference to the same AL, the getWork() method should return the same Entry object.

Q: I have my configuration parameters (such as filenames) in an external property file.  How can I get access to those values through scripting?

2005.02.10

A:  Here is one way of doing it. I have this function in a script library which I can then either copy into, or include in my Config.

// Return an External Property value

//

function getExtProp( propName, defaultValue ) // Returns Java String {

try {

retVal = main.getMetamergeConfig().getExternalProperties().getParameter(propName );

if ( retVal == null ) retVal = defaultValue

} catch (errMsg) {

retVal = defaultValue;

}

return retVal;

}

It wraps the call in a try-catch block because if the Ext Props are notdefined (i.e. assigned to a file) then the call throws and exception.This way I always get a value back.


Q: Why can't I do setParam() on my Connector?  Why are only some of the documented methods available to my Connector?

05.03.15

A: There are two Connector objects, and you might be using the wrong one.

When you see a Connector in an AssemblyLine in your ConfigEditor, you 'see' the common wrapper class, com.ibm.di.server.AssemblyLineComponent.  However, it has an interface part (for example for LDAP com.ibm.di.connector.LDAPConnector) that is specific for the type of Connector that you use.

Some of the methods you are interested in belong to the wrapping class (com.ibm.di.server.AssemblyLineComponent) and some belong to the specific target class (com.ibm.di.connector).  If you look up the javadocs, you will find the methods belonging to one and the other: Some methods have the same name in both.

The wrapper class com.ibm.di.server.AssemblyLineComponent has a field called connector that is the link between the high level common AssemblyLineComponent and the specific instance of (in the case of LDAP) com.ibm.di.connector.LDAPConnector tied to it.

So if you had trouble with this, chances are that you will have to write things like thisConnector.connector.setParam("filePath", "examples/scripting/sample.csv" )


Inheritance and object sharing

Q: After overriding an inherited Connector, I want revert to the parent behaviour. Can I do that without recreating the Connector?

A: Primary AL component inheritance can be set (and broken) via the Inherit From box/button at the top of the component Details window.

For those components that offer multiple configuration tabs (like Connectors and Functions), specific inheritance for each aspect is controlled by the Inherit From box/button at the lower right-hand part of the tab.

Any component tab can be set to:

  • No inheritance
  • Inherit from a library component
  • Inherit from the parent.

Connectors offer an additional Inheritance overview dialog that is accessed via the button found at the top left-hand part of the Connector Details window.

You can override inherited behavior and settings by editing and creating new items in a component tab, like Parameters, Attribute Maps or Hooks. To reinstate inherited settings you must do the following:

  • Parameters: Editing a parameter value break inheritance. To reinstate an inherited value, click on the parameter label and then press the Use Original Value button
  • Hook inheritance is overriden when you edit a Hook script or select/deselect the Enabled checkbox. To return to inherited Hook settings, simply select the Hook(s) and press the Delete hook button at the top of the Hook list
  • Attribute Maps: You can override an inherited Attribute mapping by manually editing it or creating a new one. To return to inherited mappings, select the Attributes and delete them by pressing the Delete attribute button at the top of the map
  • Other: All other settings are controlled via their Inherit from box/button.

 


Q: Can we share object (i.e. work object) between multiple AssemblyLines? In the other words, what is the scope of object in TDI? I expect scope of object is defined in thread boundary. Is this my understanding correct?

2005.02.21

 

A: Since each AL (and EH) runs in its own Script Engine instance, all variables are local to the AL (EH). If you want to share them, you can use various techniques:

  1. Passing them explicitly on startup: main.startAL("myAL", myEntryObj)
  2. Using the System Store:

    system.setPersistentObject( "keyname", myEntryObj ) and

    myEntryObj = system.getPersistentObject( "keyname" );

  3. Using Java Properties:

    java.lang.System.getProperties().put( "keyname", myObject ); and

    myObject = java.lang.System.getProperties().get( "keyname" );

  4. Use the MemQ object (or MemBufferQ), which allow you to open a pipe between processes (again in the same JVM).

      Method a) is for Entry objects, b) works for any serializable Java object and c) for any Java object. Method d) is slower and harder to use, but it gives you advantages if you are sharing much data.

 


Q: Is there a way to share Connectors between process ?

A: Here's an example of passing a Connector into an

AL (retrofitting the deprecated run-time provided Connector

feature). It does this by setting a Java Property in the EventHandler:

     // Save as a Java property, making it global for all

     // threads in this VM.

     java.lang.System.getProperties().put("RTConn", useConn);

Then in the AL I have a scripted Connector that looks for this property

and grabs the Raw Connector. Here is the selectEntries() method:

    function selectEntries() {

     // Get globally saved Connector (must be pre-initialized)

     RTConn = java.lang.System.getProperties().get("RTConn");

     if (RTConn == null)

        task.logmsg("** Could not get System property called \"RTConn\"")

     else {

        RTConn.selectEntries();

      }

    }


Runtime Configuratiom

Q: Can I change Connector mode at runtime (for example, from Update to Add-only mode)?  Are there better alternatives?

2005.05.02

A: You cannot change the mode of a Connector after the AssemblyLine has started.  But if you start the AL with startAL(), then you can modify the config object before you start the AL. Usually, there are better and easier ways to obtain what you are looking for, so read on.

Assume the Connector is called MyConnector.in AssemblyLine MyAL, and you want to change the mode to ADDONLY_MODE for all AssemblyLines started in this server (from now on). Here is the syntax (but read on before you do this).

connectorconfig = main.getMetamergeConfig().lookup("AssemblyLines/MyAL").getConnectorByName("MyConnector");

connectorconfig.setMode( connectorconfig.ADDONLY_MODE );

Note that you will need to be careful with your attribute maps: Probably the best is to set up (in advance and with the CE) the attribute maps for the Connector for all modes you want to set.

Now, in the spirit of 'There are many ways to do it in TDI', let me suggest a couple of alternative paths:

  • If you are working with LDAP changelogs, this behavior is directly built into TDI: The Delta Mode will perform delete/add or update dependent on what is needed.

  • Update Mode can do both Add and Update dependent on how it is configured (and that can be changed on the run).

  • When savvy users want switch Modes, they will often use preconfigured Connectors in passive state (one for each mode they want to switch to) and then use the passive Connectors as they need them (see Passive State in the Users Guide).  This solution has the advantage that  the code becomes much easier to read and document than if you change the mode on-the-fly.

High Availability/Defensive programming

Q: I can not Connect to my data-source, but the On Connection Failure hook is not called. This must be a bug, no?

2005.05.27

The On Connection Failure hook in TDI 6.0 should really have been called On Connection Loss.  It is part of the AL flow and will not be called before initialization has been successfully completed and the AL flow invoked.

To catch initial Connection errors, you should place code in your On Error hook in the Prolog section of the Connector.  Do not try to reinitialize the Connector here as you will create a recursive loop and because there are non-initialized javaobjects at this stage that can cause problems.

The easiest way to handle initial Connection errors is simply to restart the AL in the prolog On Error hook:

main.startAL("myName");

system.abortAssemblyLine("message ....");

Make sure that you pass the TCB on if this AL has one.  

Another alternative is to have kicker AssemblyLine that starts your business AssemblyLine:  If one of the Connector fail, you can catch the error and take whatever action you want to.

Finally, for the adventurous, you can try to sniff out whether a Connection will fail or not in the Prolog Before Initialization hook.  (Note, the below does not guarantee you that you will ultimately get a connection)

Something like will do the job:

maxtry = 10; // 10 * 60 secs = 10 minutes before we give up

while (--maxtry > 0) {

    try {

        thisConnector.initialize();

        thisConnector.terminate();   // remember the Connector will be initialized later since this is before initialize.

        break; // success - exit while loop

    } catch (error) {

        task.logmsg("Initialized failed ... trying " + maxtry + " more times");

        thisConnector.terminate();   // removing this will might cause you to bleed memory

        system.sleep(60);

    }

}

//

if (maxtry < 1)

    system.abortAssemblyLine("Unable to reinitialize Connector " + thisConnector.getName());

        


Q: When I reuse Connectors, where is the best practice for placing of the reconnection logic:  In the initializing Connector (IC), or in the Connector that reuses the initialized Connection (RC)?

2005.05.30

A: The Connection itself is what is reused through the @-syntax, not the Connection parameters and hooks.  

Initialization Connection errors must be handled during Initialization, there is no easy alternative here.

When it comes to the Reconnect Tab and the On Connection Failure, you will usually get two sets of them:  One for the initializing Connector (IC) and one for the reusing Connector (RC).  There is no inheritance of the hooks/tabs, so all reconnect configuration need to exist in the RC (and possibly in IC as well if IC uses the Connection as well).

However, what you might want to do, is to place your Connector in the Connector Library (LC), and have your IC and RC both inherit from LC. This way, you only have one place to update your reconnect logic should you want to.


Various

Q: Is it possible send an email (SMTP) from ITDI?

A: Yes, check out the SendMail Function (FC) and the the UserFunctions class in the Javadocs, available as the system bean in ITDI AssemblyLines.

In other words,

system.sendMail("from@mail.com", "to@mail.com", "subject, "body" , "c:\\file.txt")

sends an email message. Make sure the mail.smtp.host Java property is configured with the hostname of a valid SMTP server.

Parameters:

  • recipient - A comma separated list of recipient addresses
  • subject - The Subject field 
  • body - The message text 
  • attachment - If specified a file-path that will be attached to the message

Returns:

If null, the message was sent. Otherwise, this is the error message.


Q: I have a simple AL with an iterator Connector feeding into an update Connector.  In the output map of the update Connector I have left the Mod checkboxes unchecked but when I run the AL the Connector complains if a record with the same data already exists.

Shouldn't the connector simply compare and skip the existant record?  I get the same behaviour when I use AddOnly mode.  Do I need to add a trigger or should the default be to skip to the next record in the iteration loop?

A: If you don't want to do anything when you find an existing record, you can

just add code to the "Override Modify" hook.

any of the following code would work.

  1. //
  2. system.skipEntry();
  3. system.ignoreEntry();

If you use ignore or skip, you will at least get the count when the AL

finishes.


Q: How can I obtain the actual connector object, or at least the actual Connector name?

Inside the AL,all Connectors are wrapped with the AssemblyLineComponent class. You can always query its name with the getName() function. You can get the classname of any Java object by using the .getClass() method.

To get at the Connector Interface itself (the CI is the actual protocol/API specific part of the Connector -- LDAP, JDBC, JMS, etc.) use the .connector member.

As an example, let's say I have an AL with a Connector called SQL_People in it. The Prolog - After Init Hook of the Connector contains this code:

  task.logmsg("      Conn Name: " + thisConnector.getName() );

  task.logmsg("     Conn Class: " + thisConnector.getClass() );

  task.logmsg(" Conn Interface: " + thisConnector.connector.getName() );

  task.logmsg("       CI Class: " + thisConnector.connector.getClass() );

This will result in the following output.

15:27:34        Conn Name: SQL_People

15:27:34       Conn Class: class com.ibm.di.server.AssemblyLineComponent

15:27:34   Conn Interface: SQL_People

15:27:34         CI Class: class com.ibm.di.connector.JDBCConnector

Note how both the ALComp and the CI return the AL Comp's name with the getName() method. Btw. thisConnector is a special variable that references the Connector where this script is run.


Q: How can I transform/hash a (password) string in cleartext with MD5?

2005.06.22

From Rudy Sutjiato

function md5Hash(inputString)

{

var msgDigest = java.security.MessageDigest.getInstance("MD5"); var javaStr = new java.lang.String(inputString); var digestStr = msgDigest.digest(javaStr.getBytes());

return system.base64Encode(digestStr);

}

Call the above function anywhere in your config file as follow:

var hashStr = md5Hash("String to hash");


Connectors

General Connector

Q: The Connection to my source (JDBC, LDAP, JNDI etc) fails, but the On Connection Failure hook is never called. 2005.03.08

A: The On Connection Failure hook is part of the AssemblyLine flow and will only be called after you have started to process your data after all the Connectors have been initialized. The hook should really be called On Connection Loss as it is not called if you never have managed to initialize your Connector.

What you should do is to call the On Error hook in the Connector prolog to catch initial connection errors.


Domino Change Detection Connector

Q: Is it possible to clear the state store that the Domino Change Detection Connector uses?

I need to script the equivalent of pressing the 'delete' button on the config page, in order to have the connector iterate through all the entries in the Domino database again.

2005.06.22

A:  The following code in your AL Prolog - Before Init will do the trick:

system.deletePersistentObject(YOUR_SYSTEM_STORE_KEY);


Domino User Connector

Q: Is it possible to specify the user's mail template when creating the user?

2004.04.12

A: Yes, this is possible from TDI 6.1 and later versions. See the TDI Reference Guide for more information.


Q: Is it possible to change only the last name of a user already registered in Domino (keeping existing mailbox etc.)?

2004.04.12

A:

  • In order to just change the user last name (without creating a second user), you need to also supply the FirstName of the user in the conn Entry - map the "FirstName" Attribute.
  • Assume I have a user in Domino with the name Charlie Brown...  his FullName attribute is