You are here: TDI Users>Integrator Web>FaQ>TDIFAQ (27 Apr 2011, EddieHartman)
RabbitHole FAQ
|
TDI
FAQ
Many of the questions here are clipped out of the TDI User Forum. Be sure and search there as well.
Also note that many items here are a bit dated. But old answers are better than none.
And don't forget to ask Google, remembering that TDI is pure Java - so JDBC + TDI = JDBC + Java, and LDAP + TDI = LDAP + Java, and ...
TDI: What is it?Q: What is TDI?2005.02.21 A: Depending on who you ask, you will get a lot different answers on this one. But everybody agrees on it being (at least) an edge integration tool that has proved extremely useful in Identity Integration scenarios. TDI is a very lightweight edge integration tool that installs quickly and scales well. It lets data flow in an AssemblyLine, pulling data from one or more sources and pushing them out to one or more target systems. 'Sources' and 'targets' include a multitude of systems and protocols. TDI consists of a server run-time environment and a graphical editor to build, test and maintain the rules that the server executes. It's all Java, and runs on most platforms. The GUI integrates seamlessly, but independently with the server. This enables the developer to continuously test connectivity, attribute mapping and transformation logic against live sources, and to do this incrementally and with immediate visual feedback.Q: What is the difference between TDI and other integration frameworks?2005.02.21 A: TDI makes very little assumptions about your data. This means that it is flexible, but that it will not solve your business problems out of the box: No assumptions about your business logic are made. However, it does amazingly much out of the box when it comes to edge integration: Grabbing data from one or more sources and merging them out to new data-sources. This means that you will usually use TDI together with something that has more business rules build in: Sometime even another integration framework such as DB2II or WBI. Since TDI gives you a rich scripting environment and easily lets you write your own Java-classes, it can be used to solve very complex scenarios: It is a matter of how much you want to build versus what to buy.Q: What systems can TDI Connect to?2005.02.21 A: The short answer is 'Look in the items available under in the official reference manual. (The links above might be outdated). Also, if you have TDI installed, do a Help | About Tivoli Directory Integrator Components. As of 2005.02.21, here is the current list: ---- Connectors
TDI How to/Can I ...InstallationQ: How much memory and how much CPU power do I need to run TDI?2005.05.02 A: It depends. The server itself will run on a 486 CPU, and if you have 128 MB free memory after the OS/JVM have gotten their toll, you are in business. However, certain AssemblyLines that build up data in memory (non-SAX based XML, highly recursive solutions etc) will need more memory.Q: Can I Install and run TDI as non-root under Unix?A: Running as non-root should/can/may work. Here are a couple of things you might need to be careful about (and there might be others. We will try to collect them for later use so let us know because this is not tested):
Q: Can I do silent/non-interactive installs of TDI 6.0?2005.08.26 A: Yes and this should work for earlier versions. Silent/batch installs are done by creating a response file and using it as a parameter to your install-program/script. To actually run a silent install using a previously created response file:
ConfigurationQ:Can I/How do I use DB2 as a system store?2005.04.11 A: Yes you can since TDI 6.0. If you look at the global.properties file, there are some CREATE_TABLE statements for using setting up the system store. If you use the right syntax, you can use non-cloudscape databases as system store. DB2 has been tested, but no other RDBMS has for the time being. Here is the DB2 syntax:
## Location of the DB2 database (networked mode)
com.ibm.di.store.database=jdbc:db2://168.199.48.4:3700/tdidb
com.ibm.di.store.jdbc.driver=com.ibm.db2.jcc.DB2Driver
com.ibm.di.store.jdbc.urlprefix=jdbc:db2:
com.ibm.di.store.jdbc.user=db2inst1
com.ibm.di.store.jdbc.password=******
com.ibm.di.store.start.mode=automatic
com.ibm.di.store.port=3700
com.ibm.di.store.sysibm=true
# the varchar(length) for the ID columns used in system store and PES
Connector tables
com.ibm.di.store.varchar.length=512
# create statements for DB2 system store tables
com.ibm.di.store.create.delta.systable=CREATE TABLE {0} (ID
VARCHAR(VARCHAR_LENGTH) NOT NULL, SEQUENCEID int, VERSION int)
com.ibm.di.store.create.delta.store=CREATE TABLE {0} (ID
VARCHAR(VARCHAR_LENGTH) NOT NULL, SEQUENCEID int, ENTRY LONG VARCHAR
FOR BIT DATA )
com.ibm.di.store.create.property.store=CREATE TABLE {0} (ID
VARCHAR(VARCHAR_LENGTH) NOT NULL, ENTRY LONG VARCHAR FOR BIT DATA )
com.ibm.di.store.create.checkpoint.store=CREATE TABLE {0} (ID
VARCHAR(VARCHAR_LENGTH) NOT NULL, ALSTATE LONG VARCHAR FOR BIT DATA,
ENTRY LONG VARCHAR FOR BIT DATA, TCB LONG VARCHAR FOR BIT DATA )
com.ibm.di.store.create.sandbox.store=CREATE TABLE {0} (ID
VARCHAR(VARCHAR_LENGTH) NOT NULL, ENTRY LONG VARCHAR FOR BIT DATA )
#mozTocId159908 Q: I want to use my additional java-classes supplied in my own jar-files. Where should I put them and how does the loader work?2005.05.31 A: Generally speaking, you can simply drop the jars you want to access under the jar-directory where you have installed TDI as they will be loaded by the loader. If you want to use existing directories, read on. There are two loader-mechanisms in action in TDI:
Q: Is it possible to synchronize the Connector library between several TDI servers to provide the same set of Connectors on them?2005.10.17 Here are some suggestions:
Q: I have edited global.properties, and I want those changes to survive upgrades/fixpacks. Is there a way?2005.10.19 A: For 6.0, you can not directly achieve this. However, note that:
Q: Can I start AMC2 as a Windows Service?A: Apparently it can be done
Here is an article from the external newsgroup on that topic. We
have not even tried this, so use at your own risk. But some of
the relevant parameters might be documented below.
I use a free Java Service Wrapper from:
http://wrapper.tanukisoftware.org. Below is my wrapper.conf for AMC2 as a service. It took a few hours to get this working so enjoy it! I really want to do this for Webshere Express, but that looks complex enough to take a week. My wrapper.conf for AMC2 is as follows: (Edit it for your directories. #******************************************************************** # Wrapper Properties #******************************************************************** # Java Application wrapper.java.command=D:\Program Files\IBM\ITDI/_jvm/jre/bin/java # Java Main class. This class must implement the WrapperListener interface # or guarantee that the WrapperManager class is initialized. Helper # classes are provided to do this for you. See the Integration section # of the documentation for details. wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp # Java Classpath (include wrapper.jar) Add class path elements as # needed starting from 1 wrapper.java.classpath.1=./wrapper.jar wrapper.java.classpath.2=D:\Progra~1\IBM\ITDI\jars\webserver.jar # Java Library Path (location of Wrapper.DLL or libwrapper.so) wrapper.java.library.path.1=D:\Progra~1\IBM\ITDI # Java Additional Parameters #wrapper.java.additional.1= # Initial Java Heap Size (in MB) #wrapper.java.initmemory=3 # Maximum Java Heap Size (in MB) #wrapper.java.maxmemory=64 # Application parameters. Add parameters as needed starting from 1 wrapper.app.parameter.1=com.ibm.di.amc.tomcat.EmbeddedTomcat wrapper.app.parameter.2="D:\Program Files\IBM\ITDI/ibm_help/eclipse/plugins/org.eclipse.tomcat_4.0.6.2" amc2 wrapper.app.parameter.3="D:\Program Files\IBM\ITDI/ibm_help/eclipse/plugins/org.eclipse.tomcat_4.0.6.2/webapps/amc2" "9091" #******************************************************************** # Wrapper Logging Properties #******************************************************************** # Format of output for the console. (See docs for formats) wrapper.console.format=PM # Log Level for console output. (See docs for log levels) wrapper.console.loglevel=INFO # Log file to use for wrapper output logging. wrapper.logfile=/ITDI Logs/wrapper.log # Format of output for the log file. (See docs for formats) wrapper.logfile.format=LPTM # Log Level for log file output. (See docs for log levels) wrapper.logfile.loglevel=INFO # Maximum size that the log file will be allowed to grow to before # the log is rolled. Size is specified in bytes. The default value # of 0, disables log rolling. May abbreviate with the 'k' (kb) or # 'm' (mb) suffix. For example: 10m = 10 megabytes. wrapper.logfile.maxsize=0 # Maximum number of rolled log files which will be allowed before old # files are deleted. The default value of 0 implies no limit. wrapper.logfile.maxfiles=0 # Log Level for sys/event log output. (See docs for log levels) wrapper.syslog.loglevel=NONE #******************************************************************** # Wrapper Windows Properties #******************************************************************** # Title to use when running as a console wrapper.console.title=AMC2 #******************************************************************** # Wrapper Windows NT/2000/XP Service Properties #******************************************************************** # WARNING - Do not modify any of these properties when an application # using this configuration file has been installed as a service. # Please uninstall the service before modifying this section. The # service can then be reinstalled. # Name of the service wrapper.ntservice.name=AMC2START # Display name of the service wrapper.ntservice.displayname=Start AMC2 Service # Description of the service wrapper.ntservice.description=Start AMC2 Service # Service dependencies. Add dependencies as needed starting from 1 wrapper.ntservice.dependency.1= # Mode in which the service is installed. AUTO_START or DEMAND_START wrapper.ntservice.starttype=AUTO_START # Allow the service to interact with the desktop. wrapper.ntservice.interactive=false -------------: Starting/Invoking TDI/ALsQ: How do I pass command line parameters from the ibmdisrv command to AssemblyLines?The customer doesn't want to use customized a external property file to achieve this because they will have a number of TDI depolyment and they want to use single TDI definition and change TDI behavior by command line option. A: The following code will get you command line options: // Only for 6.0 task.logmsg( "Param r : " + main.commandLineParam( "r" ) ); // For both 6.0 and 5.x task.logmsg( "Param r : " + main.params.get( "r" ) ); So for 5.2, you use the main.params.get() method. Now, to ensure that you don't activate a parameter that TDI uses, try parameters 0 to 9: ibmdisrv -c myConfig.xml -0 param1 -1 param2 Then you can grab them from within your Config (replace r by the paramter number...)Q: How do I start an AssemblyLine from another AssemblyLine/EventHandler?2005.02.28 A: Check out the section on AssemblyLine parameter passing in the User Guide, it will show you how to use startAL().Q: How do I start an EventHandler from another AL/EH?2005.02.28 A: eh = main.startEventTrigger("EHName", <0 or 1>); // eh is then a Thread, which you can wait for completion by // doing eh.join() for exampleQ: Do I need join() in order to call an AssemblyLine from another AssemblyLine?(This answer is related to scripting TDI and from TDI starting other AssemblyLines) The customer found that calling an AssemblyLine from another AssemblyLine aborts when caller AssemblyLine ends or dies if customer doesn't use join() when calling Assemblylines. Does this behavior follow product design? Should we use join() to call Assemblylines from other Assemblylines not to produce this kind of result? A: How is the main AL being started? If you start the Server with the -w command param, then the Server stops when the specified ALs are complete -- even if they start other ALs. If you do not use -w then the Server will not terminate before all threads (ALs and EHs) are completed. Please note that when the CE (Config Editor) runs an AL, it calls the Server with the -w option. It will not do this anymore in the 6.0 release. So it could fail from the CE, but work if you start from the Command line yourself. Normally, you do not need to use join(), so you can have multiple ALs running in parallel.Stopping/Terminating TDI/ALsQ: How can an AssemblyLine terminate itself?2005.02.24 A: The clean (graceful) way to abort an AL is by running the shutdown() method of the AssemblyLine class. From the AL in question,task.shutdown();
would do the trick. 'On shutdown request' hooks would be invoked,
making it possible for user-created Java objects to be
closed/terminated. Connections are properly closed, and the epilogs are
executed.
A more brutal way to shutdown an AL is through the system.abortAssemblyLine("reason");
The 'on shutdown request' hooks won't get called, though. The epilog
will be called, unless it is already executed.
And also, you are probably wondering how to shutdown one AL from
another AL. You can grab the running AL objects through MOBJ in TDI
5.x, and through the serverAPI in TDI 6. If you need to shutdown an AL
from an external process, then you would use JMX or AMC in TDI 5.x. In
TDI 6, you could add the RMI-based server API to your choices.
This only works for an AL trying to kill itself, if you want to have
other processes within the same JVM kill it, see How can I terminate ALs and/or EHs
from other
AL/EHs?
Q: How can an EventHandler terminate itself?2005.02.28 First, a word from a sponsor: You really want to get away from EventHandlers and start using AssemblyLines and Connectors. Of course, not all Connectors support Server mode, so that's not always an options. However, we want you all to check out the Server mode availability ... But back to the answer: A: The EH can kill itself by using the EH action "Terminate Handler!" This is equivalent to the scriptsystem.abortAL();
You can also call
task.setExitRequested(true);
All of the above will terminate the current instance of the EH, e.g.
for a HTTP EH, it will terminate the current session, but the main EH
will still listen for incoming HTTP requests.
If you want to terminate the HTTP server thread (the one that listens
to incoming requests), you can use the code
task.getServer().setExitRequested(true);
Q: How can I terminate ALs and/or EHs from other AL/EHs?A: See the answer of the question below that deals with killing them all, killing one is just a special case.Q: How can I terminate my Server2005.02.28 A: If you are running 6.0:var session =
new
Packages.com.ibm.di.api.APIEngine.getLocalSession();
session.shutDownServer()
You can always terminate the Server by killing the JVM, but that is
probably what you asked about....
f you are running 5.2 and you don't want to kill the process through
the OS, you will need to kill all the EHs and ALs running on te server.
Answer
to 'Everything you wanted to know about killing it all':
2005.02.24
A: version
6.0:
Version 6.0 lets you use the server api (com.ibm.di.api.local or
com.ibm.di.api.remote) and it has a set of powerful methods. Here is a
short script example:
The below script kills all EHs one by one, and then the server itself.
var session =
new
Packages.com.ibm.di.api.APIEngine.getLocalSession();
// Stop EventHandlers
ehs = session.getEventHandlers();
task.logmsg("RUNNING EHs: " + ehs.length);
for(i = 0; i < ehs.length; i++) {
var eh = ehs[i];
task.logmsg("STOPPING " + eh.getName());
eh.stop();
}
// Kill the server
session.shutDownServer()
// Stop
EventHandlers
x = task.runningEHs();
task.logmsg("RUNNING EHs: " + x.size());
for(i = 0; i < x.size(); i++) {
var eh = x.get(i);
// If I am an EH dont commit suicide
if ( eh == task )
continue;
task.logmsg("Stopping " + eh.getName());
eh.setExitRequested(true);
}
// Stop AssemblyLines
x = task.runningALs();
task.logmsg("RUNNING ALs: " + x.size());
for(i = 0; i < x.size(); i++) {
var al = x.get(i);
// If I am an AL dont commit suicide
if ( al == task )
continue;
task.logmsg("STOPPING " + al.getName());
al.shutdown();
}
Q:How do I get the error status of the AssemblyLine from an EventHandler that started the AL?A: I know the getResult() method to get the result of AssemblyLine execution from the EventHandler. However, for example, when an AssemblyLine calls the filesystem Connector with a CSV parser, and the CSV parser produces exeption by illegal format of data, we can't get error status of the AssemblyLine from the caller EventHandler. Is there any way to get error status of Assemblylines in the EventHandler in cases like this? A: Calling al.getStats().getError() gets you the java.lang.Exception object if one exists. This is the exception (error code and message) that caused the AL failure.LoggingQ: How can I see my TDI 6.0 AssemblyLine logs using AMC2?A: The logs generated by the SystemLog appender will be visible for AMC2, so you need to add a SystemLog appender to your server or AssemblyLine.Q: I want to be able to log *just* my messages, to create an audit log. How can I do that?(I don't want all the TDI messages) 2005.04.19 2005.04.20 Here are a couple of suggestions:
First define
(in the
prologe for example):
myLog =
Packages.org.apache.log4j.Logger.getLogger("myLog");
and then, instead of task.logmsg() use
myLog.info(msg);
myLog.error() etc ...
In addition you will need (as spelled out by Jason Williams) to define
the myLog appender itself in the log4j.properties file
to something like: (NOTE! The Threshold must be defined for this to
work.
)
log4j.debug=true
log4j.rootCategory=INFO, Default, myLog log4j.appender.Default=org.apache.log4j.FileAppender log4j.appender.Default.file=ibmdi.log log4j.appender.Default.layout=org.apache.log4j.PatternLayout log4j.appender.Default.layout.ConversionPattern=%d{ISO8601} %-5p [%c] - %m%n log4j.appender.Default.append=false log4j.logger.com.ibm.di.config=WARN log4j.logger.com.ibm.di.loader=WARN log4j.appender.myLog=org.apache.log4j.FileAppender log4j.appender.myLog.File=myInfo.log log4j.appender.myLog.layout=org.apache.log4j.PatternLayout log4j.appender.myLog.layout.ConversionPattern=%c %d{ISO8601} -- %p -- %m%n log4j.appender.myLog.append=false log4j.appender.myLog.Threshold=WARN Q: What's the correct approach in TDI 5.2 for writing output from many components to one destination? Can I configure multiple ALs and EHs to use the same logging specification through the TDI GUI.2003 That is, is it allowable to fill out the logging tabs for several ALs and EHs with exactly the same details. So if I specifiy the same output file for each AL, and the same Appender type and format, and the ALs execute in parallel, will they cooperate and share writing to the file, or will each AL think it owns the file and overwrite each others output? ALso if each is an Appender of type IDIFileRoller or DailyRollingFile, when and who decides to roll the file? My goal is to ensure that I capture all the output from an EventHandler and the ALs it calls. I would also like to be able to get the log file to roll periodically (Daily or on a size basis). A: When starting an AL from an EH using the Start AL Action, you have the Internal Logfile checkbox to indicate that you want to them to share log appenders (including, sharing the same files for output if appenders have been set up for file output).You can also do this by hand, however, there are limitations in the current version. For more information on how to get around the limit in parameters in the main.startAL() function, check out my posting in this forum with the subject: "Re: IDI 5.2 logging: relative vs absolute file names AND consolidating "main-"ALs and "sub-"ALs" There I describe how to use a java.util. Vector to pass any number of params to an AL you want to start, including the logging context of the calling process. ScriptingVariables and PropertiesQ: How do I get/set Java properties?2005.04.13 A: You use java.lang.System methods:// Get all
properties:
java.lang.System.getProperties()
//
get a property
task.logmsg("com.ibm.di.store.database
= " + java.lang.System.getProperty("com.ibm.di.store.database"));
// set a
property
java.lang.System.setProperty("com.ibm.di.store.jdbc.driver","com.ibm.db2j.jdbc.DB2jDriver");
Q: How do I get a variable to persist across iterator iterations when several spawned ALs are involved?I'm running an AL, which spawns off an AL, which spawns off an AL which needs to access a variable from the second AL across every iteration. I can pass the variable okay via the IWE (Intital Work Entry), but I can't figure out how to take the value from the IWE and persist it across every iteration. Is there some sort of global variable concept or something like work with a broader scope? A: Check out the system.setPersistentObject and getPersistentObject (and deletePersistentObject) if you want this info really persisted -- these methods use the System Store. Or you can try setting Java Properties, which means you can pass stuff around in the same JVM. For example,java.lang.System.getProperties().put("myKey",
someJavaObject)
will save someJavaObject to Java properties using "myKey" to
access it later with:
java.lang.System.getProperties().get("myKey");
This means the object is only kept in memory and your solution
will perform better than persisting it to the System Store (cloudscape)
database.
Of course, you can always set an attribute equal to task in the
calling al and then pass it to the child AL. The task variable is a
pointer to the current AL. Then you can call the task.getWork()
function. As long as you keep passing in a reference to the same
AL, the getWork() method should return the same Entry object.
Q: I have my configuration parameters (such as filenames) in an external property file. How can I get access to those values through scripting?2005.02.10 A: Here is one way of doing it. I have this function in a script library which I can then either copy into, or include in my Config.// Return an
External
Property value
//
function getExtProp( propName, defaultValue ) // Returns Java String {
try {
retVal = main.getMetamergeConfig().getExternalProperties().getParameter(propName );
if ( retVal == null )
retVal = defaultValue
} catch (errMsg) {
retVal = defaultValue;
}
return retVal;
}
It wraps the call in a try-catch block because if the Ext Props are notdefined (i.e. assigned to a file) then the call throws and exception.This way I always get a value back. Q: Why can't I do setParam() on my Connector? Why are only some of the documented methods available to my Connector?05.03.15 A: There are two Connector objects, and you might be using the wrong one. When you see a Connector in an AssemblyLine in your ConfigEditor, you 'see' the common wrapper class, com.ibm.di.server.AssemblyLineComponent. However, it has an interface part (for example for LDAP com.ibm.di.connector.LDAPConnector) that is specific for the type of Connector that you use. Some of the methods you are interested in belong to the wrapping class (com.ibm.di.server.AssemblyLineComponent) and some belong to the specific target class (com.ibm.di.connector). If you look up the javadocs, you will find the methods belonging to one and the other: Some methods have the same name in both. The wrapper class com.ibm.di.server.AssemblyLineComponent has a field called connector that is the link between the high level common AssemblyLineComponent and the specific instance of (in the case of LDAP) com.ibm.di.connector.LDAPConnector tied to it. So if you had trouble with this, chances are that you will have to write things like thisConnector.connector.setParam("filePath", "examples/scripting/sample.csv" )Inheritance and object sharingQ: After overriding an inherited Connector, I want revert to the parent behaviour. Can I do that without recreating the Connector?A: Primary AL component inheritance can be set (and broken) via the Inherit From box/button at the top of the component Details window. For those components that offer multiple configuration tabs (like Connectors and Functions), specific inheritance for each aspect is controlled by the Inherit From box/button at the lower right-hand part of the tab. Any component tab can be set to:
Q: Can we share object (i.e. work object) between multiple AssemblyLines? In the other words, what is the scope of object in TDI? I expect scope of object is defined in thread boundary. Is this my understanding correct?2005.02.21 A: Since each AL (and EH) runs in its own Script Engine instance, all variables are local to the AL (EH). If you want to share them, you can use various techniques:
Q: Is there a way to share Connectors between process ?A: Here's an example of passing a Connector into an AL (retrofitting the deprecated run-time provided Connector feature). It does this by setting a Java Property in the EventHandler: // Save as a Java property, making it global for all // threads in this VM. java.lang.System.getProperties().put("RTConn", useConn); Then in the AL I have a scripted Connector that looks for this property and grabs the Raw Connector. Here is the selectEntries() method: function selectEntries() { // Get globally saved Connector (must be pre-initialized) RTConn = java.lang.System.getProperties().get("RTConn"); if (RTConn == null) task.logmsg("** Could not get System property called \"RTConn\"") else { RTConn.selectEntries(); } }Runtime ConfiguratiomQ: Can I change Connector mode at runtime (for example, from Update to Add-only mode)? Are there better alternatives?2005.05.02 A: You cannot change the mode of a Connector after the AssemblyLine has started. But if you start the AL with startAL(), then you can modify the config object before you start the AL. Usually, there are better and easier ways to obtain what you are looking for, so read on. Assume the Connector is called MyConnector.in AssemblyLine MyAL, and you want to change the mode to ADDONLY_MODE for all AssemblyLines started in this server (from now on). Here is the syntax (but read on before you do this).connectorconfig
=
main.getMetamergeConfig().lookup("AssemblyLines/MyAL").getConnectorByName("MyConnector");
connectorconfig.setMode( connectorconfig.ADDONLY_MODE );
Note that you will need to be careful with your attribute maps:
Probably the best is to set up (in advance and with the CE) the
attribute maps for the Connector for all modes you want to set.
Now, in the spirit of 'There are many ways to do it in TDI', let me
suggest a couple of alternative paths:
High Availability/Defensive programmingQ: I can not Connect to my data-source, but the On Connection Failure hook is not called. This must be a bug, no?2005.05.27 The On Connection Failure hook in TDI 6.0 should really have been called On Connection Loss. It is part of the AL flow and will not be called before initialization has been successfully completed and the AL flow invoked. To catch initial Connection errors, you should place code in your On Error hook in the Prolog section of the Connector. Do not try to reinitialize the Connector here as you will create a recursive loop and because there are non-initialized javaobjects at this stage that can cause problems. The easiest way to handle initial Connection errors is simply to restart the AL in the prolog On Error hook:
main.startAL("myName");
system.abortAssemblyLine("message ....");
Make sure that you pass the TCB on if this AL has one.
Another alternative is to have kicker AssemblyLine that starts your
business AssemblyLine: If one of the Connector fail, you can
catch the error and take whatever action you want to.
Finally, for the adventurous, you can try to sniff out whether a
Connection will fail or not in the Prolog Before
Initialization hook.
(Note, the below does not guarantee you that you will
ultimately
get a connection)
Something like will do the job:
maxtry = 10; // 10 * 60 secs = 10 minutes
before
we give up
while (--maxtry > 0) {
try {
thisConnector.initialize();
thisConnector.terminate();
// remember the Connector will be initialized later since this is
before initialize.
break; //
success - exit while loop
} catch
(error) {
task.logmsg("Initialized failed ... trying " + maxtry + " more times");
thisConnector.terminate(); // removing this will might cause
you
to bleed memory
system.sleep(60);
}
}
//
if (maxtry < 1)
system.abortAssemblyLine("Unable to reinitialize Connector " +
thisConnector.getName());
Q: When I reuse Connectors, where is the best practice for placing of the reconnection logic: In the initializing Connector (IC), or in the Connector that reuses the initialized Connection (RC)?2005.05.30 A: The Connection itself is what is reused through the @-syntax, not the Connection parameters and hooks. Initialization Connection errors must be handled during Initialization, there is no easy alternative here. When it comes to the Reconnect Tab and the On Connection Failure, you will usually get two sets of them: One for the initializing Connector (IC) and one for the reusing Connector (RC). There is no inheritance of the hooks/tabs, so all reconnect configuration need to exist in the RC (and possibly in IC as well if IC uses the Connection as well). However, what you might want to do, is to place your Connector in the Connector Library (LC), and have your IC and RC both inherit from LC. This way, you only have one place to update your reconnect logic should you want to.VariousQ: Is it possible send an email (SMTP) from ITDI?A: Yes, check out the SendMail Function (FC) and the the UserFunctions class in the Javadocs, available as the system bean in ITDI AssemblyLines. In other words,system.sendMail("from@mail.com",
"to@mail.com",
"subject, "body" , "c:\\file.txt")
sends an email message. Make sure the mail.smtp.host Java property is
configured with the hostname of a valid SMTP server.
Parameters:
Q: I have a simple AL with an iterator Connector feeding into an update Connector. In the output map of the update Connector I have left the Mod checkboxes unchecked but when I run the AL the Connector complains if a record with the same data already exists.Shouldn't the connector simply compare and skip the existant record? I get the same behaviour when I use AddOnly mode. Do I need to add a trigger or should the default be to skip to the next record in the iteration loop? A: If you don't want to do anything when you find an existing record, you can just add code to the "Override Modify" hook. any of the following code would work.
Q: How can I obtain the actual connector object, or at least the actual Connector name?Inside the AL,all Connectors are wrapped with the AssemblyLineComponent class. You can always query its name with the getName() function. You can get the classname of any Java object by using the .getClass() method. To get at the Connector Interface itself (the CI is the actual protocol/API specific part of the Connector -- LDAP, JDBC, JMS, etc.) use the .connector member. As an example, let's say I have an AL with a Connector called SQL_People in it. The Prolog - After Init Hook of the Connector contains this code:
task.logmsg("
Conn Name: " + thisConnector.getName()
);
task.logmsg("
Conn Class: " + thisConnector.getClass()
);
task.logmsg(" Conn
Interface: " + thisConnector.connector.getName()
);
task.logmsg("
CI Class: " + thisConnector.connector.getClass()
);
This will result in the following output.
15:27:34
Conn
Name: SQL_People
15:27:34
Conn
Class: class com.ibm.di.server.AssemblyLineComponent
15:27:34
Conn Interface: SQL_People
15:27:34
CI Class: class com.ibm.di.connector.JDBCConnector
Note how both the ALComp and the CI return the AL Comp's name
with the getName() method. Btw. thisConnector
is a special variable that references the Connector where this script is run.
Q: How can I transform/hash a (password) string in cleartext with MD5?2005.06.22 From Rudy Sutjiatofunction
md5Hash(inputString)
{
var msgDigest = java.security.MessageDigest.getInstance("MD5");
var javaStr = new java.lang.String(inputString);
var digestStr = msgDigest.digest(javaStr.getBytes());
return system.base64Encode(digestStr);
}
var hashStr =
md5Hash("String to hash");
ConnectorsGeneral ConnectorQ: The Connection to my source (JDBC, LDAP, JNDI etc) fails, but the On Connection Failure hook is never called. 2005.03.08 A: The On Connection Failure hook is part of the AssemblyLine flow and will only be called after you have started to process your data after all the Connectors have been initialized. The hook should really be called On Connection Loss as it is not called if you never have managed to initialize your Connector. What you should do is to call the On Error hook in the Connector prolog to catch initial connection errors.Domino Change Detection ConnectorQ: Is it possible to clear the state store that the Domino Change Detection Connector uses?I need to script the equivalent of pressing the 'delete' button on the config page, in order to have the connector iterate through all the entries in the Domino database again. 2005.06.22 A: The following code in your AL Prolog - Before Init will do the trick:system.deletePersistentObject(YOUR_SYSTEM_STORE_KEY);
Domino User ConnectorQ: Is it possible to specify the user's mail template when creating the user?2004.04.12 A: Yes, this is possible from TDI 6.1 and later versions. See the TDI Reference Guide for more information.Q: Is it possible to change only the last name of a user already registered in Domino (keeping existing mailbox etc.)?2004.04.12 A:
File ConnectorQ:I have problems reading a UTF-16 encocoded XML file (on AIX)I'm using IDI on AIX to read a XML-file that's UTF-16 encoded. This gives some problems, which we are trying to solve in several ways, one of them is the native2ascii utility. Most of the data is read fine, but any accented characters gets very wrong. A: You can specify the encoding in the XML Parser. The supported charset encoding is defined by the Java implementation. Here is an excerpt from the documentation for the Charset class ( http://java.sun.com/j2se/1.4.2/docs/api/java/nio/charset/Charset.html )*Standard charsets*
Every implementation of the Java platform is required to support
the following standard charsets. Consult the release documentation
for your implementation to see if any other charsets are supported.
Charset and Description
US-ASCII Seven-bit ASCII,
a.k.a. ISO646-US,
a.k.a. the Basic Latin
block of the Unicode character set
ISO-8859-1
ISO Latin Alphabet No.
1, a.k.a. ISO-LATIN-1
UTF-8 Eight-bit UCS
Transformation Format
UTF-16BE Sixteen-bit UCS
Transformation
Format, big-endian byte order
UTF-16LE Sixteen-bit UCS
Transformation
Format, little-endian byte
order
UTF-16 Sixteen-bit UCS
Transformation Format,
byte order identified by
an optional byte-order mark
Q: Can I control the end-of-line character(s) in the File Connector? Q: To write data into file, we choose CR+LF or LF to signal the end of a line. UNIX uses LF and Windows based systems use CR+LF. Can we specify control code for linefeed explicitly in the FileSystem Connector independent of the OS platform. (for example, can we specify to use CR+LF in TDI running on a UNIX machine?) A: You can control how the JVM writes files by setting the following Java property: java.lang.System.setProperty("line.separator","\r\n"); for example, in the "Prolog - Before Init" Hook. Then all files will be written with CR+LF as separator. Note that unless you reset the java property, all files in this JVM instance will be written with CR+LF as line separator. As such, it might be a better idea to postprocess the file with a nice little script written in your favorite tool, e.g. perl, sed...Q: Is there a function to check the type of the value when reading the CSV file using the filesystem Connector with the CSV parser? For example, is there a function to generate an error when there is a value that cannot be converted to an integer where an integer value is expected?CSV files are files consisting of text snippets; the CSV Parser returns every element of the file invariably as String. Type casting to an Integer would typically occur in Attribute Mapping (Advanced Attribute Mapping), as inret.value =
conn.numGet("someAttribute");
where someAttribute is the name of the field in the CSV file. After
Attribute Mapping the attribute now lives as a number.
Q: Is there a method that may be used to validate if the header record of a CSV file contains the expected field name.The issue I have is the customer has changed the field names without telling the operations group and as such IDI spat the dummy. I would prefer to be able detect there has been a change and abort the AL.
Q: May I risk double encryption if I read from LDAP and then produce an LDIF file to bulkload into others servers?I am trying to build an ldif file with a limited set of attributes that can be used for bulkloading another set of servers. I am using the TDI ldap Connector ( to connect to the source server) and the ldif parser to build this ldif file to be fed on the target server. The password encryption algorithm on the source and the target server is SHA. Will the userpassword attribute be doubly encrypted when I use this LDIF file? A: When a common hash algorythm is used, the value of userPassword attribute is expressed like {sha}xxxxxxxxxxxx (xxxxxxxxxxxx is hasshed value). If you modify userPassword attribute with the value includes hash scheme like {sha}, the ldap server should simply update the value without double encryption.HTTP ConnectorQ: Is there a way of getting access to the tcp.* attributes of the HTTPConnector?2005.02.10 A: No. There is the TCP Connector, but most people prefer not having to deal with it for http ...JDBC ConnectorQ:Tell me more about the rollback capability of the JDBC ConnectorAre there any conditions that needs to be met for a rollback to occur? For example, if you have configured to commit on Connector close time, and if an error occured during Iteration process, will a rollback of the DB occur? Or will it commit despite the error? Is there also a method provided to rollback manually? A: There are three commit modes available for the JDBC Connector (although they will depend on the data source you are connecting to). These options are:
Q: I want to do my own SQL-calls. Can I?2005.03.02 (This is well documented in the official documentation) A: You can use the ConfigEditor to set the SQL statement used when initializing an Iterator, but not for lookups, adds and deletes (this will be available later). However, assuming your JDBC Connector is called DBConn you can always use:DBConn.connector.execSQL("INSERT
INTO...");
Of course, you could replace the INSERT command with anything.
Another method, execSQLSelect(), is more suited for performing a select
and enumerating the result set (and used by the Connector when
initializing an Iterator).
Q: How can I call stored procedure from my JDBC Connector?2005.03.02 The JDBC Connectors "connection" property gives you access to the JDBC Connection object created when the connector has successfully initialized. In other words, if your JDBC connector is names DBconn in your AL,var con =
DBconn.connector.connection;
will give you access to the JDBC Connection object (instance of
java.sql.Connection).
Here's a code example illustrating how you can invoke a stored
procedure on that database:
// Stored
procedure call
command = "{call DBName.dbo.spProcedureName(?,?)}";
try {
cstmt = con.prepareCall(command);
// Assign IN parameters (use positional
placement)
cstmt.setString(1, "Christian");
cstmt.setString(2, "Chateauvieux");
cstmt.execute();
cstmt.close();
// TDI will close the connection, but
you might want
to force a close now.
DBConn.close();
}
catch(e) {
main.logmsg(e);
}
Q: I have problems with the JDBC when upgrading from 5.1.2 to 5.2. Has anything changed that is not documented?We developed a complex AL with several JDBC connectors in IDI 5.1.2. I expected that running the AL with the new version 5.2 should be ok. The first tests showed that this is not the case. Starting our JDBC connector, we got an error: "Can't start a cloned connection while in manual transaction mode" I searched in the internet, found, that I have to define an additional JDBC connection property "selectMethod=cursor" Then it worked. So obviously the JDBC connector changed in its behavior. The readme file mentions that there is also the new jdbcCommit. But what's the default value when I run my 5.1.2 AL in IDI 5.2? Are there any other changes in the JDBC connector? Is there an common overview about the connector changes? A: The JDBC Connector has been changed in 5.2. In order to allow for the new Commit Mode parameter, the connection is initialized with autoCommit(false) and now calls commit() as specified by the parameter. This will affect some existing deployments AND SHOULD HAVE BEEN DOCUMENTED -LDAP ConnectorQ: Is it possible to retrieve LDAP operational attributes such as modifyTimestamp with the LDAP Connector? I can search on this attribute, but I can't seem to retrieve the attribute value.2005.04.19 A: Operational attributes are only returned by the LDAP server when they are explicitly requested. So, you'll have to specify the "Return Attributes" parameter in the LDAP Connector to get these. Note that in 6.0 and before, you will have to explicitly list all attributes you are interested in: If you don't you will get only the operational attributes that you listed ... If you want all default attributes + modifyTimestamp, you could use * like in:*
modifytimestamp
Notes ConnectorQ: Can I use update Rich Text Lotus Notes documents via the Notes Connector ?2005.05.30 A: Sorry, the answer is no. The entire Java Notes toolkit is organized in the following way - if you want to create a "child" object you must call appropriate "createXXX" method of the "parent" object. So RichTextItem object can not be created unbound. However since the Connector exposes two of the Notes root objects: "Databse" and "Session", the user can navigate to the right Document (clumsy in scripting) and create RichTextItem bypassing almost the entire Connector. However, the current implementation (6.0) will 'dumb down' RichTextItems in the AttributeMap in both directions - input and output.EventHandlersWeb Services EH (TDI 5.2)Q: How do I handle SOAP fault in the TDI 5.2 WebService EH?In the Reference Guide version 5.2, the following is mentioned on how to return the SOAP Fault to the client using setError() method. (from page 157..)Note: If you need to return a SOAP Fault to the client as a result of your script processing within the Prolog or Epilog, then you can use the setError(java.lang.String aMsg) method of the WS EventHandler in the following way: var wsEventHandler = event.getProperty("event.originator"); wsEventHandler.setError("Some message..."); These two lines of code would cause the WS EventHandler to continue executing the Prolog/Epilog script until the end of the script is reached, and then WS EventHandler would generate a SOAP Fault message with <faultstring> containing " Some message.. ". You might need to do that if you implement a security mechanism in the Prolog for example.d 1. Can we only use <faultstring> with the setError() method? 2. If so, how can we return other SOAP Fault to the client? (Such as faultcode, faultactor, detail..etc.) A: Unfortunately this is a limitation of TDI 5.2. Web services functionality has been complete redone and dramatically extended and improved for the next release. Password CatchersNovellCan I use TDI for solutions that involve synchronizing Novell Passwords?2005.06.15 Unfortunately you cannot. As you know, TDI provides certain plugins that can grab password-changes in TDS, Active Directory etc. Development has looked into this earlier, and we have some problem getting access to the necessary APIs. (If you know that it can be done, let me know!) Novell has published some documentation on password synchronizing in their solutions, and have tools/functionality for it: http://www.novell.com/documentation/dirxml20/index.html?page=/documentation/dirxml20/admin/data/botfldy.html One thought you might want to consider- but might not be relevant: If the customer already has Novell products that can set passwords in AD, then TDI could catch the new AD password and propagate that to anywhere else.ADQ: Various questions on the AD password catcherI know that with idi 5.1 you could use the AD password plugin to capture user password changes when they do a ctrl + alt + del. The password is then stored in either LDAP or MQ. We're using TDI today as a supported method to update TAM for EBIZ GSO entries. Q1. for 5.2, is it possible to store the password back in the AD directory so that we can pick up the password change to update the GSO entries?A: Yes, you trap all password setting, not just ctrl+alt+del. If you write the pw to AD, you will also catch this change. Q1a. if thats possible, then can AD accept the schema changes to store the id and passwords?, i.e.: schemaPersonObjectName
ibm-diPerson
schemaUseridAttributeName ibm-diUserId
schemaPasswordAttributeName ibm-diPassword
schemaExtendedDataAttributeName ibm-diExtendedData
A: ??
1b. i guess the challenge then would be that if there are 4 domain
controllers, then which server would we be storing the data in?
A:
Aye, there's the
rub. You can of course install the pw catcher on all for dc's.
Q2. is there a way for this storage to work, if we do not use
either
LDAP or MQ for the temporary storage before the assembly line kicks in
to get the list of user ids to update passwords with?
A:
No.
I have a couple of questions regarding the Password Synchronizer Plug-in for Windows. A customer needs to intercept password changes on their PDC so that the password change can be added to the Rational ClearQuest userid/passwd repository. Periodically, their PDC is synched (one way) to their iPlanet LDAP server via iPlanet. Once the password is written to either the PDC or LDAP it is irretrievably lost due to the one-way encryption used. My two questions are: Q: If the Windows Password Sync Plug-in is installed on the client's PDC will it also intercept password changes made from a user's desktop machine or will the Plug-in need to be installed on each desktop client?A: The plugin need only be installed on the domain controller. Any domain user pw changes will be caught and sent to IDIQ: Is possible to re-route the intercepted AD password via an AL to some place other than LDAP or MQE? The ideal solution would be to sent it on to ClearQuest (my target system) since once it is written to LDAP it cannot be unencrypted (client uses SHA).A: The plugins only support either the deprecated LDAP method, or the preferred MQe transport method. So you will need to set up an IDI server to catch the pw change and then write it to ClearQuest.LDAP/GeneralQ: When user changes password he is prompted for the old and new password. Both passwords are usually provided for a password change. This means that both passwords are temporarily stored on an LDAP server. Right?A: Only the new password is caught and transmitted by the password catcher. So if you want to have the old password as well, you will need to store it somewhere in a form that you can decrypt yourself and use.Q: What is happening whenever the LDAP server where the passwords are temporarily stored doesn't answer (because it has... say crashed)? Is there a way to recover from such a situation?A: As far as what to do if a data source is unavailable, you will need to design your Config to handle situations like this. For example, reading nondestructively from the JMS queue. Or you write these password changes to a local store (like a Cloudscape database table) which you can be sure is always available. Then a secondary AL can be used to pick up these change events from your local "FIFO" table and driven to target systems. If this AL fails, it just picks up where it left off, not deleting from the local store until the pw has been successfully dispatched.Q: Where are the IDI 5.2 Password Synchronization Plug-ins mentioned in Appendix A of the 5.2 Reference Guide?The plug-ins are available for download via the Passport Advantage site.Systems/TechnologiesActive DirectoryQ: I cannot update Active Directory passwords. What can be wrong?I have an AssemblyLine with an LDAP connector that is supposed to update an Active Directory password. The connector is configured NOT to use SSL (at this stage), and the option "Auto Map AD Password" is checked. The work entry object contains a UserId attribute which contains the sAMAccountName, as well as an attribute called userPassword, which contains the password with which to update AD. I don't have an output map specified. Link criteria is for the sAMAccountName to equal the work entries UserId attribute. Problem is that when I run the AL I get a message saying: [UpdateAD] Task is not allowed to modify entries The connector is in update mode. I have looked at the examples, and seem to be doing things correctly, but this error is perplexing. Can you guys shed any light on this one ? A: You have to use SSL in order to update AD passwords Also, you will need to specify an Output Map, since a Connector uses its local storage object (conn) for all communications with the underlying data source. Attribute Maps are the link between the ALs data store (work) and those used for data source access.Q: Using TDI, can I create an AD group and add memberships?2005.06.10 A: With TDI, the easiest is to use the LDAP Connector to create group objects and add members to them. Also, when the groups already exist, you can add values to the memberOf attribute of the user records of AD to set them as members of these groups.Q: I am looking to copy some of our groups in Active Directory to our LDAP server, but the problem I have is that the AD group members are based on a DN that uses CN, but our ldap server uses a DN based on UID. What is the best way to convert the AD groups?Example: AD group: CN=TestGroup,ou=groups,dc=us,dc=company,dc=com member=CN=Joe User,ou=people,ou=sales,dc=us,dc=company,dc=com member=CN=Sally Smith,ou=people,ou=marketing,dc=us,dc=company,dc=com LDAP group: CN=TestGroup,ou=groups,dc=company.com uniquemember=uid=juser,ou=people,dc=company.com uniquemember=uid=ssmith,ou=people,dc=company.com I think I would have to write code to go through each member of the AD group and do a search against AD using the user's CN to get the user's samaccountname. But I am not sure how to do this. I think I know a way I can do this, but I have a question. I am thinking about going through each member of the AD group and then I want to open another AD connector and do a search using the user's CN and find the user's samaccountname, which I will use to populate the membership of the LDAP group. My question is: From my connector in my assembly line, how do I initiate another connector, passing the user's CN and getting back the user's samaccountname. A: The easiest way is to have a second AD Connector in your AL, but set to Passive State. That means it will be powered up when the AL starts, but never actually used by the built-in AL logic. Since all Connectors in your AL are available as script variables (which is why you need to name them like you would variables -- e.g. no spaces or funny characters), you can then access them directly from your scripts. Set up the Link Criteria to lookup an entry based on an attribute that you set up in the work Entry. For example, if you are looping through the members of a group, in this look you can do something like this:
...
work.setAttribute("$dn",
work.getAttribute("members").getValue(i));
returnEntry =
ADLookupConnector.lookup(work);
if (returnEntry = null) {
...
There is also a lookup() function that takes two strings: the
name of the attribute you are basing the search on, and the
value. This method implies an exact match.
Q: Is it possible to retrieve all changes within an AD Cluster (multiple replicated Domain Controlers (DC)) with one Changelog Connector connected to only one of these AD Servers ?2005.05.19 All DCs in a domain hold a full replica of the domain partition, which has all user and group data. TDI therefore can get all changes for that domain from any domain DC. Global Catalog DCs hold partial replicas of domains other than their own, but to get full changes from those domains you need to monitor a DC in those domains. Typically companies have 1-3 domains, so the number of TDI connections (not including redundancy considerations) is fairly small.Q: I need to modify AD accounts to not expire passwords and not allow users to change their account password. Can TDI do that?2005.06.10 Yes it can, it simply done by setting the userAccountControl attribute. For legal values (and much more tuning, see http://support.microsoft.com/default.aspx?scid=kb;en-us;Q305144 )eTrust CA-ACF2Q: Can I use TDI to integrate ACF2 and a LDAP directory such as TDS?2005.06.15 2005.05.24 A: Yes. You can use TDI to synchronize TDS and ACF2, and it has been done in several customer deployments. Typically, the changes propagated into ACF2 are made using CA-LDAP interfaces. You can update nearly all ACF2 attributes, enroll new users, etc. using these interfaces. The changes propagated from ACF2 rely on CA-LDS. LDS stands for LDAP Directory Services, and it writes all changes occurring in ACF2 out to an LDAP directory, at a place you can configure. This mechanism can be used to propagate ACF2 adds, modifies, and deletes, as well as password updates. Note that Suspending an AFC2 account seems to be a problem. There does not seem to be a way to do it through LDAP. One way of doing it is to force a suspend by writing code to try to log on multiple times with a bad password, thus triggering password suspend. It is also possible to communicate with ACF2 using the TSO command line Connector that is included with the zOS version of TDI 6.0, but we have no knowledge of such soultions in production.HTTPQ: Do I need configure a HTTP client Connector in the AssemblyLine in order to use system.httpget() at all?My customer tried to use system.httpget() to retrieve some data from web server. But he just got "java.lang.Exception: Cannot get configuration for Connector: metamerge.HTTPClient2". It seems that TDI produces exception because there is no configuration for HTTP client. Should we configure HTTP client connector to use system.httpget()? (The customer doesn't want to use HTTP client connector itself. He just would like to use system.httpget() on other kind of connector than HTTP connector. A: No, you don't need any Connectors at all. There was a bug prior to version 6.0, so please check if there exists an APAR if you run into this. The bug exists in these functions: system.httpGet(), httpPost() and httpRequest(). Note that these functions use the HTTPClient Connector, so the problem may lie there... Sorry.LDAPQ: Can I modify my LDAP search base in runtime?2005.02.14 A: Yes. As a matter of fact you can modify most parameters that you see in the CE runtime if you know their name. (And their name can be found by clicking on their label and looking up their ... name). For example, to set the search base, assuming that this is in a hook before search is performed:thisConnector.connector.setParam("ldapSearchBase","ou=people");
Note that if you want to do the following for an iterator, you must
either set the parameter before Connector initialization (for example
in the Before Initialize hook), or reinitalize the Connector:
thisConnector.connector.terminate();
thisConnector.connector.initialize(null);
Q: I have problems modifying attributes in LDAP.I am trying to use an ldap Connector to modify values of multivalue attributes in the ldap. I am wondering if anyone might have a sample of code they have used to modify attributes in an ldap connector using advanced mapping or hooks that I could take a look at? A: (Longish Answer)
A suggestion:
I'm using the TDS event handler to monitor users being added to the
directory. I've set my action items as follows:
[Set Property/Attribute]
entry.inputdn = event.getString("targetdn")
[Run AssemblyLine]
/AssemblyLine/IdentityProvider (EventEntry) doesn't
wait
My AssemblyLine defines an initial work entry attribute called inputdn.
My EventHandler receives the add user notification ok and the targetdn
is correct if I dump out the event object. My AssemblyLine is started but
the inputdn is empty rather than containing the value in targetdn from
the EventHandler.
If instead I use
[Set Property/Attribute]
entry.inputdn = "cn=testuser,o=ibm,c=au"
then my AssemblyLine receives this correctly so I'm guessing that the
syntax for retrieving the targetdn (ie. event.getString("targetdn")) is
incorrect. Any suggestions?
Comment to the above suggestion:
This is not the only way to handle this, but I have used the
following with multivalue attributes in an ldap connector.
Q: How can I populate slave LDAP servers with only part of an LDIF dump (new entries)An LDIF file is normally used to export data from one directory and import into another. However, this exports the whole directory content. Is it possible to create an LDIF file that contains only the changes (since the last LDIF export)? If yes, do you know where I can find instructions on how to do this? A: Assuming you are updating your directory through TDI. Simply tell the Connector that modifies your LDAP to also 'write' the changes it is about to make by in a file. You can then use this file instead to other directories that are supposed to be in the same state, i.e populated with the same directory data. In TDI, you would need some scripting. The good news is that it's one line of code: myPassiveFileConnector.add(current). This way, you invoke the in myPassiveFileConnector (case sensitive) of your AssemblyLine, configured in AddOnly or Passive mode, from the 'before applying changes' hook of your LDAP update connector. This will work if the update connector has got 'compute changes' ticked on (I would recommended it anyways). In the myPassiveFileConnector (supposingly pointing to a changeFile, in say, LDIF) set an attribute map like attribute copy from work attribute ret.value = work.getString(*)Q: I have 2 CVS files - and old and a new HR feed. Based on the delta from this I need to and add or a update in LDAP. How do I handle this the best way?A: The Delta feature is key to the solution; it's available for Connectors in Iterator mode. Delta allows you to detect changes between successive iterations, and marks the returned work Entry with an operation code: "add", "modify" or "delete", depending on whether this entry is new since the last run, has been changed, or was not found during the last iteration. In order to pass these changes to another system (or systems) you then have a number of options. For example, you can use two Connectors that are both configured for your output data source (i.e. the one that needs to know about the Delta changes). One is set to Update mode -- which will take care of adds and modifies -- and the other to Delete mode. In the Before Execute Hook of your Update mode Connector, you will have code like this: if (work.getOperation().equals("delete")) system.ignoreEntry(); ignoreEntry() means that this Connector stops processing the current Entry and passes it on to the next Connector in the AL. In the Delete mode Connector you have the complimentary code snippet (note the NOT operator -> !): if (work.getOperation().equals("delete")) system.ignoreEntry(); So now, the Update mode Connector will handle all Entries except "delete"s, while the Delete mode Connector processes these. You could also do this same job with just a single Connector for your output data source. Set it to any mode that you like (just remember that you must configure LinkCriteria in order for the Connector to be able to perform an update or delete), and put the Connector in Passive mode. Passive Mode means that the Connector will get fired up along with all the other AL Connectors at the start of the AssemblyLine execution, but that the Connector itself will not be powered by the built-in AssemblyLine behavior. It will just sit there "passively" until the AL completes, and then it will close its connection. However, you will drive this Connector directly from script (e.g. Hook or Script Component). So, you can add a Script Component to the AL at the point where you want to do the actual add/modify/delete. This script will look something like this (when you see "myConnector", substitute this with the name of your output Connector -- all your AL Connectors are available as script variables): if (work.getOperation().equals("delete") myConnector.deleteEntry(work) else myConnector.update(work); Alternately, if the LDAP source contains all the changes from previous LDIF one could just use the ldap to handle the lookups ( the delta) instead of trying to handle a delta between 2 files. That means that the HR feed will be the iterator and the ldap will be in lookup mode in order to find changes. Then use the hooks to control if the object exist or not.Q: I get an Object Class Violation when doing an AddOnly from Domino to TDS (Update works fine). Whats wrong?I'm having trouble using an "Add Only" (Update works fine) connection to add users to the directory, but when I do this, the add fails complaining about "Object Class Violation." I tried just adding the required attributes and then I included all the ibm specific attributes as computed fields for this realm and added a $dn (like in my other records) as a computed FQDN, but still no dice. Any ideas? I include a dump of the conn before the add and a dump of the error when it fails as follows:10:56:14
>>>
Adding: Ben Lopez
A: The
objectclass
attribute lists objectclasses associated with users. But
in addition to standard "user" attributes like uid, cn, sn, etc., it
looks like you've included attributes from the ibm-realm objectclass
(ibm-realmusertemplate, ibm-realmadmingroup, etc.). These
attributes are not part of the objectclasses assigned to the entry, thus the
objectclass violation error.
The ibm-realm attributes should not be used in a user entry.
The ibm-realm objectclass is used with TDS (directory server) web admin to
define a collection of users and groups -- tells web admin where to
create and look for users and groups so that the person using web admin doesn't
need to know so much about LDAP or how users/groups are being
administered in the directory. Other applications could make use of the
realm definition too.
10:56:14 * Begin Entry Dump
10:56:14 Operation: generic
10:56:14 [Attributes]
10:56:14 ibm-realmgroupcontainer (replace):
'ou=groups,o=techflow'
10:56:14 ibm-realmusercontainer (replace):
'ou=users,o=techflow'
10:56:14 sn (replace): 'Lopez'
10:56:14 ibm-realmusertemplate (replace):
'cn=templateuser,cn=tfrealm,o=techflow'
10:56:14 $dn (replace): 'cn=Ben Lopez,ou=users,o=techflow'
10:56:14 ibm-realmadmingroup (replace):
'cn=TFRealm,o=techflow'
10:56:14 uid (replace): 'BLopez'
10:56:14 objectclass (replace): 'top' 'inetOrgPerson'
'ePerson'
'eDominoAux' 'organizationalPerson' 'Person'
10:56:14 cn (replace): 'Ben Lopez'
10:56:14 * End Entry Dump
10:56:14 >>> Add failed for: Ben with
exception [LDAP:
error code 65 -
Object Class Violation]
10:56:14 * Begin Entry Dump
10:56:14 Operation: generic
10:56:14 [Attributes]
10:56:14 status (replace): 'fail'
10:56:14 connectorname (replace): 'LdapAddUser'
10:56:14 operation (replace): 'addonly'
10:56:14 exception (replace):
'javax.naming.directory.SchemaViolationException: [LDAP: error code 65 -
Object Class Violation]; remaining name 'cn=Ben
Lopez,ou=users,o=techflow''
10:56:14 message (replace): '[LDAP: error code 65 - Object
Class
Violation]'
10:56:14 class (replace):
'javax.naming.directory.SchemaViolationException'
10:56:14 * End Entry Dump
Mail (snmp), EH and ConnectorQ: I need to get access to email attachments using the Mailbox EH (or Connector). Can I do that?2005.02.19 2005.02.21 A: Here's a script that displays the various part of a message body (received using the Mailbox Connector in Iterator mode). Note that we used the wildcard in the Attribute Map ("*") in order to get everything picked up by the Connector:task.logmsg("");
task.logmsg("========================================");
task.logmsg("---> " + work.getString( "mail.subject"
));
task.logmsg("---------------------------------------------------------------------------------");
bodyparts = work.getAttribute
("mail.bodyparts"
);
if (bodyparts == null) {
task.logmsg(" ** Has no attachements
**");
system.skipEntry();
}
for (var i = 0; i <
bodyparts.size(); i++) {
attch = bodyparts.getValue( i
);
task.logmsg(
"Attachment #" + i
+ " : " + attch.getSize() );
dh =
attch.getDataHandler();
if (dh == null) {
task.logmsg(" ** No DataHandler **");
system.skipEntry();
}
}
task.logmsg(" name : " + dh.getName() + " content-type : " + dh.getContentType() );
DB2Q:I have been trying to Connect to DB2, but I've had very little success. I am using DB2 8.1. When I tried to use the type 2 driver I got an error about type driver not being supported. With type 4 the application freezes. However it will only freeze if it is actually able to connect to the DB. Any ideas on what is going on?A: Steps of connection to DB2
netstat -an |
grep tcp
=> to list the port number
Q: Can I Connect to DB2 on iSeries (AS400)?2005.05.13 Yes, here is what you need to do: You will need the correct driver for UDB on AS/400. There is are a couple of JDBC drivers but you need the one that runs on a non iSeries platform. This is downloadable in the IBM Toolbox for Java (http://www-1.ibm.com/servers/eserver/iseries/toolbox/ ). The driver is a jar file called jt400.jar that is zipped into the package. There is also documentation for the driver including JavaDocs on all the properties supported by the driver that can be specified in the URL in the iSeries Java Toolbox. As explained in the TDI reference for the JDBC connector, this file should be placed in the{TDI Installation Dirctory}\_jvm\jre\lib\ext directory. You have to shutdown and restart TDI after you move the driver there. According to the documentation for the driver, the driver specific configuration parameters that have to be provided on the TDI JDBC configuration panel would be: JDBC URL: jdbc:as400://hostname/default-schema where you supply the hostname of the iseries server and the name of the default schema to use. The schema part of the URL is supposed to be optional. JDBC Driver: com.ibm.as400.access.AS400JDBCDriver where you supply the driver name for TDI to load out of its \ext directory.Q: Why am I not getting the "IBMSNAP_*" attributes when reading from DB2 anymore?I used to have them in TDI 5.2, but they are gone in 6.0. 2005.02.18 A: These attributes have been moved in the 6.0 version of the Connector into Properties. So, for example, you can do this: conn.getPropety("IBMSNAP_OPERATION") To get the operation. And you can use scripted Att map to create Attributes out of them if desired. This is documented, but may be easy to overlook at first glance. See the Reference manual for the RDBMS Changelog Connector, where you find this on the first page:For
each Entry returned, control information (counters, operation,
time/date) is moved into Entry properties. All non-control information
fields in the change table are copied as is to the Entry as attributes.
The Entry objects operation (as returned by getOperation) is set to the
corresponding changelog operation (Add, Delete or Modify).
Oh, and here is a complete list of special properties:
Q: Can I encrypt the data between DB2 server and TDI?2005.10.17A: Yes, this can be done. On the DB2 side:UPDATE ADMIN CONFIGURATION　　AUTHENTICATION=DATA_ENCRYPT
On the TDI Side:
Put the following statements in the "Before initialize" hook of the JDBC Connector:
thisConnector.connector.setParam("jdbcUseProperties", "true");
thisConnector.connector.setParam("jdbc.securityMechanism", Packages.com.ibm.db2.jcc.DB2BaseDataSource.ENCRYPTED_USER_PASSWORD_AND_DATA_SECURITY); DominoQ: How do I choose which Notes template (NTF) is applied for the user's mail-file when creating him/her with the Domino User Connector?2005.02.21 A: The short answer is that this is not possible to do with the Domino Users Connector. The reason is that the underlying Notes Java API does not provide such an option. An advanced user might try to implement that on its own, using both a Domino Users Connector and a Lotus Notes Connector and scripting through the public objects provided by the Lotus Notes Connector - this however requires good knowledge in the Notes Java API and the process of user registration.Q: Domino 6.x has an option of automatically setting the HTTPpassword when the user changes the password in the ID-file. Is this change catched by the interceptor ?2005.06.13 A: The answer is: No. The Domino password interceptor is NOT catching this kind of password change. If you carefully read the Domino Password Interceptor Guide you will find a hint on that. And if you have the chance to look into the Domino Interceptor and the way how it works, you will realize, that the solution is based on changes within the Person Document Form of the Domino Directory (names.nsf). This Person Document Form is changed to catch password changes and forward them on a kind of "On Save" event. This means the Interceptor only gets fired, if the change is done using the Person Form (using the native Domino Client or Web Client). It will not be fired if the password change is done "under the cover" by the Domino Server API.Q: I would like to know if there is a way to provide the "Server responding to request" for a Domino Delete Adminp request "Get Mail File Information for Deletion" through TDI or does it get set automatically when TDI code is set to delete the mail file and all replicas using this line- conn.setAttribute("DEL_DeleteMailFile", "2");Jun 02 2006TDSWe want to process TDS events sequentially. We want to control each thread to start processing after the thread to handle former change number ends. How can we do that?It seems like TDS event handler kicks each thread to handle change event immediately when it detected it from TDS. And my understanding is that TDS event handler doesn't kick each thread to process TDS event sequentially after former thread to handle former change number ends. The thread to handle latter change number ends its process earlier than that of former change number doesn't fits our requirement. I couldn't find suitable method in Switchboard class in IDI Java Doc. Does anyone have any good solution to this matter? I'm sure LDAP changelog connector is one solution. But I feel it's a little bit troublesome to iterate to find new changenumber and track latest changenumber manually. A: Set up the EventHander to start up the AssemblyLine for processing the change. Have this AL use the ChangeLog Connector to start reading changes. Note that in the 5.2 system,the Changelog Connector automatically keeps track of the last changenumber by using the System Store. Check out the docs on the System Store feature, or the Changelog Connector for more information. When you start up the AL,make sure you tell the EventHandler to wait until it is finished before continuing. Doing this in script means calling the join() method:
var al = main.startAL(
"ProcessChangesInLDAP" );
al.join();
The above example assumes you have an AL called "ProcessChangesInLDAP".
RACFQ: Can I use TDI to access RACF?The easiest way to access RACF information is via LDAP. Here is a link to a doc in the z/os library that has a chapter on how RACF data is represented through the LDAP protocol. There is an IBM document named "OS/390 V2R10.0 SecureWay Security Server LDAP Server Administration and Usage Guide" Chapter 17 deals with accessing RACF information and how the data is mapped to LDAP attribute names. This requires the z/OS administrator to install the z/OS LDAP server and configure the SDBM backend. It does not require DB2 to be installed or set up on z/OS.Getting OrganizedQ: Does TDI has support for team collaboration through for example a revision managing system?Q: Does TDI manages exclusive access of the configuration at a time.2005.02.10 A: There is no revision managing system integrated with the ConfigEditor and neither does it lock config files. However, since the config files are XML, you could adding CVS/CMVC/RCS tags s.a "$ID:" in the comment tab of AssemblyLines. By doing so you will be able to lock your files using your favorite revision managing system. The config file being XML also lets you easily 'diff' different versions of a config file. Keep in mind that unless you use 'includes', everything is in the same config-file, and you will not be able to use automatic versioning for multiple ALs/Libraries in the same Config. A workaround is to have multiple config-files, all using common libraries and containing only few AssemblyLines. Since TDI 6.0 supports multiple config-files in one server, this wont force you to have multiple JVMs running.Q: Can I setup IDI as multiple Windows services on the one machine? I want to keep each 'solution' in a different configfile.2005.02.10 A: Not really. But in version 6.0 you can have multiple configuration files in one server (the one running as a service).TDI How does it workQ: Where can I find a list of error codes return by <insert your favorite Connector here...>?2005.02.22 A: The Connector Interface (that usually speaks to a driver), will return the error codes that the driver returns, and that often returns the error codes thrown by the underlying system. So you will need to go into the target system to find those messages. For example, TDI's LDAP Connector returns the error code that was thrown by the directory server. Most of the LDAP codes are standardized, so a quick look at http://www.directory-info.com/LDAP/LDAPErrorCodes.html should provide you with the info you're looking for.Q: What is the purpose/function for the "Java Class" and the "Native Syntax" in the "Schema tab for a Connector"(It seems like the above is not being used for any type checking.. ) A: The Native Syntax columns is not used to determine the type of Java object to create for a given attribute. This is hardwired into each component. This column of the Schema tab should not be editable (but was in early versions of TDI)Expanding TDIWriting your own ConnectorsQ: How do I add code to the UI (buttons) of my homebrewed Connectors?2005.04.14 Calling this a FAQ is maybe stretching the term, but since it is not well documented elsewhere : A: There is a parameter called formscript in the idi.inf file (that you can find in miadmin.jar) This parameter takes any type of javascript, and is typically used to define functions that can be used e.g. when buttons are pushed. See e.g. ComplexTypesGenerator/idi.inf for an example of formscript. Older Connectors use a different scheme, but it is not recommended: For example, the Contexts button in the LDAP Connector is placed inside the file ASForms.txt. (The file can be extracted from miadmin.jar)Writing your own ParsersQ: In a scripted Parser, how can I control the encoding (e.g. UTF-8) of the output stream?2005.10.05 A: Character set encoding is handled by Parsers, and not the Connectors that use them. As a result, many Parser include a parameter in their Config tab for setting the encoding to use. This is not possible with the Script Connector since there is no Config tab -- just the Parser code itself. Scripted Parsers have a number of pre-defined variables, like entry that provides access to the Entry object to be written. For Parser output (e.g. in the writeEntry() method), there is a variable called out: A BufferedWriter object that has been set up for the Parser to use for writing. This BufferedWriter is attached to the output stream that is setup by TDI for the Parser. However, this stream is configured to use the default character encoding of the underlying platform/OS, which may not be what you want. So instead you must define your own writer. This can be done in a single line of JavaScript from within Parser code:utf8Out = new
java.io.BufferedWriter(new
java.io.OutputStreamWriter(parser.getOutputStream(), "UTF-8"));
The "parser" object referred to in the above code snippet references
the TDI Parser object itself, allowing you to get hold of the output
stream. Using this you can set up your own OutputStreamWriter with the
desired encoding setting. It is advisable to wrap your
OutputStreamWriter in a BufferedWriter since the Connector could be
writing to some device that where each output can be costly (e.g. to a
file). The BufferedWriter provides a more efficient way of doing your
output. Use the OutputStreamWriter dicrectly if you don't care about
efficiency or are afraid of flush problems (see further down).
After the above code snippet, you use the new variable
utf8Out to write out Entry data:
utf8Out.write(
entry.getString("TDS_cn") );
You must not use the out
writer anymore, as this will conflict with the output you send to your
own BufferedWriter.
But there is a drawback with using your own BufferedWriter: You must
ensure that the buffer is flushed out during the
Connector/Parser
close phase. This is handled automatically for the default
BufferedWriter used by a Scripted Parser (out), but must be
done manually for your own writer when the Parser is closed.
The best way of doing so is to call the .flush() method at
the bottom of writeEntry()
to ensure that each Entry is written correctly.
Some parsers (like a SAX XML parser), need to write extra information
when the parsed stream is closed.
Unfortunately, Scripted Parsers cannot implement the closeParser()
method, so you have to work around this by either:
utf8Out.newLine();
var
closeEntry = system.newEntry(); closeEntry.setAttribute( "$$FlushAndClose", "yes" ); thisConnector.connector.getParser().writeEntry(closeEntry); Then in the writeEntry() method
you look for this Attribute:
function
writeEntry() {
if (entry.getAttribute( "$$FlushAndClose" ) = null) { utf8Out.close(); // flushes and closes } JVM QuestionsQ: Will TDI use existing JVMs? If I upgrade my JVM, might then TDI fail?2005.02.07 A: We have tried to make TDI as wellbehaved as possible: TDI comes with its own JVM (located in _jvm). It is selfcontained, and will not interfer or be confused by other JVMs. Watch the ibmditk and ibmdisrv scripts to see how things are set. TDI is using it's own loader so it will ignore system wide java-libraires (like ext/ ) However, if you under Windows have, as part of another installation, changed sytem dlls that Java uses, then there might be conflicts but this is nothing we can do anything about.Q: Can I change the JVM, for example using the new Sun JVM?2005.02.07 A: Yes you can. Edit the batch-files/scripts to point to a new JVM. Note:
|
Topic revision: r6 - 27 Apr 2011, EddieHartman
- This page was cached on 04 Aug 2023 - 20:56.
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors. Ideas, requests, problems regarding TDI Users? Send feedback