What is the System Store?

The System Store is a database that TDI uses to 'remember' stuff like:

• the last change read, when you use one of the Change Detection Connectors (also called CDCs);
• the snapshots of data used to compute changes when using the Delta Engine together with an Iterator Connector;
• data read using the System Store Connector, which writes and reads full Entry objects with any number of Attributes;
• objects (of any kind) saved using the system.setPersistenObject() call. These can further be retrieved with system.getPersistenObject(), and removed with system.deletePersistentObject();
• Sandbox data when you run an AL in Record or Replay modes;

Plus a few other things.

By default, TDI gives you the Derby database system, which is a fully functional open source RDBMS. You can change the System Store used by the TDI server at any time by simply right-clicking on that server in the Servers View and selecting the 'Edit system store settings' option.

This opens up the Server System Store page with its settings. At the top of the page, next to the title, is a drop-down arrow that lets you choose from a list of supported RDBMS's to use as the System Store.

When you select one of these options the various settings below are populated with default values for that system, including all the CREATE statements for standard TDI tables. Only the topmost settings on this page need to be edited to include the actual hostname for your database server,  as well as database name, port used and authentication credentials. Once this has been changed then the new System Store can be tested using the Test Connection button. When TDI is restarted then the server will use the new System Store settings.

Note also that you can define the System Store settings for a Project as well. The Project-specific setting is found in the 'Solution Logging and Settings' item under that Project.

The right-most tab labeled 'System Store' will give you access to the Config Instance System Store settings.

Now whenever an AL in this Config uses the System Store then it will be working with the database system assigned to the Project.

If you use the Browse Server Stores feature to view the contents of the System Store, then the 'last change' information saved by Change Detection Connectors will be easily read in the ID_PS_Default table. This is also where persistent objects that you save in your script is kept.

Each item here will have the key it was saved with - for example, the Iterator State Key value of a CDC, or the key used for a system.setPersistentObject() call - as well as a legible value. However, if you try to access this table using a JDBC or Database Connector then you will see that the value column (called 'Entry' in the table) is a BLOB. This column actually contains serialized Entry objects. If you still want to access this information directly using JDBC then you will have to deserialize the Entry value like this:

     // For example, in the After GetNext Hook you can grab 'Entry' from conn
     // and deserialize it to a new Attribute called 'Value'.
     entryObj = conn.getObject("Entry"); // get the actual Java object value
     conn.Value = com.ibm.di.store.StoreFactory.deserializeObject(entryObj);
     // Now 'Value' can be mapped into Work and is human readable

Finally, it's important to note that System Store settings are stored in the solution.properties file, where you will also find the setting to have the TDI Server automatically launch the Derby RDBMs when it starts up. This property is commented out by default as shown here.

        #com.ibm.di.store.start.mode=automatic

Simply remove the hash mark (#) to make startup automatic. This option is not used when other database systems are selected for the System Store.

Importing a client certificate from an https service

I have cobbled together a simple AL that can be run from the commandline in order to import a client certificate from an HTTPS service. For example:

ibmdisrv -c configs/tdiingOL.xml -r importcert -0 https://supportcenter.checkpoint.com

The above call results in this output (including the standard TDI is startup messages):

CTGDKD024I Remote API successfully started on port:1099, bound to:'SessionFactory'. SSL and Client Authentication are enabled.
com.ibm.tdi.rest started on /sdi
com.ibm.tdi.rest: com.ibm.tdi.rest.cache.enabled=false
CTGDIS1957I Added certificate with subject: CN=supportcenter.checkpoint.com, OU=MIS-US, O=Check Point Software Technologies Inc., L=San Carlos, ST=California, C=US.

As you can see from the commandline above, the config xml file (tdiingOL.xml) should be copied to the configs folder of your Solution Directory.

This config can be downloaded from here.

Note that you will need TDI 7.1.1 Fixpack 4, or SDI 7.2 Fixpack 2 in order for this to work.

JSON and XML Tutorial - Part 2

Well, this one is focused on JSON and I rolled it into a quick video.

https://www.youtube.com/watch?v=VHmf76FMI-U

Enjoy, and please comment if anything is unclear (or unsaid).

Easily implementing a monitoring API to your TDI solution

All you have to do is make AssemblyLines with the names of the operations you want to expose.

For example, I have a solution that catches incoming events over TCP  and dispatches the payload data to one or more 'event triggered' AssemblyLines. Like all server mode based ALs, this one uses multiple threads to deal with client traffic (the AL Pool), and shared log output can get a little convoluted. This makes monitoring and troubleshooting tougher.

So I leveraged a web server functionality first found in SDI 7.2.0.2 and TDI 7.1.1.4 that lets me run an AL named 'status' in a Config named 'MyRestAPI" by dialing up this URL in a browser.

   https://localhost:1098/sdi/MyRestAPI/status

The MyRestAPI.xml file needs to be in the <tdi soldir>/configs folder.

My 'status' AL has a single script that grabs some objects shared between the AL threads.

      // Get the shared objects - both are Javascript objects

      metricsObj = java.lang.System.getProperties().get("metricsObj");

      errorsObj = java.lang.System.getProperties().get("errorsObj");

      // Make the return payload

      returnPayload = {

            status: (metrics == null) ? "Not running" : "Ok",

            metrics: metricsObj,

            errors: errorsObj

      }

      // Set up HTTP attributes for the reply to the client

      work["http.body"] = toJson(returnPayload);

      work["http.content-type"] = "application/json";

      work["http.responseCode"] = "200";

Whenever an event is serviced by the TCP Listener AL, the metricsObj object is updated by calling metrics.gather(endmsecs - startmsecs), with the msec variables having been set before and after dispatching an event to its handler AL. A scripted logmsg() function keeps track of "ERROR", "FATAL" and "WARN" level log messages in the shared 'errorsObj' object.

The result message looks like this:

{

"status" : "Ok",

"metrics": [

{ 

"eventName": "Auth Voilation",

"duration": {

"max": 1,

"min": 1, 

"avg": 1, 

"total": 1

},

"responses": 1 

},

{

"eventName": "Health Metric",

"duration": { 

"max": 16,

"min": 0,

"avg": 0.3785425101214575, 

"total": 187

},

"responses": 494 

},

"errors": [], 

}

I have also added an 'admin' AL that looks for the query string parameter 'pause', and if the value is 'true' it sets a flag in metrics that causes the capture() function hang until its value is 'false' again. Your imagination is the limit here. That of course and time.

Note that the port used by SDI web services, as well as other comms settings, are found in the solution.properties file.

## Web container

web.server.port=1098

web.server.ssl.on=true

web.server.ssl.client.auth.on=false

Below these are the properties for the SDI dashboard (https://localhost:1098/dashboard).

Passing arguments to your AL when launching it from the command-line

In the ibmdisrv command-line used to launch your AssemblyLine, you can use one of the user-defineable options -0 through -9 to pass information into the AL.

For example, in order to pass the name of an input file to the AssemblyLine named 'ReadFile' you could use this command-line:

  ibmdisrv -c MyConfig.xml -r ReadFile -0 c:/Input/HRExport.csv

Then in the Prolog - Before Initialize Hook of the File Iterator used to read this file you can retrieve the filename from the command-line argument and apply it to the 'filePath' parameter of the Connector like this:

  fileName = main.commandlineParam("0");
  thisConnector.connector.setParam("filePath", fileName);

Of course, you will also want to harden this code a little to deal with missing command-line option values, etc.