Clustering, HA and Scaling, Oh My!

Ok, here comes some ramblings on making better TDI solutions. Risk reduction is always a matter of expense versus exposure, and this approach requires an upfront investment and adds lots of moving parts, so it's not a size that fits all and pays most dividends for larger projects, or for series of similar smaller ones. That said...

You start by deconstructing a solution into a set of individual service AssemblyLines:

1. Error Handler - First consider that all error hooks/scripts in all other ALs dispatch to a central error queue. Then you have this Error Handler AL that continuously iterates this queue and does logging/alerting/reacting as needed. This AL also emits a periodic heartbeat to an event queue*.
   
   * That's right, 2 queues. And note that you will want persistent queueing like MQ, the System Store/DB, files, etc.
   
   
2. Event Handlers (plural) - This set of service ALs catch events - e.g. detecting changes; listening for incoming messages, mail or requests; polling for new/changed files, and so forth. Each AL pushes caught events to the event queue, along with its own heartbeats. As mentioned above, all errors are dispatched to the error queue.
   
   
3. Workers (plural). These each grab the next relevant event off the queue and then performs a required action, like writing detected changes to a single target (e.g. for a sync solution you would have one Worker AL to write to Domino, one for AD, one for SAP, etc.), passing events to other systems (i.e. switching), performing searches and building responses (rss/REST/SOAP/...) or whatever. Each also reports status through the event queue and problems to the error queue.
   
   
4. Heartbeat Monitor - Polls the queues to make sure things are happening, for example that events are being processed in a timely fashion, and that heartbeats are received (and cleaned up). If the Error Handler is down, it does its own alerting and logging (it can even send events to AMC or a backup TDI Server).

This approach is suited for unit testing and provides better solution availability and maintainability than one-stop-shopping ALs tend to do. It also scales easily - just run more ALs, using inter-AL comms for coordination.

But it's definitely not for the lighthearted, or those not comfortable in the AL Debugger. However, if you do it right, you end up with a reusable the AL service framework.

And if you send it to me for publication then I'll send you a limited edition, orange plastic Metamerge pen :)

These really are great pens :)

Why TDI (version 2)

Here comes an updated version based on feedback:

---

All organizations want to reduce risk and maximize return on current investments, especially during tough times when spending for new IT dwindles and resources are redirected towards improving existing infrastructure. Given that infrastructures tend to be the result of evolution - in other words, survival of the highest switching cost and not a grand architectural plan - they seldom accommodate adjustment, much less the introduction of new software or services. Furthermore, the risk of failure or miscalculation can equate to disruption in business operations.

Enter the newer agile development methodologies whose mission is to solve problems incrementally, constantly adjusting design to meet uncovered constraints and correcting project goals to meet discovered requirements. A mindset and approach that bridges the need for executive oversight with the realities of technical innovation/renovation. TDI is designed from the bottom-up for this purpose.

That's why TDI is bundled with a growing number of IBM products and being adopted by more and more of business partners: It uniquely "closes the gap" between product functionality and client expectations. Rapid try/test/refine cycles with TDI reduce projects to manageable steps that provide feedback for stakeholders, enabling risk re-assesment and course correction.

---

In particular, thanks to Avery Salmon for stepping up to the plate here :)

TDI Elevator Pitch

I have been asked many times to write a blurb for a TDI event (e.g. a LabJam) or conference session, with specific instructions to not use much techie-speak. After numerous iterations, and much careful listening - particularly to our architect, Johan Varno - I'm relatively pleased with the following Why TDI? pitch:

---

Most organizations want to reduce risk, especially during tough times when investment in new IT dwindles and resources are redirected towards improving existing infrastructure. Given that infrastructures tend to be the result of evolution -- in other words, survival of the highest switching cost -- and that most evolutionary steps are painful, there is resistance to change.

Enter the agile development methodologies whose mission is to deliver value in stages so ROI can begin sooner and stakeholder feedback shapes each step. A mindset and approach that bridges the need for executive oversight with the uncertain realities of technical innovation/renovation. TDI is built for this purpose.

That's why TDI is bundled with a growing number of IBM products and being adopted by more and companies and integration professionals. It's a customization and deployment tool that uniquely 'closes the gap' between a product's feature list and client expectations. TDI's rapid try/test/refine approach helps reduce complex projects to manageable sprints, enabling continual risk re-assesment and course correction.

---

Comments, suggestions and even ridicule is anxiously awaited :)

Dealing with Errors

Back from summer holiday and starting to remember what I do for a living again. I thought I'd start with a quick post about error handling.

As you already know, the How-To writeup here talks about the mechanics of TDI error flows and Hooks, and from the Configs I've received, people are at the very least enabling Error Hooks so that their AssemblyLines don't stop. A good start.

However, logging enough information to manually recover from a problem is even better.

So what is enough information? That all depends on what your AL is doing and where the error occurs. If the AssemblyLine fails then you get a Java stack dump preceeded with the component name in brackets, along with the operation and underlying error returned from the target system. However if you catch the error by enabling Error Hooks then you don't see this in the log. This information is still available though and you can find it in the error variable.

Just like work and conn, error is an Entry object and it contains a number of Attributes provided by the TDI Server. These are described in the online docs and you can access them - for example, to write to log output, or to implement handling logic - using the getString() method. You can also do a task.dumpEntry(error) to easily get all the information printed to the log.

Sometimes an exception is not an error, but instead an unexpected situation like finding no match for a Lookup or Delete, or finding multiple entries (Lookup, Delete or Update modes). This results in mandatory Hooks like On No Match or On Multiple Found being invoked, and just like with Error Hooks, you have to tell TDI how to proceed or the AL stops. In these cases it is your Link Criteria that is either too narrow or too broad. As a result you will want to log enough information to determine why this is happening, and dumping out error will not get you there.

Instead, you should print out the value of work Attributes used for your Link Criteria: the right-hand operand for simple Link Criteria, usually preceded with a dollar sign ($). That way you can identify which data entries are giving you problems.

For those cases where a lookup is expected to either find no match or multiple then another approach is to add a Loop (Connector Loop) with your Connector attached. The Loop will then cycle once for each Entry found, driving these to components under this Loop. No Hook coding required.

Finally, in the case of connection errors you should note that all Connectors have a Connection Error tab where you can enable the Auto-Reconnect feature. While this is generally not interesting behavior for an initialization failure, if comms drops in the middle of AL processing, Auto-Reconnect can be used to quietly re-establish the connection and your AssemblyLine continues processing as if nothing had happened.

Note that for Iterators this will result in the Select being re-issued as well. As long as Iterator state is maintained - done automatically for all Change Detection Connectors, and you can easily implement this yourself for most other Connectors - processing will resume where it left off. If state is not used to re-align the Iterator then your AL will be reading from the start again.

AJAX made easy and interactive

Stolen from Jon Elwoods internal IBM blog:

---------------------------------------------------------

I was writing some simple AJAX functions for a web page for the migration project. They are generic enough to be used anywhere...

I created an assembly line that iterates on an HTTPServer FC. I found it hard to code all my client-side JavaScript functions directly into TDI (caused problems in expression attribute mapping) so I included them all in a funcs.js. This was included at in the header of the page:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<link rel="stylesheet" type="text/css" href="dashboard.css"/>
<script src="funcs.js"></script>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"></meta>
<title>title</title>
</head>

Contents of funcs.js:

var component

function request(attribute, value, comp) {
try {
xmlhttp = window.XMLHttpRequest?new XMLHttpRequest(): new ActiveXObject("Microsoft.XMLHTTP");
}
catch(e){
document.getElementById("main").innerHTML = "BROWSER DOES NOT SUPPORT AJAX";
}
component = comp;
xmlhttp.onreadystatechange = loadResult;
url=attribute + "=" + value;
url=url+"&sid="+Math.random();
xmlhttp.open("GET", url);
xmlhttp.send(null);
}

function loadResult() {
// if the readyState = 4 = Completed
// & http status is 200, get responseText
if ((xmlhttp.readyState == 4) && (xmlhttp.status == 200)) {
document.getElementById(component).innerHTML = xmlhttp.responseText;
}

else {
document.getElementById(component).innerHTML = "Loading...";
}
}

request() takes three parameters. Attribute and value will be used as part of the GET url. e.g. myurl.com?attribute=value. The third parameter is the component where the response xml/xhtml from TDI will be inserted.

The callBack function loadResult() will display a "Loading" message in the choosen element ID until the response is returned from TDI. I used simple <span> tags as place holders to insert my httpResponse text. e.g.:

<span id="result"> response goes here! </span>

So I would call the function:

request("myAtt","myvalue","result");

TDI simply looks at http.base for the attribute and value and returns an http.body according to my specifications. This is a very simple over view of the code. (There may be bugs and other nasties in the code so if your project comes tumbling to the ground....you have been warned!!) I will hopefully be posting a detailed tutorial on TDI and AJAX some time in the future.

The Power of the FormEntry Connector

I alluded to this in a previous post and would like to expand on it here. If you have ever been in the situation where you wanted to iterate over the xml nodes in the string value of an Attribute, or csv, or some other format, then you can solve this with JavaScript. I used to as well. Now I use the FormEntry Connector.

It's an Iterator that uses an attached Parser to read the text value you write/paste in its Config parameter (labeled 'Raw Text Data'). If you furthermore drop this Iterator into a Connector Loop, then you can map in the Connector Parameters you want, and then instruct the Loop to re-initialize and issue the search query, or just re-issue another query (no re-init).

So you map the Attribute value you want to parse as the assignment of the entryRawData parameter, attach your Parser, set up the Input Map and you're in business.

This Connector is standard issue with TDI 7.

The FormEntry.jar for TDI 6 is available for download here:

http://www.tdi-users.org/twiki/bin/view/Integrator/FormEntry

Scheduling when AssemblyLines run

When possible, use the scheduling mechanisms already in place: cronjobs, Windows scheduler, etc. Your TDI ALs can be started from the command line:

ibmdisrv -c MyConfig.xml -r MyAssemblyLine

That way if something goes wrong, your flow is restarted.

You can also do this in TDI using the Timer Connector (Iterator only). Here you can set up a scheduling mask. In the Flow section of your scheduling AL, you drop in AssemblyLine Function components (AL FCs) to launch other ALs.

One simple 'availability trick' for keeping an AL running is to start it (AL FC) inside a Loop that never ends. The AL FC is set to "Wait for completion" and will suspend until the called AssemblyLine stops. If it stops due to an error, you can catch this in the Error Hooks of the FC (& log it, send alerts, order pizza...). Regardless, the Loop -- which is Conditional and based on a script like this: ret.value = true; -- just executes the AL FC again, re-starting the stopped AssemblyLine.

So even in what is typically a long-running AL (like one listening to a changelog, or changes in Notes) you could set the Change Detection Iterator to timeout once in a while. Then you could do how keeping in the Scheduler AL, and restart the AssemblyLine again.

Here is an excellent developerWorks article on using TDI to Web 2.0-enable practically any type of data store or system, focusing on practical examples using TIM and TAM; and with Connections as the dashboard.

Thanks to Chris and Neil for sharing an impressive piece of work! Including their TDI Config and new ATOM Feed Connector.

ISM: Creating a Discovery Library Adapter to write IdML docs

If you are interested in creating IdML import files for systems like TADDM then there is a video that shows you how to assemble a DLA using TDI here.

There is also a shorter version on YouTube.

Please let me know if you have any comments, suggestions, corrections or just a bone to pick!

Happy 2009!