Bring Me the Severed Head of my Data

[This article is primarily intended for DataWeave developers. It deals with code and development strategies.] 

A while back, I wrote this article suggesting an approach to generating sample data with a simple DataWeave function. I left the conversation open ended to offer time for readers to suggest their own solution. I provided a sample (the severed head of my data) and a shell that suggested an approach to generation of sample data at any scale.

In this article, I offer one possible solution. If you have not read the OP (as it were) then give it a read and then see if you can hack out a solution that you like. If you've already given it a try, or if you simply want to see a solution dissected, then by all means, turn the page.

The premise is that you sometimes need to synthesize data for a project, and although there are tools that can readily help you do this, sometimes the characteristics of the problem domain call for a customized solution. Here are some reasons you might turn to DataWeave to help you.

Want to transform data? Then DataWeave

 If you work in corporate Information Technology, you may have been (or will be) asked to transform data from one form to another. There are many good ways to address the challenge, but my current favorite is the DataWeave language from Mulesoft.

The reason has nothing to do with the fact that it's my job to introduce developers to the language and then train them to use it well. It is much more because I've met many programming languages in my time, and this one is among the most compelling and satisfying.

You might not believe it if you have read many of the articles here, but I revere brevity. If you speak and finish exactly on the time marker, I applaud for the technical accomplishment much more so than for the speech. If you write me an email with 5 sentences instead of 5 paragraphs, I name my children after you.

When it comes to code, I realize at this point in my career, the fewer lines you write, the stronger your program. I blush with embarrassment when I think back to those days when I walked around and bragged that our project had grown to 80,000 lines of code. (Yes, I also counted the number of lines in each of the third-party libraries we bought for the project.)

Now that I know the folly of my ways, I can respect the fact that the best DataWeave expressions are the shortest.

DataWeave: Merge Objects From Different Sources

About the Merge

A common use case for DataWeave developers is that of merging elements from two different object collections into a set of composite output records. There are several ways to accomplish this, and we will look at a couple of them in this article.

As we explore this, we’ll also consider these ideas:

• The value of meaningful symbol names

• Using variables and functions to encapsulate awkward logic

• Using the groupBy() function to transform an array into a handy lookup table

This article also illustrates an approach that uses iterative construction to get our solution. It also demonstrates the value of moving complex operations into variables or functions to make our body expression more readable.

[Note: The considerations here are designed to facilitate understanding of the language. Best practice with DataWeave is to refactor important phrases to make them more generalized, and thus reusable. On the other hand, during initial development (and in team development) it pays to signal intention in the code until it is ready to be refactored for reuse.]

Spin Up Sample Data With DataWeave

 [This article is primarily intended for DataWeave developers. It may be considered a Forward to an article for the MuleSoft Developers blog that demonstrates how this problem can solved. Readers are invited to post comments to suggest the approach they believe in.]

Hopefully you don't commonly face this problem on your development team, but I see it often enough in the field that I know some folks could use a handy answer. Something that often hamstrings an API development effort is the absence of useful sample data for the project. If you use specification-driven development the way MuleSoft suggests, then there very well may be a suitable sample to get your project into coding.

Occasionally however, the samples we find in the spec are not enough either, and it falls to a developer who must generate a sample of suitable size, or variation. Using DataWeave, this is something you can do in a few minutes. I'm not going to show how in this article, I'm going to show you a framework and have you show ME how. Then I will publish an article in the MuleSoft Developers blog, showing my solution, and perhaps yours, if you suggest something magical.

Some Shifts

It's taken some time to work this out for myself, but since I began teaching for MuleSoft, I haven't really been sure what to write here. Compounded by that, I write a fair amount for my proper job. Nonetheless, I have my thoughts about technology at large,  and sometimes I want to work out some of my thoughts here before I take them back and deliver them polished at work.

So when I drop a deep weeds tech article here, I will make it clear with some hashtags and a warning above the fold. That way, you can punch through if it seems interesting, and scroll by when it's not.

(Of course, everything I publish here is fascinating, but you may want to look into it at another time, right?) 

Because a lot of what I discuss at work is for developers who work mostly with Enterprise data systems, the technology talks may get pretty gritty, especially if I bring in voices from among the many brilliant colleagues who help me understand what I'm doing.

So check the page labels and watch for fair warning before the first break.