Reduce to Dashboard

When developers use DataWeave, they often come to rely on the reduce() function to fill in any gaps left by the standard Core library. Although filter() and joinBy() and splitBy() and even groupBy() could be implemented by simply using reduce(), we favor the Core library functions as the best implementation of those patterns.

On the other hand, polishing your reduce() game will get you through tight spots sometimes when the requirement is idiosyncratic, or when the standard function just won't do.

So we'll take a data set that represents a crude inventory, and use reduce() to extract a dashboard snapshot showing projected revenue, and potential growth through untapped inventory.

(BTW, if more about DataWeave is not your cup of tea, never fear. We are talking about the onset of automation in the workplace next. So drop by again.)

Just the Good Bits

Sometimes you find a use case that has a simple and elegant solution in a particular programming language. It might come from a question in conversation, or a requirement in a project on the table. The nature of elegance in code defies firm definition, but as a famous Jurist once said, "I can't define it, but I do know what I like."

One characteristic I usually find in what most observers might call "elegant code" is brevity. Fewer lines of code mean fewer places for a software fault to lurk. DataWeave certainly lends itself to brevity, even nearly to the point of compulsion.

Recently, a student of mine raised a question after doing one of the standard exercises in the MuleSoft Developer series using DataWeave. His question surfaced a typical use case for an API. 

We had just finished a Walkthrough that has us observe how to filter the elements of an Object given a list of desired fields.

His idea was to allow a query parameter that listed the fields being requested. If an API interface enumerates the fields that may be requested, the input can be easily validated.

Once we walked through the idea, we ended with a single line of code that completely handled the task. Follow our logic, and see if you find the outcome to be elegant, or simply brief.

Everyone's New Best Friend

 Everywhere I go, people are talking about OpenAI ChatGPT. They don't evangelize, they just mention it with a wink and a smile. Of COURSE you know all about this charming chat bot. And that's why we're smiling.

I had to get a look as soon as I heard about it from this NY Times article. It didn't take long before some of the responses passed my own personal Turing Test. 

I soon found myself arguing with the bot, struggling to get it to reconsider its answer. I was arguing with an algorithm! 

And given the nascent and experimental nature of my new friend, I would swear that I saw it evolve, day to day. (Stick with me to the end, and I'll entertain you with my own idiot glee by arguing with someone who is not there. Or is it arguing with someone who WAS there, but not actually someone?)

Bring Me the Severed Head of my Data

[This article is primarily intended for DataWeave developers. It deals with code and development strategies.] 

A while back, I wrote this article suggesting an approach to generating sample data with a simple DataWeave function. I left the conversation open ended to offer time for readers to suggest their own solution. I provided a sample (the severed head of my data) and a shell that suggested an approach to generation of sample data at any scale.

In this article, I offer one possible solution. If you have not read the OP (as it were) then give it a read and then see if you can hack out a solution that you like. If you've already given it a try, or if you simply want to see a solution dissected, then by all means, turn the page.

The premise is that you sometimes need to synthesize data for a project, and although there are tools that can readily help you do this, sometimes the characteristics of the problem domain call for a customized solution. Here are some reasons you might turn to DataWeave to help you.

Want to transform data? Then DataWeave

 If you work in corporate Information Technology, you may have been (or will be) asked to transform data from one form to another. There are many good ways to address the challenge, but my current favorite is the DataWeave language from Mulesoft.

The reason has nothing to do with the fact that it's my job to introduce developers to the language and then train them to use it well. It is much more because I've met many programming languages in my time, and this one is among the most compelling and satisfying.

You might not believe it if you have read many of the articles here, but I revere brevity. If you speak and finish exactly on the time marker, I applaud for the technical accomplishment much more so than for the speech. If you write me an email with 5 sentences instead of 5 paragraphs, I name my children after you.

When it comes to code, I realize at this point in my career, the fewer lines you write, the stronger your program. I blush with embarrassment when I think back to those days when I walked around and bragged that our project had grown to 80,000 lines of code. (Yes, I also counted the number of lines in each of the third-party libraries we bought for the project.)

Now that I know the folly of my ways, I can respect the fact that the best DataWeave expressions are the shortest.

DataWeave: Merge Objects From Different Sources

About the Merge

A common use case for DataWeave developers is that of merging elements from two different object collections into a set of composite output records. There are several ways to accomplish this, and we will look at a couple of them in this article.

As we explore this, we’ll also consider these ideas:

• The value of meaningful symbol names

• Using variables and functions to encapsulate awkward logic

• Using the groupBy() function to transform an array into a handy lookup table

This article also illustrates an approach that uses iterative construction to get our solution. It also demonstrates the value of moving complex operations into variables or functions to make our body expression more readable.

[Note: The considerations here are designed to facilitate understanding of the language. Best practice with DataWeave is to refactor important phrases to make them more generalized, and thus reusable. On the other hand, during initial development (and in team development) it pays to signal intention in the code until it is ready to be refactored for reuse.]

Spin Up Sample Data With DataWeave

 [This article is primarily intended for DataWeave developers. It may be considered a Forward to an article for the MuleSoft Developers blog that demonstrates how this problem can solved. Readers are invited to post comments to suggest the approach they believe in.]

Hopefully you don't commonly face this problem on your development team, but I see it often enough in the field that I know some folks could use a handy answer. Something that often hamstrings an API development effort is the absence of useful sample data for the project. If you use specification-driven development the way MuleSoft suggests, then there very well may be a suitable sample to get your project into coding.

Occasionally however, the samples we find in the spec are not enough either, and it falls to a developer who must generate a sample of suitable size, or variation. Using DataWeave, this is something you can do in a few minutes. I'm not going to show how in this article, I'm going to show you a framework and have you show ME how. Then I will publish an article in the MuleSoft Developers blog, showing my solution, and perhaps yours, if you suggest something magical.

Some Shifts

It's taken some time to work this out for myself, but since I began teaching for MuleSoft, I haven't really been sure what to write here. Compounded by that, I write a fair amount for my proper job. Nonetheless, I have my thoughts about technology at large,  and sometimes I want to work out some of my thoughts here before I take them back and deliver them polished at work.

So when I drop a deep weeds tech article here, I will make it clear with some hashtags and a warning above the fold. That way, you can punch through if it seems interesting, and scroll by when it's not.

(Of course, everything I publish here is fascinating, but you may want to look into it at another time, right?) 

Because a lot of what I discuss at work is for developers who work mostly with Enterprise data systems, the technology talks may get pretty gritty, especially if I bring in voices from among the many brilliant colleagues who help me understand what I'm doing.

So check the page labels and watch for fair warning before the first break.

Whaddya Mean, Big Data?

It's been going on for a while now. We hear the term "Big Data" used repeatedly and often in contexts that suggest various meanings.  To some people, it seems to suggest an insidious assault on personal privacy. To others, it seems to mean the collection of companies like Facebook and Google that thrive and use their success to influence public policy.

Except among IT professionals, the term seems to mostly be used as a pejorative. Something that evokes fear, or derision, or some force to be resisted. In common usage, Big Data ignites the same sense of dread that Big Petroleum, or Big Government, or Big Pharmaceutical do.

But what does it mean when used in the Information Technology lexicon? Is it something to watch hawkishly, or is it something that holds the promise that we could know more, make better decisions, and innovate more rapidly?

Who Ya Gonna Trust?

In society of today, we face many questions about what information to trust, whose perspective to trust, and which technologies to trust. Only history will provide us with the right answers, but we have to live in the present, not blessed with the retrospective that will inform our future selves and our progeny.

One of the fabricated perils we have been hearing more about of late is "Big Data." Sometimes it's called "Silicon Valley Elite" or "Tech Elite." And one of the dangers that supposedly comes from this new threat to our way of life is Cloud Computing.

Certainly, since there's a lot of misinformation available about this important development in the world of Information Technology, it's easy to accept viewpoints derived from casual, poorly researched, and superstition-driven irritation or fear. I saw one such article this past week, and perhaps we should forgive the author (Steve Rosseau) for reacting during the heat of the irritation when faced with the possibility that a great deal of his work was lost.

Tech Security Black Monday

It's not a good day for IT security. Two vulnerabilities were announced today, one affecting WiFi for almost everyone, and another that affects public key encryption.

Those who know me are aware that I often relate to most conversations about IT security as alarmist, and often find measures taken in Enterprise security to be self-defeating.

But this time, it's different.

This time the scope of these vulnerabilities is so widespread that many are calling this Black Monday.

Robodialer Defense

Sure, you could hide your phone number from view, and refuse all calls except those from numbers on your carefully crafted whitelist. That's one way to keep robo dialers from interrupting your life and work.

But why should you skulk in the shadows, eyes cast downward in defense against callers who might simply be an annoying disruption to your "flow" but who are very likely the shock troops for disreputable or dishonest businesses?

You can read at Gizmodo, or at CBS News about measures you can take. But what irritates me about many of the suggestions is that they constitute an inconvenience for me, or for my callers. I am not satisfied with letting the polluters extort time and effort from me or from my people. (You may have guessed that from my attitudes in my earlier article, Passwords Must Die.)

So let's talk about something you can do that doesn't take much effort (beyond initial setup) and that might provide considerable entertainment. In fact, this might be the most fun you can have with your telephone receiver down.

Top 10 Web Hosts All the Same

I think the word is out! "You can't believe anything you read on the Internet." (By the way, if you haven't heard that, it's true!) And that means this article too. Don't believe it. Don't believe me, don't believe Huffington Post, don't believe RedStatesResurgent.com (band name, I call it!), don't believe Snopes, or Politifact, or even NPR.org.

Consider this the truth. If you're reading this on the Internet, you can't believe it. Don't believe any of them. Don't believe me.

So what can you believe? And what does that have to do with reviews for web hosting companies?

Location Based Services - Roaming Soon Near You

The buzz has been building for years now, and this trend has probably taken strong roots and is ready for an explosive outbreak.

Location-based services are not new. We've had Foursquare long enough that they've had time to pivot and rebrand their service into Swarm. Some observers believed it would be the Next Big Thing after Twitter, but somehow it never fully caught fire. (Although I confess that I'm a dedicated Swarm player and I do appreciate my friends who participate. I get an interesting and unique perspective on their adventures in life.)

Here's what happens next in this world...

Are You Playing Evernote?

If you haven't heard about or tried Evernote, you can probably ask your kids.

This Silicon Valley company made a big splash in schools when many teachers from middle school through high school adopted it as a favorite online study tool for students.

If I just told you what it does, you might not be impressed, or you might wonder what it offers that you don't already have. But when you talk to advocates for this note-taking and organizing tool, you might get the impression that there's a lot going on.

So here's what it's all about...

Be Not Afraid of This Cloud

We've talked here about the Cloud before and it's likely that we'll be talking about it again. Whether you're ready for it or not, no other advance in business computing has had this big an effect since the dawn of the Internet in business.

So if you want to get ahead of the curve, and be ready for a technology that empowers your business, this is a good time to talk about it. We talked earlier about the what the cloud really is. It's most certainly more than just a decision about whether you'll put your business documents on Google Docs, or store your backups on Dropbox servers. Those are just the leaky edges of a trend that will engulf IT strategies for at least a decade.

Here's how to think about the cloud...

About Cloud Storage

What do you know about Cloud Storage? Everyone is talking about it, some people are using it, some aren't sure if they are, and as with any major shift in technology or computing, there's a lot of folklore and wives tales swirling around the concept. So let's talk about it straight.

Straight Talk

The questions on everyone's lips are, "Is it safe to use cloud storage," and "how will this change what I do when I save documents and media files," and "which cloud storage choice is best for me?"

In the first place, when we talk about cloud storage, we're actually talking about a place in some company's data center where your data resides. This is in contrast to your own hard drive in your own safe, secure home computer or home office.

But is it safe? Cloud storage seems to put your data out of your reach, and in someone else's control. How can that be good? On the other hand, just how safe is your data on a hard drive at home? Let's tell the truth about that first.

Email Has Failed Us - What To Do Now?

It's been a long time for most of us since email was something other than a depressing form of slavery in our work and social lives.

Either we receive too much unsolicited and irrelevant email for us to find the useful messages, or we chain ourselves to an hour or more each day answering correspondence just to keep the pile from growing out of hand.

From the other side, it's also become common for us to send a carefully crafted message to someone and discover that they have only scanned it briefly, or that they've never even seen it in the blizzard of messages they receive. And heaven help us if they are trying to manage both work and personal email accounts separately.

I used to teach a micro-class in productivity that addressed some of this and offered tactics to keep the email pile to a manageable level. (Okay, I still teach the class, but I think the focus will begin to change.) We can talk all day about tactics, and maybe even adopt some of them as habits, but the root cause will not disappear.

There is good news however!

Google Contacts Beta

Dear Google Friends,

The new Contact interface is terrible. Please understand that I'm a strong advocate for Google - some would call me a fanboy. I really want Google to succeed and to demonstrate that a corporation can grow large and successful without turning to evil.

BUT -- the new contacts subsystem is terrible. It's sluggish, it's inconvenient, it's often inexplicable, and it's absolutely NOT an improvement on the very functional and completely satisfactory predecessor that was once integrated with Gmail.

Gamification - We're Not Playing Here

In retrospect I'm not sure why, but I kind of expected a bit more from this article at ELearning Industry -- Gamification Benefits in Workplace Training"

The title drew me in. I was hoping that it would offer some lucid and effective arguments to the instinctive (and sometimes subconscious) resistance to gamification in corporate learning system design.

Instead, the article's primary thesis seemed to be, "people play games anyway, so just give in -- and besides, it's fun!"

This really misses the point.