(Really) Big Data: from the trenches

Okay, people throw around 'big data' experience, but what is it really like? What does it feel like to manage a Petabyte of data. How do you get your hands around it? What is the magic formula that makes it all work seamlessly without Bridge Lines opened on Easter Sunday with three Vice Presidents on the line asking for status updates by the minute during an application outage?

Are you getting the feel for big data yet?

Nope.

Big data is not terabytes, 'normal'/SQL databases like Oracle or DB2 or GreenPlum or whatever can manage those, and big data vendors don't have a qualm about handling your 'big data' of two terabytes even though they are scoffing into your purchase order.

"I've got a huge data problem of x terabytes."

No, you don't. You think you do, but you can manage your data just fine and not even make Hadoop hiccough.

Now let's talk about big data.

1.7 petabytes
2.5 billion transactions per day.
Oh, and growing to SIX BILLION transactions per day.

This is my experience. When the vendor has to write a new version of HBase because their version that could handle 'any size of data, no matter how big' crashed when we hit 600 TB?

Yeah. Big data.

So, what's it like?

Storage Requirements/Cluster Sizing

1. Your data is bigger than you think it is/bigger than the server farm you planned for it.

Oh, and 0. first.

0. You have a USD million budget ... per month.

Are you still here? Because that's the kind of money you have to lay out for the transactional requirements and storage requirements you're going to need.

Get that lettuce out.

So, back to 1.

You have this formula, right? from the vendor that says: elastic replication is at 2.4 so for 600 TB you need 1.2 Petabytes of space.

Wrong.

Wrong. Wrong. WRONG.

First: throw out the vendors' formulae. They work GREAT for small data in the lab. They suck for big data IRL.

Here's what happens in industry.

You need a backup. You make a backup. A backup is the exact same size as your active HTables, because the HTables are in bz2-format already compressed.

Double the size of your cluster for that backup-operation.

Not a problem. You shunt that TWO PETABYTE BACKUP TO AWS S3?!?!?

Do you know how long that takes?

26 hours.

Do you know how long it takes to do a restore from backup?

Well, boss, we have to load the backup from S3. That will take 26 hours, then we ...

Boss: No.

me: What?

Boss: No. DR ('disaster recovery') requires an immediate switch-over.

Me: well, the only way to do that is to keep the backup local.

Boss: Okay.

Double the size of your cluster, right?

Nope.

What happens if the most recent backup is corrupted, that is, today's backup, because you're backing up every day just before the ETL-run, then right after the ETL-run, because you CANNOT have data corruption here, people, you just can't.

You have to go to the previous backup.

So now you have two FULL HTable backups locally on your 60-node cluster!

And all the other backups are shunted, month-by-month, to AWS S3.

Do you know how much 2 petabytes, then 4 petabytes, then 6 petabytes in AWS S3 costs ... per month?

So, what to do then?

You shunt the 'old' backups, older than x years old, every month, to Glacier.

Yeah, baby.

That's the first thing: your cluster is 3 times the size of what it needs to be, or else you're dead, in one month. Personal experience bears this out: first you need the wiggle room or else you stress out your poor nodes of your poor cluster, and you start getting HBase warnings and then critical error messages about space utilization, second, you need that extra space when the ETL job loads in a billion row transaction of the 2.5 billion transactions you're loading in that day.

Been there. Done that.

Disaster Recovery

Okay, what about that DR, that Disaster Recovery?

Your 60 node cluster goes down, because, first, you're not an idiot and didn't build a data center and put all those computers in there yourself, but shunted all that to Amazon and let them handle that maintenance nightmare.

Then the VP of AWS Oregon region contacts you and tells you everything's going down in that region: security patch. No exceptions.

You had a 24/7 contract with 99.999% availability with them.

Sorry, Charlie: you're going down. A hard shutdown. On Thursday.

What are you going to do?

First, you're lucky if Amazon tells you: they usually just do it and let you figure that out on your own. So that means you have to be ready at any time for the cluster to go down with no reason.

We had two separate teams monitoring our cluster: 24/7. And they opened that Bridge Line the second a critical warning fired.

And if a user called in and said the application was non-responsive?

Ooh, ouch. God help you. You have not seen panic in ops until you see it when one user calls and come to find it's because the cluster is down with no warning catching that.

Set up monitoring systems on your cluster. No joke.

With big data, your life? Over.

Throughput

Not an issue. Or, it becomes an issue when you're shunting your backup to S3 and the cluster gets really slow. We had 1600 users that we rolled out to, we stress-tested it, you know. Nobody had problems during normal operations, it's just that when you ask the cluster to do something, like ETL or backup-transfer, that engages all disks of all nodes in reads and writes.

A user request hits all your region servers, too.

Do your backups at 2 am or on the weekends. Do your ETL after 10 pm. We learned to do that.

Maintenance

Amazon is perfect; Amazon is wonderful; you'll never have to maintain nor monitor your cluster again! It's all push-of-the-button.

I will give Amazon this: we had in-house clusters with in-house teams monitoring our clusters, 'round the clock. Amazon made maintenance this: "Please replace this node."

Amazon: "Done."

But you can't ask anything other than that. Your data on that node? Gone. That's it, no negotiations. But Hadoop/HBase takes care of that for you, right? So you're good, right?

Just make sure you have your backup/backout/DR plans in place and tested with real, honest-to-God we're-restarting-the-cluster-from-this-backup data or else you'll never know until you're in hot water.

Vendors

Every vendor will promise you the Moon ... and 'we can do that.' Every vendor believes it.

Then you find out what's what. We did. Multiple times, multiple vendors. Most can't handle our big data when push came to shove, even though they promised they can handle data of any size. They couldn't. Or they couldn't handle it in a manageable way: if the ETL process takes 26 hours and it's daily, you're screwed. Our ETL process got down to 1.5 hours, but that was after some tuning our their part and on ours: we had four consultants from the vendor in-house every day for a year running. Part of our contract-agreement. If you are blazing the big data trail, your vendor is, too: we were inventing stuff on the fly just to manage the data coming in, and to ensure the data came out in quick, responsive ways.

You're going to have to do that, too, with real big data, and that costs money. Lots.

And, ... but it also costs cutting through what vendors are saying to you, and what their product can actually handle. Their sales people have their sales-pitch, but what really happened is we had to go through three revisions of their product just so it could be an Hadoop HBase-compilant database that could handle 1.7 petabytes of data.

That's all.

Oh, and grow by 2.5 billion rows per day.

Which leads to ...

Backout/Aging Data

Look, you have big data. Some of it's relevant today, some of it isn't. You have to separate the two, clearly and daily, if you're not, then a month, two months, two years down the road you're screwed, because you're now dealing with a full-to-the-gills cluster AND having to disambiguate data you've entangled, haven't you? with the promise of looking at aging data gracefully ... 'later.'

Well, later is right now, and your cluster is full and in one month it's going critical.

What are you going to do?

Have a plan to age data. Have a plan to version data. Have a data-correction plan.

These things can't keep being pushed off to be considered 'later' because 'later' will be far too late, and you'll end up crashing your cluster (bad) or corrupting your data when you slice and dice it the wrong way, come to find (much, much worse). Oh, and version your backups, tying them to the application version, because when you upgrade your application, your data gets all screwy, being old, or your new data format on your old application when somebody pulls up a special request to view three-year-old data is all screwy.

Have a very clear picture of what your users need, the vast majority of the time, and deliver that and no more.

We turned a 4+hour query that terminated when it couldn't deliver a 200k+ row query on GreenPlum...

Get that? 4+hours to learn your query failed.

No soup for you.

To a 10 second query against Hadoop HBase that returns 1M+ rows.

Got that?

We changed peoples' lives. What was impossible before for our 1600 users was now in hand in 10 seconds.

But why?

Because we studied all their queries.

One particular query was issued 85% of the time.

We built our Hadoop/HBase application around that, and shunted the other 15% of the queries other tools that could manage that load.

Also, we studied our users: all their queries were in transactions of within the last month.

We kept two years of data on-hand.

Stupid.

And that two years grew to more, month by month.

Stupider.

We had no graceful data aging/versioning/correcting plans, so, 18 months into production we were faced with a growing problem.

Growing daily.

The users do queries up to a month? No problem: here's your data in less than 10 seconds, guaranteed. You want to do research, you put in a request.

Your management has to put their foot down. They have to be very clear what this new-fangled application is delivering and the boundaries on what data they get.

Our management did, for the queries, and our users loved us. You put in a query and it takes four hours, and only 16 queries are allowed against the system to run at any one time to: anyone, anywhere can submit a query and it returns right away?

Life-changing, and we did psychological studies as well as user-experience studies, too, so I'm not exaggerating.

What our management did not do is put bounds on how far back you could go into the data set. The old application had a 5 year history, so we thought two years was good. It wasn't. Everybody only queried on today, or yesterday, or, rarely: last week or two weeks ago. We should have said: one month of data. You want more, submit a request to defrost that old stuff. We didn't and we paid for it in long, long meetings around the problem of how to separate old data from new and what to do to restore old data, if, ever (never?) a request for old data came. If we had a monthly shunt to S3 then to Glacier, that would have been a well-understood and automatic right-sizing from the get-go.

You do that for your big data set.

Last Words

Look. There's no cookbook or "Big Data for Dummies" that is going to give you all the right answers. We had to crawl through three vendors to get to one who didn't work out of the box but who could at least work with us, night and day, to get to a solution that could eventually work with our data set. So you don't have to do that. We did that for you.

You're welcome.

But you may have to do that because you're using Brand Y not our Brand X or you're using Graph databases, not Hadoop, or you're using HIVE or you're using ... whatever. Vendors think they've seen it all, and then they encounter your data-set with its own particular quirks.

Maybe, or maybe it all will magically just work for you.

And let's say it does all magically work, and let's say you've got your ETL tuned, and your HTables properly structured for fast in-and-out operations.

Then there's the day-to-day daily grind of keeping a cluster up and running. If your cluster is in-house ... good luck with that. Have your will made out and ready for when you die from stress and lack of sleep. If your cluster is from an external vendor, just be ready for the ... eh ... quarterly, at least, ... times they pull the rug out from under you, sometimes without telling you and sometimes without reasonably fair warning time, so it's nights and weekends for you to prep with all hands on deck and everybody looking at you for answers.

Then, ... what next?

Well: you have big data? It's because you have Big Bureaucracy. The two go together, invariably. That means your Big Data team is telling you they're upgrading from HBase 0.94 to HBase whatever, and that means all your data can go bye-bye. What's your transition plan? We're phasing in that change next month.

And then somebody inserts a row in the transaction, and it's ... wrong.

How do you tease a transaction out of an HTable and correct it?

An UPDATE SQL statement?

Hahaha! Good joke! You so funny!

Tweep: "I wish twitter had an edit function."

Me: Hahaha! You so funny!

And, ooh! Parallelism! We had, count'm, three thousand region servers for our MapReduce jobs. You got your hands around parallelism? Optimizing MapReduce? Monitoring the cluster as the next 2.5 billion rows are processed by your ETL-job?

And then a disk goes bad, at least once a week? Stop the job? Of course not. Replace the disk (which means replacing the entire node because it's AWS) during the op? What are the impacts of that? Do you know? What if two disks go down during an op?

Do you know what that means?

At replication of 2.4, two bad disks means one more disk going bad will get you a real possibility of data corruption.

How's your backups doing? Are they doing okay? Because if they're on the cluster now your backups are corrupted. Have you thought of that?

Think about that.

And, I think I've given enough experience-from-the-trenches for you to think on when spec'ing out your own big data cluster. Go do that and (re)discover these problems and come up with a whole host of fires you have to put out on your own, too.

Hope this helped. Share and enjoy.

cheers, geophf

(Pure) Functional Programming Claims IRL

So, THIS happened:

– question from a young programmer

So, does functional programming stack up in the real world? In industry?

Yes. Check my linkin profile. I have been programming, in industry, as long as you have been alive.

Here are some (pure) functional programming examples to back up this claim, because academics can talk all they want, but they are not in there, in the trenches, with you where it counts.

I am, because I happened to have dug a few of those trenches. You're welcome.

Case study 1: ATS-L

Worked on a project in DHS called 'ATS' ('Automated Targeting System'). The existing system ATS-C was a 100,000-line Prolog behemoth that used pure dynamic types (no type hints, nor boxed types) and every rule started with an assert and ended with a retract. And 100,000 lines.

It was impossible to know what was going on in that system, without running the code in the debugger and pulling from the dynamic environment. Consequently, the ATS-C guy had (still has) job security. Not his aim, but that is a nice plus.

It took us 72-hours to go through every line of his code to correct the Int-rollover problem when the primary key exceeded two billion for the index.

So, I was called in to 'help.' HA! But then eventually I built ATS-L. I wrote it in 10,000 lines of purely functional Prolog (yes, that is possible to do, and remain authentic to logic programming in Prolog), so every rule called gave the same truth-verification from the same set of arguments, every time.

Shocker! I know.

I had the same level of functionality of ATS-C and handled 1,000x the number of transactions per hour. And as it was purely functional Prolog, I could reason about my program in the large and in the small. Importantly, so could others, as I passed on that work after maintaining it for three years.

In short: 1/10th the SLOC with the same level of functionality with increased real-time responsiveness and a vastly reduced level of maintenance.

Oh, and I also wrote 726 unit tests and put them on an automated midnight run, generating a report every single day. If my system broke, or something changed, I knew it, and management knew it when the report was automatically emailed to them.

Case Study 2: CDDS

Worked three years in Fannie Mae devising within a team an appraisal review process, CDDS. We had a good team of five engineers and I was given the 'Sales Comparison Approach' which had 600 elements out of 2,100 data elements in over 100 data tables, one of the tables ingested 100 million elements per month. All the elements were optional. All of them, so primary key dependencies were an ... interesting problem. The upshot was that Sales Comparison Approach was an impossible task to code, as we coded it in Java, of course.

What did I do? I coded it in Java.

After I implemented the Maybe type, then the Monad type-class ... in Java.

After I completed the system and tuned it, storing only the values that were present in the submitted forms, my manager reported up the chain that SCA and CDDS would have failed if I had not been there to implement it.

How did I implement it? In Java. I didn't use one for-loop and my if-statements were not there. I used the Maybe-Monad to model semi-determinism, lifting the present data to Just x and the absent data ('null') to Nothing, and then I executed action against the monadic data.

Simple. Provable. Implemented. Done.

Oh, and I had written 1,000 of the 1,100 unit test cases. SCA had 1,000 unit test cases, the rest of the system had a total of 100 unit test cases.

My code coverage was fiiiiiiine.

Case Study 3: Sunset Dates

This one was interesting.

I worked at Freddy Mac for a year, and they had a problem, and that problem was to calculate the sunset date for a mortgage based on the most recent date from one of possibly five indicators, that changed with each possible mortgage transaction.

Three different software teams tackled this problem over a period of six months and none of them implemented a system that passed UAT.

I sat down with the UATester and kept getting part of the story. I lifted our conversations up into the categorical domain, and then dropped that into a Java-implementation (I used both monads and comonads which I had implemented).

It took me two solid months working with this tester and a front-end developer, but we passed UAT and we got the customer and their SMA to sign off on it.

Three person team, purely functional programming ... in Java won that work where standard imperative approaches failed, over and over again.

Funny story. I was seriously asked on that project: "What's a tuple?"

Case Study 4: Dependency Graphs of Program Requirements ('TMQER')

I can't compare what I wrote, in Haskell, to an alternative system, because the alternative, traditional imperative approach was never essayed. We had a set of 521 requirements for a program with many (multiple) parent and child dependencies, so it wasn't a tree, it was a graph. So, I parsed the requirements document into a Haskell Data.Graph and provided not only a distance matrix, as requested (which is not what the customer wanted at all: it was just what they said and thought they wanted), but also clustering reports of which requirements were the 'heaviest' having the most dependencies and which requirements were show-stoppers to how many follow-on requirements.

Then I uploaded my Haskell Graph into Neo4J, making heavily-clustered requirements an obvious visual cue. And we won that contract.

The project wasn't attempted in Java. The project was attempted in R, and it couldn't be done. They estimated the graph manipulation algorithm would be 200-lines of code in R, that they couldn't get working.

With comonads, I did it in one line of Haskell. One line for a graph deforestation algorithm to get to the bare essentials of what was important to the project. Wanna see it?

How hard was that? In Haskell, a pure functional programming language, not hard at all.

Not only that, that we won a contract that our competing companies said was impossible, but our VP got wind of this and started vetting my tech to other companies.

We have a contract in the works, right now, using Haskell and Neo4J on AWS that is answering questions about fuzzy relations in social networks that a company that is expert in social engineering needs us to answer.

And I can answer these questions using graph theory and purely functional programming.

Case study 5: the one that got away

Oh, and then there was the one that got away. It had to do with a neural network I built in Mercury, a purely functional logic programming language with Prolog-like syntax that was able to classify images into 'interesting' and (mostly) 'not-interesting' but 'interesting' had very specific, different meanings, and it was able to classify these images, using a pulse-coupled neural network, in ways that eliminated 99% of waste images quickly so that analysts could concentrate on doing work, as opposed to sieving through the deluge of useless images to get the the ones they needed to see.

I build a working prototype and demoed it.

This had never been done before. Ever.

Then, a Big Six came in and said, 'we can do that for you with 250 programmers and Java' and stole the project. After ten years and billions of dollars, they were unable to reproduce my work.

Pure Functional Programming Claims IRL

So, let's do a real-money tally.

ATS-L in one month, in the three years I maintained it (it is still up and running ten years later, ladies and gentlemen) made $26 million dollars in seizures and rescued three teens being human-trafficked over the border.

CDDS has been in production since the year 2010 and has verified appraisals helping Fannie Mae to make 62 Billion dollars in net profit in one quarter the year it went live, actually contributing to the rescue of Fannie Mae from insolvency.

TMQER has rescued a government run program from failure that has the funding price-tag of over 100 Million dollars of Government (your) taxpayer (your) money. You're welcome.

Sunset dates I wish I had a dollar amount, but you can estimate for me: three teams of business analysts and software engineers over a six month period said it couldn't be done (or tried it and failed). I scrapped all that code, wrote the system from first principals (Category Theory) and got it working and approved in two months. You do the math.

... Oh, and then there's my current project. I might actually be able to own this thing. Hmmmm.

So, yes, Virginia,

1. there is a Santa Clause
2. those academics are actually onto something. (Pure) functional programming actually does matter. It actually does allow you to program better, faster and more cleanly, and with these enhanced skill-sets you become the one they turn to when other teams throw up their hands at an 'impossible' task. And then you deliver, ahead of expectations on both time to deliver and budget costs.

Hm.

CoReader x CoState == Silver Bullet

Thesis:

The title reads: 'CoReader x CoState == Silver Bullet' ... of course, it isn't a silver bullet, but it surely reduces (exponential) complexity AND provides just-in-time realized constants that (e.g.) would, and do, strain and break production systems working on very large data sets.

So, it isn't a silver bullet, but it sure-as-heavens-to-betsy puts the 'oh, I'll just generate the (resource intensive) constant' approach and then 'oh, I just wrap each layer in another layer of if-then-else (kill me with code complexity)' approach up and down walls and mops them off the floor, please, and thank you.

Antithesis:

Okay, so the CoReader (co)monad is suppose to give you a constant, and it does ...

CoReader readValue context = (context, readValue) -> context

with the askC function:

askC :: (CoReader r c) -> r

and so you have your constant.

The thing is, the CoReader depends on the context to be preexisting, which can be troublesome for data that are realized from an environment that is established at some point after the program begins to run, and that problem is that if you want a realized constant, you can't have the CoReader be a final value in some program component (the context isn't there yet), and then, if you wait to establish the CoReader until after the data set is realized, then you have all this checking code and establishment code, that can be rerun at any time (and, most likely, every time) you ask for that constant, which means that you don't have a constant at all, but a new value every time you ask over a data stream.

Sub-optimal.

So, how does one get a realized constant from a data stream that takes time to establish and guarantee that, yes, that constant I asked for upstream or in another branch is the same constant that I'm getting now from my CoReader, because I'm not re-creating it each time I ask for a constant from the data stream I get after the program is started and initialized?

... e.g. (or, 'i.e.' in my particular case): a database connection takes time, and is usually not available with a system that needs a CoReader to be constant from the get-go.

My solution, after much head-banging with trying to make the CoReader, itself, be something that it isn't, is to have the CoReader not read the stream directly, because it may not be there or be ready, but to have the CoReader read a different 'stream' entirely: the State monad ...

Or actually, since I don't want to push all my computations thereafter up into the monadic domain, to use the CoState comonad, a.k.a. the Context comonad:

Context context value = Context (context -> value) context

with the very helpful putC function:

putC :: Context (context -> value) context -> newValue -> Context (context -> newValue) context

You're probably scratching your head, saying, "I don't see any putC in the literature..."

And, sure, you are correct, there's 'experiment' and 'modify' but then you see where I'm going ... a little this plus that gets you to putC and that's what we want from our CoState comonad: I'm in a context, gimme the thing I stowed away in there for later use without me having to thread all the baggage of a monad throughout my computation.

You wrap that in a CoReader and then you have the constifying functionality you need to say, hey, once the state is establish, then give me that constant value from then on.

So, given a data stream, context, that comes to us somewhere down the road, we have a CoState comonad, which I'll call box (for a mutable boxed type indexing into that stream, or 'context'), paired with the CoReader, reader, to give you a constant value in the pure-functional (not monadic) domain.

That's half of it.

The other half is this: now that you've got CoState paired with CoReader, you, functionally, know that this is the first read into the context, or if this the nth read where you're just returning the read/realized value.

So, what if you want to do something the first time, that you don't want to do anytime thereafter.

AND, what if you don't wish to encumber yourself with the old

if this-is-the-first-time-i'm-reading-and-initializing-this-stupid-constant
then do-something-special-here
else just-return-the-stupid-constant

logic that, well, encumbers too much programming logic everywhere?

The answer is surprisingly easy, once you have the CoState x CoReader pairing ...

With a little bit of monadic magic to provide the glue.

Because, the first time you read the constant, or, that is to say, just before the first time the constant value is ground to what (it is just about to) be(come), it's value is ... wait for it ...

Nothing.

As in Nothing :: Maybe.

That is, if you lift that (non)value into the monadic domain:

uninitializedValue = Nothing
value x = Just x

And, boom! You use mplus (or (+)) to do something magical the first time:

val (+) magical-initialization-code

equals your solution with dispensing with stupid boolean flags called 'inited' and stupid if-then-else tests everywhere that are now totally unnecessary, ... just redefine askC:

askC reader = realAskC reader (+) magical-initialization-code

and then you just ask for the constant, the reader gives it to you, in the pure-functional domain, and automagically does initialization code only on the first read of that constant.

Synthesis:

Real-world example.

"Yeah, geophf," you're saying, "nice whitepaper, but what can I use it for?"

Um, impress hot babes at cocktail parties?

Get a free 'tall' latte at sbux, ... that is, after you pass over a fiver?

(Okay, seriously, 5 bucks for a small coffee? What has the world come to?)

AND:

So, let's say you're working on one of the world's largest databases (every company claims to have at least one of them ...), and you're getting 10,000 records per hour, and one section is generating up to 5 indices, but you don't know which indices you need, and which you don't, as it depends on data down-stream, and since these records are highly hierarchical XML files, with each of the layers of the hierarchy optional, we're talking WAY down stream.

So, one approach, you just getSequence from the database for all the sequences you may need, and then you're good, you insert rows with the prepopulated sequence.

Hm, 10000 records per hour, 24x7 data feed, and int is how big? 2 billion ... only?

(start-rant: Okay, does EACH AND EVERY TEAM have to relearn that you exhaust int much sooner than expected so that your 'vital system' crashes with dev and production teams working round the clock to change int into a much bigger number, when, after all, these are sequences, not (necessarily) numbers (GUID are hexa-strings-of-numerals, so why not make your precious sequence strings that you will NEVER do arithmetic on anyway, so you've set it as an int, why? :end-rant)

So, from a purely optional data input, you are probably generating all these sequences, and bringing your project to a screeching halt much sooner.

How about this.

Wrap your sequence generator in a CoReader comonad, and wrap each optional insert in the Maybe monad.

What does this give you?

In a hierarchy, it give you that you don't create a sequence you don't use, ever.

And, five levels down, when you actually do do an insert of a child record who's parent wasn't there in the data feed? It 'automagically' generates the parent record from the dataflow trace, inserts that parent first, and then inserts your child record without throwing an 'integrity violation, parent not found' error, and since it's done in the monadic domain, you wrote 0, zero, zip, nada conditional code at each later.

> return layer1 (+) deferInsert1 >>= \deferred1 -> insertm (reader1 askC) [] >>
> return layer2 (+) deferInsert2 >>= \deferred2 -> insertm (reader2 askC) [deferred1] >>

etc, etc, to your heart's content.

Which is a very different thing than:

let seq1 = getSequence
    seq2 = getSequence
    seq3 = getSequence
...
in do val1 <- return layer1
        val2 <- return layer2
        val3 <- return layer3
...
        insertm seq1 val1
        insertm seq2 val2
        insertm seq3 val3
...

This example trivializes things a bit. Looking at it, it's obvious you want to defer the creation of the sequence until you know you have that row to insert into your database. Unfortunately, it is that obvious, and we intend to create thing only when needed, but then the 'real world' rears it's ugly head, and you're looking at a branching data flow where your sequence may be needed downstream if any of the branches result in an insert, and your tree is not binary, but nary, and several layers deep.

What do you do as a programmer?

Well, you just create the sequence, because you know you're going to use, because your test data says so, but the real stream is never that dogmatic and a whole lot sparser than your 'in-a-perfect-world' test cases anticipated.

And you have hundreds of tables, ... and just one of the tables has 100 million records added to it, each month.

Do you see where 'just create the sequence' can be a key factor in crippling your application soon after it goes live?

'Just create the sequence, because for each layer, a layer of complexity is added to this already complex system,' is one approach, and in industry, pretty much the standard approach.

I propose another way; in sum: the way proposed in this paper. Use the CoState x CoReader comonadic pairing to take all the complexity, put it into a paper bag, and then throw that bag into a black hole where you'll never see it again. If your data set is rife with optionality, that is: semideterminism, then wrap that semideterminism in a Maybe monad, and you have a powerful framework that is simple to understand and easy to use.

In a DBMS system I'm working on, the SLOC (lines of code) count decreased, AND the number of sequences generated decreased by a factor of 2-4. We were generating twice to four times as many sequences that we needed, ... per input record. With the CoState x CoReader x Maybe framework in place, this spurious sequence generation stopped, and the code became more declarative: I was saying what was to be done, much more so than how to go about doing it.

Math is hard

So, here's an interesting, everyday conundrum, sent to me by a reader:

Hello my mathematical genius friend. :) [should I edit that out? *blush*]

I have been sent the following mathematical joke of sorts. The person who sent it to me claims there are no flaws in it. But obviously there has to be a flaw, because the conclusion is incorrect. The problem is that I don't know how to explain the flaw---but I suspect it happens in that third line where it attempts to equate squared cents with squared dollars. Is there any way that you could explain the flaw in such a way that a seventeen year old Norwegian would understand? Don't worry, you don't have to say it in Norwegian, he speaks English.

If you don't mind having a look at this and explaining, I would be ever so obliged. And no pressure, but...pants may be on the line in this little bet I've entered into.

1. $1= 100¢ (so $0.1 = 10¢)
2. And, 100¢ = 10¢²
3. Then, 10¢² = $0.1²
4. $0.1² = $0.01
5. $0.01 = 1¢

The implied conclusion is

∴ 'a dime squared equals one penny'

Then we say 'Q.E.D.'.

Hm, if pants are on the line for my dear reader, I wonder what's on the line for moi-self (that is faux-French) (and 'faux' is French)? A review? Or two? Or three? Of my stories?

Let's leave my preening aside.

So, who sees the fallacies above that lead to the absurd conclusion?

If you do not see it, please think on this awhile before looking at the answer.

---

The answer

From basically the get-go this problem statement is erroneous and imprecise, but this comes from a fundamental laxity in understanding of what operators are and what operators do. Certainly, the first premise is correct: One hundred pennies does indeed equate to one dollar, for

1. 100¢ = $1

is a statement of fact about the conversion from one set of units (pennies or ¢) to another set (dollars or $). But already the trickster plays fast and loose, for indeed:

1. ... (so $0.1 = 10¢)

is still correct but the (implied) conclusion makes a statement about the square of dimes, not about the square of tenths of dollars.

Do you see the fallacy now?

No? Let's continue.

So 1. is true, insofar as I can throw it, and days where my back gives out (ah! me poor bones!) that's not very far, but it's far enough for this problem statement, so long as it goes no farther than that.

But it does. *sigh*

So now let's get into the lies:

2. 100¢ = (10¢)² [parentheses implied and erroneous]

This is a lie. It's a lie, lie, lie, lie, lie, lie, lie!

Huh?

The lie is this: a square of a thing is not the thing itself, and even if you know nothing about mathematics, you can prove this to yourself. Socrates did it with an unlettered and untutored slave boy, and you are further along than what Socrates had to work with.

So let's prove 2. false with an analogue.

Take a foot rule (sorry, my readers who do not follow the British Imperial system, which, oddly enough, includes Brits these days, too) and a large piece of butcher paper. Draw a line on the butcher paper measuring one foot.

a. _ = 1 foot.

Now, 'square' that line, by drawing three more lines to make a foot square on the butcher paper.

b. ❏ = 1 foot square.

So, is

c? 1 foot = 1 foot square

Obviously not! for that would be to say:

c? _ = ❏

Or, put another way, 'one thing of one thing is equivalent to one thing of entirely a different thing'. One gulp of water does not equal one gulp of bleach. One I wish to have with my breakfast, the other, I do not, as my father very unfortunately discovered the hard way one not-so-fine morning.

"But, but, but ..." you stutter angrily, "but isn't '10² = 100' a statement of fact?"

Yes, indeed, it is, but please remember '10 ≠ 10 things' is also a statement of fact. Number (with a capital 'N') is a class of classes as, e.g., Introduction to Mathematical Philosophy so clearly and succinctly explains.

The link goes right to the book, all 228 pages of it. It's a quick read, so please (re)read it.

So to state:

2. 100¢ = (10¢)² [parentheses implied and erroneous]

is to state a falsehood.

How do we correct it? Well, by replacing the (implied) error with an explicit (corrected) ordering:

2. 100¢ = 10²¢

Do you see the correction? It the former case, we erroneously squared the units along with the number, in the latter case we do not square the units, we square the number solely.

Do I need to go any further? Or, do the fallacies fall out obviously in the rest of the assertions?

For completeness sake, I will review each step.

3. (10¢)² = ($0.1)² [parentheses implied and erroneous]

No. (10 pennies) squared does not equal (1 dime) squared.
But, a statement of fact of conversion is that (10 pennies) = (1 dime), but that's as far as it goes, and no farther.

If we square pennies we have a new unit of measure called, I don't know: (¢²), and (¢²) does not equal dimes. Not in this world.

4. ($0.1)² = $0.01 [parentheses implied and erroneous]

Again, no.
Again: $(0.1²) = $0.01, but again, that is as far as you can go with that statement.

5. $0.01 = 1¢

is just a reformulation of the first statement and is true, yes, but redundant.

Remember from Frege's predicate calculus:

q |- p [read: 'q implied by p' or 'if p then q']

[Shoot! why don't they have an 'implied by' HTML character?]
But if:

¬p [read: 'not p' or 'p is not true (or provable)']

Then you can say anything you like for q, or, more correctly, you cannot say anything at all about q, because q does not depend on ¬p, it depends on p.

So, as it were: 'If I had done the laundry, we wouldn't have had this argument' and 'I didn't do the laundry' means I don't have a leg to stand on about why we had this argument, honey.

(Oops, sorry, ... but it's not like I have had this experience at all ...)

And, to the point of this article: 'If a (10 one-hundredths of a dollar) squared were to equal (10 pennies) squared, then ...'

Well, then anything, because '(10 one-hundredths of a dollar) squared = (10 pennies) squared' is false. So say away, because anything coming from a false premise is an absurd conclusion: whirled (black eyed) peas, butterflies flapping in the Amazon, and the Number 23. They may be true enough in their own right, but you can say nothing about them from the false premise.

So, let's take the absurd conclusion:

∴ 'a dime squared equals one penny'

and reformulate it to be a true statement.

Well, the first thing we have to do it to get rid of the '∴', so let's do that, and then state the plain facts:

'a dime squared equals one penny' is an absurdity.

Q.E.D.

---

What can we take away from this?

Let's examine another absurity:

'Math is hard.'

No.

No, my dear ladies and gentlemen, 'Math' isn't 'hard.' Math is simple. Math can even be easy, for we learned from the Greeks, after all, that Math is one of the humanities. Math is simply a language. A language that can describe things exactly as they are and exactly as they are not. And precisely at that. It can even describe imprecision precisely. The 'hard'ness of mathematics comes from us, when we don't wish to be precise in what we are talking or thinking about.

Being precise ... well, that can be hard, I suppose, so then perhaps it's more precise not to say 'Math is hard' but to say 'Life is hard.'

Yes. That's true. 'Life is hard' as we choose to make it.

Oh, well. I never promised you a Rose Garden.