A Big Introduction to a Big Topic

Source: genius.com

Ride the air
In whirlwind. - Milton, "Paradise Lost"

Every post on this blog has the same subliminal theme. We stand on a deck encircling the cupola of a lighthouse set on a bluff before the ocean. The waters lap at the sands below the cliff with the familiar rush and hiss of waves breaking rhythmically on the shore. We feel the sea breeze and smell its salted fragrance as the schooner cuts its way across the horizon. Impulsively and unconciously, we draw the collar of our Navy pea coat more tightly around our neck and pull our cap down a bit to ward off the cold bite of the sea's blustery wind. And, naturally, we wonder what the new day will bring as we study the wind and waves and watch the clouds. Is the zephyr a gentle breeze or a rising series of gusts carrying the scent of rain and ozone? Are the waves rolling in gently and regularly, or are they growing into foamy, thundering masses mounted with sea spray tearing off their crests? Are those clouds high, light and scattered, or are they gathering and darkening?

Just about every major technology trend today is unwittingly converging with all the others. The convergence is apparent only when they are all viewed side by side, and even then it is a subtle realization. Let's put them all in a row and see if we can isolate their commonalities and discern any underlying meaning.

The IoT - Chip firms, software developers, wireless & wired communications providers and systems houses of all shapes and sizes are drooling over the possibilities of everything in the world being connected. We stand on the verge of having all that we do both during work and leisure becoming an activity that involves interaction with the Web. 

Parties outside of the technology sphere are keenly interested in this as well, whether it be marketers and advertisers in a wide array of business segments or more sinister elements in government and the criminal world. Stated differently: there are people dedicated to bringing more and more of the world to us thru electronic means and others who are lurking to scrutinize what we do in that digital universe.

Robotics - The dream of everyone struggling along at the Fukushima or Chernobyl sites is to have a mechanism whereby they can act remotely within those astonishingly lethal environments. Plant managers ponder the possibilities of defect-free production lines where work assignments can be doled out with no risk of misinterpretation or lack of attention and managed at will thru a software update. Shippers, mil/aero firms and vehicle manufacturers envision a world where fatigue or errors of judgement are no longer part of the equation for delivering packages & cargo, travelling by air, land and sea or patrolling for guerillas. After all, robots never get tired or lose concentration on the task at hand, while remote human operators & controllers can use them to report activities, gather field observations and analyze the telemetry of their autonomic servants at leisure.

AI - But what if the robot controllers themselves need a helping hand? A mind which never needs a break, which always has the patience and energy to chase down every last detail, pursue all lines of inquiry to the end and quickly sift thru any information presented to it - no matter the volume- overcomes the physical limitations of ordinary mortals.

Consumer Electronics - though it has decidedly slowed down as a focus of technology innovation, today's HDTVs and STBs are doing more than merely supporting 4K and 8K video resolutions that the human eye can't readily distinguished from 1080p at normal viewing ranges. These boxes are increasingly moving towards decidedly very large bandwidth & high performance internet access to support HD streaming media as well as permitting the collation of information on viewing and recording patterns, including choices and temporal context. In a sense, television and internet access in the new digital home are set to become a subset of the wider IoT market.

Mobile Computing - even smartphones and tablets, up until recently the darlings and drivers of the technology market and now fading into senescence, are poised to fall under the purview of the greater IoT sphere. They will, in fact, be part of a much wider array of portable computing devices (though it is almost certain that a converged tablet/smartphone device will, in fact, be the administrative and processing centerpiece of the segment.) Wearables in the form of watches, jewelry, lapel pins, shirt collar buttons, glasses and even goofy looking VR headsets will all impose themselves on the way people interact with their surroundings and each other, but for good and evil, in socially, educationally, politically and economically constructive and destructive fashion.

Datacenters - There is a reason why so many system, chip, semiconductor IP and software companies are setting their sights on datacenters and server farms. With all of the above developing or transforming markets set to use the Web for a major portion of their functions and activities, the ability to handle the traffic and put aside any information gathered for further use is bound to dramatically expand the need for computing and storage space in server rooms and warehouses. A purely B2B market, the worldwide opportunity is a lucrative one indeed.

Semiconductors - as Moore's Law breathes its last and chip developers contend with silicon lattices that have reached their physical limits in sub-28nm nodes to extract incremental 3P (power, performance and price) improvements thru creative design techniques, the industry is endeavoring desperately to stay relevant. It is the chip companies, after all, who are facing the business end of the spear as system houses pressure them to develop component HW and SW solutions for the above emerging and evolving markets and their insatiable hunger for more speed and bandwidth at ever lower dynamic & static power, along with cost.

Part of the semiconductor industry response is to make the most of what they have in order to provide sufficiently compelling value. Incredibly cheap and amazingly functional development board families like arduino, raspberry pi, beaglebone and jetson offer SoCs, MCUs and GPUs from Atmel, NVidia, TI and others for system level prototype development of IoT and Robotics applications.

Further refinements and enhancements of the existing chip technology base include continuing applied R&D for monolithic 3D-IC, fundamental alterations to bulk CMOS such as SiGe & FD-SOI and new memory architectures like RRAM. Intel and Micron made quite a splash just a few days ago with a tantalizing and mysterious crossbar memory structure they call 3D Xpoint, which promises nonvolatility, far greater densities than DRAM and much greater speeds and endurance than Flash:

http://www.computerworld.com/article/2951869/computer-hardware/intel-and-micron-unveil-3d-xpoint-a-new-class-of-memory.html?phint=newt%3Dcomputerworld_emerging_technologies&phint=idg_eid%3D4a1ad9e203f021295e9e79b4966a526d#tk.CTWNLE_nlt_emgtech_2015-07-29

http://www.computerworld.com/article/2952644/computer-hardware/3d-xpoint-memory-could-change-computing-as-much-as-ssds.html?phint=newt%3Dcomputerworld_emerging_technologies&phint=idg_eid%3D4a1ad9e203f021295e9e79b4966a526d#tk.CTWNLE_nlt_emgtech_2015-07-29

http://www.wired.com/2015/07/deceptive-simplicity-intels-new-memory-tech/

Despite all this terrific work, semiconductors are nonetheless experiencing a relentless value squeeze. Designs are continuing to converge on massively parallel processing thru multiple heterogenous embedded CPUs, MCUs, DSPs, GPUs, VPUs and a broad variety of hardwired accelerators. Despite colossal advances in processing muscle and storage capacity, systems houses are inexorably driving chip hardware towards commoditization - hence the value and price crunch. This will provide further impetus to drive standardized core elements (I/Fs, protocol stacks, radios and such) out of the chip houses to IP providers in an effort to minimize fixed costs that add no intrinsic value to product offerings, much in the same manner that systems companies pushed system integration requirements down the technology chain to the semiconductor industry in the last twenty five years.

Commoditization threatens to sound the death knell for the semiconductor industry, reducing its participants to a level of stagnation painfully familiar to brick and nail manufacturers. Such a menace will undoubtedly add fuel to the fire for discovering a legitimate successor to silicon, such as graphene, phosphorene, a silicon-based hexagonal structure somewhat analogous to graphene called silicene, and even more exotic combinations of elements including gallium nitride and molybdenum & sulfur.

The Common Thread

But more wonderful than the lore of old men and the lore of books is the secret lore of ocean. - H.P. Lovecraft, "The White Ship"

The centerpiece of all this technology development, its very beating heart, is almost never mentioned. Yet all of these sectors are pursuing it, expanding it, driving and being driven by it.

Data.

THAT is the ocean on which we sail. The Seven Seas of High Tech are a roiling, ever-changing ferment of information in myriad forms. 

Vast pools of computing power, combined with virtualization that facilitates sharing of resources, higher infrastructure efficiency and resilience against hardware failures or lack of accessibility is fueling the Big Data era. Having machines - virtual or not - which store and compute using vast data fields on machines that are always up and running, are intrinsically fault tolerant and which can support massively parallel processing loads at low latencies are the basis of everything we are doing now and striving to create in the development and deployment of High Technology.

Source: desktopwallpapers8.com

The underlying motive force of all our electronic HW and SW involves the gathering, packaging, transmission, reception, organization, protection and interpretation of near infinite volumes of ones and zeroes. But as we push our electronic miracles into more facets of life and saturate every aspect of our society with digital gizmos, the data itself begins to have an intrinsic value. Facebook, Yahoo, Google and scores of other firms receive staggering wealth from the raw data they garner from searches, viewed articles and social media activities. Governments have expanded their reach and spheres of influence for law enforcement, people tracking, espionage and military applications.

Data Science is increasingly perceived as a means of discovering and adding value to offerings. Yesterday's statisticians with their linear regressions and bell curves are becoming tomorrow's data scientists with a vast selection of programming languages, tools, databases and both linear and non-linear methodologies. Packaged goods, fashion, retail, media, advertising and even industrial market companies are dreaming of ways to enhance everything they do and increase their appeal to consumers and B2B clients. 

The key is to find something different - making discoveries in the data of which you were unaware and for which you weren't even deliberately searching. Finding such "unknown unknowns" is the very essence of Data Science, akin to finding long lost pirate treasure under the sparkling seas of the Spanish Main. 

As the IoT grows, this computing power (both HW and SW) will inevitably get stretched, requiring ever more raw HW capacity both in terms of computing and storage. Robotics will also provide a tremendous flood of data, with AI becoming not only useful but absolutely essential in processing, interpreting and initiating action on it. Nothing in High Tech will be left untouched by the growing ocean of Big Data.

Because of its position at the heart of High Tech, Big Data is going to become a dominant topic on this blog, as it is the impetus behind all that will transform our world in the future. We'll begin the discussion with an examination of the already existing software infrastructure that is helping bring that future to realization. I guarantee you, dear readers, that this is going to be one helluva fun ride. :-)

The only joy in the world is to begin. It is good to be alive because living is beginning, always, every moment. When this sensation is lacking—as when one is in prison, or ill, or stupid, or when living has become a habit—one might as well be dead. - Cesare Pavese
___________________________________________________________________________

Postscript:

This blog and everyone after it is in memory of Nafud:
March 1997 - July 27 2015
The sweetest and best cat that ever was. Never hissed, bit or scratched, playful, lovable and wonderful.