ICS/Lotus (mostly), Linux, Travel, Skiing, Mixology, and Random Musing of Interest

Shellshock - Final Fix Released: Time to Re-Patch

Bill Malchisky  September 29 2014 12:17:00 AM
Author's Note: Thank you to the ICS community for their tremendous support of my first Shellshock post. For those that read it early, you received critical information 14-72 hours before many sites released their stories. Several readers were fully patched before big names tweeted the issue. You were well ahead of the curve. Shellshock stories released over the weekend proved outdated and incomplete. This post provides better information faster. I am grateful for your support.


As I mentioned last week, the Shellshock bug is real, but the then available fix handled all exploits but one. Very early on Saturday, 27 September 2014, a patch became available after two days of community scrutiny. Getting patched for this exploit is important to ensure a full complete production-grade solution for the Shellshock bug. Fortunately, the fix for this and two the two additional Bash exploits identified is trivial to apply.


New Exploits Identified

"The original flaw in Bash was assigned CVE-2014-6271. Shortly after that issue went public a researcher found a similar flaw that wasn’t blocked by the first fix and this was assigned CVE-2014-7169. Later, Red Hat Product Security researcher Florian Weimer found additional problems and they were assigned CVE-2014-7186 and CVE-2014-7187. It’s possible that other issues will be found in the future and assigned a CVE designator even if they are blocked by the existing patches." --Via Red Hat's Shellshock FAQ

The excellent work of Florian Weimer at Red Hat in identifying two additional moderate exploits. Because of this, you should see reference to the later exploits if you are using a GUI update tool like Ubuntu's Update Manager (image below). Regardless, follow the respective distro update step provided below to ensure you are protected. Then, test for success.

Red Hat updated their Shellshock Impact Statement for this issue.

Image:Shellshock - Final Fix Released: Time to Re-Patch


Testing for the Initial Remaining Exploit
-- CVE-2014-7169
The fix for CVE-2014-7169 ensures that the system is protected from the file creation issue. To test if your version of Bash is vulnerable to CVE-2014-7169, run the following command:

[malchw@localhost Desktop]$ cd /tmp; rm -f /tmp/echo; env 'x=() { (a)=>\' bash -c "echo date"; cat /tmp/echo


Positive Result

bash: x: line 1: syntax error near unexpected token `='

bash: x: line 1: `'

bash: error importing function definition for `x'

Sun Sep 28 00:03:50 PDT 2014


Negative Result

date

cat: /tmp/echo: No such file or directory


Notations

1. If you are running any Linux appliance, security server, or server application on Linux such as IBM Protector, ensure you test for this exploit
2. Apple Macintosh computers running OS X are in-scope, albeit casual users are a lower risk, power users should take this exploit seriously
3. No reboot is required when updating Bash
4. The fix for CVE-2014-7169 includes fixes for CVE-2014-7186 and CVE-2014-7187 if you updated Bash on or after Saturday, 27 September: indicated with RHSA-2014:1306-1, RHSA-2014:1311-1, and RHSA-2014:1312-1
(Japanese coding fix)
-- RHSA = Red Hat Security Advisory
5. Red Hat just released a Shellshock Vulnerability Detector shell script which you can run instead -- available here
6. The fix for CVE-2014-7169 is Important and should be patched; the two new moderate exploits being addressed is not justification for this type of blog post, just a bonus


Applying the Fix

Some distros released a Bash update early Sunday morning, 28 September. Ubuntu's fix hit my machines at 2:15 am EDT.
Red Hat made the fix available via RHN and all registered systems can download it easily, else you download the file from RHN for manual installation.
The confirmation section below shows you how to ensure you have the correct patch installed, as the fix version management can get confusing.
RHEL: # yum update bash
On my older RHEL 5 box: # rpm -Uvh bash-3.2-33.el5_11.4.i386.rpm
Centos: # yum update bash
Ubuntu: $update-manager -or- $sudo apt-get update

Notations

1. If you receive the message, "No Packages marked for Update", then run # yum clean all && yum bash install
2. If you are still seeing this message and you have not updated bash, pull the latest file from your distro's support site
3. Apple was notified privately by the Bash maintainer several times along with a patch to use: Apple still has not released a fix (as of this post's time-stamp)
4. Hat tip Frank for providing a Mac solution for power users, located here

UPDATES
5. Apple finally released a Bash update for Mavericks, Lion, and Mountain Lion via App Store, as of dinner time, EDT
Hat tip Theo for the patch link
6. IBM released an updated Bash patch for Protector over the weekend, replacing Friday's Bash patch
Hat tip Mathieu for the patch update


Example Output - RHEL 6.5 Client

[root@localhost ~]# yum update bash

Loaded plugins: product-id, refresh-packagekit, rhnplugin, security,

 : subscription-manager

This system is receiving updates from Red Hat Subscription Management.

This system is receiving updates from RHN Classic or RHN Satellite.

rhel-6-desktop-rpms                                      | 3.7 kB     00:00    
rhel-6-desktop-rpms/primary_db                           |  27 MB     01:10    
rhel-x86_64-client-6                                     | 1.8 kB     00:00    
rhel-x86_64-client-6/primary                             |  18 MB     00:19    
rhel-x86_64-client-6                                                10417/10417

Setting up Update Process

Resolving Dependencies

--> Running transaction check

---> Package bash.x86_64 0:4.1.2-15.el6_4 will be updated

---> Package bash.x86_64 0:4.1.2-15.el6_5.2 will be an update

--> Finished Dependency Resolution


Dependencies Resolved


================================================================================

Package    Arch         Version                Repository                 Size

================================================================================

Updating:

bash       x86_64       4.1.2-15.el6_5.2       rhel-6-desktop-rpms       905 k


Transaction Summary

================================================================================

Upgrade       1 Package(s)


Total download size: 905 k

Is this ok [y/N]: y

Downloading Packages:

bash-4.1.2-15.el6_5.2.x86_64.rpm                         | 905 kB     00:02    
Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

Updating   : bash-4.1.2-15.el6_5.2.x86_64                                 1/2
Cleanup    : bash-4.1.2-15.el6_4.x86_64                                   2/2
rhel-6-desktop-rpms/productid                            | 1.7 kB     00:00    
Verifying  : bash-4.1.2-15.el6_5.2.x86_64                                 1/2
Verifying  : bash-4.1.2-15.el6_4.x86_64                                   2/2

Updated:

bash.x86_64 0:4.1.2-15.el6_5.2                                                


Complete!



Confirmation of Success

[root@localhost ~]# cd /tmp; rm -f /tmp/echo; env 'x=() { (a)=>\' bash -c "echo date"; cat /tmp/echo

date

cat: /tmp/echo: No such file or directory

[root@localhost tmp]#


On Red Hat based systems, you want to ensure that you have the ".2" release for your respective newer version, as below for my RHEL 6.5 box
[root@localhost tmp]# rpm -qi bash-4.1.2

rpm -qi bash-4.1.2

Name        : bash                         Relocations: (not relocatable)

Version     : 4.1.2                             Vendor: Red Hat, Inc.

Release     : 15.el6_5.2  
                 Build Date: Thu 25 Sep 2014 08:10:26 AM PDT
Install Date: Sun 28 Sep 2014 12:16:27 AM PDT      Build Host: x86-023.build.eng.bos.redhat.com


Results after the first Shellshock Bash release fix -- using my CentOS 6.5 box, which fails the above test (patched after this query).
[bill@localhost tmp]$ rpm -qi bash

Name        : bash                         Relocations: (not relocatable)

Version     : 4.1.2                             Vendor: CentOS

Release     : 15.el6_5.1
                   Build Date: Wed 24 Sep 2014 07:45:54 AM PDT
Install Date: Wed 24 Sep 2014 11:05:39 PM PDT      Build Host: c6b8.bsys.dev.cen


Red Hat Bash Releases with the New Fix (Also Addressing CentOS)

* RHEL 7 - bash-4.2.45-5.el7_0.4
* RHEL 6 - bash-4.1.2-15.el6_5.2
* RHEL 5 - bash-3.2-33.el5_11.4


Additional Mitigation Options

The linked document contains several mitigations if you are waiting for approval to patch, or are unable to patch your servers.
Via Red Hat -- Mitigating the shellshock vulnerability (CVE-2014-6271 and CVE-2014-7169)

Linux Bash Bug - Shellshock - is Real: Get Patched (AIX, Mac Too)

Bill Malchisky  September 25 2014 06:00:00 AM
This is ugly, but fortunately you just have to update to a fixed Bash version and you are fine (for now). No need to reboot your system either. Red Hat is out early on this and escalated this appropriately. Their first round of updates got all but one exploit permutation, so they re-issued another bug identifier and are working to close it soon.

Their initial timeline: Red Hat announced the bug on 14 Sep, had a proposed upstream patch seven hours later (0500h 15 Sep), backported it to Bash 3.0, 3.1, 3,2, 4.0, 4.1, and 4.2 three days later on 18 Sep; announced the release 1h later and made it public with an updated issue description six hours after that. Pretty impressive. On the 24th, Red Hat provided public documentation on this matter; six hours later it was reported that the fix is missing one exploit, so they are working to resolve that as I write this post. Things move fast in the world of open source.

Impact Statement
from Red Hat, provides direct prose for the next two sections. "Bash Code Injection Vulnerability via Specially Crafted Environment Variables (CVE-2014-6271)"

Abstract Update

Red Hat has become aware that the patch for
CVE-2014-6271 is incomplete. An attacker can provide specially-crafted environment variables containing arbitrary commands that will be executed on vulnerable systems under certain conditions. The new issue has been assigned CVE-2014-7169. Red Hat is working on patches in conjunction with the upstream developers as a critical priority.

How does this impact systems

This issue affects all products which use the Bash shell and parse values of environment variables. This issue is especially dangerous as there are many possible ways Bash can be called by an application. Quite often if an application executes another binary, Bash is invoked to accomplish this. Because of the pervasive use of the Bash shell, this issue is quite serious and should be treated as such.
All versions prior to those listed as updates for this issue are vulnerable to some degree.


Test If You Have The Bug

malchw@san-domino:~/Documents/$ env x='() { :;}; echo vulnerable' bash -c "echo this is a test" env x='() { :;}; echo vulnerable' bash -c "echo this is a test"


Positive Result

vulnerable

this is a test


Negative Result

bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `x'
this is a test


Notations
1. Your response may be something similar and be just fine; the difference is getting noise versus a clean response as the positive result indicates
2. Run this check on any Apple Mac product running OS X, as tests show Macs are vulnerable
3. Anyone running AIX, Solaris, or HP-UX should also check, as Bash is available on those systems

Addition - 26 September 2014
4. Update -- IBM released a patch for Protector that addresses Shellshock; verified by a customer of its success


Mitigation

Red Hat's Security Blog has a detailed analysis of which programs utilizing Bash can cause issues and why. "Bash specially-crafted environment variables code injection attack"


Resolution

Ideally, you need to be running bash-4.1.2-15 with current RHEL versions. Despite the bug's significance, the fix is really easy.
RHEL: #yum clean all && yum update bash
On my older RHEL 5 box: # rpm -Uvh bash-3.2-33.el5.1.i386.rpm

CentOS: #yum clean all && yum update bash
Ubuntu: $update-manager -or- $sudo apt-get update

If you know the version number, you can always specify it too (package name example is for RHEL6.5)
# yum update bash-4.1.2-15.el6_5.1


-OR-
Get the update manually and update the RPM -> https://rhn.redhat.com/rhn/errata/details/Packages.do?eid=27888

Note
: the "clean all" parameter above tells yum to clean all cached data, ensuring that bash can be updated more reliably, particularly with older systems; it may be considered optional on current systems


Distro Provided Resolution Documents

CentOS posted a document on the exploit and obtaining fixes through their list serv, "[CentOS] Critical update for bash released today."
Red Hat's is here: "Resolution for Bash Code Injection Vulnerability via Specially Crafted Environment Variables (CVE-2014-6271) in Red Hat Enterprise Linux"
Novell/SUSE; bug report with patches here
Debian
Ubuntu

Example Output - CentOS 6.5

[root@localhost ~]# yum clean all && yum update bash

Loaded plugins: fastestmirror, refresh-packagekit, security

Cleaning repos: base extras updates

Cleaning up Everything

Cleaning up list of fastest mirrors

Loaded plugins: fastestmirror, refresh-packagekit, security

Determining fastest mirrors

* base: centos.chi.host-engine.com

* extras: cosmos.cites.illinois.edu

* updates: mirror.atlanticmetro.net

base                                                     | 3.7 kB     00:00    
base/primary_db                                          | 4.4 MB     00:05    
extras                                                   | 3.3 kB     00:00    
extras/primary_db                                        |  19 kB     00:00    
updates                                                  | 3.4 kB     00:00    
updates/primary_db                                       | 5.3 MB     00:06    
Setting up Update Process

Loaded plugins: fastestmirror, refresh-packagekit, security

Loading mirror speeds from cached hostfile

* base: mirrors.lga7.us.voxel.net

* extras: mirror.es.its.nyu.edu

* updates: centos.aol.com

Setting up Update Process

Resolving Dependencies

--> Running transaction check

---> Package bash.x86_64 0:4.1.2-15.el6_4 will be updated

---> Package bash.x86_64 0:4.1.2-15.el6_5.1 will be an update

--> Finished Dependency Resolution


Dependencies Resolved

================================================================================

Package       Arch            Version                   Repository        Size

================================================================================

Updating:

bash          x86_64          4.1.2-15.el6_5.1          updates          905 k


Transaction Summary

================================================================================

Upgrade       1 Package(s)


Total download size: 905 k

Is this ok [y/N]: y

Downloading Packages:

bash-4.1.2-15.el6_5.1.x86_64.rpm                         | 905 kB     00:00    
Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

Updating   : bash-4.1.2-15.el6_5.1.x86_64                                 1/2
Cleanup    : bash-4.1.2-15.el6_4.x86_64                                   2/2
Verifying  : bash-4.1.2-15.el6_5.1.x86_64                                 1/2
Verifying  : bash-4.1.2-15.el6_4.x86_64                                   2/2

Updated:

bash.x86_64 0:4.1.2-15.el6_5.1                                                


Complete!



Quick and Dirty Work-around
, provided by Jake DePoy
# iptables --append INPUT -m string --algo kmp --hex-string '|28 29 20 7B|' --jump DROP


The Red Hat Customer Portal indicates the risk with the above work-around,
"Is not a good option because the attacker can easily send one or two characters per packet and avoid this signature easily. However, it may provide an overview of automated attempts at exploiting this vulnerability."

Ryder Cup Skype Chat Announced

Bill Malchisky  September 24 2014 08:07:54 PM
Image:Ryder Cup Skype Chat Announced

It is that time again when the best pro golfers in The United States of America take on the best pro golfers in Europe for the coveted Ryder Cup. This year, it is played at the beautiful Gleneagles course in Scotland designed by pro golf legend Jack Nicklaus, who describes his course hole by hole. Play commences Friday, Saturday at 7:35am local time, or 2:35am EDT, with a more respectable Sunday start at 11:36a local time, or 6:36a EDT for singles play.

I will open a Skype chat for the event. If you would like to join, leave a comment below, DM me, e-mail me with the subject ("Ryder Cup") or send me a message on Skype directly. Good luck and may the team with the best overall record win. :)



Team USA's site is here
Team Europe's site is here

Electronic Coverage Options

Watch the live stream here;
The iOS app is here;
The Android app is here;
The TV app is here for Samsung devices.

Big News for ICS Partners!

Bill Malchisky  September 17 2014 09:43:00 PM
After three years of working with IBM, I am proud to make the first public announcement of the beta milestone of a new IBM community feedback continuity tool entitled, Voice of the Partner. ICS is behind this at the highest levels and there is a strong desire within IBM to make this a success.

Imagine that as a partner you have a tool where you can input ideas and concerns to IBM and receive a response in a meaningful way, that also ensures continuity of feedback throughout the issue's life span. This tool will be a handshake if you will, from the partner community to IBM on pressing matters where other partners can contribute, join the conversation on relevant issues to them, share their concerns and be part of the process.

The goal of Voice of the Partner is to strengthen the relationship between IBM and its valued partners.

I have been working with IBM executives to create this bi-directional communication tool through which partners input concerning matters and then receive a response through a visible dashboard including a resolution timetable, issue owner, and item status with details -- updated quarterly. Each item of course contains a link to the underlying forum details to provide context and continuity.


Timeline

We are live in Greenhouse -- ready for the beta
Image:Big News for ICS Partners!

Beta 1: 19 September -- six week duration; small participation already filled; finalize dashboard, review workflow
Beta 2:  3 November -- six week duration; direct input from expanded participants, logistics review, debugging; (Still finalizing this list--if your company is interested in filling a slot, contact me)
Launch: 15 December -- go live date for the entire partner community

I want to thank IBM for working with me to help ensure this tool became a reality. Please comment below with any questions you might have.

Using Sametime Mobile? Avoid iOS 8 for Now

Bill Malchisky  September 16 2014 04:36:15 AM
IBM released a Technote yesterday on the issues with their Sametime Mobile applications on iPhones and iPads running iOS 8 -- due for release on Wednesday, 17 September 2014. My friends Gabriella Davis and Matteo Bisi both blogged on the Technote. Beyond that, there exists a post on The Sametime Blog offering a behind-the-scenes look as to the challenges therein, written by the on-premises Sametime Product Manager - Marlon Machado. In meeting Marlon previously, I can tell you he is a good guy and I appreciate his candor in getting ahead of this, which allows customers to plan and avoid internal help desk calls. I dislike telling customers that they have to wait to upgrade their Apple mobile devices, but if your users want to run Sametime, then you need to tell them now: wait.

Here is Marlon's post, via The Sametime Blog entitled, "About IBM Sametime Mobile Apps and iOS8".

I AM Speaking at ICON UK

Bill Malchisky  September 11 2014 10:00:00 AM
Image:I AM Speaking at ICON UK

Long story short, I will be speaking this Friday, 12 September in London, for the ICON UK renaissance. You can find me acting as emcee for the Ask IBM session at 2:00pm (1400h) and then again at 3:45pm (1545h) presenting The Headless Collaborator: Sametime 9 Command Line Install.

If you are in London for this wonderful event, please do say, "Hi," or better yet, attend one of my sessions. See you Friday!

iNotes Users -- Chrome 37 Creates Compatibility Issues

Bill Malchisky  September 5 2014 01:35:57 PM
IBM released today, a new Technote for iNotes users, entitled, "Some iNotes operations fail to work correctly in Chrome browsers upgraded to Chrome version 37" and is available here.

These five key areas introducing concern stem from Google deprecating the showModalDialog API.
1. Create/Edit Mail rule
2. Contacts Form and the Print action
3. Calendar view and results window displayed when using the "Import Holidays" action (off the "More" menu)
4. Preferences (Select default folder, Preferred rooms/resources site, Change HTTP password error, Security/Show ID info)
5. Validation of entered names/address from certain input forms where "Ambiguous Name", "Name not found" and "Certificate error" dialog might occur (Preferences, To Do, Group Calendar, Group, Phone Message)

As a work-around, Google is allowing the API to be available in a manually applied patch until May 1, 2105, with developer details located here.

The Technote provides two workarounds to handle the situation for impacted users--either one, or none may be appropriate for your organization, which then also introduces the avoidance permutation.

Updated -- Below here
As IBM codes a permanent fix by removing all requests for this API call, one hopes for an iNotes hotfix soon, particularly as Mozilla expressed interest in removing this API call too.


Further Reading

The initial change pitch is cited here
Blink Intents - Issue Dashboard spreadsheet captures the link, see row 59
"Intent to Remove: window.showModalDialog()" is linked here
"Window.showModalDialog: What it is and why you should never use it"
"Issue 345831: Delete showModalDialog" is linked here, commenced on 24 Feb 2014
For more information on iNotes, the documentation portal is located here

Ubuntu: Clearing GPG -- BADSIG Errors During Update

Bill Malchisky  September 3 2014
In updating my Ubuntu LTS host OS caused me to receive several GPG BADSIG errors, visually captured in the following Update Manager error message.

Image:Ubuntu: Clearing GPG -- BADSIG Errors During Update

You can see the errors more closely via a terminal window.

malchw@san-domino:~$ sudo apt-get update


W: A error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error:
http://extras.ubuntu.com precise Release: The following signatures were invalid: BADSIG 16126D3A3E5C1192 Ubuntu Extras Archive Automatic Signing Key

W: GPG error:
http://ppa.launchpad.net precise Release: The following signatures were invalid: BADSIG B22A95F88110A93A Launchpad PPA for Bumlebee Project

Two ways to resolve -- persistent and the brute force methods. I'll demonstrate the former first, then the latter for a stubborn error.


Persistent Solution

Step 0 -- malchw@san-domino:~$ sudo apt-get clean
Step 1 Combo -- malchw@san-domino:~$ sudo apt-get update && sudo apt-get upgrade

This was enough to get me to update my files sans issue, but the BADSIG errors remained.

The following packages have been kept back:

linux-image-generic

The following packages will be upgraded:

apparmor apport apport-gtk dh-apparmor flashplugin-installer

gnome-control-center gnome-control-center-data google-chrome-stable

icedtea-6-jre-cacao icedtea-6-jre-jamvm icedtea-7-jre-jamvm krb5-locales

libavcodec-extra-53 libavformat53 libavutil-extra-51 libc-bin libc-dev-bin

libc6 libc6:i386 libc6-dev libc6-i386 libgnome-control-center1

libgssapi-krb5-2 libgssapi-krb5-2:i386 libjs-jquery libk5crypto3

libk5crypto3:i386 libkrb5-3 libkrb5-3:i386 libkrb5support0

libkrb5support0:i386 libpostproc52 libswscale2 linux-libc-dev

multiarch-support openjdk-6-jre openjdk-6-jre-headless openjdk-6-jre-lib

openjdk-7-jre openjdk-7-jre-headless postfix python-apport

python-problem-report

43 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

Need to get 158 MB of archives.

After this operation, 4,047 kB of additional disk space will be used.

Do you want to continue [Y/n]? Y


Though pleased to have the long list of updated applications completed, I wanted to remove the errors. Onward to step two.
Wanting to rule out any caching issues for installed packages, I removed that from the permutation list.

Step 2 -- malchw@san-domino:~$ sudo apt-get update -o Acquire::http::No-Cache=True
Next, run Step 1 Combo, to see if things improve. For me, I needed to continue troubleshooting.

Thanks to AskUbuntu for the next command.
Step 3 -- malchw@san-domino:~$ sudo apt-get update -o Acquire::BrokenProxy=true
Then run Step 1 Combo.

If the command comes back sans error(s), then you are finished. Else, time for more drastic actions.


Brute Force Solution

The solution depicted below is listed on over three sites sites, thus I am unable to provide an accurate attribution.

malchw@san-domino:~$ sudo apt-get clean
malchw@san-domino:~$ cd /var/lib/apt
malchw@san-domino:/var/lib/apt$ sudo mv lists lists.old
malchw@san-domino:/var/lib/apt$ sudo mkdir -p lists/partial
malchw@san-domino:/var/lib/apt$ sudo apt-get clean
malchw@san-domino:/var/lib/apt$ sudo apt-get update


Hit
http://us.archive.ubuntu.com precise-updates Release.gpg
Hit
http://us.archive.ubuntu.com precise Release.gpg                          
Hit
http://us.archive.ubuntu.com precise-backports Release.gpg                
Hit
http://us.archive.ubuntu.com precise-updates Release        
...

Ign
http://ppa.launchpad.net precise/main Translation-en_US
Ign
http://ppa.launchpad.net precise/main Translation-en
Ign
http://ppa.launchpad.net precise/main Translation-en_US
Ign
http://ppa.launchpad.net precise/main Translation-en
Reading package lists... Done


Running the clean returned no results which is normal, but the update here, proved rewarding: came back error free. So, if the persistent approach does not clear your BADSIG error(s), then the brute force option is your best bet.

Good luck.

Fixing Public Key Repository Errors on Ubuntu

Bill Malchisky  September 2 2014 12:01:00 AM
Most of have seen at one point of another a error when updating packages, stemming from an untrusted repository.

W: GPG error:
http://security.ubuntu.com precise-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 40976EAF437D05B5

Here is one fix that works reliably well. Perhaps it will save you some time.

1. Syntax: sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys
2. $ sudo apt-get update

Repeat for each instance of the same received error after the update command in number two above. For this example, I received two NO_PUBKEY errors, one of which is listed above.

malchw@san-domino:~$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 40976EAF437D05B5

Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /tmp/tmp.rFoBoQHFbu --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv-keys 40976EAF437D05B5

gpg: requesting key 437D05B5 from hkp server keyserver.ubuntu.com

gpg: key 437D05B5: public key "Ubuntu Archive Automatic Signing Key " imported

gpg: no ultimately trusted keys found

gpg: Total number processed: 1

gpg:               imported: 1


malchw@san-domino:~$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B22A95F88110A93A
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /tmp/tmp.xHZdcG77tR --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv-keys B22A95F88110A93A

gpg: requesting key 8110A93A from hkp server keyserver.ubuntu.com

gpg: key 8110A93A: public key "Launchpad PPA for Bumlebee Project" imported

gpg: Total number processed: 1

gpg:               imported: 1  (RSA: 1)


Run step two above, which came back clean. All done. Nice and easy.



IBM’s Lost Art of Installation is Costing Them Revenue

Bill Malchisky  August 25 2014 02:50:00 PM
Preface - As I started this entry in June and in doing research to articulate the point, the case for installation simplicity grew dramatically. This post commences by illustrating the present state, how IBM did better previously (and still does with a different product), three customer examples of revenue impact, and closes with an offered solution for the present state. Please understand it is my intent to assist rather than besmirch, while being professional and respectful to both the community and IBM as a whole. Your comprehension of this perspective is appreciated.


IBM Connections 5 became available mid-June and like many people, immediately began the download process. To my surprise, I lacked sufficient space in my large files partition to handle the pull. I started constructing the trend line for how this product and others progressed overtime, contrasting that with Sametime Community Server and Domino, plus the process to start upon installing each product. My experience with the Sametime 9 installation proved just as troublesome. Amazingly, these results are diverging from IBM's excellent past, not progressing towards an improved way to handle their new set of complex (but capable) product offerings. A few conversations with colleagues and customers confirmed my thoughts and in the spirit of working with IBM to improve processes and products --- as performed previously, numerous times --- I wanted to illustrate a critical opportunity for IBM.


Download Footprint

Table of Linux i386 and x64 - limited file pull for EditLive!, Connections 5, Forms Experience Builder, and WAS (19 of 63 files to include Cognos, DB2, Tivoli, and CCM)

bill@san-domino:/dl/ibm.software/connections5$ ls

Connections_5.0_Cog_Wiz_LNX.tar       pp_qsg_10.1.1_ml.tar.gz

Connections_5.0_Wizards_lin_aix.tar   QS_FOR_WAS_ND_V8.5.5.zip

DMZ_SPS_1of2_WASNDv8.5.5.zip          WASND_v8.5.5_1of3.zip

DMZ_SPS_2of2_WASNDv8.5.5.zip          WASND_v8.5.5_2of3.zip

EditLive_5.0_Conne_MP_ML_QSG.zip      WASND_v8.5.5_3of3.zip

EditLive_v5.0_Conne_MP_ML.zip         WAS_ND_v8.5.5_Liberty.zip

FEB_v8.5.0.1_Linux_x86_ML.tar.gz      WAS_V8.5.5_SUPPL_1_OF_3.zip

FEB_v8.5.0.1_Multiplatform_ML_QS.zip  WAS_V8.5.5_SUPPL_2_OF_3.zip

IBM_Connections_5.0_Lin.tar           WAS_V8.5.5_SUPPL_3_OF_3.zip

IBM_Connections_5.0_QSG.zip


Image:IBM’s Lost Art of Installation is Costing Them Revenue

Where does one even start with this? The full download set for Linux on Intel is 63 files and 41.3GB, more than I had free on my laptop while traveling. It is also very easy to miss a file, as the Red Hat Linux OS selection includes eight Multiplatform eAssembly bundles with Windows, Linux x86, Linux x64, Linux on POWER, Linux on Z, and AIX all intermingled.  Unless you have time to peruse a significant quantity of online documentation or completed technical training to install their products, you may very well end-up hiring a consultant--or spending significant quantities of time with IBM technical support.


Naming and Build Consistency

The one noticeable improvement in Connections 5 over Connections 4.5 is more filenames are descriptive. To help visualize the contrast, please see my blog post on what Connections 4.5 resembled.

Unpacking IBM_Connections_5.0_QSG.zip provides 24 translations in 24 files of CIYQ4ML_[country-code].pdf, entitled, "IBM Connections Quick Start Guide for AIX, Windows, Linux Multilingual - Version 5", which loses the detailed filename construct (as does the EditLive! 5.0 QSG) and lacks a subdirectory like the other QSG packed files. The Connections 5 QSG provides links to a Technote and a Knowledge Center post to learn your requirements and how to install the product

Two problems inside this file:
1. The system requirements URL links to "System Requirements for EditLive! For IBM Connections 4.5 IFR2" -- the wrong product
2. The Installation URL provides a dead page

Image:IBM’s Lost Art of Installation is Costing Them Revenue

In verifying the downloaded archives' contents, many of the ZIP files lack a correlated or unique sub-directory to help manage the files. I find the lack of consistency with the provided packed files irksome.  To demonstrate, I created five QSG directories, unpacking one QSG variant into each to contrast; which guide is in which directory is inconsequential for this experiment. The results are below.

bill@san-domino:/dl/ibm.software/connections5$ ls -F qsg qsg?

qsg:

CIYQ4ML_ar.pdf  CIYQ4ML_es.pdf  CIYQ4ML_kk.pdf  CIYQ4ML_ru.pdf

CIYQ4ML_bg.pdf  CIYQ4ML_fr.pdf  CIYQ4ML_ko.pdf  CIYQ4ML_sk.pdf

CIYQ4ML_ca.pdf  CIYQ4ML_hu.pdf  CIYQ4ML_nl.pdf  CIYQ4ML_sl.pdf

CIYQ4ML_de.pdf  CIYQ4ML_it.pdf  CIYQ4ML_pl.pdf  CIYQ4ML_th.pdf

CIYQ4ML_el.pdf  CIYQ4ML_iw.pdf  CIYQ4ML_pt.pdf  CIYQ4ML_zh.pdf

CIYQ4ML_en.pdf  CIYQ4ML_ja.pdf  CIYQ4ML_ro.pdf  CIYQ4ML_zh_tw.pdf


qsg2:

installdiagrams/  quickstart/


qsg3:

quickstart/  vers/


qsg4:

CIZP2ML_ar.pdf  CIZP2ML_es.pdf  CIZP2ML_kk.pdf  CIZP2ML_sk.pdf

CIZP2ML_bg.pdf  CIZP2ML_fr.pdf  CIZP2ML_ko.pdf  CIZP2ML_sl.pdf

CIZP2ML_ca.pdf  CIZP2ML_hu.pdf  CIZP2ML_nl.pdf  CIZP2ML_th.pdf

CIZP2ML_de.pdf  CIZP2ML_it.pdf  CIZP2ML_pt.pdf  CIZP2ML_zh.pdf

CIZP2ML_el.pdf  CIZP2ML_iw.pdf  CIZP2ML_ro.pdf  CIZP2ML_zh_tw.pdf

CIZP2ML_en.pdf  CIZP2ML_ja.pdf  CIZP2ML_ru.pdf


qsg5:

FormsExpBuilder_v8.5_qsg_br.pdf     FormsExpBuilder_v8.5_qsg_ko.pdf

FormsExpBuilder_v8.5_qsg_cs_CZ.pdf  FormsExpBuilder_v8.5_qsg_ro.pdf

FormsExpBuilder_v8.5_qsg_en.pdf     FormsExpBuilder_v8.5_qsg_sk.pdf

FormsExpBuilder_v8.5_qsg_fr.pdf     FormsExpBuilder_v8.5_qsg_th.pdf

FormsExpBuilder_v8.5_qsg_hr.pdf     FormsExpBuilder_v8.5_qsg_zh_CN.pdf

FormsExpBuilder_v8.5_qsg_hu.pdf     FormsExpBuilder_v8.5_qsg_zh_TW.pdf

FormsExpBuilder_v8.5_qsg_ja.pdf

bill@san-domino:/dl/ibm.software/connections5$ ls -F qsg[23]/quickstart

qsg2/quickstart:

WAS8.5_nd_qsg_br.pdf  WAS8.5_nd_qsg_fr.pdf  WAS8.5_nd_qsg_ro.pdf

WAS8.5_nd_qsg_cs.pdf  WAS8.5_nd_qsg_hu.pdf  WAS8.5_nd_qsg_ru.pdf

WAS8.5_nd_qsg_de.pdf  WAS8.5_nd_qsg_it.pdf  WAS8.5_nd_qsg_zh_CN.pdf

was8.5_nd_qsg_en.htm  WAS8.5_nd_qsg_ja.pdf  WAS8.5_nd_qsg_zh_TW.pdf

WAS8.5_nd_qsg_en.pdf  WAS8.5_nd_qsg_ko.pdf

WAS8.5_nd_qsg_es.pdf  WAS8.5_nd_qsg_pl.pdf


qsg3/quickstart:

de/  en/  es/  fi/  fr/  it/  ja/  ko/  nl/  pt/  sv/  zh-cn/  zh-tw/
bill@san-domino:/dl/ibm.software/connections5$ ls qsg3/quickstart/en

pp_qsg.htm  pp_qsg.pdf



Again, this is just with the Quick Start Guides. Unpacking the other zip/tar'ed files does not put one at ease either. With a multitude of directories, incorrect documentation, and no clear starting reference, one either tries the Knowledge Center (which omits some details), keeps Googling, or contacts IBM tech support. Neither option is attractive for a business partner, let alone a customer simply trying to decide if they want to buy the product.


Installation Complexity Worsens with Each Release

Connections 1.0 became a starting point to a new strategic solution, so IBM had levity in lacking a simplified installation approach, as Connections was just basically five independent projects fused to create something better. Once IBM released version 2.0, that was the time to start creating installation efficiency, as a means to also accelerate adoption. In theory, with each successive product getting better (e.g. 2.0, 2.5, 3.0, 4.0, 4.5, 5.0), so would the installation. Several customers to which I communicated agreed on this facet.

Rather than improving the installation process or at least offering one, IBM chose to ignore this component altogether, instead solely adding more features and capabilities to Connections. Yes, any marketplace expects new items with successive releases, but you also want avoid alienating the respective customer's internal support team that can help drive their own adoption. Although conjecture, I suspect this abject complexity is part of the reason domestic adoption is slower than IBM would like -- despite having a decent product. Why? If customers can not install it, how are they going to test it? Not everyone wants to put their data into Greenhouse to gain a perspective.

Even now with Connections 5.0, the current product manager --- whom I know and respect very much --- does not see the value add of an admin UI for on-premises, instead Connections still relies upon a complex set of XML files. The compromise is to purchase a third-party administration tool. So after a customer spends tens of thousands of dollars on consulting, IBM licensing, hardware, and server OS licenses, they have to spend more money to get an admin tool from a non-IBM source. The XML approach is appropriate for a new product, but after version 3.0 an effective admin UI should become a core capability to help drive adoption. To be fair, IBM does offer an admin UI for their cloud offering, as it is a cloud specific architecture solution.

Looking at CCM for Connections 5 -- despite it being an add-on for the main product -- provides a suboptimal acquisition experience. One has to go to Fix Central and Passport Advantage to get all four files, and the required files list is omitted from system requirements, for starters.

Sametime used to be simple to install, then IBM decided to include the DB2 and Websphere brands into the Sametime ICS product offering. Version 8.5.1 took a long time to install; it's not much better now.

Once you install all these pieces -- for CCM, Connections, Sametime, et alia, you then have to try an patch them. This is hardly a trivial matter, with so many components the wrong lower level patch can cause stability problems. In contrast, if you patch a Domino server, it takes a minute or less; to remove the patch, just run the same patch program again. That is a great workflow design.


"The Customer Is Always Right," Remains Apropos As They are Paying

Customer Case #1:
My first ST 8.5.1 customer had me write a step-by-step, screenshot-by-screenshot document customized for their installation so that they could re-create what I did for them, in-case of a disaster or business continuity scenario. The final document equated 165 slides -- taking several days to write, plus editing time and customer input. The initial plan was for them to watch me install Sametime while capturing my actions, as they needed to know everything; that changed quickly.

The process required several escalated calls with tech support to get it all resolved. The process took far too long for my liking and began costing me money with other opportunities. Really difficult for a busy customer to dedicate the time to install it.

After tuning this customer's new ST staging environment, I provided a 1000% performance gain over their ST 7 Windows installation. Despite my success, the customer went with a different solution. Devastating to me. The customer called me to give me the decision and I enquired, "Why?" "Simply put, we do not have the resources to do the installation (of the full architected solution)." "I could do it for you." "You did Phase I. We need to own it now and we can not. It's dead."

Not only did I lose out on a lucrative Phase II consulting gig, but IBM lost out on licensing for a multi-national firm wanting every Sametime component offered (except A/V, which had issues with crossing subdomains at the time).

Customer Case #2:
A local BP learned of my aforementioned success and referred me to a senior-level colleague who wanted Sametime for their medical center. I co-authored a proposal for Sametime 8.5.1 and it went stale. I asked the BP, "What happened?" The customer loved what I did with the BP's friend's company, but management became baffled why such a significant percentage of the quote was dedicated to installation. They wanted to install it themselves to save money and then realized that would not work. Despite several calls to save the opportunity, the customer went in a new direction.


Simplified Installation Epitomized - IBM Domino

Now let's contrast the above issues to Domino...

Get the file from Passport Advantage, expand the tarball, execute the installer, and the installation routine handles all of the sub-program installations for you. Nice and easy.

bill@san-domino:/ibm.software/nd9/linux64/domino$ ls
860334080 Mar 21  2013 DOMINO_SERVER_9.0_LINUX_XS_64_EN.tar
145909760 Mar 21  2013 DOMI_SE_EMEX_AO_9.0_LIN_XS_64_EN_FW.tar


Expanding the first file provides a simple lucid starting point for the customer - "install"

eclipsemodssrc.zip  install  mozillamodssrc.zip  remote_script.dat  tools/  unix_response.dat



So why is IBM unwilling to take this great concept and apply it to other products
-- particularly those that are comprehensive? The sales mantra of "Just go to the cloud," is not a substitute for having your customers enjoy owning an installation medium they prefer.


Moving to the Cloud Is Not a Panacea

The on-ramp to the cloud should not be an escape from proper design. Lacking a quick fix to some negative feedback on installation complexity several years ago, as SmartCloud was coming to maturity as a solution, IBM's initial response lay dormant and they worked on their cloud first strategic initiative, rather than improving the installation processes in parallel -- in my opinion and based upon my experience. Once SmartCloud evolved as a platform as did its offerings, the phrase, "You can always use [Connections and Sametime] in the cloud," became the official complexity work-around--again in my opinion and based upon my experience. This also became the SMB solution for Connections and Sametime, which for reasons I mentioned is not always a solution for them -- particularly Swiss companies.

Customer Case #3:
I spoke with a valued customer last week. They made it clear that "The cloud" is not a strategic initiative for them--with any vendor. They love Sametime and have every component of ST 9 installed but one -- A/V. Why? "We got the SOW for the installation of that Sametime piece and it was a non-starter. We are not paying any more than we already have to install Sametime, plus we would need two more servers." Their owner wanted to perform video conferences and thought they had that capability with Sametime. To his dismay, they started the bidding process with a different vendor.

"The on-ramp to the cloud should not be an escape from proper design."


IBM's sales premise of, "If the process is too complex, just go to the cloud," took a tight customer relationship and opened the door to a slew of competitor bids. Having a wrapped installation procedure akin to Domino's approach would prevent competition from creeping into trusted customer areas that IBM previously owned. I would also offer that providing an appliance for Connections 5 and Sametime 9 would also go a long way to keeping customer relationships in the IBM camp. The ideal situation would be an appliance image sold on IBM iron, but with the divestiture to Lenovo, the best they could offer now is an image for on-disk or VMware. Know that in 2011, I made reference to how a lack of effective marketing weakens relationships in the same way.


Come to the Table with a Solution

I learned early in my career that if you complain in lieu of helping solve, you are being spiteful. In college, I started writing installation wrappers when I got tired of running successive make commands to compile my C programs 30+ times a session; my team liked my offering and we used it for the rest of the project. Thus, Domino's ease of installation attracted me to it as a solution. To keep things simple, there are two plausible options to address the installation complexity and keep customers content with business partners offering more services.
1. Offer an installation script that checks for the DB2 version, checks the WAS version, and then asks some questions and installs the products in the correct order with little fuss, all while displaying an accurate histogram
2. Provide the aforementioned appliance option. I know two colleagues of mine have offered to create their own build for Connections and Sametime, but the IBM licensing model prevents such a solution; as licensing adjustments are non-trivial in nature, this could take time to complete, but it should be considered and offered

IBM knows what products they require to have Connections and Sametime work. They can check for the existence of previous RDBMS (DB2) installation and flag an upgrade or issue, same for the Tivoli, Cognos, and FileNet components. Otherwise, install them as needed. This takes work. It is easier to tell people, "Go to the cloud." But if I learned anything in my 20+ years in IT, the easy solution is almost never the best solution.

In my genuine spirit of collaboration, if the respective product managers would like a more lucid vision of either of the above, I am happy to talk with them.



Conclusion

Lacking the above two solutions currently results in good community members spending days, nights, weekends learning to install a product --- usually on their time --- rather than committing that time to tuning, extending, customizing the product for our customers/our boss. This is a completely backwards model, in my opinion. Domino had it right in 1990. Why complicate to absurd levels your cornerstone social product to the point that most customers are unable to install it sans a consultant and your own technical writers are unable to capture accurate the process? The Domino model is hardly passe, regardless of your feelings on Domino. How many Zero to Hero sessions did Domino have to install it? None, that I'm aware. Connections and Sametime offer 200 page slide decks at previous Lotusphere/Connect events, and the current Sametime 9 deck is 950 slides (though part covers marketing and new features). Yes, I am glad these decks are available, but they should not be required.

An application is only as strong as its weakest feature. When customers are unable to install your product, they never discover how strong or weak a product is. In the end, when they learn that maintenance, upgrades, and general administration become more complex rather than easier with each successive release, you open the door to competitors with better ideas. Any OEM that outsources documentation to tech support or their BP community to complete and perform QA, installation fortitude to the customer, and general maintenance to their own consulting arm is destined for a wake-up call. I am grateful to IBM for all that their great products provided me and scores of my colleagues over the past two decades. I can only hope that IBM makes a earnest effort to regain one of their software hallmarks -- ease of installation.


Additional Reading

IBM Sametime 8.5.2 Administration Guide - Gabriella Davis, Marie L. Scott, and Thomas Duff
Installing the Sametime Gateway - Chris Miller

Powered by IBM Lotus Domino 8 | Lotus User Group | Get Firefox! | This blog is listed on Planet Lotus   IBM Certified

© 2010 William Malchisky.