Performance


z4OK, so yes, I did go a little mad the other weekend.  I’ve hankered for a roadster for many years and this little beauty presented itself and I took the leap. She’s garaged and only comes out at the weekends. However, that does not matter because she puts a smile on my face whilst she’s standing still as well as when I behind the wheel, feeding that 2.5 litre straight six and pushing her on through some sweeping bends.

Whilst modern she retains some glorious traditional lines, she looks fast when she’s standing still and when you open her up she’s as responsive as you like and she just flies.

OpenInsight 10 (OI10), for many of us has been a project that has seen the interface chance considerably. Whilst hugely functional, the old desktop interface was looking aged and not overly intuitive for new users. For experienced developers, the interface required a fair amount of clicking down through layers to achieve things or, at best you needed to know the shortcut keys. OI10 delivers a brand-new interface that is way more intuitive to use. There is no more clicking down through layers, so much more is right there in the interface or one click away and so, so much more has been exposed in the way of new controls, properties and more.

Whilst the O4W interface is still pretty new, it’s early releases were based around a two-column approach but OI10 opens up an array of new possibilities with multi-column design and drag and drop development.

I’ve played with the latter Alpha releases and I’m now getting more and more into the beta to convert my personally written contact manager that I use on a daily basis at RevSoft. I’m fast learning that OI10 all adds up to an easier to use interface with productivity gains to be found everywhere. I cannot believe how much code I can now remove from my forms by just setting one simple property in the Property Panel – that’s usually a case of inserting a single value (numeric or text), or making a picklist selection or toggling a property. OI10 is making our application developer easier than ever and introducing standards that will no doubt deliver better applications through consistency, stability and refined code.

So, the interface enhancements are nice, the O4W design options are more powerful but people still want better performance and on more than one occasion recently I’ve had discussions about indexing large files.

At conference last year, Bob spoke about the way that the conversion tools will optimise your tables. It is still work in progress but Revelation are mastering the dark art of balancing file-sizes with thresholds and a whole load of things that I really don’t understand. Bob’s also worked on caching things and using memory better and Andrew at Sprezzatura continues to explore ways to better configure the system for Linear Hash and find performance gains.

Some people don’t think that Revelation are taking performance seriously and listening to their customer base. I know for a fact that this is not the case. You only have to sit in the car with Mike on the way back from a User Group meeting to know that he personally takes customer needs and requested extremely seriously. On more than one occasion (in fact on many occasions) I’ve been driving him across the UK and he’s bashing away on his keyboard like it’s going to give up on him in the next ten minutes. We get to our destination and he shows me an example of something a client has suggested or requested and with a big smile on his face, he tells me that I can let my client know that it’s in the next release – subject to testing and quality control of course. It’s the little details like this that have kept me loyal to Revelation for the last 20 years, in a sales role that would normally have seen half a dozen sales people come and go.

Like the motorcar currently sitting in my garage just a few feet away from me, OpenInsight is maturing into one of IT’s classics which continues to deliver on the needs of the modern application developer. Not only does it look good and it’s wonderful to work with, hidden under the hood are a number of highly sought-after enhancements that are set to deliver some of those performance gains that the OpenInsight community have been asking for.

Just yesterday, after yet another call with a client looking at index performance on files with 500,000 plus rows, Andrew told me about some more of Bob’s enhancements to OpenInsight. Well, I just had to get some highlights from the man himself and, as a teaser, this is his reply:

“I have re-written index builds and updates. The high points are:

  • Rebuild uses in-memory hashtables and removes 64k workarounds which were in the legacy build.
  • Rebuild all for a table rebuilds all indexes in one pass, rather than individual passes
  • Update_Index is rewritten so that there is less contention on the root of the index. I made changes to SI.MFS as well.

…”

I don’t fully understand indexing but Bob tells me that the current system has to make numerous passes. One test that he undertook had to make six passes through a system with 500,000 rows. His greatly refined solution now makes just one pass through 500,000 rows, rather than having to work through 3,000,000 rows. He therefore has a very high level of confidence that the rebuild process work well and performance gains will be experienced across the board. I don’t have the figures, but he tells me that the 500,000 row rebuild was much faster and that’s good enough for me.

Other enhancements include a brand new update process that makes use of multiple sessions updating many tables at the same time. This has proven to be robust and fast during internal testing and we look forward to hearing the results obtained by our beta testers in the real world and running against real databases with hundreds of thousands or millions of rows.

I’m looking forward to getting out in the Z4 with Joanna, putting the convertible roof down and enjoying the wind in our hair. In the same way, I’m looking forward to working with the fresh looking OpenInsight toolset, modernising my applications and sharing this new gem of a toolset with the wider MultiValue community and the application development community in general.

We now have a fully integrated, highly functional toolset that is easy to use, powerful and fast. I can’t wait for the official OI10 release and to hear what Mike, Carl, Bob and the team have in the pipeline for OI11.

It’s going to be a great ride for the foreseeable future.

Advertisements

DBTA – MV Vendors Look to the Role of MV in the Future


1P7A8552Database Trends and Applications have just published a new article that looks at the role of MultiValue technologies in the future.

Author Stephanie Simone, talks to six of the key leaders in the MultiValue community to get their viewpoint on the future of our great technologies into the future.

Here is what Mike Ruane, President and CEO, Revelation Software had to say on the subject.

“At Revelation Software, we are incorporating features, capabilities, and integrations into OpenInsight (our MultiValue database development environment) to address the challenges of evolving data environments.

We are providing MV developers with a browser-based rapid application development tool that lets them generate responsive forms, reports, and dashboards, and we are supporting the “develop once and deploy to the desktop, tablet, or a mobile device” design philosophy.

With a development environment that tightly integrates with the Git source code management system, we offer an environment that allows for collaborative development teams as well as source code management.

Moreover, our database development environment seamlessly integrates with cloud databases. Building applications that utilize NoSQL cloud-based data storage allows us to support large numbers of concurrent users, deliver highly responsive experiences to a globally distributed base of users, provide high availability, and also handle semi-structured and unstructured data.

Enhancing usability, OpenInsight offers a development user interface within MV that is similar to what developers outside the MV world are accustomed to, provides a database environment that utilizes industry-standard AES encryption and policies, and enables end users who work with MV data stores to view their data in today’s BYOD world.”

Your can view the entire article on DBTA’s website here.

MultiValue Vendors Pushing Boundaries – DBTA


The 25th February 2015 edition of DBTA magazine featured an article titled ‘MultiValue Vendors Pushing Boundaries’ which features Revelation Software and Mike Ruane.

There are always new buzzwords coming along. But whether you call it “SMAC” or “CAMS,” there is no doubt that today the confluence of trends (analytics, cloud, social, and mobile) is proving to be a disruptive force that is causing many to reassess their approaches to data management.

Over the years, MultiValue technologies have evolved and adapted, pushing boundaries in order to integrate with new data sources and targets, address new analytics needs, and keep pace with emerging requirements. This has enabled customers to continue to rely on their trusted, and often highly specialized, MultiValue applications and data management systems.

Revelation’s website has a link to the full DBTA article and links ot the various speakers thoughts.

Securing data over the web within an O4W solution


There are three main areas of consideration when securing Linear Hash data over the web as part of an O4W based solution.  Those are the data that resides on the web server (or data server), the data whilst in transmission and then the data displayed within the browser.

Whilst these notes are written with an O4W solution in mind, it should be recognised that the provision for securing your data actually resides outside of the O4W system itself.

Liner Hash
It is worth mentioning that most hackers are familiar with relational data of the SQL type and understandably most people would prefer to work with data that they understand.  With MultiValue database driven solutions (like OpenInsight and O4W’s linear Hash) the data structure is usually unfamiliar to many hackers and therefore a lesser target in itself.

However, if you are running a high profile system, such as a bank with client financial details, a clinical system with patient data and the like, then you will still want to consider the security of your data and to do everything that you can to avoid those nightmare news headlines.

Data at rest
One of the first considerations that you will be faced with is where exactly should you house your data.  Locating it on the web server is usually fairly easy, it’s local, there are no networking (path) issues to consider and you can easily backup the whole system.  However, locating the data on the web server itself comprises a potential security risk in itself.

It is therefore recommended that a separate Data Server be used to locate the application’s data.  This server can then be hidden behind one or  more firewalls, so if your web server is compromised in anyway, the hacker will have more than one large hurdle to overcome.

Furthermore, with Revelation Software, you can utilise a Universal Driver between the web server and the data server and configure the system so that only access to the data is permitted through the Universal Driver.  Now that provides any external or internal hacker with yet another problem to overcome.

For those who need their data secure whilst it is residing on the data server, industry standard encryption tools can be used to encrypt the data to 128-bit encryption levels, for example.  A Google search for 128-bit encryption will provide you with plenty of information and solutions for  Advanced Encryption Standard (AES) and Data Encryption Standard (DES).

In addition, from OpenInsight 9.3, the toolset itself will support Data Encryption At Rest, making it even easier for Revelation developers to secure their data in single use, LAN, WAN and Web based solutions.

Data in transit
One of the weakest points before the data reaches the browser is during transmission.  Your precious data contained within one or more data packets is passed through countless networks, computers, hubs, etc. and at anytime these packets could be intercepted and interrogated.  If you have decrypted the data within the O4W application ready for it to be read by the user, then anyone could hijack it.

Fortunately, Hyper Text Transfer Protocol Secure (HTTPS) is your friend here.  HTTPS has been around for many years and it is well known as the industry standard for secure transmission of data over the web.  It has been supported since Internet Explorer 2 was around, so that really highlights that we are talking about a technology that is well tried and very well tested.

HTTPS is a secure version of the Hyper Text Transfer Protocol (HTTP) which allows for secure ecommerce transactions, such as online banking.  The technology effectively brings together HTTP and the SSL/TLS protocol to provide encrypted communication and secure identification of a network web server.

It is usually the System Administrator (or whoever has overall responsibility for the Web server) who configures the server to use HTTPS.  This is usually simply as case of acquiring a public key certificate for the web server from a trusted certificate authority.  This certificate must be signed by that authority for the web browser to accept and use the certificate.

It is also possible for the system to be configured for client authorisation.  Configuring a system in this way can limit access to a web server and thereby only permit authorised users.  This is achieved by the System Administrator creating a certificate for each user and this certificate is then loaded into the user’s browser during the session.  The certificate usually contains some identifying information (e.g. name and email address) and this is automatically verified by the server on each reconnect.

Web browsers such as Internet Explorer (IE) and Firefox display a padlock icon to indicate to the user that the website is secure.  In addition, the web address (URL) begins with https:\\, rather than http:\\.  When a user connects to a website via HTTPS, the site encrypts the session with a

digital certificate before any data is transmitted to the user’s browser.  O4W supports HTTPS, so transmitting your data using 128-bit encryption levels is super easy.  System Administrators can find a lot more information about the acquisition and use of public key certificates by running a Google search.

Data in the browser
So, we have our data encrypted at rest on our data server and we are using HTTPS to secure the data during transmission.  What about data in the browser?

Securing data in the browser is a tough one, mainly because the user needs to view the data in human readable format on the screen.  Or do they?

And herein lays the answer.

If the user needs to view the sensitive data in the browser, then you will have to revert to good old education.  Don’t leave your screen showing the data when you leave your desk, run your password protected screen saver before leaving your desk and, if you are near a customer facing position, make sure no unauthorised person can see your screen.

However, if the data does not need to be humanly read in the browser, why display it in the first place?  Leave the data on the data server and use some other form of key to interact with that sensitive data on the server, behind your firewall.  Many systems use this approach for user IDs.  The system maintains a cookie in the user’s browser and this contains some form of unique identifier – it could simply be a random number;  1234657684346846 (a session token of some kind).  It is this non-descript information that is then passed backwards and forwards over the web to identify the user.  When this key is received by your web application, you have code that takes the key, reads it and matches it to the user’s record.  Your program code can then use the sensitive login data to run the application on the web server as required.

To summarise
As an O4W web developer, you’ll want to concern yourself with protecting sensitive data within your specific application.  For example, credit card details; you will need to both protect a credit card field and audit who has access to the field and when it is accessed.  This is usually achieved by the developer including access controls to protect fields and the display of sensitive information.  As a conscientious developer, you may want to encrypt each record individually so that it can only be decrypted with a pin number that only the user knows and you’ll want to run a check to verify that user is legitimate.

This level of security in the application is good because it prevents an administrator from potentially stealing information from the system.  However, this does present new obstacles because the developer now has to code around the in ability to freely access all of the fields. How do you report on fields you can’t decrypt? What if the user forgets their pin, can the data be reset and re-encrypted or will the loss of data be irreversible?

The developer has to choose a balance in the application’s design between ease of access to data and the level of security. More security inevitably creates complexity so it’s important consider the risks and benefits of exposing the data.

The System Administrator should be responsible to ensure the underlying environment that the application runs on will be secure. If an attacker can gain administrator access in the operating system then the entire applications security protections could be rendered useless.  By protecting the environment (server, network, and client) at the operating system level the system administrator will be protecting the application.

Developers and administrators handle different levels of security that complement each other. The developer should be responsible for the application level access to the data while an administrator should be responsible for system level access.

But O4W helps . . .
Whilst you ‘could’ leave the question of data security up to someone else, effectively passing the problem on, there are ways that O4W can help.

From OpenInsight version 9.3, Revelation Software plan to introduce data encryption at rest.  This will be delivered through a new encryption service’ that system administrators can deploy to encrypt OpenInsight (OI) and O4W data on a field by field level.  The data will naturally be encrypted using industry standard encryption routines (DES, AES, TRIPLEDES, etc.).  This encryption service will use the Windows Communication Foundation (WCF) to communicate, thus ensuring that the data is also encrypted in transit.

In terms of protecting data in O4W ‘specifically’, O4W (in 9.2.1 and above) encrypts the user passwords when stored in OpenInsight, and never transmits any password information between the browser and the server.  When validating user information, O4W encrypts the user-entered password in the browser itself, and then returns only an impenetrable hash which the server then uses in its own calculations to determine if this is a valid logon.

In addition, the O4W programming paradigm is to ‘never’ send sensitive data to the browser if it can be helped.  Instead, temporary, unique records are created with the sensitive information, and stored on the server, and only the unique identifiers for these records are transmitted back and forth to the browser.

Of course, with the 9.3 release, the encryption service can apply encryption to any fields in the OpenInsight database, including those that are used by O4W, so by using https:// and the encryption service ‘any’ record and ‘any’ field can be protected all the way from “at rest” in the server,  through the engine server, through the web server, and to the browser.

My thanks to Bryan Shumsky (Revelation Software, Inc.) and Andrew McAuley (Sprezzatura) for their help and advice without which this posting would not have been completed.